Messy plant and business data has two problems: you do not know yet what is wrong with it, and once you know, you want the flow to say so loudly instead of passing it on. OpexFlow has one tool for each:
| Tool | Question it answers |
|---|---|
Profile (editor, GET /api/flow/nodes/{id}/profile, MCP profile_node) |
What do the records at this node look like, field by field? |
quality node |
Are the values acceptable? Warn, park or drop the ones that are not, and count every broken rule. |
Profile a node
Open a deployed flow, select any node and open Profile in the inspector. The editor arms a bounded sampler on that node and refreshes the view every two seconds while it is open. Nothing is sampled for nodes nobody profiles: with the sampler off a record costs one atomic read.
What is sampled. The records the node receives (In) or emits (Out, the default; a sink defaults to In), after the privacy masker. Two buffer modes, up to 1000 records:
| Mode | Buffer | Use it for |
|---|---|---|
| Recent (default) | the newest size records |
what is flowing now |
| Sample | a uniform random sample of everything since arming | a long, even picture of a busy stream |
At most 8 nodes are sampled at once, one record is skipped when it is bigger than 64 KB, each node holds at most 4 MB, and a sampler nobody asked about for 10 minutes switches itself off and frees its memory. Nothing is stored on disk: a restart starts with every sampler off.
What you see, per field (nested objects are flattened to dot paths such as
metadata.raw.rpm; arrays are one value):
| Figure | Meaning |
|---|---|
| Type mix | integer, float, boolean, string, numeric_string (text such as 12,5 or 1.234,5 that reads as a number), timestamp, array, object. |
| Missing % | absent + null + blank text, as a share of the records. |
| Distinct | different scalar values in the buffer. |
| Numbers | min, max, mean, standard deviation (sample, n − 1), median, a 10-bin histogram and a sparkline of the values in arrival order. Number-like text is included and counted separately, so a column of "12,5" strings still shows its range. |
| Top values | the five most frequent values with their share (not for float columns, the histogram covers those). |
| Examples | up to three different values. |
| Timestamps | earliest and latest. |
Rows are badged when something looks wrong: missing values, mixed types, number-like text, a constant column, a column that is always missing.
Decimal-comma handling is the same in the profile and in the quality node: with both separators
the last one is the decimal mark (1.234,5 and 1,234.5 are 1234.5), a single comma is a decimal
comma (1,234 is 1.234), several of the same separator group thousands (1.234.567), and NaN
or inf are not numbers.
API and MCP
# arm the sampler and get the profile (waits up to wait_ms for the first record)
curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"phase":"out","mode":"recent","size":500,"wait_ms":5000}' \
http://127.0.0.1:3000/api/flow/nodes/clean/profile
# read it again (also keeps the sampler armed)
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/flow/nodes/clean/profile
# stop and free the buffer
curl -X DELETE -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/flow/nodes/clean/profileReading (GET) needs the viewer role like the debug socket; starting and stopping the sampler (POST, DELETE, and the MCP tool) needs operator. The data is masked the same way as in the debug feed. The
response lists fields[] with the figures above plus records (in the buffer), seen (offered
since arming) and armed. The MCP tool profile_node {node_id, phase?, mode?, size?, reset?, wait_ms?, stop?} does the same for an agent. An unknown node is 404, a ninth sampled node is 429.
The quality node
validate checks a record against the JSON Schema of a catalogue event type: the shape and
types of a published contract (business events). quality
is about the values: is the temperature plausible, is the timestamp fresh, did the counter go
backwards, is the batch id unique, did a machine stop reporting. Use validate at the edge where
data enters, quality wherever the numbers matter. They combine: validate first for shape, then
quality for value rules.
{
"id": "gate", "type": "quality",
"rules": [
{ "check": "not_null", "field": "machine.id", "severity": "fail" },
{ "name": "rpm_range", "check": "range", "field": "metadata.raw.rpm", "min": 0, "max": 6000 },
{ "check": "fresh", "field": "timestamp", "max_age": "10m", "max_future": "5s", "severity": "drop" },
{ "check": "monotonic", "field": "metadata.counter", "by": "machine.id" },
{ "check": "compare", "field": "end", "op": ">=", "other": "start" }
]
}Fields are dot paths into the record, a machine record (machine.id, status.state,
metadata.raw.rpm) or free JSON (order.total, lines.0.sku). The node only ever adds the
top-level _quality key, which filter, branch and the sinks ignore, so it works on both shapes
of record and never breaks the nodes after it.
Severity
severity |
A record that breaks the rule |
|---|---|
warn (default) |
continues, with _quality: {ok: false, issues: [{rule, check, field, severity, message}]}; a second quality node appends to the same list |
fail |
fails the node: parked in Dead letters with every failing rule’s message, or sent down the node’s error edge when one is wired |
drop |
discarded here, counted in the node’s drop figure |
Every rule is always evaluated, so one record reports all it breaks. fail wins over drop when a
record breaks both. A record that breaks nothing is passed on untouched.
Missing values
null, an absent field and blank text count as missing. Every rule except not_null skips a
missing value, so a range rule does not also complain about a field that a not_null rule already
covers. Add "required": true to make a missing value a violation of that rule. row_count has no
field and does not take required.
Rules
check |
Options | Passes when |
|---|---|---|
not_null |
field, allow_empty |
the value is present and not blank (allow_empty: true accepts blank text) |
type |
field, is |
is is string, number (a JSON number), integer (no fraction, 5.0 counts), boolean, object, array, timestamp (RFC 3339, YYYY-MM-DD[ T]HH:MM:SS, YYYY-MM-DD, epoch seconds or milliseconds) or numeric (a number or number-like text) |
range |
field, min, max (inclusive), gt, lt (exclusive) |
the number, or text like 12,5, is within the bounds |
regex |
field, pattern |
the text matches (unanchored: use ^…$); numbers match as their text |
allowed |
field, values |
the value equals one of values (numbers by value, text exactly) |
unique |
field, window (default 10m), by |
the value was not seen in the last window (per by value). Exactly one window apart is not a duplicate. |
fresh |
field (default timestamp), max_age, max_future |
the timestamp is at most max_age old and at most max_future ahead of the clock (each optional, one required) |
monotonic |
field, direction (increasing default, decreasing), strict, by |
the number or timestamp never goes the wrong way compared with the previous record (per by value); strict also refuses a repeat. After a reset the new level is the baseline: reported once. |
row_count |
min, max, window (default 1m), by |
records per tumbling window are within bounds. max flags the record that exceeds it. min is judged when the window closes, on the next record, so it can only warn (deploy error otherwise); silent windows between two records are reported with the next one. |
compare |
field, op (== != > >= < <=), other (a field) or value (a literal) |
numbers (or number text) compare as numbers, timestamps as time (zones honoured); text only supports == / != |
Durations are 500ms, 30s, 10m, 2h, 1d or a plain number of seconds. Every rule can have a
name (used in issues and stats); the default is {field}.{check} with #2, #3 for repeats.
A misspelt or irrelevant option (max on a regex rule) is a deploy error that names the rule.
Windows (unique, row_count) use the processing clock (when the node sees the record), so
late or out-of-order records do not disturb them; fresh judges the record’s own timestamp, so a
late record is judged on how old it is.
Counting, stats and Health
Violations are counted per rule: the node’s stats carry rules: {name: count} (shown next to each
rule in the editor, GET /api/status, MCP get_node_stats, the live socket). Health counts
records checked and records with at least one violation: GET /api/health/report shows
counters.quality_checked_total, counters.quality_violated_total and
data_quality.rule_violation_pct.
State and restarts
Unique values, the last monotonic value and the window counts live in memory, capped at 10 000
entries per rule (the oldest unique value is evicted first, a by value beyond the cap is not
tracked). A restart or a deploy that rebuilds the node empties them. That can only miss a duplicate
or a regression right after the restart, never invent one, and the first row_count window is never
judged because it may be partial. A node used by several sources keeps one state per source.
Example
plant-quality-gate in the Examples gallery: a machine feed with a not_null
that parks anonymous records, a range and an allowed-states rule that annotate, a monotonic
counter, and the sample record shows its _quality issues in a dry run.