Skip to content

Data profile and data quality

Look at what a node's records really contain (types, missing values, ranges, distributions) and enforce expectations about the values with the quality node: not null, type, range, regex, allowed values, unique, freshness, monotonic, row counts and cross-field rules.

Updated View as Markdown

Messy plant and business data has two problems: you do not know yet what is wrong with it, and once you know, you want the flow to say so loudly instead of passing it on. OpexFlow has one tool for each:

Tool Question it answers
Profile (editor, GET /api/flow/nodes/{id}/profile, MCP profile_node) What do the records at this node look like, field by field?
quality node Are the values acceptable? Warn, park or drop the ones that are not, and count every broken rule.

Profile a node

Open a deployed flow, select any node and open Profile in the inspector. The editor arms a bounded sampler on that node and refreshes the view every two seconds while it is open. Nothing is sampled for nodes nobody profiles: with the sampler off a record costs one atomic read.

What is sampled. The records the node receives (In) or emits (Out, the default; a sink defaults to In), after the privacy masker. Two buffer modes, up to 1000 records:

Mode Buffer Use it for
Recent (default) the newest size records what is flowing now
Sample a uniform random sample of everything since arming a long, even picture of a busy stream

At most 8 nodes are sampled at once, one record is skipped when it is bigger than 64 KB, each node holds at most 4 MB, and a sampler nobody asked about for 10 minutes switches itself off and frees its memory. Nothing is stored on disk: a restart starts with every sampler off.

What you see, per field (nested objects are flattened to dot paths such as metadata.raw.rpm; arrays are one value):

Figure Meaning
Type mix integer, float, boolean, string, numeric_string (text such as 12,5 or 1.234,5 that reads as a number), timestamp, array, object.
Missing % absent + null + blank text, as a share of the records.
Distinct different scalar values in the buffer.
Numbers min, max, mean, standard deviation (sample, n − 1), median, a 10-bin histogram and a sparkline of the values in arrival order. Number-like text is included and counted separately, so a column of "12,5" strings still shows its range.
Top values the five most frequent values with their share (not for float columns, the histogram covers those).
Examples up to three different values.
Timestamps earliest and latest.

Rows are badged when something looks wrong: missing values, mixed types, number-like text, a constant column, a column that is always missing.

Decimal-comma handling is the same in the profile and in the quality node: with both separators the last one is the decimal mark (1.234,5 and 1,234.5 are 1234.5), a single comma is a decimal comma (1,234 is 1.234), several of the same separator group thousands (1.234.567), and NaN or inf are not numbers.

API and MCP

# arm the sampler and get the profile (waits up to wait_ms for the first record)
curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"phase":"out","mode":"recent","size":500,"wait_ms":5000}' \
  http://127.0.0.1:3000/api/flow/nodes/clean/profile

# read it again (also keeps the sampler armed)
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/flow/nodes/clean/profile

# stop and free the buffer
curl -X DELETE -H "Authorization: Bearer $TOKEN" http://127.0.0.1:3000/api/flow/nodes/clean/profile

Reading (GET) needs the viewer role like the debug socket; starting and stopping the sampler (POST, DELETE, and the MCP tool) needs operator. The data is masked the same way as in the debug feed. The response lists fields[] with the figures above plus records (in the buffer), seen (offered since arming) and armed. The MCP tool profile_node {node_id, phase?, mode?, size?, reset?, wait_ms?, stop?} does the same for an agent. An unknown node is 404, a ninth sampled node is 429.

The quality node

validate checks a record against the JSON Schema of a catalogue event type: the shape and types of a published contract (business events). quality is about the values: is the temperature plausible, is the timestamp fresh, did the counter go backwards, is the batch id unique, did a machine stop reporting. Use validate at the edge where data enters, quality wherever the numbers matter. They combine: validate first for shape, then quality for value rules.

{
  "id": "gate", "type": "quality",
  "rules": [
    { "check": "not_null", "field": "machine.id", "severity": "fail" },
    { "name": "rpm_range", "check": "range", "field": "metadata.raw.rpm", "min": 0, "max": 6000 },
    { "check": "fresh", "field": "timestamp", "max_age": "10m", "max_future": "5s", "severity": "drop" },
    { "check": "monotonic", "field": "metadata.counter", "by": "machine.id" },
    { "check": "compare", "field": "end", "op": ">=", "other": "start" }
  ]
}

Fields are dot paths into the record, a machine record (machine.id, status.state, metadata.raw.rpm) or free JSON (order.total, lines.0.sku). The node only ever adds the top-level _quality key, which filter, branch and the sinks ignore, so it works on both shapes of record and never breaks the nodes after it.

Severity

severity A record that breaks the rule
warn (default) continues, with _quality: {ok: false, issues: [{rule, check, field, severity, message}]}; a second quality node appends to the same list
fail fails the node: parked in Dead letters with every failing rule’s message, or sent down the node’s error edge when one is wired
drop discarded here, counted in the node’s drop figure

Every rule is always evaluated, so one record reports all it breaks. fail wins over drop when a record breaks both. A record that breaks nothing is passed on untouched.

Missing values

null, an absent field and blank text count as missing. Every rule except not_null skips a missing value, so a range rule does not also complain about a field that a not_null rule already covers. Add "required": true to make a missing value a violation of that rule. row_count has no field and does not take required.

Rules

check Options Passes when
not_null field, allow_empty the value is present and not blank (allow_empty: true accepts blank text)
type field, is is is string, number (a JSON number), integer (no fraction, 5.0 counts), boolean, object, array, timestamp (RFC 3339, YYYY-MM-DD[ T]HH:MM:SS, YYYY-MM-DD, epoch seconds or milliseconds) or numeric (a number or number-like text)
range field, min, max (inclusive), gt, lt (exclusive) the number, or text like 12,5, is within the bounds
regex field, pattern the text matches (unanchored: use ^…$); numbers match as their text
allowed field, values the value equals one of values (numbers by value, text exactly)
unique field, window (default 10m), by the value was not seen in the last window (per by value). Exactly one window apart is not a duplicate.
fresh field (default timestamp), max_age, max_future the timestamp is at most max_age old and at most max_future ahead of the clock (each optional, one required)
monotonic field, direction (increasing default, decreasing), strict, by the number or timestamp never goes the wrong way compared with the previous record (per by value); strict also refuses a repeat. After a reset the new level is the baseline: reported once.
row_count min, max, window (default 1m), by records per tumbling window are within bounds. max flags the record that exceeds it. min is judged when the window closes, on the next record, so it can only warn (deploy error otherwise); silent windows between two records are reported with the next one.
compare field, op (== != > >= < <=), other (a field) or value (a literal) numbers (or number text) compare as numbers, timestamps as time (zones honoured); text only supports == / !=

Durations are 500ms, 30s, 10m, 2h, 1d or a plain number of seconds. Every rule can have a name (used in issues and stats); the default is {field}.{check} with #2, #3 for repeats. A misspelt or irrelevant option (max on a regex rule) is a deploy error that names the rule.

Windows (unique, row_count) use the processing clock (when the node sees the record), so late or out-of-order records do not disturb them; fresh judges the record’s own timestamp, so a late record is judged on how old it is.

Counting, stats and Health

Violations are counted per rule: the node’s stats carry rules: {name: count} (shown next to each rule in the editor, GET /api/status, MCP get_node_stats, the live socket). Health counts records checked and records with at least one violation: GET /api/health/report shows counters.quality_checked_total, counters.quality_violated_total and data_quality.rule_violation_pct.

State and restarts

Unique values, the last monotonic value and the window counts live in memory, capped at 10 000 entries per rule (the oldest unique value is evicted first, a by value beyond the cap is not tracked). A restart or a deploy that rebuilds the node empties them. That can only miss a duplicate or a regression right after the restart, never invent one, and the first row_count window is never judged because it may be partial. A node used by several sources keeps one state per source.

Example

plant-quality-gate in the Examples gallery: a machine feed with a not_null that parks anonymous records, a range and an allowed-states rule that annotate, a monotonic counter, and the sample record shows its _quality issues in a dry run.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close