Sensor data is irregular, noisy and arrives out of order. Six flow nodes turn it into something a model, a dashboard or a CMMS can use, with no JavaScript:
| Node | What it does |
|---|---|
| window | Aggregates a stream per key (machine, sensor) over tumbling, sliding or count windows: mean, RMS, kurtosis, percentiles, integral and 20 more. One record per window. |
| resample | Puts an irregular stream onto a fixed time grid (linear, previous or no fill, with a maximum gap). |
| rolling | Adds moving mean / std / min / max over the last N records or the last N seconds to every record. |
| outlier | z-score, IQR, MAD and Hampel tests over a rolling reference. Flag, drop, replace or route the odd values. |
| normalize | z-score, min-max or centring with fixed training statistics or running estimates. |
| deadband | Passes a record only when a value moved (report by exception) or a state changed. |
They work on both kinds of record, see the data contract:
- A machine record (it has a top-level
machineobject) stays a valid machine record. A field a node creates (rolling statistics, outlier flags, window aggregates) is written undermetadata, never at the top level, sofilter,branchand the sinks keep working. A window emits{source, timestamp = window end, machine, status, axes: [], metadata: {window, <field>: {...}}}. - Any other JSON (webhook, ingest, an
httpresponse) is read and written at the paths you give; a window emits a plain object{key, window, <field>: {...}}.
Paths are dot paths: metadata.vibration.x, axes.0.position, payload.sensors.temp. Numbers may be
JSON numbers or text with a decimal comma ("0,588399", "1.234,5", "1,234.5", the datalogger
convention); NaN, inf, empty text and null count as missing.
A typical vibration flow: source ingest:vibration -> resample (100 ms grid) -> outlier (Hampel, replace with
the median) -> normalize (the model’s training mean/std) -> window (10 s: RMS, kurtosis, peak)
-> filter metadata.x.rms > 3 -> sink push. The shipped example Vibration: clean and
summarise a sensor stream wires exactly this.
Keys, time and durations
Every node that remembers something does it per key. key is a ${path} template:
${machine.id}, ${machine.id}/${metadata.sensorId}. Left out, a machine record’s key is its
machine id and any other record shares one state. A record whose key cannot be built is dropped and
counted (on_missing_key: fail fails it instead).
Windowing and resampling use event time: the value at time (default timestamp, which a machine
record always has). It may be RFC 3339 text (2026-03-01T10:00:00.5Z), YYYY-MM-DD HH:MM:SS,
time of day HH:MM:SS.f (time_unit: hms) or epoch seconds, milliseconds, microseconds or
nanoseconds (time_unit: auto tells them apart by magnitude; set s, ms, us, ns or iso to
be explicit). A record without a readable time uses its arrival time (on_bad_time: arrival, the
default), or is dropped (drop) or failed (fail).
A duration is "500ms", "10s", "2m", "1h", "1d" (also us) or a plain number of seconds.
The per-node counters on the canvas gain a drop figure: records a data-preparation node dropped or
routed away on purpose (late, outlier, unusable, inside the deadband).
Window
{ "id": "w", "type": "window", "key": "${machine.id}",
"mode": "tumbling", "size": "10s",
"fields": [
"metadata.temp",
{ "path": "metadata.vibration.x", "as": "x",
"aggregates": ["rms", "peak_to_peak", "crest_factor", "kurtosis", "p95"] }
],
"aggregates": ["count", "mean", "min", "max"],
"out_of_order": "2s", "allowed_lateness": "0s", "late": "drop",
"idle_flush": "auto" }| Option | Meaning |
|---|---|
mode |
tumbling (default): fixed windows [start, start + size), epoch-aligned (offset shifts them). sliding: size long, one every every, overlapping. count: the last size records, one window every every records (every = size when left out). |
fields |
A path, or {path, as, aggregates}. The output name is as, else the last path segment (two fields must not end up with the same name). |
aggregates |
Aggregates of fields that do not list their own (default count, mean, min, max). |
emit_empty |
Also emit windows that received no record (count 0, other aggregates null), at most 1000 in a row. Default: empty windows are skipped. |
layout |
nested (default) {x: {rms: 1}} or flat {x_rms: 1} for SQL / CSV / ClickHouse sinks. |
max_records |
Most records buffered per key (default 10000). Beyond it the oldest are dropped and the window is flagged truncated. |
max_keys |
Most keys held (default 1000). The least recently used key is flushed (its open windows emitted partial) and removed. |
Aggregates
Computed from the window’s records, ordered by event time (ties in arrival order), two-pass, so there is no streaming round-off.
| Name | Meaning |
|---|---|
count |
numeric samples; missing counts the records without a usable number |
sum, min, max, mean |
as named |
var, std |
sample variance / deviation (divisor n-1; null below 2 samples) |
var_pop, std_pop |
population variance / deviation (divisor n). std_pop is the AC RMS (RMS with the mean removed) |
median, p50 … p99 (any pNN, 0 to 100) |
quantile by linear interpolation between order statistics, position (n-1)·p (numpy’s default) |
first, last |
by event time |
range, peak_to_peak |
max - min |
rms |
sqrt(mean(x²)) of the raw values (DC offset included) |
peak |
max(abs(x)) |
crest_factor |
peak / rms (null when rms is 0) |
kurtosis |
m4 / m2² with population moments: 3 for a normal distribution, the condition-monitoring convention. kurtosis_excess is kurtosis - 3 |
skewness |
m3 / m2^1.5 with population moments |
rate_of_change (rate) |
(last - first) / (t_last - t_first) per second |
integral |
trapezoid rule over the time-ordered samples, time in seconds |
For the sample 2, 4, 4, 4, 5, 5, 7, 9 one second apart: mean 5, var_pop 4, std_pop 2, var
32/7, median 4.5, p90 7.6, rms √29, peak 9, skewness 0.65625, kurtosis 2.78125, rate 1, integral 34.5.
Aggregates that are undefined (kurtosis of a constant signal, a rate from one sample) are null,
never NaN.
The output record
{ "source": "vibration-edge", "timestamp": "2026-03-01T10:00:10.000Z",
"machine": { "id": "pump-1" }, "status": { "state": "running" }, "axes": [],
"metadata": {
"key": "pump-1",
"window": { "start": "2026-03-01T10:00:00.000Z", "end": "2026-03-01T10:00:10.000Z",
"start_ms": 1772359200000, "end_ms": 1772359210000, "count": 10,
"mode": "tumbling", "size_ms": 10000 },
"temp": { "count": 10, "mean": 61.2, "min": 60.1, "max": 62.4 },
"x": { "rms": 0.41, "peak_to_peak": 1.9, "crest_factor": 3.1, "kurtosis": 2.9, "p95": 0.7 } } }window also carries every_ms (sliding), revision (a re-emitted window, see lateness),
partial: true (emitted before the window was complete: idle flush or key eviction),
truncated: true (the buffer cap dropped records of it) and suspect: true (see
State and restarts).
Watermark, out-of-order and late records
Per key the watermark is the newest event time seen minus out_of_order and only moves forward.
A window is emitted, once, when the watermark reaches its end. So a record up to out_of_order
behind the newest one still lands in its window; the price is that every window is emitted
out_of_order later.
A record is late when every window it belongs to is already closed.
allowed_lateness: "5s": a late record whose newest window ended less than that before the watermark is still added, and each closed window containing it is emitted again withrevision1, 2, … (downstream upserts onwindow.start). The buffer keeps the records needed for that.- Anything later is dropped and counted (
late: "drop", the default), or sent out of a second port namedlatewith_late: {window_end, watermark}added (late: "port"; wire thelateport, the deploy refuses an unwired port).
Count windows have no event-time lateness; they take records in arrival order.
Idle flush
Windows only close when a later record moves the watermark, so the last window of a sensor that
stopped would wait forever. idle_flush emits the open windows of a key that sent nothing for that
long (processing time), flagged partial: true:
"auto"(default): twicesize + out_of_order, at least 5 s; 30 s for count windows.- A duration, or
"off".
The flushed windows are closed: a straggler for them is then late. The flush runs in the connector’s series driver once a second and only on the active node of an HA pair.
Sliding and count windows
A sliding window size: "60s", every: "10s" emits every 10 s the last 60 s; each record belongs to
size / every windows (at most 1000) and the buffer is shared, so memory does not grow with the
overlap. Windows are on the epoch grid, so the first ones overlap the start of the stream: a first
record at 10:00:12 belongs to the six windows starting 09:59:20 to 10:00:10, and those hold few records
(window.count says how many).
A count window size: 100, every: 25 emits after every 25 records the last 100 (the first one
after 100). window.start / end are the event times of its first and last record.
Resample
{ "id": "rs", "type": "resample", "key": "${machine.id}", "interval": "100ms",
"fields": ["metadata.vibration.x", "metadata.vibration.y", "metadata.vibration.z"],
"fill": "linear", "max_gap": "2s", "align": "first" }An irregular stream (a datalogger that writes in bursts with 1-second timestamps) onto a uniform grid,
with the semantics of the vibration edge’s CSV replay (csv_source::resample): the grid is
anchor + k · interval, a grid time that equals a sample passes it unchanged, otherwise linear
interpolates between the two samples around it. A stream has no future, so the grid stops at the
newest sample and goes on when the next one arrives; one record is emitted per slot completed.
| Option | Meaning |
|---|---|
align |
epoch (default): grid times are multiples of interval (plus offset), so two sensors resampled separately share instants. first: the grid starts at the key’s first sample, as the edge does. |
fill |
linear (default); previous holds the last sample; none synthesises nothing: only slots hit exactly by a sample exist, every other slot is a gap. |
max_gap |
Where the two samples around a slot are further apart than this, the slot is a gap whatever the fill (default: unlimited). A gap is skipped, or with on_gap: "null" emitted with null values and _gap: true (on a machine record only metadata.* fields can be null; other slots are skipped). |
reorder |
Hold samples back this long and sort them first. Default 0: input must be in time order. A sample at or before the last one used (late or duplicate) is dropped and counted. |
max_slots |
Most grid slots one sample may produce (default 100000); a longer stretch counts as a gap, so a one-hour hole on a 1 ms grid cannot flood the flow. |
on_missing |
Every listed field must be numeric in a record for it to be a sample. skip (default) drops the record, fail fails it. Fill gaps first with a cleansing step if you want them interpolated. |
The output record is a copy of the source record at or before the slot, with the listed fields
replaced and the slot time written at time in the format it came in (RFC 3339 stays RFC 3339,
epoch milliseconds stay epoch milliseconds). timestamp-less records get the grid time at the
time path.
With reorder, idle_flush (default twice reorder, at least 5 s) releases the held samples of a key that went quiet.
Rolling
{ "id": "r", "type": "rolling", "key": "${machine.id}",
"fields": ["metadata.temp"], "records": 10,
"aggregates": ["mean", "std", "min", "max"], "into": "rolling" }Adds the statistics of the last records records (or the trailing duration of event time,
(t - duration, t]), the current record included, at into.<field>.<aggregate> (default into is
rolling; layout: "flat" writes rolling_temp_mean). Any aggregate of the table above works. The
record itself is unchanged and always passes. min_periods (default 1) nulls the statistics until the
window holds that many numeric samples; count and missing are always filled.
With duration, records may arrive out of order: each is computed against its own window, as long as it
is less than duration behind the newest record (the buffer keeps 2 · duration).
Outliers
{ "id": "o", "type": "outlier", "key": "${machine.id}",
"fields": ["metadata.temp"], "method": "hampel", "window": 30, "threshold": 3,
"action": "replace", "replace_with": "median" }Each value is judged against the last window values of its key and field (default 30), not
counting itself: a spike inside its own reference would inflate the very spread it is judged by (a
z-score in a window of n values can never exceed (n-1)/√n). Raw values enter the reference whatever the
verdict, so a level shift stops being an outlier once the window has caught up. With fewer than
min_samples (default 10) earlier values a record passes.
method |
Outlier when | Bounds |
|---|---|---|
zscore |
abs(x - mean) > threshold · std (sample std; threshold default 3) |
mean ± threshold·std |
iqr |
x < Q1 - k·IQR or x > Q3 + k·IQR (k default 1.5; quartiles as in p25 / p75) |
the fences |
mad |
modified z 0.6745 · abs(x - median) / MAD > threshold (default 3.5) |
median ± threshold·MAD/0.6745 |
hampel |
abs(x - median) > threshold · 1.4826 · MAD (default 3) |
median ± threshold·1.4826·MAD |
range |
x < min or x > max (fixed limits, either may be left out, no reference) |
the limits |
A flat reference (spread 0) makes every different value an outlier; min_scale is the smallest
spread, in the signal’s units, that counts. The robust methods (iqr, mad, hampel) are not fooled
by the outliers already sitting in the reference; the z-score is.
action |
Effect |
|---|---|
flag (default) |
The record passes with outlier: {flag, fields, scores} written at into (default outlier) on every record, so filter metadata.outlier.flag == true works. |
drop |
A record with an outlier is discarded (counted). |
replace |
The outlier value is overwritten: replace_with median (of the reference, default), limit (the violated bound), previous (the last accepted value) or null (metadata.* only on a machine record). outlier: {flag, fields, scores, replaced} is added, on records that had an outlier only. |
route |
A record with an outlier leaves through a second port named outlier (with the info above), the others through out. Wire both. |
A record is an outlier when any listed field is.
Normalise
Fixed parameters, for example the training statistics of a model. The vibration edge’s
models/inference_config.json stores norm_mean and norm_std per channel:
{ "id": "n", "type": "normalize", "method": "zscore",
"fields": ["metadata.vibration.x", "metadata.vibration.y", "metadata.vibration.z"],
"mean": [0.006168725434690714, 0.0084125567227602, 0.008863953873515129],
"std": [0.17158810794353485, 0.21506984531879425, 0.174563929438591] }mean, std, min, max are arrays aligned with fields (paste them from the training config), or
are set per field ({"path": "...", "mean": 0.1, "std": 0.2}). Running estimates per key instead:
"mode": "running".
| Option | Meaning |
|---|---|
method |
zscore (x - mean) / std; minmax lo + (x - min) / (max - min) · (hi - lo) into range (default [0, 1], clip: true clamps); center x - mean (AC coupling). |
mode: running |
Welford estimates per key (population std, as numpy’s default, which training pipelines store). alpha (0 to 1) makes them exponentially weighted so old data is forgotten; the current value is part of the estimate. Needs min_samples values (default 2); until then on_warmup drops the record (default) or passes it unscaled. |
as (per field) |
Write the result to another path and keep the original. |
on_missing |
A field that is missing or not numeric stays as it is (pass, default), drops the record or fails it. |
A zero spread gives 0 (z-score) or the low end of the range (min-max) instead of a division by zero. A
fixed z-score with std <= 0 is refused at deploy.
Deadband
{ "id": "db", "type": "deadband", "key": "${machine.id}",
"fields": [{ "path": "metadata.temp", "absolute": 0.5 },
{ "path": "metadata.load", "percent": 2 },
"status.state"],
"heartbeat": "60s" }A numeric field moved when it differs from the last value that was let through by at least absolute,
or percent of that value’s magnitude (either is enough, the same rule as the push path’s
preprocess.deadband; an unchanged value never passes). A field with neither threshold is change
detection: any difference counts, for numbers, text and booleans. Node-level absolute / percent
are the defaults of the fields that set none.
mode: any (default) passes the record when any field moved, all when every one did. The first record
of a key passes (emit_first). With heartbeat a record passes anyway after that long without one, so a
quiet but alive sensor is still seen. A passing record is unchanged and becomes the new reference of
every field; the others are dropped (counted).
State and restarts
Windows, rolling buffers, outlier references, running normalisation estimates, the resample grid and
the deadband reference live in memory per key, bounded: max_keys (default 1000, least recently
used evicted), max_records per key (window, rolling and resample reorder buffers). The state of a node
is kept across a redeploy as long as what it computes is unchanged (window size, fields, key, …);
changing that starts the node empty and the open windows of the old state are discarded.
With a data dir the state is persisted in series.db (persist: false keeps it in memory only):
- changed keys are written once a second; a graceful stop writes everything and marks the file clean;
- a graceful restart loses nothing: open windows continue, and a key that stayed quiet through the downtime is idle-flushed right after start;
- after a crash (kill -9, power loss) up to the last second of records is missing from the restored
state; windows that were open at the last write are emitted with
window.suspect: true(count can be lower than it should) and the log says so; - a node removed from the flow deletes its state at the first idle tick that finds it gone; a key unseen for 7 days is deleted;
- rows hold the buffered values (not the whole record, except
resample, which keeps the previous record as its template). They are not encrypted by the master key.
The window output is at-least-once: a crash between emitting a window and writing the state can emit it
again after the restart. Downstream, key on window.start + key.
Dry runs
Test runs (/api/flow/test, the editor’s Test) never touch live state: a throwaway state is used and a
single record seldom closes a window, so a window test shows the record buffered. Use the sandbox with
several records, or deploy and watch the node counters.