Case studiesReference integration: MES, CMMS and ERP on one event backbone

One event backbone for MES, CMMS and ERP on NATS JetStream

How OpexFlow sits on a NATS JetStream backbone between OpexMES, OpexMX and an ERP: every message is checked against its contract, accepted once, kept in Postgres and passed on, and a message that breaks the contract or arrives twice never reaches a system.

Why a backbone and not point-to-point calls

A plant with a shop-floor system, a maintenance system and an ERP ends up with a call from each to each. When OpexMX is down for a patch, OpexMES either blocks or drops the work order it just created. When the ERP resends a release, two work orders appear. When one team changes a field name, the other two find out in production.

A backbone changes three things. Producers publish a fact once, on a versioned subject, and do not know who listens. JetStream keeps the message until each consumer has acknowledged it, so a consumer that is down catches up instead of losing data. And every message has a written contract that any consumer can check before it touches the payload. OpexFlow is the consumer that carries those facts into Postgres and into the systems that need a call.

The contract

The subjects, streams and schemas come from the opex-contracts repository. Business events are wrapped in an EventEnvelope with a producer-generated eventId, an occurrence time, the producer and an optional correlation id. The payload is validated against the schema named in the envelope. Telemetry is flat and has no envelope. The three event subjects used here are low-volume.

Subjects used in the reference setup
Subject	Schema	Published by	OpexFlow does
opex.v1.event.workorder.released	workorder.released.v1	The ERP, through its connector	Keeps it in Postgres, records the ERP reference, tells the ERP once OpexMX holds the order
opex.v1.event.workorder.created	workorder.created.v1	OpexMES	Keeps it, creates the order in OpexMX, tells the ERP
opex.v1.event.alarm.raised	alarm.raised.v1	A connector, from a telemetry state change	Keeps it, opens a maintenance ticket in OpexMX
opex.v1.telemetry.<machine>	telemetry.v1	A plant gateway	Batches it into ClickHouse (see the telemetry case study)

The streams follow nats/streams.yaml of the contracts: OPEX_TELEMETRY (30 days, a 5 minute duplicate window) and OPEX_EVENTS (one year, a 24 hour duplicate window), both on file storage. The producer also sets the Nats-Msg-Id header to the eventId, so JetStream drops an exact resend inside the window before OpexFlow sees it.

A terminal showing nats stream ls with four streams, the two subject transforms of OPEX_INGEST, and nats consumer info for the OpexFlow consumer: pull mode, explicit acks, 14,336 maximum ack pending, 0 outstanding acks, 0 redelivered, 0 unprocessed. — The streams and the OpexFlow consumer after the test run: nothing pending, nothing unacknowledged.

The flows

OpexFlow has one JetStream source per subject and one flow per subject. The three event flows share the same intake. The work order created flow below adds the calls to OpexMX and the ERP.

Nodes in the work order created flow, in order

JetStream: workorder.created
validate: envelope
validate: payload
dedupe: eventId
sql: Postgres
http: OpexMX
http: ERP

Drop the silence markers. A JetStream source that has heard nothing for 60 seconds emits an offline record. The flow ignores it.
Validate the envelope against the EventEnvelope schema, then the payload against workorder.created.v1. Both schemas are registered in OpexFlow as event types, copied from opex-contracts.
Accept each eventId once, for 30 days.
Write the event and the work order to Postgres. Both statements are idempotent.
POST the work order to OpexMX with the eventId as Idempotency-Key and store the ticket number it returns.
Read the ERP reference of the work order. If the ERP released it, POST the shop-floor status to the ERP and store its acknowledgement.

The JetStream message is acknowledged only when the whole path has finished. A failure that a retry cannot fix, such as a contract violation, terminates the message and parks it in Dead letters. A failure that may pass, such as OpexMX answering 503, leaves the message unacknowledged so JetStream delivers it again. Every step before the failing one is safe to repeat, which is why the Postgres writes are upserts.

The work order created flow in the OpexFlow editor in two rows: a JetStream source, a filter, two validate nodes, a dedupe node, two Postgres nodes, a map, a POST to OpexMX, three more Postgres nodes, a released-by-ERP check and a POST of the status to the ERP. — The flow in the editor, cropped to the nodes. Under the dedupe node the counter shows one duplicate dropped. The host address in the last error under the OpexMX call is blurred.

Tested cases

The flows were run with messages published by small scripts that play OpexMES, the ERP connector and a plant gateway. Every message was checked against the opex-contracts JSON Schemas before it was published, except the one that is meant to break them. OpexMX and the ERP are stand-in services served by OpexFlow itself.

Test cases and results
Case	What happened	Result
A work order created by OpexMES	Stored in Postgres, created in OpexMX (201), status sent to the ERP when it had released the order	OpexMX ticket MXWO-WO-1001 and ERP acknowledgement ERPACK-4000012345 on the work order row
A message that breaks the contract	Type “repair” and priority “urgent” are not in the enums, createdAt is missing	Parked in Dead letters at the payload check, nothing in Postgres or OpexMX, acknowledged as terminated: 0 redelivered
The same eventId sent again	The first resend carried the same Nats-Msg-Id and was dropped by JetStream. A second resend without the header reached OpexFlow	The dedupe node dropped it: one row in Postgres and one call to OpexMX
OpexMX answers 503	Three work orders created while every call was refused: 12 refused calls in 20 seconds, the messages redelivered with a growing delay	After OpexMX returned, all three were created (3 times 201) and had their ticket. No second row in Postgres
Released after created	MES created the order six seconds before the ERP release arrived	The ERP status call went out when the release was recorded: ERPACK-4000012346
An alarm	alarm.raised.v1 from a connector	Alarm row and OpexMX ticket MXT-AL-2001

The contract violation is the failure that is easy to miss on a bus: JetStream accepted the message, because it does not read payloads. It is the consumer that has to refuse it. Here the message did not vanish. It sits in Dead letters with the record as it arrived and every rule it broke, and the JetStream consumer shows nothing pending and nothing redelivered.

The Dead letters tab with one open entry from nats:wo-created at the node wo-pay. The error reads: does not match event type workorder.created, createdAt is a required property, priority urgent is not one of low, medium or 2 other candidates, type repair is not one of preventive, corrective or 3 other candidates. — The invalid work order in Dead letters, with the reasons and the record, ready to correct and replay.

The call log of the stand-in services shows the OpexMX outage: refused calls first, then one successful call per work order once the service answered again.

The Mock services panel with the services erp (2 calls) and opexmx (20 calls) and a list of recorded calls: POST /api/workorders 201 several times, POST /api/tickets 201, POST /erp/workorders/4000012346/status 200 and earlier POST /api/workorders answering 503 three times at the same second. — Recorded calls of the stand-in OpexMX and ERP. The 503 rows at the bottom are the outage; the 201 rows above are the retries after OpexMX returned.

One work order, end to end

The ERP released work order WO-1001. The simulated OpexMES consumed the release from JetStream and answered with a created event, the way a real MES would. The times are when the events were stored in Postgres and when the stand-in services were called. The MES simulation waits one second, and OpexFlow drains the JetStream source every second.

Trace of WO-1001 across the three systems
Time	Where	What
11:01:15.237	JetStream to Postgres	workorder.released.v1 from erp:sap stored; ERP reference 4000012345 recorded
11:01:16.236	JetStream to Postgres	workorder.created.v1 from mes stored
11:01:16.244	OpexFlow to OpexMX	POST /api/workorders answered 201, ticket MXWO-WO-1001
11:01:16.250	OpexFlow to the ERP	POST /erp/workorders/4000012345/status answered 200, ERPACK-4000012345

Everything hangs on the correlation id, which is the work order number. The bus_events table in Postgres returns the events of one work order in order, whichever system produced them.

Health and alerts

The Health and SLA tab lists every external call. Over the whole test run, which includes the ClickHouse outage of the telemetry case study, the call to OpexMX shows 19 calls, 12 errors and 12 retries. The 12 are the refused calls above, and 7 calls succeeded. The Postgres statements ran 63 times without an error.

The Health and SLA tab: 30,941 runs, 892 errors, four flows listed, and an external systems table with clickhouse, postgres, POST to OpexMX with 19 calls and 12 errors, POST status to ERP with 2 calls and POST ticket to OpexMX with 1 call. — Health and SLA for the test run: flows on top, external calls below.

Scope

OpexMES, OpexMX and the ERP are simulated: scripts publish contract-valid messages, and the OpexMX and ERP endpoints are stand-in services. No real MES, CMMS or ERP was involved.
OpexFlow consumes from JetStream and calls systems over HTTP. It does not publish business events onto JetStream: its NATS sink carries machine telemetry only. The ERP release, the MES creation and the alarm are published by the producers, and a rejected message is parked in the Dead letters of OpexFlow, not on an opex.v1.dlq subject.
OpexFlow 2.2.0 reads device subjects under opexmx.ingest.* from one ingest stream. The contract subjects reach it through a stream that sources OPEX_TELEMETRY and OPEX_EVENTS and renames their subjects. The contract streams and subjects are untouched.
The ERP status call is sent when both the ERP release and the OpexMX work order exist, whichever arrives last. If both are processed at the same moment the ERP may get the status twice, which an idempotent status endpoint accepts.
Authentication and TLS of NATS, OpexMX and the ERP, the real ERP interface and the field mapping are set per installation.