Plant edge to hosted OpexFlow: records survive an outage

Why the plant link needs a queue

A plant has machines that cannot stop and a WAN link that sometimes does. Records that go to a cloud system have to survive that gap, arrive in the order they were made, and not be counted twice when a sender is unsure whether its last attempt worked. The plant network also should not accept connections from outside.

The reference deployment solves this with one rule at each end. The edge writes every record to disk before it tries to send it, and only dials out. The hosted side verifies a signature on every request and drops a record it has already stored.

The flow

On the edge, every record from the devices goes to local history and to the push sink. The push sink stores the record in a SQLite queue on disk (push_queue.db) before the first attempt, then sends records in order as an HMAC-signed POST to the hosted hook. A restart of the edge keeps the queue.

push_endpoint: "${HOSTED_URL}/api/hooks/edge"
hmac_secret: "${EDGE_HMAC_SECRET}"
hmac_key_id: "plant-sg-01"
push_queue:
  enabled: true
  max_quota_mb: 2000
  on_full: evict_oldest   # use block for business records that must never be dropped

On the hosted side, a signed webhook source checks the signature, a dedupe node drops a record it has seen, a branch routes alarm records, and the sinks store the record to history. Alarms also go to the live feed. The hook answers after the flow has run, so a record only counts as delivered once it is stored.

{
  "match": "webhook:edge",
  "webhook": {
    "signature": {
      "scheme": "hmac_sha256",
      "header": "X-Opex-Signature",
      "timestamp_header": "X-Opex-Timestamp",
      "tolerance_secs": 300,
      "secret": "${EDGE_HMAC_SECRET}"
    },
    "respond": "on_done"
  }
}

1. The edge signs "{timestamp}.{body}" and sends X-Opex-Signature and X-Opex-Timestamp. Every retry is signed again with a fresh timestamp, so a record that waited out a long outage is not rejected as old.
2. The hosted source refuses a bad signature or a timestamp outside five minutes with 401, and counts it.
3. The dedupe node keys on the sender’s key id, the machine and the record timestamp, and remembers keys for two days.
4. A branch tags the record as alarm or normal, and the record is stored.

Tested cases

The integration tests run a durable edge pusher configured as in edge.yaml against the shipped hosted flow behind the real router, over real TCP on one machine.

The wrong-secret case is the one that matters in operation. A misconfigured edge does not lose data and does not get any into the flow. It fills its queue and says why, and fixing the secret drains it.

Watching the link

• On the edge, GET /api/push/status reports queued records, consecutive failures, the last error and the age of the oldest record. Alert on a growing age.
• On the hosted side, GET /api/status lists per webhook the records received and the bad signatures. GET /api/ready is the public health check used by the container.
• A queue has a size limit. On the edge the reference uses evict_oldest, which deletes the oldest telemetry at the limit and counts and alerts it. The hosted node uses block, which refuses new records instead of dropping any. Business records should use block at both ends.
• check-link.sh in deploy/hosted injects a record at the edge and waits for it on the hosted side.

Scope

• The tests run both ends in one process on loopback. They do not cross a real WAN or Cloudflare.
• The hosted node needs a VPS or VM with a persistent volume. A Cloudflare Worker cannot run the binary, and a Cloudflare Container has an ephemeral disk that would lose the databases on restart.
• The deployment guide puts a Cloudflare Tunnel in front of the node and a Bypass policy on the hook path alone. The hook has no bearer token, and the signature is its only authentication. A service token is the alternative.
• Disk use was measured on 2,008 records: about 0.43 KB per record in the history database. A 2,000-record loop took 24 seconds on a debug build. That is a smoke result, not a capacity figure.
• The arm64 image is not exercised.