High availability

OpexFlow can run as an active-standby pair. Both nodes start from the same config; a lease in a shared store decides which one is active. Only the active node runs the processing stack (adapters, NATS ingest, schedule driver, cron scheduler, backfill). The other node is a standby: it serves the API and the UI read-only and watches the lease. If the active node dies, the standby takes over within the lease TTL.

HA is opt-in. Without an ha: section (or with enabled: false) a node is always active and nothing below applies: no lease, no middleware, no overhead.

How it works

1. Each node starts the API, the UI and the health endpoints, then competes for the lease.
2. The winner starts the full stack in-process and becomes active. The loser stays standby and retries every renew_secs.
3. The active node renews the lease every renew_secs. A crashed or partitioned leader stops renewing, the lease expires after lease_ttl_secs, and the standby acquires it and starts its own stack (it first re-reads flow.json, the event catalogue and the printer and runtime-source files from its data directory).
4. Fencing. A leader that has not confirmed a renewal for lease_ttl_secs - margin (margin = a quarter of the TTL, on a monotonic clock) demotes itself before the lease can expire for anybody else. A node that was frozen (VM pause, SIGSTOP) demotes the moment it wakes up and finds its deadline passed, without asking the store.
5. Demotion = exit. A demoted node stops polling, drains in-flight work like a normal shutdown, and exits non-zero. systemd or Docker restarts it and it comes back as a standby. In-process re-demotion (stopping and restarting all adapters live) is deliberately not done: a restart is the simplest way to be sure no task of the old role survives.

A graceful stop (SIGTERM, docker stop) behaves differently: the active node drains, then releases the lease, so the standby takes over immediately instead of waiting for the TTL.

Configuration

ha: is a top-level section, a sibling of nats: and storage::

ha:
  enabled: true
  node_id: "plant-a-1"     # default: the host name. MUST differ between the two nodes.
  backend: auto            # auto | nats | postgres
  lease_ttl_secs: 15
  renew_secs: 5            # default ttl / 3
  bucket: "opex_flow_ha"   # NATS KV bucket
  key: "leader"            # NATS KV key
  postgres_lock_key: 5715144172572004183   # Postgres advisory-lock key

opex-flow --check-config rejects an enabled ha: with no usable lease store, a TTL below 2 s, or a renewal interval that cannot keep the lease.

NATS KV backend

The lease is one key in a JetStream KV bucket whose max_age equals lease_ttl_secs, so the key of a dead leader disappears by itself. Acquiring is an atomic create; renewing is an update checked against the revision the node last wrote, and losing that check means leadership is lost. The bucket is created on first use. If it already exists its max_age must equal lease_ttl_secs (both nodes must use the same TTL); otherwise the node reports the mismatch and stays standby. For a NATS cluster pre-create the bucket with 3 replicas: nats kv add opex_flow_ha --history 1 --ttl 15s --replicas 3. The NATS server may expire the key slightly after the TTL, so budget lease_ttl_secs plus a second or two for a takeover.

The HA lease uses the connection settings of the nats: section (url, creds_file) but a connection of its own. The section must be enabled: true (and, as always, list at least one device).

Postgres backend

The active node holds a session-level advisory lock (pg_try_advisory_lock) on a dedicated connection; the server releases it when that session ends, for example when the process dies. The active node checks the session with SELECT 1 on every renewal. The standby retries every renew_secs and learns the leader’s name from the lock holder’s application_name (opex-flow-ha:<node_id>, visible when the database user may read pg_stat_activity).

A process crash releases the lock at once (the socket closes). A network partition does not: the server keeps the dead session until it notices, so lower the server’s tcp_keepalives_idle / tcp_keepalives_interval / tcp_keepalives_count (or set tcp_user_timeout) to the failover time you want. The old leader still fences itself at 3/4 of the TTL.

What a standby does

The active node answers /api/ready with 200 {"ready":true,"role":"active"} once its stack is up. While a node is promoted but still starting, it answers 503 with "role":"promoting". GET /api/status has an ha object with the full detail:

{
  "ha": {
    "enabled": true, "role": "active", "node_id": "plant-a-1", "leader": "plant-a-1",
    "backend": "nats", "serving": true, "since_ms": 1767225600000,
    "last_renew_ms": 1767225603000, "takeovers": 1, "demotions": 0,
    "lease_ttl_secs": 15, "renew_secs": 5, "last_error": null
  }
}

takeovers counts the times this node became active after seeing another node lead; demotions counts lost or unrenewed leases. Without HA the object is {"enabled":false}.

Alerts and metrics

A takeover raises a warning alert and a demotion a critical one through the configured alert channels (telemetry.alerts), and both are logged. With the telemetry feature, /metrics exposes opex_connector_ha{stat="active"} (1 on the active node) and opex_connector_ha{stat="takeovers"}. Alert on “no node reports active” and on a rising takeovers.

What moves between nodes, and what does not

The standby starts its stack only on promotion, so it reads whatever is in its own data directory at that moment. OpexFlow does not copy state between nodes; you decide how it gets there.

Practical consequences:

• Edit flows, printers, sources and event types on the active node (the standby refuses writes). Replicate flow.json, events/, printers.runtime.json, sources.runtime.json (and lookups.db) to the other node’s data_dir with rsync on a timer, syncthing, lsyncd, or a shared volume. A node that has never received a flow.json seeds one from its config, as on a fresh install.
• Single writer. Never point two running nodes at the same SQLite files. Both processes open push_queue.db and drain it, so a shared file would be delivered twice, and SQLite’s WAL mode is not safe over network filesystems. Keep every *.db local to its node.
• push_queue.db of a node that died is drained by that node after it restarts, not by the new active node. Records already in the dead node’s queue are delayed, not lost, as long as its disk survives.
• Sessions and named API tokens are node-local (auth.db). The legacy api_token and the OIDC provider work on both nodes; create named tokens on the node that will use them, or configure the same connector.api_token on both.
• A license seat is per device: activate the key on both nodes.

No message lost

HA moves processing between nodes; it does not make an unreliable source reliable. What survives a failover depends on how the data reaches OpexFlow:

For no-loss failover of machine data, feed OpexFlow through NATS JetStream (devices publish to the broker, the broker holds the stream while no node is active) and/or use retrying senders.

Deployment pattern

Two nodes, one shared NATS (clustered for production), a load balancer in front, and a health check on /api/ready:

   senders / UI ──► load balancer ──► node-a  :3000  (active, ready 200)
                    health: GET /api/ready  └► node-b  :3000  (standby, ready 503)
                                  ▲
        node-a, node-b ──── NATS JetStream (lease bucket + ingest streams)

The load balancer sends traffic only to the node that answers 200 on /api/ready. Keep /api/health for the container or process liveness check: it stays 200 on a standby.

docker-compose (two nodes, one NATS)

services:
  nats:
    image: nats:2.10
    command: ["-c", "/etc/nats/nats-server.conf"]
    volumes:
      - ./nats-server.conf:/etc/nats/nats-server.conf:ro
      - nats-data:/data

  flow-a: &flow
    image: opex-flow
    restart: always            # a demoted node exits non-zero and must come back as standby
    stop_grace_period: 45s     # the active node drains, then releases the lease
    depends_on: [nats]
    env_file: .env
    environment:
      OPEX_BIND_ADDR: 0.0.0.0
    command: ["/etc/opex-flow/config.yaml"]
    volumes:
      - ./config-a.yaml:/etc/opex-flow/config.yaml:ro   # same as config-b.yaml but ha.node_id
      - data-a:/data
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/api/health"]
      interval: 10s
    ports: ["3001:3000"]

  flow-b:
    <<: *flow
    volumes:
      - ./config-b.yaml:/etc/opex-flow/config.yaml:ro
      - data-b:/data
    ports: ["3002:3000"]

volumes: { nats-data: {}, data-a: {}, data-b: {} }

The image’s built-in HEALTHCHECK probes /api/ready, which is 503 on a healthy standby, so the compose file overrides it with /api/health. Both configs carry the same sections plus a different ha.node_id (flow-a, flow-b). The load balancer (nginx, HAProxy, a cloud LB) health-checks flow-a:3000/api/ready and flow-b:3000/api/ready.

systemd (two hosts)

Use the unit from Deployment. Two changes for HA:

[Service]
Restart=always            # a demoted node exits non-zero and must restart as a standby
RestartSec=5
StartLimitIntervalSec=0   # flapping leadership must not exhaust the start limit

Run rsync -a --delete-after active:/var/lib/opex-flow/{flow.json,events,printers.runtime.json,sources.runtime.json} standby:/var/lib/opex-flow/ (or a shared volume) on a timer so the standby has the latest flow when it is promoted.

Failover timing

Overlap guarantee: the leader stops acting at 3/4 of the TTL after its last confirmed renewal (measured on a monotonic clock from the start of that request), so, barring a clock or scheduler stall longer than the margin, two nodes are never active together. Work already accepted by a demoted node finishes during its drain (at most 30 s), so downstream effects are at-least-once, never silently dropped.

Limits

• A pair, not a cluster: one active node, one standby.
• Both nodes need access to every source and sink (same network reach).
• State replication is file-level and yours to provide (see above); there is no live replication of the SQLite databases.
• The lease is only as available as its store. Two nodes that both lose the store both demote.