Observability

The SLA / health dashboard (per flow and per external system), /metrics without the telemetry feature, SIEM export and the monthly service report are in every build: see Health, SLA, SIEM export and service reports.

Metrics, traces and the history series need a binary built with the telemetry cargo feature (cargo build --release --features telemetry), which is not in the default feature set; without it /metrics is empty and /api/obs/{traces,series,alerts} do not exist. Alert delivery and pipeline-failure alerts work in every build (see Alerts). /metrics and /api/obs/* need the API bearer token. The connector serves Prometheus metrics and keeps in-process traces + a SQLite history series, all viewable in the editor’s Metrics panel (“Telemetry disabled” appears there on builds without the feature).

• Metrics at /metrics: opex_connector_records_total, opex_connector_sink_total, opex_connector_push_duration_seconds, opex_connector_last_publish_ts, opex_connector_sink_stats{sink,stat} (per-sink published / dropped / backlog).
• Traces: one span per record through source → transform → sink, queryable via /api/obs/traces.
• History: sampled metric series in SQLite (/api/obs/series).
• Alerts: pipeline failures (every build) and sink-stall / error detection (telemetry); delivered to webhook, Telegram, email, Microsoft Teams and WhatsApp. See Alerts.
• Dashboards: the Metrics panel in the editor charts record rate, push and sink errors and per-sink idle time, and lists active alerts; point any external dashboard (Grafana, Datadog, etc.) at /metrics.

Alerts

telemetry.alerts configures where alerts go; any combination of channels may be set and every configured channel receives every alert (a failing channel never blocks the others).

telemetry:
  stall_threshold_secs: 300     # telemetry builds: sink silent this long -> alert
  sample_interval_secs: 15
  alerts:
    on_failure: true            # pipeline-failure alerts (default true; every build)
    window_secs: 300            # at most one alert per failing node per window (min 10)
    webhook_url: "https://hooks/ops"
    telegram: { bot_token: "${TG_TOKEN}", chat_id: "-100…" }
    email:
      smtp_host: smtp.example.com
      security: starttls        # starttls (587) | tls (465) | none (25)
      smtp_port: 587            # optional, defaults by security
      username: alerts@example.com
      password: "${SMTP_PASSWORD}"
      from: "Opex Flow <alerts@example.com>"
      to: [it-ops@example.com, integrations@example.com]
      subject_prefix: "[opex-flow]"   # optional
    teams:
      webhook_url: "${TEAMS_WEBHOOK_URL}"   # incoming webhook or Workflows URL
      format: card              # card (Adaptive Card, default) | text
    whatsapp:
      phone_number_id: "123456789012345"
      access_token: "${WHATSAPP_TOKEN}"
      to: ["6281234567890"]
      template: ops_alert       # an approved template
      language: en_US
      params: ["{title}", "{detail}"]   # fills {{1}}, {{2}} of the template body

The config is checked at startup (--check-config too): bad addresses, URLs, phone numbers or template placeholders refuse to start rather than fail when an alert is needed.

Pipeline-failure alerts

When a record ends in failure, the connector sends an alert naming the flow, the failing node, the error and the invocation id (open it with GET /api/pipeline/trace/{invocation_id} when the invocation was sampled, or find the record in GET /api/pipeline/dead-letters). Reported are: a node that fails the record (Fail, e.g. a transform with on_error: fail, an invalid record dead-lettered at a sink), a transient failure that will be retried (sink down, HTTP 5xx), a node timeout, an exceeded step or invocation limit, a lane panic, and a push record the cloud permanently rejected (dead letter). A transform error also alerts when its on_error policy (passthrough, the default, or drop) absorbs it, because a broken mapping must not go unnoticed; the alert says what happened to the record.

Rate limit. One alert per failing node per window (window_secs, default 5 min):

1. the first failure of a node alerts immediately, so an outage is known within seconds;
2. further failures of that node are counted, not sent;
3. when the window ends, one summary follows (“N more failure(s) in the last 5 min”, with the latest error) if any were counted; the next failure after a quiet window alerts at once.

Nodes are limited independently; across all nodes at most 30 alerts go out per minute (the rest are folded into their node’s next summary). So a flood of failures costs a handful of messages, and no failure waits longer than the window plus ~30 s to be reported.

POST /api/obs/test (bearer token, every build) sends a test alert to all configured channels and returns {ok, results:["teams: HTTP 202", …], channels:[{channel, ok, detail}]}. Secrets in URLs are never echoed in results or logs. The alert config is read at startup (POST /api/config/reload does not apply it): restart after changing a channel.

Stall and error alerts (telemetry builds)

The watcher raises stall (a sink silent past stall_threshold_secs), errors (push errors in the last 10 minutes) and pipeline (frames failed or dropped in the last 10 minutes) alerts, delivered through the same channels, re-notified every 20 minutes while active.