Health, SLA, SIEM export and service reports

Everything on this page is in the default build (no telemetry feature needed) and is configured in one top-level insight: section. It answers four operational questions: is each flow and each external system healthy and within its service level, can our monitoring / SIEM see what the connector does, what was last month’s service, and which interfaces does the connector publish.

insight:
  metrics: { enabled: true }              # GET /metrics (Prometheus text), see "Prometheus"
  sla:
    enabled: true                         # default; records per-flow / per-system measurements
    minute_retention_hours: 48            # minute buckets
    hour_retention_days: 90               # hour buckets
    evaluate_secs: 60                     # how often SLA targets are checked
    window_minutes: 15                    # rolling window of a check
    min_requests: 5                       # fewer runs in the window are not judged
  siem:                                   # see "SIEM export"
    enabled: true
    sinks:
      - { kind: syslog, address: "siem.example.com:6514", protocol: tls, facility: local3 }
      - { kind: splunk_hec, url: "https://splunk.example.com:8088/services/collector/event", token: "${SPLUNK_HEC_TOKEN}", index: ops }
  report:                                 # see "Monthly service report"
    timezone: Asia/Jakarta
    schedule: { enabled: true, day_of_month: 1, hour: 6 }
    recipients: ["ops@example.com"]       # SMTP: insight.report.email, else telemetry.alerts.email

SLA dashboard

The executor measures two things and rolls them up in <data_dir>/insight.db:

• per flow: one run = one source record through its whole pipeline. Latency is the time from the start of the run to the end of its last step; the run is an error when a step failed for good or asked for a retry (retry, node timeout, invocation deadline). Steps routed down an error edge are handled by the flow and do not count.
• per external system: every run of a node that talks to one: http / soap (the URL’s host; mock for mock services), sql, email (SMTP host), sftp (host), printer (printer id or line X) and sink nodes (sink:push, sink:postgres, …). A URL that is a ${…} template falls back to the node’s label.

Per flow and per system you get throughput (runs / calls), errors, retries (transient failures that asked for a retry; they are also counted as errors), availability % (successful ÷ attempts) and p50 / p95 / p99 latency. Latencies are kept as mergeable histograms, minute buckets for 48 hours and hour buckets for 90 days (both configurable): percentiles of any range are computed from the merged histograms, never averaged (accuracy ±4.4 %). Open it in the editor under Operations → Health and SLA (menu: Health and SLA), with sparklines per row; or call GET /api/health/sla?range=1h|6h|24h|7d|30d|90d.

Service level per flow

Set the target on the flow tab: flow menu → Service level…, or sla on the flow in the document:

{ "flows": [ { "id": "pos", "name": "POS sync", "sla": { "latency_p95_ms": 800, "success_rate": 99.5 } } ] }

Every evaluate_secs the window is judged. A metric that is out of target opens an incident and raises one alert (through the alert channels of telemetry.alerts, and the SIEM alert stream); it stays open, silently, while the metric stays out of target and closes when a window with enough runs is back inside it. An idle flow neither breaches nor recovers. An invalid sla is a deploy error.

Prometheus

GET /metrics is served in every build when insight.metrics.enabled: true (with the telemetry feature it is always on and these series are appended to the telemetry ones). Like the rest of /metrics it needs no bearer token; keep the listener private or put it behind your proxy. Series (counters are since process start):

• opex_flow_runs_total{flow,name}, opex_flow_run_errors_total, opex_flow_run_retries_total, opex_flow_run_duration_seconds (histogram)
• opex_flow_system_calls_total{system,kind}, …_errors_total, …_retries_total, opex_flow_system_call_duration_seconds
• opex_flow_sla_breach{flow,metric} 1 while a flow is out of target
• opex_flow_siem_events_total{sink,kind,result=sent|failed|dropped}, opex_flow_siem_retries_total

SIEM export

insight.siem streams three sources as JSON lines to any number of sinks, batched and retried:

The line:

{"@timestamp":"2026-10-30T12:34:56.789Z","id":"19a3c…-4f","host":"edge-1","service":"opex-flow","stream":"audit",
 "severity":"info","severity_code":6,"message":"alice flow.deploy: ok","data":{"actor":"alice","action":"flow.deploy",…}}

Sinks (kind):

• syslog: RFC 5424 over tcp, tls (RFC 5425, ca_cert optional) or udp. <PRI>1 TIMESTAMP HOST APP PID MSGID [opex-flow@32473 id=… stream=…] {json}, PRI = facility × 8 + severity, MSGID audit / alert / log. TCP and TLS use octet counting (framing: newline for plain TCP); UDP sends one datagram per message (JSON cut at 8000 bytes). facility is a name (local0..local7, daemon, auth, …) or number. TLS needs the siem-tls feature (default on).
• splunk_hec: Splunk HTTP Event Collector, Authorization: Splunk <token>, index / source / sourcetype optional.
• elastic: Elasticsearch / OpenSearch _bulk with index, api_key or username / password; documents use the event id as _id, so a retried batch does not duplicate.
• webhook: any URL; the batch as NDJSON (default) or a JSON array (format: json_array), custom headers.

Tokens and passwords are ${ENV} references. sources: [audit, alerts] on a sink limits its streams.

Batching and retry. A batch is sent at batch.max_events (100), batch.max_bytes (1 MB) or batch.flush_secs (5 s). A transient failure (connection error, HTTP 5xx / 408 / 425 / 429, a Splunk “server busy” 503) is retried with a doubling back-off (retry.initial_ms 500 → max_ms 30000, max_attempts 6); a permanent one (other 4xx) is not. Each sink has its own bounded queue (queue_capacity, 10 000): when it is full the newest events are dropped and counted, so a slow SIEM never slows the connector. The queue is in memory; the audit log itself stays in audit.db. Status and counters: GET /api/siem/status (also in the Health view); POST /api/siem/test sends one event to every sink.

Monthly service report

GET /api/reports/monthly?month=YYYY-MM&format=json|html|csv&part=… (default: the current month, marked in progress): availability, volume (runs, errors, retries), the per-flow and per-system tables with percentiles and SLA met / missed, the SLA breaches of the month with their duration, the alerts raised, and the volume per day. html is one self-contained printable page; csv has one table per part (flows, systems, daily, breaches, incidents). Totals are the sum of the stored hour buckets of the month, so they match /api/health/sla?range=30d. Months are calendar months in insight.report.timezone. In the editor: Operations → Monthly report (preview, download, Mail now).

Scheduled mail. With report.schedule.enabled, the previous month’s report is mailed once, on day_of_month (1..28) at hour in timezone (a restart afterwards catches up; enabling the schedule mid-month does not mail history). It goes through SMTP from insight.report.email (same fields as telemetry.alerts.email) or, when unset, from telemetry.alerts.email, to report.recipients (default: that config’s to), as HTML plus the CSV tables. A failed send is retried every 15 minutes. POST /api/reports/monthly/send {month?} mails one now (operator, audited report.send).

Published OpenAPI

GET /api/openapi/published (and /api/openapi/published.yaml) is an OpenAPI 3.1 document of every HTTP interface of the connector, generated from the live configuration. In the editor: Operations → API docs, a lightweight viewer with search and schema trees. Contents:

• Webhooks: one POST /api/hooks/{name} per enabled webhook:{name} source, with its signature scheme in security / components.securitySchemes (header, algorithm, signed text, required DOKU headers; never the secret), the responses (200 / 202 / 400 / 401 / 404 / 413 / 422 / 503).
• Ingest keys: POST /api/ingest/{source}/{id} per source key (proline:* becomes /api/ingest/proline/{id}).
• App actions and data sources of every published app: POST /api/apps/{slug}/actions/{action} and …/data/{source} with the role each needs.
• Mock services: every endpoint with its canned responses.
• Management API: every route of the access table with x-required-role and x-audit-action.

Request schemas come from the event catalogue: when a validate node is reachable from a webhook / ingest source, the body is that event type’s JSON Schema (components.schemas.<event type>). An app action over a collection takes the collection’s schema; a flow action lists the ${input.x} fields its params read and names the schema of the record the flow validates (x-flow-record-schema). Otherwise the body is a free JSON object. Business events are described as AsyncAPI 3.0 at GET /api/events/asyncapi; together the two documents cover every interface.

REST and MCP