Data protection

OpexFlow keeps credentials out of flows and config, masks personal data wherever it is shown, and encrypts what it stores. Three independent pieces, one shared key:

Master key

OPEX_MASTER_KEY is the one secret the connector itself needs. It is any string of at least 16 characters; generate a strong one with:

opex-flow secret keygen        # prints 64 hex characters

The connector reads it, in this order, from the environment variable OPEX_MASTER_KEY, from the file named by OPEX_MASTER_KEY_FILE (Docker / Kubernetes secrets), or from the systemd credential OPEX_MASTER_KEY ($CREDENTIALS_DIRECTORY/OPEX_MASTER_KEY):

# /etc/systemd/system/opex-flow.service.d/key.conf
[Service]
LoadCredential=OPEX_MASTER_KEY:/etc/opex-flow/master.key
# or, bound to the host's TPM2 / machine id:
# LoadCredentialEncrypted=OPEX_MASTER_KEY:/etc/opex-flow/master.key.cred

It is never written to disk by the connector. Sub-keys for data at rest, the secrets file and dedupe keys are derived from it (HKDF-SHA256), so none of them can be used in place of another. Back the key up separately from the data directory. Without it the encrypted stores and secrets.enc cannot be read; the connector refuses to start on encrypted data without the key rather than silently ignoring it. A different key than the one the data was written with is refused the same way (the error names the key id, which is also logged at startup).

Key rotation is not automatic: to change the key, stop the connector, export what you need (dead letters you still want to replay, secrets via opex-flow secret list and your records), and start with the new key on a fresh data directory.

Secrets

A flow or config never holds a credential. It holds a reference, resolved when it is used:

${secret:erp_token}                    # local store first, then Vault (see secrets.order)
${secret:local:erp_token}              # the local encrypted store only
${secret:vault:erp/netsuite#token}     # Vault / OpenBao KV v2: path erp/netsuite, field token

A secret reference works wherever ${ENV} does:

• Config values: any string, resolved when the config loads (connector.hmac_secret, connector.api_token, sink passwords, SMTP credentials, …).
• http node auth: the secret fields (token, password, client_secret, consumer_*, token_*) take a single ${secret:name} instead of ${ENV}; a literal is still refused on Deploy.
• http node templates: url, headers and body.
• Signed webhooks: webhook.signature.secret, resolved at deploy.

In an http node, a secret must also be admitted by transform.executor.http_env_allowlist, which takes secret: entries beside environment names: secret:erp_token, secret:erp_* or secret:*. A secret nobody admitted is a deploy/run error, the same deny-by-default as environment variables.

Local store

<data_dir>/secrets.enc holds the secrets, encrypted with a sub-key of OPEX_MASTER_KEY (file mode 0600). Manage it with the CLI, or in the UI under Account menu, Secrets (admin):

printf %s "$TOKEN" | opex-flow secret set erp_token     # value from stdin, never from the command line
opex-flow secret list                                   # names and change times, never values
opex-flow secret rm erp_token

opex-flow secret ... works on OPEX_DATA_DIR, else on the directory of --config (default ./config.yaml); pass --data-dir to be explicit. A running connector notices a changed file within a second. Values are write-only: GET /api/secrets lists names, PUT /api/secrets/{name} with {"value": "..."} stores one, DELETE removes one (admin role, audited, request bodies never logged). Writing needs the master key (409 without it).

Vault / OpenBao

secrets:
  cache_ttl_secs: 300            # how long a Vault value is cached (0 = never)
  order: [local, vault]          # lookup order of a bare ${secret:name}
  vault:
    addr: https://vault.example.com:8200
    mount: secret                # KV v2 mount (default "secret")
    prefix: opex-flow/           # prepended to every path
    namespace: ops               # Enterprise namespace, optional
    # token auth ...
    token: ${VAULT_TOKEN}        # or token_file: /run/secrets/vault-token (Vault Agent sink)
    # ... or AppRole
    # role_id: 7f3a...
    # secret_id: ${VAULT_SECRET_ID}
    # approle_mount: approle
    ca_cert: /etc/opex-flow/vault-ca.pem   # private PKI, optional

The secrets: section itself accepts ${ENV} only (it configures the providers). A reference ${secret:vault:erp/netsuite#token} reads secret/data/opex-flow/erp/netsuite and returns its field token; without #field the field value is used, or the only field of the secret. An AppRole token is renewed at 80 % of its lease and once more on a 403.

What a secret never does

A resolved value is never logged and never returned by an API. Every value the connector resolves is remembered in a redaction set and scrubbed (***) from error texts, traces, dead letters, alerts, debug events, dry-run results and the audit log. GET /api/config shows the file as written (references, not values) with secret-looking keys redacted as before. GET /api/flow, exports and MCP drafts only ever contain the reference.

Masking personal data

Mask by field name in the config, and tag fields in the event catalogue:

privacy:
  mask:
    - "@pii"                # preset: *email*, *phone*, *mobile*, nik, ktp, *passport*, date_of_birth,
                            # full_name, *account_no*, address, password, ...
    - "@pii-values"         # preset: e-mail addresses, card-like and 16-digit numbers inside any string
    - customer.tax_id       # a key `tax_id` below a key `customer`, at any depth
    - "*voucher*"           # * wildcards, case-insensitive
    - "regex:ORD-\\d{8}"    # a value pattern, replaced inside any string

A name without a dot matches that key at any depth (arrays are transparent); a dotted name matches a path suffix. A matching value (object, array or scalar) becomes the string ***. There is no substring match unless you write the *: nik does not touch technician.

Event types mark personal fields in their JSON Schema with "x-sensitive": true:

{ "properties": { "email": { "type": "string", "format": "email", "x-sensitive": true } } }

The shipped types tag their personal fields (guest.updated: name, e-mail, phone, nationality, birth date; vendor.updated: tax id, e-mail, bank account number and holder; employee.updated: name, e-mail; feedback.received: comment). Inside an event envelope of that type, those fields are masked wherever the event is shown. The AsyncAPI export carries the tags: x-sensitive stays on the property, and every message has x-sensitive-fields (data.email, …) and a personal-data tag; GET /api/events returns sensitive_fields per type.

Masking is applied where data is shown:

The record a dead letter stores stays whole (encrypted at rest), so a replay still sends the real data. When an operator edits a dead letter, a field that still reads *** keeps its stored value, so fixing one field never overwrites a personal one with the mask; to really change a masked field, send a value other than ***. Dry runs started from a masked last-record carry ***: leave record out of POST /api/flow/test to run on the real last record and read the (masked) result. A change to privacy.mask applies on the next config load (POST /api/config/reload).

Not masked: the subject and correlation_id columns used to search the event log are stored and filterable as written (put identifiers there, not personal data), and GET /api/history returns machine telemetry (no personal data by design).

Encryption at rest

With OPEX_MASTER_KEY set, these columns are stored as enc1:<key id>:<base64> (AES-256-GCM, a random nonce per value, the table and column bound as associated data):

Rows written before the key was set are encrypted in place at the next start (the log line Encrypted N existing row(s) names each column), so enabling it needs no export or import. A database that already holds encrypted rows is not opened without the matching key.

Without a master key the stores stay plaintext and the connector logs a warning at every start:

WARN OPEX_MASTER_KEY is not set: dead letters, event history, the push queue, job logs and stored history
     are written UNENCRYPTED, and the local secrets file is unavailable. ...

What stays outside column encryption: indexed columns used for lookups (device id, timestamps, event type, subject, correlation id, source, dead-letter error text), reconcile.db (ledger keys and amounts), lookups.db, flow.json, the event catalogue files and license.json. None holds credentials (those are references) and the personal-data fields of events are encrypted in events.db, but a stolen disk would show them. Where that matters, combine this with full-disk encryption of the data directory (LUKS/dm-crypt on Linux, BitLocker on Windows, an encrypted volume or cloud-provider disk encryption for containers), which covers every file including SQLite WAL files and temporary pages. The two layers are independent: disk encryption protects a lost disk; column encryption also protects database copies, backups and anyone who can read the data directory but does not hold the key.

Back up with sqlite3 data.db ".backup out.db" as before: backups stay encrypted.

Encryption in transit: put TLS in front of the API (reverse proxy), use https:// endpoints, and postgres.sslmode: verify-full; see Security.