Observability

The gateway gives you two observability surfaces: structured logs emitted by tracing and audit events persisted to the gateway_events table. Between them you can answer most "what happened when, and why" questions without instrumenting anything else.

What you do not get — yet — is a Prometheus scrape endpoint or an OTLP trace exporter. Both are on the roadmap; today the gateway is observable through logs and the audit table only. Plan accordingly.

Structured Logging

Logging is provided by tracing plus tracing-subscriber::EnvFilter. The filter directive comes from the RUST_LOG environment variable.

The gateway initializes the subscriber once at startup with the default fmt layer, which writes human-readable lines to stderr. Structured JSON output is not enabled in the current build — if you need machine-parsed logs, capture stderr through a log shipper that does the parsing (Vector, Fluent Bit, the OTel Collector's filelog receiver) until JSON output lands.

Filter recipes

Common RUST_LOG values:

Restart the process to apply a new filter — there is no live-reload endpoint. See Configuration for the precedence rules.

Notable log lines

The gateway emits a handful of high-signal lines every operator should recognize on sight.

Pool-timeout signal

When the database pool is exhausted, the gateway logs a line like:

ERROR sqlx::pool: database pool acquire timed out — request path starved

This means a request held a connection longer than the pool deadline (or there are simply not enough connections for the offered load). Two ways to react:

1. Raise pool capacity. The pool size is configured by sqlx's defaults; for sustained high load, scale the database tier (more replicas, larger instance, more connection headroom).
2. Scale the gateway horizontally. If you are already on Postgres, adding gateway replicas spreads the connection demand. SQLite cannot benefit from this — switch to Postgres before adding replicas.

If you see this line in steady state, do not paper over it with a longer timeout. It signals that the database is the bottleneck.

Audit Events

Every state-changing or invocation-shaped action persists a row to gateway_events with a typed payload. This is the authoritative log of what the gateway has done. The payload column is JSON; the event_type column is the variant tag below.

Querying the audit table

There is no REST endpoint for audit history yet — read directly from the database. The most useful starting query:

SELECT id, event_type, created_at, payload
  FROM gateway_events
  WHERE event_type = 'WorkflowInvocationFailed'
  ORDER BY created_at DESC
  LIMIT 50;

Other recipes:

-- All events for a single execution, in order
SELECT event_type, created_at, payload
  FROM gateway_events
  WHERE payload->>'execution_id' = 'exec-1234'   -- Postgres JSONB syntax
  ORDER BY created_at;

-- CLI rejections, last 24 hours
SELECT created_at, payload->>'tool_name' AS tool, payload->>'rejection_reason' AS reason
  FROM gateway_events
  WHERE event_type = 'CliToolSemanticRejected'
    AND created_at > NOW() - INTERVAL '1 day'
  ORDER BY created_at DESC;

-- Credential exchange failure rate, last hour
SELECT
    COUNT(*) FILTER (WHERE event_type = 'CredentialExchangeFailed')   AS failed,
    COUNT(*) FILTER (WHERE event_type = 'CredentialExchangeCompleted') AS ok
  FROM gateway_events
  WHERE created_at > NOW() - INTERVAL '1 hour';

For SQLite, replace payload->>'…' with json_extract(payload, '$.…') and the INTERVAL math with datetime('now', '-1 hour').

Wiring the audit feed to a SIEM

The simplest pipeline:

gateway_events  --(periodic SELECT … WHERE id > $cursor)-->  Vector / Logstash
                                                               |
                                                               v
                                                       Elasticsearch / Splunk

Track the largest id you have shipped, poll on a 10–30 second cadence, and forward new rows. The table is append-only; there are no in-place updates to reconcile.

For higher fidelity, run a logical replication slot (Postgres) or periodically .dump the audit table (SQLite) into your archive bucket. The goal is the same either way: get the audit trail off the gateway's primary database before it grows large enough to slow operator queries.

Honest Gaps

Next Steps

• Lock down log routing: Configuration — the precedence rules for RUST_LOG.
• Diagnose specific failure modes: Troubleshooting.