Operations runbook

Running the control plane in production. Configuration is entirely environment variables — .env.example in the repo is the canonical list, and the internal deployment guide covers the CI/CD and Railway path. This document explains the operational subsystems and how to tune them.

Identity

The CP has its own AITP identity (Ed25519), served at /.well-known/aitp-manifest and used to sign the revocation list.

• CP_AID_SEED_HEX — 32-byte hex seed. Required in production. Without it the seed is regenerated on every boot, so the CP's AID changes on restart and any peer that pinned the old key breaks. Generate once and store it as a secret:

  node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

• CP_BASE_URL — public URL embedded in the CP's own manifest. Set it to the externally reachable origin.
• The CP's own manifest has a 86400s (24h) TTL (MANIFEST_TTL_SECS) and is rebuilt in place once it comes within 3600s of expiry (MANIFEST_REBUILD_MARGIN_SECS), on the next call to getCpManifestJson() after that point — a long-lived process never serves a permanently expired manifest and does not need a periodic restart to stay fresh. These are hardcoded constants in src/lib/identity/cp-agent.ts, not environment variables — there is nothing to configure here. (This applies only to the CP's own self-published manifest at /.well-known/aitp-manifest; it has no bearing on agent-submitted manifests handled by POST /api/registry/enroll, a separate code path.)

Rotating CP_AID_SEED_HEX rotates the control-plane identity — treat it like a signing key, not a config toggle.

Authentication & exposure

• API_KEYS — comma-separated allowlist for gated routes. Required in production: empty API_KEYS in prod makes every gated route return 503 SERVER_MISCONFIGURED (fail-safe). Empty in non-prod disables auth and logs a one-time warning.
• ENROLLMENT_SECRET — server-side HMAC key for minting/verifying one-time enrollment tokens. Required. Callers never see it.
• CORS_ORIGIN — allowed browser origin (the UI console's origin). Set it to a single origin, e.g. https://console.example.com. Applied per-request at runtime by the proxy, so it can be changed via the deploy environment without rebuilding the image. Defaults to http://localhost:3000 if unset.

See api.md for the full auth matrix.

Verifying the request gate

Auth, rate limiting, CORS and x-request-id injection all live in one file (src/proxy.ts). Unit tests call its exported function directly, which proves the logic but cannot prove Next actually attached it — a gate file in a location Next does not recognise builds green, emits no warning, and leaves every /api/* route unauthenticated and unthrottled.

npm run verify:gate

Builds the app and boots it, then asserts the whole contract over HTTP: 15 checks covering rejection of unauthenticated requests, acceptance of valid keys, all three rate-limit buckets with their Retry-After / X-RateLimit-* headers, probe-path exemption, preflight handling, fail-closed behaviour when API_KEYS is unset in production, and that the gate is attached with an unchanged matcher.

Two properties make it worth more than a smoke test:

• It runs with a different CORS_ORIGIN than it built with, and asserts the served header matches the runtime value. Asserting mere presence would pass on an artifact that had frozen the value at build time.
• It enumerates the public route set from the built manifest, never from PUBLIC_PATHS, and diffs it against scripts/request-gate-baseline.json. Re-deriving the expectation from the code under test would be a tautology that reports green while the gate is open.

If a route's classification legitimately changes, review every line of the printed diff — each public entry is a route reachable with no credentials — then regenerate:

node scripts/verify-request-gate.mjs --build --update-baseline

CI runs this on every push; a non-zero exit fails the build.

Rate limiting

In-memory, per-process token buckets on every /api/* route except the probes (/api/health, /api/readyz, /api/metrics). Over-limit → 429 RATE_LIMITED with Retry-After and X-RateLimit-* headers.

• RATE_LIMIT_WINDOW_MS (default 60000) is the accumulation window.
• RATE_LIMIT_ENABLED=false disables the limiter entirely (dev / load tests).
• Set any individual limit to 0 to disable that bucket.

Buckets are per-process. Behind multiple replicas the effective limit is roughly N × limit. For a hard global limit, put a shared limiter at the edge.

Client-IP trust (important behind a proxy)

X-Forwarded-For is client-controllable, so per-IP buckets are spoofable unless you tell the CP which hop to trust:

• CLIENT_IP_HEADER — a single trusted header your edge sets to the real client IP (e.g. cf-connecting-ip, x-vercel-forwarded-for). Wins when set.
• TRUSTED_PROXY_HOPS — number of trusted proxies appending to XFF; the client IP is read this many entries from the right. Default 0 = XFF not trusted at all (leftmost is spoofable).

Misconfigure these and per-IP limits either bucket every request under one key or are trivially bypassed. Match them to your actual edge.

SSE capacity

GET /api/events/stream holds an in-process subscription per open stream.

• MAX_SSE_CONNECTIONS (default 500) caps concurrent streams per process; over the cap returns 503 SSE_CAPACITY. Clients should back off and retry.
• MAX_AUDIT_EVENTS_MEMORY (default 500) sizes the bus's total in-memory retention (older events are evicted and counted as dropped). Each new subscriber replays at most the last 100 events before going live, regardless of this setting.

If you front the CP with a fan-out proxy that opens its own upstream pool, raise MAX_SSE_CONNECTIONS accordingly.

Webhook delivery

Each delivery retries up to WEBHOOK_RETRY_ATTEMPTS (default 3) with exponential backoff. A per-endpoint circuit breaker sits in front of the retries:

• WEBHOOK_BREAKER_FAILURE_THRESHOLD (default 5) — consecutive failures before the breaker opens and deliveries to that endpoint are skipped.
• WEBHOOK_BREAKER_RESET_MS (default 60000) — how long the breaker stays open before a half-open probe is allowed.

Inspect or reset a breaker via GET /api/webhooks/:id/circuit-breaker and POST /api/webhooks/:id/circuit-breaker/reset (see api.md).

Data retention

A periodic sweep keeps storage bounded. It is multi-instance safe via a Postgres advisory lock (pg_try_advisory_xact_lock), so replicas don't duplicate work.

• RETENTION_ENABLED (default true) — master switch.
• RETENTION_INTERVAL_MS (default 1800000 / 30 min) — sweep cadence.
• RETENTION_BATCH_LIMIT (default 10000) — max rows deleted per sweep, so a sweep never locks a table for minutes.

What is swept (set any TTL to 0 to keep that table indefinitely):

Despite its name, EXPIRED_AGENT_GRACE_DAYS GCs operator-deregistered agents, not expired ones — expired rows are left in place so they can be re-enrolled. Authoritative records (revocation_entries, issued_tcts, delegations, trust_anchors, pinned_keys) are never swept.

Observability

• Metrics: GET /api/metrics exposes Prometheus text format (public, exempt from rate limiting).
• Logs: structured JSON via pino. LOG_LEVEL ∈ trace|debug|info|warn|error|fatal (default info). Every request/response carries x-request-id for correlation.
• Tracing (OpenTelemetry): off by default. Set OTEL_ENABLED=true to export spans to the OTLP HTTP endpoint at OTEL_EXPORTER_OTLP_ENDPOINT (path /v1/traces is appended unless OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set). OTEL_SERVICE_NAME defaults to aitp-control-plane. HTTP, pg, and fetch are auto-instrumented.

Health, readiness & graceful shutdown

• GET /api/health — liveness + DB ping. Stays 200 even while draining.
• GET /api/readyz — readiness (DB reachable, identity initialized).

On SIGTERM the process enters a drain window: /api/readyz flips to 503 { "ready": false, "reason": "shutting_down" } so a load balancer pulls the pod out of rotation, while /api/health stays 200 so the orchestrator doesn't hard-kill it mid-drain. Point your LB/orchestrator readiness probe at /api/readyz and the liveness probe at /api/health.

Database

• DATABASE_URL — Postgres connection string (required).
• DB_POOL_MAX (default 20) — connection pool size.
• Migrations run via npm run db:migrate from a checkout; the runtime image does not bundle drizzle-kit. See the internal deployment guide for the migration step against a hosted database.

Multi-tenancy

Namespaces (namespace column, X-Aitp-Namespace header, ?namespace= filter) are an opt-in scoping convention, not an enforced boundary. GET /api/registry/agents without ?namespace= returns rows across all tenants by design — registry discovery is an operational, non-normative layer in AITP, not a protocol-defined isolation boundary. If you need isolation, your callers must set the namespace on both discovery and enrollment; the CP enforces no implicit boundary.