Skip to content

Operational alerts

Applies to OML v0.6.x · Status: Stable · Last verified 2026-11-06

Use this page when an alert fires or oml doctor shows a failing line. Each section gives the steps, one command per step. Where OML has no command, the step says what you do instead.

Two rules hold throughout. Never edit a store by hand. Capture ids, digests and counts as evidence, never memory text.

Fork check failing

Use this when /readyz fails after a restart and writes return system.unavailable. The stored chain head did not verify against the last signed head. The server refuses writes rather than fork the chain. There is no override.

  1. Read the chain state.
oml doctor
  1. Verify the chain from genesis. The output names the last sequence that verifies.
oml audit verify-chain --genesis --signing-key <64-hex-public-key> --tenant <tenant>
  1. If someone restored Postgres, continue the restore steps instead of this page.
  2. If a Postgres failover came first, the standby that won was behind the signed head. Acknowledged writes are lost. Treat it as a recovery event: see disaster recovery.
  3. If neither happened, suspect corruption or tampering. Keep the output of steps 1 and 2 and follow security before any recovery act.

Do not force readiness, re-point at another Postgres or delete the chain-head row.

No reviewer can approve

Use this when writes to a workarea stay held and nobody can approve them. A memory needs a reviewer key (role dreamer) that did not write it.

  1. Check the workarea and its crew.
oml workareas show <workarea>
  1. Check the keys that review there.
oml keys show <reviewer-key>
  1. If there is no reviewer key with a grant on the workarea, issue one.
oml keys issue <key-name> --role dreamer --grant <workarea>:dreamer
  1. Confirm the reviewer sees the queue.
oml review list

An admin key cannot review. The server refuses it with access.role_exclusive.

Held memories are waiting too long

Use this when oml_review_wait_seconds climbs or agents report memories that never appear.

  1. List what the reviewer key may judge, oldest first.
oml review list
  1. If reviewers are available but outpaced, approve or reject in bulk. See review held memories.
  2. If the queue is mostly one source, deny that key from the whole tree. A denial wins over every grant.
oml deny set <key> universe
  1. If no reviewer can see the queue, go to no reviewer can approve.

The lake falls behind

Use this when oml_lake_lag_seconds grows without stopping. Nothing client-facing degrades. Analytics age with the lag.

  1. Read the lag and the failure counters on the writer's listener.
curl -s http://127.0.0.1:9466/metrics | grep -E 'oml_lake_(lag|commit_failures|dead_letters)'
  1. If the writer is not running, restart it. See the lake.
  2. If oml_lake_commit_failures_total grows, read the writer's log for the failing table and fix the object store or its credentials.
  3. If oml_lake_dead_letters_total grows, the writer skipped rows after its retry budget. Read the log lines for the failing event. Re-enqueueing a dead letter is not available. Fix the cause, then restart the writer.
  4. If batches complete but slowly, check oml_lake_maintenance_runs_total. Overdue compaction slows commits.

Lead takeovers repeat

Use this when writes fail with system.unavailable outside a deploy. Every takeover refuses writes until the new lead holds the lease and passes the fork check.

  1. Count the refused writes in the request log.
journalctl -u oml-server -o cat | jq -c 'select(.error_code=="system.unavailable")'
  1. Check Postgres health first. A failing lease heartbeat is the usual cause.
  2. If a rolling update is running, wait for it to finish. See upgrades.
  3. If two lead-eligible processes run, stop the extra one. They cannot fork the chain, only contend.
  4. While writes fail, do not restart the fleet. Each restart opens another refusal window.

Repeated 429 answers

Use this when clients get limit.rate (429) repeatedly. This build has two caps that answer it: unused OAuth client registrations (10,000) and recall feedback labels (1,000 a UTC day). It has no per-key or per-tenant rate limit.

  1. Find the route in the request log.
journalctl -u oml-server -o cat | jq -c 'select(.status==429) | {route, key_name, client_ip}'
  1. If the route is /oauth/register, the cap is on unused registrations; they lapse after 24 hours.
  2. If one key floods the server, deny it.
oml deny set <key> universe

Rotate a signing key or a keyring

Planned and emergency rotation are deployment steps. OML has no command for either. The server journals system.signing_key_rotated when a new signing key starts. See the event catalog.

  1. For a compromise, stop the lead first. Writes refuse everywhere.
  2. Rebuild the lead's host or pod from a trusted image.
  3. Rotate every credential the lead held: the Postgres role, the key-store grant and the object-store credentials.
  4. Replace the signing key at its custody point (the Kubernetes Secret or the mounted file) and start the lead.
  5. Verify the chain with the new public key.
oml audit verify-chain --genesis --signing-key <new-64-hex-public-key> --tenant <tenant>

Rotation does not heal the past. Anything the attacker signed still verifies under the old key. The rotation event and the last head you stored outside the deployment bound the span to review.

Revoke a key

OML has no revoke or rotate command. To stop a key, deny it, then issue a new one.

  1. Deny the key across the whole tree.
oml deny set <key> universe
  1. Issue the replacement.
oml keys issue <key-name> --role <role> --grant <workarea>:<role>
  1. Check the old key's denial.
oml keys show <old-key>