Troubleshoot the server

Applies to OML v0.6.x · Status: Stable · Last verified 2026-11-06

Run this first, with an admin or operator key:

oml doctor

It prints each check with ok, warn or fail. For client-side symptoms, see troubleshoot the client.

/readyz answers 503

/readyz names three checks. Any one that is not ok keeps it at 503.

• postgres: unreachable: the serving pool cannot reach the database. Fix the database or OML_POSTGRES__URL.
• fork_check: failed: the stored chain head did not verify. Writes return system.unavailable and reads continue. Do not force readiness. See fork check failing.
• names_assigned: unassigned: the start could not name a row that predates names. Every route except the two probes returns system.unavailable. Restart the lead after the upgrade finishes. See upgrades.

The lead exits with postgres.role_split

The lead checks its two database roles before and after migrating. It exits before binding a listener when the split is wrong, and logs role_split_refused with the reason.

Reason	Cause	Fix
owner_url_missing	postgres.owner_url is unset.	Set OML_POSTGRES__OWNER_URL to the owner role.
same_role	Both URLs log in as one role.	Create a separate owner role. See upgrades.
different_databases	The two logins see different databases.	Point both URLs at one database.
serving_role_privileged	The serving role is a superuser or holds CREATEDB, CREATEROLE, REPLICATION, BYPASSRLS, or can create in the database.	ALTER ROLE <role> NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS; then REVOKE CREATE ON DATABASE <database> FROM <role>;
serving_role_owns	The serving role owns the database or an object in it.	Move ownership to the owner role with REASSIGN OWNED.
serving_role_reads_migrations	The serving role holds a privilege on _sqlx_migrations.	Revoke it.
serving_role_ungranted	The serving role has no SELECT on journal.	Let the migrations run under the owner role, then restart.
table_unfenced, definer_function	A table lacks forced row-level security or a policy, or a function in public is SECURITY DEFINER.	No setting fixes these. Restore the schema from the migrations.

If the log says the check could not read the catalog, the owner URL could not read the catalog. Fix the owner role's access.

The lead stops during migrations

The lead runs migrations at boot under postgres.owner_url, then serves under postgres.url. Migrations only move forward, so rolling the binary back never rolls the schema back.

• postgres owner connect: …: the owner login failed. Fix the owner URL or password.
• migrate on boot: …: a migration failed. Read the message, fix the cause and restart. Restore from backup if the schema is half-applied. See restore.

The proxy's access log records credentials

With oauth.issuer set, the Location header of POST /oauth/authorize and the bodies of POST /oauth/authorize and POST /oauth/token carry codes and tokens. An access log that records them keeps live credentials.

1. Configure the proxy to omit that header and those bodies. See deployment.
2. Delete the log lines already written.

oml doctor cannot read a proxy's configuration, so it cannot check this.

Search model files are missing

The reranker needs config.json, tokenizer.json and model.safetensors in search.rerank.model_dir. The server checks each file against a pinned SHA-256 and stops at start on a mismatch. oml doctor warns when reranks are skipped.

Either place the files, or let the server download them once.

OML_SEARCH__RERANK__FETCH=true oml-server-lead

Set it back to false afterward. false never uses the network. See configuration.

The lake writer is behind or stopped

Check the writer's own health first.

oml-lakewriter doctor
curl -s http://127.0.0.1:9466/metrics | grep -E 'oml_lake_(lag|commit_failures|dead_letters)'

lake.writer fails when no writer holds the lease. Start it again. For lag that keeps growing, see the lake falls behind and the lake.

On Kubernetes

Install OML on Kubernetes describes the chart. Probes: startup checks /healthz every 2 seconds for up to 60 failures, which covers migrations; liveness checks /healthz every 10 seconds; readiness checks /readyz every 5 seconds and answers 503 until PostgreSQL is reachable and the startup fork check has passed. The timings are chosen, not measured.

You see	Cause	Do this
The pod waits in CreateContainerConfigError, naming the Secret	The Secret or its key is missing	Create it as in step 2 of the install. The pod starts on its own.
CrashLoopBackOff; the log ends fatal: postgres connect: ...	The URL in the Secret does not reach PostgreSQL	Fix the URL or the network path. With the NetworkPolicy on, check networkPolicy.postgres.
CrashLoopBackOff; the log ends fatal: keyring: keyring file permissions are not 0600 (kms-file shape)	Something changed the keyring file's mode, such as fsGroupChangePolicy set back to Always or a volume that maps modes	Restore OnRootMismatch, or move the claim to a volume that keeps file modes. Set the file back to 0600 from a pod that mounts the claim as uid 999. Never replace the file: it holds the signing key.
/readyz answers 503 with fork_check: failed; no token is printed	The server loaded a different signing key than the one that signed the chain	Reattach the claim that holds the deployment's key (runbooks). Never delete the chain head.
No enr_ line in the pod log	A tenant exists, so the deployment was claimed, or the fork check failed	A claimed deployment prints no token. Issue further keys with the admin key.

The chart deploys one server under Recreate: no read followers, no rolling update, no lake writer, no bundled PostgreSQL. The signing key sits on a volume and the enrollment token in the pod log. It has no oauth.issuer value and no resource numbers.