---
title: Troubleshooting
---

Each entry gives a symptom, the check that confirms its cause, and the fix. To follow
one request through the ledger, query log and traces, see
[debug a request](/docs/pt-BR/guides/debugging).

## Start here

Three commands answer most questions before you read any log:

```bash
curl -s http://localhost:7777/v1/health | jq '{status, ch_ok, ch_error, schema, problems}'
docker exec loom-server loom doctor              # the same problems, plus pre-boot checks
curl -s -XPOST http://localhost:7777/v1/ops.health -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H 'Content-Type: application/json' -d '{"org":"<org>"}' | jq   # ClickHouse parts, workers, stubs (ORG_ADMIN)
```

Every entry in `problems` carries a `fix`; read it first. On a client machine,
`loom doctor --client` checks the server URL and token the SDK and hooks resolve. What
each field means is in [monitoring](/docs/pt-BR/operations/monitoring#health-checks).

## Server and ClickHouse

### Every operation fails and /v1/health returns 503

- **Cause:** ClickHouse is unreachable from the server. The boot also stops at the first
  migration.
- **Check:** `curl -s localhost:7777/v1/health | jq .ch_error`, then
  `docker exec loom-server loom doctor` (the `clickhouse_reachable` line).
- **Fix:** correct `LOOM_CH_HOST` / `LOOM_CH_PORT` / `LOOM_CH_USER` / `LOOM_CH_PASSWORD`.
  Port 8443 implies TLS; set `LOOM_CH_SECURE=true` for TLS on any other port. On
  ClickHouse Cloud, check that the service is running (idle services pause) and that its
  IP allow-list admits the Loom host.

### loom up times out, or the container exits at boot

- **Cause:** usually a migration or ClickHouse error, a malformed `.env` value, or a
  process outside the compose files without `LOOM_INSTANCE_LABEL`.
- **Check:** `docker logs loom-server`; the failing migration file or setting is named
  there. Validate the env file with
  `docker compose --env-file .env -f dev/docker-compose.cloud.yml config >/dev/null`
  (use the compose file you boot).
- **Fix:** the error in the log. A deployment (`LOOM_DEPLOYMENT=1`) with auth on and the
  checked-in dev `LOOM_JWT_SECRET` refuses to start by design: set a real secret
  ([deployment](/docs/pt-BR/operations/deployment#required-environment)).

### Writes fail with TOO_MANY_PARTS, or merges are failing

- **Symptom:** writes fail with ClickHouse error 252 `TOO_MANY_PARTS`, or `ops.health`
  reports `clickhouse_parts: fail`.
- **Check:** whether merges are failing, which is almost always the cause, not the write
  rate. Read the merge errors in `system.part_log` first:

  ```sql
  SELECT table, error, count() AS failures, any(exception)
  FROM system.part_log
  WHERE database = 'loom' AND event_type = 'MergeParts' AND error != 0
    AND event_time > now() - INTERVAL 1 HOUR
  GROUP BY table, error;
  ```

- **Fix:** if the exception names a skip index the server cannot merge, exclude that
  index from merge-time materialization. The setting is metadata-only and reversible;
  inserted parts still build the index:

  ```sql
  ALTER TABLE loom.<table> MODIFY SETTING
    exclude_materialize_skip_indexes_on_merge = '<existing list>,<index name>';
  ```

  Part counts fall within minutes once merges succeed. Do not drop the index first
  (rebuilding it later rewrites the table), and do not only raise
  `parts_to_throw_insert`: with merges stopped the count keeps growing. If no merge is
  failing, look for a writer inserting one row per statement (`system.query_log` grouped
  by `log_comment`, [query attribution](/docs/pt-BR/operations/monitoring#logs-traces-and-query-attribution)).

### Writes take seconds after a ClickHouse Cloud replica restart

- **Symptom:** write-heavy operations (`memory.set`, `run.*`, `conversation.append`)
  take seconds while reads stay fast, with no errors, often right after a deploy that is
  not the cause.
- **Check:** replica uptime, and system-log re-attachment in progress:

  ```sql
  SELECT hostName(), uptime() FROM clusterAllReplicas(default, system.one);
  SELECT count() FROM system.query_log
  WHERE event_time > now() - INTERVAL 15 MINUTE AND query ILIKE 'ATTACH TABLE system.%';
  ```

- **Fix:** none needed on the Loom side. After a restart the service re-attaches its
  system tables for a while, and inserts recover as that finishes (typically within an
  hour). Do not roll back a deploy because of it; contact ClickHouse Cloud support if it
  persists.

### Requests fail with BACKEND_TIMEOUT or OVERLOADED

- **Cause:** a ClickHouse statement ran past its class deadline (504
  `BACKEND_TIMEOUT`), or every pooled ClickHouse connection of the process stayed busy
  (503 `OVERLOADED`). Under a sustained write rate, batched inserts (audit rows,
  transcript turns, the agent_events flush) hold their connection for up to
  `LOOM_CH_BATCHED_INSERT_WAIT_MAX_MS` and then the flush, and background ones (the
  usage-event and write-mark flushes) for up to
  `LOOM_CH_BATCHED_INSERT_BACKGROUND_WAIT_MAX_MS` and the flush
  ([batched inserts](/docs/pt-BR/architecture/clickhouse#batched-inserts)).
  `OVERLOADED` can also mean a statement found ClickHouse short of the server's or the
  ClickHouse user's memory (code 241, shared by every query; the server log has
  `ClickHouse is short of memory` with ClickHouse's line, once per 5 s).
  From `conversation.delete` or `conversation.compact`, `OVERLOADED` can also mean writes
  to the thread's embeddings kept landing for 8 reconcile rounds; resend once the
  thread's writes settle.
- **Check:** the slowest sources in `system.query_log` by `log_comment`
  ([monitoring](/docs/pt-BR/operations/monitoring#logs-traces-and-query-attribution)); `ops.health` for part
  pressure; for memory, the statements of `system.query_log` with `exception_code = 241`
  and the largest `memory_usage` beside them, against the server's
  `max_server_memory_usage` and the user's `max_memory_usage_for_user`.
- **Fix:** the query or worker that dominates; for memory, that statement, or more
  memory for the ClickHouse service. A Loom far from its ClickHouse (a WAN)
  needs `LOOM_LATENCY_BUDGET_MULTIPLIER` near 20 so its latency budgets do not fire on
  every call. For `OVERLOADED` under a sustained write rate, size the pool for the
  connections batched inserts hold
  ([batched inserts](/docs/pt-BR/architecture/clickhouse#batched-inserts)): raise
  `LOOM_CH_POOL_MAXSIZE` (a restart), switch batching off for the switched group with
  `loom settings set ch_batched_inserts false` (`LOOM_CH_BATCHED_INSERTS=false` at
  startup; the waited async inserts still wait), or lower the insert waits live with
  `loom settings set ch_batched_insert_wait_max_ms <ms>` (and
  `ch_batched_insert_background_wait_max_ms`).
  [Runtime settings](/docs/pt-BR/operations/runtime-settings) covers live overrides.
  The [errors reference](/docs/pt-BR/reference/errors) lists every code.

## Providers and search quality

### Searches fail with 503 EMBED_TIMEOUT

- **Cause:** the embedding provider did not answer the query embedding within
  `LOOM_MEMORY_SEARCH_EMBED_BOUND_S` (2.5 s) on both of its attempts by default (one
  retry, `LOOM_MEMORY_SEARCH_EMBED_BOUND_RETRIES`). A query whose vector is in the
  embedding cache still answers.
- **Check:** the server log's `query embed exceeded …s (attempt n/m)` warnings, and
  `loom.agent_events` rows with `error_code = 'EMBED_TIMEOUT'`. A provider that hangs
  sets no health flag; `/v1/health` shows `provider_failing:<name>` only when it also
  answers errors. Also check the provider's status page.
- **Fix:** a raw HTTP or MCP caller retries once; an SDK caller has already retried, so it
  backs off. If it repeats, the provider is slow or down: wait it out, or raise the bound
  for a provider that is slow but answering. Each failed search holds the server for up to
  one bound per attempt.

### Search returns arbitrary results, or extraction writes nothing

- **Symptom:** `set_from_messages` writes no facts, `memory.ask` returns empty answers,
  or search ranks by character overlap.
- **Cause:** the process runs on a fake provider: configured as fake
  (`LOOM_LLM_PROVIDER=fake`, which every stock `.env` sets, or `fake-ngram-*`
  embeddings in a checkout's `.env` or an upgraded install), or configured as real
  without its key.
- **Check:** `curl -s localhost:7777/v1/health | jq '.problems[].code'` shows
  `llm_key_missing`, `embedder_key_missing`, `fake_llm` or `fake_embedder`
  (`fake_llm: true`, `fake_embedding: true` in the body).
- **Fix:** set the key named in the problem's `fix` (`OPENAI_API_KEY`,
  `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`) and the provider block in `.env`, then recreate
  the containers (`docker compose … up -d --force-recreate`). Setting a key alone does not
  change a provider the `.env` pins to `fake`. On the local stack the vector width is
  fixed when the tables are created, so switching from the fake embedder needs
  `make down-all && make dev`. Rows embedded by the fake embedder need `loom reembed`.
  Set `LOOM_REQUIRE_REAL_PROVIDERS=true` so this reads `status: degraded` next time.

### Search quality dropped after an embedder change

- **Cause:** a new embedder moves every similarity score. Rows still embedded by the old
  model sit in a different vector space.
- **Check:** `loom reembed --dry-run` counts off-model rows; `loom.search.top_similarity`
  shows the shift.
- **Fix:** `loom reembed` ([maintenance](/docs/pt-BR/operations/maintenance#re-embed-and-re-extract)), then
  re-baseline your alert on `loom.search.top_similarity`. With
  `LOOM_ADAPTIVE_THRESHOLDS` on (the default) the similarity gates follow the new
  embedder after the boot probe; `loom recalibrate` reruns it immediately.

### A write is missing from search, or was not stored

- **Check:** `loom drops --namespace NS` for what the ingest pipeline discarded or
  truncated, and the write's own response (`status`, conflict verdicts,
  `pipeline_errors`). `loom.embed.pending` counts rows still waiting for a vector.
- **Fix:** follow the drop reason ([maintenance](/docs/pt-BR/operations/maintenance#memory-ingest-drops));
  for a conflicted write, resolve it in the dashboard's Review queue
  ([conflicts and refusal](/docs/pt-BR/features/memory/conflicts-and-refusal)).

## Auth and tokens

### Requests fail with 401 AUTH_INVALID or 503 AUTH_UNAVAILABLE

- **Symptom:** 401 `AUTH_INVALID`, or 503 `AUTH_UNAVAILABLE` with `Retry-After: 5`.
- **Check:** the `message` in the error detail (expired, bad signature, revoked, missing
  bearer). On a client machine, `loom doctor --client` tests the token the hooks and the
  SDK would send.
- **Fix:**
  - A bad signature means the token was minted with a different `LOOM_JWT_SECRET` (secret
    rotated, or minted against another deployment): re-mint with
    `docker exec loom-server loom mint-token …`.
  - An expired or revoked token needs a new one; `loom revoke-token --list` shows the
    revocations.
  - `AUTH_UNAVAILABLE` right after boot means the revocation list has not loaded yet
    (`revocation_not_loaded`); the server refuses tokens until it has, so retry after a
    few seconds and check ClickHouse if it persists.
  - A stale `LOOM_API_KEY` in the environment overrides `~/.loom/token` for the SDK and
    the plugins.
  - After turning auth on for the first time, every call needs a token: mint one as
    above.

### An id you know exists reads as missing

- **Cause:** an op addressed only by an org-wide id, such as `memory.get_by_id`,
  answers as for a missing row rather than a 403 that would confirm the id exists. The
  usual cause is a token without the op's role on the row's namespace. The ops that
  resolve the id before the read also answer so when the caller reaches more than one
  namespace holding the id, as after a `namespace.restore`.
  [Row-scoped operations](/docs/pt-BR/architecture/security#row-scoped-operations) has the rule
  and its exceptions.
- **Fix:** use a token with the op's role on the row's namespace
  ([tokens and grants](/docs/pt-BR/guides/tokens-and-grants)); for an id in several
  namespaces, address the row by key in its namespace (`memory.get`, or
  `memory.update` for a write).

## Schema and upgrades

### Migrations pending or checksum drift

- **Symptom:** `/v1/health` reports `pending_migrations`, or
  `loom migrate --dry-run --strict` exits 1 (it does so on ledger-only rows or checksum
  drift, not on pending files alone), or `redeploy.sh`'s preflight stops.
- **Check:** `loom migrate --dry-run --strict` prints pending files, then `ledger-only`
  rows (ledger filenames the repository no longer has) and `checksum drift` (applied
  files whose text no longer matches the recorded SHA-256).
- **Fix:**
  - Pending files: `loom migrate`, or restart with `LOOM_BOOT_MIGRATE=true`.
  - Checksum drift or a ledger-only row means a migration file was changed (or renamed
    and changed) after some server applied it, usually a server run from an unmerged branch. Compare
    the applied DDL with the repository file. If the effect is identical, record the
    current file (the newest row wins):

    ```sql
    INSERT INTO loom._migrations (filename, checksum, applied_by)
    VALUES ('<file>', '<sha256 of the file>', 'operator');
    ```

    A ledger-only row whose DDL exists under the new name can be removed with
    `DELETE FROM loom._migrations WHERE filename = '<old name>'`.

  - If the effect differs, write a new migration that reconciles the schema; never edit
    an applied file ([migrations](/docs/pt-BR/development/migrations)).
  - A boot that fails on a table "with rows in the old layout" needs
    `scripts/ops/table_rebuild.py` for that table first
    ([upgrades](/docs/pt-BR/operations/upgrades#how-migrations-run)).

### Requests fail with 503 SCHEMA_PENDING

- **Cause:** the statement names a table that a migration this build carries would
  create, and the service has not applied it (a process booted with
  `LOOM_BOOT_MIGRATE=false`, or a migration failed).
- **Fix:** as for pending migrations above.

## Cells

### A process with LOOM_CELL unset is gated

- **Symptom:** every request but the health probes answers 503, `/v1/health/ready`
  fails its `cell` check, no worker runs, and the reason says the deployment "is a cell
  in enforced mode and LOOM_CELL is unset", "holds a partial cell state (... present)"
  or that its "cell state is not read yet". A CLI command that declares an org refuses
  with "this deployment is a cell in enforced mode (cell ...) and LOOM_CELL is unset",
  "holds a partial cell state and LOOM_CELL is unset", or "cell state could not be read
  (...)"; for the first, set `LOOM_CELL` on the CLI, and for the last, act on the reason
  it names: a missing grant ([upgrade step 15](/docs/pt-BR/operations/upgrades#upgrade-procedure)) or a
  Postgres that does not answer.
- **Cause:** under the Postgres authority every server process reads the deployment's
  cell state (`loom.cell_first`, `loom.tenancy_mode`, `loom.cell_self`, `loom.tenancy`).
  With `LOOM_CELL` unset it serves only while none of them holds a row, or while they
  hold a first cell's full state in `recording`
  ([cells](/docs/pt-BR/architecture/cells#the-tenancy-check-and-placement)).
- **Check:** whether the router's directory holds a row for this deployment. Skip this
  on a deployment with no router. Start in a psql session on `LOOM_PG_DSN`:

  ```sql
  SELECT deployment_id AS id FROM loom.authority \gset
  \connect '<router DSN>'
  SELECT cell, is_first, state, placed FROM loom_router.cells WHERE deployment_id = :'id';
  ```

- **Fix**, for "not read yet": the process booted with Postgres migrations pending
  (`LOOM_BOOT_MIGRATE` off) or has not reached Postgres since; apply the migrations
  (`/v1/health` names them) or restore the connection, and the next read, within a
  second, opens or gates it as below.
- **Fix** otherwise, by what the check shows:
  - **The deployment is a cell** (the directory holds its row, and it is neither an
    unattached later cell nor a first cell being backed out, below): set `LOOM_CELL` to the
    cell's name; for a partial state, run `loom cell prepare` again, which completes it.
  - **The deployment is not meant to be a cell** (no directory row, or no router), or
    **an unattached later cell** (its row reads `is_first = false`,
    `state = 'attaching'` and `placed = 0`) that you are turning back into a single
    deployment: for a registered cell, use the
    [fenced state cleanup](/docs/pt-BR/operations/cell-control#clear-a-registered-unattached-cells-state).
    Do not run the SQL below for a registered cell. For a deployment whose directory
    has no row, first positively stop/fence every control client and outstanding router
    control request and prevent restarts; with no online acceptance controller, clear
    the state in one transaction, in a psql session on `LOOM_PG_DSN`
    (`\connect` back to it after the check) as the role that owns schema `loom`'s tables
    (the role that migrates; altering a trigger needs the table's owner). The
    deployment is then `open`. An unattached later cell's directory row stays live until
    `loom cell remove --cell <cell>` removes it, which a new prepare of the deployment
    needs. On a lock failure, `ROLLBACK` if psql is still open, then rerun the whole step.

    ```sql
    \set ON_ERROR_STOP on
    BEGIN;
    SET LOCAL lock_timeout = '100ms';
    -- the attach lock: held by a running attach, prepare or update
    SELECT pg_advisory_xact_lock(7813586394088235880);
    SELECT pg_advisory_xact_lock(7813586394288842610);  -- the migration lock
    ALTER TABLE loom.cell_first DISABLE TRIGGER cell_first_immutable;
    DELETE FROM loom.cell_first;
    DELETE FROM loom.tenancy_mode;
    DELETE FROM loom.cell_self;
    DELETE FROM loom.tenancy;
    ALTER TABLE loom.cell_first ENABLE ALWAYS TRIGGER cell_first_immutable;
    ALTER TABLE loom.cell_first ENABLE ALWAYS TRIGGER cell_first_no_truncate;
    COMMIT;
    ```

    Then drop the org triggers and the `loom.tenancy` guard, in the same session and as the
    same role, after the state transaction (so a migrating boot's `ensure` does not create
    them again); on a lock failure, `ROLLBACK` if psql is still open, then rerun the step:

    ```sql
    \set ON_ERROR_STOP on
    BEGIN;
    SET LOCAL lock_timeout = '100ms';
    SELECT pg_advisory_xact_lock(7813586394288842610);  -- the migration lock
    SELECT format('DROP TRIGGER %I ON loom.%I', t.tgname, c.relname)
    FROM pg_trigger t JOIN pg_class c ON c.oid = t.tgrelid
    JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE n.nspname = 'loom'
      AND t.tgname IN ('tenancy_insert', 'tenancy_update', 'tenancy_record') \gexec
    COMMIT;
    ```

    A lock failure at the state transaction's attach lock means an attach, a prepare or an
    update is running against this Postgres, which can take hours: rerunning the step
    meanwhile fails the same way. This names the session that holds the lock:

    ```sql
    SELECT a.pid, a.usename, a.application_name, a.client_addr, a.backend_start
    FROM pg_locks l JOIN pg_stat_activity a ON a.pid = l.pid
    WHERE l.locktype = 'advisory' AND l.granted
      AND l.classid = (7813586394088235880 >> 32)::oid
      AND l.objid = (7813586394088235880 & 4294967295)::oid AND l.objsubid = 1
      AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database());
    ```

  - **A first cell being backed out** (its row reads `is_first = true`,
    `state = 'attaching'` and `placed = 0`, whatever its `mode`, since an attach refused at
    its step-3 lock leaves it `enforced`; and `SELECT mode FROM loom.tenancy_mode` on its
    Postgres reads `recording`): back it out in this order. Unset `LOOM_CELL` on every
    process, each of which then serves as in `recording`; run the
    [fenced state cleanup](/docs/pt-BR/operations/cell-control#clear-a-registered-unattached-cells-state),
    with `LOOM_CELL` still set on that control client; then run `loom cell remove --cell <cell>` with the router's operator
    credential and the cell's `LOOM_PG_DSN`, which refuses until that Postgres reads `open`
    and records the row's deployment id. Removing the row first would answer every new org
    503, and clearing the state first would gate every process with `LOOM_CELL` set. The
    router cannot read the cell's Postgres, so a `DELETE` of the row called on the router
    directly skips that check, and leaves a still prepared cell answering its new orgs 503
    and unable to register again until the steps above run. The cleared stores keep no cell
    state, and the deployment is prepared again from `loom cell prepare`. A first cell whose
    row reads `placed > 0` goes forward, never back: run its attach again (the removal
    refuses it).

  - **A restored copy of a cell's Postgres** on another deployment carries the source
    cell's deployment id, so the check returns the live source cell's row. Do not clear
    it, and do not serve it beside the source.

### A CLI with LOOM_CELL set refuses a command that writes for no org

- **Symptom:** a command that writes for no org or revokes (`loom secrets reseal`,
  `loom settings set`, `loom ch-setup`, `loom revoke-token` and the rest) exits with
  "refusing: this Postgres's first row names cell ..., not LOOM_CELL=...", "refusing:
  this cell's state could not be read (...)" or "refusing: this ClickHouse could not be
  read for the cell check (...)".
- **Cause:** with `LOOM_CELL` set such a command reads the cell's `first` row from its
  `LOOM_PG_DSN`, and checks its ClickHouse against that Postgres, before it changes
  anything ([cells](/docs/pt-BR/architecture/cells#the-tenancy-check-and-placement)). A first
  row naming another cell means the CLI is pointed at that cell's stores. A revocation
  that cannot give access back runs through a Postgres outage instead, printing that the
  cell was not checked.
- **Fix:** never clear the state the refusal names: it is another cell's. Point
  `LOOM_PG_DSN` and the ClickHouse settings at this cell's own stores, or set `LOOM_CELL`
  to the cell the first row names if that is the cell you meant. For a state that could
  not be read, act on the reason it names: a missing grant
  ([upgrade step 15](/docs/pt-BR/operations/upgrades#upgrade-procedure)) or a Postgres or ClickHouse that does
  not answer.

### A boot logs "cell triggers: not applied"

- **Symptom:** a boot of a cell in any state but `open` logs
  `cell triggers: not applied on loom.<table> (...)`, and `loom doctor` names the table's
  missing or disabled trigger (`loom.cell_first`'s in any such state, an org table's trigger
  or the `loom.tenancy` guard in `enforced`).
- **Cause:** the boot creates a cell's org triggers and guard, and sets `loom.cell_first`'s
  back to always, with a lock wait of 100 ms a try, three tries a second apart, and serves
  past a table in two cases. The log names the sessions that held or queued a conflicting
  lock when the table was still busy: a long transaction or a bulk write on it, or an
  autovacuum, which the short tries do not make yield and which is left to finish (a wait
  long enough to cancel it would queue every writer of the table behind the boot). It says
  `this role cannot alter it` when the migrating role does not own the table, which
  `loom doctor` names as an `owner` defect. Until a later boot finishes the table, a writer
  that runs no tenancy check is not refused there; Loom's own writes are unaffected.
- **Fix:** for a busy table, end the transaction the log names if it is stale, or wait for
  the autovacuum to finish (`pg_stat_progress_vacuum`); for a table the role cannot alter,
  run the boot as the role that owns schema `loom`'s tables, or give the table to that
  role with `ALTER TABLE loom.<table> OWNER TO <role>`. Then restart one pod with
  `LOOM_BOOT_MIGRATE` on, and repeat until `loom doctor` names no missing or disabled
  trigger.

### The cell_marker readiness check names the tenancy marker

- **Symptom:** `/v1/health/ready` of a process with `LOOM_CELL` set reports the
  `cell_marker` check with "pooled Postgres sessions do not keep the tenancy marker":
  failing on an `enforced` or `partial` cell, passing with that reason on a first cell still
  `recording`, and the process's cell heartbeat reports `marked` as `false`.
- **Cause:** a connection pooler between the process and Postgres. A pooler opens its server
  connections with its own startup parameters, so they lack `loom.tenancy_checked`; a pooler
  that forwards client parameters with `SET` reads as unmarked too, since the refresh runs
  `RESET ALL` before it reads the marker. Once a cell has its org triggers, they refuse
  every write of such a session for an org the cell does not hold, Loom's own included.
- **Fix:** point `LOOM_PG_DSN` at Postgres directly, with no pooler of any mode, as a cell
  requires ([Postgres](/docs/pt-BR/operations/postgres)), then restart the processes: a `LOOM_PG_DSN` change
  takes a restart. The check reads again on each cell refresh (`LOOM_TENANCY_MODE_CACHE_S`)
  and clears on the first refresh after the restart.

### A boot fails with "function loom.tenancy_refuse_… does not exist"

- **Symptom:** a migrating boot of an `enforced` cell fails creating a cell trigger with
  `function loom.tenancy_refuse_unheld() does not exist` (or `…_unmarked()`), and
  `loom doctor` names the function absent in any state.
- **Cause:** the function was dropped by hand after its migration ran (011 for the org
  triggers, 012 for the `loom.tenancy` guard; a `DROP FUNCTION … CASCADE` drops its triggers
  with it). The migration ledger already lists the file, so no boot creates the function
  again, and until it is back, unchecked writers are not refused. A `loom doctor` of a new
  release that names the function absent before the release's first migrating boot needs no
  fix: that boot runs the file.
- **Fix:** run the migration file (`schemas/pg_migrations/011_tenancy_triggers.sql` or
  `012_tenancy_guard.sql`) with `psql` as the role that owns schema `loom`'s tables, then,
  on an `enforced` cell, restart one pod with `LOOM_BOOT_MIGRATE` on: its boot creates the
  triggers (on any other cell, attach creates them later).

### Writes are slow, or a script is refused, on a deployment that is not a cell

- **Symptom:** on a deployment whose cell state is `open` (every process serves), Loom's own
  writes, and any script's or bulk loader's, take longer per row, a role without `SELECT` on
  `loom.cell_first` gets "permission denied for table cell_first", or a script's
  `loom.tenancy` insert fails with SQLSTATE `LT001`.
- **Cause:** the org triggers are still on its tables: a schema-only restore of a prepared
  cell, or a cleanup that skipped the step that drops them. In `open` they let every row
  through, but each unmarked write still reads the state. The `loom.tenancy` guard,
  `tenancy_record`, refuses a script's tenancy row with SQLSTATE `LT001`: the row itself
  makes the state `partial`.
- **Check:**

  ```sql
  SELECT c.relname, t.tgname FROM pg_trigger t JOIN pg_class c ON c.oid = t.tgrelid
  JOIN pg_namespace n ON n.oid = c.relnamespace
  WHERE n.nspname = 'loom'
    AND t.tgname IN ('tenancy_insert', 'tenancy_update', 'tenancy_record');
  ```

- **Fix:** run the drop step of
  [A process with LOOM_CELL unset is gated](#a-process-with-loom_cell-unset-is-gated)
  as the role that owns schema `loom`'s tables.

## Workers

### A worker fails every pass

- **Symptom:** `loom.worker.consecutive_failures` stays at 3 or more; the workers log
  repeats `<name>: pass failed` or `pass exceeded its deadline`, then `DEGRADED`.
- **Check:** `GET /v1/admin/workers.status` for the worker's last error, and
  `docker logs loom-workers 2>&1 | grep -E '<name>: (pass|DEGRADED)'` for the traceback.
- **Fix:** follow the exception. Repeated deadline cancellations mean the pass is too
  large for its interval (a big backlog, a slow provider). A provider error usually means
  a missing key or quota. For an LLM worker on the fake provider, disable the worker or
  configure a real one. To stop one worker while you investigate, set its flag to false
  and recreate the workers container; the others keep running.

### No worker runs at all

- **Check:** `/v1/health?include=workers` → `workers_leader` on each process; one must
  hold the lease. `workers_lease_absent` at warn means no process holds a live one. A
  `workers_lease_unreachable` problem names why a process could not ask ClickHouse;
  `workers_lease_given_up` that its workers child kept failing to start.
- **Fix:** run every replica with `LOOM_BACKGROUND_WORKERS=auto` and `LOOM_DEPLOYMENT=1`; a
  process with `off` never runs them, and one outside the deployment runs none under
  `auto`. Set `on` only on a ClickHouse no deployment uses: it runs its own copy and takes
  no lease. During an upgrade an older build leaves the lease to a live newer one.

## Host

### The host or ClickHouse runs out of disk

- **Symptom:** the Loom host's disk fills (containers fail to write, image builds fail),
  or ClickHouse reports low free space.
- **Check:** on the host, `docker system df` and the size of `var/code_repos`; in
  ClickHouse, the per-table sizes and `system.disks` queries in
  [maintenance](/docs/pt-BR/operations/maintenance#disk-pressure-hygiene). When the disk is full, write
  diagnostic output to `/dev/shm`.
- **Fix:** cap Docker's JSON logs, remove build cache and dangling images
  (`docker builder prune`, `docker image prune`). Do not prune stopped containers if a
  stopped container is your rollback. In ClickHouse, let merges collapse versions after
  large erasures, compact the code vector tables with `loom code compact --apply`, or opt
  into per-table retention.

## Local stack

- **The dashboard returns 503.** The server runs from source (`make server`)
  without a built frontend. Run `make frontend-build`; `/v1/admin/*` answers either way.
- **The stack ignores your `.env`.** A `docker compose` command without
  `--env-file .env` boots the compose file's fallbacks (auth on, an OpenAI provider with
  no key). Use `make dev` or pass the flag.
- **A server started from source serves other code.** See
  [local development](/docs/pt-BR/development/local-development#running-the-server-from-source).

## Clients, plugins and MCP

- **The coding-agent plugins** have their own symptom tables (a hook that finds no
  Node, no primer, captures not arriving, tools listed twice):
  [Claude Code](/docs/pt-BR/features/integrations/claude-code) and
  [Codex](/docs/pt-BR/features/integrations/codex). The plugins' `loom-status` skill runs the
  client check without a Loom checkout.
- **The ledger has no row for the request.** The request never arrived.
  `loom doctor --client` names the server URL and token the SDK and hooks resolve, and
  where each came from; a stale `LOOM_API_KEY` or `LOOM_BASE_URL` in the environment
  wins over `~/.loom/`. `GET /v1/whoami` with the token shows the subject, org,
  namespaces and roles the server verified. A 403 on a namespace the token does not name,
  or `IDENTITY_INCOMPLETE` from a scoped token with no namespace in the body, is
  authorization, not a bug ([tokens and grants](/docs/pt-BR/guides/tokens-and-grants#namespace-access)).
  Stranded `loom-flush-*.json` files in the temp directory are plugin captures that never
  reached the server ([headless bots](/docs/pt-BR/guides/headless-bots#5-when-the-bot-is-quiet)).
- **An MCP tool is missing from the listing.** Listings vary by the MCP URL's
  `?toolset=`, never by caller. Under `all`, an experimental op that neither the core nor
  the atomic roster lists is left out of the listing; it is still callable by name or
  through `loom_op`. The `code.*` tools appear on the default toolset only with
  `LOOM_MCP_CODE_TOOLS=true` ([MCP toolsets](/docs/pt-BR/reference/mcp#toolsets)).
- **An MCP call returns `isError: true`.** A refusal other than 401 (`LOCKED`,
  `PINNED_CONFLICT`, `BLACKLISTED`, `RATE_LIMITED`, `UNKNOWN_ARGUMENT`, …) is a tool
  result with `isError` and the HTTP error detail in `structuredContent`; a 401 or a 5xx
  is a JSON-RPC error. Match on `code`.
- **A field you sent was ignored.** The response names it in `ignored_args`, with
  `did_you_mean` when a known field is close; a write operation refuses a close
  misspelling with 422 `UNKNOWN_ARGUMENT`
  ([unknown and removed fields](/docs/pt-BR/reference/http-api#unknown-and-removed-fields)).
- **429 `RATE_LIMITED`.** See the buckets in
  [monitoring](/docs/pt-BR/operations/monitoring#rate-limits-and-traffic-classes); `Retry-After` says when.

## Filing a bug

`loom report` prints the bug form's Environment field with the `loom doctor` diagnostic
(presence booleans only, never secret values); `--open` opens the bug form with the
versions filled in ([filing an issue](https://github.com/ClickHouse/loom/blob/main/CONTRIBUTING.md#filing-an-issue)).
