SQLite lock contention + stale-lock timeout after 2026.9.6 migration (shared state DB)

Troubleshooting
1

Version: OpenClaw 2026.9.6 (eb377ac), systemd user service, single host, Telegram channel, two agents (main + ppq-agent)

Summary:
After updating to 2026.9.6, the Gateway’s shared SQLite database (~/.openclaw/agents/main/agent/openclaw-agent.sqlite) developed persistent, unresolvable write-lock contention. Response latency went from ~2s to 30-40s+ per message. Restarts, config changes, and the built-in memory repair tool did not resolve it. Also found a separate, clearly reproducible bug: the memory-core short-term-lock does not check holder-PID liveness before timing out.

Root cause timing (strong circumstantial evidence):
The 2026.9.6 update’s migration step (completed 00:12:44 local time) consolidated several previously-independent JSON state stores — Telegram allowFrom/pairing requests, memory-core host events, exec-approvals, workspace-state (for both main and ppq-agent) — into the single SQLite file that also stores session transcripts. The very first “slow SQLite transaction hold” log entry of any kind appeared 44 seconds after that migration completed:

operation=agent.write, elapsedMs=34817, isMainThread=true

Nothing resembling this log line exists anywhere before that timestamp. In the following 24h, we logged 286 such slow-hold events; zero occurred before the migration.

Confirmed NOT proportional to activity:
Checked logs during a genuine ~4.5-minute idle window (zero chat messages, zero tool calls): still logged a 10.089s unlabeled slow hold. This rules out “just heavy usage” as the explanation — the contention is ambient/structural, not load-driven.

WAL never stabilizes:

• Manually checkpointed WAL once via PRAGMA wal_checkpoint(TRUNCATE): 62MB → 672KB.
• Refilled to 62MB within ~1 hour under light, ordinary usage.
• After a full gateway restart (clean process, fresh state): refilled to 62MB within ~90 seconds.
• Current state as of openclaw memory status --fix: Agent database: 219.5 MiB · WAL 61.6 MiB.

Diagnostic gap — stalls are unlabeled even at max verbosity:
Set logging.level to debug, then trace. Smaller stalls (2-4s) get labeled "session transcript export snapshot". The larger, dominant stalls (10-14s) never receive an operation label at any logging level tested — no way to identify the actual code path from available logs.

Hypothesis tested and disproven:
Suspected the memory-core embedding bootstrap (failing with “no OpenAI key configured”, degrading to keyword-only) was a contributing writer. Set memory.search.provider to "none" (confirmed applied), restarted cleanly. WAL ballooned to 62MB just as fast as before — this hypothesis is ruled out.

Separate, clearly reproducible bug — stale-lock timeout ignores holder liveness:

  1. openclaw gateway stop → confirmed systemctl --user is-active openclaw-gateway → inactive
  2. openclaw memory status --fix --agent main → failed:
Reason: Timed out waiting for memory workspace lock at
plugin-state:memory-core/short-term-locks/c217adbdc7b2c84179b48b694e8536ea04e4e461e824d3865660bcfc69430129
(held by 2117536…e1c7)

3. Independently confirmed via ps -p 2117536 that this PID no longer existed — it was the previous (fully stopped) gateway process.
4. openclaw gateway start → new PID (2119238) → same memory status --fix --agent main command now succeeds instantly with Repair: no changes.
This shows the short-term-lock mechanism waits out its full timeout against a lock record whose holder PID is already dead, rather than checking liveness and failing fast / auto-clearing. It’s reproducible on demand via the stop → fix → (timeout) → start → fix → (succeeds) sequence above. Note: fixing/observing this did not reduce WAL size or slow-hold frequency — it appears to be a related-but-separate issue from the core contention problem.

What we’d like investigated:

  1. Why does WAL checkpointing fail to keep pace with write volume since the 2026.9.6 migration? Is autocheckpoint disabled, or is a long-lived read transaction preventing truncation?
  2. Should memory-core’s short-term-lock check holder-PID liveness before waiting out the full timeout?
  3. Could the dominant 10-14s slow-hold operations get proper labels in logs (even at trace level), to make this diagnosable without needing an internal profiler?
  4. Was consolidating multiple independent subsystem state stores (Telegram, memory-core, exec-approvals, workspace-state) into one shared SQLite file — the same file used for session transcripts — an intentional tradeoff? If so, is there guidance on mitigating the resulting lock contention (e.g., separate files per subsystem, connection pooling, WAL tuning)?

Happy to provide the full backup/state directory or further logs if useful for reproduction.