Skip to main content

Memory governor

The deacon’s memory governor (assessMemoryPressure in src/lib/cloister/memory-governor.ts) watches available RAM and CPU pressure and reports what it sees to the activity feed through the memory-pressure patrol. Its verdict holds nothing back by itself: the preemptive scheduler’s automatic resume (preemption.ts:resumeYieldedAgents) and the shed() reclaim function (stopping merged Docker stacks, pausing idle work agents) have no production caller. CPU pressure acts at the dispatch doors instead; see CPU pressure. The governor never blocks conversations, pan start, or dashboard Start. A separate, unrelated check — evaluateSpawnGuardrails in routes/agents/shared.ts — is what actually gates POST /api/agents (the operator’s start button, the planning auto-handoff, and its deferred retry).

Reserve defaults

Four reserves, in resources.governor_{hard,soft,watch,recovery}_reserve_gb, each computed as min(max(pct × totalGb, floorGb), cap × totalGb) — a percentage of total RAM, with an absolute floor for small hosts and a cap so the floor never swallows a small host whole: At or above 40 GB, the cap and floor never bind and these match the pre-scaling (PAN-2500) values, so nothing changes on large hosts. The POST /api/agents spawn guardrail has its own pair, resources.memory_warn_gb / resources.memory_block_gb, scaled the same way but simpler — min(4, totalGb / 8) and min(2, totalGb / 16): Both tables come from src/lib/config-yaml/governor-reserves.ts (computeGovernorReserveDefaultsGb, computeSpawnMemoryThresholdDefaultsGb).

Normalizing a user-set config

If a config.yaml override breaks the reserves’ ordering invariants, normalizeGovernorReserves (same module, called from config-yaml/merge.ts) corrects it and logs one warning naming the key it changed:
  1. hard >= soft → hard = soft × 0.5.
  2. recovery <= soft → recovery = soft + 1.
  3. watch <= soft → watch = soft + 1.
  4. Any of the four at or above this host’s total RAM → all four reset to the scaled defaults above (a recovery reserve at or above total RAM can never be reached, which is what locked the governor into permanent shedding before PAN-4267).

Linux vs macOS measurement

Linux reads /proc/meminfo’s MemAvailable and /proc/pressure/memory for PSI (pressure stall information); a swap-free-percent hold applies when PSI shows live stalls. macOS has neither. computeDarwinAvailableMemoryBytes (src/lib/system-health/darwin.ts) is the one available-memory calculation shared by the header collector and the governor’s reader (readProcMemoryDarwin in src/dashboard/server/services/proc-memory.ts): it prefers memory_pressure -Q’s free percentage of total RAM, falling back to the Activity Monitor vm_stat formula (total - (anonymous - purgeable + wired + compressor) × page size) when memory_pressure is unavailable. macOS also allocates swap on demand, so a low free-swap share is not pressure there — the governor ignores swap runway on darwin (swapGrowsOnDemand). Its stall signal instead is the kernel’s own kern.memorystatus_vm_pressure_level sysctl: level 4 (critical) sheds immediately regardless of the memory reserves, and level 1 (normal) counts as calm for the holding re-admit window — the same role Linux’s PSI plays.

Small Macs

Before PAN-4267, an 8-16 GB Mac could get a recovery reserve at or above its total RAM, so the governor never cleared HARD. The scaled defaults above fix this for a fresh install. If you still need to override the reserves by hand (e.g. on a host between the documented sizes), add to ~/.overdeck/config.yaml:
Then pan restart. normalizeGovernorReserves will correct anything in this override that violates the ordering invariants above and log why.

CPU pressure

On Linux the governor and the dispatch doors read CPU Pressure Stall Information (/proc/pressure/cpu, some avg60): the share of the last minute during which a runnable task waited for a CPU. Unlike load average, it does not count tasks blocked on IO. Where PSI is unavailable (macOS, old kernels), load per core (governor_cpu_soft_load_per_core / governor_cpu_recovery_load_per_core) is the fallback. When CPU pressure is at or above the hold threshold:
  • pan lane start and POST /api/lanes refuse with HTTP 429 cpu-saturated. Pass pan lane start --force (or "force": true) to launch anyway.
  • Starting an agent from the dashboard shows an acknowledgeable cpu_saturated warning. Your confirm covers it; the planning auto-handoff does not, so it waits and retries.
  • Flywheel-started work agents are held only if you turn on governor_cpu_hold_dispatch.
Conversations, pan start, and dashboard Start are never blocked on CPU. Runaway processes (for example a tool call that outlived its agent) are reported in the activity feed with the kill -TERM -<pgid> command to stop them; Overdeck never kills them itself. The deacon log (~/.overdeck/logs/deacon.log) gets one cpu-pressure sample line every 5 minutes with PSI some avg10/avg60 and load per core, so you can calibrate the hold threshold for your host. New weights and nice levels apply to units and agents started after the change.