From bff13f2909ff2e89fba66c56e60410add69ac780 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 12 Nov 2026 17:38:04 +0100 Subject: [PATCH 1/3] Cap all terminals together at two thirds of RAM in their own slice Terminal scopes move to app-wtd-terminals.slice within app-wtd.slice, which the daemon caps with MemoryMax at start. A runaway set of terminals then hits an OOM inside that slice instead of pushing the session into memory pressure, where systemd-oomd kills the largest app cgroup, typically the browser. Also warm the hub up before the baseline of the 8 MiB stderr memory test, whose first-session growth alone exceeded the bound. --- README.md | 2 +- openspec/specs/daemon/spec.md | 6 ++- openspec/specs/platform/spec.md | 10 +++-- src/daemon/main/index.ts | 7 +++- src/daemon/main/memory.test.ts | 2 +- src/platform/scopes/index.ts | 3 ++ src/platform/scopes/scopes.test.ts | 43 ++++++++++++++++++-- src/platform/scopes/scopes.ts | 20 ++++++++- test/process/daemon-terminals-scopes.test.ts | 12 +++++- test/process/hub-router-remote.test.ts | 6 +++ test/support/scopes.ts | 3 ++ 11 files changed, 99 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 959d21f..6245d49 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,7 @@ A host's `port` is the loopback port on this machine through which its own hub i ## Memory and out-of-memory kills -With a systemd user manager that outlives your session (the process runs under `user@.service`, as anything started from a desktop session does, or lingering is on), every process worktree-term starts gets a cgroup of its own in the slice `app-wtd.slice`: the daemon `wtd--daemon-.scope`, the hub `wtd--hub-.scope`, and each terminal, with everything it starts, `wtd--term--.scope`, where `` hashes the state directory. systemd-oomd and the kernel then kill only the terminal that ran out of memory, never the daemon, the hub or another terminal; its tab says `killed: out of memory`, and `memory.log` gets an `oom-kill` line and each terminal's cgroup figures. A daemon run as `worktree-term-daemon.service` stays in that unit. Terminal shells also get an `oom_score_adj` of at least 500, so a system-wide OOM picks a terminal before the daemon or hub. +With a systemd user manager that outlives your session (the process runs under `user@.service`, as anything started from a desktop session does, or lingering is on), every process worktree-term starts gets a cgroup of its own in the slice `app-wtd.slice`: the daemon `wtd--daemon-.scope`, the hub `wtd--hub-.scope`, and each terminal, with everything it starts, `wtd--term--.scope` in `app-wtd-terminals.slice` within it, where `` hashes the state directory. The daemon caps that slice at two thirds of physical memory (`MemoryMax`, until the user manager stops), so all terminals together cannot push the rest of the desktop into memory pressure; at the cap the kernel kills the terminal with the highest OOM score, not the browser. A terminal that ran out of memory is killed alone, never the daemon, the hub or another terminal; its tab says `killed: out of memory`, and `memory.log` gets an `oom-kill` line and each terminal's cgroup figures. A daemon run as `worktree-term-daemon.service` stays in that unit. Terminal shells also get an `oom_score_adj` of at least 500, so a system-wide OOM picks a terminal before the daemon or hub. Without such a user manager (no systemd, an SSH session without lingering, `systemd-run` failing), everything runs in the cgroup it was started in, as before, and memory limits are ignored. The daemon says which on its standard error at start: `terminal scopes: on`, or `terminal scopes: off ()`. diff --git a/openspec/specs/daemon/spec.md b/openspec/specs/daemon/spec.md index ae44c95..a9744f2 100644 --- a/openspec/specs/daemon/spec.md +++ b/openspec/specs/daemon/spec.md @@ -626,12 +626,16 @@ Once it holds the start lock and before it serves its socket, the daemon SHALL f - **THEN** it serves from the cgroup it was started in ### Requirement: Terminal scopes -At start the daemon SHALL write one line to its standard error: `terminal scopes: on` when it has a lasting user manager, else `terminal scopes: off ()`. While terminal scopes are on, the daemon SHALL start each terminal's process in its terminal scope with the `memory` limits of its `createTerm`, so that every process the terminal starts, detached ones included, belongs to that scope and to no other terminal's; and at start it SHALL reset every failed unit whose name begins `wtd--term-`. While terminal scopes are off, terminals SHALL start as without scopes and their limits SHALL be ignored, the first `createTerm` with a limit writing `memory limits ignored: terminal scopes are off` to standard error once per daemon. In either case, where `/proc` exists, the shell and every process it starts SHALL have an `oom_score_adj` of at least 500, set before the shell runs. +At start the daemon SHALL write one line to its standard error: `terminal scopes: on` when it has a lasting user manager, else `terminal scopes: off ()`. While terminal scopes are on, the daemon SHALL start each terminal's process in its terminal scope with the `memory` limits of its `createTerm`, so that every process the terminal starts, detached ones included, belongs to that scope and to no other terminal's; and at start it SHALL cap the terminal slice (see `platform`) at two thirds of physical memory, rounded down to a whole MiB, reporting a failure to cap it on its standard error, and reset every failed unit whose name begins `wtd--term-`. While terminal scopes are off, terminals SHALL start as without scopes and their limits SHALL be ignored, the first `createTerm` with a limit writing `memory limits ignored: terminal scopes are off` to standard error once per daemon. In either case, where `/proc` exists, the shell and every process it starts SHALL have an `oom_score_adj` of at least 500, set before the shell runs. #### Scenario: Terminal processes in the terminal's scope - **WHEN** terminal scopes are on and a terminal's shell runs `sleep 1000 &`, `sh -c 'sleep 1001' &` and `setsid sleep 1002 &` - **THEN** the shell and the three `sleep` processes have that terminal's `wtd--term--.scope` cgroup, which is neither the daemon's nor another terminal's +#### Scenario: Terminals capped together +- **WHEN** terminal scopes are on and a terminal is created +- **THEN** its scope's parent, `app-wtd-terminals.slice`, has a `memory.max` of two thirds of physical memory rounded down to a whole MiB, and the daemon's cgroup is outside that slice + #### Scenario: Limits applied - **WHEN** terminal scopes are on and a terminal is created with `memory` `{max: "64M", swapMax: "0"}` - **THEN** its scope's `memory.max` is 67108864 and its `memory.swap.max` is 0 diff --git a/openspec/specs/platform/spec.md b/openspec/specs/platform/spec.md index ef0ebe9..9292a14 100644 --- a/openspec/specs/platform/spec.md +++ b/openspec/specs/platform/spec.md @@ -271,8 +271,8 @@ A memory log SHALL be appended to one JSON object per line, each line written by ### Requirement: Process scopes A process SHALL find a **lasting user manager** at most once in its lifetime, as both of: the scope probe — the systemd-run command with `--user --scope --quiet --collect --slice=app-wtd.slice -- true` — exits 0 within 2 s; and the process's own cgroup (the `0::` line of `/proc/self/cgroup`) lies under `/user.slice/user-.slice/user@.service/`, or `/var/lib/systemd/linger/` exists. Otherwise the outcome SHALL name the first reason in one line (the probe's failure, its timeout, or `not under the user manager and not lingering`). A probe still running after 2 s SHALL be killed with `SIGKILL` and reaped before the outcome is given. The systemd-run command SHALL be `$WTD_SYSTEMD_RUN` when set and non-empty, else `systemd-run`; the busctl command `$WTD_BUSCTL` when set and non-empty, else `busctl`; the systemctl command as dialing states. Once the user manager is found lasting, the process SHALL run the scope probe once more with `--expand-environment=no` added after `--collect`; the option is **supported** when that run exits 0 within 2 s and unsupported when it exits non-zero saying `unrecognized option`; any other outcome — another failure, or a run still going after 2 s, which SHALL be killed and reaped — SHALL count as no lasting user manager, the reason naming it. -Every scope SHALL be in the slice `app-wtd.slice`. Its unit name SHALL begin `wtd--`, `` being the first 8 lowercase hexadecimal digits of the SHA-256 of the state directory path: `wtd--daemon-.scope` for a daemon, `wtd--hub-.scope` for a hub, `` being 8 random lowercase hexadecimal digits, and `wtd--term--.scope` for a terminal, `` being its daemon's instance id. -A process **entering its scope** SHALL ask the user manager, through the busctl command, to start the transient scope with its own pid as the only member, in that slice, and wait for the start to complete. A process **started in a scope** SHALL be started through the systemd-run command with `--user --scope --quiet --unit= --slice=app-wtd.slice -p OOMPolicy=kill`, then `--expand-environment=no` when the option is supported, a `-p MemoryHigh=`, `-p MemoryMax=` or `-p MemorySwapMax=` for each memory limit given, `--`, and its command, so that it keeps its pid, environment and arguments as given. +Every scope SHALL be in the slice `app-wtd.slice`; a terminal scope SHALL be in the **terminal slice** `app-wtd-terminals.slice` within it. **Capping the terminal slice** at `` bytes SHALL run the systemctl command with `--user set-property --runtime app-wtd-terminals.slice MemoryMax=`, failing when it cannot start, exits non-zero or outlasts 2 s. Its unit name SHALL begin `wtd--`, `` being the first 8 lowercase hexadecimal digits of the SHA-256 of the state directory path: `wtd--daemon-.scope` for a daemon, `wtd--hub-.scope` for a hub, `` being 8 random lowercase hexadecimal digits, and `wtd--term--.scope` for a terminal, `` being its daemon's instance id. +A process **entering its scope** SHALL ask the user manager, through the busctl command, to start the transient scope with its own pid as the only member, in that slice, and wait for the start to complete. A process **started in a scope** SHALL be started through the systemd-run command with `--user --scope --quiet --unit= --slice=app-wtd-terminals.slice -p OOMPolicy=kill`, then `--expand-environment=no` when the option is supported, a `-p MemoryHigh=`, `-p MemoryMax=` or `-p MemorySwapMax=` for each memory limit given, `--`, and its command, so that it keeps its pid, environment and arguments as given. A scope's **outcome** SHALL be read by querying its `ActiveState` and `Result` through the systemctl command every 50 ms until `ActiveState` is `inactive` or `failed`, for up to 2 s; it SHALL be out of memory exactly when `Result` is `oom-kill` by then. A scope found `failed` SHALL then be reset with `reset-failed`. A query that fails or yields output that does not parse SHALL count as not out of memory and SHALL be reported. #### Scenario: Lasting under the user manager @@ -297,7 +297,11 @@ A scope's **outcome** SHALL be read by querying its `ActiveState` and `Result` t #### Scenario: Started in a scope - **WHEN** a command is started in a terminal scope with limits `max` `64M` and `swapMax` `0` -- **THEN** its process runs with its own pid in the unit's cgroup under `app-wtd.slice`, whose `memory.max` is 67108864, `memory.swap.max` 0 and `memory.oom.group` 1 +- **THEN** its process runs with its own pid in the unit's cgroup under `app-wtd.slice/app-wtd-terminals.slice`, whose `memory.max` is 67108864, `memory.swap.max` 0 and `memory.oom.group` 1 + +#### Scenario: Terminal slice capped +- **WHEN** the terminal slice is capped at `` bytes and a command is then started in a terminal scope +- **THEN** the scope's parent cgroup has a `memory.max` of `` #### Scenario: Command arguments kept as given - **WHEN** a command is started in a terminal scope with arguments holding `$$` and `${HOME}` diff --git a/src/daemon/main/index.ts b/src/daemon/main/index.ts index ae28725..1240558 100644 --- a/src/daemon/main/index.ts +++ b/src/daemon/main/index.ts @@ -13,6 +13,7 @@ import { type MemorySampler, } from '../../platform/memory/index.js'; import { + capTerminalSlice, cgroupPath, daemonUnit, enterScope, @@ -22,6 +23,7 @@ import { outOfMemory, resetFailedScopes, scopeCommand, + terminalSliceMax, terminalUnit, unitPrefix, type UserManager, @@ -52,8 +54,8 @@ type Report = (error: unknown) => void; type UnitOf = ((termId: number) => string) | null; /** - * Moves the daemon into a scope of its own unless it runs as its systemd unit, and lowers its `oom_score_adj` as far - * as permitted; whether terminals get scopes of their own, said on standard error. + * Moves the daemon into a scope of its own unless it runs as its systemd unit, caps the terminal slice, and lowers + * its `oom_score_adj` as far as permitted; whether terminals get scopes of their own, said on standard error. */ const placeDaemon = async (paths: HostPaths, report: Report): Promise => { const roots = hostScopeRoots(); @@ -62,6 +64,7 @@ const placeDaemon = async (paths: HostPaths, report: Report): Promise { describe('terminal scope figures', () => { const UNIT = 'wtd-01234567-term-inst-1.scope'; - const SCOPE_PATH = `/user.slice/user-1.slice/user@1.service/app.slice/app-wtd.slice/${UNIT}`; + const SCOPE_PATH = `/user.slice/user-1.slice/user@1.service/app.slice/app-wtd.slice/app-wtd-terminals.slice/${UNIT}`; const FIGURES: ProcessCgroup = { path: SCOPE_PATH, current: 1000, diff --git a/src/platform/scopes/index.ts b/src/platform/scopes/index.ts index 5850c14..f6f344c 100644 --- a/src/platform/scopes/index.ts +++ b/src/platform/scopes/index.ts @@ -1,4 +1,5 @@ export { + capTerminalSlice, cgroupPath, daemonUnit, enterScope, @@ -10,6 +11,8 @@ export { resetFailedScopes, SCOPE_SLICE, scopeCommand, + TERMINAL_SLICE, + terminalSliceMax, terminalUnit, unitPrefix, type Report, diff --git a/src/platform/scopes/scopes.test.ts b/src/platform/scopes/scopes.test.ts index 1b91540..b3fe7ed 100644 --- a/src/platform/scopes/scopes.test.ts +++ b/src/platform/scopes/scopes.test.ts @@ -1,7 +1,7 @@ import { spawn, spawnSync, type ChildProcess } from 'node:child_process'; import { createHash } from 'node:crypto'; import { existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; -import { tmpdir } from 'node:os'; +import { tmpdir, totalmem } from 'node:os'; import { join } from 'node:path'; import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; import { waitUntil } from '../../../test/support/daemon-host.js'; @@ -18,6 +18,7 @@ import { withUserManager, } from '../../../test/support/scopes.js'; import { + capTerminalSlice, daemonUnit, enterScope, findUserManager, @@ -28,6 +29,8 @@ import { resetFailedScopes, SCOPE_SLICE, scopeCommand, + TERMINAL_SLICE, + terminalSliceMax, terminalUnit, unitPrefix, type ScopeRoots, @@ -239,14 +242,36 @@ describe('unit names', () => { expect(terminalUnit(dir, 'a1B2_c-3', 7)).toBe(`wtd-${hash()}-term-a1B2_c-3-7.scope`); }); - it('use the slice app-wtd.slice', () => { + it('use the slice app-wtd.slice, and for terminals app-wtd-terminals.slice within it', () => { expect(SCOPE_SLICE).toBe('app-wtd.slice'); + expect(TERMINAL_SLICE).toBe('app-wtd-terminals.slice'); + }); +}); + +describe('terminal slice cap', () => { + it('is two thirds of physical memory, rounded down to a whole MiB', () => { + const MiB = 1024 * 1024; + expect(terminalSliceMax(3 * MiB)).toBe(2 * MiB); + expect(terminalSliceMax(4 * MiB)).toBe(2 * MiB); + expect(terminalSliceMax(3 * MiB - 1)).toBe(MiB); + expect(terminalSliceMax()).toBe(Math.floor((totalmem() * 2) / 3 / MiB) * MiB); + }); + + it('sets MemoryMax on the terminal slice through systemctl, until the user manager stops', async () => { + const ctl = systemctl(''); + await capTerminalSlice(1234); + expect(ctl.calls()).toEqual([['--user', 'set-property', '--runtime', 'app-wtd-terminals.slice', 'MemoryMax=1234']]); + }); + + it('rejects naming the failure of systemctl', async () => { + vi.stubEnv('WTD_SYSTEMCTL', fake('systemctl', "echo 'Failed to set unit properties: Access denied' >&2; exit 1").program); + await expect(capTerminalSlice(1234)).rejects.toThrow('Access denied'); }); }); describe('scope command', () => { const unit = 'wtd-01234567-term-i-1.scope'; - const head = ['--user', '--scope', '--quiet', `--unit=${unit}`, '--slice=app-wtd.slice', '-p', 'OOMPolicy=kill']; + const head = ['--user', '--scope', '--quiet', `--unit=${unit}`, '--slice=app-wtd-terminals.slice', '-p', 'OOMPolicy=kill']; it('runs the command through systemd-run with each limit given as a property', () => { vi.stubEnv('WTD_SYSTEMD_RUN', '/opt/fake run'); @@ -447,12 +472,22 @@ describe.skipIf(NO_USER_MANAGER !== null)(withUserManager('scopes of the real us const pid = Number(printed); expect(pid).toBe(child.pid); const path = cgroupOf(pid); - expect(path.endsWith(`/app-wtd.slice/${unit}`)).toBe(true); + expect(path.endsWith(`/app-wtd.slice/app-wtd-terminals.slice/${unit}`)).toBe(true); expect(cgroupFile(path, 'memory.max')).toBe('67108864'); expect(cgroupFile(path, 'memory.swap.max')).toBe('0'); expect(cgroupFile(path, 'memory.oom.group')).toBe('1'); }); + it('caps the slice of terminal scopes as a whole', async () => { + const max = terminalSliceMax(); + await capTerminalSlice(max); + const unit = terminalUnit(dir, 'capped', 1); + const [file = '', ...args] = scopeCommand(unit, {}, ['sh', '-c', 'cat /proc/self/cgroup'], noExpand); + const path = /^0::(\/.*)$/m.exec(spawnSync(file, args, { encoding: 'utf8' }).stdout)?.[1] ?? ''; + expect(path.endsWith(`/${TERMINAL_SLICE}/${unit}`)).toBe(true); + expect(cgroupFile(path.slice(0, path.lastIndexOf('/')), 'memory.max')).toBe(String(max)); + }); + it('moves a running process into a hub scope of its own', async () => { const child = spawn('sleep', ['30'], { stdio: 'ignore' }); children.push(child); diff --git a/src/platform/scopes/scopes.ts b/src/platform/scopes/scopes.ts index a5e2c8b..8001c88 100644 --- a/src/platform/scopes/scopes.ts +++ b/src/platform/scopes/scopes.ts @@ -1,13 +1,19 @@ import { spawn } from 'node:child_process'; import { createHash, randomBytes } from 'node:crypto'; import { access, readFile, writeFile } from 'node:fs/promises'; -import { userInfo } from 'node:os'; +import { totalmem, userInfo } from 'node:os'; import { join } from 'node:path'; import { setTimeout as sleep } from 'node:timers/promises'; /** The slice every scope lives in. */ export const SCOPE_SLICE = 'app-wtd.slice'; +/** The slice within `SCOPE_SLICE` that terminal scopes live in, capped as a whole. */ +export const TERMINAL_SLICE = 'app-wtd-terminals.slice'; + +/** The share of physical memory all terminals together may use. */ +const TERMINALS_SHARE = 2 / 3; + const PROBE_TIMEOUT_MS = 2000; const OUTCOME_TIMEOUT_MS = 2000; const START_TIMEOUT_MS = 2000; @@ -225,7 +231,7 @@ export const scopeCommand = (unit: string, limits: ScopeLimits, command: readonl '--scope', '--quiet', `--unit=${unit}`, - `--slice=${SCOPE_SLICE}`, + `--slice=${TERMINAL_SLICE}`, '-p', 'OOMPolicy=kill', ...(noExpand ? ['--expand-environment=no'] : []), @@ -322,6 +328,16 @@ export const outOfMemory = async (unit: string, report: Report): Promise Math.floor((total * TERMINALS_SHARE) / MiB) * MiB; + +/** Caps the terminal slice at `max` bytes until the user manager stops, so terminals cannot starve the rest of the session. */ +export const capTerminalSlice = async (max: number): Promise => { + await systemctl(['set-property', '--runtime', TERMINAL_SLICE, `MemoryMax=${String(max)}`]); +}; + /** Resets every failed unit whose name begins with `prefix`. */ export const resetFailedScopes = async (prefix: string, report: Report): Promise => { await systemctl(['reset-failed', `${prefix}*`]).catch(report); diff --git a/test/process/daemon-terminals-scopes.test.ts b/test/process/daemon-terminals-scopes.test.ts index 1059178..52af83c 100644 --- a/test/process/daemon-terminals-scopes.test.ts +++ b/test/process/daemon-terminals-scopes.test.ts @@ -1,5 +1,6 @@ import { spawnSync } from 'node:child_process'; import { existsSync, readFileSync } from 'node:fs'; +import { totalmem } from 'node:os'; import { join } from 'node:path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { DaemonClient } from '../support/daemon-client.js'; @@ -29,6 +30,7 @@ import { scopePath, SLICE, SMALL, + TERMINAL_SLICE, terminalUnit, unitState, userManagerEnv, @@ -123,12 +125,20 @@ describe.skipIf(NO_USER_MANAGER !== null)(withUserManager('terminal scopes on'), const sleeps = await Promise.all(['1000', '1001', '1002'].map((s) => pidOf(['sleep', s]))); const other = await run('echo $$ > other.pid; exec sleep 60'); const scope = cgroupOf(shell); - expect(scope.endsWith(`/${SLICE}/${terminalUnit(host.stateDir, instance, termId)}`)).toBe(true); + expect(scope.endsWith(`/${SLICE}/${TERMINAL_SLICE}/${terminalUnit(host.stateDir, instance, termId)}`)).toBe(true); for (const pid of sleeps) expect(cgroupOf(pid)).toBe(scope); expect(cgroupOf(daemon.pid)).not.toBe(scope); expect(cgroupOf(await pidIn('other.pid')).endsWith(`/${terminalUnit(host.stateDir, instance, other)}`)).toBe(true); }); + it('caps all its terminals together at two thirds of physical memory, leaving its own scope uncapped', async () => { + await run('echo $$ > sh.pid; exec sleep 60'); + const scope = cgroupOf(await pidIn('sh.pid')); + const MiB = 1024 * 1024; + expect(cgroupFile(scope.slice(0, scope.lastIndexOf('/')), 'memory.max')).toBe(String(Math.floor((totalmem() * 2) / 3 / MiB) * MiB)); + expect(cgroupOf(daemon.pid)).not.toContain(`/${TERMINAL_SLICE}/`); + }); + it("applies the createTerm's memory limits to the terminal's scope", async () => { await run('echo $$ > sh.pid; exec sleep 60', SMALL); const scope = cgroupOf(await pidIn('sh.pid')); diff --git a/test/process/hub-router-remote.test.ts b/test/process/hub-router-remote.test.ts index 770b82b..c811dbb 100644 --- a/test/process/hub-router-remote.test.ts +++ b/test/process/hub-router-remote.test.ts @@ -141,6 +141,12 @@ describe('remote hosts', () => { }); it('keeps its memory bounded and the reason short when SSH writes 8 MiB to standard error without a newline', async () => { + // A first remote going down warms the hub up, so the baseline leaves out its one-off growth. + fx.ssh.failHost('warm', 'ssh: Could not resolve hostname warm'); + configure([{ name: 'warm', ssh: 'warm' }]); + const warm = await fx.host.session(); + await warm.waitHost(1, (h) => h.status === 'down', { timeout: 30_000 }); + warm.close(); const before = residentBytes(fx.hub.pid); fx.ssh.spewHost('box', 8 * MiB); configure([{ name: 'box', ssh: 'box' }]); diff --git a/test/support/scopes.ts b/test/support/scopes.ts index 34d8db8..bcea927 100644 --- a/test/support/scopes.ts +++ b/test/support/scopes.ts @@ -7,6 +7,9 @@ import { join } from 'node:path'; /** The slice every scope lives in. */ export const SLICE = 'app-wtd.slice'; +/** The slice within `SLICE` that terminal scopes live in. */ +export const TERMINAL_SLICE = 'app-wtd-terminals.slice'; + export const SKIP_REASON = 'skipped: no lasting systemd user manager'; /** Bash that allocates 300 MiB in the shell itself. */ From eeae6eddbf89dddd519f6a915f5c82d8b96de700 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 12 Nov 2026 18:49:16 +0100 Subject: [PATCH 2/3] Wait for the daemon unit name to be free when another checkout's suite holds it The systemd-unit placement test must start its daemon as worktree-term-daemon.service, a name shared with every other checkout's suite on the host. It now retries for up to 30 s while systemd reports the unit already loaded. --- test/process/daemon-terminals-scopes.test.ts | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/test/process/daemon-terminals-scopes.test.ts b/test/process/daemon-terminals-scopes.test.ts index 52af83c..28fba6c 100644 --- a/test/process/daemon-terminals-scopes.test.ts +++ b/test/process/daemon-terminals-scopes.test.ts @@ -327,12 +327,18 @@ describe.skipIf(NO_USER_MANAGER !== null)(withUserManager('daemon placement'), ( it.skipIf(unitTaken)(`leaves a daemon running as ${UNIT} in that unit's cgroup`, async () => { const env = host.env(userManagerEnv()); const setenv = Object.entries(env).flatMap(([key, value]) => ['-E', `${key}=${value ?? ''}`]); - const started = spawnSync( - 'systemd-run', - ['--user', '--quiet', '--collect', `--unit=${UNIT}`, ...setenv, process.execPath, BUNDLE, 'daemon'], - { - encoding: 'utf8', + const started = await waitUntil( + () => { + const ran = spawnSync( + 'systemd-run', + ['--user', '--quiet', '--collect', `--unit=${UNIT}`, ...setenv, process.execPath, BUNDLE, 'daemon'], + { encoding: 'utf8' }, + ); + // Another checkout's suite on this host may hold the unit for a moment. + return ran.stderr.includes('already loaded') ? undefined : ran; }, + `${UNIT} to be free`, + 30_000, ); expect(started.status, started.stderr).toBe(0); try { From dacdc46a9f702bbcc3191ccc956ead6625f2ef02 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 12 Nov 2026 18:57:27 +0100 Subject: [PATCH 3/3] State in the README what the terminal slice cap does not promise It reserves no memory for the rest of the desktop, the kernel kills at it only when reclaim fails and may kill several terminals in turn, and setting it can fail, which the daemon reports and runs on. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6245d49..364fc06 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,7 @@ A host's `port` is the loopback port on this machine through which its own hub i ## Memory and out-of-memory kills -With a systemd user manager that outlives your session (the process runs under `user@.service`, as anything started from a desktop session does, or lingering is on), every process worktree-term starts gets a cgroup of its own in the slice `app-wtd.slice`: the daemon `wtd--daemon-.scope`, the hub `wtd--hub-.scope`, and each terminal, with everything it starts, `wtd--term--.scope` in `app-wtd-terminals.slice` within it, where `` hashes the state directory. The daemon caps that slice at two thirds of physical memory (`MemoryMax`, until the user manager stops), so all terminals together cannot push the rest of the desktop into memory pressure; at the cap the kernel kills the terminal with the highest OOM score, not the browser. A terminal that ran out of memory is killed alone, never the daemon, the hub or another terminal; its tab says `killed: out of memory`, and `memory.log` gets an `oom-kill` line and each terminal's cgroup figures. A daemon run as `worktree-term-daemon.service` stays in that unit. Terminal shells also get an `oom_score_adj` of at least 500, so a system-wide OOM picks a terminal before the daemon or hub. +With a systemd user manager that outlives your session (the process runs under `user@.service`, as anything started from a desktop session does, or lingering is on), every process worktree-term starts gets a cgroup of its own in the slice `app-wtd.slice`: the daemon `wtd--daemon-.scope`, the hub `wtd--hub-.scope`, and each terminal, with everything it starts, `wtd--term--.scope` in `app-wtd-terminals.slice` within it, where `` hashes the state directory. The daemon caps that slice at two thirds of physical memory (`MemoryMax`, until the user manager stops), so all terminals together use at most that share; the cap reserves nothing for the rest of the desktop, which can still run short if it needs more than the remaining third. At the cap, when the kernel cannot reclaim enough, it kills the terminal with the highest OOM score, not the browser, and then the next while the terminals still exceed it. A failure to set the cap is reported on the daemon's standard error, and the daemon runs on without it. A terminal that ran out of memory is killed whole, with everything it started, but never the daemon or the hub, and at its own limit no other terminal; its tab says `killed: out of memory`, and `memory.log` gets an `oom-kill` line and each terminal's cgroup figures. A daemon run as `worktree-term-daemon.service` stays in that unit. Terminal shells also get an `oom_score_adj` of at least 500, so a system-wide OOM picks a terminal before the daemon or hub. Without such a user manager (no systemd, an SSH session without lingering, `systemd-run` failing), everything runs in the cgroup it was started in, as before, and memory limits are ignored. The daemon says which on its standard error at start: `terminal scopes: on`, or `terminal scopes: off ()`.