Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ A host's `port` is the loopback port on this machine through which its own hub i

## Memory and out-of-memory kills

With a systemd user manager that outlives your session (the process runs under `user@<uid>.service`, as anything started from a desktop session does, or lingering is on), every process worktree-term starts gets a cgroup of its own in the slice `app-wtd.slice`: the daemon `wtd-<h>-daemon-<r>.scope`, the hub `wtd-<h>-hub-<r>.scope`, and each terminal, with everything it starts, `wtd-<h>-term-<instance>-<id>.scope`, where `<h>` hashes the state directory. systemd-oomd and the kernel then kill only the terminal that ran out of memory, never the daemon, the hub or another terminal; its tab says `killed: out of memory`, and `memory.log` gets an `oom-kill` line and each terminal's cgroup figures. A daemon run as `worktree-term-daemon.service` stays in that unit. Terminal shells also get an `oom_score_adj` of at least 500, so a system-wide OOM picks a terminal before the daemon or hub.
With a systemd user manager that outlives your session (the process runs under `user@<uid>.service`, as anything started from a desktop session does, or lingering is on), every process worktree-term starts gets a cgroup of its own in the slice `app-wtd.slice`: the daemon `wtd-<h>-daemon-<r>.scope`, the hub `wtd-<h>-hub-<r>.scope`, and each terminal, with everything it starts, `wtd-<h>-term-<instance>-<id>.scope` in `app-wtd-terminals.slice` within it, where `<h>` hashes the state directory. The daemon caps that slice at two thirds of physical memory (`MemoryMax`, until the user manager stops), so all terminals together cannot push the rest of the desktop into memory pressure; at the cap the kernel kills the terminal with the highest OOM score, not the browser. A terminal that ran out of memory is killed alone, never the daemon, the hub or another terminal; its tab says `killed: out of memory`, and `memory.log` gets an `oom-kill` line and each terminal's cgroup figures. A daemon run as `worktree-term-daemon.service` stays in that unit. Terminal shells also get an `oom_score_adj` of at least 500, so a system-wide OOM picks a terminal before the daemon or hub.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

Without such a user manager (no systemd, an SSH session without lingering, `systemd-run` failing), everything runs in the cgroup it was started in, as before, and memory limits are ignored. The daemon says which on its standard error at start: `terminal scopes: on`, or `terminal scopes: off (<reason>)`.

Expand Down
6 changes: 5 additions & 1 deletion openspec/specs/daemon/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -626,12 +626,16 @@ Once it holds the start lock and before it serves its socket, the daemon SHALL f
- **THEN** it serves from the cgroup it was started in

### Requirement: Terminal scopes
At start the daemon SHALL write one line to its standard error: `terminal scopes: on` when it has a lasting user manager, else `terminal scopes: off (<reason>)`. While terminal scopes are on, the daemon SHALL start each terminal's process in its terminal scope with the `memory` limits of its `createTerm`, so that every process the terminal starts, detached ones included, belongs to that scope and to no other terminal's; and at start it SHALL reset every failed unit whose name begins `wtd-<h>-term-`. While terminal scopes are off, terminals SHALL start as without scopes and their limits SHALL be ignored, the first `createTerm` with a limit writing `memory limits ignored: terminal scopes are off` to standard error once per daemon. In either case, where `/proc` exists, the shell and every process it starts SHALL have an `oom_score_adj` of at least 500, set before the shell runs.
At start the daemon SHALL write one line to its standard error: `terminal scopes: on` when it has a lasting user manager, else `terminal scopes: off (<reason>)`. While terminal scopes are on, the daemon SHALL start each terminal's process in its terminal scope with the `memory` limits of its `createTerm`, so that every process the terminal starts, detached ones included, belongs to that scope and to no other terminal's; and at start it SHALL cap the terminal slice (see `platform`) at two thirds of physical memory, rounded down to a whole MiB, reporting a failure to cap it on its standard error, and reset every failed unit whose name begins `wtd-<h>-term-`. While terminal scopes are off, terminals SHALL start as without scopes and their limits SHALL be ignored, the first `createTerm` with a limit writing `memory limits ignored: terminal scopes are off` to standard error once per daemon. In either case, where `/proc` exists, the shell and every process it starts SHALL have an `oom_score_adj` of at least 500, set before the shell runs.

#### Scenario: Terminal processes in the terminal's scope
- **WHEN** terminal scopes are on and a terminal's shell runs `sleep 1000 &`, `sh -c 'sleep 1001' &` and `setsid sleep 1002 &`
- **THEN** the shell and the three `sleep` processes have that terminal's `wtd-<h>-term-<instance>-<termId>.scope` cgroup, which is neither the daemon's nor another terminal's

#### Scenario: Terminals capped together
- **WHEN** terminal scopes are on and a terminal is created
- **THEN** its scope's parent, `app-wtd-terminals.slice`, has a `memory.max` of two thirds of physical memory rounded down to a whole MiB, and the daemon's cgroup is outside that slice

#### Scenario: Limits applied
- **WHEN** terminal scopes are on and a terminal is created with `memory` `{max: "64M", swapMax: "0"}`
- **THEN** its scope's `memory.max` is 67108864 and its `memory.swap.max` is 0
Expand Down
10 changes: 7 additions & 3 deletions openspec/specs/platform/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,8 +271,8 @@ A memory log SHALL be appended to one JSON object per line, each line written by

### Requirement: Process scopes
A process SHALL find a **lasting user manager** at most once in its lifetime, as both of: the scope probe — the systemd-run command with `--user --scope --quiet --collect --slice=app-wtd.slice -- true` — exits 0 within 2 s; and the process's own cgroup (the `0::` line of `/proc/self/cgroup`) lies under `/user.slice/user-<uid>.slice/user@<uid>.service/`, or `/var/lib/systemd/linger/<user name>` exists. Otherwise the outcome SHALL name the first reason in one line (the probe's failure, its timeout, or `not under the user manager and not lingering`). A probe still running after 2 s SHALL be killed with `SIGKILL` and reaped before the outcome is given. The systemd-run command SHALL be `$WTD_SYSTEMD_RUN` when set and non-empty, else `systemd-run`; the busctl command `$WTD_BUSCTL` when set and non-empty, else `busctl`; the systemctl command as dialing states. Once the user manager is found lasting, the process SHALL run the scope probe once more with `--expand-environment=no` added after `--collect`; the option is **supported** when that run exits 0 within 2 s and unsupported when it exits non-zero saying `unrecognized option`; any other outcome — another failure, or a run still going after 2 s, which SHALL be killed and reaped — SHALL count as no lasting user manager, the reason naming it.
Every scope SHALL be in the slice `app-wtd.slice`. Its unit name SHALL begin `wtd-<h>-`, `<h>` being the first 8 lowercase hexadecimal digits of the SHA-256 of the state directory path: `wtd-<h>-daemon-<r>.scope` for a daemon, `wtd-<h>-hub-<r>.scope` for a hub, `<r>` being 8 random lowercase hexadecimal digits, and `wtd-<h>-term-<instance>-<termId>.scope` for a terminal, `<instance>` being its daemon's instance id.
A process **entering its scope** SHALL ask the user manager, through the busctl command, to start the transient scope with its own pid as the only member, in that slice, and wait for the start to complete. A process **started in a scope** SHALL be started through the systemd-run command with `--user --scope --quiet --unit=<unit> --slice=app-wtd.slice -p OOMPolicy=kill`, then `--expand-environment=no` when the option is supported, a `-p MemoryHigh=<v>`, `-p MemoryMax=<v>` or `-p MemorySwapMax=<v>` for each memory limit given, `--`, and its command, so that it keeps its pid, environment and arguments as given.
Every scope SHALL be in the slice `app-wtd.slice`; a terminal scope SHALL be in the **terminal slice** `app-wtd-terminals.slice` within it. **Capping the terminal slice** at `<n>` bytes SHALL run the systemctl command with `--user set-property --runtime app-wtd-terminals.slice MemoryMax=<n>`, failing when it cannot start, exits non-zero or outlasts 2 s. Its unit name SHALL begin `wtd-<h>-`, `<h>` being the first 8 lowercase hexadecimal digits of the SHA-256 of the state directory path: `wtd-<h>-daemon-<r>.scope` for a daemon, `wtd-<h>-hub-<r>.scope` for a hub, `<r>` being 8 random lowercase hexadecimal digits, and `wtd-<h>-term-<instance>-<termId>.scope` for a terminal, `<instance>` being its daemon's instance id.
A process **entering its scope** SHALL ask the user manager, through the busctl command, to start the transient scope with its own pid as the only member, in that slice, and wait for the start to complete. A process **started in a scope** SHALL be started through the systemd-run command with `--user --scope --quiet --unit=<unit> --slice=app-wtd-terminals.slice -p OOMPolicy=kill`, then `--expand-environment=no` when the option is supported, a `-p MemoryHigh=<v>`, `-p MemoryMax=<v>` or `-p MemorySwapMax=<v>` for each memory limit given, `--`, and its command, so that it keeps its pid, environment and arguments as given.
A scope's **outcome** SHALL be read by querying its `ActiveState` and `Result` through the systemctl command every 50 ms until `ActiveState` is `inactive` or `failed`, for up to 2 s; it SHALL be out of memory exactly when `Result` is `oom-kill` by then. A scope found `failed` SHALL then be reset with `reset-failed`. A query that fails or yields output that does not parse SHALL count as not out of memory and SHALL be reported.

#### Scenario: Lasting under the user manager
Expand All @@ -297,7 +297,11 @@ A scope's **outcome** SHALL be read by querying its `ActiveState` and `Result` t

#### Scenario: Started in a scope
- **WHEN** a command is started in a terminal scope with limits `max` `64M` and `swapMax` `0`
- **THEN** its process runs with its own pid in the unit's cgroup under `app-wtd.slice`, whose `memory.max` is 67108864, `memory.swap.max` 0 and `memory.oom.group` 1
- **THEN** its process runs with its own pid in the unit's cgroup under `app-wtd.slice/app-wtd-terminals.slice`, whose `memory.max` is 67108864, `memory.swap.max` 0 and `memory.oom.group` 1

#### Scenario: Terminal slice capped
- **WHEN** the terminal slice is capped at `<n>` bytes and a command is then started in a terminal scope
- **THEN** the scope's parent cgroup has a `memory.max` of `<n>`

#### Scenario: Command arguments kept as given
- **WHEN** a command is started in a terminal scope with arguments holding `$$` and `${HOME}`
Expand Down
7 changes: 5 additions & 2 deletions src/daemon/main/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import {
type MemorySampler,
} from '../../platform/memory/index.js';
import {
capTerminalSlice,
cgroupPath,
daemonUnit,
enterScope,
Expand All @@ -22,6 +23,7 @@ import {
outOfMemory,
resetFailedScopes,
scopeCommand,
terminalSliceMax,
terminalUnit,
unitPrefix,
type UserManager,
Expand Down Expand Up @@ -52,8 +54,8 @@ type Report = (error: unknown) => void;
type UnitOf = ((termId: number) => string) | null;

/**
* Moves the daemon into a scope of its own unless it runs as its systemd unit, and lowers its `oom_score_adj` as far
* as permitted; whether terminals get scopes of their own, said on standard error.
* Moves the daemon into a scope of its own unless it runs as its systemd unit, caps the terminal slice, and lowers
* its `oom_score_adj` as far as permitted; whether terminals get scopes of their own, said on standard error.
*/
const placeDaemon = async (paths: HostPaths, report: Report): Promise<UserManager> => {
const roots = hostScopeRoots();
Expand All @@ -62,6 +64,7 @@ const placeDaemon = async (paths: HostPaths, report: Report): Promise<UserManage
if (!((await cgroupPath(roots))?.endsWith(`/${UNIT_NAME}`) ?? false)) {
await enterScope(daemonUnit(paths.stateDir), process.pid).catch(report);
}
await capTerminalSlice(terminalSliceMax()).catch(report);
await resetFailedScopes(`${unitPrefix(paths.stateDir)}term-`, report);
}
await lowerOomScoreAdj().catch(report);
Expand Down
2 changes: 1 addition & 1 deletion src/daemon/main/memory.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,7 @@ describe('closed terminals', () => {

describe('terminal scope figures', () => {
const UNIT = 'wtd-01234567-term-inst-1.scope';
const SCOPE_PATH = `/user.slice/user-1.slice/user@1.service/app.slice/app-wtd.slice/${UNIT}`;
const SCOPE_PATH = `/user.slice/user-1.slice/user@1.service/app.slice/app-wtd.slice/app-wtd-terminals.slice/${UNIT}`;
const FIGURES: ProcessCgroup = {
path: SCOPE_PATH,
current: 1000,
Expand Down
3 changes: 3 additions & 0 deletions src/platform/scopes/index.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
export {
capTerminalSlice,
cgroupPath,
daemonUnit,
enterScope,
Expand All @@ -10,6 +11,8 @@ export {
resetFailedScopes,
SCOPE_SLICE,
scopeCommand,
TERMINAL_SLICE,
terminalSliceMax,
terminalUnit,
unitPrefix,
type Report,
Expand Down
43 changes: 39 additions & 4 deletions src/platform/scopes/scopes.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { spawn, spawnSync, type ChildProcess } from 'node:child_process';
import { createHash } from 'node:crypto';
import { existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { tmpdir, totalmem } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
import { waitUntil } from '../../../test/support/daemon-host.js';
Expand All @@ -18,6 +18,7 @@ import {
withUserManager,
} from '../../../test/support/scopes.js';
import {
capTerminalSlice,
daemonUnit,
enterScope,
findUserManager,
Expand All @@ -28,6 +29,8 @@ import {
resetFailedScopes,
SCOPE_SLICE,
scopeCommand,
TERMINAL_SLICE,
terminalSliceMax,
terminalUnit,
unitPrefix,
type ScopeRoots,
Expand Down Expand Up @@ -239,14 +242,36 @@ describe('unit names', () => {
expect(terminalUnit(dir, 'a1B2_c-3', 7)).toBe(`wtd-${hash()}-term-a1B2_c-3-7.scope`);
});

it('use the slice app-wtd.slice', () => {
it('use the slice app-wtd.slice, and for terminals app-wtd-terminals.slice within it', () => {
expect(SCOPE_SLICE).toBe('app-wtd.slice');
expect(TERMINAL_SLICE).toBe('app-wtd-terminals.slice');
});
});

describe('terminal slice cap', () => {
it('is two thirds of physical memory, rounded down to a whole MiB', () => {
const MiB = 1024 * 1024;
expect(terminalSliceMax(3 * MiB)).toBe(2 * MiB);
expect(terminalSliceMax(4 * MiB)).toBe(2 * MiB);
expect(terminalSliceMax(3 * MiB - 1)).toBe(MiB);
expect(terminalSliceMax()).toBe(Math.floor((totalmem() * 2) / 3 / MiB) * MiB);
});

it('sets MemoryMax on the terminal slice through systemctl, until the user manager stops', async () => {
const ctl = systemctl('');
await capTerminalSlice(1234);
expect(ctl.calls()).toEqual([['--user', 'set-property', '--runtime', 'app-wtd-terminals.slice', 'MemoryMax=1234']]);
});

it('rejects naming the failure of systemctl', async () => {
vi.stubEnv('WTD_SYSTEMCTL', fake('systemctl', "echo 'Failed to set unit properties: Access denied' >&2; exit 1").program);
await expect(capTerminalSlice(1234)).rejects.toThrow('Access denied');
});
});

describe('scope command', () => {
const unit = 'wtd-01234567-term-i-1.scope';
const head = ['--user', '--scope', '--quiet', `--unit=${unit}`, '--slice=app-wtd.slice', '-p', 'OOMPolicy=kill'];
const head = ['--user', '--scope', '--quiet', `--unit=${unit}`, '--slice=app-wtd-terminals.slice', '-p', 'OOMPolicy=kill'];

it('runs the command through systemd-run with each limit given as a property', () => {
vi.stubEnv('WTD_SYSTEMD_RUN', '/opt/fake run');
Expand Down Expand Up @@ -447,12 +472,22 @@ describe.skipIf(NO_USER_MANAGER !== null)(withUserManager('scopes of the real us
const pid = Number(printed);
expect(pid).toBe(child.pid);
const path = cgroupOf(pid);
expect(path.endsWith(`/app-wtd.slice/${unit}`)).toBe(true);
expect(path.endsWith(`/app-wtd.slice/app-wtd-terminals.slice/${unit}`)).toBe(true);
expect(cgroupFile(path, 'memory.max')).toBe('67108864');
expect(cgroupFile(path, 'memory.swap.max')).toBe('0');
expect(cgroupFile(path, 'memory.oom.group')).toBe('1');
});

it('caps the slice of terminal scopes as a whole', async () => {
const max = terminalSliceMax();
await capTerminalSlice(max);
const unit = terminalUnit(dir, 'capped', 1);
const [file = '', ...args] = scopeCommand(unit, {}, ['sh', '-c', 'cat /proc/self/cgroup'], noExpand);
const path = /^0::(\/.*)$/m.exec(spawnSync(file, args, { encoding: 'utf8' }).stdout)?.[1] ?? '';
expect(path.endsWith(`/${TERMINAL_SLICE}/${unit}`)).toBe(true);
expect(cgroupFile(path.slice(0, path.lastIndexOf('/')), 'memory.max')).toBe(String(max));
});

it('moves a running process into a hub scope of its own', async () => {
const child = spawn('sleep', ['30'], { stdio: 'ignore' });
children.push(child);
Expand Down
20 changes: 18 additions & 2 deletions src/platform/scopes/scopes.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,19 @@
import { spawn } from 'node:child_process';
import { createHash, randomBytes } from 'node:crypto';
import { access, readFile, writeFile } from 'node:fs/promises';
import { userInfo } from 'node:os';
import { totalmem, userInfo } from 'node:os';
import { join } from 'node:path';
import { setTimeout as sleep } from 'node:timers/promises';

/** The slice every scope lives in. */
export const SCOPE_SLICE = 'app-wtd.slice';

/** The slice within `SCOPE_SLICE` that terminal scopes live in, capped as a whole. */
export const TERMINAL_SLICE = 'app-wtd-terminals.slice';

/** The share of physical memory all terminals together may use. */
const TERMINALS_SHARE = 2 / 3;

const PROBE_TIMEOUT_MS = 2000;
const OUTCOME_TIMEOUT_MS = 2000;
const START_TIMEOUT_MS = 2000;
Expand Down Expand Up @@ -225,7 +231,7 @@ export const scopeCommand = (unit: string, limits: ScopeLimits, command: readonl
'--scope',
'--quiet',
`--unit=${unit}`,
`--slice=${SCOPE_SLICE}`,
`--slice=${TERMINAL_SLICE}`,
'-p',
'OOMPolicy=kill',
...(noExpand ? ['--expand-environment=no'] : []),
Expand Down Expand Up @@ -322,6 +328,16 @@ export const outOfMemory = async (unit: string, report: Report): Promise<boolean
}
};

const MiB = 1024 * 1024;

/** The `MemoryMax` of the terminal slice for `total` bytes of physical memory, in whole MiB as the kernel keeps whole pages. */
export const terminalSliceMax = (total = totalmem()): number => Math.floor((total * TERMINALS_SHARE) / MiB) * MiB;

/** Caps the terminal slice at `max` bytes until the user manager stops, so terminals cannot starve the rest of the session. */
export const capTerminalSlice = async (max: number): Promise<void> => {
await systemctl(['set-property', '--runtime', TERMINAL_SLICE, `MemoryMax=${String(max)}`]);
};

/** Resets every failed unit whose name begins with `prefix`. */
export const resetFailedScopes = async (prefix: string, report: Report): Promise<void> => {
await systemctl(['reset-failed', `${prefix}*`]).catch(report);
Expand Down
Loading
Loading