Skip to content

shell: keep an empty word that has a double-quoted expansion - #44904

Closed
robobun wants to merge 3 commits into
mainfrom
robobun/7037d9d3/shell-quoted-empty-expansion
Closed

robobun wants to merge 3 commits into
mainfrom
robobun/7037d9d3/shell-quoted-empty-expansion

Conversation

@robobun

@robobun robobun commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • A word with a double-quoted expansion that expands to nothing gives no argv word: cmd "$UNSET", cmd "$(true)". bash and dash pass one empty word. So [ -n "$UNSET" ] is true.
  • The lexer pushes the same Token::Var for $X and "$X" (src/shell_parser/parse.rs:2652). Only a literal "" sets the flag that Cmd::child_done reads (src/runtime/shell/states/Cmd.rs:390).

Fix

  • The lexer records the quote state on the Var and VarArgv tokens. The parser builds QuotedVar and QuotedVarArgv atoms. Cmd::child_done keeps an empty word that has one.
  • ~"$UNSET" is a literal ~, as in bash, never $HOME. An empty argv[0] prints bun: command not found: (the commit of shell: empty-string argv[0] should fail with "command not found" #32280, included here).
  • Verified: 32 new or changed cells in test/js/bun/shell/ fail on main. Also ran the whole directory.
  • Self-reviewed: 5 changes asked, 5 made.

Background

  • A word is an ast::Atom, a list of SimpleAtom parts. Cmd::child_done drops an empty word that had no quotes. fix(shell): preserve empty string arguments in Bun shell #27231 kept the literal "", '' and ${""}.
  • Considered: set has_quoted_empty for a quoted expansion. Then rm -rf ~"$UNSET" becomes rm -rf $HOME, as ~"" is $HOME today.

Downsides

  • Behaviour change, no flag: cmd "$UNSET" passes "", as sh does. rimraf "$UNSET" (rimraf 4 and 5) then deletes the cwd, and "$UNSET" cmd fails with command not found.
  • Do not merge before shell: fail an empty operand with ENOENT instead of acting on the cwd #38002. On Windows the rm and cp builtins take "" for the cwd: in CI, rm -rf "$UNSET" deletes a file in the cwd and cp -R "$UNSET" out copies the cwd into out. Those two cells fail on the Windows lanes until then.
  • "$(printf ' ')" becomes "" (bash: one space). Cost per statement with no empty word: +4 to +7 instructions of about 7700, 0 allocations. .text +2304 bytes.
Notes

Matrix. 12 spellings, each through Bun.$ and through bun exec, beside bash 5.2.37 and dash: "$X" (set to the empty string), "$UNSET", "$(true)", "`true`", "$9", "$A$B", "$A"$B, $A"$B", "$A""$B", "$A$(true)", "$(true)$(true)", $(true)"$(true)". This branch: 24 of 24 rows equal bash. main: 0 of 24. $UNSET, $(true), `true` and $UNSET$(true) without quotes: 8 of 8 rows still give no word. The value can come from an assignment, from process.env, or from .env().

What a script sees (release builds of main e655c58 and of this branch, Linux):

Script main this branch bash 5.2
args "$UNSET" x 1 word 2 words 2 words
[ -n "$UNSET" ] true false false
grep -c "$UNSET" file reads stdin, pattern file counts the lines counts the lines
"$UNSET" cmd runs cmd command not found, exit 1 command not found, exit 127
"$(exit 3)" exit 3 command not found, exit 1 command not found, exit 127
~"$UNSET" no word ~ ~
cd "$UNSET" goes to $HOME stays stays
export "$UNSET" prints every export no output, exit 0 error, exit 1
rm -rf "$UNSET" usage, exit 1 removes nothing, exit 0 removes nothing, exit 0
rimraf "$UNSET" (rimraf 4, 5) usage, exit 1 deletes the cwd deletes the cwd

The last row is the same with bun run --shell=system, npm, and every sh. Bun's shell is the default for package.json scripts on Windows.

Windows and #38002. With the cp builtin (the default on Windows, BUN_ENABLE_EXPERIMENTAL_SHELL_BUILTINS=1 elsewhere), cp -R "$UNSET" out prints the usage on main. On this branch it copies the cwd into itself until it crashes (exit 139 on Linux with the variable set). #38002 reports that rm -rf "" deletes the files in the cwd on Windows. Both are reachable on main through "" and ${""}. The rm -rf "$UNSET_VAR" and cp -R "$UNSET_VAR" out cells have no Windows skip, so the Windows lanes show when the order is safe. On the Windows lanes of build 126553 both fail, as expected before #38002. The directory holds keep.txt and sub\inner.txt. After rm -rf "$UNSET_VAR", keep.txt is deleted, the exit code is 1 and stderr is rm: \sub: Invalid argument. After cp -R "$UNSET_VAR" out, the directory has a new out entry.

Still different from bash. Four todo cells record what bash prints.

  • "$(printf ' ')" is "" (bash: one space). A quoted command substitution loses its trailing blanks. main dropped the word, so [ -n "$(printf ' ')" ] goes from true to false (bash: true).
  • a\<newline>"$UNSET" is two words, a and "" (bash: a). A backslash-newline ends the word here. main dropped the second word.
  • A word gives at most one empty field: ""$(printf ' ')"" is one empty word (bash: two).
  • "$UNSET"~ and ~"$UNSET"/ expand to $HOME and $HOME/, as on main (bash: ~ and ~/). ~"" is $HOME (pinned by test/regression/issue/17294.test.ts) and ~$UNSET gives no word, as on main.
  • {"$UNSET",a} loses the empty variant (shell: keep an empty brace expansion word in every position #40709, shell: emit one argv word per brace variant, including empties #30998). echo hi > "$UNSET" says ambiguous redirect. An unquoted $VAR is not field split.

Not in this PR. A double-quote pair that holds only a backslash-newline still gives no word (bash: one empty word). A first version kept it. It made ~$UNSET"\<newline>" expand to $HOME, and line continuations belong to #29670.

Other PRs.

Design. has_quoted_empty keeps its meaning (a literal "" was seen). The new fact is static, so Cmd::child_done and the tilde arm read it from the word (Atom::has_quoted_expansion) only when the result is empty. QuotedVar is a variant and not a bool on Var: the bool makes SimpleAtom 32 bytes.

Measured. Release builds of e655c58 and of this branch, same toolchain, built back to back. valgrind, perf, strace and bloaty are not on this machine.

  • sizes: Token 12 B, SimpleAtom 24 B, Atom 32 B (const asserts), Lexer 200 B. All as on main.
  • tokens and atoms (shellInternals.lex and parse, 18 scripts, among them echo "$HOME" hi, echo a b c, true "$A" b $C and echo "$(echo hi)" x): +0 tokens, +0 atoms.
  • instructions per statement inside bun_shell_parser and bun_runtime::shell functions (bun exec of 800 and of 1600 statements under qemu-user with an exec trace, the slope, the same count on each repeat): echo a b c 7696 to 7700, echo "$HOME" hi 7697 to 7704, echo a $UNSET b 9315 to 9351 (the word check runs), echo a "$UNSET" b 9570 to 9683 (the kept word).
  • calls into mimalloc per statement (same runs): 22 to 22, 20 to 20, 20 to 20, and 20 to 21 with one more mi_free for the kept word. Native under gdb: 20.0 to 20.0 and 15.0 to 15.0 calls, 675.84 to 675.84 and 602.84 to 602.84 requested bytes.
  • arena bytes per parse: not read with allocated_bytes(), because a test of the crate alone does not link here. Token count, atom count and the three sizes are the same.
  • syscalls per statement (qemu -strace): 1 to 1.
  • release binary: .text +2304 B, .rodata +0 B, file +4096 B. SimpleAtom +2 variants, Token +0, host functions +0.

Suites. Release builds: all of test/js/bun/shell/ and test/regression/issue/17294.test.ts. This branch: 1013 pass, 2 fail. main: 981 pass, 34 fail. The 2 are the ls permission cells, which fail as root. The other 32 are the new and changed cells. Debug build: bunshell, lex, parse, exec, env.positionals and 17294 pass. cargo check for x86_64-pc-windows-msvc and aarch64-apple-darwin passes.


no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/shell/bunshell.test.ts

@robobun

robobun commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 6:50 AM PT - Oct 16th, 2026

❌ @robobun, your commit e5aa9b8 has 1 failures in Build #126553 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 44904

That installs a local version of the PR into your bun-44904 executable, so you can run:

bun-44904 --bun

@robobun

robobun commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

Reproduced on 1.4.3-canary.1+367d939d9 and on main, beside bash 5.2.37 and dash. args is a script that prints its argument count and each argument in brackets:

args "$UNSET" x     # Bun: <1>[x]    bash, dash: <2>[][x]
args "$(true)" x    # Bun: <1>[x]    bash, dash: <2>[][x]
X=; args "$X" x     # Bun: <1>[x]    bash, dash: <2>[][x]

The fix is this PR, #44904.

State: this PR waits for #38002. Do not merge it before that.

In build 126553 the only test that fails on each attempt is test/js/bun/shell/bunshell.test.ts, on the two Windows lanes. The other failures in that build passed on a retry. The two cells that fail are the cells that guard the merge order:

  • rm -rf "$UNSET_VAR": on Windows the rm builtin takes the empty operand for the cwd. keep.txt is deleted from the cwd.
  • cp -R "$UNSET_VAR" out: the cp builtin copies the cwd into out.

#38002 makes both builtins reject an empty operand. After it merges, this branch needs a rebase. Both cells are expected to pass then.

@robobun

robobun commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

#46691 makes ${NAME} lex to the same tokens as $NAME (it replaces #45032, which is closed). With this PR and #46691 together, "${X}" with X not set gives one empty word. #46691 has .todo tests for that case. The PR that lands second can turn them on.

`transition_to_exec` was taking the early-exit path (exit 0, no error)
when argv[0] was an empty string. After NUL-termination an empty string
is a one-byte `[0u8]`, so the `a.len() > 1` guard lumped it in with the
no-args case. The Zig reference only early-exits when args[0] is the
Optional-null sentinel (i.e. args was empty); an actual empty-string
word flows through to which() and reports "command not found" with
exit 1, matching bash.
A word with a double-quoted expansion ("$VAR", "$N", "$(cmd)") that
expands to nothing was dropped from argv. bash and dash keep it as one
empty word.

The lexer now records on the Var and VarArgv tokens that they were inside
double quotes. The parser turns them into QuotedVar and QuotedVarArgv
atoms. Cmd keeps an empty word when its atom has a quoted expansion.

A leading tilde before an empty quoted expansion stays a literal tilde,
as in bash, and never becomes $HOME.
@robobun
robobun force-pushed the robobun/7037d9d3/shell-quoted-empty-expansion branch from 51e3593 to 360d3b0 Compare October 16, 2026 08:41
Comment thread src/runtime/shell/states/Cmd.rs Resolved
Comment thread src/runtime/shell/states/Expansion.rs Resolved
Comment thread src/shell_parser/parse.rs Resolved
Comment thread src/shell_parser/parse.rs Resolved
robobun added a commit that referenced this pull request Oct 16, 2026
…ange)

This branch is parked. #45490 has the `$?` and `$$` part. `${NAME}` is
#45032. The two commits below this one are the earlier, larger shape.
A review of them found these items. Do them before this work is
proposed.

Commit 1 (expansions):
- `param_text_braces` goes up in any quote state and comes down only
  in the state of the word. `echo ${S:-"${Y%z}"} mid; echo second`
  prints `set` and drops the rest. Count only where the `}` arm would
  close the word.
- A word with no `}` must be a lexer error (`Unclosed parameter
  expansion`), as for `$(`. Delete the rollback, the `param_unclosed`
  memo and the `Eof` sentinel: 896 KB of unclosed heads lex in 5.4 s.
- A quoted word must stay in `CharState::Double` up to its `}`. An
  inner `"` only flips a bit that decides if `}` closes. `\}` is a
  `}` there. Today `"${X:-"$(echo p  q)"}"` gives 2 arguments (dash
  and bash: 1), `"${X:-"it's done"}"` stays text, and `"${X:-a\}b}"`
  gives `a\b}` (dash: `a}b`).
- `Step::EndField` next to `~`, `{a,b}` or `*`:
  `~/.cache${S:+ --force}` gives `[/.cache][~--force]`,
  `{a,b}${X:-c }` gives an extra empty argument, and
  `echo *.js${S:+ --flag}` fails with `no matches found: --flag`.
  Check the trailing break before the brace and glob passes, and
  apply the leading tilde to the first field.
- `${X:-"" b}` loses the empty argument: keep a quoted-empty flag for
  each field.
- `SubShellKind::ParamWord` makes `eat_subshell` hold an unreachable
  arm (mordant `param_wider_than_callers`): keep the word context in
  its own lexer field. `append_var_argv -> bool` folds a
  `self_exe_path()` failure into "not set" (mordant
  `error_collapsed_to_bool`).
- `ParamHead.skip` and the `open_params` stack of the parser cost 5
  instructions for each `parse_atom` call. `Expansion::next` can find
  the matching `ParamEnd` with a depth counter.
- `if !head.op.falls_back_to_value() { out.truncate(len) }` is dead.
- `ends_in_param_ref` copies the head grammar of the lexer by hand:
  add a generated test that a plain interpolated string and a string
  reference give the same non-text tokens after `$` and `${` plus
  each tail of up to 2 bytes of `${}?:-+a1\#!@`. `$\FOO${"BAR"}`
  expands `$FOOBAR` on main too.
- Tests to add: `-` and `+` with a set, an empty and an unset
  parameter and with argument numbers; nested forms where the used
  and the unused word differ; the word guards (`[[`, `]]`, `#`, a
  digit before `>`, a quoted `}`); the table again in a script with a
  non-ASCII byte; a checked-in dash reference table.
- With `${NAME}`, an unset `"${X}"` gives no word: `[ -d "${X}" ]`
  passes and `cd "${X}"` goes to `$HOME`. #44904 makes it one empty
  word, so it must land first.
- A prefix assignment is not seen: `A=x B=${A:-y} cmd` gives `y`.

Commit 2 (parse errors): no maintainer asked for it. If it returns:
one line for each error, a kind on `LexError`, and the hint about
`--shell=system` printed by the caller only for a package.json
script. It also rejects text that runs today: `"${{ x }}"`, `"$#"`,
`sh -c "echo hi $@" -- a b`.
@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as stale: this PR conflicts with main, and its last commit is from 2026-10-16.

The branch stays, so the work is not lost. To pick it up again, rebase the branch on main and reopen this PR.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant