Windows: Command Failed at startup when the username contains ampersands

[jviotti]

This happens as soon as you open the app:

Command failed: C:\Users\Foo & Bar\AppData\Local\Temp\9DB4.tmp.bat 'C:\Users\Foo' is not recognized as an internal or external command, operable program or batch file. The system cannot find the path specified. (code 1

This can be more easily reproduced by placing drivelist in a directory containing an ampersand and executing the example file:

git clone https://github.com/resin-io-modules/drivelist
move drivelist "Foo & Bar"
cd "Foo & Bar"
node example.js

This file is executed by child_process.execFile(). Simple tricks like quoting the whole path don't work. I also tried escaping the ampersand with ^&.

I did make it work by setting the execFile cwd option to the directory name of the script and then simply executing the basename, however that won't play well with asar archives. And to put another constraint to this: .execFile is the only spawning function we can use, since its the only one that is patched by Electron.

[lurch]

Blimey, sounds fairly nasty. Do we know if this has affected any users in the wild? I guess the nature of this bug (and the related ones) means that Etcher would fail before it gets the chance to log any errors?

[lurch]

FWIW, if I try replicating your example above on Linux (in /tmp), it works okay as long as I do npm install after doing cd "Foo & Bar". These all work:

node example.js
cd ..
node ./Foo\ \&\ Bar/example.js
node Foo\ \&\ Bar/example.js
node "Foo & Bar"/example.js
node "Foo & Bar/example.js"
cd ~
node /tmp/Foo\ \&\ Bar/example.js
node "/tmp/Foo & Bar/example.js"
node /tmp/"Foo & Bar"/example.js

and as we're not using any 'special' characters or variables here, these all work identically with double or single cream... I mean quotes! 😉

[jviotti]

Blimey, sounds fairly nasty. Do we know if this has affected any users in the wild? I guess the nature of this bug (and the related ones) means that Etcher would fail before it gets the chance to log any errors?

Yeah, I discovered this one through Sentry :/ The error happens after the app is started, so Corvus does get a chance to report it.

[lurch]

I've been attempting to dig into this some more, using drivelist as a nice & small test case...
(in the hope that this might also help with #1376 )

On Linux, it looks like I can use any crazy characters in the pathname that I like, and everything "just works" :

cd /tmp/Foo\ \!\"£\$%\^\&\*\(\)\'\\\:\;@#~\`\|\\\[\]\{\}\?\<\>-_\=+\$\{PATH\}\ Bar/
$ node example.js 
[
  {
    "device": "/dev/sda",
    "description": "SanDisk SD8SN8U1",
    "size": 1024209543168,
    "mountpoints": [
      {
        "path": "/"
      },
      {
        "path": "/home"
      },
      {
        "path": "/boot/efi"
      },
      {
        "path": "/var/lib/docker/aufs"
      }
    ],
    "raw": "/dev/sda",
    "protected": false,
    "system": true
  }
]

Without all the crazy backslash-escapes auto-created by tab-completion, the actual directory name looks like

/tmp/Foo !"£$%^&*()'\:;@#~`|\[]{}?<>-_=+${PATH} Bar

and from my home-dir, I can run any of:

node '/tmp/Foo !"£$%^&*()'\''\:;@#~`|\[]{}?<>-_=+${PATH} Bar/example.js'
node /tmp/Foo\ \!\"£\$%\^\&\*\(\)\'\\\:\;@#~\`\|\\\[\]\{\}\?\<\>-_\=+\$\{PATH\}\ Bar/example.js
node "/tmp/Foo "\!"\"£\$%^&*()'\\:;@#~\`|\\[]{}?<>-_=+\${PATH} Bar/example.js"
node "/tmp/Foo "'!'"\"£\$%^&*()'\\:;@#~\`|\\[]{}?<>-_=+\${PATH} Bar/example.js"

with identical results. (apparently the ! causes problems (history expansion) in double-quoted strings in an interactive terminal, but not in shell scripts )

On Windows, these are the illegal filename-characters: \ / : * ? " < > |, and here's what I found (about non-illegal filename characters) from my investigations in a Windows7 VM with nodejs v6.10.3 installed :

Folder	node example.js (from within the folder)	node folder\example.js	folder\scripts\win32.bat
FooBar	works	C:\FooBar\... "C:\FooBar\..." C:\Foo""Bar\...	C:\FooBar\... "C:\FooBar\..." C:\"FooBar"\... C:\Foo""Bar\...
Foo Bar	works	"C:\Foo Bar\..." C:\Foo" "Bar\...	"C:\Foo Bar\..." C:\"Foo Bar"\... C:\Foo" "Bar\... "C:\Foo"" ""Bar\..."
Foo&Bar	fails	nothing works	"C:\Foo&Bar\..." C:\"Foo&Bar"\ C:\Foo^&Bar\ C:\Foo"&"Bar\
Foo & Bar
Foo!Bar
Foo$Bar
Foo%Bar
Foo(Bar
Foo)Bar
Foo[Bar
Foo]Bar
Foo{Bar
Foo}Bar
Foo;Bar
Foo@Bar
Foo^Bar
Foo_Bar
Foo`Bar
Foo~Bar
Foo£Bar
Foo+Bar
Foo=Bar
Foo'Bar
Foo-Bar

(WIP... will complete later)

In each case, the folder is a clone of the drivelist repo, inside of which I've done npm install (which I've then copy'n'pasted to a whole bunch of different names). Explaining the first row of the big table above...
For the first row, I can cd FooBar (first column), and node example.js works (second column). I can then cd %USERPROFILE% and node C:\FooBar\example.js and node "C:\FooBar\example.js" both work (third column). Still in my home-dir, C:\FooBar\scripts\win32.bat, "C:\FooBar\scripts\win32.bat" and C:\"FooBar"\scripts\win32.bat all work (fourth column) indicating that cmd.exe allows extra syntaxes that node.exe doesn't. Obviously some of the escaping I'm using is quite contrived, I'm just trying to get a feel for what syntaxes do and don't work in what contexts ;-)

Possibly relevant / useful links:
sindresorhus/open#47
nodejs/node#6803

...I've been trying to play around with this (while trying very hard to to fall too deep down the rabbit-hole! 🐰 ) and I'm beginning to suspect that this mis-handling of & might actually be a bug isolated to the child-process execFile function in NodeJS only on Windows... :-/ (Linux always works, and AFAICT from playing around in the node REPL, &s in the folder-name do work on Windows if I use .exec instead of .execFile - which obviously doesn't help much in light of @jviotti 's ".execFile is the only spawning function we can use, since its the only one that is patched by Electron." comment. I'm beginning to wonder if using cmd /c ... with .execFile might be a workaround?)

[jviotti]

Thanks a lot for helping me out with this.

I'm beginning to wonder if using cmd /c ... with .execFile might be a workaround?)

The problem is still asar archives. The only way to execute a file inside an asar is to pass its path as the first argument to .execFile (since Electron understands that, and will handle extraction). Any other variations means that the file will not execute.

---

There is a solution though, which will actually make my work on the CLI way easier: inline the drivelist scripts in JavaScript files that manually extract them out and execute them.

The logic put there can execute the file using my cwd workaround, and then it all should work. What do you think?

[lurch]

The problem is still asar archives. The only way to execute a file inside an asar is to pass its path as the first argument to .execFile (since Electron understands that, and will handle extraction). Any other variations means that the file will not execute.

Ahhh, thanks. I didn't realise that the .execFile restriction was due to the way that Etcher dynamically unpacks things from the asar.

There is a solution though, which will actually make my work on the CLI way easier: inline the drivelist scripts in JavaScript files that manually extract them out and execute them.

Huh? That sounds hideous :-/ But I guess it might work, as long as we don't run into permissions / threading / forking issues.
However I wouldn't want to see the drivelist scripts 'manually' embedded inside other Javascript files, as that would make them totally unmaintainable. We should continue to edit the platform-specific scripts as standalone files, and then have some step of the build-process that then embeds them inside another JS file (e.g. as base64 or something).

[jviotti]

Check balena-io-modules/drivelist#164. I created a small script that creates a JSON file out of them, so we can still keep separate for developing purposes. I'll test this thoroughly before merging.