Enable filenames with colon (:) on drvfs

[doehrm]

Insider build 14986

Trying to create a filename with a colon on a drive mounted with drvfs is not working whereas I can create the file on lxfs:

$ mount | grep home
home on /home type lxfs (rw,noatime)

$ touch foo+bar:0185.somext

$ ls -l
-rw-rw-rw- 1 doehrm doehrm 0 Dec 16 23:26 foo+bar:0185.somext

$ mount | grep \/e
E: on /mnt/e type drvfs (rw,noatime)

$ touch foo+bar:0185.somext
touch: setting times of 'foo+bar:0185.somext': No such file or directory

$ echo "Foo Bar" > foo+bar:0185.somext
bash: foo+bar:0185.somext: Invalid argument

$ echo "Foo Bar" > foo+bar\:0185.somext
bash: foo+bar:0185.somext: Invalid argument

The drive connected is an external USB drive formatted with NTFS

Am I missing something?

--
Markus

[therealkenc]

Unfortunately NTFS defines a set of reserved characters, while Unix-ish filesystems (ext, ufs2, zfs, hfs), on which '/' is the only special character, does not. Part of that is because of Microsoft still being tied to the boat anchor that is DOS.

As an aside, Unix is tied somewhat to the boat anchor that is bash, and ":" is de-facto problematic also; because, there is no way to escape ':' in a PATH environment variable. Curious what your use case is, because you pretty much have to avoid ':' on Linux too (but by convention only).

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

[doehrm]

The use case is "chromium-os" ;-)

I build a special version for an older laptop of mine. The official build host is an Ubuntu 16.04.1. The whole build is done in a chroot'ed environment so basically only a few tools need to be installed on the OS (python, git, curl and a few others), the rest is downloaded and compiled in the chroot.

While downloading the repository descriptions I come across a few files that cannot be sync'ed because of the colon:

error: unable to create file frontend/client/src/autotest/public/Open+Sans:300.woff (Invalid argument)
error: unable to create file frontend/client/src/autotest/public/Roboto+Bold:700.woff (Invalid argument)
error: unable to create file frontend/client/src/autotest/public/Roboto+Light:300.woff (Invalid argument)
error: unable to create file frontend/client/src/autotest/public/Roboto+Medium:500.woff (Invalid argument)
error: unable to create file frontend/client/src/autotest/public/Roboto+Regular:400.woff (Invalid argument)
error: unable to create file server/site_tests/display_EdidStress/test_data/edids/weekly/SCT_272_STEELCASE_m:s_HDMI.txt

The original sources really contain that colon:

https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/frontend/client/src/autotest/public

My aim was to have a "somewhat native" possibility to compile a new version on my laptop rather than to have Hyper-V and virtual network cards and switches messing with my non-trivial network/VPN setup (if you ever set up half a dozen VPNs (SSTP, OpenVPN and PPTP mixed with ipv4/ipv6) and you install Hyper-V and you wonder, why nothing is working any more, you'll understand ;-) )

But I understand that it'd be problematic allowing it, despite I'm not looking to have it as a directory name but "just a file name".

[aseering]

Tangent -- : was the Mac path separator prior to OS X. Graphical Mac applications still allow filenames to contain / and not :. Basically, : is weird everywhere.

Anyway, it's an interesting question whether DrvFs should (for compatibility purposes) be allowed to construct filenames that wouldn't be allowed for regular Windows applications, and if so, what the semantics of such files should be.

[aseering]

@doehrm -- can you not build in WSL's $HOME? As you mention, it supports :. And it should have better performance as well.

[doehrm]

I still work on an OS that has filenames such as NODE::DKA0:[FOO.BAR.BAR2]foobar.dat;5 where a colon is a separator, no possibility to create a filename with a colon there and I guess, that's also where deep in the Windows (NT 4.0) kernel it [was|is] forbidden too.

And yes, I'll try that next - as soon as I can get rid of some older insider builds still stored on the disk and make some space.

[doehrm]

Interestingly enough, I could clone the repository using GIT on Windows:

C:\Users\curry\test>..\Git\bin\git.exe clone https://chromium.googlesource.com/chromiumos/third_party/autotest
Cloning into 'autotest'...
remote: Sending approximately 434.56 MiB ...
remote: Counting objects: 9793, done
remote: Finding sources: 100% (16/16)
remote: Total 177259 (delta 128226), reused 177251 (delta 128226)
Receiving objects: 100% (177259/177259), 434.38 MiB | 698.00 KiB/s, done.
Resolving deltas: 100% (128226/128226), done.
Checking out files: 100% (9006/9006), done.

and it seems there is some sort of semantics already existing:

c:\Users\curry\test\autotest\frontend\client\src\autotest\public>dir /r
 Volume in drive C has no label.
 Volume Serial Number is 8203-5049

 Directory of C:\Users\curry\test\autotest\frontend\client\src\autotest\public

17.12.16  02:01    <DIR>          .
17.12.16  02:01    <DIR>          ..
[...]
17.12.16  02:01                 0 Open+Sans
                           21 744 Open+Sans:300.woff:$DATA
17.12.16  02:01                 0 Roboto+Bold
                           19 812 Roboto+Bold:700.woff:$DATA
17.12.16  02:01                 0 Roboto+Light
                           21 080 Roboto+Light:300.woff:$DATA
17.12.16  02:01                 0 Roboto+Medium
                           20 636 Roboto+Medium:500.woff:$DATA
17.12.16  02:01                 0 Roboto+Regular
                           21 132 Roboto+Regular:400.woff:$DATA
[...]
              21 File(s)         66 494 bytes
               2 Dir(s)  46 544 334 848 bytes free

A plain dir shows the first line (size 0, short filename up to the colon), dir /r reveals the complete name with a "$DATA" at the end.

[fpqc]

@therealkenc The real reason doesn't have anything to do with DOS iirc. The ":" character is reserved for alternate data streams in NTFS and actually appears to be related maybe to the ancient OS/2 subsystem.

[doehrm]

I'd not say OS/2 but rather VMS (OpenVMS). Windows and VMS share quite a few things (http://www3.sympatico.ca/n.rieck/docs/Windows-NT_is_VMS_re-implemented.html).

[doehrm]

I was now able to download all the code to $HOME. When I try to enter the chroot the system either freezes (and I have to hard reboot) or BSODs :-( with APC_INDEX_MISMATCH.

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 1, {7fce84d0c357, 0, 1, ffffc701f5f0dc80}

Page 10ad38 not present in the dump file. Type ".hh dbgerr004" for details
Probably caused by : ntkrnlmp.exe ( nt!KiSystemServiceExitPico+194 )

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

APC_INDEX_MISMATCH (1)
This is a kernel internal error. The most common reason to see this
bugcheck is when a filesystem or a driver has a mismatched number of
calls to disable and re-enable APCs. The key data item is the
Thread->CombinedApcDisable field. This consists of two separate 16-bit
fields, the SpecialApcDisable and the KernelApcDisable. A negative value
of either indicates that a driver has disabled special or normal APCs
(respectively) without re-enabling them; a positive value indicates that
a driver has enabled special or normal APCs (respectively) too many times.
Arguments:
Arg1: 00007fce84d0c357, Address of system call function or worker routine
Arg2: 0000000000000000, Thread->ApcStateIndex
Arg3: 0000000000000001, (Thread->SpecialApcDisable << 16) | Thread->KernelApcDisable
Arg4: ffffc701f5f0dc80, Call type (0 - system call, 1 - worker routine)

Debugging Details:
------------------

Page 10ad38 not present in the dump file. Type ".hh dbgerr004" for details

FAULTING_IP: 
+8c0
00007fce`84d0c357 ??              ???

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT
BUGCHECK_STR:  0x1
CURRENT_IRQL:  0
ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre
LAST_CONTROL_TRANSFER:  from fffff80084d60929 to fffff80084d55240

STACK_TEXT:  
ffffc701`f5f0dab8 fffff800`84d60929 : 00000000`00000001 00007fce`84d0c357 00000000`00000000 00000000`00000001 : nt!KeBugCheckEx
ffffc701`f5f0dac0 fffff800`84d6082b : ffffc701`f5f0dc00 ffffb703`396c6200 00000000`00000000 00000000`00000000 : nt!KiBugCheckDispatch+0x69
ffffc701`f5f0dc00 00007fce`84d0c357 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceExitPico+0x194
00007fff`ce8dcf98 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007fce`84d0c357

STACK_COMMAND:  kb

FOLLOWUP_IP: 
nt!KiSystemServiceExitPico+194
fffff800`84d6082b 4883ec50        sub     rsp,50h

SYMBOL_STACK_INDEX:  2
SYMBOL_NAME:  nt!KiSystemServiceExitPico+194
FOLLOWUP_NAME:  MachineOwner
MODULE_NAME: nt
IMAGE_NAME:  ntkrnlmp.exe
DEBUG_FLR_IMAGE_TIMESTAMP:  58427289
BUCKET_ID_FUNC_OFFSET:  194
FAILURE_BUCKET_ID:  0x1_SysCallNum_0_nt!KiSystemServiceExitPico
BUCKET_ID:  0x1_SysCallNum_0_nt!KiSystemServiceExitPico
ANALYSIS_SOURCE:  KM
FAILURE_ID_HASH_STRING:  km:0x1_syscallnum_0_nt!kisystemserviceexitpico
FAILURE_ID_HASH:  {48293086-2975-82e2-87d1-2bab5cd63ecf}

I switched on a full dump and will reproduce the problem again (if that helps).

[fpqc]

@doehrm Please email secure@microsoft.com ATTN: Ben Hillis ( @benhillis ) or Russ Alexander ( @russalex )with a copy of a minidump.

Also, yes, that's right, alternate data streams is a VMS-inspired thing. Extended attributes are the OS/2 legacy.

[doehrm]

Done so, thanx @fpqc for pointing me in the direction and sorry for messing this issue up with nonrelated BSOD's.

Since NTFS seems to have a way to handle those files, doesn't/couldn't/shouldn't that mean DrvFS should too?

[fpqc]

@doehrm The path "a:b" refers to an alternate data stream of the file "a" on NTFS. It is not a valid path in Win32. Powershell has a cmdlet (streams) that lets you see what ADS are attached to a file. The default streamtype is simply called "$DATA" Look familiar?

https://msdn.microsoft.com/en-us/library/windows/desktop/aa364404(v=vs.85).aspx

iirc, GeoHot demonstrated an attack on Windows using ADS a few years ago.

[doehrm]

or it's internally "mapped" to a Unicode character (U+F000 - U+F0FF)...

[fpqc]

The result of your dir /r indicates that alternate data streams were stored.

It says "I created an empty file Open+Sans, then created an alternate $DATA stream for that file called 300.woff that contains the actual desired content of the file". Git for Windows shouldn't be doing that. It is completely broken behavior.

I just tested it myself. That's what it's doing. That's also probably why it's going crazy with BSODs. Nobody (external to Microsoft, at least) has, to my knowledge, tested DrvFS functionality with Alternate named $DATA streams thrown into the mix.

It works in LXFS because the WSL drivers are intercepting all File I/O within the environment and escaping the character as necessary when it ends up on the disk.

[benhillis]

@doehrm - I received your dump and took a look. The cause of the bluescreen was a locking issue in a terminal ioctl. The fix just reached the rs_prerelease branch so it should be in the next Windows Insider build.

Thank you for reporting the issue and sending us the memory dump!