Symlink Emulation fails.

[fduarte42]

Expected behavior

Symlinked file should show content of original file

Actual behavior

Sometimes container sees the following content:
XSym 0021 607cac914a72ec8c3c3f1d6d7eb98214 ../../../autoload.php

If I enter the bash in the container and remove the file, the file is corrupted and a listing shows this
root@de7aff095c91:/var/www/html/vendor/typo3/cms/vendor# ls -la
ls: cannot access autoload.php: No such file or directory
total 4
drwxr-xr-x 2 root root 0 Oct 8 11:27 .
drwxr-xr-x 2 root root 4096 Oct 8 11:25 ..
-????????? ? ? ? ? ? autoload.php

And now windows can't do anything with the file either. After a chkdsk /F the corruption ist gone.

I testet this on 2 different machines and also in a VM. Same problems.

Information

I used my container image fduarte/docker-php based on library/php:apache:7.0 to install Typo3. I see this if I start to reload the backend over and over again. Sometimes it happens. After a reload it is gone again.

I'm using docker for windows 1.12.2-rc1 beta27.1 (build 7538) on Windows 10 pro 1607 (build 14393.222)

[cuchac]

We have exactly same problem. We trigger this bug often by using symlinks in git repositories. When switching between different branches containing different symlinks, we almost always trigger this unaccessable/unremovable symlink. We have to restart docker to fix this.

[embray]

I am seeing a similar issue. This happened with me when running the autoconf ./configure script for a particular project that was, for various reasons, making config.log into a symlink. It can be reproduced as follows:

In C:\test I have two files:
configure.ac:

AC_INIT([test], [1.0], [[email protected]])
AC_OUTPUT([Makefile])

and Makefile.in which is empty (it doesn't matter either way).

Now I run docker -t -i --rm -v C:\test:/test ubuntu:16.10

In the container:

# cd /test
# apt-get update && apt-get -y install autoconf
# autoconf  # should generate `configure`
# mkdir logs && ln -s logs/config.log config.log
# ./configure

Expected Behavior

A normal, successful run of this barebones ./configure outputs:

# ./configure
configure: creating ./config.status
config.status: creating Makefile

This is the result I get in the same Docker container when not running in a host-mounted volume.

Actual Behavior

# ./configure
configure: creating ./config.status
./config.status: line 506: config.log: Too many levels of symbolic links

The error does't actually make sense since it's not a recursive symlink or anything. If I try to read the symlink (e.g. $ cat config.log) I get the expected contents of config.log (up to where the error occurred). Now the real mystery: If I delete the file pointed to by the symlink and re-run ./configure I get:

# rm logs/config.log
# ./configure
configure: creating ./config.status
config.status: creating Makefile

so it seems to work. However:

# ls -l config.log
-rwxr-xr-x 1 root root 3000 Feb 16 14:27 config.log

the symlink is replaced with what appears to be a normal file. logs/config.log does not exist at all. If I read the file with cat config.log it has the expected contents. However, if I then delete it:

# rm -f config.log
# ls -l
# ls -l
total 113
-rwxr-xr-x 1 root root     0 Feb 16 14:27 Makefile
-rwxr-xr-x 1 root root     0 Feb 16 14:20 Makefile.in
drwxr-xr-x 2 root root     0 Feb 16 14:21 autom4te.cache
-rwxr-xr-x 1 root root  3000 Feb 16 14:27 config.log
-rwxr-xr-x 1 root root 23799 Feb 16 14:27 config.status
-rwxr-xr-x 1 root root 83812 Feb 16 14:21 configure
-rwxr-xr-x 1 root root    65 Feb 16 14:19 configure.ac
drwxr-xr-x 2 root root     0 Feb 16 14:26 logs
# cat config.log
cat: config.log: No such file or directory

the dirent for it keeps coming back, but the file itself is missing. It can't be written to either:

# echo "Hello" > config.log
bash: config.log: No such file or directory

and it otherwise exhibits the same behaviors as the OP mentioned (sometimes I also get the ????s in the ls output).

The file is visible from Windows, but cannot be deleted, opened, edited, or have its permissions changed in any way. It's completely corrupt. In my Cygwin shell it shows as:

-rw-r-----  1 Unknown+User Unknown+Group 3.0K Feb 16 14:02 config.log

Restarting the Docker VM seems to fix it sometimes. Other times I can only delete the file after a full reboot.

This all seems to have something to do with the fact that the ./configure script does some complicated redirections to config.log which, in my case, is a symlink. I wasn't able to come up with a simpler example though. My wild guess is that this is actually a bug in the cifs kernel module in Linux but I'm not sure.

[rn]

@embray thanks for the detailed report. I can repro this with the latest beta on the latest windows insider build. FWIW I can also reproduce this directly in the Linux VM, ie not inside a container.

We use SMB/CIFS mounts from Linux and there we specify the mfsymlinks option to emulate symlinks on the Linux side only. That's the reason you can't see them on the Windows host.

However, the other issues, including the inability to remove files/symlinks seem like a bug in the Linux kernel implementation + interaction with the CIFS server on the host.

The Linux kernel code implementing the symlink emulation has been largely unchanged since 2014 but there seems to be a bug somewhere...

[embray]

FWIW I can also reproduce this directly in the Linux VM, ie not inside a container.

Cool--I was planning on testing that as well but had to move on to other things for today. That definitely increases my suspicion that this is actually a Linux bug.

[rn]

@embray I could reproduce the issue on a Ubuntu Linux VM and will send details to the relevant linux kernel mailing list. I'd like to credit you for the easy repro so if you send me your email address I'll cc you (otherwise will just refer to your GH profile). My email address is rolf<dot>neugebauer<at>docker<dot>com

[rlpowell]

Any idea what's going on here? This burns me at least once a day, forcing a docker restart. Did this get pushed out to lkml, and if so, can we get a link to the thread?

[embray]

Back in February @rneugeba sent an e-mail about it to a couple mailing lists (and CC'd me) but I don't think it ever got any respone? And now that I look, I don't see it in the archives of those mailing lists so I wonder if it got bounced.

[rn]

There are two mailing lists, one is closed and got bounced. The other is open but I never got a response. The mailing list archives are hard to find. Let me check again...

[stucki]

Please try to use the master branch of https://github.com/TYPO3/CmsComposerInstallers. It should no longer create the problematic file vendor/autoload.php as a symlink.

composer require typo3/cms-composer-installers "~1.3.0@dev"

[fduarte42]

If we add the requirement with composer require we get the following error, but adding it manually to to composer.json works. (hint: the control.cmd is a wrapper script for our docker image, the command ist executed inside the image as normal)

[doxxx]

Another method to resolve the corruption is to use the net sessions command on the Windows host to forcibly disconnect the Linux VM's SMB connection. In an Administrator mode Command or Powershell window, run net sessions to find the computer name that's connected -- usually something like \\10.0.75.2. Then run net sessions <computer name> /delete to disconnect it. You should now be able to delete the files on the Windows host.

[SimonWpt]

Nothing new about a solution for the bug?

[mterrel]

+1 I see this corrupt link file very consistently when using npm inside a docker container on Windows where one particular package.json intentionally references a dependency by a local (relative) path.

[docker-robott]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[rfay]

/remove-lifecycle stale

[MK-42]

currently trying to reproduce this issue outside of our composer installation.

Looks like that symlinks created inside containers get unreliable under heavy io-load, not presenting the actual symlinked content, but a dummy-file

XSYM
[four digit number]
[some hexadecimal code]
[relative symlink-target path]

But I can't reproduce this standalone atm.

What occured during testing on a clean ubuntu-container after installing stress:
running stress --hdd 1000 --hdd-bytes 100 to create some I/O load on the mounted volume will lead to orphan files, that are listed via ls -la (and can be seen in windows explorer on the host), but produce a No such file or directory when cating or rming them (from inside the container). They also can't be deleted from windows explorer (error about permissions, asks for admin again and again, never deleting the file). running net sessions and disconnecting the linux vm via net sessions \\10.0.75.2 /delete made them disappear on their own. Probably some bug in the smb-service?

[andre-spindler]

I have another, similar issue:
I use typo3 and have a local project specific extension/package in my project path. Composer then creates the symlink to the package in the extension dir. But this symlink is not handled correctly and the extension isd not found. I have to set "options->symlink" to "false" to make composer create a copy of the package to get it work.

[docker-robott]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[scaytrase]

Still actual

Docker version 18.09.2, build 6247962
Docker desktop: Version 2.0.0.3 (31259)
Channel: stable
Build: 8858db3

/remove-lifecycle stale

[docker-robott]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[scaytrase]

Still actual
docker-compose version 1.23.2, build 1110ad01
Docker version 18.09.2, build 6247962

/remove-lifecycle stale

[rfay]

/lifecycle frozen