bind mount crashing

[michaellopez]

I'm trying to use a host bind mount to store mariadb data. It crashes when I bind mount from the host.

I'm changing uid and gid on mysql user and group inside the image to match what I have in OS X. This is what I'm trying to debug with:

$ mkdir data
$ sudo chmod 777 data
$ docker run --rm -it -v $(pwd)/data:/var/lib/mysql -e MYSQL_RANDOM_ROOT_PASSWORD=1 mariadb:10.1.14 /bin/bash
root@e4a910a2dc22:/# usermod -o -u 501 mysql #501 is my uid in os x
root@e4a910a2dc22:/# groupmod -o -g 20 mysql #20 is my gid in os x
root@e4a910a2dc22:/# mysql_install_db --user=mysql --datadir="/var/lib/mysql" --rpm
2016-05-16 10:38:15 139680758392768 [Note] /usr/sbin/mysqld (mysqld 10.1.14-MariaDB-1~jessie) starting as process 80 ...
2016-05-16 10:38:15 139680758392768 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-05-16 10:38:15 139680758392768 [Note] InnoDB: The InnoDB memory heap is disabled
2016-05-16 10:38:15 139680758392768 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-05-16 10:38:15 139680758392768 [Note] InnoDB: Memory barrier is not used
2016-05-16 10:38:15 139680758392768 [Note] InnoDB: Compressed tables use zlib 1.2.8
2016-05-16 10:38:15 139680758392768 [Note] InnoDB: Using Linux native AIO
2016-05-16 10:38:15 139680758392768 [Note] InnoDB: Using SSE crc32 instructions
2016-05-16 10:38:15 139680758392768 [Note] InnoDB: Initializing buffer pool, size = 256.0M
2016-05-16 10:38:15 139680758392768 [Note] InnoDB: Completed initialization of buffer pool
2016-05-16 10:38:15 139680758392768 [ERROR] InnoDB: Tried to read 16384 bytes at offset 81920. Was only able to read 0.
2016-05-16 10:38:15 7f09f5fc67c0  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: Fatal error: cannot read from file. OS error number 13.
2016-05-16 10:38:15 7f09f5fc67c0  InnoDB: Assertion failure in thread 139680758392768 in file os0file.cc line 3177
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
160516 10:38:15 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.14-MariaDB-1~jessie
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=0
max_threads=102
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759828 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560a17fe98be]
/usr/sbin/mysqld(handle_fatal_signal+0x34d)[0x560a17b2b92d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f09f5ba88d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f09f3c99067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f09f3c9a448]
/usr/sbin/mysqld(+0x76ba6e)[0x560a17cf5a6e]
/usr/sbin/mysqld(+0x85bc07)[0x560a17de5c07]
/usr/sbin/mysqld(+0x7e6e24)[0x560a17d70e24]
/usr/sbin/mysqld(+0x70bc10)[0x560a17c95c10]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x64)[0x560a17b2db94]
/usr/sbin/mysqld(+0x42b23a)[0x560a179b523a]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x92a)[0x560a179b6a0a]
/usr/sbin/mysqld(+0x388356)[0x560a17912356]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x9e3)[0x560a179165b3]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f09f3c85b45]
/usr/sbin/mysqld(+0x38008d)[0x560a1790a08d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Installation of system tables failed!  Examine the logs in
/var/lib/mysql for more information.

The problem could be conflicting information in an external
my.cnf files. You can ignore these by doing:

    shell> /usr/scripts/scripts/mysql_install_db --defaults-file=~/.my.cnf

You can also try to start the mysqld daemon with:

    shell> /usr/sbin/mysqld --skip-grant --general-log &

and use the command line tool /usr/bin/mysql
to connect to the mysql database and look at the grant tables:

    shell> /usr/bin/mysql -u root mysql
    mysql> show tables;

Try 'mysqld --help' if you have problems with paths.  Using
--general-log gives you a log in /var/lib/mysql that may be helpful.

The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
MariaDB is hosted on launchpad; You can find the latest source and
email lists at http://launchpad.net/maria

Please check all of the above before submitting a bug report
at http://mariadb.org/jira

root@e4a910a2dc22:/# ls -la /var/lib/mysql/
total 110596
drwxrwxrwx  6 mysql dialout      204 May 16 10:38 .
drwxr-xr-x 17 root  root        4096 May 12 18:05 ..
-rw-rw----  1 mysql dialout 50331648 May 16 10:35 ib_logfile1
-rw-rw----  1 mysql dialout 50331648 May 16 10:35 ib_logfile101
-rw-rw----  1 mysql dialout 12582912 May 16 10:35 ibdata1
drwx------  2 mysql dialout       68 May 16 10:35 mysql
root@34997a7e4238:/# id mysql
uid=501(mysql) gid=20(dialout) groups=20(dialout)
root@34997a7e4238:/# cat /etc/group | grep mysql
mysql:x:20:

As you can see, it seems to write initial files properly, but InnoDB seems to choke on reading them, claiming it does not have access rights to the directory. What else can I do to grant it access?

docker run --rm -e MYSQL_RANDOM_ROOT_PASSWORD=1 mariadb:10.1.14 works without problem.

$ docker -v
Docker version 1.11.1, build 5604cbe
$ docker-machine -v
docker-machine version 0.7.0, build a650a40

[michaellopez]

@yosifkit @tianon
Is there any documentation and/or examples of the proper way of using this --user flag from #59? Is there anything else that needs to be done for it to work properly with bind mounts form host? Do I understand correctly that 1000:50 is uid and gid of the docker user in boot2docker? But shouldn't my users uid and gid be used instead?

Whatever I try, I can't get this to work 😢 . I think that many more people should be complaining if this was not working, so what am I missing?

$ docker pull mariadb:10.1.14
10.1.14: Pulling from library/mariadb

Digest: sha256:4ba05f5d84328374f938895eb7a0d64ad340d245b44983d7cfb22f3274cf655f
Status: Image is up to date for mariadb:10.1.14
$ docker run -e MYSQL_RANDOM_ROOT_PASSWORD=1 -v $PWD/data:/var/lib/mysql mariadb:10.1.14 --user 1000:50
$ echo $?
1
$ docker ps -a
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                     PORTS                                                                NAMES
0c7921fc1b20        mariadb:10.1.14                       "docker-entrypoint.sh"   3 seconds ago       Exited (1) 2 seconds ago                                                                        nostalgic_noyce
$ docker logs 0c7921fc1b20
$

I don't get any logs or error messages. Trying to debug it interactively it seems that it chokes on line 26 in docker-entrypoint.sh

With my limited bash knowledge, I could make it work by changing line 26 to DATADIR="$(echo $(_datadir "$@"))" or removing -o pipefail from the line 2

And even after doing the workarounds above, I'm back to my problems in this issue. Any help would be appreciated. Thanks.

[yosifkit]

Well, to use the --user flag it would need to be passed to docker, not mysql:

docker run --user 1000:50 -e MYSQL_RANDOM_ROOT_PASSWORD=1 -v $PWD/data:/var/lib/mysql mariadb:10.1.14

That shouldn't be any different than when you were changing the uid/gid of the mysql user.

[michaellopez]

@yosifkit After days of debugging I think I have finally found what was causing my issue. I thought I would document it here so that others may benefit from my struggle, I hope you don't mind.

Now that I fully understand #59 I can say that it did indeed help a lot, thanks. But it was only one of two pieces of my puzzle.

We use NFS backed volumes both for local development and production. My initial error in this issue was caused by the combination of NFS backed volumes and innodb_flush_method = O_DIRECT in /etc/mysql/my.cnf.

Running sed -i 's/^innodb_flush_method/#innodb_flush_method/' /etc/mysql/my.cnf before starting the daemon allows our containers to start up properly and function normally as far as we can initially tell. I did a quick google on innodb_flush_method and the information I found is really above my head, so I can't really tell with confidence what this change brings about internally, but I think it is "data integrity is not as safe as with O_DIRECT". But we'll test this out and respond in this thread again if something goes bad.

@yosifkit Do you now the impact of removing this configuration? Can it be merged into the official image?

[yosifkit]

@michaellopez, we don't want to change the default config. You should be able to just add --innodb_flush_method=NULL to your cmd and not have to do any seding.

$ docker run -d -e MYSQL_ROOT_PASSWORD=foo mysql --innodb_flush_method=NULL

[michaellopez]

@yosifkit Thanks for the tip!