Startup error

[measwel]

I am trying to start a cassandra node in docker with:

 DC1_CASSANDRA_SEED:
      image: cassandra:3.10
      # image: 'bitnami/cassandra:latest'
      # sleep if starting for the first time. For each next node, sleep extra 60 sec.
      command: bash -c 'if [ -z "$$(ls -A /var/lib/cassandra/)" ] ; then sleep 0; fi && /docker-entrypoint.sh cassandra -f'
      
      # map local folder for data persistency
      volumes:
        - ./volumes/cassandra_seed_data:/var/lib/cassandra:rw
        # - ./volumes/cassandra_seed_data:/bitnami 
      # Docker constainer environment variable. We are using the
      # CASSANDRA_CLUSTER_NAME to name the cluster. This needs to be the same
      # across clusters. We are also declaring that DC1_CASSANDRA_SEED is a seed node.
      environment:
          - CASSANDRA_CLUSTER_NAME=arbotti_cluster
          - CASSANDRA_SEEDS=DC1_CASSANDRA_SEED
          - CASSANDRA_PASSWORD_SEEDER=yes
          - CASSANDRA_PASSWORD=password123  
      # Exposing ports for inter cluster communication
      expose:
            - 7000
            - 7001
            - 7199
            - 9160
      # Exposing ports on host => host_port:docker_port
      ports:
        - "9042:9042"
      # Cassandra ulimt recommended settings
      ulimits:
          memlock: -1
          nproc: 32768
          nofile: 100000
      
      # labels:
      # kompose.service.type: nodeport
        
      networks:
        - arbotti_DC1

Every time this ends with the error:
ERROR [main] 2017-11-28 22:17:46,792 CassandraDaemon.java:752 - Has no permission to create directory /var/lib/cassandra/data

What should I do to enable the Daemon to make the directory?

[tianon]

Before we step down from root, we adjust permissions on /var/lib/cassandra appropriately:

cassandra/3.11/docker-entrypoint.sh

Line 11 in b77e932

chown -R cassandra /var/lib/cassandra /var/log/cassandra "$CASSANDRA_CONFIG"

So, I'm not sure how this could fail. 😕

[measwel]

1. Does the directory have to be there? There is no /var/lib/cassandra on my system.
2. Does there have to be a cassandra user?

[tianon]

Those are both inside the container.

[measwel]

I wasn't sure. Could it be that the docker image itself has wrong permissions?

[measwel]

Cassandra starts okay, when I map a named volume instead of a volume on the host:

      volumes:
        # - ./volumes/cassandra_seed_data:/var/lib/cassandra:rw 
        - cassandra_data:/var/lib/cassandra:rw  <== THIS WORKS

As I understand, I have to map a volume on the host to have data persistency?
But mapping a volume on the host leads to:

ERROR [main] 2017-11-28 22:17:46,792 CassandraDaemon.java:752 - Has no permission to create directory /var/lib/cassandra/data

[tianon]

Are you running on Docker for Windows, Docker for Mac, or Docker Toolbox (and thus Docker Machine with vboxsf file share)? Permissions in those environments are always a bit of a struggle.

[measwel]

I am running Docker in Vagrant VM ( Ubuntu 17.04 ).
I can add that the problem seems to happen only for cassandra ( also for the bitami image by the way ). Other containers map their data fine to host directories.

I noticed this:

DC1_CASSANDRA_SEED_1  | INFO  [main] 2017-11-28 23:08:32,455 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml
                        time="2017-11-28T23:08:32Z" level=error msg="Unable to fetch disk info for &cmd.retryStorage{remoteStorage:(*cmd.networkStorage)(0xc420164c10), maxRetryAttempts:1, retryUnit:1000000000, retryCap:30000000000, offline:false, offlineTimestamp:time.Time{wall:0x365af809, ext:63647507311, loc:(*time.Location)(nil)}}" cause="disk not found" source="[xl-v1.go:199:getDisksInfo()]" 

[qedrakmar]

Let's confirm a couple things:

If you run it as normal, and attach to the container, can you see a /var/lib/cassandra ? If so, what are the permissions on it?
Does "./volumes/cassandra_seed_data" exist relative to where the compose file is running?
What are the permissions on the . and volumes and cassandra_seed_data directories?
Is this folder completely contained within the VM, or is any part of the tree a shared folder from the host to the VM?

[measwel]

1. Permissions inside running container:
   4 drwxrwxrwx 6 cassandra cassandra 4096 Nov 30 22:24 cassandra
   4 drwxr-xr-x 8 cassandra cassandra 4096 Nov 30 22:25 data

2. ./volumes/cassandra_seed_data exists when I try to mount the local directory with permissions:
   0 drwxr-xr-x 1 vagrant vagrant 68 Dec 1 16:39 cassandra_seed_data

3. The local cassandra_seed_data is part of a folder which is being shared with the host machine. Perhaps that is the root of the problem? Should I map it to a folder which is not shared with the host?

UPDATE: I can confirm now that the problem is gone where I map a local directory OUTSIDE the directory which is mapped to the host machine. :)

[qedrakmar]

As mentioned in #48 (comment), the shared folder behavior could be circumvented by using NFS from the host into the VM instead of Vagrant's shared folders. (Even though the other issue references VBox, it's about the same behavior for Vagrant.)

[measwel]

Thank you. My problem is solved by simply mapping the database to a folder outside the shared folders. I am fine with that.