Creating a cluster in docker swarm

[nafajardo]

Hi all,

I am trying to make a cluster in docker swarm, but am having no headway and cant seem to wrap my head around why my methods are not working.

I have a two node cluster, and am trying to use the following dockercompose:

networks:
  casnet:
    driver: overlay
services:
  cassandra-masters:
    image: cassandra:latest
    environment:
      - CASSANDRA_BROADCAST_ADDRESS = cassandra-masters
    deploy:
      mode: replicated
      replicas: 2
    networks:
      - casnet
  cassandra-slaves:
    image: cassandra:latest
    environment:
      - CASSANDRA_SEEDS = cassandra-masters
      - CASSANDRA_BROADCAST_ADDRESS = cassandra-slaves
    deploy:
      mode: replicated
      replicas: 3
    depends_on:
      - cassandra-masters
    networks:
      - casnet

The environment variables used to be static IPs of the nodes, but after switching to an overlay network it needs to grab the node IP dynamically. I would appreciate help, or a working dockercompose file ;)

[nafajardo]

Using the solution from #94 doesn't work on a multi node cluster (or at least my multi node cluster).

[yosifkit]

It might be because of #150.

[nafajardo]

Correct me if I am wrong; I took a look at that issue, and it seems like he needed to make a modification to the Dockerfile entrypoint script. This modification was made due to the IP address being assigned to the Cassandra instance being the loopback IP and not the one used for communication between nodes (even though he was running a single node Cassandra instance).

I think this can be considered separate from my issue since I just want to get a working dockercompose file on an instance of Dockerswarm running X nodes, and not necessarily using an overlay network. So even putting all the services (or raw containers depending on whatever the final solution is) on a network that has a driver set as host is viable.

I am able to get a working Cassandra Ring with the following commands:

Ran on node A:
docker run --name casseed -d -e CASSANDRA_BROADCAST_ADDRESS=<node A IP address> -p 7000:7000 cassandra:latest

Ran on node B:
docker run --name casslave -d -e CASSANDRA_BROADCAST_ADDRESS=<node B IP address> -p 7000:7000 -e CASSANDRA_SEEDS=<node A IP address> cassandra:latest

[tianon]

I was able to get this working by simplifying your YAML to the following:

services:
  cassandra-masters:
    image: cassandra:latest
    environment:
      - MAX_HEAP_SIZE=128m
      - HEAP_NEWSIZE=32m
      - CASSANDRA_BROADCAST_ADDRESS=cassandra-masters
    deploy:
      mode: replicated
      replicas: 1
  cassandra-slaves:
    image: cassandra:latest
    environment:
      - MAX_HEAP_SIZE=128m
      - HEAP_NEWSIZE=32m
      - CASSANDRA_SEEDS=cassandra-masters
      - CASSANDRA_BROADCAST_ADDRESS=cassandra-slaves
    deploy:
      mode: replicated
      replicas: 1
    depends_on:
      - cassandra-masters

By using replicas: 2 on that first entry, you're asking Docker to start up two instances of Cassandra, and they don't have a CASSANDRA_SEEDS value so they won't set up any clustering (so you're starting off with split-brain right off the bat before you even attempt to add any "slaves"). Your second stanza will then start up three new Cassandra instances, which will then randomly join one of the two initial instances.

Also, I believe the spaces around the = in your environment: sections are a typo -- I don't believe either Compose or Swarm support spaces around the = there.

Given that this works without compose (and that I'm able to get it working with compose), I'm going to close, and would suggest that if you need further help getting this working, you might want to try the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

[nafajardo]

Hey sorry about the late response, but thank you so much! I am not currently in the same development environment, but I will be sure to point this resource to the team that took this on. Thank you again!