Node stuck on UJ

[giladgaron]

Hello.
I'm trying to test a 4 node setup on 4 different VMs (Centos 7, Docker 1.10.1)
I have two seed nodes and two other nodes.
My startup is as follows (No data in Cassandra, fresh container):
1st node:
docker run --name cassandra1 -d -e CASSANDRA_BROADCAST_ADDRESS=10.26.194.33 -e CASSANDRA_SEEDS=10.26.194.33,10.26.194.47 -p 7000:7000 -p 9042:9042 cassandra:latest

• waiting for the node to become UN

2nd node:
docker run --name cassandra2 -d -e CASSANDRA_BROADCAST_ADDRESS=10.26.194.47 -e CASSANDRA_SEEDS=10.26.194.33,10.26.194.47 -p 7000:7000 -p 9042:9042 cassandra:latest

• waiting for the node to become UN

3rd node:
docker run --name cassandra3 -d -e CASSANDRA_BROADCAST_ADDRESS=10.26.194.78 -e CASSANDRA_SEEDS=10.26.194.33,10.26.194.47 -p 7000:7000 -p 9042:9042 cassandra:latest

• waiting for the node to become UN

4th node:
docker run --name cassandra4 -d -e CASSANDRA_BROADCAST_ADDRESS=10.26.194.16 -e CASSANDRA_SEEDS=10.26.194.33,10.26.194.47 -p 7000:7000 -p 9042:9042 cassandra:latest

The 4th node is stuck on UJ, no error are shown on the logs:

INFO  [main] 2016-02-14 17:17:09,173 StorageService.java:1181 - JOINING: sleeping 30000 ms for pending range setup
DEBUG [PendingRangeCalculator:1] 2016-02-14 17:17:09,223 PendingRangeCalculatorService.java:64 - finished calculation for 3 keyspaces in 49ms
DEBUG [GossipTasks:1] 2016-02-14 17:17:09,399 FailureDetector.java:293 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2016-02-14 17:17:09,399 FailureDetector.java:293 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2016-02-14 17:17:09,399 FailureDetector.java:293 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2016-02-14 17:17:10,401 FailureDetector.java:293 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2016-02-14 17:17:10,401 FailureDetector.java:293 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2016-02-14 17:17:10,401 FailureDetector.java:293 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2016-02-14 17:17:11,402 FailureDetector.java:293 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2016-02-14 17:17:11,402 FailureDetector.java:293 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2016-02-14 17:17:11,402 FailureDetector.java:293 - Still not marking nodes down due to local pause
DEBUG [GossipStage:1] 2016-02-14 17:17:25,428 FailureDetector.java:456 - Ignoring interval time of 2003749978 for /10.26.194.78
DEBUG [GossipStage:1] 2016-02-14 17:17:25,429 FailureDetector.java:456 - Ignoring interval time of 2627652047 for /10.26.194.47
DEBUG [GossipStage:1] 2016-02-14 17:17:28,432 FailureDetector.java:456 - Ignoring interval time of 2005226789 for /10.26.194.78
INFO  [main] 2016-02-14 17:17:39,176 StorageService.java:1181 - JOINING: Starting to bootstrap...
INFO  [main] 2016-02-14 17:17:39,744 StreamResultFuture.java:88 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Executing streaming plan for Bootstrap
DEBUG [main] 2016-02-14 17:17:39,744 StreamCoordinator.java:144 - Connecting next session d94d19d0-d33e-11e5-9066-091830ac5256 with 10.26.194.78.
INFO  [StreamConnectionEstablisher:1] 2016-02-14 17:17:39,753 StreamSession.java:238 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Starting streaming to /10.26.194.78
DEBUG [StreamConnectionEstablisher:1] 2016-02-14 17:17:39,753 ConnectionHandler.java:82 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Sending stream init for incoming stream
DEBUG [StreamConnectionEstablisher:1] 2016-02-14 17:17:39,771 ConnectionHandler.java:87 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Sending stream init for outgoing stream
INFO  [StreamConnectionEstablisher:1] 2016-02-14 17:17:39,779 StreamCoordinator.java:266 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256, ID#0] Beginning stream session with /10.26.194.78
DEBUG [STREAM-OUT-/10.26.194.78] 2016-02-14 17:17:39,779 ConnectionHandler.java:334 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Sending Prepare (3 requests,  0 files}
DEBUG [STREAM-IN-/10.26.194.78] 2016-02-14 17:17:39,827 ConnectionHandler.java:266 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Received Prepare (0 requests,  0 files}
DEBUG [STREAM-IN-/10.26.194.78] 2016-02-14 17:17:39,828 ConnectionHandler.java:266 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Received Complete
DEBUG [STREAM-IN-/10.26.194.78] 2016-02-14 17:17:39,828 ConnectionHandler.java:110 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Closing stream connection handler on /10.26.194.78
INFO  [STREAM-IN-/10.26.194.78] 2016-02-14 17:17:39,828 StreamResultFuture.java:185 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Session with /10.26.194.78 is complete
DEBUG [STREAM-OUT-/10.26.194.78] 2016-02-14 17:17:39,832 ConnectionHandler.java:334 - [Stream #d94d19d0-d33e-11e5-9066-091830ac5256] Sending Complete

Any idea what I'm doing wrong?

[tianon]

Sorry for the super massive delay. My best guess would be reachability on that fourth node (or reachability from it to the two seed nodes). I'd recommend double-checking the networking between all the nodes (perhaps try connecting cqlsh to each of the other three nodes from that fourth node in a container).

Just for reference, there's a pretty sweet little example Swarm file for setting up a swarm cluster semi-automatically in #94 (comment). 👍

In the future, these sorts of questions/requests would be more appropriately posted to the Docker Community Forums, the Docker Community Slack, or Stack Overflow.