Skip to content

Commit 1707094

Browse files
Merge pull request #308 from amazon-mq/jepsen-306-hardening
Improve Jepsen suite
2 parents fac1253 + ccfd98b commit 1707094

10 files changed

Lines changed: 245 additions & 98 deletions

File tree

.github/workflows/jepsen.yaml

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,9 @@ concurrency:
4343
jobs:
4444
jepsen:
4545
runs-on: ubuntu-latest
46+
# Safety net: a wedged run must fail fast, not burn hours. The build plus a
47+
# bounded 120s test settles well inside this.
48+
timeout-minutes: 45
4649
strategy:
4750
fail-fast: false
4851
matrix:
@@ -67,10 +70,28 @@ jobs:
6770
otp-version: ${{ matrix.otp-version }}
6871
elixir-version: ${{ matrix.elixir-version }}
6972
- name: Clone rabbitmq-server
70-
run: git clone --depth 1 --branch ${{ matrix.rmq-version }} https://github.com/amazon-mq/upstream-to-rabbitmq-server.git ${{ github.workspace }}/rabbitmq-server
73+
# The server is built and run as part of the cluster, so pin it to a
74+
# commit SHA rather than the mutable streams-tiered-storage branch tip:
75+
# a CI run then builds exactly the server it was tested against, and a
76+
# force-push to the branch cannot change what executes here. Bump this
77+
# SHA periodically to track the branch (that re-introduces server-drift
78+
# coverage at a controlled point). Shallow-fetch the SHA directly so the
79+
# checkout keeps working once the branch tip has moved past the pin.
80+
env:
81+
RMQ_SERVER_SHA: 91051d3c678faa6cc68a75d6b561ceca05e12025
82+
run: |
83+
git init -q "${{ github.workspace }}/rabbitmq-server"
84+
git -C "${{ github.workspace }}/rabbitmq-server" remote add origin \
85+
https://github.com/amazon-mq/upstream-to-rabbitmq-server.git
86+
git -C "${{ github.workspace }}/rabbitmq-server" fetch -q --depth 1 origin "$RMQ_SERVER_SHA"
87+
git -C "${{ github.workspace }}/rabbitmq-server" checkout -q "$RMQ_SERVER_SHA"
7188
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
7289
with:
7390
path: rabbitmq-server/deps/rabbitmq_stream_s3
91+
# The job then runs a full server build plus a lein dependency
92+
# download and a RabbitMQ cluster; do not leave the token in .git/config
93+
# for that third-party code to read.
94+
persist-credentials: false
7495
- name: Run Jepsen
7596
working-directory: rabbitmq-server/deps/rabbitmq_stream_s3
7697
run: jepsen/ci/run-jepsen.sh

jepsen/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,9 @@ docker compose exec control \
4343

4444
The partition nemesis runs against a 5-node cluster with the kafka anomaly checkers and retention wide. A run under partition faults reports `:valid? true`, with fault-induced failures correctly classified as indeterminate rather than lost or duplicate, and broker logs collected back to the control node.
4545

46-
The storage-tier nemeses make data really tier to S3 and exercise the read-from-S3 path under fault. Small segments and fragments plus padded payloads force segments to roll, upload and trim within a short run; MinIO needs a KMS key for the plugin's SSE uploads (see `docker/docker-compose.yml`). The `--faults` option selects `s3-outage` and `s3-latency` (via Toxiproxy), `trim` (periodic `evaluate_local_retention`, which trims uploaded segments locally so reads of older offsets fall to the S3 tier), and `leader-move` (repeatedly transfer every stream's leader to a replica, relocating each writer and bumping the stream epoch so a deposed writer's in-flight upload must be fenced).
46+
The storage-tier nemeses make data really tier to S3 and exercise the read-from-S3 path under fault. Small segments and fragments plus padded payloads force segments to roll, upload and trim within a short run; MinIO needs a KMS key for the plugin's SSE uploads (see `docker/docker-compose.yml`). The `--faults` option selects `s3-outage` and `s3-latency` (via Toxiproxy), `trim` (periodic `evaluate_local_retention`, which trims uploaded segments locally so reads of older offsets fall to the S3 tier), and `leader-move` (each tick, transfer a random stream's leader to a replica, relocating that writer and bumping the stream epoch so a deposed writer's in-flight upload must be fenced).
4747

48-
Two extra checkers run alongside the kafka anomaly checkers. The `:tiering` coverage checker scrapes the plugin's S3 counters before teardown and fails the run unless fragments were uploaded, and with `trim` that reads were served from S3, and with `leader-move` that the epoch advanced; this keeps a run from passing green while silently not using the remote tier or not moving leaders. The `:durability` checker reads every stream end-to-end after the run and fails if any acknowledged write is lost or duplicated; it is authoritative under `leader-move`, where the kafka offset analyzers are downgraded to advisory (see "Why the Kafka workload").
48+
Two extra checkers run alongside the kafka anomaly checkers. The `:tiering` coverage checker scrapes the plugin's S3 counters before teardown and fails the run unless fragments were uploaded, and with `trim` that reads were served from S3, and with `leader-move` that the epoch advanced; this keeps a run from passing green while silently not using the remote tier or not moving leaders. The `s3-outage` fault is instead proven injected by the Toxiproxy status check in the nemesis (it fails on a non-2xx response), since the plugin tolerates an outage gracefully and need not surface an error counter. The `:durability` checker reads every stream end-to-end after the run and fails if any acknowledged write is lost or duplicated; it is authoritative under `leader-move`, where the kafka offset analyzers are downgraded to advisory (see "Why the Kafka workload").
4949

5050
A run under `partition,s3-outage,s3-latency,trim` stays `:valid? true` while serving thousands of reads from the remote tier:
5151

jepsen/docker/Dockerfile-control

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
# Jepsen control node: runs `lein run test ...` and drives the DB nodes over
22
# SSH. Needs a JDK (for the Stream Java client + jepsen), leiningen, and an
33
# SSH client.
4-
FROM clojure:temurin-21-lein
4+
# Pinned by digest for supply-chain reproducibility (clojure:temurin-21-lein).
5+
FROM clojure@sha256:f5ef76151930464b31b97d518469b802de1c82507d6205d6ecf96cf6cb6d17d2
56

67
RUN apt-get update && apt-get install -y --no-install-recommends \
78
git openssh-client sshpass iputils-ping dnsutils gnuplot graphviz \

jepsen/docker/Dockerfile-node

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
# plus the bits jepsen needs: sshd for control access and iptables/sudo for
33
# the partition nemesis. The broker itself is installed at run time by db.clj
44
# from /shared/rabbitmq.tar.xz (the generic-unix tarball).
5-
FROM erlang:27
5+
# Pinned by digest for supply-chain reproducibility (erlang:27).
6+
FROM erlang@sha256:b439336c5035e414008770118719907d08b12c23e576b34a5efc6516653de0f4
67

78
RUN apt-get update && apt-get install -y --no-install-recommends \
89
openssh-server sudo iproute2 iptables procps ca-certificates \

jepsen/docker/docker-compose.yml

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,8 @@ services:
5252
n5: {<<: *node, hostname: n5, container_name: n5}
5353

5454
minio:
55-
image: minio/minio:latest
55+
# Pinned by digest for supply-chain reproducibility (was minio/minio:latest).
56+
image: minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
5657
environment:
5758
MINIO_ROOT_USER: jepsenjepsen
5859
MINIO_ROOT_PASSWORD: jepsenjepsenjepsen
@@ -69,7 +70,8 @@ services:
6970
jepsen:
7071

7172
toxiproxy:
72-
image: ghcr.io/shopify/toxiproxy:latest
73+
# Pinned by digest (was ghcr.io/shopify/toxiproxy:latest).
74+
image: ghcr.io/shopify/toxiproxy@sha256:9378ed52a28bc50edc1350f936f518f31fa95f0d15917d6eb40b8e376d1a214e
7375
# The toxiproxy image is FROM scratch (no shell), so the passthrough proxy
7476
# is declared in a config file loaded at startup rather than created by an
7577
# init container. -host=0.0.0.0 exposes the admin API (8474) for S3 fault
@@ -91,7 +93,8 @@ services:
9193
# comes from MC_HOST_j, and we talk path-style to minio:443 with --insecure
9294
# (the cert is for *.s3.jepsen.local, not "minio"). restart: on-failure
9395
# retries until MinIO is accepting connections, then exits 0.
94-
image: minio/mc:latest
96+
# Pinned by digest (was minio/mc:latest).
97+
image: minio/mc@sha256:a7fe349ef4bd8521fb8497f55c6042871b2ae640607cf99d9bede5e9bdf11727
9598
environment:
9699
MC_HOST_j: "https://jepsenjepsen:jepsenjepsenjepsen@minio:443"
97100
entrypoint: ["mc"]

jepsen/jepsen.streams3/src/jepsen/streams3/checker.clj

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -80,12 +80,18 @@
8080
lost (sort (remove present vs))
8181
dups (->> vals frequencies
8282
(keep (fn [[v c]] (when (> c 1) v)))
83-
sort)]
83+
sort)
84+
;; Offsets each duplicated value sits at, so we can
85+
;; tell a real stream duplicate (distinct offsets)
86+
;; from a read redelivery (the same offset twice).
87+
dup-offsets (into {}
88+
(for [v dups]
89+
[v (mapv first (filter #(= (second %) v) pairs))]))]
8490
:when (or (nil? pairs) (seq lost) (seq dups))]
8591
(cond-> {:key k}
8692
(nil? pairs) (assoc :not-read true)
8793
(seq lost) (assoc :lost lost)
88-
(seq dups) (assoc :duplicated dups)))
94+
(seq dups) (assoc :duplicated dups :duplicate-offsets dup-offsets)))
8995
problems (cond-> []
9096
(empty? reads) (conj :no-authoritative-reads)
9197
(seq errs) (conj :acked-writes-lost-or-duplicated))]

jepsen/jepsen.streams3/src/jepsen/streams3/client.clj

Lines changed: 83 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -100,16 +100,33 @@
100100
(defonce ^:private pub-counters (atom {})) ; k -> AtomicLong (next publishingId)
101101
(defonce ^:private env-generation (atom 0)) ; bumped each Environment rebuild
102102

103+
(defn- retryable?
104+
"True when the throwable (or a cause in its chain) is a StreamException -- a
105+
transient broker-side condition (stream/leader not yet available right after
106+
create or during a partition) worth retrying. A programming error (NPE,
107+
ClassCast) is not a StreamException, so it falls through and crashes fast. The
108+
walk is nil-safe and depth-bounded so a cyclic or self-referential cause chain
109+
(which async stream-client errors can produce) cannot spin forever -- under the
110+
leader-move crash storm an unbounded walk hung the whole run."
111+
[^Throwable e]
112+
(->> (iterate (fn [^Throwable t] (when t (.getCause t))) e)
113+
(take 64)
114+
(take-while some?)
115+
(some #(instance? com.rabbitmq.stream.StreamException %))
116+
boolean))
117+
103118
(defn with-retry
104-
"Calls thunk, retrying on any exception until op-retry-ms elapses, then
105-
rethrows. For transient 'leader/stream not available' errors right after
106-
create or during a partition."
119+
"Calls thunk, retrying transient stream errors (see retryable?) until
120+
op-retry-ms elapses, then rethrows. A non-transient exception is rethrown
121+
immediately, so a client bug fails fast instead of after a 10s retry loop."
107122
[thunk]
108123
(let [deadline (+ (System/currentTimeMillis) op-retry-ms)]
109124
(loop []
110125
(let [r (try {:val (thunk)}
111126
(catch Exception e
112-
(if (< (System/currentTimeMillis) deadline) :retry (throw e))))]
127+
(if (and (retryable? e) (< (System/currentTimeMillis) deadline))
128+
:retry
129+
(throw e))))]
113130
(if (= r :retry) (do (Thread/sleep 250) (recur)) (:val r))))))
114131

115132
(defn get-env ^Environment [test]
@@ -139,6 +156,22 @@
139156
(reset! pub-counters {})
140157
(swap! env-generation inc))))
141158

159+
(defn reset-run-state!
160+
"Resets all run-scoped shared connection state (the Environment, declared
161+
streams, producers, publishingId counters, generation) so a second test in the
162+
same JVM (a REPL session or `lein run test-all`) starts clean rather than
163+
carrying stale stream declarations or publishingId counters against a
164+
freshly-wiped cluster. (tiering-stats and authoritative-reads are reset
165+
alongside this in db/setup!.)"
166+
[]
167+
(locking shared-env
168+
(when-let [e @shared-env] (try (.close e) (catch Exception _)))
169+
(reset! shared-env nil)
170+
(reset! declared-streams #{})
171+
(reset! shared-producers {})
172+
(reset! pub-counters {})
173+
(reset! env-generation 0)))
174+
142175
(defn ensure-stream! [env k]
143176
(when-not (contains? @declared-streams k)
144177
;; create() is idempotent for a matching stream.
@@ -253,11 +286,11 @@
253286
;; Authoritative end-to-end read (for the durability checker)
254287
;; ---------------------------------------------------------------------------
255288

256-
;; Reading the whole history of ~90 streams, some served from S3, needs a far
257-
;; more generous quiescence wait than the steady-state catch-up above.
258-
(def auth-quiet-ms 3000) ; deliveries idle this long => every stream at its tail
259-
(def auth-grace-ms 8000) ; tolerate this long before any delivery (S3 warm-up)
260-
(def auth-cap-ms 180000) ; hard cap so a genuinely stuck stream cannot hang
289+
;; Default cap on the authoritative read (overridable via --auth-read-timeout-sec).
290+
;; The read waits for an observable target (each consumer reaching the stream's
291+
;; committed offset), not idle time, so it cannot truncate mid-backfill; the cap
292+
;; only bounds a stream that genuinely never reaches its target.
293+
(def auth-read-cap-ms-default 180000)
261294

262295
;; The authoritative read of every jepsen stream, captured once after the run
263296
;; while the brokers are still up (db/log-files), for the durability checker:
@@ -266,53 +299,59 @@
266299
;; offsets unreliable.
267300
(defonce authoritative-reads (atom {}))
268301

269-
(defn- await-quiescent!
270-
"Waits until the total buffered count across queues holds steady for
271-
auth-quiet-ms (every stream reached its tail), bounded by auth-cap-ms, with an
272-
auth-grace-ms grace before the first delivery for streams served from S3."
273-
[queues]
274-
(let [total (fn [] (reduce (fn [a ^java.util.Collection q] (+ a (.size q))) 0 queues))
275-
start (System/currentTimeMillis)
276-
deadline (+ start auth-cap-ms)]
277-
(loop [last-total (total), last-change start]
278-
(let [now (System/currentTimeMillis)
279-
t (total)]
280-
(cond
281-
(>= now deadline) nil
282-
(not= t last-total) (do (Thread/sleep 100) (recur t now))
283-
(and (> (- now last-change) auth-quiet-ms)
284-
(or (pos? t) (> (- now start) auth-grace-ms))) nil
285-
:else (do (Thread/sleep 100) (recur t last-change)))))))
302+
(defn- await-targets!
303+
"Waits until every stream's max delivered offset reaches its committed-offset
304+
target, bounded by cap-ms. `state` is {key {:q queue :maxoff AtomicLong}};
305+
`targets` is {key committed-offset}. A stream that never reaches its target
306+
waits out the cap, after which the durability checker reports the loss."
307+
[state targets cap-ms]
308+
(let [deadline (+ (System/currentTimeMillis) cap-ms)
309+
met? (fn [] (every? (fn [[k {:keys [^AtomicLong maxoff]}]]
310+
(>= (.get maxoff) (get targets k -1)))
311+
state))]
312+
(loop []
313+
(when (and (not (met?)) (< (System/currentTimeMillis) deadline))
314+
(Thread/sleep 100)
315+
(recur)))))
286316

287317
(defn read-streams-fully
288-
"Authoritative end-to-end read of stream keys `ks` for the durability checker.
289-
Builds a FRESH Environment (the run's shared one may be wedged from topology
290-
churn), subscribes each stream from the beginning, waits for deliveries to go
291-
quiet (every stream at its tail), and returns {key -> vector of [offset value]
292-
in delivery order}. Closes the Environment before returning."
293-
[test ks]
294-
(let [uris (java.util.ArrayList.
295-
(map #(str "rabbitmq-stream://guest:guest@" (name %) ":" stream-port)
296-
(:nodes test)))
297-
env (-> (Environment/builder) (.uris uris) (.build))
298-
queues (reduce (fn [m k] (assoc m k (ConcurrentLinkedQueue.))) {} ks)]
318+
"Authoritative end-to-end read of the streams in `targets` (key -> committed
319+
offset) for the durability checker. Builds a FRESH Environment (the run's
320+
shared one may be wedged from topology churn), subscribes each stream from the
321+
beginning, and waits until each consumer has delivered up to the stream's
322+
committed offset -- an observable target rather than an idle-time guess, so a
323+
mid-backfill quiet period cannot truncate the read and report false loss.
324+
Returns {key -> vector of [offset value] in delivery order}. Closes the
325+
Environment before returning."
326+
[test targets]
327+
(let [cap-ms (* 1000 (or (:auth-read-timeout-sec test)
328+
(quot auth-read-cap-ms-default 1000)))
329+
uris (java.util.ArrayList.
330+
(map #(str "rabbitmq-stream://guest:guest@" (name %) ":" stream-port)
331+
(:nodes test)))
332+
env (-> (Environment/builder) (.uris uris) (.build))
333+
state (into {} (map (fn [k] [k {:q (ConcurrentLinkedQueue.) :maxoff (AtomicLong. -1)}])
334+
(keys targets)))]
299335
(try
300-
(doseq [[k q] queues]
336+
(doseq [[k {:keys [^ConcurrentLinkedQueue q ^AtomicLong maxoff]}] state]
301337
(-> env (.consumerBuilder) (.stream (stream-name k))
302338
(.offset (OffsetSpecification/first))
303339
(.messageHandler
304340
(reify com.rabbitmq.stream.MessageHandler
305341
(handle [_ ctx msg]
306-
(.add q [(.offset ctx) (decode-value (.getBodyAsBinary msg))]))))
342+
;; Deliveries arrive in offset order from `first`, so set wins.
343+
(.add q [(.offset ctx) (decode-value (.getBodyAsBinary msg))])
344+
(.set maxoff (.offset ctx)))))
307345
(.build)))
308-
(await-quiescent! (vals queues))
309-
(into {} (map (fn [[k q]] [k (drain! q)]) queues))
346+
(await-targets! state targets cap-ms)
347+
(into {} (map (fn [[k {:keys [^ConcurrentLinkedQueue q]}]] [k (drain! q)]) state))
310348
(finally (.close env)))))
311349

312350
(defn capture-authoritative-reads!
313-
"Reads keys `ks` end-to-end and stores the result in authoritative-reads."
314-
[test ks]
315-
(reset! authoritative-reads (read-streams-fully test ks)))
351+
"Reads the streams in `targets` (key -> committed offset) end-to-end and stores
352+
the result in authoritative-reads."
353+
[test targets]
354+
(reset! authoritative-reads (read-streams-fully test targets)))
316355

317356
(defrecord Client [node ^Environment env env-gen consumers buffers assigned]
318357
client/Client

jepsen/jepsen.streams3/src/jepsen/streams3/core.clj

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,19 @@
2727
[nil "--final-time-limit SEC"
2828
"Cap on the final-reads phase (the kafka final-polls loop is unbounded)"
2929
:default 180 :parse-fn parse-long]
30+
[nil "--final-settle-sec SEC"
31+
"Quiet settle period after healing faults, before the final reads"
32+
:default 15 :parse-fn parse-long]
33+
[nil "--auth-read-timeout-sec SEC"
34+
"Cap on the durability checker's end-to-end read (per committed-offset target)"
35+
:default 180 :parse-fn parse-long]
3036
[nil "--faults FAULTS"
3137
"Comma-separated: partition, s3-outage, s3-latency, trim, leader-move"
3238
:default "partition"]])
3339

3440
(defn streams3-test
3541
[opts]
42+
(nem/validate-faults! opts)
3643
(let [wl (workload/workload opts)]
3744
(merge
3845
tests/noop-test
@@ -77,7 +84,7 @@
7784
(gen/nemesis [{:type :info :f :stop-partition}
7885
{:type :info :f :stop-s3-outage}
7986
{:type :info :f :stop-s3-latency}])
80-
(gen/sleep 15)
87+
(gen/sleep (:final-settle-sec opts))
8188
;; With trimming enabled, trim once more after healing so the
8289
;; final reads drain from the now-trimmed (S3-only) tier and
8390
;; actually exercise the read-from-S3 path.

0 commit comments

Comments
 (0)