Streams as a work queue are uncovered, and XLEN is not queue depth

[alberttwong]

Filing rather than opening a PR because this wants a new reference file plus a redis-core eval re-run, which is a maintainer-scoped decision. Happy to write it if you want it.

The gap

Streams appear in the skills exactly twice, as the same table row in two files:

skills/redis-core/SKILL.md:33                        | Event log, fan-out messaging | Stream | Persistent, consumer groups |
skills/redis-core/references/choose-data-structure.md:13 | Event logs, messaging   | Stream | Persistent, consumer groups |

That is the whole of it. Across all eight skills there is no XADD, no XREADGROUP, no XACK, no XPENDING, no XAUTOCLAIM, no XGROUP. So the skills recommend a Stream for a work queue and then say nothing about operating one — and the operational details are where the surprises live.

The one that cost us most: XLEN is not queue depth

Redis keeps acknowledged entries in a stream until something trims them. So XLEN counts history, not backlog. A perfectly healthy, fully drained queue reports a large and growing XLEN, and a monitor built on it reports a problem that does not exist — or, worse, is tuned around that number until it stops reporting problems that do.

Queue depth is three separate numbers, and they mean different things:

Number	Source	Means
backlog	consumer-group lag from XINFO GROUPS	entries never delivered to this group — a worker is not running, or not keeping up
pending	XPENDING	delivered, not yet acked — a worker is mid-job, or died holding one
dead	your own dead-letter stream	given up on. Nothing else about the affected record looks wrong, which is what makes this the one to alert on at zero

We banded these three and deliberately left XLEN unbanded. The dead-letter count is the signal worth a hard zero: a job that was abandoned leaves a record that is otherwise complete, so nothing downstream reports anything.

Other things worth a paragraph each

• A forgotten consumer silently steals work. A leftover XREADGROUP process — a dev worker, a stale container — takes jobs from the same group and acks them. The tests or the job that expected them fail intermittently in a way that looks exactly like flakiness. A per-environment stream suffix makes this impossible rather than unlikely; we needed one.
• Reclaim is not automatic. A consumer that dies holding an entry leaves it pending forever unless something runs XAUTOCLAIM. pending climbing with backlog at zero is the specific shape of "a worker is dying mid-job".
• Trimming is a policy decision, not a default. MAXLEN/MINID on XADD, or periodic XTRIM. Without one the stream is an unbounded log, and the memory cost is invisible until it isn't.
• A job's payload should be a key, not a document. Ours carries a record id and a fingerprint; re-reading the record at consume time is what makes a retry correct rather than a replay of stale data.
• The fingerprint has to cover everything that changes the output. Ours covered the input text but not the embedding model or dimension, so changing the dimension re-queued every record and the worker skipped all of them as unchanged. The queue drained to zero, reported no failures, and every stored vector was still the old length. Nothing was broken except that nothing had happened.

Suggested shape

skills/redis-core/references/streams-consumer-groups.md, linked from the SKILL.md quick reference next to the data-structure table — the table is what sends someone to a Stream in the first place, so it is the right place to say "and here is what running one involves".

Per CONTRIBUTING this would need evals; evals/redis-core/core/ exists, so it is a suite re-run plus probably one new eval on the depth question, since that is the part a model gets wrong by default (XLEN is the obvious answer and it is the wrong one).

Context: these came out of building a production app on Redis Cloud with an embed worker on a Stream. Related PRs from the same project: #53, #54, #55.

🤖 Generated with Claude Code