Skip to content

Latest commit

 

History

History
279 lines (239 loc) · 15.4 KB

File metadata and controls

279 lines (239 loc) · 15.4 KB

1.0.1

  • Document how to run the tests. They talk to a real Redis on port 6399, which the readme never said, so a clone-and-dart test failed with a connection error and no explanation. Readme only; no code change.

1.0.0

The API is stable. No code changes since 0.12.0, which settled the last three freeze-blockers; this freezes the surface after re-verifying them against a live Redis.

Verified by execution: two workers constructed with the same workerId in one isolate throw a StateError rather than silently stealing each other's in-flight tasks; run() after close() throws instead of starting a zombie loop; an unreachable server surfaces as TaskQueueConnectionException, and a server-side refusal as TaskQueueServerException, both under the sealed TaskQueueException with the original error on .cause. The full suite (47 tests) passes against a real Redis, exercising weighted draining, the atomic Lua promote/retry/dead-letter scripts, crash recovery, and delayed scheduling.

One limitation is by design and documented: a worker that dies and is never restarted under the same id leaves its in-flight list for a supervisor to requeue — recovery keys off a stable workerId, which is the price of not needing a heartbeat. Depends only on package:redis (4.x).

0.12.0

Three breaking changes, all closing gaps that would have been frozen into 1.0.0. Each is small to migrate.

  • workerId is now required on Worker.connect. It used to default to the host name, which is silently wrong the moment a second worker starts on the same machine: the two share an in-flight list, and each one's startup recovery treats the other's live task as an orphan and requeues it, so the task runs twice. Reproduced before the change (two workers, one task, both ran it), and the default is what produced it. There is no safe default — the id has to be stable across restarts and unique per worker — so the choice now belongs to the caller, documented at the parameter. Passing the same id to two live workers in one isolate throws a StateError instead of corrupting the in-flight lists. To migrate, pass the value you were relying on: workerId: Platform.localHostname restores the old behaviour exactly (and is correct if you run one worker per host).
  • Redis failures are now TaskQueueExceptions. enqueue, stats, deadLetters, replayDeadLetter, purgeDeadLetters, both connects and the worker loop used to surface whatever package:redis raised. That is not a usable contract: RedisError, RedisRuntimeError and TransactionError do not implement Exception, so on Exception catch missed them entirely, and a dropped socket arrived as a SocketException, a StateError, or a bare String depending on where it broke. They now become TaskQueueConnectionException (Redis unreachable, after the existing one-shot reconnect) or TaskQueueServerException (Redis answered with an error), both under a sealed TaskQueueException, with the original kept on .cause. A server-side refusal also no longer triggers a reconnect-and-retry — the socket is alive, so the retry only collected the same answer.
  • Worker.close() is terminal. It awaits stop() first, so the loop is not left polling a socket that is closing, releases the worker id, and is a no-op when called twice. run() after close() throws a StateError rather than starting a second loop on a dead connection.

Ten tests cover the new behaviour, including the duplicate-execution scenario that motivated the first change.

0.11.0

Settles the class-modifier and export questions ahead of a 1.0.0 freeze. No behaviour changes.

  • Mark the six exported classes final: QueueClient, Worker, Task, TaskContext, DeadLetter and QueueStats. None was designed to be extended or implemented, and nothing in the package, its tests or its examples subtypes any of them. Sealing them keeps the rest of 1.x additive: the roadmap adds members to exactly these types, and every addition would otherwise break anyone who had implemented them. Breaking only for code that subclassed one, which was never a supported shape; mock a QueueClient or Worker by wrapping it rather than implementing it.
  • Name both remaining exports explicitly. client.dart and worker.dart were re-exported whole, so a symbol that became public in either would have joined the API by accident. The exported set is unchanged.
  • Fix the delayed-set key in the README, which documented <prefix>:<queue>:delayed while the real key is <prefix>:queue:<queue>:delayed. Anyone monitoring Redis by the documented pattern would have watched a key that does not exist.

0.10.1

  • QueueClient and Worker now reconnect after a dropped Redis connection instead of failing every call for the rest of their lifetime. Neither this package nor the underlying redis client ever redialed a dead socket, so a Redis restart, a managed-Redis failover, or a proxy's idle timeout broke a long-running worker or a reused client permanently, even minutes after Redis was healthy again. Reproduced against a real Redis: Worker.run() crashed with an unhandled stream is closed exception on the exact try/catch-free pattern README.md shows, and QueueClient.enqueue(), reused across calls the way README.md recommends, kept failing on the same client well after Redis had come back up. Every Redis command in both classes now retries once through a fresh connection before giving up, and run()'s poll loop reconnects and reruns orphan recovery instead of exiting when that retry fails too, so a dropped connection ends one call, not the worker. No API change: connect() takes the same arguments, and behavior is unchanged as long as the connection never drops.

0.10.0

  • stop() is now awaitable: it returns a future that completes once the worker has drained, the task in progress finished and the loop exited. await worker.stop() is the graceful-shutdown path for a SIGTERM or a rolling deploy. Existing worker.stop(); statements are unaffected; the returned future is simply ignored. To be clear about what this does and does not change: a task already in flight was never cut off, since the run loop awaits the handler before checking whether it should stop. What was missing was a way to wait for that drain without separately holding the future from run(). This adds it.

0.9.0

  • QueueClient.stats() reports queue depth: pending and delayed counts per queue, plus the total in-flight and dead-letter counts, as a QueueStats. It reads counters rather than tasks, so it is cheap enough to poll for a dashboard or a backlog alert, and it discovers the active queues with SCAN rather than KEYS, so it does not block Redis. Pass queues: [...] to count a fixed set instead, which reports an empty queue as zero rather than omitting it. This is the "is the system keeping up" question the queue could not answer before: a growing pending count means the workers are behind, a growing dead-letter count means something is failing for good.

0.8.0

  • The dead-letter list is now inspectable and manageable, which the README has claimed since the start without an API to back it. A dead-lettered task is stored with the error that gave up on it, not as a bare envelope, and QueueClient gains three methods: deadLetters({limit}) returns the entries as DeadLetter (task, queue, error text, attempt count, dead-at time), replayDeadLetter(id) re-enqueues one for a fresh set of attempts, and purgeDeadLetters() clears them. A replay removes the entry and re-enqueues in one atomic step, so it can't be dropped without landing back on its queue or double-enqueued by two concurrent callers.
  • Additive and backward-compatible: DeadLetter.decode reads a pre-0.8.0 entry (a bare envelope with no stored error) too, so an existing dead-letter list still reads after the upgrade.

0.7.1

  • Make task ids unique across producer processes. The id was identityHashCode(this) plus a per-process counter, and both reset or repeat when a new process starts, so two producers could mint the same id. Since the id is the deduplication key an idempotent handler writes against, a collision meant one task was silently taken for a repeat of another and skipped: data loss in the exact place the package tells you to rely on the id. Measured: of 100,000 fresh clients, several produced an identical first id. Ids now pair a per-client 96-bit secure-random prefix with the counter, so the prefix separates processes and the counter orders within one. No new dependency; no API change. A test enqueues from 200 clients and asserts every id is distinct.

0.7.0

  • Breaking: onError and onDeadLetter now receive the run's TaskContext as their second argument, and the attempt/willRetry named parameters are gone: context.attempt and !context.isLastAttempt carry the same information, so there is one shape to learn instead of two. The reason is context.id. The observers had the task type and the error but no id, so a failure log or a dead-letter alert could name a kind of job and never the job itself. That is the first thing anyone wants during triage, and it was the one place 0.6.0's TaskContext had not reached: the handler got it, the observability seam did not. Migration: (task, error, stack, {attempt, willRetry}) becomes (task, context, error, stack), reading context.attempt and context.isLastAttempt.

0.6.1

  • Install instructions now say pub add instead of pinning a version. The pinned number was stale by several releases and would have been stale again after the next one: the README ships frozen in the archive, so a hand-edited version line is wrong the moment anything is published. This one cannot go out of date.

0.6.0

  • Breaking: a handler now takes the run's context as a second argument, (Task task, TaskContext context). Existing handlers migrate by adding the parameter: (task) async { ... } becomes (task, _) async { ... }.
  • Handlers can see which run they are in. TaskContext carries the task's id, the queue it came from, the 1-based attempt, maxAttempts (the first run plus its retries) and isLastAttempt, which is true on exactly the run whose failure dead-letters the task. The id is the point. Delivery is at-least-once, so the package has always asked handlers to be idempotent, but until now it gave them nothing stable to deduplicate on: the id lived in the envelope and never reached the handler, which left callers inventing their own key inside the payload. It is assigned at enqueue and is unchanged across retries and crash recoveries, so it is the value to record with the effect. attempt and isLastAttempt cover the other half, taking a slower or safer path on a late try and getting one last chance to record something before the task is given up on.
  • Examples now demonstrate the delivery guarantees instead of describing them. example/crash_recovery.dart kills a worker mid-task in a child process and shows the task completing after recovery. example/at_least_once.dart stages the same crash twice, with a careless handler and with one keyed off context.id, and counts the effect: applied twice, then once. There is also an example/README.md covering the worker-id contract, the idempotency pattern, watching the dead-letter list, and where this queue does not fit.

0.5.0

  • Crash-safe at-least-once delivery. The worker now claims a task by atomically moving it (LMOVE) onto a per-worker in-flight list and only removes it once the task is done, retried, or dead-lettered. A worker that dies mid-task leaves the envelope on its in-flight list and requeues it on its next run, so a crash, OOM kill, or lost node no longer loses the task in progress. Previously the worker popped with BRPOP, so a task being handled when the process died was gone. Delivery is at-least-once: a task can run again after a crash, so handlers must be idempotent.
  • New workerId on Worker.connect (default: the host name) names the in-flight list a restarted worker recovers from. Set it to something stable across restarts (a pod or service name); two workers must never share one. See the README's "Recovery and worker ids" for the one case this doesn't cover on its own (a worker that never restarts under the same id).
  • Real weighted fair scheduling. The worker draws each queue's turn in proportion to its weight with a rotating cursor over the weighted order, so {'critical': 6, 'default': 3, 'low': 1} is served roughly 6:3:1 under load and, unlike strict priority, a flood of critical jobs can't fully starve low. The previous BRPOP over a repeated key list was really strict priority; the weights only set the order.
  • Docs: the flow and state diagrams now show the in-flight list and the crash-recovery path.

0.4.1

  • Docs: replace the two README mermaid diagrams with rendered PNGs. pub.dev does not render mermaid, so the diagrams showed as raw source there; they now display as images on both pub.dev and GitHub.

0.4.0

  • Add observability hooks to Worker.connect. onError fires on every handler failure, with the 1-based attempt number and whether a retry follows; onDeadLetter fires when a task is given up on after its retries. Both default to null and are isolated, so a throwing callback cannot take the worker down. Before this, a handler exception was swallowed silently, which is rarely what a production queue wants.
  • Restore scheduled tasks (enqueue with processAt or processIn), which the 0.3.1 release removed by accident. 0.3.1 was published as a docs-only change but also dropped the 0.3.0 scheduling feature; if you schedule tasks for a future time, move to 0.4.0 (or pin 0.3.0) rather than 0.3.1.

0.3.0

  • Scheduled tasks. enqueue takes an optional processAt (an absolute time) or processIn (a delay from now), set at most one, to hold a task until a future time.
  • No new machinery: a scheduled task is scored into the same per-queue delayed sorted set the 0.2.0 retry backoff uses, and the same atomic Lua due-mover promotes it once due. The worker is unchanged.
  • The inherited bounds apply: a due task starts up to about a second past its due time (the mover runs once per poll-loop pass), and only while a worker polling that queue is running.
  • enqueue without either parameter behaves exactly as in 0.2.0.

0.2.0

  • Exponential backoff for retries. A failed task is no longer re-enqueued immediately: it goes into a per-queue delayed sorted set (<prefix>:<queue>:delayed) scored with the time it becomes due, and waits min(cap, base * 2^(retry-1)) plus jitter before being retried.
  • Configurable backoff on Worker.connect: backoffBase (default 1s), backoffCap (default 60s), and backoffJitter (default 0.1).
  • The worker's poll loop now runs a due-mover each pass that promotes delayed tasks whose score has passed back onto their pending list. The move is a single atomic Redis Lua script (ZRANGEBYSCORE + ZREM + LPUSH), so a task can't be lost or duplicated, even with multiple workers.
  • Dead-letter behaviour is unchanged; weighted-queue behaviour is unchanged.

0.1.0

  • Initial release.
  • Enqueue tasks from a producer, process them in a worker.
  • Retries with a dead-letter list after maxRetries.
  • Weighted queues to avoid starvation.