rabbitmq-stream-s3

Tiered storage plugin for RabbitMQ streams. Uploads committed stream data to Amazon S3 in the background. Consumers read from local disk when data is available and fall back to S3 for older data.

Streams retain data in S3 indefinitely (or according to a remote retention policy) at low cost, while local disk only holds recent data. Consumers see a single continuous stream regardless of where the data lives.

For a detailed description of the design, start with docs/README.md.

Project Maturity

rabbitmq-stream-s3 has matured considerably and is now at a beta level of stability. The core design has settled and the main subsystems (upload path, read path, garbage collection, retention, and manifest replication) are implemented and exercised by an extensive test suite, formal models, and a Jepsen harness.

It remains pre-1.0, so some interfaces and behavior may still change.

Important: rabbitmq_prometheus dependency

This plugin depends on rabbitmq_prometheus. Enabling rabbitmq_stream_s3 implicitly enables rabbitmq_prometheus, which opens the Prometheus metrics endpoint on port 15692.

Prerequisites

This project currently requires specific development branches of the rabbitmq-server and osiris repositories:

rabbitmq-server

Branch: streams-tiered-storage

osiris

Branch: tiered-storage-abstractions

See Tiered Storage Support for RabbitMQ Streams.

Build

1. Clone the RabbitMQ server repository:

git clone https://github.com/amazon-mq/upstream-to-rabbitmq-server.git
cd upstream-to-rabbitmq-server

2. Switch to the required branch:

git checkout streams-tiered-storage

3. Build:

make

For more information on building and developing RabbitMQ plugins, see plugin-development.

Configure

See docs/operations.md for all configuration options.

The minimum required configuration is:

stream_s3.bucket = my-rabbitmq-streams-bucket
stream_s3.region = us-east-1

Documentation

• docs/README.md: overview and reading guide
• docs/user-guide.md: behavior (no implementation details)
• docs/concepts.md: streaming primitives and how the plugin extends them
• docs/manifest.md: manifest tree structure and concurrency control

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.