Ceph RGW bucket notifications to Kafka and AMQP

Triggering pipelines from Ceph RGW bucket notifications

Use Ceph RGW bucket notifications when a downstream pipeline must start as soon as an object is created or deleted, and when repeatedly listing a bucket would be too slow or too costly at your object count. The Object Gateway publishes an event for each matching change to a topic, and the topic delivers that event to an endpoint your pipeline consumes. Topics are created through the SNS-compatible API that RGW exposes, and a bucket is then linked to a topic with a notification configuration through the S3 API. Confirm the exact calls, event names and endpoint options in the Object Gateway documentation for your Ceph release, since these details have changed between releases.

The Object Gateway supports three endpoint kinds: HTTP, AMQP and Kafka. Choose by what the consumer already runs and how it handles failure. An HTTP endpoint suits a webhook into an application that can process events idempotently. A Kafka endpoint suits several independent consumers that each need to read the same event stream and replay it. An AMQP endpoint suits teams that already operate a broker such as RabbitMQ and want RGW events to reach their existing queues. Before you configure anything, list each consumer, its protocol, its expected throughput and whether it needs replay. Those answers decide the endpoint, and they determine how much retry and storage behavior you must plan on the broker side.

Notifications are not a full audit trail. Events describe changes made after the configuration exists, so existing objects need a separate backfill, and event filters limit what reaches each topic. Plan a periodic reconciliation, such as a scheduled listing or inventory report, that compares bucket contents with what the pipeline has processed. This catches gaps from misconfigured filters, endpoint outages or operations your release does not emit events for. Ask your Ceph administrator to confirm which operations generate events in your release, and assign one team to own the topic, endpoint and reconciliation job together.

Endpoint and delivery options and when each fits

This table compares the three endpoint types RGW can notify and the persistent and non-persistent delivery modes. Read each row against your consumer's skills and uptime needs, then use the confirmation column to check the Ceph release notes for your target version before you commit to an endpoint.

Option	Typical consumer	What your team operates	Main risk	Confirm before choosing
HTTP endpoint	A service or webhook receiver that accepts POST requests and can be scaled behind a load balancer	The receiving HTTP service, its TLS certificates, authentication and the retry behavior of its own handlers	Events are lost or delayed when the receiver is down, unless persistent delivery is enabled and the queue is sized for the outage	Supported TLS and authentication options for the endpoint URI in your Ceph release, and how the receiver handles duplicate deliveries
Kafka endpoint	Stream processing or analytics pipelines already built around Kafka topics	A Kafka cluster (KRaft mode on current releases), topics, partitions, retention, ACLs and broker upgrades	Broker unavailability or misconfigured SASL or TLS blocks delivery, and the RGW side may hold events until the broker returns	Supported Kafka broker versions, SSL and SASL mechanisms, and certificate handling for your Ceph release
AMQP endpoint	Teams running RabbitMQ or another AMQP 0-9-1 broker for task queues and fan-out	An AMQP broker, exchanges, queues, bindings, vhosts and credentials	Routing mistakes between exchange and queue leave events unconsumed, and broker upgrades can change queue behavior	Supported AMQP versions and exchange types, TLS and SASL settings, and whether the broker's queue type matches your durability needs
Non-persistent delivery	Consumers that tolerate missing events, such as cache invalidation or best-effort indexing	Only the endpoint itself, with no queue to monitor on the Ceph side	An event is dropped if the endpoint cannot accept it in time, so the pipeline silently diverges from the bucket	How the release notes describe the non-persistent path and whether dropped events are logged or counted
Persistent delivery	Pipelines that must process every change, such as audit trails, replication or billing	The persistent queue on the RGW side, plus monitoring of its backlog and the endpoint's health	A long outage fills the queue, and once the endpoint stays down past the retry window, operations can return 503 errors to clients	How the release notes define the persistent queue, its retry behavior and the 503 condition, and how to alert before the queue fills
Event filters and topics	Teams that route only specific prefixes, suffixes, tags or event types to each endpoint	Topic definitions, notification configuration per bucket and filter rules kept under change control	Overly broad filters create load on the endpoint, and narrow filters miss events that consumers expected	Which filter types your release supports and whether filters apply the same way across all endpoint types
Security controls	Security teams reviewing what leaves the storage network and how it is authenticated	Credentials for each endpoint, certificate rotation and the network path from RGW to the endpoint	Plaintext or unauthenticated endpoints expose object metadata, and credentials stored in topic definitions can be overlooked	TLS verification options, SASL mechanisms and where endpoint credentials are stored in your Ceph release
Upgrade impact	Platform teams planning Ceph or broker upgrades on the same timeline	Regression tests for each endpoint type, a staging broker or receiver, and a rollback plan	Changes in supported broker versions or endpoint settings can break delivery after a Ceph upgrade	Release notes for the Ceph version you plan to run, and the broker versions it lists as supported

Choosing between HTTP, Kafka and AMQP consumers

Choose Kafka when your platform team already runs brokers, consumer groups and monitoring for event streams. Events from RGW then join streams your teams already watch, and most of the consumer work matches what you do for any other Kafka topic. The new work sits in the access model, so plan it before the first event is published. The Kafka security overview describes the TLS and SASL options your clients and brokers can use.

Exposure: the endpoint is a Kafka listener reachable from the gateway network, secured with TLS. Application APIs stay outside the event path.
Acknowledgment: consumers commit offsets after they finish processing, so replays after a failure are expected and consumers must be idempotent.
Authentication: the documented SASL mechanisms include SCRAM-SHA-512 and OAUTHBEARER, which are worth evaluating against your identity setup. Confirm which mechanisms your RGW release can use.
Authorization: in KRaft clusters, ACLs live in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Grant the gateway identity explicit write access and keep that setting off.
Operational load: the platform team owns partitions, retention, broker upgrades and Java runtime versions, which are Java 17 for brokers from Kafka 4.0.

Confirm your Kafka version, KRaft status and authentication mechanism before you commit to this path.

How to set up a notification pipeline step by step

This sequence takes a bucket from no event handling to a tested pipeline that sends object create and delete events to a Kafka, AMQP or HTTP receiver. Each step ends with a check you can record before the pipeline carries production traffic.

Confirm the Ceph release and RGW endpoint
The platform owner records the Ceph release running the cluster and the RGW endpoints that clients use. Ceph's actively maintained releases are Tentacle and Squid, so confirm your release is one of them and that the notification behavior you read about matches its documentation. Record the zone, the buckets in scope and the account that will own the notification configuration. Check that this account can list its buckets through the endpoint over HTTPS before going further.
Prepare the broker or receiver with TLS and authentication
Setting up Kafka, RabbitMQ or an HTTP receiver is handled separately from the Ceph side, and your team or DeployOpen completes it before RGW connects. The receiver needs a stable address, a TLS certificate that RGW hosts trust, and a dedicated credential limited to the target topic or queue. Confirm the authentication mechanism your receiver supports, such as SSL or SASL for Kafka. Check from an RGW host that a test connection succeeds over the intended TLS settings and that the credential is refused for other topics.
Create the topic with its endpoint and delivery mode
Create the notification topic in RGW with the receiver endpoint, its credential reference and the delivery mode. Decide early whether events must survive an endpoint outage, which requires persistent delivery with retries, or whether best-effort delivery is acceptable for the workload. Record the topic name, endpoint scheme and owner. Use the topic commands and attribute names from the Object Gateway documentation for your release. Check by reading the topic back and confirming the endpoint, persistence setting and owner match the plan.
Bind the topic to the bucket with event types and filters
Attach a notification configuration to the bucket that names the topic, the S3 event types you need, such as object creation and removal, and any key prefix or suffix filters. Keep the first configuration narrow, with one prefix and two event types, so test results are easy to read. Add broader filters after the first test passes. Check by reading the bucket notification configuration back and confirming each event type and filter appears exactly as you wrote it in the plan.
Test with a controlled upload and delete
Upload one test object whose key matches the filter, then delete it. At the receiver, confirm that a create event and a remove event arrived with the bucket name, object key and event name. Then upload a second object outside the filter and confirm that no event is produced for it. Check the RGW logs for delivery errors during the test, and keep the test objects and receiver messages as evidence for your change record.
Rehearse an endpoint outage before go-live
Stop the receiver or block its network path for a short period on a test topic, then restore it. Watch the RGW logs and the receiver to confirm which events were queued, how they were redelivered and whether any were lost. Ask your Ceph release documentation what RGW does when an endpoint stays unavailable beyond its retry behavior, including any 503 responses returned to clients, and write the answer into the runbook. Check that the outcome matches what your application owners expect before enabling production buckets.

How persistent delivery and retries change the failure picture

The failure picture changes most when you choose between synchronous and persistent delivery for a topic. With synchronous delivery, the Object Gateway attempts to send the notification as part of the S3 request, so an unreachable endpoint shows up in the request path. With persistent delivery, the gateway records the event on the cluster and retries delivery afterward, so the endpoint outage moves out of the client's request and into the background queue. The Object Gateway documentation describes both modes. Before you choose, confirm in the release you run how each mode reports a failed send and which errors reach the client.

Persistent delivery means events queue on the cluster while the endpoint is unavailable, and that queue consumes capacity and needs monitoring. Track the number of pending events, the age of the oldest one and the free space in the pool that holds the queue. Set alerts before the queue becomes a capacity problem for the cluster. The 503 errors that clients receive when an endpoint stays down too long are a separate risk. The point at which they start, and whether they apply to persistent topics, depends on the release, so confirm it in the release notes and test it in a non-production environment by stopping a test endpoint for a controlled period.

Do not design the consumer on the assumption that every event arrives once and in order. Retries can deliver the same event more than once, and the documentation should be checked for what it guarantees about ordering and loss in your release. Make consumers idempotent by keying their writes on a stable identifier from the event, and plan how you will replay or reconcile objects when a queue has been dropped or an endpoint has been out longer than your retry window. Ask the team responsible for the pipeline to write down the expected behavior before go-live.

What goes wrong when an endpoint is down too long

Bucket notifications decouple object writes from delivery, so an endpoint outage rarely fails a write directly. The pressure shows up elsewhere: queues grow, clients see errors, consumers reprocess events, and filters or credentials quietly stop matching. Each card below names the symptom, the first check and the team that acts on it.

Endpoint unreachable for an extended period

When a Kafka broker, AMQP host or HTTP receiver stays unreachable, RGW keeps trying delivery according to the topic's persistence settings. Notifications for the outage window are at risk once retries stop or the queue reaches its limit, so the events are lost without an obvious error in the application. First check the RGW logs for connection failures to the endpoint named in the topic, then confirm the endpoint's own health from the RGW host. The platform team owns the RGW side, and the owner of the broker or receiver owns the endpoint.

503 responses reaching S3 clients

Clients writing objects can receive HTTP 503 errors when persistent notification delivery cannot keep up or the endpoint stays down past the point RGW is configured to tolerate. The write itself is the symptom your application sees, so the failure looks like a storage problem even though the cause is in the notification path. Check whether the 503s correlate with the endpoint outage in RGW logs and compare the timing with the topic's configuration before changing storage settings. The storage team owns the RGW configuration, and the application team owns client retry behavior.

Backlog growth in the persistent queue

During a long outage, undelivered notifications accumulate in the persistent queue that RGW keeps for each topic. A backlog that grows faster than it drains means that when the endpoint returns, delivery may take a long time and downstream pipelines fall behind the bucket state. Track queue size against the time since the last successful delivery, and confirm the queue location and capacity in your own environment before the outage, since limits depend on your RGW setup. The platform team owns the queue and drain plan, and the pipeline owner decides what lag is acceptable.

Duplicate or out-of-order consumer processing

Retried deliveries can reach a consumer more than once, and a backlog drained after recovery can arrive out of order with newer events already processed. Consumers that assume each event arrives exactly once and in sequence will write wrong state, such as applying a delete before the earlier create. Check whether each event carries the object key, version and event time so the consumer can detect repeats and stale updates, and design idempotent writes. The pipeline or application team owns consumer logic, and the data platform team validates the results.

Filter mismatches that drop events

Event filters on prefix, suffix or metadata can silently exclude events that consumers expect, especially when object keys or metadata naming change during an upgrade or a new upload tool is introduced. No error appears, so the gap is found late, often when a downstream report is missing records. Compare the filter rules on the topic with a sample of real object keys and test each filter with a known upload before relying on it. The team that owns the bucket naming standard and the team that owns the notification configuration should review changes together.

Certificate or SASL failures after a broker upgrade

After a Kafka upgrade, the endpoint may start rejecting connections from RGW if TLS certificates, SASL mechanisms or ACLs no longer match the client settings in the topic. Kafka 4.0 supports only KRaft mode and requires Java 17 for brokers, so the upgrade path itself can change what the endpoint accepts. Check the broker logs for authentication or handshake errors, then confirm the certificate chain and SASL mechanism configured for the topic match what the upgraded brokers expect. The Kafka operations team owns the broker change, and the platform team owns the RGW topic credentials.

Questions teams ask before setting up RGW notifications

How much effort does an RGW notification pipeline take to build?

Effort depends mainly on how many topics, consumers and event types you need, how much volume they carry, and who owns the message broker. A single topic feeding one consumer is a small build. Each added topic adds filter design, a delivery path and an owner for failures. Event volume affects broker sizing, the partition or queue layout, and how much retry and backlog handling you must test. Broker ownership matters most. If your organization already runs Kafka or RabbitMQ, the work is mainly configuration, access rules and consumer code. If the broker is new, the platform work comes first and the notification work follows it. A fixed-scope quote should price each topic and consumer separately, so you can see what changes when the count grows.

Can our internal team set up bucket notifications without outside help?

Yes, if your team already runs Ceph RGW and a messaging platform, and has time to test failure cases. The hard part is verifying behavior when an endpoint goes down, because that is where most surprises appear. Teams usually need experience in three areas: S3 administration on RGW, operating Kafka or RabbitMQ, and writing consumers that handle duplicate or late events. Check whether your engineers have recently done a broker upgrade or an access control change, since those tasks share skills with event delivery. Many organizations split the work. Internal staff own the consumers and business logic, while an outside team handles the platform build and the test plan. DeployOpen implements open-source platforms inside your infrastructure, so the platform part can be scoped and delivered separately.

What does the notification project need from the storage and messaging teams?

The storage team and the messaging team each hold part of the setup, and the project stalls when either one is missing. Plan for these inputs:

Storage team: RGW user and bucket ownership, the endpoint and TLS settings the gateway uses, and permission to change bucket notification configuration.
Messaging team: broker addresses, topic and partition or queue design, and a client identity for the gateway with the access rights to publish.
Both teams: agreed topic names, retention limits, and a named owner for alerts when delivery backs up.

For Kafka, clients authenticate with SSL or SASL. In KRaft clusters, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so confirm access rules before testing starts. Check each setting against the notification documentation for your Ceph release.

How long do testing and cutover take for bucket notifications?

The schedule follows the steps involved rather than a fixed number of days, so the useful question is which steps your environment requires. Testing usually covers successful delivery to each topic, filtered events that should never fire, behavior when an endpoint is down briefly, and behavior when it stays down long enough for RGW to return errors to clients. It also covers recovery of queued events and a load test at production volume. Cutover adds a step where producers and consumers switch over with a documented rollback path. Each step takes longer when consumers are not ready or when broker access needs an approval chain. Ask the proposal for the step list and the acceptance criteria for each step. Your own calendar will depend on change windows and on how many environments you test in.

What should we ask for in a fixed-scope notification quote?

Ask for a written scope that names each topic, each consumer and each event type, along with the Ceph and broker versions the design targets. Most DeployOpen engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts, so the document should state exactly what is delivered. Ask for:

A list of topics, filters and endpoints, with an owner for each.
The target versions and the upgrade path if your broker is behind. Kafka's downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases.
The test plan, including endpoint outage scenarios and acceptance criteria.
Exclusions, such as consumer business logic, monitoring changes or broker hosting.
Handover documents, and whether ongoing support is a separate retainer.

Separate one-time build work from recurring support so you can compare the quote with your own team's cost.

Do bucket notifications replace lifecycle rules or backup jobs?

No. Notifications tell downstream systems that an object changed, so they suit indexing, auditing, and triggering processing. They do not expire, tier or copy data on their own. Lifecycle rules handle expiry and transitions between storage classes. Backups protect against accidental deletion, corruption and site loss, and an event cannot undo any of these. A delete event that reaches a consumer does not restore the object. Keep these jobs separate in the design and document which system owns each outcome. If a consumer uses events to keep a copy in sync, test what happens when events are dropped or delayed, because the copy will drift unless something reconciles it.

Which broker should we choose for RGW events, Kafka or RabbitMQ?

Base the choice on the skills you already have, the support model you need, and the consumers that will read the events. Apache Kafka is licensed under Apache-2.0 and has no paid edition from the project itself. Kafka 4.0 supports only KRaft mode, so ZooKeeper-based clusters must migrate first. RabbitMQ server and its core plugins are licensed under MPL 2.0. RabbitMQ 4.3.6 was released on September 16, 2026, with community support until January 31, 2027, and longer support requires a valid commercial support license. Confirm which brokers your Ceph release supports for notifications in its documentation, and test the endpoint types you plan to use before you commit to one.

Securing the notification path

A bucket notification leaves RGW, crosses a network and lands in a broker or HTTP endpoint, and each hop needs its own authentication, encryption and access rule. The cards below cover those hops in order, then the data that travels along them. Events often carry object names and metadata, so the notification path falls under the same data governance as the objects themselves.

Endpoint authentication and TLS by consumer

Each consumer type authenticates and encrypts differently, so the topic endpoint has to be configured per type. For Kafka, HTTP and AMQP endpoints, confirm which TLS and authentication options your chosen RGW release supports, including whether certificate verification can be turned off and whether client certificates are accepted. Test each endpoint with the exact URI scheme you plan to use in production, since a fallback to plaintext can go unnoticed. Record the result in the release-specific test plan before go-live.

Kafka security options and SASL mechanisms

Kafka security is optional, so a cluster can run unsecured or mix secured and unsecured listeners. Use a dedicated secured listener for RGW with SSL/TLS, and choose one SASL mechanism from the documented set: GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER. Check which of these your RGW release can present as a client before you pick one, because the broker choice has to match what RGW supports. Avoid leaving an unsecured listener open for convenience during testing.

RabbitMQ access control and licensing

RabbitMQ server and its tier 1 core plugins are licensed under MPL 2.0, which governs how you distribute modified source, not who may connect to your broker. Access is controlled by users, virtual hosts and permissions you define, so create a dedicated RGW user limited to the exchange or queue it publishes to. Confirm which AMQP version and TLS settings your RGW release supports, since AMQP 1.0 became a core protocol in RabbitMQ 4.0 and the older AMQP 0-9-1 path remains the more common choice. Review the license terms with your legal team before packaging or redistributing any build.

Least-privilege credentials for RGW

The RGW side needs only enough access to publish events to the configured topic, not broad administrative rights on the broker or the object store. Create a separate broker account for each topic or zone, scope its permissions to the target resources, and rotate its secret on a schedule your security team sets. Keep the credential out of plain configuration files where your secret management allows it. Ask your RGW release notes which authentication method is supported for each endpoint type before you design the account model.

Object names and metadata in payloads

Event records typically contain the bucket name, object key, size, event type and selected metadata, and an object key can reveal customer identifiers or document types. Decide which fields a downstream consumer truly needs, and filter or redact the rest before data leaves the zone, using event filters where your release supports them. For example, a key such as a patient or account identifier should be reviewed before it is published to a shared topic. Document the field list so the data owner can approve it.

Retained events follow object governance

Persistent delivery stores events until the endpoint accepts them, and the broker keeps messages according to its own retention settings. That data is a copy of information about your objects, so it falls under the same classification, retention, access review and deletion rules as the objects. Set broker retention to the shortest period your consumers need, and confirm that deletion of an object or bucket is reflected in downstream copies according to your policy. Include broker storage, backups and logs in the data inventory.

Limits to confirm in your environment

Notification behavior depends on the exact Ceph release, the broker you target and the network between RGW and consumers. Confirm each item below against your own environment before you commit to a pipeline design, because the answers change retry handling, event coverage and where failures surface.

Ceph release and support window

Confirm which release you will run. The Ceph releases page lists Tentacle (20.2.x, first released November 18, 2025, estimated end of life June 1, 2027) and Squid (19.2.x, first released September 26, 2024, estimated end of life October 31, 2026) as actively maintained. Check the end-of-life date against your pipeline's planned lifetime, since a notification design that outlasts its release will need an upgrade first.

Supported event types and filters

Not every object event type or filter expression is available in every release. Ask for the list of event names and the filter syntax (prefix, suffix and metadata or tag filters) that your target release accepts, and test each one against a non-production bucket. Record which events you need, such as creates, deletes and multipart completions, so gaps are found before the pipeline depends on them.

Persistent queue behavior and 503 threshold

Persistent delivery retries events while an endpoint is unavailable, but the queue size, retry interval and the point at which RGW returns 503 to the client can differ by release. Ask for the exact persistent queue limits and the outage duration that triggers 503 responses, then reproduce an endpoint outage in a test cluster. Size the queue for your longest expected broker outage, and decide how you will alert when it starts to fill.

Broker versions and client libraries

Kafka and RabbitMQ each have their own support windows and protocol rules. For Kafka, confirm the broker version you will connect to against the supported releases list (4.3.1, 4.2.2 and 4.1.2) and check that the authentication mechanism you choose is accepted by your Ceph build. For AMQP, confirm the AMQP version your RabbitMQ release supports, because AMQP 1.0 became a core protocol in RabbitMQ 4.0.

Network paths and firewall rules

RGW hosts must reach each broker or HTTP endpoint directly, and the path must work for every RGW daemon, including those behind an ingress or on a separate subnet. Map the source addresses, destination ports and TLS requirements for each consumer, and confirm that firewall rules allow return traffic for any acknowledgments. Test from every RGW host, not only from one, because a single blocked host can cause events to queue unevenly.

Multisite zone event behavior

In a multisite deployment, confirm which zone emits the notification for a given write and whether replicated objects also fire events in the secondary zone. Duplicate or missing events are a common result when this is not planned. Ask the vendor documentation for your release how failover changes the event source, and rehearse the behavior before you rely on it for downstream processing.

How DeployOpen scopes and delivers notification pipelines

DeployOpen plans, implements, tests, documents and supports self-hosted open-source platforms inside your own infrastructure, including the Ceph Object Gateway and the message brokers that receive its bucket events. For bucket notifications, we start from the event consumers your pipelines need, then build and verify each topic, endpoint and retry path against them before handover.

How we scope

Written scope lists each bucket, event type, key filter and consumer that will receive notifications.
The scope names the target endpoint types, such as Kafka, AMQP or HTTP, and who operates each one.
We confirm which RGW release and Kafka or RabbitMQ versions your environment runs before work starts.

How we build

Topics, notification rules and event filters are defined as configuration and checked against test buckets first.
Persistent delivery is configured with retry behavior tested against a deliberately stopped endpoint.
Broker security, including TLS and authentication to the endpoint, is set up in your network and verified end to end.

How we hand over

Delivered documentation covers topic definitions, filter rules, endpoint settings and the tested failure behavior.
A runbook explains how to find queued or failed notifications and how to clear backlog after an outage.
Your team walks through test results and alert thresholds before the engagement closes.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Retainers can cover RGW and broker upgrades, so notification behavior is retested at each version change.
Ask any quote to state which endpoint, monitoring and on-call responsibilities sit with DeployOpen and which stay with your team.