Centralized Ceph logging with Loki and Alloy

Collect daemon logs from every Ceph host in one place by enabling Loki and Alloy through cephadm, or by shipping them to a log platform you already run.

Book a callTalk to us
On this page

Centralized Ceph logging with Loki and Alloy: when a cluster needs it

When a Ceph cluster spans many hosts, each daemon writes its logs to its own node. A failed write or a slow request can involve OSDs on three hosts, a monitor on a fourth and a gateway on a fifth, so the on-call engineer ends up opening a shell on each machine and lining up timestamps by hand. cephadm can deploy Loki and Alloy as services in the cluster to collect daemon logs into one place. The alternative is to skip the bundled stack and ship logs to a log platform your organization already runs. Either way, the decision is about who operates the log store and where log data is allowed to live, not about whether Ceph logs are useful.

Loki stores compressed log chunks and an index in object storage and is queried by label, so it suits high volumes of daemon logs when you want retention you control. Alloy, Grafana's vendor-neutral distribution of the OpenTelemetry Collector, is the shipper Grafana supports for Loki. Teams usually need this when they run a cluster across several racks or sites, when security or audit reviews ask for retained daemon logs, or when they cannot keep relying on access to individual hosts. Before you commit, confirm which Ceph release and cephadm version you run, which object store will hold the chunks, and whether the bundled services fit your air-gap and retention rules. Grafana licensing states that Grafana Labs moved Loki and its other core projects from Apache License 2.0 to AGPLv3, effective with an announcement on April 20, 2021. AGPLv3 carries network-use obligations for modified versions, so have your legal team review it before you change or embed the software.

Plan the log store as a service in its own right. Loki's retention is off by default, which means logs accumulate until you enable it, so set retention deliberately and record who approved the period. Loki has no built-in authentication, so the query endpoint needs an authenticating reverse proxy in front of it, and multi-tenant access depends on the X-Scope-OrgID header being set correctly. Ask your implementation partner or your own team to confirm the cephadm release notes for Loki and Alloy services in your Ceph version, the sizing of object storage for your daily log volume and the process for upgrading the log stack separately from the cluster. Those answers decide whether the feature stays a convenience or becomes a dependency that your incident process relies on.

Choosing between cephadm-managed Loki and an existing log platform

A cluster with no log platform today gets the most complete picture from cephadm-managed Loki and Alloy. Both run as services placed on hosts you choose, and your team owns the storage, retention and access model. Choose the Loki deployment mode from measured volume, not from the cluster size. The documented monolithic mode fits up to about 20 GB of logs per day. Simple scalable mode is documented for volumes approaching 1 TB per day, but the Loki documentation says it is being deprecated and will be removed with Loki 4.0, so a new build should not depend on it. Very large volumes point to microservices.

  • What cephadm deploys: the Loki and Alloy services when you enable them, placed on the hosts you select. Confirm the service specifications for your release in the cephadm documentation.
  • Mode selection: start with monolithic for volumes up to about 20 GB per day, and plan the path if growth moves you beyond that range.
  • Storage backend: Loki stores chunks and its index in object storage. Ceph RGW is S3-compatible, but the Loki documentation notes that S3-compatible stores are not typically recommended for production, so confirm this with Grafana Labs for your design.
  • Retention: retention is disabled by default, so logs accumulate until you set it explicitly.
  • Access: Loki has no built-in authentication, so an authenticating reverse proxy must sit in front of it.

Before choosing a mode, measure daily log volume from a representative sample period on your hosts.

Configuration options and when each fits

This table compares the logging paths, object stores, index choices, retention and tenancy settings that shape a Loki and Alloy design for Ceph daemon logs. Read each row as a decision: the right column names when an option fits and what to confirm before you commit. The main takeaway is that the logging path matters less than where Loki keeps its chunks and index, and how retention and tenant headers are set from the start.

DecisionOptionWhen it fits
Logging pathcephadm-managed Loki with AlloyYou want the Ceph cluster to carry its own log store and have capacity to run Loki and Alloy as cephadm services on hosts you control.
Logging pathAlloy forwarding to an existing LokiYour organization already runs a Loki cluster with tenant and retention rules, and Ceph logs should join it without a second store.
Logging pathAlloy forwarding to another log platformYour central platform is not Loki. Confirm which Alloy or OpenTelemetry outputs that platform accepts before you design around it.
ShipperFluent Bit, Fluentd or Docker logging driverYou already standardize on one of these shippers. Loki accepts data from them, but Alloy is Grafana's supported log shipper for Loki.
Object storeAmazon S3, Google Cloud Storage or Azure Blob StorageGrafana's recommended production object stores. Choose this when the cloud account and data residency rules allow it.
Object storeMinIO or local filesystemSupported by Loki but not typically recommended for production. Suitable for labs or small evaluations, so confirm your durability needs first.
Index storeTSDB indexThe recommended index for Loki 2.8 and newer. Older index stores such as DynamoDB, Bigtable and Cassandra are deprecated, so do not plan new builds around them.
RetentionCompactor retention enabled with a set periodRetention is disabled by default, so logs are kept indefinitely until you enable it. Set global, per tenant or per stream rules, each at least 24 hours.
Tenancyauth_enabled with X-Scope-OrgID per sourceUse when several teams or clusters share one Loki. Every request must carry the tenant header, and an authenticating reverse proxy must sit in front of Loki.
Deployment modeMonolithic, simple scalable or microservicesMonolithic suits small log volumes. Simple scalable is deprecated and removed in Loki 4.0, so confirm the path before choosing it for new work.

Enabling Loki and Alloy in cephadm step by step

This sequence takes a cephadm-managed cluster from no central logging to Loki and Alloy running as orchestrated services, with retention enforced and every host's daemon logs searchable behind an authenticating proxy. Each step closes with a check you can run before moving on. Confirm exact service spec fields against the cephadm and Loki documentation for your versions.

  1. Confirm release support and host health

    Check that the cluster runs Squid or Tentacle, the two actively maintained releases, and note their estimated end-of-life dates: October 31, 2026 for Squid and June 1, 2027 for Tentacle. If the cluster runs an older release, plan the upgrade before adding logging, since cephadm was introduced in v15.2.0 and does not support releases before that. Run ceph orch host ls and ceph health detail to confirm every host is online and the cluster has no unresolved errors. Done when every host reports as online and the orchestrator answers without errors.

  2. Choose and provision the log object store

    Loki keeps compressed log chunks and its index in object storage, so decide which store will hold them before enabling the service. Loki's recommended production stores are Amazon S3, Google Cloud Storage and Azure Blob Storage, while S3-compatible stores such as MinIO are supported but not typically recommended for production, so confirm the choice against the documentation for your Loki version. A Ceph Object Gateway bucket is workable only if you accept that a cluster outage would also take out your logs. Create the bucket and a dedicated credential with write access, then write and read a test object with that credential. Done when the test object round-trips successfully.

  3. Enable Loki and Alloy through the orchestrator

    Apply a service specification for Loki and another for Alloy through the cephadm orchestrator with ceph orch apply -i, using the service spec fields documented for your Ceph release. Let cephadm place the containers on the hosts you select, and keep the spec files in version control so placement and storage settings can be reviewed and rebuilt. Use ceph orch ls to confirm both services are listed with the expected daemon counts, and ceph orch ps to see the daemons in a running state. Done when both services report their expected daemon counts and every daemon is running.

  4. Point Alloy at daemon logs on each host

    Configure Alloy to read the Ceph daemon logs on every host and attach labels that identify the host, daemon type and daemon ID, so each stream can be filtered later. Alloy is Grafana's supported log shipper for Loki and a vendor-neutral distribution of the OpenTelemetry Collector. Confirm the log paths that your cephadm release writes for each daemon type, and make sure the Alloy container can read them. Reconcile the service and check that each host appears in the labels. Done when a query for each host returns recent lines from its OSD, monitor and manager daemons.

  5. Set retention with retention_enabled and stream rules

    Retention in Loki is disabled by default, so logs accumulate indefinitely until you enable it. Set retention_enabled in the compactor configuration, then define retention globally, per tenant or per stream selector, with each stream rule lasting at least 24 hours. Have your compliance owner set the period, since it should reflect incident review and audit obligations. Confirm the compactor is running and has loaded the settings. Done when the compactor reports retention as active and a test stream rule matches only the labels you intended.

  6. Configure the authenticating reverse proxy

    Loki has no built-in authentication layer, so place an authenticating reverse proxy such as nginx, HAProxy, OAuth2 Proxy or Pomerium in front of it. If you set auth_enabled, Loki becomes multi-tenant and expects an X-Scope-OrgID header on every request, so the proxy must set or validate tenant identity. Make sure Alloy and Grafana send their tenant headers through the proxy. Done when a request without credentials is refused and a request with valid credentials reaches Loki under the expected tenant.

  7. Verify logs from every host arrive with labels

    Run a query for each host and daemon type, and confirm that entries arrive with the labels defined in the Alloy configuration. Compare the number of hosts listed by ceph orch host ls with the number of distinct host labels in Loki, and check that timestamps are current. Trigger a known, low-risk event, such as a daemon restart in a maintenance window, to confirm the line appears within the latency you expect. Record the label names and tenant identifiers so on-call engineers can query logs during an incident. Done when every host has recent logs under the expected labels.

Securing the log path and access to Ceph logs

Logs from every Ceph host are a concentrated record of hostnames, client identifiers and error detail, so the path from daemon to Loki needs the same access control as the cluster itself. The cards below cover the gateway in front of Loki, tenant separation, storage credentials, retention and patching. Each one is a decision to make before the log platform goes into production.

Authenticate every request at the edge

Loki has no built-in authentication layer, so an authenticating reverse proxy must sit in front of it. Options named in the Loki documentation include nginx, HAProxy, OAuth2 Proxy and Pomerium. Confirm which proxy your identity provider integrates with, that it terminates TLS, and that Alloy and any Grafana data source authenticate through it. Direct access to the Loki port should be blocked at the network level.

Treat log content as sensitive data

Ceph daemon logs can include host names, rados object names, client identifiers and detailed error text that reveal how your storage is laid out. Decide who may read which hosts and which daemon types before you grant access. Review sample log lines from each daemon type during rollout, and note which fields you need to mask or drop in Alloy before shipping.

Separate tenants with auth enabled

Loki is multi-tenant only when auth_enabled is set to true, and it identifies each tenant through the X-Scope-OrgID header on every request. Enable it before the first log is written, because changing it later affects how existing data is read. Map each team or environment to a tenant and confirm that the proxy sets or validates that header so a client cannot choose another tenant's ID.

Scope object store credentials and encryption

Loki keeps compressed chunks and its index in object storage, so the credentials it uses are a direct route to every log. Create a dedicated bucket and a least-privilege key for Loki, rotate it on a schedule, and keep it out of plain-text configuration where your secrets tooling allows. Confirm with your storage vendor how data is encrypted at rest and in transit, since Loki does not provide that by itself.

Use retention as data minimization

Retention is disabled by default, which means logs are kept indefinitely until retention_enabled is set. Decide how long each tenant's logs are needed for operations and for any policy you must meet, then configure retention globally, per tenant or per stream selector. Each stream rule must be at least 24 hours. Delete on a schedule that your privacy and records team has approved, and confirm the same period applies to backups.

Patch Loki, Alloy and Ceph together

The log path has three moving parts that each receive their own releases: Loki, Alloy and the Ceph cluster that produces the logs. Track the Loki release list, the Alloy release notes and the Ceph active release schedule, and assign an owner for each. Test log shipping after every upgrade, because a broken Alloy pipeline can go unnoticed until an incident needs the logs.

How retention and log volume shape the design

Retention is the first design decision for this stack, and the default works against you: Loki's retention is disabled by default, so logs are kept indefinitely until you set retention_enabled to true. Retention is enforced by the compactor, which means the compactor must be running and configured before any deletion happens. You can apply retention globally, per tenant, or per stream selector. Each stream rule must keep logs for at least 24 hours. Decide up front which Ceph daemon logs belong in long-term storage (for example, monitors and OSDs for incident review) and which can expire quickly, because a single global period is rarely the right answer for both. Confirm the exact configuration keys against the Loki retention documentation for the release you deploy.

Log volume is what drives storage and compute, and it moves with three factors you control. The first is verbosity: Ceph daemons log more at higher debug levels, and a cluster under recovery or repeated slow requests can produce far more output than a healthy one, so measure during normal operation and during a rebuild. The second is the number of daemons and hosts shipping logs, since every OSD, monitor and manager adds a stream. The third is the Loki deployment mode you choose. The monolithic mode suits small volumes, while the simple scalable mode separates read, write and backend targets; Loki's documentation marks simple scalable as deprecated ahead of its removal in the 4.0 release, so plan a path to microservices for growth. Object storage capacity, ingester memory and query load all follow from these inputs.

Sizing must come from measured log volume in your own environment, not from published throughput ceilings or generic examples. Before you commit to hardware or a bucket size, run the candidate verbosity settings on a representative cluster for at least one full business cycle, record bytes per day per host and per daemon type, and add headroom for incident periods. Ask your implementation partner for a written sizing worksheet that states the assumptions behind each figure, and ask for a retention test that confirms old data is actually removed. Also confirm your object store choice: Loki's production guidance names Amazon S3, Google Cloud Storage and Azure Blob Storage, while S3-compatible stores such as Ceph RGW itself are supported but not typically recommended for production, so validate that choice against the Loki storage documentation before relying on it.

Limits to confirm before you commit

Several parts of this design depend on versions, licenses and vendor roadmaps that change between releases. Confirm each item below with the vendor or in your own environment before you fix the architecture, the storage sizing or the retention policy.

Cephadm service support in your release

Confirm that cephadm service definitions for Loki and Alloy exist in the Ceph release you run, not only in the release you plan to move to. Check whether they are included by default, whether they are marked as experimental, and which container images they pull. If they are missing, you will need to run Loki and Alloy outside cephadm, which changes how you handle upgrades, certificates and host placement. Note the Ceph release you run and the one planned for the upgrade, then check both.

Loki license obligations under AGPLv3

Grafana Loki is licensed under AGPLv3, which Grafana Labs adopted in April 2021 after Loki had been under Apache License 2.0. Ask your legal team whether AGPLv3 obligations apply when you modify Loki or offer it to users over a network, such as a log search portal shared with a partner or customer. Confirm whether your organization has a policy on AGPL components before you deploy, because the answer affects whether you can patch Loki privately and what you must publish if you do.

Loki, Alloy and object store compatibility

Check that the Loki and Alloy versions you select support your object store, and that the store is in the recommended list for production. Amazon S3, Google Cloud Storage and Azure Blob Storage are the recommended production stores. S3-compatible stores such as MinIO are supported, but the docs do not typically recommend them for production. If your object store is a Ceph RGW cluster, test the chunk and index write path under your expected ingest volume before you commit to it.

Simple scalable mode before Loki 4.0

The Loki documentation states that simple scalable deployment mode is being deprecated and will be removed with the Loki 4.0 release. Ask the vendor for the planned migration path and its timing, and whether a supported upgrade exists from simple scalable to microservices or another mode. If you size for simple scalable now, budget for a deployment change and a storage migration later. Monolithic mode is documented for smaller volumes, so confirm where your expected daily volume falls among the documented modes.

Ingest path into your existing log platform

If you already run a log platform, confirm whether it can receive data from Alloy directly, or whether it needs a different shipper such as Fluent Bit, Fluentd or an OpenTelemetry pipeline. Loki also accepts OpenTelemetry logs over HTTP, so a shared collector may simplify the design. Check the supported input formats, field mapping for Ceph daemon log lines, and whether the platform's license or ingest pricing changes when you add a second copy of the logs.

Retention rules and their owners

Retention in Loki is disabled by default, so logs stay indefinitely until you enable it in the compactor. Decide who owns the retention policy, whether it differs by Ceph cluster or by tenant, and how it maps to any log retention requirement your auditors test. Confirm the minimum stream rule duration in your Loki version, and confirm that retention deletes only what you intend to delete before you enable it on production logs.

Questions teams ask before bringing in outside help

How much does a centralized Ceph logging setup cost to run?

No price for this work is set in the supplied facts, so the figure depends on your scope and environment. The cost splits into three parts: object storage for log chunks and the index, compute for the Loki and Alloy components, and staff time for setup, retention decisions and on-call support. Storage and compute recur every month and grow with log volume and retention length. Setup labor is one-time, while upkeep such as upgrades and tuning recurs. When you request a quote, ask for each part separately, the storage assumptions behind it, and what is included in ongoing support. DeployOpen issues a quote from a written scope rather than a list price.

How long does a Loki and Alloy logging project take?

Duration depends on how many hosts ship logs, which object store you use, how many tenants and identity integrations you need, and whether dashboards and alerts must be built. The Loki deployment mode you choose also matters: the documentation describes a single-binary monolithic mode for smaller volumes, a simple scalable mode for larger ones, and microservices for very large clusters. Note that simple scalable mode is being deprecated and will be removed with Loki 4.0, so a new build should account for that. Ask for a written timeline tied to the scope, and confirm which items are in the first phase.

What does the client need to provide for this work?

You need to provide several things before work starts. The main items are an object store for Loki chunks and index, such as an S3-compatible service on your Ceph cluster or another approved store, since the documentation says local filesystem and some S3-compatible stores are not typically recommended for production. You also need DNS records and TLS certificates for the endpoints, an identity source for the authenticating reverse proxy that must sit in front of Loki, and a named operator who can respond to alerts about ingestion or storage problems. Confirm network paths from every Ceph host to the Alloy and Loki endpoints during planning.

Can our team build this ourselves using the Ceph and Grafana documentation?

Yes, a capable team can build it from the documentation, and the cephadm and Loki guides cover the core steps. The harder parts are rarely the first install. They include choosing retention per tenant or stream, securing Loki with an authenticating proxy because it has no built-in authentication layer, sizing object storage for your daily volume, and testing that logs survive a host failure. Consider whether your team has time to own upgrades and incident response for the logging stack, since that work recurs. A partner can cover the design and build while your staff learn the operating model.

When is shipping logs to an existing log platform cheaper to run?

Shipping to an existing platform is usually cheaper when your organization already has a log platform with spare licensed capacity, trained staff and retention rules that meet your requirements. In that case you avoid running a second storage system and its upgrades. Alloy can forward logs from Ceph hosts to other destinations, and Loki also accepts Fluent Bit, Fluentd and OpenTelemetry logs over HTTP, so the collection layer can often stay the same. A new Loki stack is more likely to make sense when you need open-source control over the data path, want Ceph logs separated from other workloads, or lack an existing platform with suitable access controls. Compare total running cost, not license line items alone, and check the licensing terms of any platform you choose. Grafana Loki itself is licensed under AGPLv3, which your legal team should review.

How long do we keep Ceph logs, and who decides?

Retention is a policy decision that your security, compliance and operations teams should make together before build. In Loki, retention is applied by the compactor and is disabled by default, so logs stay indefinitely until you enable it. Retention can be set globally, per tenant or per stream selector, and each stream rule must be at least 24 hours. Longer retention raises object storage use and recurring cost, so the answer should match what auditors and incident responders need. Confirm the retention periods in writing and record who approves changes.

How do we keep the logging endpoint secure?

Treat the logging endpoint as sensitive, because Ceph logs can reveal hostnames, cluster topology and error details. Loki has no built-in authentication layer, so you must place an authenticating reverse proxy in front of it, such as nginx, HAProxy, OAuth2 Proxy or Pomerium. If you enable multi-tenancy, each request carries an X-Scope-OrgID header that identifies the tenant, and the proxy should control which identities can set it. Plan encrypted transport, access reviews for the Grafana data source, and a clear rule for which team can query which tenant.

What DeployOpen confirms in writing before deployment

Most engagements are fixed-scope, so the written scope is agreed before any work starts, and a quote follows from that document. For centralized logging, the scope should name the Ceph release the cluster runs, the hosts that will ship logs, and whether cephadm manages the Loki and Alloy services or whether they run separately. It should also record the object store that will hold Loki chunks and index data, the retention rules (including whether retention_enabled will be set and which stream selectors get shorter or longer periods), and the tenant model if auth_enabled will be true and each team sends an X-Scope-OrgID header.

The scope should also name the authenticating reverse proxy that sits in front of Loki, since Loki has no built-in authentication layer, and the Grafana access model that goes with it. If your organization already runs a log platform, the scope must say so and define what Alloy forwards to it, in what format, and who owns the receiving side. The handover documentation should be listed as a deliverable, including the retention settings, tenant mapping, proxy configuration and the steps to restore a log bucket. Confirm the Ceph release and the Loki version against the supported versions at the time of contract, since both change.

DeployOpen prepares the logging system, its configuration and the evidence an assessor would ask to see, such as retention settings, access controls and change records. It does not issue audit reports, attestations or certifications, and independent auditors or certification bodies make those determinations. If a compliance review depends on log retention, ask your assessor in advance which records they expect, and write those artifacts into the scope so they are produced and handed over before deployment closes.

How DeployOpen delivers centralized Ceph logging

DeployOpen plans and builds centralized logging for Ceph in infrastructure you control, starting from which daemons, hosts and retention periods matter to your team. The work covers Loki and Alloy as cephadm services or shipping into a log platform you already run, with access, retention and runbooks agreed in writing before the build starts.

How we scope

  • Inventory which Ceph daemons, hosts and log sources must be collected, and where they run today.
  • Decide between cephadm-managed Loki and Alloy or shipping logs to an existing log platform.
  • Agree retention, tenant layout, Grafana access and the storage backend for log data in writing.

How we build

  • Enable and configure the Loki and Alloy services through cephadm, using the service specifications agreed in scope.
  • Set retention in the Loki compactor and confirm the object store holds log data as planned.
  • Test that logs from every host arrive, that labels make daemon logs searchable, and that alerts on log loss work.

How we hand over

  • Document the deployed architecture, service specifications, retention settings and access model in a handover pack.
  • Provide runbooks for adding hosts, rotating credentials, changing retention and recovering the log store.
  • Walk your operations team through dashboards, queries and the procedure for escalating a logging fault.

How we support

  • Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
  • Ongoing support, Loki and Ceph upgrades and continuous testing are available as a retainer.
  • Confirm with DeployOpen which upgrade path and log platform changes are covered in your retainer terms.

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com