Loki security hardening

Isolation in Loki depends on the tenant header, not on the stored data. When auth_enabled is true, Loki treats each request as belonging to the tenant named in the X-Scope-OrgID header, and it does not check whether the caller is entitled to that tenant. Any client that can set the header can read another tenant's logs. The control therefore sits outside Loki, at the layer that receives requests from people and shippers. Decide first which component owns that decision, and make sure nothing else can reach Loki's HTTP port without passing through it.

The reverse proxy must derive the tenant from the authenticated identity and overwrite whatever the client sent. Strip any inbound X-Scope-OrgID before the request reaches Loki, then set it from a mapping keyed on the identity provider's user or group claim. For people querying through Grafana, that mapping usually means one tenant per team or per environment, with a written list of who may see which tenant. Loki's authentication documentation names nginx, HAProxy, OAuth2 Proxy and Pomerium as examples of proxies to run in front of it. Confirm how your chosen proxy removes and sets request headers, and test that a forged header still resolves to the tenant the identity allows.

Shippers need the same discipline. Each log shipper writes on behalf of a source system, so give each source its own credential at the proxy, and let that credential map to one tenant. Grafana Alloy, Fluent Bit, Fluentd, the Docker logging driver and OpenTelemetry HTTP clients all need the tenant set in some way, so confirm for each one how it sends the header and whether the value can be pinned in configuration. Avoid a shared write credential that can write to every tenant. A compromised host holding that credential could plant data in other tenants and obscure its own activity there. Record which source maps to which tenant in your inventory.

Test cross-tenant reads before go-live and after every change to the proxy or the Grafana data source. Create two test tenants, write a marker line into each, and then query for tenant B's marker from a session authenticated as tenant A. The result should be empty, and it should stay empty when the test client adds a forged X-Scope-OrgID header. Also try reaching Loki's port directly from the network segment where the proxy sits, and confirm the connection is refused. Repeat the test with a shipper credential and with a user whose group membership has just changed. Keep the test scripts in version control so they run again after each upgrade.

Be clear about what AGPLv3 Loki provides. It does not supply label-based access control or cross-tenant query features on its own. Grafana Enterprise Logs (GEL) documents token authorization, label-based access control and cross-tenant queries as its own capabilities, so confirm those features in the GEL documentation before assuming they fit your model. If you need per-user restrictions within one tenant, or one query spanning several tenants, the open-source design has to meet that requirement with proxy policy and separate tenants, and the options are narrower. Ask your implementation partner to write down which of these capabilities your design depends on.

Loki retention is not on by default. The compactor applies retention, and it only does so once retention_enabled is set. Until then, logs stay in object storage indefinitely, which is the opposite of what most security and privacy owners expect when they approve a logging stack. Before the first production log arrives, the team should confirm that retention is enabled, that the compactor is running with the expected configuration, and that someone has checked the bucket size after the first retention cycle rather than assuming it worked.

Retention can be set globally, per tenant, or per stream selector. A global value is the simplest starting point, but it applies the same clock to every log the stack holds. Per-tenant rules let a team keep application logs for one period and audit-relevant logs for another, provided each tenant is mapped to a clear owner. Per-stream selector rules match streams by label, so they work only when the labels that identify sensitive sources are consistent across shippers. Each stream rule must be at least 24 hours, so the team should write down the shortest period each data class can tolerate and check that the configuration reflects it.

Sensitive streams are easier to govern when they are isolated. Routing them to their own tenant, or at least to a distinct stream selector, keeps their retention, access and deletion requests separate from general operational logs. This matters most when the sensitive stream has a shorter or longer required period than everything else. Without isolation, a single global setting forces a compromise that either keeps sensitive data longer than the policy allows or deletes operational logs that engineers still need. The team should decide the tenant or label scheme before ingestion starts, because moving historical data between tenants later is a separate exercise.

Retention in Loki does not reach everything that holds a copy of a log line. The team must confirm how data already written to object storage will be removed when a retention period changes, and whether the chosen method meets the deletion requirement for that data class. Backups of the bucket, replicas held by log shippers such as Grafana Alloy or Fluent Bit buffers, and any exports or dashboards snapshots in Grafana may keep data after Loki has deleted it. Each copy needs an owner, a retention period and a documented deletion step. Confirm the behavior for each with the relevant vendor documentation and by testing in your own environment, since backup tooling and object storage lifecycle features differ by provider.

Each retention decision should be recorded in a form that security and privacy owners can review and that an independent auditor can later trace. The record should name the tenant or stream selector, the period, the legal or policy basis, the systems holding copies, the deletion method, and the person who approved the setting. Store the Loki configuration in version control so every change has a commit, reviewer and date. Schedule a periodic check that the live configuration still matches the record. DeployOpen can prepare this evidence as part of a hardening engagement, but the retention period itself is a decision the client's own owners must make and sign.

Test the Loki hardening by trying to break it on purpose, and treat every control as unproven until a failing request is refused. The goal is to show that each control fails closed: when a token is missing, a header is forged, a port is reached directly, or a bucket policy is too broad, the request is denied and the denial is recorded. Run these tests in a non-production environment that mirrors the production topology, with the same reverse proxy, TLS configuration, object storage bucket and compactor settings. Write each test case with a name, the control it covers, the exact request or action, the expected result and the observed result.

Start with the front door. Send a push request and a query request to the endpoint that sits behind the authenticating reverse proxy, with no credentials, and confirm both are refused. Repeat with an expired or invalid token. Loki has no built-in authentication layer, so the proxy is the only thing standing between a caller and the API, and these cases show whether it is actually enforcing that. Then attempt to reach the Loki HTTP and gRPC ports directly from a network segment outside the proxy. The connection should fail at the network layer. Confirm this from every network path that exists, including the cluster network and any load balancer that could bypass the proxy.

Next test tenant isolation. Authenticate as tenant A, then send a request that sets X-Scope-OrgID to tenant B. The proxy must overwrite or reject that header, because Loki trusts whatever tenant identifier it receives when auth_enabled is true. A query that returns tenant B data is a critical failure. Repeat for push requests, where a forged header could place logs into another tenant. Then test object storage. Using credentials for one identity, attempt to read and list objects that belong to another tenant or to the Loki system, and attempt an unencrypted write if your policy requires server-side encryption. Each attempt should be denied by the bucket policy or IAM role, not only by Loki's own configuration. Confirm the bucket policy and encryption requirement with your storage vendor's documentation and your own environment.

Test TLS on each hop: client to proxy, proxy to Loki, Loki to object storage and the components that talk to each other. Use a tool to connect without TLS, with an unsupported protocol version and with a certificate that does not match, and confirm each is refused. Then test retention in a test tenant only. Set a short, explicit retention rule for a stream selector, enable retention in the compactor, push known test entries, and confirm that entries past the rule are removed after the compactor runs. Retention is disabled by default, so also confirm that it is actually enabled. Finally, check that logs and errors do not expose tokens or credentials. Trigger authentication failures, a failed object storage call and a misconfigured shipper, then search the Loki, proxy and shipper logs for token strings, authorization headers and access keys.

Record every test as evidence: the test ID, date, environment, version of Loki and the proxy, the command or request used, the expected and observed result, and the name of the person who ran it. Store raw outputs, such as HTTP status codes, proxy logs and bucket policy simulation results, alongside a short summary. Keep the test plan under version control so changes are traceable. This record shows what you tested and what happened. It does not constitute an audit report, an attestation or a certification, and it should not be presented as one. If a compliance assessor needs evidence for a specific control, ask them which artifacts they require and map your records to those requirements.