How APISIX access logs reach Loki with the loki-logger plugin
The loki-logger plugin pushes APISIX request logs to Loki over HTTP, so gateway traffic can be queried in Grafana next to logs from the services behind it. Today the platform team's gateway logs stay on the gateway, where they are hard to correlate with application and identity logs during an incident. The plugin is enabled per route, so the team decides which routes ship logs and which fields leave the gateway. Confirm the plugin's configuration keys, batching options and supported fields against the APISIX documentation for the version you run, because plugin options change between releases. Before rollout, agree a log format that keeps client identifiers, tokens and sensitive query strings out of the stream, since every field written to Loki becomes searchable by anyone with read access.
Loki separates data by tenant only when multi-tenancy is enabled. With auth_enabled set to true, Loki reads the tenant from the X-Scope-OrgID header on every request, and a push without that header is rejected. The loki-logger plugin must therefore send the header with each batch, and the value should map to a tenant your team has agreed with the owners of the other data in that Loki cluster. If the gateway serves several business units, decide whether routes map to one tenant each or whether a shared tenant with stream labels is enough, because that choice determines who can query which traffic and how retention is set per tenant. Confirm the header configuration key and the label set in the plugin reference before you test. The Loki multi-tenancy documentation explains the header model in full.
Loki has no built-in authentication layer, so the push endpoint must sit behind an authenticating reverse proxy. Loki's documentation names nginx, HAProxy, OAuth2 Proxy and Pomerium as examples. In this design, APISIX sends each push to the proxy with credentials the proxy checks, and the proxy forwards the tenant header to Loki unchanged. Plan the proxy's own access control, TLS and certificate rotation as part of the build, and make sure the proxy does not let a client choose a tenant it is not entitled to write to. Test the path with three checks: a rejected unauthenticated push, an accepted authenticated push, and a query in Grafana that returns the expected entries. Ask your team to confirm which credential methods the proxy supports before you write the plugin configuration, since that choice drives the plugin settings.
What you need before enabling the loki-logger plugin
Most delays in a Loki logging rollout come from decisions made before any gateway configuration is touched. The cards below cover the version, the Loki side, the tenant model, the access layer and the network path, so the plugin can be enabled once and verified in one pass.
Confirm the APISIX version
Check the running release against the downloads page before writing plugin configuration, since option names and defaults can change between releases. The downloads page lists APISIX 3.19.0 as the latest release, but it does not mark any release as long-term support, so your team should decide which line it will standardize on and track its patch schedule. Record the exact version in the change ticket so the plugin behavior you test matches what runs in production.
Reachable Loki push endpoint
The gateway nodes must be able to reach the Loki push endpoint over HTTP or HTTPS, and that endpoint must be the write path of your chosen deployment, not a Grafana query address. Confirm the URL with the team that runs Loki, and test it from each gateway node with a sample push so that firewall rules, DNS and proxy settings are proven before the plugin is enabled. A failed push at this stage is far easier to diagnose than a silent gap in logs later.
Auth status and tenant design
Decide whether Loki runs with auth_enabled set to true. When it is true, Loki identifies tenants with the X-Scope-OrgID header on every request, so the plugin must send a tenant value that matches the team or environment that owns the data. Agree on tenant names, for example one per environment or per business unit, before rollout, because changing them later splits history across tenants and complicates retention and access rules.
Authenticating reverse proxy
Loki has no built-in authentication layer, so the push endpoint needs an authenticating reverse proxy in front of it, such as nginx, HAProxy, OAuth2 Proxy or Pomerium. Decide which credential the gateway presents, how that credential is stored and rotated, and which path the proxy exposes to the gateway. Without this layer, anyone who can reach the Loki port can write or read logs, which is a material exposure for gateway logs that contain client identifiers.
TLS certificates and trust
Plan the certificate chain for the Loki endpoint and make sure every gateway node trusts the issuing authority, including any internal CA used in restricted networks. Decide who issues, renews and deploys the certificates, and record the renewal date so an expired certificate does not stop log delivery. Confirm the TLS settings with your security team, because the gateway holds the credential used to write to Loki.
Deployment mode for log volume
Choose the Loki deployment mode before the first push, based on the daily log volume you expect from the gateway. Loki documents monolithic mode for up to about 20 GB of logs per day, simple scalable mode for larger volumes, and microservices mode for very large clusters. Simple scalable mode is being deprecated and will be removed with Loki 4.0, so a new deployment should plan for a path beyond it. Measure the current access log size per day on a representative route set to make the choice on evidence.
Configure the loki-logger plugin step by step
This sequence takes APISIX request logs from one test route to a queryable Loki tenant in Grafana. Each step ends with a check, so the team confirms the step worked before the plugin reaches production routes.
Confirm Loki accepts pushes and tenant headers
The platform or observability engineer confirms the Loki side before touching the gateway. Loki must run with multi-tenancy enabled, and every request must carry the tenant header, which is set through X-Scope-OrgID. Because Loki has no built-in authentication, send the test push through the authenticating reverse proxy you plan to use in production. Done when a test line pushed for one tenant is returned by a query for that tenant and does not appear when a different tenant is queried. The Loki multi-tenancy docs cover the header behavior.
Enable the plugin on a test route
The gateway engineer creates or selects a dedicated test route, then enables loki-logger on that route only through the Admin API. In traditional deployment mode the Admin API listens on port 9180, and client traffic enters on port 9080. Restrict the Admin API to administrators and automation, because the plugin configuration will hold the Loki endpoint and tenant settings. Done when the route configuration returned by the Admin API lists loki-logger as enabled, and a request to the test route through port 9080 returns the same response it returned before the change.
Set the Loki endpoint, tenant and labels
The engineer enters the Loki push endpoint, which should be the reverse proxy address, the tenant identifier for this traffic, and a small set of static labels such as gateway name, environment and route group. Loki indexes labels, so avoid per-request values like request IDs, client IP addresses or user identifiers as labels. Keep those in the log body instead. Confirm the exact configuration keys and their accepted values in the APISIX loki-logger plugin reference before you apply them. Done when the Admin API accepts the configuration without validation errors and the APISIX error log shows no delivery errors after a test request.
Tune batch size and flush settings
The engineer sets the batch size, flush or inactive timeout, and retry settings that the plugin reference documents for loki-logger. Larger batches reduce the number of push requests to Loki but delay when lines become queryable and hold more entries in memory until they are sent. Ask the Loki operator what ingestion limits apply to the tenant so batch sizes stay within them. Send a short burst of test requests and watch the timing. Done when the test lines arrive in Loki in grouped pushes within the flush window you expect, and the gateway logs show no dropped or repeatedly retried batches.
Define the log format
The engineer defines the log format with the APISIX variables the team needs, typically the method, path, status, latency, route identifier, upstream address and consumer name. Exclude headers that carry credentials, such as Authorization and cookies, and check query strings for secrets before logging them. Write the entry as structured JSON so analysts can filter on fields with LogQL parsing instead of creating extra labels. Done when a sample line in Grafana Explore parses as JSON, contains the expected fields, and contains no token or cookie values.
Verify entries and labels in Grafana
The observability engineer adds or checks the Loki data source in Grafana and sets the same tenant header on it, so Grafana queries the tenant that receives the gateway logs. Run a stream selector on the gateway labels, then add a filter on a field such as status. Send a known set of test requests, including one that returns an error, and compare the results with what was sent. Done when every test request appears with the expected labels, the status filter returns only the matching lines, and a query under a different tenant returns none of them.
Confirm retention before wider rollout
Before the plugin is enabled on production routes, the Loki operator confirms the retention policy for the tenant. Loki keeps logs indefinitely unless the compactor retention is turned on, and retention can be set globally, per tenant or per stream selector. Agree the retention period, the storage capacity and the owner of the tenant with the security and compliance team, because gateway logs often contain audit evidence. Done when the retention setting for the tenant is written down, the compactor configuration matches it, and the owners have signed off on the rollout scope.
How Loki storage, retention and deployment mode affect gateway logs
Decide how Loki will store and keep gateway logs before the APISIX logger is switched on, because those choices determine cost, query speed and how long evidence stays available. Loki stores compressed log chunks and its index on object storage. Its documentation recommends TSDB as the index store for Loki 2.8 and newer, and it lists Amazon S3, Google Cloud Storage and Azure Blob Storage as recommended production object stores. S3-compatible stores such as MinIO and the local filesystem are supported, but the docs note they are not typically recommended for production. Confirm which object store your infrastructure team will operate, who holds its credentials, and how it is backed up.
Match the deployment mode to expected log volume. The monolithic mode is a single binary that the Loki documentation places at up to about 20 GB of logs per day, so a busy gateway that logs every request will often exceed it. The simple scalable mode, which separates read, write and backend targets, is documented for volumes approaching 1 TB per day, but the Loki documentation states that it is being deprecated and will be removed with the Loki 4.0 release. A new deployment should therefore plan for the microservices mode or a supported successor, and the team should confirm the current guidance in the Loki deployment modes documentation before committing to a layout. Measure the actual request log size from a pilot before choosing.
Retention is disabled by default, so Loki keeps logs indefinitely until you configure it. Retention is applied by the compactor, which must have retention_enabled set, and it can be global, per tenant or per stream selector, with each stream rule at least 24 hours. Before go-live, the team should agree on the retention period for gateway access logs with its security and compliance owners, decide whether any tenant needs a different period, and confirm where the period is recorded as evidence. Ask the Loki operator to test that the compactor actually deletes data on the chosen schedule, because an unconfigured compactor leaves storage growing without limit.
Mapping APISIX log fields to Loki labels and log lines
This table compares common APISIX request fields and sets out whether each one should become a Loki label or stay in the log line. Labels select streams, so keep them to values drawn from a small, stable set, and leave identifiers that vary per request in the log content, where LogQL can filter them. Check each variable name against the loki-logger log format options in your APISIX version, and count your own routes and consumers before you decide.
| Field | Loki label or log line | Reason | Cardinality note |
|---|---|---|---|
route_id | Label, set through the plugin's label configuration | Scopes queries to one API surface that you already manage | Bounded by route count, so usually safe as a label |
service_id | Label only if services are few and stable | Groups routes that share an upstream or policy | Often empty if routes do not use services; confirm your usage first |
status | Label or log line | Supports alerting on error rates with a small set of values | Few distinct values, so a label keeps error-rate queries cheap |
consumer_name | Log line by default | Shows which client called without creating a stream per client | Grows with every API client, and a label here is a common cause of stream explosion |
upstream_addr | Log line | Shows which backend served a request during scaling or failover | Changes as instances are added or replaced, so labels would churn streams |
Request URI, client IP, request_id | Log line only | Needed to investigate a single request and correlate it with other telemetry | Unique per request, so never use as a label |
Tenant (X-Scope-OrgID header) | Fixed per gateway, or per route if the plugin supports it | Keeps each team or environment isolated in Loki | Not a label. Confirm whether the plugin accepts a per-route value or whether you need separate routes or gateway instances |
Choosing a log format for querying gateway traffic
Platform operators ask fleet-level questions: is the error rate rising, which routes are slow, and is one gateway node behaving differently from the others. These queries need a small, stable set of labels and numeric timing fields that can be parsed from the log line. Labels should have a bounded set of values, because Loki creates a separate stream for every unique label combination.
- Status and route identifiers: Consider the response status and the route identifier as labels, since both have a limited set of values that changes only when configuration changes.
- Timing fields in the line: Keep request time and upstream response time in the log body so percentile and average queries can extract them after parsing. Timing values have many distinct values, so they should not be labels.
- Gateway instance: A node or pod identifier helps isolate one unhealthy data plane instance, and its count is limited by fleet size.
- Fields to leave out: Client IP addresses, full URIs containing identifiers, request IDs and user agents belong in the body, not in labels.
After the first rollout, check the active stream count in Loki and confirm which variables your APISIX version emits to the plugin.
Security teams need to answer who was allowed or denied, through which route, and when. That requires the authentication outcome and the identity that produced it, recorded consistently across every route. The log format should capture these facts without copying credentials into a system that many people can search.
- Authentication outcome: Record the status code and the plugin result for each request, so 401 and 403 responses can be separated from upstream failures.
- Consumer identity: Include the consumer name or subject identifier in the line. Use an opaque identifier where the name would reveal personal data.
- Client address: Keep the client IP in the body for investigations. Storing it as a label would create a stream for nearly every client.
- Credentials: Exclude the Authorization header, tokens, API keys and cookies from every field.
- Access and retention: Loki has no built-in authentication layer, so place an authenticating reverse proxy in front of it. Retention is disabled by default, so set it deliberately to match your audit requirements.
Before relying on these logs as evidence, confirm with your security and compliance owners which fields they need and how long they must be kept.
Developers debug one route at a time. They need to see what the gateway received, where it sent the request, what the upstream returned, and how to connect the entry to a trace or application log. The fields that serve this purpose are mostly per-request values, which makes them unsuitable as labels.
- Route and upstream details: Include the route identifier, upstream address and upstream status in the line, so a developer can see whether the gateway or the service produced the error.
- Request correlation: Put the request or trace identifier in the body so it can be matched with application logs and traces. Using it as a label would create a new stream for every request.
- Method and path template: Log the route's path template instead of the raw path where possible, so URLs containing identifiers do not fragment searches.
- Fields to leave out: Request and response bodies, cookies, query strings that may carry secrets and full headers should stay out of the default format.
- Query cost: Filter by stream labels first, then apply line filters or parsed fields within a narrow time range, because body searches read more data than label lookups.
In a staging environment, send a few sample requests through each route and confirm that the resulting log lines contain the fields your developers will search for.
Securing the log path from gateway to Loki
Access logs leave the gateway, cross the network and land in a store that has no login of its own. Each hop needs a decision about encryption, identity, tenant isolation, what gets written and how long it stays.
Encrypt the hop to Loki
Send log traffic over TLS from the APISIX loki-logger to the endpoint in front of Loki, not over plain HTTP between hosts. Confirm the certificate chain that APISIX trusts for the Loki endpoint, decide whether the logger should verify the server certificate strictly, and plan how that certificate is renewed. A failed certificate check usually shows up as silently dropped log batches, so test the path with a known request before relying on it.
Put an authenticating proxy in front of Loki
Loki has no built-in authentication layer, so the reverse proxy in front of it is the only access control for pushes and queries. Choose one such as nginx, HAProxy, OAuth2 Proxy or Pomerium, and restrict which clients can reach the push path. Decide whether the gateway authenticates with a dedicated service credential, and confirm that Grafana users and the gateway cannot reach each other's tokens.
Set and lock the tenant header
When auth_enabled is true, Loki reads the tenant from the X-Scope-OrgID header on every request. Set that header at the proxy or gateway, not in client-supplied input, so a caller cannot write logs into or read logs from another tenant by changing it. Check that the proxy strips any incoming value before adding its own, and document who is allowed to change the tenant mapping.
Keep secrets and bodies out of log lines
Access logs can capture Authorization headers, API key query strings, cookies and request bodies. Build the custom log format so it records method, route, status, latency and consumer identifier, and leaves out token values and payloads. Test with a dummy bearer token and a sample body, then search Loki for both to confirm neither appears. Treat this as a review item for each new route.
Protect the Admin API and stored credentials
The logger's Loki endpoint and any credentials it uses are stored in APISIX plugin configuration, which the Admin API on port 9180 can read and change in traditional mode. Restrict that port to administrator networks, use strong Admin API keys, and limit who can edit logger settings. Confirm where the configuration is stored, who has access to etcd or the configuration file, and how changes are audited.
Match retention to internal policy
Loki keeps logs indefinitely unless retention is enabled, because retention is disabled by default. Decide the retention period from your internal policy, then set retention_enabled and the period globally, per tenant or per stream selector. Each stream rule must cover at least 24 hours. Confirm that the compactor runs, and that backups and object storage lifecycle rules do not keep copies longer than the policy allows.
Why APISIX logs fail to appear in Loki
This table lists the failures that most often stop APISIX request logs from appearing in Loki, with the likely cause and the fix for each. Read it when queries return nothing, pushes are rejected, or data disappears sooner than expected. Start with tenant and authentication checks, since they account for most empty results.
| Symptom | Likely cause | Fix |
|---|---|---|
| Queries return no data | The X-Scope-OrgID header is missing or does not match the tenant the query uses, so logs are stored under a different tenant | Set the tenant in the loki-logger configuration to the same value Grafana sends, then confirm with a test query for that tenant |
| Pushes rejected with 401 or 403 | Loki has no built-in authentication, so the reverse proxy in front of it rejects unauthenticated or wrongly credentialed APISIX requests | Configure the loki-logger plugin with credentials the authenticating proxy accepts, and check the proxy log for the rejected request |
| Queries are slow or time out | High label cardinality, such as request IDs, client IPs or full URLs used as labels, creates a very large number of streams | Keep labels to a small set of stable values like route or service, and move per-request fields into the log line for filtering |
| Log lines are missing during an outage | Batches are dropped when the Loki endpoint is unreachable after the plugin's retries are exhausted | Check the plugin retry and batch settings, monitor APISIX error logs for failed sends, and plan a second log stream if gaps are unacceptable |
| Data disappears earlier than expected | Retention is enabled in the compactor with a shorter period than assumed, or a per-tenant or per-stream rule applies to these logs | Review the compactor retention configuration, including tenant and stream overrides, and confirm the period with the teams that query the data |
| TLS handshake or certificate errors | The plugin cannot verify the Loki or proxy certificate because the CA is missing or the hostname does not match | Add the correct CA certificate to the gateway trust store and confirm the certificate name matches the endpoint; avoid disabling verification |
| Logs arrive with unexpected fields or formats | The log format in the plugin configuration does not match what the team expects to query | Define the log format explicitly, test one request end to end, and agree the field names with the teams writing LogQL queries |
Questions teams ask before connecting APISIX to Loki
What drives the cost of connecting APISIX logs to Loki?
The cost is driven mostly by the Loki deployment you choose, the volume of gateway logs you keep, and how long you keep them. Loki documents three deployment modes: monolithic for a single binary handling up to about 20 GB of logs per day, simple scalable for larger volumes, and microservices for very large clusters. Object storage holds compressed chunks and the index, so storage spend grows with retained volume. The APISIX side is comparatively small: the loki-logger plugin configuration, a reverse proxy in front of Loki, and the Grafana access you need. Ask any quote to separate the one-time build from recurring infrastructure, storage, and support.
How long does an APISIX to Loki logging setup take?
The duration depends on your environment and the scope you agree. A single gateway pushing logs to an existing Loki tenant is a much smaller job than a new Loki cluster on object storage with a multi-tenant layout, a authenticating proxy, and dashboards for several teams. The schedule depends on access approvals, the number of gateway nodes, label design decisions, and how many tenants need separate data. DeployOpen agrees timelines in writing as part of a fixed scope, so ask for the schedule to be tied to named deliverables and to the access you must provide.
What does our team need to provide for this work?
Plan for four kinds of access and decisions before work starts:
- Loki admin access, or an administrator who can enable multi-tenancy, set retention, and manage the storage backend.
- Gateway access to the APISIX Admin API and the routes or global rules where the logger plugin will run.
- A tenant plan that maps each X-Scope-OrgID value to a team, environment, or data class.
- Grafana access with permission to add the Loki data source and build or import dashboards.
If any of these sits with another group, agree on the owner and approval path early.
Can our internal team set this up without a partner?
Yes, an internal team with APISIX and Loki experience can do this, and the configuration itself is compact. The harder parts are the decisions: whether Loki runs with auth_enabled set to true, how tenants are split, how retention is applied through the compactor, and how logs reach Loki without an unauthenticated write path. Loki has no built-in authentication layer and its documentation says to run an authenticating reverse proxy in front of it. Consider a partner if your team lacks Loki operations experience, needs an independent review of log content, or must meet a tight deadline.
Does the AGPLv3 license on Loki affect our decision?
It can, and the answer depends on how you use and distribute Loki. Grafana Labs announced on April 20, 2021 that Loki and its other core projects moved from Apache License 2.0 to AGPLv3. Running Loki internally for your own logs is a different situation from offering modified Loki functionality to outside users over a network, which AGPLv3 addresses directly. Ask your legal or licensing team to review the Grafana Labs licensing terms against your intended use, and confirm whether a commercial Grafana Enterprise Logs option fits your needs. APISIX itself is licensed under Apache-2.0, so the two licenses apply to different components.
Will gateway logs in Loki expose tokens or personal data?
They can if the log format is not chosen carefully. The access log format decides which request headers, query strings, and client identifiers reach Loki, so review the fields before enabling the logger on production routes. Remove or avoid logging authorization headers, cookies, and sensitive query parameters, and restrict who can read each tenant in Grafana. Because Loki has no built-in authentication, the reverse proxy in front of it must enforce access, and tenant headers must be set so one team cannot read another's gateway traffic. Confirm the retention period with your data owners, since retention is disabled by default in Loki.
How should we design labels and log format for querying gateway traffic?
Keep stream labels few and stable, such as the gateway instance, environment, and route group, and put variable values like client IDs, paths, and status codes into the log line where they can be filtered with LogQL. High-cardinality labels make Loki slower and more expensive to index, so decide the label set before rollout and test queries against a sample of real traffic. Choose a JSON log format if dashboards need field-level filters, and document the field names so every team builds queries the same way.
What ongoing work follows the initial connection?
After go-live, the recurring work covers storage growth, retention checks, tenant access reviews, upgrades to APISIX and Loki, and dashboard maintenance. Loki's simple scalable mode is documented as deprecated and due for removal with the Loki 4.0 release, so plan a deployment mode migration if you use it. Track Loki release notes for version changes, and schedule a periodic review of log fields to catch new sensitive data. DeployOpen offers ongoing support and upgrades as retainers, and those terms are agreed separately from the initial scope.
How DeployOpen scopes and delivers the APISIX to Loki integration
DeployOpen plans, builds, documents and supports the path from APISIX request logs to Grafana Loki inside infrastructure you control. Each engagement starts from a written scope that names the routes, log fields, tenant design and retention rules before any configuration changes are made.
How we scope
- We list the APISIX routes and consumers whose request logs will be shipped to Loki.
- We agree the log fields, tenant header values and label strategy in writing before build.
- We confirm your Loki deployment mode, object store and retention requirements with your team.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
How we build
- We configure the loki-logger plugin on the agreed routes and test batching behavior under load.
- We set the X-Scope-OrgID tenant mapping when your Loki instance runs with multi-tenancy enabled.
- We place an authenticating reverse proxy in front of Loki, since Loki has no built-in authentication layer.
- We test that sensitive headers and tokens are excluded from logged request fields.
How we hand over
- We document the plugin settings, tenant mapping, log format and label choices for your engineers.
- We provide example LogQL queries that your team can use to investigate gateway traffic.
- We walk your platform and security staff through failure handling, including what happens when Loki is unreachable.
- We hand over configuration as code so changes can be reviewed and rolled back.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- We can review APISIX and Loki release notes when you plan an upgrade of either component.
- We can revisit retention settings and tenant design as log volume and team needs change.
- Support scope and response arrangements are agreed in writing with your retainer.
