What Loki upgrade and support work involves
Loki upgrade and support work moves a running self-hosted stack to a newer version without losing logs or query behavior, then keeps it operating. Four kinds of trigger start this work: a new Loki release your team needs, a deprecation with a removal date, an end-of-life component, or a change to the index store. Three of these are current. Simple scalable deployment mode is being deprecated and will be removed with Loki 4.0. Promtail reached end of life on March 2, 2026, so its log shipping needs a move to Grafana Alloy. Index stores other than TSDB are deprecated for Loki 2.8 and newer.
Planning starts with the gap between your current version and the target. Read the release notes for every version in between, list configuration keys that changed, and review your schema_config and storage settings with the change in mind. Identify which component upgrades first, such as the compactor and ingesters, and check that your Helm chart values or binary versions match the target. Write the rollback path before the change, and confirm with the vendor documentation which steps are reversible once new index data is written.
Testing after the upgrade is where most gaps show up. Run a fixed set of LogQL queries against known time ranges and compare results with a pre-upgrade baseline. Retention is disabled by default, so confirm that the compactor applies the global, per-tenant and per-stream rules you intended, using a test tenant before production. Check that X-Scope-OrgID routing still isolates tenants and that the authenticating reverse proxy in front of Loki still blocks unauthenticated requests.
Support after the upgrade covers monitoring compactor and ingester health, tracking new releases and patches, and scheduling maintenance windows. For a team that already runs Loki with spare capacity and a tested rollback, doing routine minor upgrades in-house is usually sensible. A partner adds the most value for a deployment mode change, an index store move, or a deadline such as the Loki 4.0 removal. Ask any partner for a written, fixed scope, and ask whether ongoing support is a separate retainer.
What the work includes
The work runs in sequence, from assessment through ongoing support. Each package ends with a named deliverable that your team keeps.
Upgrade assessment
Compare your running Loki version with the target release and its notes. The GitHub releases page lists v3.7.8, published September 17, 2026, as the latest at the time of writing, so confirm the target with your team. Note deployment mode and index store, and flag any simple scalable mode use, which the Loki documentation says will be removed in Loki 4.0. Deliverable: an assessment and draft change plan.
Version and config review
Review the Loki configuration against the target release, field by field. Look at schema config, the TSDB index settings, compactor and retention blocks, auth_enabled and the storage section, and any deprecated options the release notes name. Produce a configuration diff that shows every changed line and the reason for it, so your team can review changes before they reach a cluster.
Staging rehearsal
Run the upgrade on a staging copy built from production settings, including the same deployment mode, object storage backend and tenant layout. Restart components in the order your plan sets and time each step, recording errors and anything that differs from the assessment. The output is a test report and a rollback runbook that has been exercised in staging. Repeat the rehearsal after any change to the plan, because one clean run does not cover a changed configuration.
Production change window
Execute the approved change plan in an agreed window, with the rollback runbook open and one named owner for each step. Before the window, confirm that log shippers can buffer through the planned restarts. During the change, watch write and query paths and object storage errors. Set the go and rollback decision points before the window opens. You receive a change record of what ran and when.
Post-upgrade query and retention checks
Test queries against known log ranges and tenants, comparing results and latency with the pre-upgrade baseline. Confirm retention still deletes what it should: the compactor must run, retention_enabled must match the plan, and each stream rule must keep its period. Send requests with each X-Scope-OrgID to confirm tenants see only their own logs. You receive a test report covering queries and retention.
Ongoing support
Ongoing support covers release tracking, patch planning for each new Loki version, and help with Helm chart updates and log shipper changes when they affect the stack. It is available as a retainer, and its scope, response expectations and exclusions are agreed in writing. Each change ends with an updated operations note, so your on-call team has the current configuration and runbooks.
What drives cost and duration
This table lists the factors that change how much work a Loki upgrade and ongoing support take, why each one matters, and how to keep it down before you request a quote.
| Driver | Why it matters | How to keep it down |
|---|---|---|
| Deployment mode | Monolithic, simple scalable and microservices modes have different components to upgrade and test. Simple scalable mode is being deprecated and will be removed with Loki 4.0, which adds a mode migration to the upgrade. | Confirm your current mode and the target version before scoping. Plan any move off simple scalable mode as a separate, scheduled change. |
| Number of tenants and retention rules | Each tenant and each retention rule is a query and deletion path to verify after upgrade. Per-tenant and per-stream rules multiply the test cases. | Keep retention rules in version-controlled configuration and limit the number of tenants to what the business needs. |
| Object store and backend type | Amazon S3, Google Cloud Storage, Azure Blob Storage and S3-compatible or filesystem stores each have different permissions, lifecycle behavior and test steps. Non-recommended production stores add risk to every change. | Standardize on one supported object store per environment. Record the bucket policies and lifecycle settings so the upgrade test can compare them. |
| Helm or non-Helm install | A Helm-based install moves chart values and templates with each release. A non-Helm install needs manual configuration changes and a custom rollout procedure. | Pin chart and application versions together, keep values files in Git, and ask for the chart diff as part of the quote. |
| Log shippers in use | Grafana Alloy, Fluent Bit, Fluentd, the Docker logging driver and Promtail each have their own configuration and compatibility checks. Promtail reached end of life on March 2, 2026, so any remaining Promtail agents need migration work. | Inventory every shipper and its version before the upgrade. Move Promtail agents to Grafana Alloy as a separate workstream. |
| Schema or index configuration changes | Moving to a new schema or index version can require a dated schema configuration entry and careful sequencing so existing data stays queryable. Older index stores are deprecated. | Confirm whether the target version needs a schema change. Ask for a written plan that covers the cutover date and rollback. |
| Grafana and auth proxy dependencies | Grafana data sources, dashboards and the authenticating reverse proxy in front of Loki each need regression checks. Loki has no built-in authentication, so proxy changes affect every query path. | List Grafana versions, data source settings and proxy rules. Ask the provider to test queries through the same path your users take. |
| Number of environments | Each environment, such as development, test and production, needs its own upgrade run and verification. Differences between environments often cause the most rework. | Keep environments as close in configuration as possible and upgrade lower environments first. Ask for the environment count and sequence in the quote. |
Upgrade and support phases
The sequence moves a Loki stack from its current version to a target version with a documented inventory, a tested rehearsal and a verified production cutover, then hands it to ongoing support.
Inventory the current deployment
Record the running Loki version, deployment mode, Helm chart version and values, object storage layout, index store and schema configuration, retention settings, and every log shipper and Grafana data source that sends or reads logs. Done when the inventory is reviewed with your team and every shipper and query path is named.
Review the target version and breaking changes
Select the target Loki release and read its release notes, configuration changes and deprecations against your inventory. Confirm whether your deployment mode is affected by the documented deprecation of simple scalable mode before Loki 4.0. Output is a written change list with each item marked as applies, does not apply, or needs a test.
Design schema and index changes
Decide whether any index or schema change is needed, including moving to TSDB, which is the recommended index store for Loki 2.8 and newer, and plan the schema period cutover so existing data remains queryable. Confirm with the Loki storage documentation and your own data. Output is a signed-off schema design with a rollback note for the index change.
Rehearse the upgrade in staging
Apply the Helm chart and configuration changes to a staging copy that uses representative data, then run query tests across every tenant, retention tests against the compactor and a check that shippers still write. Output is a rehearsal record with pass or fail results per test and a list of fixes applied before production.
Cut over production with rollback criteria
Agree rollback criteria in writing before the change, such as ingestion errors, query failures or retention jobs that stop completing. Run the production change in the agreed window, watch ingestion and query health, and roll back if a criterion is met. Done when the change record shows the new version serving writes and reads.
Verify and hand over
Confirm after the change that logs arrive from each shipper, queries return expected results for each tenant, retention is still applied, and dashboards and alerts work. Deliver the updated inventory, runbook, rollback notes and test records to your team. Done when your owners accept the handover in writing.
Provide ongoing support
Support covers monitoring of the stack, planned patch and version upgrades, chart updates, retention and shipper reviews, and incident response as agreed in the support scope. Ongoing support is available as a retainer, and the scope, response expectations and reporting are confirmed in writing before work starts.
What usually goes wrong
Most Loki upgrade problems come from a small set of recurring causes: schema changes, retention settings, query behavior, configuration drift, access paths, outdated shippers and missing rollback plans. Each card names the check that prevents it.
Schema and index changes out of order
Loki reads data according to the schema configuration recorded for each date range, so adding a new schema period or moving to TSDB must be planned as an ordered change. Apply the new period with a future start date, roll out the new Loki version everywhere first, and confirm that queries across the boundary return data from both index types before relying on it.
Retention that silently does not run
Retention is disabled by default, so logs are kept indefinitely until retention_enabled is set, and the compactor is the component that applies it. After every upgrade, confirm the compactor is running, retention is enabled in the expected configuration, and deletion rules match the intended global, tenant or stream scope, with each stream rule at least 24 hours.
Queries returning different results
Upgrades can change how queries behave, so the same LogQL may return different counts or ordering after the move. Keep a fixed set of saved queries and expected results from before the upgrade, rerun them on a staging copy of the new version, and record differences for review before production cutover.
Helm values drifting from the running config
When Helm values are edited directly on a cluster, the running configuration no longer matches the chart values kept in version control, and the next upgrade can overwrite or reintroduce old settings. Render the chart with the new version, diff it against the live release, and keep one source of truth for every value that controls storage, retention and tenancy.
Authentication path breaking after endpoint changes
Loki has no built-in authentication, so the authenticating reverse proxy in front of it and the Grafana data source both depend on specific endpoints and headers. Changes to routes, the X-Scope-OrgID header handling or the data source URL can block users or expose tenants, so test a login, a tenant-scoped query and a denied cross-tenant request after each change.
Shippers still on Promtail
Promtail reached end of life on March 2, 2026, with no further updates or commercial support, so agents left on it will not receive fixes during a Loki upgrade. Inventory every host and cluster sending logs, migrate them to Grafana Alloy, and confirm that labels and timestamps arrive as before so existing dashboards and alerts keep matching.
No tested rollback path
A Loki upgrade that changes storage formats or index layout may not be reversible by simply redeploying the previous chart, so the rollback path must be defined before the change. Write down which components and data changes are one-way, back up the configuration and index state, and rehearse a restore on a non-production copy at least once.
How the work changes by starting point
Simple scalable mode is being deprecated, and the Loki documentation states it will be removed with the Loki 4.0 release. An install in this mode must move to monolithic or microservices mode before that release. Monolithic mode is documented for up to about 20 GB of logs per day, so the target mode depends on current volume.
- Daily volume: measure ingest over a representative period and compare it with the documented mode guidance.
- Target mode: choose monolithic or microservices before writing the migration plan.
- Read and write path: test ingestion and queries after the read, write and backend targets move.
- Retention state: confirm whether retention is enabled and which streams it covers.
Answer first: What is your peak daily ingest? Who owns the Loki 4.0 timeline?
Loki's documentation recommends TSDB as the index store for Loki 2.8 and newer. DynamoDB, Bigtable and Cassandra are deprecated, so an install on any of them needs a planned move to TSDB. The schema configuration and object store layout should be mapped before any change.
- Schema configuration: record each schema period and the date it took effect.
- Historical data: decide whether old data stays on the previous index or is reindexed, and test queries across that boundary.
- Store access: confirm credentials, IAM policies and network paths to the old store remain valid until it is retired.
- Retention: confirm compaction and retention still apply to data written under both index formats.
Answer first: Which index store and Loki version run today? How long must data written under the old index stay queryable?
A Helm-managed cluster makes the chart version and values files the upgrade unit, so the chart update, image tag and configuration changes must be reviewed together. A binary or systemd install upgrades the binary and unit files on each host. Rollback depends on keeping the previous binary available.
- Chart differences: compare default values between chart versions before upgrading.
- Values files: store overrides in version control and test them in a staging cluster.
- Rollout order: plan component restarts and check ingestion lag after each step.
- Host configuration: for systemd, track config files, unit files and binary paths on each host.
Answer first: Who owns the chart and the values repository? Is there a staging environment with comparable log volume?
In an air-gapped environment, Loki binaries, container images and chart packages must cross a controlled transfer process, be mirrored into internal registries and be validated before any node uses them. Nothing is pulled from public sources during the upgrade window, so every dependency must be listed in advance.
- Artifact inventory: list every binary, image, chart and dependency for the target version.
- Verification: check each artifact against the checksums or signatures the publisher provides before import.
- Registry completeness: confirm the internal mirror holds every image tag the chart references.
- Object storage: confirm the internal store is suitable for production. The Loki docs note that S3-compatible stores and the local filesystem are supported but not typically recommended for production.
Answer first: Who approves transfers across the boundary? How many approval steps does a new artifact set need?
Grafana Enterprise Logs is built on Loki and adds token authorization, label-based access control and cross-tenant queries. Its long-term support line is version 3.6, which reaches end of life on February 1, 2029. An install in front of GEL needs its own upgrade target and license checks, separate from the open-source path.
- License and entitlement: confirm the agreement covers each version you plan to run.
- Access control mapping: list which controls depend on GEL features and which depend on a reverse proxy.
- Version path: compare the 3.6 long-term support line with your planned upgrade cadence.
- Client configuration: check that shippers, queries and dashboards still work with GEL authorization and tenant settings.
Answer first: Are you running AGPLv3 Loki or GEL today? Which access controls must still work after the upgrade?
Recurring costs after the upgrade
This table lists the recurring cost items that follow a Loki upgrade, what each one covers, who usually pays for it and what to confirm before agreeing to a support quote.
| Cost item | What it covers | Who pays and what to confirm |
|---|---|---|
| Support retainer | Ongoing vendor-independent support for the Loki version in production, including patch review, incident response hours and help with query or configuration issues | Usually the organization, paid to a partner on a retainer. Confirm response hours, which Loki versions are covered, whether upgrades are included and how extra hours are billed |
| Scheduled upgrade windows | Recurring planned windows to move to new Loki releases, covering schema and index checks, Helm chart changes, rollback steps and post-upgrade query tests | Internal staff time plus any partner hours. Confirm how many windows per year the quote includes and who signs off on the change calendar |
| Staging environment upkeep | A non-production copy of the stack that mirrors production object storage layout, Helm values and tenant setup, used to rehearse each upgrade | Organization, for compute and storage. Confirm whether the staging copy uses production-sized data and how often it is refreshed |
| Object storage and compute carried over | Chunks, index and compactor storage in S3, Google Cloud Storage, Azure Blob Storage or another approved store, plus the nodes running the chosen deployment mode | Organization, billed by the infrastructure provider or the data center team. Cost rises with daily log volume and retention period; confirm retention settings before sizing |
| Internal on-call and skills | Staff who receive alerts from Loki, read compactor and ingester health, and rotate through upgrade and retention tasks | Organization, as internal salary time. Confirm how many people are trained on Loki operations and whether the on-call rota covers the logging stack |
| Edition decision: AGPLv3 Loki or Grafana Enterprise Logs | AGPLv3 Loki carries no license fee but leaves token authorization and label-based access control to your own design. Grafana Enterprise Logs, a commercial product from Grafana Labs, adds those features and cross-tenant queries under a license | Organization, through a Grafana Labs license for Enterprise Logs only. Confirm the license cost, term and support level directly with Grafana Labs before the support quote is finalized |
| Authentication proxy and Grafana access | The authenticating reverse proxy Loki requires in front of it, plus the Grafana instance and identity tools that query logs, each needing patching and certificates | Organization, through internal platform time or a support scope that includes them. Confirm which components the quote covers and who renews certificates |
What the client needs to provide
Before we can scope a Loki upgrade, we need read access to the running configuration and the Helm values that produced it, along with the current chart version and any overrides. Without the actual values, we would be estimating the upgrade from memory of a default install. We also need the object store and backend in use (for example, which S3-compatible or cloud bucket holds chunks and the TSDB index), and a current inventory of tenants and their retention rules, including any per-stream selectors applied by the compactor.
A staging environment that matches production closely enough to test queries is the input that most affects the plan. Confirm its Loki version, index schema, object store type and rough data volume, and whether it sits behind the same reverse proxy and Grafana data source configuration. Name an owner for each of Loki, the reverse proxy and Grafana, because upgrade steps touch all three and someone must approve changes to each.
Decide the target Loki version and whether the upgrade must come before any change tied to Loki 4.0. Simple scalable mode is documented as being removed in 4.0, so if you run it, confirm the migration sequence with your own team first. Agree on change windows and one named approver who can accept a maintenance slot.
Rollback criteria need sign-off from more than the platform team. Ask your security or compliance lead to approve any change that affects tenant isolation or log retention, and ask the owner of log-driven incident response to confirm which query failures would block operations. Record the agreed criteria before scoping begins so the test plan can verify them.
Questions buyers ask about Loki upgrades and support
How much does a Loki upgrade cost, and what drives the price?
The cost depends on the written scope, and DeployOpen does not publish a price list. The main drivers are the number of environments, the deployment mode (monolithic, simple scalable or microservices), the number of tenants and dashboards to check, and whether the work includes Helm chart changes, a Promtail to Grafana Alloy move or an index migration to TSDB. Ask the quote to separate one-time upgrade work from any recurring retainer for support, upgrades and continuous testing.
How long does a Loki upgrade take from planning to go-live?
The timeline is set by the version gap, the deployment mode, the number of environments, how much test coverage already exists and the change windows your organization allows. Your target release matters too: the Loki GitHub releases page lists v3.7.8, published September 17, 2026, but the right target depends on your current version and on your own compatibility testing. If you run simple scalable mode, note that the Loki documentation says it is being deprecated and will be removed with Loki 4.0, which can add planning work. The written scope will state the phases and dates.
Can our own team run a Loki upgrade without outside help?
A small monolithic install can often be upgraded by a team with strong Helm and Kubernetes experience, but the work still needs a named owner, a staging environment that mirrors production, object storage access, a tested rollback path and people who own the Grafana dashboards and alerts. Engineers should understand the compactor, the schema configuration and how the index store is laid out before they touch production. Teams still on Promtail should plan the move to Grafana Alloy now, since Promtail reached end of life on March 2, 2026.
Does a Loki upgrade include testing retention and queries afterward?
Retention and query checks can be included when they are written into the scope, and they should be. Retention is disabled by default and is applied by the compactor, so confirm that retention_enabled and the global, per-tenant and per-stream rules still behave as intended after the upgrade. Query testing covers the LogQL used by dashboards and alerts, ingest counts, and tenant isolation through the X-Scope-OrgID header. Ask the provider to list which queries, tenants and retention rules are in the test plan.
What does support cover for a Loki stack after go-live?
Ongoing support is available as a retainer, and its contents should be defined in writing before work starts. A typical scope covers version upgrades, Helm chart updates, patching, retention and backup checks, and testing after changes, with DeployOpen working inside infrastructure you control. The cost depends on the number of environments, the deployment mode, how often you want upgrades reviewed and how much query and retention testing you want each cycle. Confirm response expectations and any escalation path in the written agreement rather than assuming them.
Can DeployOpen take over an existing Loki stack that someone else built?
DeployOpen can assess an existing Loki stack and take over its operation in infrastructure you control, but the first step is an inventory. That inventory should record the Loki version, the deployment mode, the object store and index type, the log shippers in use, retention settings, how authentication sits in front of the endpoints and which Grafana dashboards depend on the stack. Gaps found in that review, such as an unsupported index store or an unauthenticated endpoint, are scoped and priced as their own work. Takeover terms, handover documents and any retainer are set out in the written scope.
How DeployOpen delivers Loki upgrades and support
DeployOpen plans, implements, tests and documents Loki upgrades inside your own infrastructure, starting from a written scope agreed before work begins. Ongoing support is available as a retainer for teams that want the stack kept current after handover.
How we scope
- Inventory your current Loki version, deployment mode, index store, and object storage configuration before any change is planned.
- Agree in writing which upgrade steps, schema or index changes, and Helm chart updates are included in the engagement.
- Confirm the retention rules, tenants, log shippers and Grafana queries that must keep working after the upgrade.
How we build
- Stage the upgrade in a non-production environment that mirrors your production topology and storage backend.
- Plan any move to the TSDB index and check the schema configuration before the new version rolls out.
- Update Helm chart values and version pins through reviewed changes so each step can be traced and reversed.
How we hand over
- Test representative LogQL queries, ingestion from each log shipper, and Grafana dashboards against the upgraded cluster.
- Verify that retention rules still delete data as intended, since the compactor controls deletion after the upgrade.
- Deliver runbooks, version records, rollback notes and configuration documentation your team can operate without us.
How we support
- Ongoing support can run as a retainer covering upgrade planning, patching and fixes for your Loki stack.
- Review each new Loki release against your configuration, including changes to deployment modes and deprecations.
- Agree retainer coverage and upgrade frequency in writing before support begins.
