Self-hosted LGTM stack: Loki, Grafana, Tempo and Mimir

How the self-hosted LGTM stack fits together, which links Grafana builds between logs, traces and metrics, what the AGPL licensing means, and what each component costs to run.

Book a callTalk to us
On this page

Self-hosted LGTM stack: what it is and when a team chooses it

A platform team that needs logs, traces and metrics in one place, under its own control, often ends up with the LGTM stack: Grafana for dashboards and exploration, Loki for logs, Tempo for traces and Mimir for metrics. Each component is a separate open-source project that you run in your own cloud accounts, data centers or air-gapped network. Metrics usually reach Mimir from Prometheus or a Prometheus-compatible collector through the Prometheus remote write API, and Grafana reads all four signals through its data sources. The connections between signals, such as jumping from a trace span to its logs, use links that Grafana's Tempo data source builds to Loki, Prometheus and Pyroscope. This page does not describe any connector beyond those documented data source links and remote write.

The stack fits teams that want one query and dashboard layer across signals, that must keep telemetry inside a defined boundary, and that can staff the operation of several stateful services. It is a weaker fit where a single managed service already covers the need, or where the team has no one to own storage, upgrades and capacity. Before committing, confirm three things with your own data: expected log volume per day, the number of active metric series per tenant, and how many teams will share the platform, because these drive which Loki deployment mode and which Mimir architecture you need. Also confirm how users will sign in, since Grafana's SAML and team sync features depend on the edition you choose.

Licensing is the first decision that affects long-term cost. Grafana, Loki and Tempo moved from Apache License 2.0 to AGPLv3 on April 20, 2021, and Mimir is distributed under AGPL-3.0-only, so your legal team should review how your organization uses and modifies these components before rollout. Plugins, SDKs and agent software carry their own licenses, and Grafana Enterprise is a separate commercial option. Costs then come from infrastructure, object storage, the people who run the platform and any support contract, rather than from license fees for the core projects. Ask any implementation partner to price each component and each environment separately, and to state which costs recur monthly and which are one-time build effort.

What each component does and where it stores data

The LGTM stack splits observability into a query layer and one storage backend per signal. Grafana reads from each backend, so the storage choice and deployment mode for each component decide what you operate, back up and pay for.

Grafana as the query layer

Grafana is the dashboard, query and alerting front end. It stores its own configuration in a database, and the database choice is the first production decision: SQLite works for small installs, while production and high availability need MySQL 8.0 or later or PostgreSQL 12 or later. Grafana holds no telemetry itself, so check which data sources each team can reach and how Loki, Tempo and Mimir are authenticated before you publish dashboards.

Loki for log storage

Loki stores compressed log chunks and its index in object storage, which is where most of its storage cost sits. Choose a deployment mode by daily log volume: monolithic runs as a single binary, while simple scalable mode is being deprecated and removed with Loki 4.0, so plan microservices or a supported path for growth. Retention is disabled by default, so logs are kept indefinitely until you enable it in the compactor. Loki has no built-in authentication, so an authenticating reverse proxy must sit in front of it.

Tempo for traces

Tempo is the trace backend in the stack and is the part of this design that links signals together. Tempo spans link to logs in Loki, to metrics in Prometheus and to profiles in Pyroscope, so trace-to-log and trace-to-metric navigation depends on the data source settings you define in Grafana. Confirm the Tempo storage backend, retention and deployment layout with the Tempo documentation for your release, because this topic's supplied facts do not cover them. Plan the tenant and access model at the same time as the log and metric tenants.

Mimir for long-term Prometheus metrics

Mimir receives Prometheus remote writes through its distributor and answers PromQL queries through the query frontend, so Prometheus remains the scraper. Each tenant's series are stored in their own TSDB in object storage such as Amazon S3, Google Cloud Storage or Azure Storage. The classic architecture uses stateful ingesters with local write-ahead logs, while Mimir 3.0 and later prefer Kafka as the ingest pipeline. Mimir requires a tenant header on every request and depends on an external proxy for authentication, so decide on the ingest architecture before sizing the cluster.

Which links Grafana builds between logs, traces and metrics

The Tempo data source is where the cross-signal links in this stack are configured. From a trace span, Grafana can jump to the matching Loki logs, to related metrics in a Prometheus-compatible store and to profiles in Pyroscope. The Prometheus data source is the piece that makes the metrics side work with Grafana Mimir, because it speaks the Prometheus querying API. Point a Prometheus-type data source at your Mimir query path, and Grafana treats it the same way it treats a single Prometheus server. Confirm the exact URL and tenant header settings for your Mimir deployment in the Grafana Prometheus data source documentation before you wire it up, since multi-tenant Mimir needs the tenant ID on every request.

These links are only as good as the labels behind them. Trace-to-log jumps depend on the trace identifier and the service and namespace labels appearing in Loki streams, and trace-to-metric jumps depend on matching service labels in Mimir. If Alloy or your OpenTelemetry Collector attaches different service names to logs, traces and metrics, the links will open empty results and nobody will notice until an incident. Decide the label names once, such as service.name and the resource attributes your teams already use, and check them in a test environment with one instrumented service before rolling out to every team.

The service graph needs more than the Tempo data source. It draws its edges from span metrics, so you need a linked Prometheus-compatible metrics source that receives those metrics, and that source must be the one the Tempo data source references. Plan for the metrics pipeline that generates span metrics and confirm it reaches Mimir under the tenant you expect. Without that link, the trace view loads but the service graph stays blank. Treat the service graph as a verification step during the build, not a feature you assume is working after install.

Choosing a Loki and Mimir deployment mode

Monolithic mode runs Loki as a single binary that handles ingestion, querying and storage coordination in one process. Loki's documentation places it at up to about 20 GB of logs per day, which makes it the simplest option for a pilot, a single team or a small set of applications. It has the fewest moving parts to install, upgrade and back up, so it suits teams building their first stack.

  • Operational footprint: One process to monitor, restart and size, with no internal service mesh to manage.
  • Storage: Chunks and the index still go to object storage, so plan the bucket and its access policy even at this scale.
  • Growth limit: Once daily volume approaches the documented ceiling, the move to a distributed mode is a redesign, not a config change.
  • Failure domain: A single process means one restart or crash affects all ingestion and queries.
  • Retention: Retention is disabled by default, so set it deliberately in the compactor even on a small instance.

Before committing, confirm the daily volume you will actually ingest, including peaks, against the Loki deployment modes guidance.

Configuration sequence from storage to Grafana data sources

This sequence takes a self-hosted LGTM stack from empty object storage to a Grafana instance where logs, traces and metrics can be queried and linked together. Each step ends with a check you can run before moving on.

  1. Provision object storage and buckets

    Your platform or storage team creates one bucket or prefix each for Loki chunks and index, Tempo traces and Mimir blocks, using the object store your organization already operates, such as Amazon S3, Google Cloud Storage, Azure Blob Storage or an S3-compatible store. Loki's docs list S3, Google Cloud Storage and Azure Blob Storage as its recommended production stores, and note that MinIO and local filesystems are supported but not typically recommended for production. Give each service its own credentials with access limited to its bucket. The step is done when a test write and read succeed from each service's host with those credentials, and the bucket versioning and lifecycle settings have been reviewed against your retention plan.

  2. Deploy Loki with auth enabled and TSDB index

    Deploy Loki in the deployment mode that matches your expected log volume, starting from the monolithic mode for small volumes as described in the Loki deployment docs. Set auth_enabled: true so every request must carry an X-Scope-OrgID tenant header, configure the schema with the TSDB index (the recommended index store for Loki 2.8 and newer), and point storage at the bucket from the previous step. Loki has no built-in authentication, so place an authenticating reverse proxy such as nginx, HAProxy, OAuth2 Proxy or Pomerium in front of it. Check the result by pushing a test log line with a tenant header through the proxy, then querying it back. Confirm that a request without the header is rejected. Retention stays off until you enable it in the compactor, so set that as a deliberate step, not a default.

  3. Deploy Mimir and confirm the tenant header

    Install Grafana Mimir in monolithic mode for a first environment, or use the mimir-distributed Helm chart if you run Kubernetes, since Grafana publishes it to install, configure and upgrade Mimir there. Mimir requires a tenant ID header on every HTTP request and leaves authentication to an external reverse proxy, so place the same kind of authenticating proxy used for Loki in front of it. Point the storage backend at the Mimir bucket. Confirm the install by sending a test remote write with a tenant header and running a PromQL query for that series through the query path. A request without the header should fail. Record the tenant IDs you plan to use, because the same identifiers will be needed in Grafana data source settings and in Alloy or Prometheus configuration.

  4. Deploy Tempo for trace storage

    Deploy Tempo with its trace storage pointed at a separate bucket or prefix from Loki and Mimir, and confirm its receiver accepts OTLP from your applications or collector. Tempo is the trace backend in this stack, so the Tempo data source in Grafana will query it directly. Verify the deployment by sending a test trace through the receiver and finding it by trace ID in Tempo's API or from a test query in Grafana after the next steps are complete. Also decide now whether trace search will rely on TraceQL, since that affects how much you invest in span attributes later. Record Tempo's query endpoint and the tenant behavior of your chosen configuration so the data source step uses the correct URL and headers.

  5. Configure Alloy or Prometheus remote write

    Configure one collector to send each signal to its backend: Grafana Alloy (a vendor-neutral OpenTelemetry Collector distribution that Grafana supports for Loki) or Prometheus with remote write. Prometheus writes to Mimir through the remote write API to the distributor, and PromQL queries are served by the query frontend, so set the remote write URL to the distributor and include the tenant header in the remote write configuration. Send logs to Loki's push endpoint with its tenant header, and traces to Tempo's OTLP receiver. Prometheus local storage is not clustered or replicated and has a default 15-day retention, so do not treat it as the long-term store. The check is a rate or count query in Mimir returning the series your collector is sending, plus a Loki query showing recent log lines from the same host.

  6. Add Loki, Tempo and Prometheus-compatible data sources with provisioning

    Define the Loki, Tempo and Prometheus-compatible data sources in Grafana's provisioning YAML so the configuration is versioned and reproducible rather than set by hand in the UI. The Prometheus data source works with any system that implements the Prometheus querying API, including Mimir, so the Mimir entry uses the Prometheus type with the tenant header configured under custom HTTP headers. Set each data source's URL, tenant headers and default flags, and keep credentials in Grafana's secure fields or environment variables, not in the file. Grafana loads these files at startup, so restart after changes and confirm all three data sources report healthy under Connections. Then open Explore and run one query against each backend to confirm the data returns with the correct tenant.

  7. Wire trace-to-log and trace-to-metric links

    Enable the links in the Tempo data source settings so a span can open related logs in Loki, related metrics in the Prometheus-compatible data source, and linked profiles in Pyroscope if you run it. Set the Loki link to filter on the service name and trace ID fields that your log pipeline attaches, and set the metrics link to a query your Mimir data source can answer. Because the links depend on matching labels, use the same service name in traces, logs and metrics. Confirm the wiring by opening a trace in Explore, selecting a span, and following the logs link to see lines for that trace ID. Then follow the metrics link to confirm that a PromQL query returns data for the same service. If either link returns nothing, check the label names first.

What the AGPLv3 licensing means for an internal stack

On April 20, 2021, Grafana Labs moved Grafana, Loki and Tempo from the Apache License 2.0 to the GNU Affero General Public License v3 (AGPLv3), and Mimir is distributed under AGPL-3.0-only. Running these components inside your own network for your own staff is the common case, but the license still shapes what you may change, how you share modified code and which obligations apply if users interact with a modified service over a network. The Grafana Labs licensing page is the starting point for the vendor's own statement of terms, and it should be read alongside the license text itself.

Not everything in a Grafana stack carries the same license. Plugins, SDKs and agent software are published under other terms, so each plugin you add for a data source or panel, and each collector you deploy, needs its own license review before it goes into the build. Grafana Enterprise is offered as a non-AGPL commercial option, which changes the obligation picture if your legal team concludes that AGPL terms are not acceptable for a modified or customer-facing deployment. Ask Grafana Labs which terms apply to the specific edition, plugin and version you plan to run, and record the answers in your architecture decision log.

Have your counsel review the obligations that apply to your own distribution and modifications before you build custom changes, custom plugins or a service that external parties use. Counsel should confirm whether internal use triggers any source offer duties, whether patches to Loki or Mimir need to be published, and how contractors and implementation partners handle code they write against these components. This review is a one-time legal task that should be repeated when you upgrade to a new major version, add a plugin or change how the stack is exposed to users outside your organization.

What each component costs to run and what drives it

This table compares, for each part of the stack, the one-time work, the recurring cost and the factors that move the bill. Read it as a checklist for a quote: each row names what the partner or your team should price separately. The main takeaway is that storage, ingest volume and series count drive most recurring cost, while database, high availability and pipeline design drive most one-time work.

ComponentOne-time workRecurring costCost drivers
GrafanaDatabase setup (MySQL or PostgreSQL), HA node build, load balancer, identity integration, provisioning as codeCompute for two or more active nodes, managed or self-run database, load balancer, patching and upgradesNumber of concurrent users, HA design, database size, plugins and alerting load. Ask the quote to separate database and load balancer costs.
LokiDeployment mode choice, object storage bucket and access policy, retention rules, authenticating reverse proxy, log shipper setupObject storage capacity and requests, compute for write, read and compactor roles, reverse proxy operationDaily log volume, retention period, number of streams and labels, deployment mode. Confirm which storage tier and request pricing the quote assumes.
TempoTrace storage backend choice, retention, span-to-log and span-to-metric links, collector pipeline for tracesObject storage for trace blocks, compute for ingest and query, network traffic from collectorsTrace volume and sampling rate, retention length, query load. Ask how sampling is priced into the scope.
MimirArchitecture choice (classic or Kafka ingest storage), object storage, tenant model, reverse proxy for authentication, Helm chart configurationObject storage, compute for distributors, ingesters or Kafka brokers, query components, operating the Kafka pipeline if usedActive series count, sample rate, number of tenants, retention, ingest storage design. Confirm whether Kafka is run by you, by a managed service or by the partner.
Prometheus collectorsScrape configuration, service discovery, remote write to Mimir, Alertmanager cluster setup, local retention sizingServer or pod compute, local disk for the default 15-day retention, Alertmanager nodes, upgrades on the release cycleNumber of targets and series per server, scrape interval, local retention settings, number of Prometheus instances. Confirm which Prometheus release line the quote covers.

Security controls for a shared LGTM stack

A shared LGTM stack holds logs, traces and metrics from many teams, so the controls that matter sit at the edges: who can reach each component, how tenants stay apart, and how long data is kept. Most of these controls are configuration choices you own, and several components expect you to supply them.

Put an authenticating proxy in front of Loki

Loki has no built-in authentication layer. Its documentation expects an authenticating reverse proxy such as nginx, HAProxy, OAuth2 Proxy or Pomerium in front of it, so every push and query path needs to pass through that proxy. Check that the write path used by your shippers and the read path used by Grafana both go through it, and that no internal port is reachable directly. Treat the proxy as part of the Loki security boundary and test it as such.

Put an authenticating proxy in front of Mimir

Mimir requires a tenant ID header on every HTTP request and leaves authentication and authorization to an external reverse proxy. Prometheus remote write and Grafana queries both reach Mimir, so the proxy has to control which callers can write to which tenant and which can read. Confirm that the proxy sets or validates the tenant header itself, so a client cannot choose another team's tenant by editing a request.

Use X-Scope-OrgID for tenant isolation

Loki becomes multi-tenant when auth_enabled is set to true, and it identifies each tenant with the X-Scope-OrgID header on every request. Mimir uses the same header model for tenant separation of series and queries. Isolation depends on that header being assigned by a trusted layer, not by the client. Decide the tenant boundary (per team, per business unit or per environment) before onboarding sources, because changing it later means migrating or re-labeling data.

Harden Grafana cookies and content policy

The Grafana security hardening guidance recommends secure and SameSite cookies, a Content Security Policy, Trusted Types and domain verification against DNS rebinding. Each setting is a config change that can break embedded panels, plugins or reverse proxy paths, so apply them in a staging instance and test every data source and plugin you rely on. Record the final values so the hardened configuration can be reviewed and rebuilt.

Require authentication on the metrics endpoint

Grafana exposes a metrics endpoint for its own health data, and the hardening guidance recommends requiring authentication for it and hiding the version from unauthenticated users. Leaving either open lets anyone on the network learn your version and internal state. Scope this check to the Grafana instance itself and confirm, from outside the trusted network, that the endpoint returns an authentication challenge instead of data.

Know the Grafana authentication limits by edition

In Grafana's authentication docs, LDAP, generic OAuth, Entra ID, Okta, GitHub, GitLab, Google OAuth, auth proxy and JWT are available in all editions. SAML, team sync, active sync and SCIM provisioning are available only in Grafana Enterprise and Grafana Cloud. If your identity policy requires SAML or automated provisioning, that decision affects the edition you run, so confirm it before design work starts.

Set retention as a governance control

Loki retention is disabled by default, so logs are kept indefinitely until retention_enabled is set on the compactor. Retention can be global, per tenant or per stream selector, and each stream rule must be at least 24 hours. Mimir and Prometheus also have retention settings that determine how long metrics remain queryable. Write retention periods down per tenant and data type, because they define what evidence and investigation history you can produce later.

Common failures when connecting the signals

This table lists the failures teams hit most often when wiring Loki, Tempo, Mimir and Grafana together, with the usual cause and the fix for each. Read it as a checklist to run before go-live, since most of these problems look like empty dashboards or silent data loss rather than errors. The main takeaway is that tenancy, retention and storage choices made at the start are the ones that are hardest to correct later.

FailureLikely causeFix
Empty or rejected queriesMulti-tenant Loki or Mimir receives requests without the X-Scope-OrgID tenant header, or Grafana data source has a different tenant IDSet the tenant header on every data source and writer, and confirm the tenant ID matches across Alloy, Grafana and the backend
Logs grow indefinitelyLoki retention is disabled by default, so the compactor keeps logs until retention_enabled is setEnable retention deliberately, set global, per-tenant or per-stream rules, and confirm the compactor runs
Retention rules silently ignoredA stream retention rule is shorter than 24 hours, which Loki does not accept as a valid ruleUse rules of at least 24 hours and check the compactor logs after a configuration change
Grafana fails in high availabilitySQLite is used as the Grafana configuration database, which the install docs say does not scale for productionMove to MySQL 8.0 or later or PostgreSQL 12 or later before adding a second active Grafana node
Trace-to-log or trace-to-metric links show nothingLabel names in Loki or Prometheus do not match the tags configured in the Tempo data source link settingsAlign service and resource label names across signals, then test each link from a real trace span
Duplicate or missing alert notificationsPrometheus sends alerts to a load balancer in front of Alertmanager instead of to each instancePoint Prometheus at every Alertmanager instance and configure Alertmanager clustering with its cluster flags
Data lost after a node failurePrometheus local storage is treated as replicated, but the storage docs say it is a single node and not clusteredShip samples through remote write to Mimir or another long-term store, and back up or replicate the local data separately

Questions buyers ask before choosing a partner for an LGTM stack

What drives the cost of a self-hosted LGTM stack?

The biggest drivers are the scale of data you ingest and keep, the number of environments, and how much integration work sits around the stack. Infrastructure is paid to your cloud or data center provider, object storage for Loki and Mimir recurs with retention, and Grafana, Loki, Tempo and Mimir themselves carry no license fee for the open-source editions. Implementation effort is a one-time cost that depends on the number of signals, data sources, identity integration and high availability requirements. Ask any partner for a quote that separates one-time build work from recurring infrastructure, upgrades and support, and that states which components are in scope.

How long does an LGTM stack build take, and what sets the duration?

Duration depends on the deployment mode you choose, how many signals you onboard, and how much of the surrounding estate must be connected. Loki offers monolithic, simple scalable and microservices modes, and Mimir can run in monolithic mode or as distributed components, so the architecture decision alone changes the work. Other drivers include the object storage and identity provider you already run, network approvals, the number of data sources Grafana must reach, and whether alerting must route to existing systems. Ask for a timeline tied to a written scope and named milestones rather than a generic estimate.

What must our team provide before a partner starts?

Most of the prerequisites sit in your environment, so they should be confirmed before the build starts. The usual list includes:

  • Object storage for Loki chunks and indexes and for Mimir blocks, with a bucket policy and access keys or roles
  • An identity provider for Grafana sign-in, plus a decision on which edition supports the group mapping you need
  • Network paths from collectors to Loki, Tempo and Mimir, plus DNS, certificates and load balancing
  • A reverse proxy that handles authentication in front of Loki and Mimir, because neither enforces it on its own
  • Named owners for on-call, alert routing and upgrade approvals
Can an internal team run an LGTM stack without a partner?

Yes, many internal teams run it, provided they have skills in Kubernetes or virtual machine operations, object storage, Prometheus-style metrics, and log and trace pipelines. The operational load includes upgrades, which Grafana releases in a regular cadence, retention and compaction tuning, capacity planning, backups of Grafana's own database, and on-call for the stack itself. Grafana's high availability setup needs a shared MySQL or PostgreSQL database rather than SQLite, and alerting needs extra configuration to avoid duplicate notifications. Assess your team's capacity honestly and consider whether an internal team, a partner for the first build, or a retainer for ongoing operations fits best.

How does licensing affect internal use of Grafana, Loki, Tempo and Mimir?

Grafana Labs moved Grafana, Loki and Tempo from Apache License 2.0 to AGPLv3 on April 20, 2021, and Mimir is distributed under AGPL-3.0-only. Internal use within your organization is generally a different question from offering the software to outside users as a service, so have your legal team review how AGPL terms apply to your modifications and deployment model. Prometheus remains under Apache-2.0. Grafana Enterprise is offered as a non-AGPL commercial option, and some authentication features such as SAML and team sync are limited to Enterprise and Grafana Cloud. Confirm the licensing position in writing before you commit to a design.

How do Grafana Cloud's pricing tiers compare with self-hosting?

The Grafana pricing page lists a free tier, a Pro tier that starts with a 19 US dollar monthly platform fee plus usage, and an Enterprise tier with a minimum commitment of 25,000 US dollars per year. Grafana Cloud shifts infrastructure and upgrades to Grafana Labs and bills by usage, so the comparison depends on your ingest volume, retention, and the staff hours you would otherwise spend on operations. Self-hosting moves that cost to your cloud bills, storage and people, and it keeps data in your environment. Ask for a side-by-side model based on your expected log, trace and series volumes, and check the pricing page for current usage rates.

What does a quote for an LGTM stack engagement include?

A quote follows from a written scope, so the document should list the components, environments, signals, data sources and integrations included. It should also state the deliverables, such as configuration, provisioning files, dashboards, runbooks, and handover sessions, and the assumptions the partner relies on from your side. Ask whether the price covers high availability testing, retention settings, identity integration and a post-go-live review, and whether ongoing support, upgrades and continuous testing are priced separately as a retainer. Clear assumptions make it easier to compare offers from different partners.

How DeployOpen scopes and supports your LGTM stack

DeployOpen plans, implements, hardens, documents and supports self-hosted Grafana, Loki, Tempo and Mimir inside your own cloud accounts, data centers or air-gapped networks. Engagements start from a written, fixed scope, and ongoing operation can be covered by a retainer.

How we scope

  • Written scope names the signals, data volumes, retention periods and tenants the stack must cover.
  • Scope records which Loki deployment mode and Mimir architecture fit your expected log and metric volumes.
  • Scope lists the data sources, identity provider and reverse proxies that sit in front of each component.

How we build

  • We deploy Grafana with an external database and the Loki, Tempo and Mimir backends on your object storage.
  • We configure trace-to-log, trace-to-metric and metric-to-trace links, then test them against real query paths.
  • We define data sources, dashboards and alert resources as provisioning files so the setup can be reviewed and repeated.

How we hand over

  • Handover includes configuration files, network diagrams, tenant and retention settings, and access model documentation.
  • Runbooks cover component restarts, storage capacity checks, retention changes and backup and restore steps for each backend.
  • Your team receives a walkthrough of upgrade steps, including how Grafana, Loki, Tempo and Mimir release cycles differ.

How we support

  • Retainers can cover patching, version upgrades, capacity reviews and changes to tenants, dashboards and alert routing.
  • Upgrades are planned against each project's release notes and tested in a non-production environment first.
  • Support scope, response expectations and included work are written into the retainer agreement before it starts.

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com