Prometheus SSO and authentication

Prometheus pulls metrics from its targets on a schedule, so scrape traffic is machine-to-machine and cannot complete an interactive login. A browser-based SSO flow expects a user, a redirect and a session cookie, and none of those exist when Prometheus wakes up to scrape an exporter every few seconds. If the authenticating reverse proxy you place in front of the Prometheus web UI also sits in front of the scrape path, scrapes will fail with redirects or login pages. That is why the SSO design has to split traffic into two paths before anything else is configured.

There are two sound ways to handle the scrape path. The first keeps scrapes on a private network and controls access there, using network segmentation, firewall rules and dedicated scrape subnets so that only the Prometheus servers can reach exporter endpoints. The second keeps the endpoints reachable but requires the scraper to authenticate on the target side, using TLS with client certificates or HTTP basic authentication. Prometheus supports both TLS client certificates and basic authentication with bcrypt-hashed passwords, but basic authentication without TLS sends credentials in cleartext, so it should not be used on any path that leaves a trusted network. Your choice should follow from where your targets live and who else can reach them.

Scrape configuration is where the credentials end up. Basic authentication passwords, bearer tokens and TLS key files are referenced in the Prometheus configuration file, and the configuration is often copied into version control, shared across teams and visible to anyone who can read the Prometheus host. Treat these values as you would any other service credential: store them in a secrets manager or a file mounted with restricted permissions, rotate them on a schedule, give each scrape job its own identity where the target allows it, and make sure the configuration file itself is not published to a repository. The same applies to the credentials used by remote write or alert receivers if they appear in related configuration.

Automation needs its own answer too. CI pipelines, deployment jobs and dashboards that query the Prometheus API usually cannot complete an interactive login either, so you need to know exactly how your reverse proxy treats those calls. Confirm whether it accepts API tokens or client certificates for non-browser requests, whether it returns a redirect or a 401 when a token is missing, and whether a token issued for one service can reach endpoints it should not. Also confirm that the proxy does not pass through the admin and lifecycle endpoints, which are disabled by default and should stay that way. Ask your proxy vendor or platform team to demonstrate each path with a test request before go-live, and record the results as part of the access design.

The goal of testing is to prove that the authenticating proxy is the only way into Prometheus, Alertmanager and their APIs. Start with unauthenticated requests. Send them to the web UI, the query API and the Alertmanager UI and API from a machine outside the proxy's trust zone. Each should return a redirect to the identity provider or a rejection, never data. Then try to reach the Prometheus and Alertmanager listen ports directly, bypassing the proxy. Those connections should fail at the network layer. Do not assume the proxy's defaults match your needs. Confirm in your own configuration which paths the proxy protects, whether API paths return 401 or a redirect, and whether any health or metrics path is exempt by design.

Next, test authorization. Sign in as a user who is not in the group that the proxy requires and attempt every protected path, including query, graph, rules, targets and Alertmanager silences. Each attempt should be denied. Then confirm that scraping still works. Scrapers are usually machine clients that should not depend on a browser login, so check that the up metric stays at 1 for each target after the proxy is in place. Confirm which scrape path you use: TLS with client certificates, basic authentication over TLS, or a dedicated internal network path. Basic authentication must never be accepted over plain HTTP on a network path. Send a request with credentials over an unencrypted listener and verify that Prometheus refuses it or that no such listener exists.

Admin and lifecycle endpoints need their own checks. Prometheus disables its admin HTTP APIs by default, and the lifecycle endpoints for reload and shutdown are off unless their flags are set. Send requests to the admin API, the reload endpoint and the shutdown endpoint, and confirm the responses are disabled responses. If your team has enabled either flag, record why, confirm the proxy restricts those paths to a named operator group, and test that group separately. CSRF is a separate risk. Prometheus has no built-in CSRF protection on mutating endpoints, so test them through the proxy. Submit a state-changing request from a page on another origin while signed in, and confirm the proxy rejects it. Check the session cookie's SameSite and Secure attributes and the proxy's own CSRF handling. Repeat this for Alertmanager silence creation, which also changes state.

Failure tests show whether the setup fails closed. Simulate an identity provider outage by blocking the provider from the proxy. New logins should be refused, and no request should pass without a valid session. Confirm how existing sessions behave during the outage, because this depends on the proxy's configuration. Then test expired sessions. Use a short session lifetime in a test environment and check that the UI and API both require re-authentication when it lapses. Finally, restart the proxy during a test window and confirm that no request reaches Prometheus unauthenticated while it starts. Monitoring should also keep collecting, since scrapes should not depend on the browser login path. Record each result, the configuration version tested and who reviewed it, so the evidence can be reused in a compliance assessment or an upgrade test.

The first ongoing task is confirming that the authentication path itself stays healthy, because a failed reverse proxy or identity provider locks engineers out of the dashboards and alerts they need during an incident. Monitor the proxy's availability, response times and authentication error rates, and monitor the identity provider's login and token endpoints as a separate dependency. Add a synthetic check that signs in through the proxy and queries the Prometheus API, then alert when it fails. Also verify that Alertmanager's UI and API still work through the proxy, since a proxy that forwards traffic but mishandles headers can break them silently. Ask your proxy vendor which health and error metrics it exposes and how to scrape them.

Certificates and credentials need an owner and a renewal date. List every TLS certificate in the chain, including the proxy's certificate, the certificates on Prometheus and Alertmanager, and any client certificates used by scrapers or API clients. Rotate them on a schedule your team can rehearse, and reload configuration after each change so scrapes do not fail in a gap. Basic authentication passwords for scrape endpoints and API users are stored as bcrypt hashes, so rotation means generating a new hash, updating the matching secret in the scrape configuration and retiring the old password. Keep plaintext copies in a secrets manager and out of version control. Basic authentication should only run over TLS, because without it the credentials travel in cleartext.

Access review keeps the access model accurate after launch. At a regular interval, compare group membership in the identity provider against the documented roles, remove leavers and reduce access for people who changed teams. Pull the proxy's access logs and check who reached administrative endpoints, and whether any service account authenticates from an unexpected source. Confirm that the proxy maps group claims to Prometheus access correctly, because a stale or mistaken mapping can grant more access than intended. Agree on log retention with your compliance lead, since the required period depends on the frameworks you are assessed against and this page does not set one.

Include the proxy in every Prometheus upgrade plan. Prometheus starts a new minor release cycle every 6 weeks, and after those 6 weeks a minor release generally no longer receives bug fixes, so a minor line needs to move on roughly that cadence. The long-term support line receives bug fixes for one year. The current LTS line is 3.13, supported until July 31, 2027, which makes it the simpler choice for teams that cannot upgrade every few weeks. Whichever track you choose, set a calendar, test the login flow, header forwarding and group mapping against the new version in staging, and schedule the proxy's own patches in the same window.

Three documents make the system recoverable by someone other than the people who built it. The access model lists each group, the Prometheus and Alertmanager capabilities it grants, and who approves changes. The break-glass procedure names the account or path that works when the identity provider is unavailable, who may use it, how each use is logged and how the credentials are reset afterward. The runbook for identity provider outages describes how the health checks above detect the failure, how on-call reaches Prometheus and Alertmanager during the outage, how alert notifications keep flowing without depending on SSO, and who communicates with stakeholders. Store break-glass credentials sealed, and test the runbook on a schedule your team sets.