Prometheus security hardening

The core risk is simple: exporters such as SNMP and Blackbox take their targets from URL parameters, and the Prometheus security model warns that anyone with HTTP access to them can make them send requests to arbitrary endpoints. A Blackbox exporter that will probe any address it is handed can be used to reach internal services, cloud metadata endpoints or hosts on a network segment the requester should never touch. Hardening this class of exporter is therefore about controlling who can reach the exporter, which targets it can be asked to contact, and what its configuration permits. Treat each exporter as a small proxy with network reach, not as a passive metrics endpoint. Start with exposure. Bind each exporter to localhost or to a private interface that only the scraping Prometheus server can reach, so it is never listening on a routable address by default. Then apply network policy so that only the Prometheus server's address is allowed to connect to the exporter port. In a Kubernetes or cloud environment, this means a network policy or security group rule that names the Prometheus workload or subnet explicitly. Ask your team to list every host that currently reaches each exporter port, because scrapers from old monitoring systems often remain allowed long after a migration. Next, put authentication in front of each exporter. Prometheus can present TLS client certificates or basic authentication credentials when it scrapes, and many exporters can be configured to require them. Confirm which authentication options your exporter version supports, because they vary by exporter and release. Where an exporter cannot authenticate natively, a reverse proxy that enforces mutual TLS or credentials is the usual fallback. Keep credentials out of plain scrape configuration files where possible, since a leaked scrape file would otherwise hand over both the target list and the secret. The most specific control is restricting which targets each scrape job may request. Many exporters support an allow-list of modules or targets, or let you define named modules whose probe parameters are fixed, so a job can only ask for a set of approved destinations. The exact mechanism depends on the exporter and version, so confirm it against that exporter's own documentation before relying on it. Pair this with a review of the exporter's module definitions: remove unused modules, confirm that each one specifies what it may probe, and check that no module passes arbitrary parameters through to the request. Finally, make these checks repeatable. Record the allowed scrapers, the authentication method, the permitted modules and the target list for each exporter in your change records, and test from a host outside the allow-list to confirm that the connection is refused. Re-run the same test after each exporter upgrade, because changed defaults or new parameters can silently reopen a path that was closed. Ask any implementation partner to show the exact configuration and the negative test results for each exporter in scope.

The goal of testing Prometheus hardening is to prove that every listener refuses access it should refuse, and that a failed check leaves the target down and visible rather than open. Run three groups of checks against a staging copy of the production configuration, then repeat the external scan after every change to the web configuration, the scrape configuration or the exporter deployment. Keep the test plan in the same repository as the configuration so the two cannot drift apart. The first group is an external reachability scan. From a host that should have no network path to Prometheus, scan every port the deployment exposes: the Prometheus web and API port, any exporter ports such as SNMP and Blackbox, and any sidecar or gateway listeners. Build the port list from the actual running processes and the host firewall rules, not from the design document, because listeners added during troubleshooting are the ones most often forgotten. Confirm that the scan from the unauthorized host returns no open application port, and that the same scan from an authorized monitoring subnet shows only the ports that subnet is meant to reach. The second group is functional checks. Confirm that a scrape succeeds when the configured TLS client certificate or basic authentication credentials are presented, and that the same request fails without them. Run both sides from the Prometheus host and check the target health page and the up metric, not only the HTTP status. Then run the negative checks: requests to admin and lifecycle endpoints should return errors unless those features were deliberately enabled, a cleartext basic authentication request must be refused on any listener where TLS is required, and an exporter called from an unauthorized host must not be able to reach an internal address it was not meant to query. Test the last case with an address you control, such as a logging endpoint on a test network, so the result is observable without probing production systems.

A safe failure looks like this: the bad request is rejected, the target is marked down, and an alert fires that someone is paged or ticketed for. The unsafe outcome is a server that accepts an unauthenticated request, returns metrics to an unknown caller or relays requests to arbitrary addresses. Write the expected result for each test before running it, so a pass means the outcome matched the written expectation, not that the output looked reasonable. Confirm the alert actually fires by stopping a test exporter or changing its credentials on purpose, and verify that the alert reaches the receiver your on-call process uses. A rejected scrape that produces no alert is a monitoring gap, and it should be treated as a failed test. Record the results in a form an auditor can follow. For each check, keep the date, the tester, the source host, the command or scan profile, the expected outcome, the observed outcome and a pointer to the raw output. Record the Prometheus version and the exact configuration version used for the run, including the web configuration file, scrape configuration and any reverse proxy settings. Auditors will ask which version was tested and whether that version is the one in production, so the test record and the deployed configuration should carry the same identifier. Store the configuration under version control and tag the commit that was tested. Confirm with your team which checks belong in your evidence set and how often they run. The scan, functional and negative checks can be scripted and rerun after each upgrade, which is useful because minor releases change defaults and the 6-week release cycle means changes arrive regularly. Ask the implementation partner, if one is used, to deliver the test scripts, the expected-results register and the evidence template as part of the agreed scope, and confirm in writing who reruns the suite after future upgrades.

Hardening is not finished at go-live. Three recurring tasks decide whether the controls keep working: rotating TLS certificates and scrape credentials before they expire, checking security settings after every upgrade, and keeping documentation current enough that someone else can run the system. Start by listing every TLS certificate and every credential used for scrapes, remote write, the web interface and Alertmanager, with its expiry date and the person accountable for renewal. Expiry is one of the most common ways a sound deployment breaks, because a lapsed certificate or password can stop scrapes or alert delivery and show up first as missing data. Confirm the renewal lead time with your certificate authority or internal PKI team, and schedule renewals well ahead of each expiry date.

Upgrades require a deliberate choice. A new Prometheus minor release cycle begins every six weeks, and after those six weeks a minor release generally stops receiving bug fixes, so a minor line you deploy will age quickly. The alternative is the long-term support line. The current LTS line is 3.13, supported until July 31, 2027, and LTS releases receive bug fixes for one year. The minor cycle delivers fixes and features sooner but requires an upgrade roughly every six weeks to stay on a supported release. The LTS line requires fewer upgrades, but you must plan the move to a later LTS line before 3.13 reaches end of support. Whichever path you choose, rerun the hardening checks after each upgrade: the TLS minimum version, the basic authentication file, the admin and lifecycle flags, exporter target restrictions, and TLS on Alertmanager and remote write. Read the release notes for changes to flags and web configuration before each upgrade.

Documentation lets the next engineer, auditor or partner verify the deployment without reverse-engineering it. Keep five records current. First, the endpoint inventory: every scrape target, exporter, Prometheus web endpoint, Alertmanager instance and remote write destination, with its protocol and listening interface. Second, the access model: who can reach each endpoint, through which authenticating proxy or network path, and which people hold admin rights. Third, the configuration files, including the main Prometheus configuration and the web configuration, kept under version control with reviewed changes. Fourth, the verification results: dated records of what was tested, how, and the outcome, so later changes can be compared against a known baseline. Fifth, the owner of each credential, with the system it unlocks and its renewal date. Keep secret values in your approved vault and reference them from the documentation without copying them into it.

DeployOpen can run this ongoing work as a retainer, covering certificate and credential renewals, post-upgrade verification, and updates to the inventory and access model. This is an option, not a requirement. If your own platform team will own the work, the same five records still apply, and the post-upgrade checklist should be agreed before the first upgrade. If you are weighing a retainer, ask for the scope in writing before work starts, including which upgrade and verification steps it covers and how credential ownership is split between your team and the provider.