How Prometheus infrastructure monitoring fits into your estate
Your infrastructure team needs host, endpoint and network device metrics in one place, and the supported method is for Prometheus to pull metrics over HTTP from exporters. The node exporter runs on hosts, the blackbox exporter probes endpoints from the outside, and the SNMP exporter polls network devices. Prometheus finds these targets either through service discovery or through static configuration, and it stores what it scrapes in its own time series database. Each exporter is a separate process with its own configuration file, its own listening port and its own failure modes. Installing the binaries is the simple part. The work lies in deciding which targets each Prometheus server scrapes, how the blackbox modules define a successful probe, which SNMP modules and authentication profiles map to which device families, and who or what is allowed to reach each exporter over the network. Those decisions shape the quality of every dashboard and alert that comes later. Target design deserves attention before any configuration is written. Teams usually need to decide whether one Prometheus server covers a single site, a data center, or a mixed cloud and on-premises estate, and whether scrape traffic crosses network zones that your security team controls. Ask your network team to confirm which subnets can reach the SNMP and node exporter ports, and which firewalls sit between Prometheus and the devices it polls. Access control is the risk to plan for first. The Prometheus security model warns that the SNMP and blackbox exporters take their targets from URL parameters, so anyone with HTTP access to them can direct them to send requests to arbitrary endpoints. Bind exporters to trusted interfaces, restrict access at the network layer, and confirm with your security team how scrape requests are authenticated before any exporter is exposed beyond the monitoring network. Finally, confirm the Prometheus version you will run before you design around it. Release timing and long-term support lines affect how long a given design remains supported, so check the current release cycle page and the GitHub releases with your platform team before you commit to a version and an upgrade plan.
What your team needs before any exporter is deployed
Node, blackbox and SNMP exporters are only as useful as the inventory, network paths and credentials behind them. Settle these items first, because each one determines what Prometheus can scrape, who can approve a change and how long metrics will be kept.
A Prometheus server version
Choose the Prometheus version before any exporter configuration is written. Check the current minor release and the current long-term support line on the release cycle page, because a minor release generally stops receiving bug fixes six weeks after it ships. Confirm with your platform team whether you will run the newest minor line or an LTS line that receives fixes for one year, and record the decision with its end-of-support date.
A target inventory with named owners
List every server, endpoint and network device you intend to scrape, with its role, environment and a named owner who can answer questions and accept alert routing. For SNMP targets, record the device model and the OIDs or MIBs your team expects to read. Without an owner per target, scrape failures and noisy alerts have nobody to go to, and inventory gaps surface only after an outage.
Network paths from Prometheus to targets
Map each path from the Prometheus host to every node exporter port, blackbox probe destination and SNMP agent. Include firewalls, VLAN boundaries, proxies and any NAT on the route, and confirm which direction each connection opens. Blackbox probes also need outbound paths to the endpoints being checked, which are often different from the paths used for scraping. Test each path with the same protocol and port the exporter uses.
SNMP credentials held in a secrets store
SNMP version 2c community strings and SNMPv3 users are secrets that grant read access to device data. Store them in an approved secrets manager, not in plain text in the exporter configuration or in version control, and rotate them through the same process you use for other service credentials. For SNMPv3, decide who creates the users, which authentication and privacy protocols the devices support and how the exporter receives the values at runtime.
TLS certificates for scrape endpoints
Decide which scrape endpoints will use TLS and which certificate authority signs them. Prometheus supports TLS and client certificate authentication, with TLS 1.2 as the minimum, so plan how each exporter presents a certificate and how Prometheus validates it. Assign ownership for issuance, renewal and revocation, and confirm that expiry alerts will reach someone who can act before a certificate lapses.
Storage sized for retention or remote write
Local Prometheus storage keeps data for 15 days by default, and that setting can be changed by time or size. Estimate series counts and scrape intervals with your team, then size the disk for the retention you choose. If you need longer history, decide before deployment whether to send samples to a remote write system, because local storage is single-node and is not clustered or replicated. Confirm who owns backups and disk growth alerts.
Ownership of hosts, devices and access lists
Name the team that owns each host and network device, the person who approves changes to firewall and access control lists, and the process for requesting a new network path. Exporter rollout depends on these approvals, and blackbox and SNMP exporters often need changes outside the monitoring team's control. Agree on a change record for each new path so that access granted for monitoring can be audited and removed later.
Why probe design decides what Prometheus can see
The exporter you choose decides what question Prometheus is answering. The node exporter runs on each host and exposes that host's own metrics, such as CPU, memory, disk and network counters, which Prometheus scrapes directly over HTTP. The blackbox exporter and the SNMP exporter work differently. Prometheus does not scrape them for their own state. It scrapes them with a target supplied as a URL parameter, and the exporter then does work on that target's behalf: the blackbox exporter probes it over HTTP, HTTPS, TCP, ICMP or DNS depending on the module, and the SNMP exporter queries the device over SNMP using a module that defines which object identifiers to walk. Each of these is a separate question, and each returns a separate answer.
The usual layout is one scrape job per module. A blackbox job for HTTP checks, another for TCP checks and another for ICMP keeps the module, the relabeling rules and the timeout visible in one place, and it lets alert rules filter on the job label without guessing which probe type produced a result. The same logic applies to SNMP, where a job per device class or per module keeps authentication settings and walk scope separate. Confirm the module names, the auth and module parameter behavior, and any relabeling pattern against the current blackbox and SNMP exporter documentation, because options change between releases and the examples in older tutorials may not match your version.
A probe result describes the path from wherever the exporter runs, not the view of a customer on the internet. If the exporter sits in a data center, the result reflects that data center's DNS resolver, routing, firewall rules and egress path. A check can report success while a user in another region sees a failure caused by a CDN edge, a regional DNS record or a mobile carrier route. Before you rely on probe results for availability reporting, decide which network position matters for each service and place exporters there deliberately. Running more than one exporter in different locations is often the only way to separate a site problem from a path problem, but the cost of extra instances, their network access and their maintenance should be part of the plan.
Protect the exporters themselves. The Prometheus security model warns that exporters such as SNMP and Blackbox take targets from URL parameters, so anyone with HTTP access to them can make them send requests to arbitrary endpoints. Bind the exporters to a management network or a scrape-only interface, restrict which sources can reach them, and consider an allow-list of permitted target ranges enforced by the network or a proxy. Confirm with your security team how SNMP community strings or v3 credentials will be stored and rotated, since these are read from configuration and can be exposed through misconfigured access.
How to configure the exporters in sequence
This sequence takes an estate from target inventory to a staged rollout of the node, blackbox and SNMP exporters. Each step closes with a check you can see in the Prometheus UI or in a query result, so a fault shows up on the pilot host before it reaches the next wave.
Inventory targets and define labels
The platform or infrastructure team lists every host, endpoint and network device in scope, grouped by role, site and device family. For each target, record the owning team and the exporter that will cover it. Agree on the label names before writing any configuration, because these labels later drive dashboards, alert routing and relabeling. This step is done when the inventory has been reviewed and every target has a role label, a site label and a named owner. Confirm that label names match your existing naming conventions before the pilot starts.
Deploy node exporter on a pilot host
Install node exporter on one non-critical Linux host and run it under your service manager. Confirm the listening port and any flags in the current node exporter documentation for the version you install. Add the host to the Prometheus configuration as a static target carrying the role label from step one, then reload Prometheus using the method your change process allows. Check the Targets page in the Prometheus UI: the target should show as up. A query for the up series with that job label should return 1 for the pilot host.
Define blackbox modules and relabeling
Define one blackbox module for each probe type you need, such as HTTP or ICMP, and set timeouts and expected responses in the exporter's module configuration. Then write a scrape job that passes each probed address as the target parameter and the module as a module parameter. Use relabeling to keep the probed address in the instance label and to point the scrape itself at the blackbox exporter's address. Confirm the parameter and relabel names against the blackbox exporter documentation. Check: run one probe against a known endpoint and confirm probe_success returns 1 in the expression browser.
Define SNMP modules per device family
Group network devices by family, such as switches, routers and firewalls, and create one SNMP module per family that names the MIBs and OIDs that family must return. Generate and validate module definitions with the tooling described in the snmp_exporter documentation for your version. Route each device through its family module using the same target-parameter pattern as blackbox. Check: query one device of each family and confirm the expected interface metrics appear with the device address label, and that the exporter reports a successful scrape for that device.
Keep SNMP credentials out of the config
Store SNMP community strings or SNMPv3 credentials in a separate secret store or a restricted file, and reference that location from the exporter configuration. Keep credential values out of the Prometheus configuration and out of version control. Limit who can read the secret and who can send requests to the exporter, because the Prometheus security model warns that anyone with HTTP access to the SNMP and Blackbox exporters can direct them at arbitrary endpoints. Check: a search of the version-controlled configuration finds no credential strings, and a request to an exporter from outside the approved scrape sources is refused.
Add alert rules for probe and scrape health
Write alert rules for probe failure, based on probe_success, and for scrape health, based on the up series per job. Route them through Alertmanager with the team labels from step one. Agree the evaluation window and severity with the on-call team, and check rule syntax against the Prometheus documentation for your version. Check: point one probe at an unreachable address on purpose, then confirm in the Rules and Alerts pages that the alert moves from pending to firing and that the notification reaches the expected receiver. Restore the probe afterwards.
Roll out in waves
Expand from the pilot to groups of targets by role and site, one wave at a time. Hold each wave until its targets show as up, probes return success, and every new alert has an explanation the owning team accepts. Before the first wave, record the scrape duration and sample volume for each job so that any change in load and storage use after each wave is visible. Check: the Targets page shows every target in the wave as up, and storage and query load match the baseline you recorded before the wave started.
How targets, labels and modules map between the systems
This table maps each monitored source to its Prometheus job, scrape parameter and label scheme, so the team can see where host, endpoint and device metrics enter the system. Read each row as one pattern to copy, then check the label names against the design agreed for your estate. Label names are a design choice, so agree them before the first rollout rather than renaming series after dashboards and alerts depend on them.
| Source | Prometheus job and parameter | Labels to carry | Dropped for cardinality |
|---|---|---|---|
| Host metrics | One node exporter job per host group, keyed by instance (host:port) | instance, owner, environment, site | Per-process and per-container series unless a dashboard needs them |
| Endpoint URLs | Blackbox job per probe type, with module (HTTP or TCP) passed as a URL parameter | target label kept through relabeling, owner, environment | Redirect chains and raw query strings in the target value |
| Network devices | SNMP job per device profile, with module set to that profile and the device address as the target | instance (device address), owner, environment, site | Interface descriptions and high-churn counters not used by any alert |
| Owner label | Set once per job or static target group, from a ownership register | owner (team or service name) | Free-text values; use a fixed list of team names |
| Environment label | Set per scrape group, matching the deployment or network segment | environment (for example, production, staging) | Values not in the agreed list, such as ticket numbers |
| Exporter access control | Only Prometheus reaches exporter ports; probe and SNMP targets are fixed in config | Not applicable | Client-supplied targets from URL parameters on exporters |
| Label naming | Decide names, value sets and the instance convention with the team before rollout | Agreed names documented in the rules file | Labels added ad hoc by individual teams |
How to protect blackbox and SNMP exporters from misuse
Blackbox and SNMP exporters take their targets from URL parameters, which makes them useful to Prometheus and useful to anyone who can reach them. The cards below cover network placement, authentication, admin endpoints and device credentials, the controls that keep these exporters from becoming a request relay inside your network.
Restrict listeners to Prometheus
Bind each exporter to an address and port that only the Prometheus servers can reach, and filter that port at the host firewall and network layer. The Prometheus security model warns that anyone with HTTP access to Blackbox or SNMP exporters can make them send requests to arbitrary endpoints. Check from an unrelated host on the same subnet that the exporter port does not answer, and review the rule after any change to the scrape network.
Allowlist probe targets
Where the exporter supports it, define the permitted probe targets and modules in configuration rather than accepting any target from the query string. Blackbox modules describe how to probe, while the target itself arrives as a parameter, so a module that permits arbitrary destinations is the main exposure. Ask the exporter version you deploy which restrictions it supports, and confirm them with a test request to an address outside your allowlist.
Keep exporters off public networks
Place exporters on internal segments with no route from the internet, and do not publish them through load balancers, ingress controllers or NAT rules. A probe exporter exposed to the internet can be used to scan internal ranges or reach metadata endpoints in cloud accounts. Confirm the security group or firewall policy by testing from outside your network, not only by reading the rule set.
Enable TLS and client certificates
Serve exporter endpoints over TLS, with a minimum of TLS 1.2 as stated in the Prometheus security model, and require client certificates so that only Prometheus presents a valid identity. Plan certificate issuance, distribution to each Prometheus server and rotation before go-live, because an expired certificate stops scrapes silently. Confirm with your exporter version that the web configuration file supports these options.
Avoid basic authentication without TLS
HTTP basic authentication with bcrypt-hashed passwords is supported, but the credential is sent with every request and is exposed in cleartext when TLS is absent. Use basic authentication only over TLS, or skip it in favor of client certificates on internal networks. Check that no configuration, log line or dashboard echoes the password hash or the plaintext credential during troubleshooting.
Put an authenticating proxy in front of the UI
Prometheus has no built-in single sign-on, and its security model notes there is no built-in CSRF protection on mutating endpoints. Route operator access to the web UI and API through an authenticating reverse proxy that enforces your identity provider, and keep the Prometheus port itself reachable only from that proxy and the scraping hosts. Test that a direct request to the Prometheus port without a session is refused.
Keep admin APIs disabled
Admin HTTP APIs are off by default. The --web.enable-admin-api flag enables destructive operations such as deleting series, and --web.enable-lifecycle enables reload and shutdown endpoints. Leave both disabled unless a documented procedure needs them, and if you enable them for a maintenance window, record the change and turn them off afterward. Verify the running flags on each server rather than relying on the deployment template.
Use SNMPv3 and inject secrets at deploy time
Use SNMPv3 with authentication and privacy wherever network devices support it, since SNMPv1 and v2c community strings travel in cleartext. Store SNMP credentials and exporter passwords in your secrets manager and inject them when the configuration is rendered, not in version-controlled files or unit definitions. Confirm which SNMPv3 security levels each device model supports, and plan a credential rotation procedure that updates both device and exporter in one change.
Common failure points when exporters go live
This table lists the failures teams most often hit when node, blackbox and SNMP exporters move from a test setup into production, with the cause and the check that confirms each one. Read each row as a symptom you may see in Prometheus, then work through the cause and fix in the order given. The main takeaway is that most of these failures are visible in target health and probe results long before an alert would fire, so check those first.
| Failure | Likely cause | Fix and check to run |
|---|---|---|
| Blackbox probes report failure | The exporter cannot reach the target from its network position, often because a firewall, proxy or DNS view differs from the Prometheus host. | Run the probe from the exporter host with the same module. Check probe_success and probe_http_status_code or probe_dns_lookup_time_seconds, then fix routing, DNS or egress rules for that network segment. |
| Every probe shows the exporter's own address | Relabeling copies the exporter address into the instance or target parameter, so the probe address is lost or overwritten. | Confirm the __param_target relabel rule maps the target address and that __address__ points at the exporter. Check the target labels on the Targets page and the probe_success series for the expected instance values. |
| SNMP scrapes time out | The configured module walks more OIDs than the device can answer quickly, or the device rate-limits repeated polls. | Reduce the walk to the required OIDs or split large walks into smaller modules. Check scrape_duration_seconds and the SNMP exporter debug output for timeouts, then lower the polling frequency for that device. |
| SNMP module missing or unknown OIDs after a generator change | A regenerated snmp.yml dropped a MIB or module, or the MIB files used by the generator do not match the device firmware. | Diff the generated module list against the previous file before reload. Check the exporter debug output for unknown module errors and confirm the MIB set used by the generator matches the vendor versions in production. |
| Cardinality grows after rollout | Interface descriptions, per-probe labels or unbounded target identifiers are copied into series labels, multiplying series per device. | Count series per metric name in the Prometheus status pages, drop or keep unneeded labels with metric_relabel_configs, and avoid putting free-text descriptions into labels. Review new series counts after each change. |
| Scrape targets disappear after a service discovery change | A discovery source changed its labels or filters, so targets no longer match the keep rules, or the file or API source stopped returning entries. | Check the Targets page for dropped targets and their discovery labels. Compare the service discovery output with the previous configuration and confirm the keep rules still match the new label names. |
| Unauthenticated exporter access found in an audit | Blackbox and SNMP exporters accept targets from URL parameters with no authentication, so anyone who can reach the HTTP port can direct requests at arbitrary addresses. | Restrict the exporter listen address and network access to Prometheus only, and place an authenticating proxy or TLS and basic authentication in front where the platform allows it. Verify with an unauthenticated request from an unapproved host and review the exporter access logs. |
How to test exporter data before alerts depend on it
Before any alert rule is trusted, test each exporter path against a known answer, working from collection through to notification. Start with scrape state. In the Prometheus targets view, confirm that every expected node, probe and SNMP target is reported as up, and that the job and instance labels match the static configuration or service discovery you wrote. A target that is missing from the list is a different failure from one that is down, and the check should record both separately so the team knows whether the problem is in configuration or in the network path.
Node and blackbox checks: For node_exporter, pick a few hosts and compare a value that changes slowly, such as total memory or filesystem size, with the value the host reports for itself. For blackbox_exporter, point a probe at a test endpoint you control that returns a known status code, then at one that is deliberately unreachable. Confirm that probe_success changes as expected and that the timeout and expected status settings in the module behave as you configured them. Because the blackbox and SNMP exporters take targets from URL parameters, restrict who can reach them while these tests run.
SNMP checks: Choose a few interfaces on each network device and compare their operational status and traffic counters with the device's own status view taken at about the same time. Expect small timing differences. Prometheus values reliability over precision, and its docs say it is not the right tool where per-request accuracy is required, so compare trends and state changes instead, such as an interface moving from up to down or a counter continuing to rise. Record the expected result for each check before running it so the outcome can be repeated at sign-off.
Alert and routing checks: Deliberately cause a failure, such as stopping a test exporter or failing a probe, and confirm that the rule fires in Prometheus and reaches Alertmanager. Check that Alertmanager groups it with related alerts as configured, and that silences and inhibition rules suppress only what they should. Confirm the notification reaches the named owner. During rollout, send test alerts to a non-paging receiver so that verification does not wake on-call staff, and move the routes to paging receivers only after the team has signed off on the results.
Questions buyers ask before hiring a partner for exporter work
How much does it cost to hire a partner for Prometheus exporter work?
Prometheus itself carries no license fee. It is licensed under Apache-2.0 and has no paid enterprise edition, so your cost is the work plus the infrastructure and people that run the monitoring. Expect four components: the compute and storage that host Prometheus and any long-term storage, the implementation labor, the ongoing operation by your team or a retainer, and internal time from your staff for access and approvals. Implementation is one-time, while infrastructure, storage and operations recur. Ask each bidder for a quote built from a written scope that lists target types, device profiles, network paths and approval steps, and ask them to separate one-time work from recurring support.
What makes exporter monitoring projects take longer than expected?
The schedule is driven less by the exporter install than by the number of target types, the device profiles that need their own SNMP modules or probe definitions, the network paths the probes must cross, and the approval steps your security and network teams require. A node exporter on a Linux server is usually quick to deploy once access exists. A network estate with several vendor families, segmented management networks and change windows takes longer to validate. Ask for a list of approvals and access requests during scoping, because waiting on those often causes the biggest delays. The written plan should show these items as separate tasks so you can see where the time goes.
What do we need to provide before a partner starts exporter work?
Your team provides the inventory, access, credentials and approvals, and the work cannot start until they are in place. Inventory means the list of servers, endpoints and network devices, plus the addresses or URLs the blackbox exporter should probe, with owners and environments identified. Access means network routes from the Prometheus host to each target and an agreed place to run the exporters. Credentials cover SNMP access, which should be read-only and held in your secrets system. Approvals include change tickets, firewall requests and sign-off from the teams that own each device class. Confirm the format and turnaround of each item in the written plan before work begins.
Can our own team set up Prometheus node, blackbox and SNMP exporters?
Yes. A team with Linux and SNMP experience can run a pilot itself, especially if the first scope covers a small set of servers and one or two network device types. The node exporter is a simple service to install and run. The blackbox exporter needs module definitions and a clear probe design, and the SNMP exporter needs the right MIB and module configuration for each device family. The harder parts are protecting exporters that accept targets from URL parameters and keeping configuration consistent as the estate grows. A pilot is a good way to test your skills before a wider rollout. Bring in a partner if the pilot exposes gaps in those areas.
Is Prometheus suitable for exact billing or audit-grade measurements?
No, not as a system of record. The Prometheus documentation says the project values reliability over precision and is not a good choice where 100% accuracy is required, such as per-request billing. For operational monitoring of servers, endpoints and network devices, that trade-off is usually acceptable. Finance teams and auditors should not treat Prometheus data as the evidence source unless your compliance owner has agreed to it. Confirm which measurements need a different, separately controlled data path, and keep those outside the monitoring design. A partner can help decide which metrics feed alerts and dashboards, but your organization and its assessors decide what counts as evidence.
How is the scope and timeline agreed for an exporter engagement?
Most DeployOpen engagements are fixed-scope, so the scope, deliverables and timelines are agreed in writing before work starts. For exporter work, the written plan should name the target types in scope, the device profiles and SNMP modules, the blackbox probes and their modules, the network paths to validate, the approval steps with an owner for each, and the verification you will accept at handover. It should also list exclusions, such as devices added later or dashboards beyond the agreed set. Changes after signing go through a documented change process that can adjust scope, cost and timing. Ask any partner to show a sample plan before you sign.
How do we stop exporter endpoints from being abused through URL parameters?
Start by restricting who can reach the exporters. The Prometheus security model warns that SNMP and Blackbox exporters take targets from URL parameters, so anyone with HTTP access can make them send requests to arbitrary endpoints. Limit network access to the exporters and the Prometheus server, use TLS and authentication where the exporter supports them, and limit which targets callers can name. Confirm with the documentation for your exporter version how it handles target restrictions, because capabilities differ by release. Test the controls with a review or scan before go-live. Your own team can begin this work, and a partner can review the design.
How DeployOpen scopes and delivers exporter work
DeployOpen scopes exporter work from a written list of servers, endpoints and network devices, then builds node, blackbox and SNMP exporter configuration inside your own infrastructure. Because Prometheus pulls from each target, the agreed scope fixes which targets are scraped, which probe and SNMP modules apply, and how exporter access is restricted before the build starts. Engagements are fixed-scope, with deliverables and timelines agreed in writing first.
How we scope
- Inventory the servers, endpoints and network devices to cover, with owners and scrape paths agreed in writing.
- Separate endpoint reachability checks that need blackbox probes from device metrics that need SNMP modules.
- Agree the written scope covering deliverables, exclusions and who owns alert rules and dashboards before work starts.
How we build
- Configure node exporter collectors per host class, keeping enabled collectors to those the alerting plan uses.
- Write blackbox modules and SNMP module definitions in version-controlled configuration, tested against sample targets before rollout.
- Restrict access to blackbox and SNMP exporter endpoints, because their targets come from URL parameters.
How we hand over
- Deliver exporter configuration, module files and scrape job definitions as documented, reviewable artifacts in your repository.
- Document covered targets, probe and SNMP modules in use, and the steps to add a new device.
- Walk your operations team through targets, scrape health checks and how exporter changes move through review.
How we support
- Run ongoing support, module updates and exporter upgrades under a retainer if your team wants continued help.
- Review scrape health and target lists when devices are added, retired or re-addressed, as the retainer agrees.
- Confirm exporter version support windows with the upstream project before scheduling upgrades.
