OpenTelemetry metrics into Prometheus with OTLP

Prometheus normally pulls. It discovers targets through service discovery or static configuration, then fetches their metrics over HTTP on a schedule it controls. The OTLP path reverses that direction: senders push metrics to Prometheus through its OTLP receiver. That single change affects who holds the endpoint, how targets are found, what happens when a sender is slow or down, and where configuration lives, so the team should settle it before writing any config. A scrape target is something Prometheus can see and check. A pushed sender is something Prometheus only hears from when it sends.

There are two common shapes. In the first, applications or agents send OTLP directly to Prometheus. This is the simplest path with the fewest moving parts, but every sender needs the Prometheus endpoint, and every sender must handle its own retries and buffering. In the second, a Collector runs in agent mode next to workloads or in gateway mode in front of Prometheus, receiving metrics from many sources and forwarding them. The Collector is a vendor-agnostic receiver, processor and exporter, so it can sit between instrumented code and Prometheus without changing the applications. The gateway shape centralizes control, and most teams choose it once more than a handful of services are involved.

In the gateway shape, the Collector is usually where batching, filtering, authentication and retry behavior are set. Batching reduces request overhead, filtering removes series nobody uses, authenticator extensions can check incoming connections and add credentials on outgoing requests, and exporter retry settings decide how much data survives a short Prometheus outage. Teams should confirm which of these the Collector build they plan to use provides, because component stability varies and is documented in each component's README. Confirm the exact component names and options against the Collector configuration documentation for the release you choose, not against examples copied from older guides.

Prometheus is designed to value reliability over precision. Its documentation says it is not a good choice where 100% accuracy is required, such as per-request billing. Pushed metrics do not change that. A gateway can retry and a sender can buffer, but a metric pipeline still samples and aggregates, and a short gap or duplicate can occur during failure recovery. If the use case needs billing-grade accuracy, the team should first ask whether metrics are the right signal at all, and whether that data belongs in logs, traces, or a transactional store with its own reconciliation. Writing that answer down before the build starts avoids rework.

Before writing configuration, decide the shape with the people who will run it. Ask who owns the endpoint, who receives the alert when the gateway backs up, how credentials rotate, and whether applications already emit OTLP or need a Collector to translate. Record the choice in the design document, including the expected number of senders and the failure behavior you accept. Those answers determine the batching, retry, and authentication settings that follow, and they are the basis for any quote or plan an implementation partner produces.

When several OpenTelemetry Collector replicas send metrics for the same series to Prometheus over OTLP, the samples do not reliably arrive in timestamp order. Prometheus expects each series to grow forward in time. By default, a sample older than the newest sample it already holds for that series is rejected. Running more than one replica is a common way to get redundancy, so this rejection is a normal outcome of the design unless you configure Prometheus to accept late samples within a bounded window.

Each replica batches and sends on its own schedule, and network delay or retries can make one replica deliver a later sample before another replica delivers an earlier one. Replicas that receive or collect the same workload can also send the same series twice with identical or nearly identical timestamps. In queries, delayed samples appear as gaps that later fill in, and duplicates can show up as doubled lines, jitter or spikes in rate calculations. Prometheus does not deduplicate these for you, so decide early whether duplicates are prevented upstream in the Collector pipeline or tolerated in the data.

Enabling out-of-order ingestion means Prometheus must hold accepted late samples in memory until they are compacted into blocks. A wider out-of-order time window accepts older samples but keeps more data in the head block, which increases memory use. A narrower window protects memory but rejects samples from slow replicas. Set the window using the Prometheus documentation for your release, since the setting and its behavior can change between versions. Then test with the replica count and send interval you plan to run, because no fixed value fits every workload. Confirm the memory headroom under that test load before you choose the value.

Local storage is a single node that is not clustered or replicated, so a larger head and extra memory pressure land on one machine. That affects how you size memory, disk and the retention period (15 days by default, configurable by time or size), and whether you send data onward through remote write. Because Prometheus favors reliability over precision, accept that some late or duplicate samples will be handled rather than perfectly reconciled. Monitor the out-of-order and rejected sample counters that Prometheus exposes, along with OTLP send failures and retries in the Collector, and alert when either rises. Confirm the metric names for your release before you build the dashboards.

Write the acceptance criteria into the scope before anyone sends a test metric. The criteria should name the name translation strategy under test, the promoted resource attributes, the send rate the load test must reach, and the alert rules and Grafana panels that must still resolve. Record who signs off each criterion and what happens if one fails. A test plan without agreed pass conditions tends to end with a debate about whether the results are good enough, so settle that question on paper first.

Start with a known test metric from one sender, such as a single Collector instance emitting a counter with a fixed set of resource and data point attributes. In Prometheus, confirm the series name after translation, the metric type, the labels, and whether each promoted resource attribute appears as a label. Compare the result with the translation rules in your chosen configuration, and check the names with a PromQL query rather than only the explore view. Then repeat the test with a second Collector replica sending the same series. This exposes out-of-order handling and duplicate series behavior, which are the failure modes most likely to surprise a team that has only tested one sender.

Run the load test at the planned send rate for long enough to reach steady state, and watch Prometheus while it runs. Track resident memory, the head series count and how quickly it grows, and the rejected sample counters for out-of-order, too-old and invalid samples. Promoted attributes multiply the number of series, so check the head series figure against the sizing you expect from the attribute cardinality of your real workloads. Confirm the exact metric names for rejected samples in the documentation for your Prometheus version, because the names of these counters have changed across releases. Ask your vendor or the project documentation what head series and memory levels are reasonable for your hardware before you set pass thresholds.

Keep the existing scrape path running in parallel until dashboards and alerts are validated. Point the same alert rules and Grafana queries at both sources, then confirm that each PromQL expression resolves under the chosen name translation and returns the same results within the tolerance your acceptance criteria allow. Check recording rules and any label matchers that depend on the old names. Cutover should happen only when every criterion in the scope has passed and been signed off. If a criterion fails, the scrape path remains the source of truth while the translation strategy, promoted attributes or replica settings are corrected and retested.