When the OpenTelemetry Collector agent vs gateway choice matters
The agent pattern fits a small, stable estate, the gateway pattern fits teams that need centralized credentials and policy, and tiers fit estates that need both plus trace-aware sampling. The OpenTelemetry Collector is a single binary, so this choice is a deployment decision, not a product decision: the same build can run as a sidecar or DaemonSet, as a standalone gateway, or in both roles. Platform teams should settle the pattern before they pick exporters, because the pattern determines where credentials live, where filtering and sampling happen and how many components must be patched and monitored.
The agent pattern suits a estate with few services, stable topology and a clear one-to-one mapping between a Collector and the workload it serves. The OpenTelemetry docs describe it as straightforward to start with, but limited in scalability and inflexible when deployments are complex or change often. The gateway pattern suits teams that want one otlp endpoint for all applications, centrally managed exporter credentials, and policy such as filtering or sampling applied in one place. The docs list its costs as one more component that can fail, added latency with cascaded Collectors and higher overall resource usage. Tiers make sense when agents handle local collection and a gateway tier handles policy, and when tail sampling needs every span of a trace in one place. In that case, the load-balancing exporter with routing_key set to traceID is the setting to test first, and the OpenTelemetry gateway deployment docs describe the pattern.
Before you commit, confirm three things with your own environment: how many services and hosts will send telemetry, how often workloads and their Collector configurations change, and which team will own the gateway tier's uptime and upgrades. OpenTelemetry is not an observability backend, so storage, query and retention choices sit outside this page and must be decided separately for traces, metrics and logs. Also check whether your configuration must use authenticator extensions, such as OIDC on receivers or oauth2client on exporters, since those requirements can push you toward a central tier even for a small estate.
OpenTelemetry Collector agent vs gateway deployment, factor by factor
This table compares agent, gateway and tiered Collector layouts on the factors the OpenTelemetry docs call out, with the condition under which each factor starts to matter. Read each row against your own fleet size, credential model and sampling needs, and treat the docs' listed pros and cons as the starting point for your design review.
| Factor | Agent (sidecar or DaemonSet) | Gateway (central OTLP endpoint) |
|---|---|---|
| Placement | Runs next to each application or on the same host, such as a sidecar or DaemonSet | Standalone Collectors receive telemetry from applications or other Collectors at one OTLP endpoint |
| Scalability | Docs list limited scalability, which matters as the number of hosts and services grows | Scales by adding Collector instances behind the endpoint, which adds a component to operate |
| Where credentials are held | Each agent holds its own exporter credentials, so rotation touches every node | Credentials are centrally managed at the gateway, which the docs cite as a separation of concerns |
| Where filtering and sampling run | Runs locally, close to the source, with limited central visibility | Centralized policy such as filtering or sampling, and tail sampling needs trace-aware load balancing |
| Added latency | Typically one hop from the application to the backend | Docs list added latency from cascaded Collectors, which matters when a tier is chained behind another |
| Resource use | Consumes resources on every node, which the docs note as a cost across a large fleet | Docs list higher overall resource usage, which matters when gateway tiers multiply |
| What can fail | A failed agent affects only its host or workload, but configuration drift across nodes is harder to see | Docs list one more component that can fail, so gateway availability becomes a shared dependency |
| Rolling out change | Config changes must reach every node or sidecar, so rollout follows the orchestrator's pace | Changes land on fewer instances, but a bad change at the gateway affects all sources at once |
Why the agent pattern is easy to start and hard to scale
The agent pattern is the quickest way to get OpenTelemetry Collector data flowing, because each Collector runs next to the workload it serves, as a sidecar in a pod or as a DaemonSet on each node. That gives a clear one-to-one mapping: a service's telemetry leaves through the Collector beside it, and a team can see exactly which process produced which spans, metrics and logs. The OpenTelemetry docs describe this layout as straightforward to start with, which is why it often becomes the first deployment on a new cluster or VM fleet. The same Collector binary runs in either role, so the choice is about where it sits, not which software you install.
The docs list two limits that matter once the estate grows. First, scalability is limited: every agent is another process consuming CPU and memory on its host or pod, and the approach does not give you a single place to absorb bursts or to scale the processing layer independently of the workloads. Second, the pattern is inflexible for complex or evolving deployments. Each agent carries its own configuration, so a change to credentials, filtering or sampling policy has to reach every node, and a partial rollout leaves some services emitting data under the old rules. For a regulated team, that means an audit question such as "which exporter endpoint and credential is every service using today?" becomes a fleet inventory exercise. Before you commit, ask your platform team how configuration reaches agents. Confirm whether your GitOps tooling, Helm values or configuration management pushes Collector changes, how long a rollout takes across your largest cluster, and how a failed rollout is detected and rolled back. If those answers are unclear, the agent pattern will become harder to operate as the number of teams and environments grows.
What the gateway pattern adds to your operating surface
The gateway pattern centralizes credentials, policy and sampling, but the OpenTelemetry docs list costs that land on your operations team. Each card below names a condition under which that cost becomes real and what to confirm in your own environment before committing.
One more component that can fail
A gateway is a standalone Collector service that every agent or application depends on for export. If one gateway instance or its load balancer fails, telemetry from many workloads can stop at once, so the agent pattern's per-host failure domain is replaced by a shared one. This bites most when a single gateway serves all clusters or all production services. Confirm how your agents buffer and retry when the gateway is unreachable, how quickly a failed instance is replaced, and who is paged when the tier stops accepting data.
Added latency from cascaded Collectors
When an agent forwards to a gateway, and the gateway forwards again to a backend, each hop adds processing and network time. The OpenTelemetry docs flag this as a downside of the gateway pattern, and it matters most for workloads that need near-real-time alerting from traces or metrics. The delay is usually small per hop but compounds with batching, retries and queueing at each tier. Measure end-to-end time from span creation to backend visibility in a staging environment that matches your topology, and confirm which hops batch and where queues fill under load.
Higher overall resource use
Running an agent tier and a gateway tier means more Collector processes consuming CPU and memory than either pattern alone, and the gateway adds memory for queues and tail sampling buffers. The cost is shared across the platform team's budget, not just the application teams that send the data. It grows with trace volume, the number of processors enabled and the retention window of any sampling decision buffer. Ask your team to estimate the footprint of each tier from load tests with representative telemetry, not from defaults.
Sizing and monitoring the gateway tier
A gateway needs capacity planning that agents do not, because its load is the sum of every sender. Scaling decisions depend on queue depth, refused data, export failures and CPU, and trace-aware load balancing adds a routing layer whose own health must be watched. Without these signals, a gateway can drop data silently while every agent reports success. Confirm which Collector internal metrics your team will alert on, the headroom you will keep above observed peak load, and how scaling events affect trace routing for tail sampling.
How centralized credentials and policy work at the gateway
The main reason to put a gateway in front of your telemetry is that it separates concerns. The OpenTelemetry docs describe the gateway as a single OTLP endpoint served by one or more standalone Collectors, where credentials can be managed centrally and policy such as filtering or sampling is applied in one place instead of in every agent. In practice, this means the teams sending telemetry do not each need their own copy of backend tokens, and a change to a sampling rule or a redaction filter does not require redeploying every application host. This benefit grows with the number of senders. If a handful of services in one team send telemetry to one backend, the agent pattern may be enough. If dozens of teams, business units or tenants send data, the gateway is where you enforce the rules once.
The Collector handles this with extensions rather than with application code. Authenticator extensions can check incoming connections on receivers, for example by validating OIDC tokens presented by the sender, so only authorized clients get through the gateway. On the outbound side, the oauth2client extension can attach credentials to requests sent by exporters, so the gateway holds the backend credential and the senders never see it. Both are configured under the extensions section and then referenced from the receiver or exporter that uses them. Before you design around them, confirm which authenticator and client extensions your chosen Collector distribution includes, and check the stability level documented in each component's README, since core components have mixed stability levels. The OpenTelemetry Collector configuration docs are the natural next read for the exact keys.
Running the OpenTelemetry Collector agent vs gateway over time
This table compares what each Collector pattern asks of your team after go-live: upgrades, configuration rollout, health monitoring, capacity planning and skills. Read each row against your own fleet size and change process, because the operating load differs more in change management than in day-one setup. The main takeaway is that agents spread risk across many small changes, while a gateway concentrates risk in fewer components that need careful drain and rollout planning.
| Area | Agent pattern | Gateway pattern |
|---|---|---|
| Upgrades | Agent fleets upgrade node by node, so a bad release affects one host or pod at a time, but the rollout touches every node. Releases such as v0.161.0 (September 16, 2026), v0.162.0 (September 29, 2026) and v0.163.0 (October 16, 2026) each need validation across the fleet. | Gateway tiers need drain and rollout planning so in-flight telemetry is not dropped. Fewer instances change, but each change affects more traffic. Confirm the drain behavior of your load balancing and queue settings before upgrading. |
| Configuration rollout | Each node carries its own config, so a change must reach every agent. Config drift between hosts is a common failure mode to check for. | One config change covers the central tier, but tail sampling and load-balancing rules must stay consistent across all gateway instances and any cascaded Collectors. |
| Health monitoring | Monitor the agent on every host or pod, including exporter failures and queue backlogs. Alert volume grows with node count. | Monitor a small set of gateway instances closely, including the single OTLP endpoint, queue sizes and downstream backend errors. Gateway failure affects many applications at once. |
| Capacity planning | Each agent uses resources on its own host, so capacity follows the workload on each node. Limited scalability is listed in the OpenTelemetry docs for this pattern. | Gateway sizing follows total telemetry volume and any tail sampling buffers. The docs list higher overall resource usage and added latency from cascaded Collectors as costs to measure. |
| Security and release checks | Each node runs a copy of the Collector binary, so artifact verification must run on every deployed version. Releases ship signed artifacts, SHA256 checksums and SBOM files that your security team can check. | Fewer binaries to verify, but the central tier handles credentials and policy, so review its authenticator and exporter settings on each release. Confirm component stability levels in each pipeline before upgrading. |
| Skills and ownership | Platform or node teams manage agent configuration. The pattern is straightforward to start, but changes often need coordination with each application team. | A platform team owns the gateway, the load-balancing exporter and sampling policy. Plan for on-call coverage, since the gateway is one more component that can fail. |
Why tail sampling needs trace-aware load balancing
Tail sampling only works if every span of a trace reaches the same Collector. A tail sampling processor decides whether to keep a trace after its spans have arrived, so it needs the complete set of spans for that trace in one place. If a gateway Collector receives only part of a trace, it cannot judge the whole trace correctly and will keep or drop the wrong data. The OpenTelemetry gateway deployment guidance addresses this with the load-balancing exporter, which should be configured with routing_key set to traceID. That setting sends all spans sharing a trace ID to the same backend Collector. This is a gateway or tier requirement. An agent running beside a single application does not need it, because it does not fan spans out across several Collectors.
The same guidance recommends routing by service when span-to-metrics aggregations matter. Metrics derived from spans, such as request counts and latency histograms per service, are only accurate if all spans from a given service land on one Collector that aggregates them. Splitting one service across Collectors can produce duplicated or partial metrics. Choose the routing key deliberately: trace ID for tail sampling, service for span-to-metrics. If you need both on the same tier, plan for two layers of load balancing or accept a trade-off, and document which pipeline depends on which key.
Routing is only as stable as the Collector pool behind it. When Collectors are added, removed or restarted, the load-balancing exporter must resolve the new membership and re-route traces, and traces in flight can be split across old and new owners. Before relying on tail sampling in production, test the behavior in a staging tier: scale the gateway up and down, restart one Collector during steady traffic, and confirm that sampled traces remain complete and span-to-metrics counts stay consistent. Confirm with your Collector version's release notes how the exporter resolves its backends in your environment, and record the results as acceptance evidence.
How to move from agent-only to a tiered layout
This sequence takes a team from agents on every host to a tiered layout with a gateway tier, one step at a time, so each change is verified before the next one adds risk. The end state is a gateway tier that handles credentials and policy, with trace-aware routing in place before any tail sampling runs.
Inventory workloads and telemetry types
The platform or observability team lists every workload that emits telemetry, the hosts or clusters it runs on, and whether it sends traces, metrics or logs. For each one, record which backend receives the data, who owns the credentials used to send it, and whether any sampling or filtering is applied today. Ask the application owners to confirm the list, since they know which services emit data outside the standard path. The step is done when the inventory has a named owner for every workload and every credential.
Start with agents where mapping is one-to-one
The platform team deploys the Collector as a sidecar or DaemonSet for workloads where one agent maps cleanly to one application or host. The OpenTelemetry docs describe this pattern as straightforward to start with, so it is the right first step for a pilot. Confirm the agent configuration, the exporter target and the resource limits with the owners of the affected hosts before rollout. The step is done when telemetry from the pilot workloads arrives at the backend and the agents run within the limits the team set.
Add a gateway tier for credentials and policy
Once credentials or central policy such as filtering or sampling must be managed in one place, the security or platform team stands up one or more standalone Collectors behind a single OTLP endpoint. Agents or applications are repointed to that endpoint, and the gateway holds the outbound credentials, typically through authenticator extensions. Confirm with the security team which authentication method applies to incoming and outgoing connections. The step is done when credentials are no longer stored on each host and the gateway passes test telemetry to the backend.
Confirm the gateway can absorb added latency
Because the OpenTelemetry docs list added latency from cascaded Collectors and extra resource use as downsides of the gateway pattern, the platform team measures the delay and resource use of the gateway path against the agents' baseline. This is also the point to decide whether any cascaded tier is needed at all. Ask the backend owner to confirm the ingest rate and retry behavior they expect. The step is done when the team has recorded the latency and resource numbers and accepted them as the baseline.
Introduce trace-aware load balancing before sampling
Before any tail sampling runs, the platform team places a load-balancing exporter in front of the gateway tier with routing_key set to traceID, so all spans of a trace reach the same Collector. Routing by service is the alternative the docs describe for keeping span-to-metrics aggregations accurate, and the team chooses the key that fits its backends. Confirm the routing key and the list of gateway instances with the people who own the sampling policy. The step is done when a sample of traces shows complete spans on a single gateway instance.
Enable tail sampling on the gateway
After routing is verified, the owners of the sampling policy enable tail sampling on the gateway tier, starting with a narrow policy that applies to a subset of services. The policy owner should document which traces are kept and dropped, and confirm this with the teams that rely on those traces for incident response. Compare trace volume and missing-span reports before and after the change. The step is done when the policy is documented, approved by the service owners and producing the expected volume.
Monitor gateway health and capacity
The platform team sets up monitoring for the gateway tier itself, covering queue usage, dropped data, export failures, memory and CPU, and the health of each instance behind the load balancer. Because the gateway is one more component that can fail, the team also defines who is paged when it does. Confirm the alert thresholds and the on-call rota with the operations owners before the tier takes production traffic. The step is done when alerts fire in a test scenario and the team has a documented capacity plan for scaling the gateway.
Questions buyers ask before choosing the Collector deployment pattern
Which Collector pattern costs more to run over time?
The gateway tier usually carries the recurring cost of a single, shared component, while the agent pattern spreads recurring work across every host or pod. Cost depends less on the pattern than on who operates it. Ask who runs the agent fleet (configuration rollouts, version upgrades, restarts) and who runs the gateway tier (scaling, failover, capacity for traces and metrics). The OpenTelemetry docs list added latency from cascaded Collectors and higher overall resource usage as gateway downsides, so budget for the extra compute and the on-call rotation that a central tier adds. Ask any provider to price the fleet and the gateway separately, with upgrades and on-call listed as recurring line items.
Can our platform team run the Collector ourselves without outside help?
Yes, many teams do, and the Collector is a single binary with documented agent and gateway patterns. Whether it is practical depends on your Kubernetes or host automation, your on-call capacity and how many configuration changes you expect each quarter. Teams often struggle with three things: keeping agent configuration consistent across hosts, testing component stability levels (documented in each component's README) before enabling them, and maintaining a tail sampling setup that routes by trace ID. If you already run GitOps tooling and have someone who owns telemetry, self-service is realistic. If not, DeployOpen can plan, implement and document either pattern under a fixed scope agreed in writing, and offer ongoing support or upgrades as a retainer.
What do we need to provide before work starts on a Collector rollout?
Expect to supply access and decisions, not just a list of hosts. On the platform side, you need Kubernetes access or host access, the node and namespace layout, and the network paths from applications to the Collector endpoints. On the identity side, collect the identity provider details if receivers must authenticate with OIDC through an authenticator extension, and the credential flow for exporters that use the oauth2client extension. On the policy side, decide which telemetry is kept, filtered or sampled, and who signs off on those rules. Missing sampling decisions are the most common reason a scope drifts, so settle them before the fixed scope is written.
How long does migrating from direct exports to a Collector take?
Plan the migration in phases rather than as a single cutover. A common sequence is to deploy the Collector beside existing exporters, send a small set of services through it, verify that traces, metrics and logs arrive intact, then move remaining workloads and retire old exporters. Each phase has its own exit criteria, such as dropped-data checks and dashboard parity. Elapsed time depends on how many services emit telemetry, how many teams must approve changes and how many backends are involved. Ask for a phased plan with exit criteria and a written timeline in the quote rather than a single date, because the timeline should follow the agreed scope.
Does tail sampling force us to run a gateway?
Tail sampling decides after a trace completes, so all spans of a trace must reach the same Collector. The OpenTelemetry gateway deployment docs describe the load-balancing exporter with routing_key set to traceID as the way to send every span of a trace to the same Collector. That means a gateway tier, or an equivalent routing layer, is required for multi-instance tail sampling. Routing by service keeps span-to-metrics aggregations accurate. An agent-only setup can still do head sampling or per-host filtering without this tier, so the answer depends on the sampling policy you choose.
What should we ask for in a written quote for either pattern?
Ask for the scope in terms of components, not hours. A useful quote lists the deployment pattern or tiers, the Collector distribution and version policy, the receivers, processors and exporters in use, the authenticator extensions and how credentials are stored, the sampling policy and its owner, and the testing performed before cutover. It should also separate one-time work (design, implementation, documentation, migration) from recurring work (upgrades, support, on-call, policy changes). Ask what evidence will be delivered for compliance reviews, and which items are excluded. DeployOpen does not publish a price list, so the quote follows from the written scope.
Does a Collector rollout help with compliance evidence?
A Collector rollout can support compliance readiness by centralizing credentials, documenting which telemetry leaves each environment and recording configuration changes. It does not create compliance proof by itself. DeployOpen prepares systems, controls and evidence for assessments, but independent auditors and certification bodies issue any audit report, attestation or certification. Ask the auditor or framework owner which evidence they expect, such as configuration baselines, access records for the gateway and retention settings, and confirm those items are in the written scope before work starts.
How DeployOpen scopes and runs Collector deployments
DeployOpen plans, implements, integrates, documents and supports self-hosted open-source platforms inside the client's own infrastructure, including OpenTelemetry Collector pipelines. We start from the telemetry sources, the backends and the credential and policy requirements, then decide with the client which Collector pattern fits each part of the estate.
How we scope
- Inventory which services, hosts and clusters emit traces, metrics and logs and where each signal must land.
- Record which teams own credentials, sampling rules and filtering policy for each signal type.
- Agree in writing whether the engagement covers agent, gateway or tiered layouts, with deliverables and timelines fixed before work starts.
How we build
- Deploy agent Collectors as sidecars or DaemonSets only where per-host or per-workload isolation is needed.
- Configure gateway Collectors behind one OTLP endpoint, with authenticator extensions for incoming and outgoing credentials.
- Set up the load-balancing exporter with routing_key set to traceID when tail sampling runs behind a gateway, and test trace completeness before cutover.
How we hand over
- Document each Collector tier, its configuration files, its owners and the failure modes the OpenTelemetry docs flag for each pattern.
- Provide runbooks for scaling gateway replicas, rotating credentials and rolling back a Collector version.
- Walk the client's operations team through alerts on Collector health so they can run the pipeline without DeployOpen on call.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped in writing like the original engagement.
- Re-test the pipeline when the Collector moves to a new release, such as after a minor version that changes a component's stability level.
- Confirm component stability from each component's README before relying on it in production, and record that check in the handover.
