When ListenOTLP in NiFi is the right OpenTelemetry receiver
A security or platform team that already runs NiFi for log pipelines can use ListenOTLP as one ingestion point for OTLP logs, metrics and traces, then route each signal to the backends that need it. The decision is whether that extra receiving role belongs in NiFi or in an OpenTelemetry Collector. OpenTelemetry is an observability framework and toolkit for generating, exporting and collecting telemetry, and it is not an observability backend, as the OpenTelemetry overview explains. NiFi would therefore be the receiving and routing layer only. Storage, querying and dashboards still belong to separate tools chosen for each signal.
NiFi is usually the stronger fit when the same team already operates the NiFi cluster, has change control and provenance requirements for every data path, and needs to filter, enrich or deduplicate telemetry alongside syslog and other log feeds. Keeping all inbound data in one governed platform simplifies audit evidence and on-call ownership. The Collector is usually the stronger fit when most telemetry comes from applications instrumented with OpenTelemetry SDKs, when agents or gateways must process data close to the source, and when you need Collector-specific features such as authenticator extensions. Those extensions can authenticate incoming connections with OIDC and add OAuth2 credentials to outgoing exporter requests. The Collector is Apache-2.0 licensed and vendor-agnostic, so it is a common choice for an OTLP-native pipeline.
Before committing, confirm in the NiFi version you run that ListenOTLP offers the gRPC and HTTP listeners, TLS and client authentication settings, and output formats your design needs. Check the signal types it emits and how each maps to your backends. Also check the maturity of the SDKs your services use: the OpenTelemetry status page shows that trace SDKs are stable in most languages, while log SDK maturity varies and core Collector components have mixed stability. Cost comes from NiFi cluster capacity, licensing of your backends, and the engineering time to build, test and operate the flow. Ask any partner for a written scope that names the signals, languages, volumes and acceptance tests before work starts.
Configuration options and when each fits
This table compares the main ListenOTLP decisions: transport, TLS and client authentication, listening address and port, and how signals are separated for routing. Read each row as a design choice, then confirm the property names and defaults in the ListenOTLP documentation for your exact NiFi 2 release before you build.
| Decision | Option or setting | When it fits | Confirm before building |
|---|---|---|---|
| Transport | gRPC endpoint | Agents and SDKs configured for OTLP over gRPC, where you want a single long-lived connection per sender and the sender's default protocol is gRPC. | Which OTLP transports the processor exposes in your release, and the port each one uses. |
| Transport | HTTP endpoint | Senders behind HTTP proxies or load balancers that handle HTTP more easily than gRPC, or runtimes where gRPC support is limited. | Accepted content types and whether protobuf and JSON encodings are both supported. |
| Transport | Both protocols in one listener | Mixed fleets during a migration, where some senders use gRPC and others use HTTP and you want one ingest point. | Whether one listener can serve both protocols or whether you need two listener instances with separate ports. |
| TLS | Server certificate only | Internal networks where senders trust your certificate authority and client identity is handled by network controls. | Which keystore and truststore properties apply, and how certificate rotation reloads without a restart. |
| TLS | Mutual TLS with client certificates | Senders in other zones or tenants, or any source where you need to tie each connection to a known client identity. | Client authentication mode values, how the truststore is built, and how revoked clients are removed. |
| Bind address | Specific interface address | Hosts with a separate telemetry network, so the listener is reachable only from that segment. | The bind address property and whether the value must match a cluster node address on every node. |
| Bind address | All interfaces | Cluster nodes behind a load balancer where the balancer reaches each node on its own address. | Firewall rules at the host and network layer, since an open bind address is reachable from every attached network. |
| Port | Standard OTLP ports or a custom port | Standard ports keep sender configuration simple. Custom ports help when another service already uses the default on the same host. | Port conflicts with other NiFi listeners on the same node, and any port your sender tooling expects by default. |
| Signal separation | One listener, routing by signal type | Small volumes or one shared destination, where a single flow can branch logs, metrics and traces with a router. | How the processor marks each signal type on output and which attributes carry the signal name in your release. |
| Signal separation | Separate listeners per signal | Different ports or security settings per signal, or different service levels for traces than for logs. | Whether separate listeners on one node compete for the same resources, and how load balancers map ports to signals. |
Setting up an OTLP receiver flow in NiFi
Build the receiver in six steps: confirm the runtime, bind the listener, secure it, route each signal, connect backends with queue limits, and test end to end. Each step ends with a check you can record in the change ticket, so the flow is proven before any production agent sends data to it.
Confirm the NiFi 2 runtime and Java 21
The engineer who owns the NiFi hosts confirms that each node runs NiFi 2.x on Java 21, the minimum runtime listed in the NiFi administration guide. Check the Java version the NiFi service actually uses by reading the service definition or process environment, since a login shell may report a different version. If you are still on NiFi 1.x, the migration guidance requires moving to 1.27.0 first. Decide now whether the optional Python processors are needed, because they run on Python 3.10, 3.11 or 3.12 and are a beta feature. The step is done when each node starts cleanly and the version shown in the UI matches the change record.
Bind the listener to the intended interface
Choose the network interface and port the OTLP listener will accept traffic on, usually a private address that application hosts or a load balancer can reach. Remember that NiFi binds its UI to 127.0.0.1 when no security configuration is provided, so set the listener address deliberately before you open any firewall rule. Confirm the firewall allows the chosen port only from the sender networks you expect. The step is done when a listening socket appears on the chosen address (for example, from
ss -ltnon the node) and a connection from an unapproved network is refused.Enable TLS and client authentication
The security engineer attaches an SSL context service to the listener that holds the server certificate and a trust store containing the certificate authority for your telemetry senders. Turn on client certificate verification so only senders with a certificate from that authority can deliver data. NiFi user login does not protect this listener, so treat the certificate trust chain as the access control and document who issues sender certificates. The OpenTelemetry Collector can also authenticate its own connections with authenticator extensions such as OIDC, which you should confirm for your sender design. The step is done when a test handshake succeeds with a valid client certificate and fails without one.
Route telemetry by signal type
The flow builder routes flow files by the signal they carry, so logs, metrics and traces each take their own path. Read the attribute the listener sets to identify the signal in the component documentation for your NiFi version, and confirm the attribute name there before writing routing rules. Add an explicit path for unmatched or malformed requests and a failure relationship for each branch so no data ends in an unhandled state. The step is done when a sample of each signal type appears in provenance on its intended branch and no flow file reaches an unconnected relationship.
Connect backends with queue back pressure
Connect each backend processor to its signal branch and set object count and size thresholds on the queues in front of it. These thresholds decide how much data NiFi holds while a backend is slow, and when they are reached NiFi applies back pressure upstream to the listener. Set the values from a test run and the sizing guidance in the NiFi administration guide, not from a default you have not measured. Confirm the retry and failure behavior of each backend processor so that delayed data is retried, not dropped. The step is done when stopping one test backend fills its queue to the threshold, the listener slows intake, and provenance shows the retries once the backend returns.
Verify with a test exporter
Send test logs, metrics and traces from an OpenTelemetry exporter or the Collector over both gRPC and HTTP, using a sender certificate issued for the test. Use the OpenTelemetry Collector configuration reference to write the test exporter so that each signal type is sent separately. Confirm that every signal arrives at its backend with resource and attribute fields intact, and that a request without a valid certificate or sent to the wrong port fails on the sender. The step is done when each signal appears once in its backend, the sender reports successful delivery, and the rejected requests are recorded in the expected error path.
Securing the OTLP listener and what NiFi must protect
An OTLP listener accepts telemetry from every application that can reach it, so the listener, the NiFi UI and any Collector in front of NiFi each need deliberate controls. The cards below cover the decisions that determine who can send data in, who can change the flow, and what happens when a sender cannot authenticate.
TLS on the listener
Decide whether the ListenOTLP endpoints (gRPC and HTTP) will terminate TLS inside NiFi, and which certificate chain senders will trust. Telemetry carries hostnames, service names, log bodies and sometimes user identifiers, so plaintext on a shared network exposes more than most teams expect. Confirm the certificate lifecycle, including renewal and how new certificates reach each cluster node, before go-live.
Client authentication on senders
TLS encrypts the channel but does not prove which workload is sending. Decide whether to require client certificates, bearer tokens or network-level restrictions, and confirm that the chosen mechanism is supported by each sending agent or SDK. Check how a sender with a revoked or expired credential is rejected, and whether that rejection is logged where your security team will see it.
Loopback default when unconfigured
When no security configuration is provided, NiFi binds to 127.0.0.1, so the UI is reachable only through the loopback interface. Teams that move a test flow to a shared host often expose the UI or listener by changing the bind address without enabling security first. Treat any change to the bind address as a security change that goes through review, and verify the bound addresses after each deployment.
NiFi UI access controls
The NiFi UI can change flows, so anyone who reaches it can alter what happens to telemetry. NiFi supports single-user, LDAP, OpenID Connect, SAML 2.0 and X.509 client certificate authentication, and the Kerberos provider is deprecated. Choose a provider your identity team already operates, map groups to policies that separate viewing flows from modifying them, and confirm who may read sensitive parameters.
Collector authenticator extensions
If an OpenTelemetry Collector sits in front of NiFi, its authenticator extensions can check incoming connections on receivers, for example with OIDC, and attach credentials to outgoing exporter requests through the oauth2client extension. This lets the Collector enforce identity before traffic reaches NiFi. Confirm the extension names, their versions and their stability level in each component's README before you rely on them.
Protecting the flow and its secrets
The flow definition, parameter contexts and controller service properties often hold backend credentials and API tokens for SIEM and storage targets. Decide where sensitive values are stored, who can view them, and how they are rotated without editing every node by hand. Review these settings with the same rigor as the listener itself, because a leaked backend token bypasses every inbound control.
Where an OpenTelemetry Collector fits instead
The main question is whether NiFi should be the only OTLP receiver or whether an OpenTelemetry Collector should sit in front of it as the OTLP edge. The Collector is vendor-agnostic and can run as an agent next to services or as a gateway that accepts data from many sources, so it is a natural place to receive OTLP, apply batching and retries, and export to several backends. NiFi is strongest after that point, where it enriches records, parses unstructured logs, filters noise and routes events to SIEM, search and data lake targets with provenance. Before you commit, check the Collector release notes you plan to use: the project publishes signed artifacts, SHA256 checksums and SBOM files with each release, which supports your supply chain review. Confirm those artifacts for the exact version you pick rather than relying on this page.
Be aware that Collector components do not share one maturity level. The OpenTelemetry status page reports mixed stability across core components, and each component README states its own level. Trace SDKs are stable in most languages, but log SDK maturity varies more. For each signal you plan to receive, confirm the receiver, processor and exporter stability in the Collector version you deploy, and check whether authenticator extensions such as OIDC on receivers or the oauth2client extension on exporters cover your access model. If a needed component is still in development, NiFi's ListenOTLP path may carry more of that load for you, but you then own its operational behavior, testing and upgrades.
The choice depends on your team's skills and existing pipelines more than on any general ranking. If your platform team already operates Collectors and your applications export OTLP natively, a Collector edge is usually the less disruptive path, and NiFi can take over at the enrichment and routing stage. If your security or data engineering team already runs NiFi flows for syslog and log normalization, a single NiFi receiver can keep fewer moving parts, provided its listener sizing, authentication and cluster behavior are tested for your telemetry volume. Write down who will patch, configure and troubleshoot each component before deciding, because an unmaintained edge is a bigger risk than either choice.
Limits to confirm before you commit
Receiving OTLP in NiFi depends on four things you must confirm in your environment: the NiFi version, the signals and encodings it accepts, the maturity of the SDKs and components you run, and the upgrade path you will need later. Each card below names one check and what to do with the answer before a design is signed off.
OTLP signal and encoding support
Confirm in the ListenOTLP component documentation for your exact NiFi version which signals (logs, metrics, traces) it accepts and which transport and encoding each signal uses. Check whether gRPC and HTTP endpoints are both available and whether protobuf and JSON payloads are handled the same way. Test each signal with a sample exporter from your own services before you commit to a routing design, because support can differ by signal and version.
Log SDK maturity by language
Trace APIs and SDKs are stable in most languages, but log SDK maturity varies, and several languages are still in development. List the languages your applications use and check the status of logs for each one before you plan to send application logs through OTLP. Where a language lags, plan a fallback such as file tailing or syslog for those services.
Component stability levels
Core OpenTelemetry Collector components carry mixed stability levels, documented in each component's README. If an OpenTelemetry Collector sits in front of NiFi or receives data alongside it, review the stability of each receiver, processor and exporter in your pipeline. Record the stability level in your design so that a later change in a component's status is visible to the team that owns the pipeline.
Throughput under load
Published guidance does not give a throughput figure for ListenOTLP, so measure it yourself. Run a load test with payloads and concurrency that match your peak telemetry rate, and watch back pressure, queue depth, repository disk use and JVM heap on each node. Ask the vendor or your NiFi team for any sizing guidance that applies to your version before you set cluster size.
Authentication on receivers
Decide how clients prove their identity before telemetry reaches a flow. Collector authenticator extensions can check OIDC tokens on receivers, but confirm whether your NiFi version offers an equivalent for ListenOTLP and whether TLS client certificates are required. Unauthenticated endpoints accept data from any network that can reach them, so restrict listener addresses and ports as part of the design.
NiFi upgrade path and registry
NiFi 2 changes the flow configuration file from flow.xml.gz to flow.json.gz, and the migration guidance requires moving to NiFi 1.27.0 before 2.0.0. NiFi Registry is deprecated and planned for removal in NiFi 3.0, so flows versioned through it need a move to Git-based Flow Registry Clients. Plan the ListenOTLP flow with these steps in mind, and confirm each step against the migration guidance for your starting version.
Questions teams ask before building OTLP ingestion in NiFi
What drives the cost of an OTLP receiver built in NiFi?
The cost follows the written scope, mainly how many signal types (logs, metrics, traces) you accept, how many backends you route to, how many environments you need, and how much of the existing NiFi cluster and its security setup must change. Receiving one signal type from one producer group is a much smaller job than receiving all three for dozens of services. Ask any partner, including DeployOpen, for a quote that lists each signal, each destination, each environment and each certificate or authentication integration as a separate line. DeployOpen does not publish a price list, and a quote follows from that written scope.
How long does it take to get a working OTLP receiver in NiFi?
Duration depends on the number of environments, whether a NiFi cluster already exists, how quickly certificates and network paths are approved, and how many backends must receive data before the work is accepted. The fastest path is usually a single non-production endpoint that accepts sample telemetry from one test service, followed by production rollout. Ask for a schedule with separate milestones for endpoint setup, sample validation, routing tests and production cutover, and confirm which of those depend on your change approvals.
What must our team provide before the work starts?
Plan to supply the following before build starts:
- Target environments, including the NiFi cluster or a new node group, host sizing and the network zones where producers and backends sit
- TLS certificates and their issuing authority for the gRPC and HTTP endpoints, plus the trust chain that clients will use
- Credentials and access to each backend (for example a logs platform, a metrics store and a trace store), with agreed test tenants
- Sample telemetry from real producers, covering the attributes, resource fields and volumes that matter to your routing rules
- A named owner for producer changes, because clients must point their exporters at the new endpoint
Confirm these items in writing during scoping.
Can our own engineers build an OTLP receiver in NiFi without a partner?
Yes, if your team already runs NiFi in production and has people who can work with TLS, Java runtime settings, flow design and the OpenTelemetry data model. The hardest parts are usually not the processor configuration but the authentication design, the handling of malformed or unexpected payloads, and proving that routing does not drop signals during backend slowdowns. Assess those skills honestly, then decide whether to build internally, buy a fixed-scope build, or retain a partner for ongoing testing and upgrades.
What is the difference between a fixed-scope engagement and a retainer?
A fixed-scope engagement agrees the scope, deliverables and timeline in writing before work starts, which suits a defined first build such as one OTLP endpoint with two backends. A retainer covers ongoing work such as NiFi upgrades, flow changes as producers evolve, continuous testing and support. Most teams start with a fixed-scope build and move to a retainer once the receiver is running. Ask for the boundaries of each in writing, including who handles NiFi version changes and who approves changes to routing rules.
Should we use ListenOTLP in NiFi or an OpenTelemetry Collector?
Use NiFi when telemetry must join flows that already move other data, such as log normalization or enrichment in the same cluster, and when your team already operates NiFi well. Use an OpenTelemetry Collector when the main job is receiving, batching and exporting OTLP for many services with a broad set of receivers and exporters. The OpenTelemetry Collector is a vendor-agnostic way to receive, process and export telemetry and runs as an agent or gateway. Many organizations run the Collector close to producers and send selected data into NiFi for downstream processing, so the decision is often about placement rather than either-or. Confirm the current component stability in each Collector README before you commit.
What operational risks should we check before going live?
Check how the receiver behaves when a backend slows down, when a producer sends malformed or oversized payloads, when certificates expire, and when a NiFi node leaves the cluster. Confirm that back pressure reaches the producers in a way you accept, that dropped data is recorded, and that you can see throughput and errors per signal type. Also confirm that authentication is enforced on every endpoint, because the OpenTelemetry Collector supports authenticator extensions for incoming connections, and NiFi access controls should be reviewed separately. Turn these checks into test cases with pass criteria before production cutover.
How DeployOpen scopes and delivers OTLP ingestion for NiFi
DeployOpen scopes OTLP ingestion as a fixed-scope engagement with a written statement of which telemetry signals, transports and backends are in and out before any flow is built. For a ListenOTLP build, that scope usually names the signals to receive (logs, metrics, traces or a subset), whether gRPC, HTTP or both are exposed, the sizing assumptions for each listener, the backends each signal is routed to, and the authentication model for senders. It also states which NiFi version and Java runtime the work targets, since the migration from NiFi 1.x to 2.x changes flow configuration files and some component names. Deliverables are agreed in the same document, typically the flow definitions, the listener and TLS configuration, routing rules, runbook notes and test evidence. Ask for each deliverable to be listed by name so acceptance can be checked against it.
Testing happens against the client's own backends, not against a generic stand-in. DeployOpen sends representative OTLP traffic from sender tools or existing applications into a non-production NiFi, then confirms that each signal reaches each destination with the expected attributes, that malformed or unsupported payloads are handled as designed, and that back pressure behaves correctly when one backend slows. The client supplies the test endpoints, sample data that matches production shape, and the people who can confirm what a correct record looks like in their SIEM, observability or storage platform. Where an OpenTelemetry Collector is part of the design, its configuration is tested in the same run. DeployOpen implements everything inside your cloud accounts, data centers or restricted networks. It prepares systems, configuration and evidence for your assessors, but it does not issue audit reports, attestations or certifications.
Ongoing work is handled separately. Upgrades to newer NiFi 2.x releases, changes to sender fleets, new backends and tuning after real traffic arrives are better priced as a retainer than as an open-ended extension of the original build. Ask any provider to state in writing what the retainer covers, how release changes to ListenOTLP and the OpenTelemetry components are tested, and how incidents are reported. Ask for a quote that separates one-time build effort from recurring support, and confirm who holds the NiFi, certificate and backend credentials after handover.
