Route one log stream to several SIEM targets with NiFi

Routing one log stream to several SIEM and storage targets with NiFi: when a single pipe is not enough

Fan-out in NiFi is a design decision, not a processor setting. When one log source must feed a SIEM for detection, a search cluster for investigation and an archive for retention, the outcome depends on how the flow separates those needs: which events go where, in what format, and what happens when one destination slows down. Teams usually need this pattern once a single forwarder can no longer serve all three audiences, typically because the SIEM is priced or tuned for a subset of events, the search cluster needs full fidelity for investigators, and the archive must keep every record unchanged for retention. In NiFi, a processor exposes named relationships such as success, failure and matched or unmatched. Connecting several downstream paths to the same relationship copies the flowfile to each one, so one ingest path can drive several targets while each branch applies its own filter, record writer and delivery settings.

The decisions that matter most sit upstream of the processors. Decide the canonical event schema before the first branch, because each destination will otherwise drift toward its own field names. Decide which target is authoritative, since the archive is usually the copy you must never lose, and let the SIEM branch be the one allowed to drop or sample events under pressure. Give each branch its own connection queue with explicit back pressure thresholds, so a slow SIEM fills its queue and throttles the upstream listener without stalling the archive. Provenance records each send and each failure per branch, which lets you prove later which events reached which target. Confirm in your own test environment how your NiFi version behaves when one connection is full and how your listener protocol responds, because the answer decides whether you can keep delivery guarantees without losing syslog or other UDP traffic.

Before go-live, test the failure paths rather than the happy path. Stop each destination in turn, replay a known set of events, and check that the archive received everything, the search cluster caught up and the SIEM branch behaved as the rules say it should. Review the Apache NiFi System Administrator's Guide for the repository and back pressure settings that apply to your version before sizing queues. Ask your implementation partner to document each branch's contract in writing: its filter logic, output format, acceptable loss and recovery steps. That document is what your detection, operations and compliance teams will need when a target misbehaves at 2 a.m.

Choosing how to split the stream per destination

A single flow receives events, classifies them with RouteOnAttribute or QueryRecord, and sends each branch to its own put processor. This is the smallest design to build and read. All destinations share one process group, one set of back pressure settings and one provenance trail, so a slow target can hold the whole flow when its connection fills.

Ordering: Events stay in arrival order inside each branch, but branches drain at different speeds, so the same event can appear in the SIEM before or after the search cluster.
Duplicates: Retries from one destination can resend events, and a branch that fails after a partial send may create duplicates unless the target supports idempotent keys.
Effort to change one target: Low for filter changes, but editing a shared route touches every branch, so regression testing covers all destinations.
Back pressure: A full connection on one branch can stop the upstream queue, which delays the other destinations as well.
Scale: Works well for a few targets with similar speed and reliability.

Confirm with your SIEM and search cluster owners how long each can be unavailable before events must be buffered or dropped, because that limit decides whether a shared connection is acceptable.

How to build the fan-out flow in NiFi 2

This sequence takes one log stream from intake through per-target formatting, filtering and delivery, so each SIEM and storage target receives the events it should with known back-pressure and failure behavior. Each step ends with a check you can record before moving on.

Define the event schema and per-target formats
Before placing any processor, agree in writing on the normalized event schema, the field names detection teams rely on, and the output format each target needs, such as JSON for the search cluster, CEF or the SIEM's expected format for the SIEM, and compressed, partitioned files for long-term storage. Record which events go to which target and which fields may be dropped for each. Confirm these decisions with the SIEM owner, the search cluster owner and the records or compliance owner who sets retention. The step is done when every target has a signed-off schema and a list of required fields and filters.
Set up the intake with TLS
Create the listener or consumer that receives the stream, such as a ListenTCP, ListenHTTP or a Kafka consumer, and enable TLS on it with certificates issued by your own CA. Restrict who can send to the endpoint, and confirm the listener binds to the interface your collectors can reach. NiFi binds its own UI to 127.0.0.1 when no security configuration is provided, so configure HTTPS and authentication for the UI before exposing any cluster-facing port. Send a test event from a known source and confirm it appears in the intake queue with a provenance record. The step is done when a test message from each sending source is visible in provenance.
Add routing with filters
Parse the incoming records with a record reader that matches the schema from step one, then use RouteOnAttribute or QueryRecord to split the stream into one relationship per target, with each relationship carrying only the events and fields that target needs. Keep a separate catch-all relationship for events that match no filter so nothing disappears silently. Write the filter logic as named, documented expressions so reviewers can read them. The step is done when a sample file of mixed events produces the expected split across relationships.
Configure each target's delivery processor
For each destination, add the delivery processor that matches its interface, such as a record writer that produces the agreed format feeding a search index processor, a SIEM forwarder, or an object storage writer for long-term retention. Use credentials from a parameter context or a credential store and not in flow definitions. Confirm the processor's success and failure relationships are both connected. Check the processor documentation for the exact property names and the delivery behavior of the version you run. The step is done when each target processor accepts a test batch and returns it to a success relationship.
Set queue and back-pressure thresholds per target
Set the connection queue limits on each route to the delivery processor, separately for each target, so a slow SIEM does not hold up the search cluster or the archive. Decide the object count and data size thresholds for each connection with the target owner, and document what happens when the queue fills, which is that upstream processors slow down and intake may back up. Account for the fact that a shared upstream queue can slow every target, so decide whether the archive route should be isolated. Verify back pressure by stopping one target in a test environment and watching its connection queue reach its threshold while the other routes keep moving. The step is done when that test produces the expected queue behavior and the team has recorded the thresholds.
Enable error and dead-letter relationships
Connect the failure and retry relationships of every delivery processor to a dead-letter path, which can be a separate connection to a storage location or a queue reviewed by operators. Route records that fail parsing to the same path with the original content and the error attribute retained. Set the retry behavior deliberately, since retries can duplicate events at a target that does not deduplicate, and make sure your dead-letter path has its own retention and access controls. The step is done when a deliberately malformed event and a rejected delivery both land in the dead-letter path with the error recorded.
Verify delivery with test events and provenance
Send a known set of test events, including valid, filtered, malformed and oversized records, and compare the count received by each target with the count expected from your filter rules. Use provenance to trace a sample event from intake to each target and confirm the routing decision and the delivery event for each. Confirm the counts match and that every event is accounted for by a delivery, a filter decision or a dead-letter record. Repeat the test after any change to filters or queue settings. The step is done when the per-target counts reconcile with the expected totals and the reconciliation is saved as evidence for the change record.

Keeping delivery guarantees when targets disagree

When one log stream fans out to a SIEM, a search cluster and long-term storage, each target fails and recovers on its own schedule. The cards below cover how NiFi's queues and provenance give at-least-once delivery, where duplicates come from, how target-side keys absorb them, and how a slow destination pushes pressure back to intake.

At-least-once from NiFi queues

NiFi keeps each FlowFile in a connection queue backed by the content and FlowFile repositories until the next processor or destination confirms the work. A FlowFile is removed only after a successful transfer, so a node restart or failed send leaves it queued for retry. Check that the repositories sit on durable disks and that your fan-out uses the same FlowFile for each branch so one failure does not drop the other copies.

Where duplicates come from

Duplicates appear when a destination accepts a record but the acknowledgment is lost or times out, so NiFi retries a record that was already stored. Retries after a cluster node failure or a rerun of a bulk load produce the same effect. Expect duplicates at the boundary of every send, and measure them by comparing event counts per destination against the source during a controlled failover test.

Stable event IDs for idempotent writes

Derive a deterministic document ID from the source fields, for example a hash of the host, timestamp, and raw message, and set it on every send. Search indexes that accept an ID overwrite rather than duplicate, and SIEM ingestion may offer its own deduplication keys. Confirm with each vendor whether a repeated ID is upserted, dropped or counted against license volume, since that behavior differs by product.

Per-target duplicate handling

Storage that cannot take a stable ID needs a different answer, such as writing files with a batch ID and an event ID so a later compaction job removes repeats. Object storage or a lakehouse table may suit a deduplication step at query time. Decide per destination which duplicates are tolerable for long-term records and which must be removed before analysts or detections see them.

Back pressure from a slow target

NiFi connections have object-count and data-size thresholds. When a destination slows, its queue fills, and once the threshold is reached the upstream processor stops being scheduled, which propagates to the listener and eventually the sender. Set thresholds per branch so one slow SIEM does not hold back the search cluster, and decide whether intake should block, drop or spill to disk for each source.

Testing guarantees under failure

Test each target by stopping it during a replay of known input, then compare the counts at each destination with the source. Watch queue depth, provenance for failed sends, and the time the intake listener stays blocked. Ask the team to document the expected loss window for every source type, because a syslog UDP listener cannot apply back pressure the way a TCP or HTTPS listener can.

Configuration options and when each fits

This table compares the NiFi settings that decide how one log stream behaves when it fans out to a SIEM, a search cluster and archive storage. Read each row as a decision: the option, what it controls, when it fits and what it costs you in delivery or operations. The main takeaway is that back pressure and archive ordering matter more than raw throughput once one target slows down.

Option	What it controls	When it fits	Trade-off
Back-pressure thresholds on connections	Object count and data size allowed to queue before upstream processors pause	Use when a SIEM or search target slows and you need upstream intake to slow instead of dropping events	Large thresholds hide slowdowns and use more disk in the content repository; small ones pause intake sooner
Prioritizers on connections	Order in which queued FlowFiles leave a connection	Use when security alerts must reach the SIEM ahead of bulk archive traffic	Priority ordering can starve low-priority data during long backlogs, so confirm archive lag is acceptable
Load-balanced connections in a cluster	How FlowFiles spread across cluster nodes before delivery	Use when one node cannot keep up with a high-volume fan-out and events do not need node affinity	Balancing across nodes can change per-source ordering and adds network transfer between nodes
Per-target record readers and writers	Format and schema applied separately for each destination	Use when the SIEM needs CEF or a normalized schema, the search cluster needs JSON and the archive needs compressed Parquet or Avro	Each extra format is another schema to version and test, and conversion errors need their own routing
Per-target filters and queries	Which records each destination receives	Use when the SIEM receives only detection-relevant events and storage receives everything	Filtering before the archive branch makes the archive incomplete; the choice must match your retention policy
Retry and penalty settings on delivery processors	How many attempts a failed delivery gets and how long a FlowFile waits before retry	Use for transient network or authentication failures on a target that recovers on its own	Long penalties delay delivery to healthy targets only if they share a path; excessive retries amplify load on a struggling target
Archive copy before or after filtering	Whether long-term storage receives the full-fidelity stream or only the filtered subset	Write the archive before filtering when retention or evidence requirements call for every raw event	Archive before filtering stores more data and cost; archive after filtering is cheaper but cannot restore dropped events
Failure and retry relationships per branch	Whether a failed branch blocks the other branches	Use separate failure relationships so the archive keeps writing while the SIEM branch is paused	Independent branches need explicit monitoring, because a silent branch failure is easy to miss

How back pressure from one slow target reaches the source

A slow destination slows every destination that shares its path, unless the flow is built to separate them. In NiFi, processors pass FlowFiles through connections, and each connection has back pressure thresholds set by object count and data size. When a PutSplunk, PutElasticsearchRecord or archive writer falls behind, its inbound connection fills. Once a threshold is reached, the upstream processor is no longer scheduled to run, so the processor feeding that connection stops. If the splitter or router that fans out the stream is upstream of that connection, it stops too, and the ingest processor behind it eventually stops reading from the listener or the queue. A single shared path therefore lets one target stall the whole stream.

The usual fix is to give each target its own connection and, where a target's failure should not hold up others, its own branch with its own queue and retry handling. The source then sees back pressure only on the branch that is actually full. Confirm how your listener behaves when NiFi stops accepting input (for example, whether a TCP sender is held, a UDP listener drops packets, or an upstream broker buffers) because that determines whether a slow SIEM causes data loss at the edge. Test this with a deliberately throttled target in a non-production cluster and record the queue depths and provenance events you see.

The design has to decide, per stream, which destination is allowed to block. A common choice is that the SIEM may hold the source, because detection gaps are the higher risk, while the long-term archive is written asynchronously and allowed to fall behind. The opposite choice may suit a compliance archive that must never miss a record. Write that decision down with the owners of each target, set queue limits that match the time you can tolerate a stall, and set alerts on queue depth so the stall is seen before the disk fills. Confirm the exact threshold and retry settings in the component documentation for your NiFi version rather than assuming defaults.

Security and operational considerations for multi-target routing

A flow that fans one log stream out to several SIEM and storage targets concentrates credentials, network paths and change rights in one place. The cards below cover the controls that keep each target authenticated and encrypted, keep secrets out of flow definitions, limit who can alter routing, preserve an audit trail and show when a target is falling behind.

TLS and authentication per target

Each destination has its own endpoint, certificate chain and credential, so configure transport security on every outbound processor rather than assuming one setting covers all targets. Confirm which TLS protocol versions and trust stores each SIEM, search cluster and object store accepts, and whether mutual TLS is required. Record the certificate expiry date for each target in your change calendar, because a single expired certificate can stop one branch of the fan-out while the others keep flowing.

Credentials in sensitive parameters

Store API keys, passwords and tokens for each target as sensitive parameters or in a secrets-backed parameter provider, and reference them from processors instead of typing them into properties. Sensitive values are encrypted in the flow definition and masked in the UI. Check that exported flow files and Git-versioned copies contain only parameter references, and that the encryption key used for sensitive values is stored and backed up separately from the flow.

Access policies for routing changes

Changing a RouteOnAttribute rule or a destination URL can silently redirect security data, so limit edit rights on process groups that hold routing to a small named group. Separate roles for viewing, operating and modifying components, and require changes to go through a Git-based flow registry with review. Ask your team to document who may start, stop and reconfigure each target path, and test that a user without modify rights cannot change a destination.

Provenance retention for audit

Provenance records show which events were sent to which target, when, and whether delivery succeeded, which is the evidence auditors and incident responders ask for. Set the provenance repository size and retention period to match your audit requirement, and confirm whether those records are sufficient on their own or must be exported to a log store. Decide who can query provenance, because event contents can include sensitive fields even when the flow masks them.

Queue depth and delivery error monitoring

When one target slows, its connection queue fills, back pressure reaches the upstream processors, and other targets can be starved if they share a path. Alert on queue depth and count of failure relationships per destination, not only on the flow as a whole. Monitor connection queues per target, and confirm how your team will see a stalled target before the input queue reaches its limit.

Questions teams ask before routing logs to several targets

What drives the cost of routing one log stream to several SIEM and storage targets?

The cost comes from the number of destinations, the number of distinct formats and filters, and how much of the flow must be tested against each target. Each destination adds its own format mapping, field normalization, authentication setup, and error handling, so the effort rises with every target. One-time costs cover the design, the flow build, target connection setup, and testing against real volumes. Recurring costs cover running the platform, patching NiFi, updating mappings when a SIEM changes its schema, and the license or storage charges that the targets themselves bill. Ask any partner for a quote that separates these items and lists which targets, formats and filters are in scope.

How long does a multi-destination routing project take?

Duration depends mainly on how many destinations exist, how well their ingest formats are documented, and how quickly your team can grant access and approve test data. Teams with one SIEM and one storage target usually finish faster than teams that also need a search cluster with separate index rules and a retention policy for archive copies. Delays most often come from waiting for target credentials, firewall changes, and sign-off on field mappings, not from the NiFi flow itself. Ask for a written timeline that names the dependencies on your side, such as network access and a test window on each target.

What does our team need to provide before the build starts?

The build needs access to each destination in a test environment, sample logs for every source type, and a written list of what each target must receive. Specifically, you should prepare:

Test credentials or service accounts for the SIEM, the search cluster and the long-term storage bucket or table.
Representative sample logs covering normal traffic, rare event types, and malformed lines.
The current field schema or ingest format each target expects, and any content your detection or retention teams rely on.
A named owner for each destination who can approve mappings and test results.

Can our internal team build multi-destination log routing alone?

A team with NiFi operations experience and a clear understanding of its log sources can build it internally, provided it has time to test each destination and maintain the flow afterward. The skills needed are NiFi flow design, record parsing, error routing, cluster operations, and enough knowledge of each target's ingest rules. Internal teams often underestimate the testing load and the work of keeping mappings current as targets change. A partner is most useful when you need a fixed scope, a second set of hands for testing, or documented handover. Confirm your team's skills and capacity against the list of targets before deciding.

How are duplicate events handled when the same log goes to several targets?

Duplicates are expected when one stream fans out, so the design should decide per target whether a repeated event is acceptable. NiFi can route a single event to several relationships, and each destination should use its own idempotency approach where the target supports one, such as stable document identifiers in the search cluster. For SIEM ingest, the usual control is deduplication before fan-out, while the archive copy may keep every event for audit. Confirm with each target owner whether they expect deduplicated input and how they will detect repeats. Test with replayed data to verify the result.

What happens to the other destinations when one target goes down or slows?

NiFi uses back pressure on connections, so a slow destination can cause its queue to fill and eventually slow the upstream processors that feed it. The design choice is whether a slow target should hold up the whole flow or whether its events should be queued, rerouted or failed separately. Separate connections and queues per destination, with explicit failure paths, keep one outage from blocking the others, but they need disk capacity and monitoring. Confirm queue thresholds, retry policy, and where failed events are stored and replayed. Test a deliberate outage on each target during the build.

Does the routing design affect our security and compliance evidence?

Yes. Each destination receives data that may have different access rules, retention periods and encryption requirements, so the routing design must reflect those controls. Teams should document which fields go to which target, who can change the routes, how credentials are stored, and how delivery failures are recorded for audit. NiFi provenance can support this evidence, but retention and access to provenance data need their own decisions. This page does not issue audit reports or certifications. Share the design with your auditors early, and ask them which records they expect to see.

Which NiFi version should we use for a new routing flow?

NiFi 1.x can no longer be maintained because of its dependency versions, and the project's guidance is to use NiFi 2 for new work. The NiFi downloads page states that NiFi 1.28 is the last minor release of the version 1 series, and that 1.28.1 reached end of support on December 8, 2024. A new routing flow should therefore be built on NiFi 2, and any existing 1.x flows should have a planned migration path. Confirm the current 2.x release and the Java runtime your environment supports before the build begins.

How DeployOpen scopes and delivers this routing work

DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, including NiFi flows that fan one log stream out to several SIEM and storage targets. Engagements are fixed-scope, with the destinations, filters, formats and delivery behavior agreed in writing before the build starts.

How we scope

Inventory every destination, its ingest format, its authentication method and its tolerance for delay or duplicate events.
Record which events go to which target, the filter and format rules per destination, and the retention expected in each store.
Agree in writing how back pressure, failed deliveries and replay from provenance will be handled before any flow is built.

How we build

Build the routing flow in NiFi 2 with one ingest path, per-destination filtering and format conversion, and separate failure relationships for each target.
Test back pressure by slowing one target on purpose and confirming that the other destinations keep receiving events within agreed limits.
Store flow definitions in a Git-based Flow Registry Client so changes to routing rules are reviewed and traceable.

How we hand over

Deliver a written runbook covering each destination, queue thresholds, replay steps and the owner for every SIEM and storage endpoint.
Document the delivery guarantees actually achieved, including where duplicates are possible and how downstream deduplication is expected to work.
Walk your operations and detection teams through the flow, the provenance views and the test results before sign-off.

How we support

Ongoing support, NiFi upgrades and continuous testing are available as retainers after handover.
A retainer can cover monitoring of queue depth, failed deliveries and destination health, with changes handled through the agreed scope.
Confirm with your vendors the current supported ingest formats and authentication options for each SIEM and storage target before scoping.

Routing one log stream to several SIEM and storage targets with NiFi