Reducing SIEM ingest volume with Apache NiFi
Cutting licensed SIEM ingest by filtering, sampling, deduplicating and summarizing logs in NiFi before they reach Splunk or another SIEM, while archiving full-fidelity copies to cheaper storage.
Apache NiFi for security and log data pipelines
Using NiFi to collect, parse, enrich, filter and route logs and telemetry to SIEM, observability and data lake platforms.
Collecting syslog at scale with Apache NiFi
Receiving syslog from network and server fleets with ListenSyslog: TCP versus UDP, RFC 5424 and 3164 parsing, invalid message handling, load balancing listeners across a cluster and TLS.
Parsing CEF and unstructured logs in NiFi with Grok
Turning vendor logs into structured records with CEFReader, GrokReader and ExtractGrok: pattern libraries, handling lines that do not match, schema choices and normalizing fields for detection teams.
Receiving OpenTelemetry data in Apache NiFi with ListenOTLP
Using NiFi as an OTLP receiver for logs, metrics and traces: gRPC and HTTP endpoints, protocol detection, routing telemetry to several backends and where an OpenTelemetry Collector fits instead.
GeoIP enrichment of security logs in Apache NiFi
Adding location and ISP data to IP fields with GeoEnrichIPRecord and ISPEnrichIP: MaxMind database updates, record path mapping and enrichment cost at high event rates.
Routing one log stream to several SIEM and storage targets with NiFi
Sending the same events to a SIEM, a search cluster and long-term storage with different filters and formats per destination, back pressure when one target slows and keeping delivery guarantees.
Building a security data lake with NiFi and Apache Iceberg
Landing normalized security events in Iceberg tables on object storage with PutIcebergRecord: partitioning, file sizes, catalog choice and querying with Trino alongside the SIEM.
Collecting logs at remote sites with MiNiFi and NiFi
Using MiNiFi agents to tail and forward logs from branch offices, plants and isolated networks into a central NiFi over Site-to-Site, with buffering, compression and provenance from the edge.
