Why teams build a security data lake with NiFi and Apache Iceberg
Teams build a security data lake when SIEM licensing or retention limits force a choice between cost and history. Ingesting every event into the SIEM at full fidelity can become expensive, yet hunters and auditors often need months or years of raw security events that were never meant to sit in a hot search tier. A common answer is to keep the SIEM for detection and active investigation, and to land a complete, normalized copy of the events in open table storage that is cheaper to keep. Before committing, confirm with your SIEM vendor what your license actually meters, how long your current retention runs, and which fields your detection teams rely on, because those answers decide how much of the stream belongs in the lake and how much stays in the SIEM.
In this design, NiFi normalizes events into a record schema and writes them through the PutIcebergRecord processor into Apache Iceberg tables on object storage. Trino then queries those tables with SQL, so analysts can join historical events with current SIEM results in one investigation workflow. Because Iceberg tables are open and multiple engines can read them at once, the same data can serve hunting, audit extracts and other analytics without a second copy. The catalog you choose, such as a REST catalog, Hive Metastore or AWS Glue, becomes part of your control plane, so confirm which catalog your Trino and NiFi versions support before you commit to it.
Iceberg adds several features that matter for security data. Schema evolution lets you add fields as vendor log formats change without rewriting existing tables, hidden partitioning lets queries filter on event time without users knowing the physical layout, and snapshots allow time travel to the table state at a past point and rollback after a bad write. These features do not make the lake an audit record on their own. Iceberg does not validate that events are complete, does not prevent duplicate writes from a retried flow, and does not control who can read sensitive fields. Those controls sit in NiFi flow design, catalog and storage permissions, and Trino access control, and they need their own testing.
Choosing an Iceberg catalog for security data
The Hive Metastore is the catalog most platform teams already run when they have Hadoop or older Hive-based lakes. Trino's Iceberg connector can use it as the catalog for Iceberg tables, so a team that already operates the metastore and its database often extends the same service to security tables rather than introducing a new component. The operational question is whether that metastore is supported, backed up and reachable from both the NiFi write path and the Trino coordinator and workers.
- Operational fit: Works best where the metastore is already a maintained shared service with an owner and a recovery procedure.
- Access path: Confirm network routes from NiFi writers, Trino and any other query engines to the metastore endpoint.
- Backing database: Plan backups and restore tests for the database behind the metastore, since table pointers depend on it.
- Engine coverage: Confirm which other engines in your environment read Iceberg tables through this metastore.
- Selection: Set
iceberg.catalog.typeto the Hive Metastore value that matches your Trino version.
Before committing, ask your platform team to confirm the metastore version and its supported Iceberg behavior in your environment.
AWS Glue is the managed catalog option for teams whose security data lake sits in Amazon S3 and whose analytics runs on AWS services. Trino's Iceberg connector supports Glue as a catalog type, which removes the need to operate a metastore service yourself. Authorization, auditing and cross-account access then follow your AWS account structure, which is often the deciding factor for regulated teams.
- Managed service: Less catalog infrastructure to run, patch and back up, with the provider operating the service.
- Identity model: Table access is governed by AWS IAM policies, so the security team must map who writes and who reads.
- Account boundaries: Plan how log archive accounts, security tooling accounts and analytics accounts share tables.
- Ecosystem: Athena also reads Iceberg tables with time travel, which may matter if analysts use AWS-native tools alongside Trino.
- Cost: Catalog request and API charges recur with usage, so ask the provider for current pricing for your volume.
Confirm with your AWS team that the Glue catalog, the IAM roles used by NiFi and Trino, and the S3 bucket policies all match your data classification rules.
REST catalogs implement the Iceberg REST catalog API, which lets several engines share one catalog service without each one holding metastore credentials. Nessie adds Git-like branches and commits to catalog state, so changes to security tables can be staged and reviewed before they become visible to queries. Trino's Iceberg connector supports both as catalog types, and these options suit teams that run their own catalog service on Kubernetes or in a data center they control.
- Shared endpoint: A REST catalog gives NiFi, Trino and other engines one interface, which simplifies credential rotation.
- Branching with Nessie: Staging a schema or partitioning change on a branch can support review before production tables change.
- Self-operated risk: You own availability, upgrades and backups of the catalog service, so plan it as a tier-one dependency.
- Authentication: Confirm how the catalog endpoint authenticates clients and how that identity maps to your access policies.
- Compatibility: Check that the NiFi writer version you run supports the REST catalog path you choose.
Ask the catalog vendor or project to confirm the Iceberg format versions and Trino connector behavior they support in your target release.
A JDBC catalog stores Iceberg pointers in a relational database that you already operate, which can be a practical choice when a managed database with backups and access controls is available. Snowflake is a different model: Snowflake Iceberg tables can be read by external engines through the Horizon Iceberg REST Catalog API, and Trino's Snowflake connector can query and create tables in an external Snowflake account. Choose this path when Snowflake is already the governed analytics platform and the SIEM team wants one place for access policy.
- JDBC database: The catalog database becomes a critical dependency, so test restore and locking behavior under concurrent writes.
- Snowflake-managed tables: Snowflake stores Parquet data in Iceberg format on Snowflake-managed storage or an external volume, which changes where files live.
- Read path: Trino reads Snowflake-managed Iceberg tables through the Horizon REST API, so confirm that endpoint and its credentials.
- Connector limits: The Trino Snowflake connector accesses one Snowflake database per catalog, which shapes how you split security data.
- Cost and licensing: Snowflake compute and storage charges recur with usage, so request a quote based on your ingest and query profile.
Confirm with your database and Snowflake administrators which option matches your data residency and access review process.
Air-gapped and tightly segmented networks change the catalog decision because managed cloud catalogs may be unavailable or not approved. In these environments the catalog usually runs on infrastructure you control, such as a self-hosted REST catalog, a Hive Metastore or a JDBC database inside the enclave. The same catalog must serve NiFi, Trino and any offline analysts, and every component needs local TLS trust and credential management.
- Connectivity: Plan every catalog and object storage endpoint inside the enclave, with no dependency on external services.
- Certificates: Decide how the internal certificate authority issues and rotates client and server certificates for catalog connections.
- Recovery: Keep a tested copy of catalog state and metadata in the same boundary as the data it describes.
- Change control: Route catalog schema and partition changes through your approval process, since branch-based review may not be available.
- Version alignment: Pin Trino, the catalog service and the NiFi writer to versions you have tested together.
Before finalizing the design, ask the team responsible for the enclave to confirm which catalog types are approved and how they will be patched.
Partitioning, file size and catalog options and when each fits
This table compares the main table design choices for a security data lake built by NiFi into Iceberg, with notes on high-volume firewall logs and low-volume identity events. Read each row as a decision: the option, its trade-off and the kind of event stream it suits. The main takeaway is that partitioning and file size should follow query patterns and ingest rate, not a default setting.
| Choice | Option | Trade-off | When it fits |
|---|---|---|---|
| Partition layout | Event time, day or hour | Keeps time-bounded searches cheap and matches how detection teams query, but a single hot partition can hold many small files during bursts | High-volume firewall logs, where most investigations filter by recent time windows |
| Partition layout | Source and time combined | Narrows scans to one device or tenant, but multiplies partition count and small files when many sources send little data | Identity events from a few directory or SSO sources, where analysts usually pivot by source |
| Hidden partitioning | Partition transform on a timestamp column | Queries filter on the event timestamp without knowing the partition scheme, and the layout can change later, though engines must support it | Tables queried from Trino and other engines by analysts who should not need partition column names |
| Explicit partition columns | Separate date or source columns | Simple for every engine to read, but every query must include the partition columns or it scans more data | Tables read by tools that do not yet handle hidden partitioning, confirmed in your engine versions |
| Target file size | Larger files from fewer, bigger commits | Fewer files reduce planning and scan overhead, but data arrives later and memory use in NiFi grows | Steady firewall volume where latency to the lake can be measured in minutes or longer |
| Target file size | Smaller files from frequent commits | Data is visible sooner, but many small files slow queries and inflate metadata | Low-volume identity events that still need near-current lake data, usually with compaction scheduled |
| Compaction cadence | Scheduled rewrite of small files | Restores scan performance and adds compute and storage churn that must be scheduled outside peak ingest | Any table with frequent small commits, with the schedule tuned against your own file counts |
| File format | Parquet (Trino default) | Columnar and well supported by Trino and most readers, so it is the usual starting point | Most security event tables queried for investigation and hunting |
| File format | ORC or Avro | Supported by Trino's Iceberg connector, but Parquet is the default and gets the most routine testing in common pipelines | Only when an existing reader or team standard requires it, confirmed for each engine |
| Catalog type | REST, Hive Metastore, AWS Glue, JDBC, Nessie or Snowflake | Selected with iceberg.catalog.type in Trino; each needs its own availability, access and backup plan | Choose by where other engines already read metadata, then confirm NiFi's PutIcebergRecord catalog support |
Setting up the Iceberg landing path in NiFi
This sequence takes a NiFi 2 cluster from a verified runtime to a working path that lands normalized security events in an Iceberg table on object storage, then confirms the table is readable from Trino and that earlier snapshots can be queried. Each step ends with a check so problems surface before production traffic reaches the table.
Confirm the NiFi 2 and Java 21 runtime
Confirm that the NiFi nodes run NiFi 2 on a Java 21 runtime, the minimum the NiFi administration guide lists, and that any optional Python-based processors you plan to use run on a supported Python version. Your platform team checks the installed versions on every node, not only the first one, and confirms that the cluster starts cleanly with its chosen security configuration. The step is done when every node reports the same NiFi version and Java version and the cluster shows all nodes connected in the UI. Because the Iceberg processors depend on the runtime, a mismatch found here is far cheaper to fix than one found during a test flow.
Provision object storage and the catalog
Create the bucket or container that will hold Iceberg data and metadata files, and stand up the catalog service you selected, such as a REST catalog, AWS Glue, Hive Metastore or a JDBC-backed catalog. Your storage and data platform owners set access so that NiFi can write, and Trino can read, under the same catalog namespace. Ask your vendor or storage team to confirm which catalog types your chosen Trino release and NiFi processor support in your environment, because the catalog is shared by every engine. The step is done when a test client can create and list a namespace in the catalog and write a small object to the bucket with the credentials NiFi will use.
Define the normalized record schema
Write the schema for normalized security events as an Avro or equivalent record definition, covering timestamp, source, event category, severity, actor, host, IP fields and the raw message reference. Configure a record reader, such as the reader that matches your input format, so that NiFi parses incoming events into this schema before any write. Security engineers and data owners agree on field names, types and nullable fields, since changing them later means schema evolution work on the table. The step is done when a sample file from each source parses through the reader with no failures and the output records match the agreed fields.
Create the Iceberg table with a partition spec
Create the Iceberg table from the agreed schema and define its partition spec, typically on a timestamp transform such as day plus a low-cardinality source field, chosen by the query patterns your analysts and detection teams run. Iceberg's hidden partitioning means queries do not need to filter on a separate partition column, but the spec still determines file layout and pruning. Confirm the table format version against what Trino's Iceberg connector supports in your release. The step is done when the table appears in the catalog with the expected schema and partition spec, and a metadata query shows no data files yet.
Configure PutIcebergRecord and its controller services
Add PutIcebergRecord to the flow and configure its controller services: the record reader from the schema step, the catalog service that matches the catalog you provisioned, and the storage credentials for the bucket. Set the target namespace and table name, and review the file size and commit behavior options against your expected event volume, since many small files slow queries and increase metadata work. Keep the failure relationship routed to a dead-letter path with provenance enabled. The step is done when the processor validates with no errors and the controller services show enabled in the UI.
Run a test flow with known events
Feed a controlled set of sample events, including a few deliberately malformed lines, through the flow into the Iceberg table. Use a fixed count of valid records so the expected row count is known in advance. Check the NiFi bulletin board and the provenance view for each record to confirm that valid events reached PutIcebergRecord and that malformed events went to the failure path. The step is done when the table row count matches the valid records you sent, no records are lost between the source and the table, and the failure queue holds only the malformed events.
Verify with a Trino query
Point a Trino Iceberg catalog at the same catalog service and query the table from Trino, checking the row count, the schema and a filter that exercises the partition spec. Your data platform team confirms that the Trino catalog properties, including the iceberg.catalog.type setting, match what NiFi uses. Run the same query used by analysts alongside the SIEM to confirm the results agree with the test data. The step is done when Trino returns the expected count and fields, and the filtered query reads only the partitions you expected.
Confirm time travel to an earlier snapshot
Run a time-travel query in Trino against a snapshot taken before the test load, then against the snapshot after it, and confirm the row counts differ as expected. Record the snapshot identifiers in the runbook so responders know how to reach a known state after a bad load. Ask your platform team to confirm the snapshot retention and expiry settings, since expired snapshots cannot be queried or rolled back to. The step is done when both historical queries return the counts you predicted, and the runbook lists the snapshot IDs and retention policy.
How partitioning and file sizes decide query cost
Many small files and poor partitioning are the most common reasons a security data lake gets slow and expensive to query. Trino has to plan and open every data file that a query touches, and Iceberg keeps a manifest entry for each one. When NiFi commits a steady trickle of tiny Parquet files, each query pays for more file listings, more manifest reads and more row-group opens, even when the actual event volume is modest. Partitioning shapes how much of that work a query can skip. Iceberg's hidden partitioning lets you partition on a derived value such as the event day or a timestamp bucket without exposing the transform in every query, but the partition spec still decides whether a search on one tenant or one source type reads a few files or the whole table.
The write path in NiFi controls file size. PutIcebergRecord commits a snapshot each time it finishes a batch, so the batching settings you choose (how many records or how much time a flow file accumulates before it is written) set how often files land. Larger batches produce bigger files and fewer snapshots, which helps query planning, but they also delay when an event becomes visible to detection analysts. Smaller batches lower that latency and increase file count. There is no universal target. Confirm the exact sizing and compaction schedule with your own event rates, peak bursts and the query patterns your detection engineers actually run, then measure scan time and file counts on a representative week of data before you fix the numbers.
Securing the lake, the catalog and query access
A security data lake concentrates sensitive event data in one place, so access has to be controlled at four points: the object storage bucket, the Iceberg catalog, the Trino query layer and the NiFi flows that write to it. Each card below covers one of those points or the retention rules that govern the data itself.
Object storage access and encryption
The Parquet files and Iceberg metadata sit in object storage, so the bucket policy is the last line of defense if a catalog or query credential leaks. Give NiFi's writer a dedicated principal that can write only to the lake prefix, and give Trino a separate read-only principal. Confirm with your storage vendor how server-side encryption keys are managed, who can rotate them, and whether access logs for the bucket are retained in a system outside the lake.
Catalog authentication
The catalog decides which tables exist and where their metadata files live, so write access to it is more sensitive than read access to the data. Restrict catalog write operations to the NiFi service identity and the maintenance jobs that compact files, and keep interactive users read-only. Confirm which authentication method your chosen catalog supports, because the options differ between Hive Metastore, AWS Glue, JDBC, REST and Nessie catalogs.
Trino authentication and access control
Trino supports password file, LDAP, OAuth 2.0, Kerberos, certificate and JWT authentication, so pick the one your identity provider already issues tokens for rather than maintaining local passwords. Choose file-based rules for a small team or Apache Ranger or Open Policy Agent when access must follow group membership and audit requirements. Test each rule against the security event tables before analysts depend on it, because an overly broad grant exposes every raw field.
NiFi sensitive values and flow access
The NiFi flows that write to Iceberg hold catalog and storage credentials, which must be stored as sensitive parameters rather than plain processor properties. Limit who can view and change the flows in NiFi's authorization policies, and keep the write path separate from the parsing flows that analysts may need to edit. Confirm how your NiFi version protects sensitive values in flow definitions before exporting them to Git.
Retention and deletion of raw data
Raw security events often contain personal data and credentials that must not be kept longer than the investigation and legal hold requirements allow. Iceberg snapshots keep old data files available for time travel, so expiring snapshots and removing orphaned files is part of the deletion process, not an optional tidy-up. Document the retention period per table, schedule the expiry job, and confirm with your legal and records teams before setting values.
Questions teams ask before building a security data lake
What drives the cost and effort of a security data lake built with NiFi and Iceberg?
Scope drives most of the cost: how many log sources you onboard, how many distinct event formats need parsing, how much history you backfill, and how many query consumers depend on the tables. Cost splits into one-time work (flow design, schema and partition design, catalog and storage setup, Trino configuration, testing and documentation) and recurring work (compaction, snapshot expiry, upgrades and on-call). Object storage, compute for NiFi and Trino, and the catalog service are infrastructure costs you pay to your provider. Ask any partner for a written scope that lists sources, target tables, catalog choice, query consumers and retention, because a quote follows from that list. DeployOpen does not publish prices.
How long does it take to validate a first landing path into Iceberg?
The duration depends on scope, so no fixed figure applies. A first path is faster when it covers one log source, one normalized schema and one table, and slower when it adds several parsers, a new catalog, network isolation or a Trino query layer. Validation usually means confirming that records land with the expected schema, that partitions and file sizes behave as planned, that a query returns the same counts as the source, and that failures route to a dead-letter path. Ask for milestones in the written scope rather than a single end date.
What does the work need from our team?
The client team typically supplies the following:
- Access to the cloud accounts, data center or air-gapped network where the lake will run
- Named owners for the log sources, the normalized event schema and the retention rules
- Decisions on the catalog, object storage and identity model
- Sample events from each source, including malformed ones, so parsing can be tested against real data
- Reviewers for the detection and analytics teams who will query the tables
Without these inputs the schema and retention choices stall, which is usually the longest delay.
Can an internal team build a security data lake alone?
Yes, a capable internal team can build one, provided it has skills in NiFi flow design, Iceberg table design, object storage administration and SQL query engines. The main risks are underestimating schema evolution, small-file growth and catalog operations. Teams that build in-house usually assign one engineer to the data path and one to the platform, and they plan a review before the lake feeds detections. A partner is useful when the team lacks one of those skills or needs the work finished within a fixed scope. Compare both options against the same written requirements.
How does the security data lake sit beside the SIEM?
The SIEM keeps the events that analysts search and alert on, while the Iceberg lake keeps full-fidelity or longer-retention copies that are cheaper to store and query in bulk. NiFi sends each event to both destinations with filters suited to each one, so the lake does not replace detection content. Trino can query the Iceberg tables alongside other sources for investigations and retrospective hunts. Confirm with your SIEM vendor which ingest and export options your license supports, and confirm how audit requirements define the copy you must keep.
What does recurring operations look like for the lake?
Recurring work includes monitoring NiFi flow health and back pressure, watching Trino query performance, compacting small files, expiring old snapshots, removing orphan files, testing backups and restores of the catalog, and applying upgrades. NiFi 2 and Trino both release on their own schedules, so plan a regular upgrade window and test table reads after each change. Iceberg rollback and time travel help recover from bad writes, but they need snapshot retention set deliberately. Ask any partner whether this work is covered by a retainer and what response times apply in writing.
Which catalog should we choose for the Iceberg tables?
The catalog stores table metadata and is shared by the writer and readers, so the choice affects every query engine. Trino's Iceberg connector supports Hive Metastore, AWS Glue, JDBC, REST, Nessie and Snowflake catalogs, selected with iceberg.catalog.type. Pick the catalog your query engines already support and that your identity and backup model can cover. Confirm how NiFi's PutIcebergRecord processor connects to your chosen catalog before you commit to a design.
Running the lake after go-live
The lake needs recurring maintenance from the first day of production, because every NiFi commit to an Iceberg table adds data files and snapshots. Schedule compaction of small files per partition, expire old snapshots on a retention policy agreed with the detection and compliance teams, and run orphan file cleanup only after snapshot expiration, so files still referenced by retained snapshots are never removed. Confirm with your catalog and engine versions which maintenance procedures they support, and test each one against a copy of a production table before enabling it on the live lake. Maintenance jobs should be owned, logged and visible in the same monitoring stack as the ingest flows.
Schema changes arrive with new log sources, and Iceberg supports schema evolution without rewriting tables. Still, the NiFi record schema in PutIcebergRecord and the Trino view definitions that detection teams query must be changed in the same review. Adopt a process where a new source is added to a staging table first, its fields are mapped to the normalized model, and the change is promoted only after queries that depend on those fields return expected results. Keep schema versions in your Git-versioned flow definitions so that a rollback of the flow and a rollback of the table state can be matched to the same change record.
Monitor NiFi backpressure on the queues that feed the Iceberg writers, and alert on repeated write failures, catalog timeouts and commit conflicts, since a stalled writer can silently delay events in the lake while the SIEM still receives them. Restore tests matter as much as backups: periodically roll a non-production copy back to a recent snapshot or query a historical snapshot with time travel, then compare row counts and key fields against the source. Record the results, the time taken and any gaps. Ask your platform owner to confirm which snapshot and metadata files must be included in your object storage backup, because the lake is only recoverable if both the data and the catalog state are restored together.
How DeployOpen scopes and delivers a NiFi security data lake
DeployOpen plans, builds, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For a security data lake, that means agreeing the NiFi flows, Iceberg table design, catalog and Trino query layer in writing before build work starts, then handing over a system your own team can operate.
How we scope
- Inventory the log sources, event formats and daily volumes that the NiFi flows must parse and normalize into Iceberg tables.
- Agree the partition strategy, target file sizes, catalog type and object storage location in a written design record.
- Define which SIEM searches stay in the SIEM and which queries move to Trino, with the security team confirming each use case.
How we build
- Build NiFi flows that land normalized events through PutIcebergRecord, with error routing for records that fail schema checks.
- Configure the Iceberg catalog and Trino Iceberg connector, then test snapshot, time travel and rollback behavior on sample data.
- Test authentication, authorization and encryption for NiFi, Trino and the object storage path against the agreed design.
How we hand over
- Deliver flow definitions versioned in a Git repository, with parameter and secret handling documented for your environment.
- Provide runbooks for compaction, snapshot expiry, retention, catalog backup and adding a new log source to the lake.
- Walk your platform and security engineers through the operating model and record open items against the written scope.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer, including NiFi and Trino version changes.
- Ask any quote to itemize one-time build work separately from recurring infrastructure, licensing and support costs.
