Proven ways to build NiFi flows that handle errors, duplicates, ordering, validation, enrichment and transformation, based on the processors in the NiFi 2 components documentation.
Designing failure paths that do not lose or loop data forever: failure relationships, penalization, RetryFlowFile with a maximum, dead-letter queues, FlowFile expiration and alerting on retries exceeded.
Removing duplicates within files and across files with DeduplicateRecord and DetectDuplicate: hash sets versus bloom filters, distributed map caches in a cluster, cache sizing and expiry.
Avoiding millions of tiny FlowFiles and tiny output files: MergeRecord and MergeContent bin settings, max bin age, correlation attributes and sizing outputs for object storage, lakehouse tables and warehouses.
Checking incoming data before it reaches targets with ValidateRecord, ValidateJson and ValidateCsv: schema sources, strict versus lenient types, routing invalid records and reporting rejects back to senders.
Adding reference data to records with LookupRecord and lookup services, or joining two FlowFiles with ForkEnrichment and JoinEnrichment: database, REST, CSV and cache lookups and their performance costs.
Choosing between JoltTransformRecord, JoltTransformJSON, JSLTTransformJSON, UpdateRecord and QueryRecord for reshaping JSON, with the trade-offs in readability, performance and testing.
Using SQL inside a flow with QueryRecord to filter rows, route by condition, rename fields and aggregate, with multiple query relationships from one processor and its limits on large inputs.
Protecting downstream APIs and databases with ControlRate and back pressure: rate by count, size or attribute, accuracy trade-offs with short time windows, and rate limiting HTTP intake after the ListenHTTP change in 2.4.
Keeping records in order when flows branch, retry or load balance: connection prioritizers, EnforceOrder by group, partition-by-attribute load balancing and the limits of ordering across nodes.
Reading spreadsheets and CSV exports reliably with ExcelReader and CSVReader: XLSX and XLS support, password-protected workbooks, multiple sheets, schema inference versus explicit schemas and bad rows.
Holding data until a related event arrives with the Wait and Notify processors: release signals in a distributed cache, counting fragments, timeouts and alternatives for batch completion checks.
Keeping target tables in step with changing source records using UpdateDatabaseTable and PutDatabaseRecord: adding columns, flat record requirements, when to block changes and audit trails for schema changes.
Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.