Apache NiFi MiNiFi edge agents
Collecting data at remote sites, plants and devices with MiNiFi Java or C++ agents feeding a central NiFi cluster: choosing between the two agents, central management, provenance from the edge and network constraints.
Apache NiFi features
Pages on NiFi capabilities that are evaluated on their own: edge agents, Python processors, provenance and parameter management, based on the NiFi documentation.
Apache NiFi Python processors
Building NiFi processors in Python with the FlowFileTransform, RecordTransform and FlowFileSource APIs: per-processor virtual environments, dependency declaration, Python version requirements and when Java is still the better choice.
Apache NiFi data provenance and lineage
Using NiFi provenance to trace every FlowFile from receipt to delivery or drop, replaying failed data, sizing and retaining the provenance repository, and what provenance can show auditors about data handling.
Apache NiFi parameter contexts and secrets
Promoting flows across environments with parameter contexts: sensitive versus non-sensitive parameters, inheritance, and parameter providers that pull values from HashiCorp Vault and other external stores.
Apache NiFi Stateless execution engine
When to run a process group on the Stateless engine instead of the traditional engine: transactional sources such as Kafka, JMS or SQS, single-destination flows, data that is not persisted across restarts, failure ports, timeouts and merging caveats.
Record-oriented processing in Apache NiFi
Why NiFi flows built on record readers and writers outperform split-per-line designs: CSV, JSON, Avro, Parquet, XML and Excel readers, ConvertRecord, UpdateRecord and QueryRecord, schema inference versus explicit schemas, and when splitting is still needed.
Apache NiFi with Confluent, Apicurio and AWS Glue schema registries
Resolving Avro, JSON and Protobuf schemas from an external registry in NiFi record flows: the Confluent, Apicurio and AWS Glue registry services, encoded schema references in Kafka messages, and replacing the removed Hortonworks Schema Registry service.
Apache NiFi Site-to-Site data transfer between clusters
Moving data between NiFi instances, data centers and edge sites with the Site-to-Site protocol and remote process groups: port discovery, certificate-based security, load distribution across remote nodes, batching, checksums and reverse proxy setups.
Enforcing flow standards with Apache NiFi flow analysis rules
Using flow analysis rules to stop risky flow designs before they reach production: recommendation versus policy rules, disallowing component types, restricting back pressure and FlowFile expiration settings, and requiring TLS on listeners.
