Apache NiFi cloud storage integration

How Apache NiFi connects to S3, Azure and Google Cloud Storage

NiFi reaches Amazon S3, Azure Blob or Data Lake Storage and Google Cloud Storage through its own bundled processors and controller services, so no separate plugin is needed. The team that moves files into object storage with cron jobs and shell scripts can replace that work with a flow that lists, fetches and puts objects on a schedule, with retries, back pressure and provenance handled by NiFi. The list, fetch and put pattern is the core of each integration: a listing processor records what it has seen in NiFi state, a fetch processor downloads each object as a FlowFile, and a put processor writes FlowFiles to a destination bucket or container. Object keys and container names travel as FlowFile attributes, so routing, renaming and path construction happen in the flow rather than in a script. Each listing and put processor also needs a matching controller service that holds the cloud credentials, which keeps secrets out of individual processor properties. Everything runs inside your own cloud accounts and inside your NiFi cluster. NiFi does not run in a vendor-hosted service, and the storage credentials, bucket policies and network paths are yours to design and audit. Before building, confirm the exact processor names, controller service names and property names in the component documentation for your NiFi 2 version, because these bundles change between releases and some properties differ by cloud. Check which credential types each controller service accepts, such as access keys, managed identities, service accounts or workload identity, and confirm them against your organization's policy. Also confirm which listing processors are designed to run on the primary node only in a cluster, since that determines how duplicate pickups are prevented. The risks sit mostly in the boundaries. Keys that contain characters the destination does not accept, objects that change after listing, and large files that exhaust heap during fetch are the usual failure points, and each needs a test with representative objects before cutover. Plan the flow so that a failed put leaves the source object in place until the destination write is confirmed. Cross-cloud transfer flows, which read from one provider and write to another, add a second set of credentials, egress costs and network routes that you should document as part of the design.

What you need before building cloud storage flows

Before any S3, Azure or Google Cloud Storage flow runs, the platform, identities and network paths must be in place. The cards below cover the runtime, the NiFi version, provider access, network trust and cluster state, which together decide whether the flows can be built and operated safely.

NiFi 2 on Java 21

The NiFi administration guide lists Java 21 as the minimum runtime, so the host image, JVM and service wrapper must all be checked against that before installation. Confirm that the Python version used by any optional Python-based processors is one the guide lists, and remember that those processors are described there as beta. Build cloud flows on NiFi 2 from the start, because a 1.x line cannot be maintained further.

Why not stay on 1.x

The NiFi project states that the 1.x line depends on Jetty 9.4, Spring Framework 5.3 and AngularJS 1.8, none of which can be upgraded within that line. Flows that use cloud processors on 1.x would therefore inherit an unmaintained base. Plan the move to 2.x as a prerequisite, and check the migration guidance for the 1.27.0 step and the flow configuration file change before you start.

Bucket, container and account access

Each target needs a named bucket, Azure container or Data Lake Storage file system, and a Google Cloud Storage bucket, plus a documented owner. Ask the cloud team which prefixes NiFi may list, read and write, and whether object versioning, lifecycle rules or retention locks will affect files NiFi moves or deletes. Agree on these before the first flow runs, because permission errors are the most common early failure.

A separate identity per cloud

Each provider needs its own credential, such as an access key or IAM role for S3, a managed identity or service principal for Azure, and a service account for Google Cloud Storage. The client must supply these identities, scope them to the required buckets or containers, and decide how they are issued and rotated. Store them in the controller services that NiFi uses for credentials, not in flow parameters that are exported with the flow.

Outbound routes to each endpoint

Every NiFi node must reach the storage endpoints for each provider over the network path that the security team approves, including DNS resolution. Confirm whether traffic goes directly, through a proxy, or through a private endpoint, and test each route from every node, not only from one host. Record the firewall rules and the owner who approves changes to them.

Trust chain for private endpoints

When a private endpoint, custom DNS name or inspecting proxy is used, the certificate presented to NiFi must chain to a root that the NiFi truststore trusts. Export the chain from the endpoint, import it into the truststore on each node, and test the handshake before adding processors. Keep the truststore under change control, because a missing intermediate certificate often shows up only after a renewal.

Where cluster state lives

A clustered NiFi needs a decision on coordination, either embedded or external ZooKeeper, or Kubernetes using ConfigMaps or Leases for state and leader election. This decision affects which node lists a bucket, because list processors store their state in the cluster. Choose the coordination model with the platform team, confirm it in writing, and make sure it is in place before cloud list and fetch processors are scheduled.

How each cloud store authenticates from NiFi

NiFi's S3 processors authenticate through a controller service rather than through properties on each processor, so credentials are configured once, protected as sensitive values, and shared by every list, fetch and put processor that references it. Open the controller service properties in the component documentation to see which credential modes the release you run supports, since these options change between versions and should not be taken from memory. Plan the IAM side around one bucket and, where you can, one prefix, granting only the object and list actions the flow needs.

Credential source: Confirm in the controller service properties whether the release accepts static keys, a named profile, or an instance or container role, and prefer a role where your runtime supports it.
Bucket scope: Grant list on the bucket with a prefix condition and object read or write only under the prefixes the flow uses, instead of account-wide S3 access.
Write actions: Add write permissions only to the flows that deliver files, and keep list and read in a separate policy for pickup flows.
Encryption keys: If the bucket uses a customer managed KMS key, grant the matching decrypt or encrypt permissions to the same role.
Rotation: Record who rotates static keys, how often, and how a rotation is applied to a running cluster without a stopped flow.

Confirm with the cloud team which account owns the bucket, whether bucket policies or a VPC endpoint restrict access, and how audit logging of object access is enabled.

Configuration sequence for a cloud storage flow

This sequence takes a cloud storage flow from an identity with narrow permissions to a verified transfer between NiFi and an object store. Each step closes with a check, so a fault is found at the step that introduced it rather than after data has moved.

Create a narrowly scoped cloud identity
The cloud team creates a dedicated identity for NiFi in the target account, such as an IAM role, managed identity or service account, and grants only the list, read and write actions on the specific buckets, containers or prefixes the flow needs. Deletion rights stay out of the first version unless a flow requires them. Confirm the exact permission names for your provider in its current documentation, because they differ by service. The step is done when the identity can perform each required action in a test and is denied an action outside the scope, such as writing to a bucket the flow should not touch.
Create and test a controller service per provider
In NiFi, the flow engineer creates one controller service for each provider the flow touches, and stores the credential material as sensitive properties or references it from the secrets source your organization uses. Keep credentials out of processor properties and flow parameters that are not marked sensitive. Use the service's built-in verification where it is offered, then run a small test that lists a known prefix. The step is done when the service reports valid and the test listing returns the expected object names.
Configure the list processor with cluster state
The flow engineer adds the list processor for the source bucket or container, sets it to run on the primary node in a cluster, and confirms that it keeps its listing state so each object is emitted once. Set the prefix, a file filter and the listing strategy deliberately, since these decide what is picked up on each run. Leave the run schedule conservative during testing. The check is that a single test object appears once in the output queue after the first run and does not reappear on the next run unless it was changed.
Add the fetch processor for the listed key
The flow engineer connects the list output to the fetch processor and sets the bucket, container and key attributes from the list FlowFile, so the fetch reads the exact object the listing produced. Route the fetch on its failure relationship to a separate path, because missing or permission-denied objects are handled differently from transient network errors. The check is that the content of the fetched FlowFile matches the source object, which you can confirm by comparing a checksum of the original file with one computed on the NiFi side.
Add the put processor with a destination prefix
The flow engineer adds the put processor for the target provider and sets a destination prefix that separates test output from production output, then maps the object key from the source attributes or from a parameter. Choose whether the destination should overwrite existing objects or fail on conflict, and document that choice for the operations team. The check is that the object appears under the expected prefix and that its checksum matches the source file after transfer.
Route success and failure to retry or error handling
The flow engineer routes the put success relationship to the completion path and sends failure to a retry loop with a bounded number of attempts, followed by an error destination that keeps the FlowFile and its attributes for inspection. Decide which failures are retried automatically, such as throttling or timeouts, and which need a person, such as an access denial. The check is a deliberate failure test, for example using an identity that lacks write permission, which must land in the error destination with a clear message and must not drop the file silently.
Run a cross-cloud transfer test
For a flow that moves files between providers, the flow engineer runs the full chain from the source controller service through the target controller service with a test set of files that includes a large object, an object with unusual characters in its name and a zero-byte object. Compare counts and checksums at both ends and confirm that no object is listed twice across repeated runs. The step is done when the reconciliation report from the test matches the source inventory exactly, and the results are saved with the flow version in your Git-based registry for later audit.

How object keys and metadata map onto FlowFile attributes

This table compares each part of a cloud object, such as its key, bucket or container, size, timestamps, checksum and custom metadata, with the FlowFile attribute or destination field that carries it. Read each row as a question to answer before the flow goes live. The main takeaway is that attribute names and checksum behavior differ by processor, so confirm them in each component's documentation and test them with real objects.

Source attribute	Target field	Transformation or prefix rule	Check to confirm
Object key	FlowFile filename and object key attribute	Keep the full key so folder structure survives, or strip a configured prefix before writing	Confirm the attribute name in the List and Fetch processor documentation for each provider
Bucket or container name	Bucket or container attribute on the FlowFile	Carry it forward as an attribute so downstream Put steps can route to the same or a different target	Confirm the attribute is present after both list and fetch steps
Object size	Size attribute in bytes	Compare against bytes written after transfer; do not rely on it for multipart or compressed objects without testing	Compare source size with destination size on a sample of large objects
Last-modified time	Timestamp attribute used for incremental listing	Convert the provider timestamp to the format your state logic expects	Confirm the time zone and precision, and that listing state advances correctly across a cluster
Content type	MIME type attribute	Pass through as-is, or set it explicitly on write when the source type is missing or wrong	Confirm the destination accepts the type and that it matches on a sample object
ETag or checksum	Checksum or ETag attribute, plus an optional hash attribute computed in NiFi	An ETag is not always an MD5 digest, so compare with a hash you compute in the flow when integrity matters	Upload a known file, download it, and compare hashes end to end
Custom object metadata	User-defined attributes with a fixed prefix	Map each custom key to a named attribute; apply a prefix to avoid collisions with NiFi's own attributes	Confirm which keys the fetch processor returns and how the put processor writes them back
Destination key and metadata	Object key and metadata set on the put processor	Build the destination key with Expression Language from source attributes, and keep the same prefix rules for each target cloud	Verify the written key path and metadata in the target console after a test run

Designing transfer flows between two clouds

Decide first whether a flow should copy objects directly from one cloud to another through NiFi or stage them in a store you already control. A direct transfer, with FetchS3Object reading from Amazon S3 and PutGCSObject writing to Google Cloud Storage, is the simpler design when volumes are modest and the flow runs once or on a steady schedule. Staging is the better choice when the same data feeds several consumers, when the transfer must be replayed, or when the copy has to survive a NiFi outage without re-reading the source. Egress charges are usually the largest cost in a cross-cloud transfer, and they are billed by the source provider on the bytes that leave it. Confirm the rates in each provider's price list and multiply them by your own monthly volume, object count and retry rate, because a flow that retries whole files pays for those bytes again. Bandwidth between the NiFi nodes and each cloud, and the number of objects, also shape the design, since many small objects cost more in per-request charges and in NiFi's per-FlowFile overhead than a few large ones.

A fetch and put pair does not stream bytes past NiFi. Each FlowFile's content is written into the content repository on the node that handled it, and it stays there until the downstream put succeeds and the FlowFile is dropped. Large objects therefore occupy disk for as long as they wait in a queue, and a slow destination can fill the content repository and stall unrelated flows. Size the content repository for the largest in-flight objects multiplied by the number of concurrent transfers, not for the average file. Then set connection back-pressure thresholds on the queues between FetchS3Object and PutGCSObject so that upstream listing stops before the disk fills. Confirm the repository and back-pressure settings against the NiFi administration guide for your version and test them with an object at your largest realistic size, because the failure mode is a node that stops accepting work rather than an error message.

Securing credentials and transfers to cloud storage

Cloud storage flows put long-lived keys and bulk data movement in one place, so the controls around them matter as much as the processors. The cards below cover where credentials live, how far each identity reaches, how connections are protected, who can change them, and what evidence the audit team can review.

Keep credentials in controller services

Store access keys, secrets and tokens only in the controller services for S3, Azure and Google Cloud Storage, and mark each secret property as sensitive so NiFi encrypts it at rest and masks it in the UI. Do not place keys in processor properties, flow parameters that are not marked sensitive, FlowFile attributes or flow definitions committed to Git. Check the sensitive properties protection setup in the NiFi administration guide before you rely on it. Confirm which credential options each provider's controller service supports in your NiFi version.

Scope each identity to one target

Create a separate cloud identity for each flow or environment, limited to one bucket or container and to the rights the flow needs. A pickup flow that only lists and reads should not hold write or delete permissions, and a delivery flow should not be able to read other buckets. Ask your cloud team which policy language and resource scoping each provider offers, because the exact permission names are provider specific and should be confirmed against current provider documentation.

Enforce TLS to every endpoint

Require HTTPS for every storage endpoint the processors call, including private endpoints and any custom or regional URLs you configure. Where your organization uses an internal certificate authority or a storage gateway with its own certificate, import the trust chain into the NiFi truststore and pin only that trust, rather than disabling certificate checks. Test each endpoint from every cluster node, since a node with a missing trust entry fails only its share of the work.

Restrict controller service changes

Controller services hold the credentials, so the users who can create, modify or enable them are the users who can redirect data or reuse keys. Use NiFi authorization policies to give controller service and policy administration to a small named group, and give flow developers read and operate rights on the process groups they build. Review these policies after each change in team membership, and confirm the authorizer and policy model for your NiFi version in the administration guide.

Route provenance and logs to auditors

Decide where NiFi provenance events, bulletin output and application logs for cloud transfers are stored, who can read them, and how long they are kept. Provenance records show which object was fetched or written and by which flow, which is the evidence most often requested for data movement. Confirm with your audit team which log destination and retention they can access, and test that the records for a sample transfer can be exported before go-live.

Rotate keys or use short-lived identities

Plan a rotation schedule for any static access key and document the steps to update the controller service without stopping unrelated flows. Where the cloud platform and your NiFi deployment support workload or short-lived identities, prefer them over stored keys, since they expire without manual action. Ask the cloud provider which identity federation or token options apply to each storage service and confirm that your NiFi version and the chosen controller service support them before you design around them.

Common failures in cloud storage flows and how to fix them

This table lists the failures teams most often hit when NiFi lists, fetches and puts objects in S3, Azure and Google Cloud Storage. Each row gives the likely cause, the first diagnostic step and the fix, so you can work from a symptom to a checked change. Start with duplicate pickups and access errors, since they are the most common and the easiest to confirm.

Failure	Likely cause and diagnostic step	Fix
Duplicate pickups after a restart or node change	The listing processor keeps its state on one node, or it runs on all nodes instead of the primary node. Check the processor's scheduling and whether its state survives a restart. Compare object keys in the duplicate FlowFiles' provenance.	Schedule the listing processor on the primary node only, use cluster-scoped state, and confirm the state is retained across a restart before going live.
Access denied on list or get	The identity used by the controller service lacks list permission on the bucket or container, or get permission on the prefix. Review the error message in the bulletin and test the same credentials with the provider's own CLI against the same prefix.	Grant list rights on the bucket or container and read rights on the required prefixes, then retest. Scope rights to the prefixes the flow uses rather than the whole account.
Timeouts or throttling on large objects	Very large objects, slow links or high concurrency can exceed timeouts or trigger provider rate limits. Look for timeout exceptions or repeated retry errors in the logs and check the request rate against the provider's documented limits.	Raise timeouts where the link justifies it, reduce concurrent tasks, use multipart or chunked transfer where the processor supports it, and spread work across prefixes.
Checksum mismatch after transfer	A partial write, an interrupted transfer or a processor that does not verify the stored checksum can leave a damaged object. Compare the stored checksum attribute with a checksum computed on the source file.	Enable checksum verification where the processor offers it, route mismatches to a retry path, and delete and rewrite the affected object rather than trusting the copy.
Expired or rotated credentials stop the controller service	Access keys, client secrets or service account keys were rotated or expired, so the controller service fails to authenticate and every dependent processor stops. Check the controller service state and the authentication error in the logs.	Update the credential in the controller service, then enable it again. Store credentials in a managed secret store, and agree a rotation schedule with the team that owns the cloud account.
Clock skew breaks signed requests	Signed requests to S3, Azure and Google Cloud Storage include timestamps, and a node whose clock drifts beyond the provider's tolerance will see signature or time-window errors. Compare the node's time with a trusted time source on every cluster member.	Synchronize clocks on all NiFi nodes with NTP or the platform's time service, then monitor drift as part of host health checks.

Questions buyers ask about cloud storage integration with NiFi

How much effort does a three-cloud NiFi storage integration take?

Effort depends mostly on how many buckets, containers and file systems the flows touch, how many files move each day, how much the files must be transformed, and how long internal approvals take. A flow that lists and fetches from one S3 bucket and writes to one Blob container is a small, bounded task. A design that spans S3, Azure Data Lake Storage and Google Cloud Storage, with separate credentials per account, record-level conversion and checksum checks, adds design, test and review work in each dimension. Ask the partner to break the estimate down by cloud, by flow and by environment, so you can see which factor drives the total.

Can our in-house team build cloud storage flows in NiFi?

Yes, if the team has someone comfortable with NiFi processors, controller services, the expression language and Git-based flow versioning, plus engineers who understand IAM roles, storage access policies and network egress in each cloud. Most gaps appear in credential handling, state management for listing processors in a cluster and error routing for partial failures. Assess those skills honestly before starting. A partner can build the first flows alongside your staff so the know-how stays in-house, which is usually cheaper to plan than a rebuild after a failed handover.

What must we provide before an integration project starts?

Plan to supply cloud accounts or subscriptions and projects where the NiFi nodes will read and write, identities with the least privilege the flows need, network routes or private endpoints from the NiFi cluster to each storage service, and a secrets store for keys and tokens. You should also provide representative sample files, including malformed and oversized ones, plus expected file naming, folder layout and retention rules. A written test plan that defines success for each flow, such as object counts, checksums and error handling, keeps acceptance concrete.

How long does a cloud storage integration take?

The timeline is set by readiness rather than by NiFi itself. Account and identity provisioning, network path approvals, security review of credential handling, and the number of environments to promote through usually take longer than building the processors. Transformation complexity and the volume of sample data for testing add further time. Request a schedule that marks which dependencies are on your side, so the start date reflects when inputs arrive, not when the build team is free.

Is NiFi the right tool compared with a managed transfer service?

NiFi suits teams that need custom routing, transformation, enrichment and error handling between several storage systems and other sources, and that want to run the pipeline in infrastructure they control. A managed transfer service can be simpler for plain copy jobs between two cloud accounts with little logic. Compare the two on transformation needs, number of destinations, how much you must control where data is processed, and who will operate the pipeline after go-live. Running NiFi also means operating its cluster, so include that cost in the comparison.

What should a fixed-scope quote for this work list?

A fixed-scope quote should name each cloud service and account in scope, each flow with its trigger (list, fetch, put or cross-cloud transfer), the processors and controller services it uses, and the credential method for each. It should list the environments included, the test cases and acceptance criteria, the documentation and handover sessions, and the items the client must supply. It should also state what is excluded, such as custom processors, changes to the source systems or audit support. Ask whether the quote is one-time or includes recurring support, because operations and upgrades recur after go-live.

How do we avoid duplicate or missed files when listing cloud storage?

Use the list processor for each cloud service with state tracking, run it on the primary node or with cluster-aware state, and only then fetch the listed objects. Verify the behavior with a test that adds, replaces and deletes objects during a run. Confirm in your own environment how each processor persists state across restarts and cluster changes, and document the expected behavior for reruns.

How DeployOpen scopes and delivers cloud storage flows

DeployOpen plans, builds, documents and supports Apache NiFi flows that move files between NiFi and Amazon S3, Azure Blob or Data Lake Storage, and Google Cloud Storage, inside the cloud accounts and networks you control. Engagements are fixed-scope, with the scope, deliverables and timeline agreed in writing before work starts.

How we scope

Inventory every bucket, container and prefix the flows will read or write, with owners and retention rules.
Document the credential model for each cloud, including which controller service holds keys, roles or managed identities.
Agree in writing which NiFi version and cluster topology the flows will run on, and which cloud regions and accounts are in scope.

How we build

Build list and fetch patterns with state tracking so clustered nodes do not pick up the same object twice.
Configure cloud credentials through controller services and parameter contexts, with no secrets stored in flow definitions.
Test cross-cloud transfer flows with checksums, failure routing and retry behavior before they are promoted to production.

How we hand over

Version the flows in a Git repository so changes are reviewed, tracked and reproducible across environments.
Provide runbooks covering credential rotation, bucket permissions, failure queues and the steps to replay a failed transfer.
Walk your operations and security teams through the flows, the provenance trail and the monitoring you will own afterward.

How we support

Offer ongoing support, upgrades and continuous testing as a retainer, scoped separately from the build.
Review cloud provider SDK and NiFi component changes during planned upgrade windows, with a test run before production.
Confirm with your team which support hours, escalation paths and service levels apply, as these are set in the written agreement.

Apache NiFi with Amazon S3, Azure and Google Cloud Storage