Apache NiFi and MongoDB integration

How Apache NiFi and MongoDB integration works

A team that needs MongoDB collections fed from or drained into NiFi flows works with two processors: GetMongoRecord reads query results from a collection, and PutMongoRecord writes batches of records into one. Neither processor carries its own connection details. Both depend on a shared MongoDBControllerService, so the connection string, credentials and TLS settings live in one place that every flow in the cluster can reuse. GetMongoRecord is the record-based version of GetMongo, and it writes its results through a Record Writer, which determines the output format. PutMongoRecord reads incoming FlowFiles through a Record Reader and a schema, then inserts or upserts the records. Because the reader and schema define the field types sent to MongoDB, they are the first place to check when documents come out malformed.

The most important limit to plan around is that PutMongoRecord does not delete documents. It supports insert and upsert only, so any flow that must remove documents from a collection needs a separate mechanism, such as a database-side process or a change to the source data model that marks records as inactive. Upsert also needs a clear key decision before the build starts: the team must know which fields identify a document, and confirm in the processor documentation how those fields are matched and what happens when a record has no match. Batch size, controlled by the Batch Size property, is the setting that protects the database under load, so it should be tested against the target cluster rather than left at its default. Confirm the current behavior of each property against the PutMongoRecord component documentation for the NiFi version you run.

What you need before building a MongoDB flow in NiFi

A MongoDB flow in NiFi depends on four things decided before the first processor is placed: the NiFi 2 deployment that runs it, the MongoDB deployment and account it connects to, the network path and TLS settings between them, and the schema that record readers and writers will use. Settle these first, because each one changes how PutMongoRecord and GetMongoRecord behave in production.

NiFi 2 runtime and cluster shape

Confirm the NiFi 2 release and Java runtime your team will run, since the administration guide lists Java 21 as the minimum. Decide whether the flow runs on one node or a cluster, and whether it is scheduled on the primary node only. A MongoDB write flow that runs on every node at once multiplies load on the collection, so the scheduling choice belongs in the design, not in a later fix. Confirm the NiFi release against the component documentation for PutMongoRecord before you pin a version.

MongoDB deployment and server versions

Identify the MongoDB deployment type, whether a standalone server, a replica set or a sharded cluster, and the server version your collections run on. Confirm the supported MongoDB server versions for the NiFi release in use against the component documentation, because support can differ between releases. Ask your database team whether the target is a primary, a secondary or a load-balanced endpoint, since that determines where reads and writes land.

Dedicated database account and roles

Create a dedicated MongoDB user for the flow instead of reusing an administrative or application account. Grant only the roles needed for the operations the flow performs, for example read on the source collection and insert or update on the target collection. Remember that PutMongoRecord does not delete documents, so the account does not need delete rights for this flow. Record which database and collections the account can reach, because that list is the first thing an auditor will ask for.

Network path and TLS settings

Map the network path from the NiFi nodes to the MongoDB endpoints, including firewalls, DNS names and any proxy or load balancer in between. Decide whether the connection uses TLS, which certificate authority signs the MongoDB server certificate, and whether NiFi nodes must present client certificates. Each NiFi node needs the same trust configuration, so verify connectivity from every node, not only from the one you test on.

Shared MongoDB controller service plan

Plan one MongoDB controller service per MongoDB deployment and share it across processors, rather than placing connection URIs on each processor. NiFi deprecated the Mongo URI property on PutMongo in 1.22.0 in favor of MongoDBControllerService implementations, so new flows should start there. Decide who owns the controller service, where its credentials are stored, and which process groups may reference it, because a change to it affects every flow that uses it.

Record schema source and format

Decide where the record schema comes from before building the reader. PutMongoRecord uses a configured record reader and schema to insert or upsert records, so the schema must match the documents you intend to write. Choose whether the schema is embedded in the reader, read from the incoming data, or maintained in a schema registry, and agree on field names, types and nested structures with the owners of the source data. Mismatched types are the most common cause of rejected batches.

Configuring the MongoDB controller service and record processors

This sequence connects NiFi to one MongoDB deployment, defines the record format once, and proves that reads and writes land in the right collection before any production traffic runs through the flow.

Create the MongoDB controller service
Create a MongoDBControllerService in the NiFi canvas controller settings, not inside each processor. Set the connection URI, the database and the authentication details, and store the password as a sensitive value. Use a service account limited to the collections this flow needs. Confirm the exact connection properties for your MongoDB version and deployment type (standalone, replica set or sharded cluster) against the NiFi component documentation before you enable the service. The step is done when the service shows as enabled and a connection test from the controller service configuration succeeds. Because the service is shared, any processor that references it will fail if the service is disabled, so note which flows depend on it.
Define the record reader and writer schemas
Add a Record Reader and a Record Writer as controller services and give them the same schema, or a schema that deliberately differs where a transformation is intended. For PutMongoRecord, the reader parses incoming FlowFiles into records, so its schema determines which fields reach MongoDB and their types. For GetMongoRecord, the writer decides how query results are serialized into the FlowFile, typically JSON. Use an explicit schema rather than schema inference for production flows, so that a source change does not silently alter document shape. Confirm the reader and writer options in the NiFi record processing documentation for your release. The step is done when a sample FlowFile parses in the reader and the writer produces the expected JSON in a test run.
Configure GetMongoRecord for queries
Place GetMongoRecord on the canvas, reference the MongoDB controller service, and set the database and collection names. Enter the query in the query property using a JSON filter that matches the documents you need, and choose the Record Writer you created. Keep the query narrow at first, for example a filter on a single date range or a small key set, so that the first run returns a bounded result. Note that GetMongoRecord reads from the collection, so confirm the account has read permission on it. Run the processor once and inspect the output FlowFile. The step is done when the FlowFile contains the expected number of records, which you can compare against a count run directly in the MongoDB shell with the same filter.
Configure PutMongoRecord for inserts and upserts
Place PutMongoRecord, reference the same MongoDB controller service, and set the target database and collection. Select the Record Reader that matches the incoming data, and choose the write mode that fits the use case: insert for append-only collections, or upsert when records must update existing documents by key. Confirm how the upsert key is defined in the processor properties for your NiFi release, and test it on a copy of the collection first. Remember that PutMongoRecord does not support deletes, so any removal of documents must be handled outside this flow. The step is done when a sample FlowFile writes without routing to the failure relationship and the processor's configuration reflects the chosen write mode.
Set batch size to protect the database
Set the Batch Size property on PutMongoRecord to a value that your MongoDB operations team has reviewed against the cluster's write capacity, replication lag and other workloads. The processor sends records in batches, so this value sets how many documents go out per write, and a larger batch increases load on the primary and on secondaries. Start with a conservative value and raise it only after observing write latency and replication lag under real load. Ask your DBA or platform team which limits they want enforced. The step is done when a load test with production-like volume shows stable write latency and no growing queue of FlowFiles waiting upstream of the processor.
Connect the processors into a test flow
Wire GetMongoRecord to PutMongoRecord, but send the output to a test collection rather than the production target. Route the success and failure relationships of PutMongoRecord to visible sinks, such as a log processor or a file on a test path, so that failed batches are not dropped. Keep the flow stopped until every processor shows a valid configuration with no warnings. This connection is where most field-mapping problems appear, because the reader schema must match the document shape the writer produces. The step is done when the flow starts without validation errors and the test run completes end to end.
Verify with provenance and a document count
After the test run, open the data provenance view for PutMongoRecord and confirm that the events show the expected record count and no unexpected failures. Then count the documents in the test collection in the MongoDB shell and compare the number with the number of records that left GetMongoRecord. Check a sample of documents for field names, types and nested structure. If you ran an upsert test, run it a second time with the same input and confirm the count does not increase. Record the results and the processor configuration in your change record so the same checks can be repeated after upgrades. The step is done when the counts match, the sampled documents are correct and the rerun behaves as designed.

Choosing insert or upsert for each collection

Event collections hold facts that happened once, such as log entries, clickstream events or transaction notifications. Each record should exist exactly once, and there is usually no natural update to apply later. For these collections, insert is the natural choice because every write adds a new document and nothing needs to be matched against existing data. The cost of this simplicity is that a replayed batch will create duplicates unless something upstream prevents the replay.

Replay exposure: A retried batch after a timeout can insert the same events twice, so confirm how NiFi handles failed batches before the write is designed.
Document identity: Confirm whether MongoDB generates _id values for inserted documents or whether the flow supplies them, because that choice decides whether duplicates can be detected later.
Re-run cost: Re-running a source query without a deduplication step will multiply events in the collection.
Batch boundaries: Keep batch size moderate so that a failed batch is small enough to reason about and clean up.
Downstream counts: Reports that count documents will drift if duplicates are not removed.

Before going live, ask the team to name the field or process that makes each event unique and to test a deliberate re-run in a non-production collection.

Sizing batches so MongoDB stays healthy

The Batch Size property on PutMongoRecord sets how many records NiFi groups into each write to the target collection, so it is the main lever you have over how hard the flow hits the database. Larger batches mean fewer round trips and usually higher throughput, but each write takes longer to acknowledge, holds more work on the primary, and produces a larger burst of oplog entries for secondaries to replicate. Smaller batches are gentler on the primary and make failures easier to isolate, at the cost of more network calls and more overhead per record. The right value sits between those two pressures, and it depends on your document size, indexes and write concern, not on a number taken from a guide.

Write pressure in a NiFi flow is multiplied in ways that are easy to miss. Each concurrent task on PutMongoRecord writes its own batches, and every node in a NiFi cluster runs its own copy of the processor against the same collection. A setting that looks modest on one node can produce several times the write load across a cluster with a high concurrent task count. Before you change the batch size, check the number of concurrent tasks, the number of nodes and whether the upstream queue can feed the processor fast enough to keep it saturated. Confirm these values in the processor configuration of the environment you will run in, not from a development setup.

Set the batch size from measurements. Run the flow against a copy of production-like data, then watch write latency, primary CPU and disk, queue depth in your MongoDB monitoring, and replication lag on secondaries at several batch sizes and concurrency levels. Choose the largest setting that keeps replication lag and latency inside limits your database owners have agreed, and leave headroom for other applications that share the cluster. Ask your MongoDB administrators which thresholds they use for these metrics before the test starts, and repeat the measurement after any change to indexes, document shape or cluster topology.

Mapping record fields to MongoDB documents

This table compares how each part of a NiFi record becomes a MongoDB document field during PutMongoRecord writes, and where the reader and writer schemas decide the stored result. Read each row as a question to answer in a test flow before the flow reaches production data. The main takeaway is that the upsert identifier and the timestamp type are the two mappings that most often cause silent duplicates or wrong query results.

Source field	Target field and type handling	Key use	What to confirm in testing
Business identifier (for example, order_id)	Mapped to the same name or to _id; the reader schema should declare it as the type the collection already stores	Primary match key for upsert, so it must be present and unique in every record	Re-running the same batch updates existing documents instead of creating duplicates
Nested record fields	Written as embedded documents or arrays, depending on how the record schema nests them	Not used as a key; keep key fields at the top level for simple matching	Nested values appear as expected objects in MongoDB, not as escaped JSON strings
Numeric fields (integer, long, decimal)	Stored as the BSON numeric type implied by the schema; check decimals for precision loss	Not normally a key, but a poor key choice if values are floats	Totals and counts query correctly as numbers in MongoDB, not as text
Timestamp and date fields	Confirm whether the reader parses them as timestamps or strings; string dates sort and compare incorrectly	Can support incremental queries on the collection, but are not reliable identity keys	Date range queries in MongoDB return the expected documents after the write
Null and missing values	Decide whether null is written as a stored null or omitted from the document, based on the writer configuration	Never use a key field that may be null in the source	Downstream queries that test for field existence behave as your consumers expect
Schema changes in the source	New or renamed fields may be ignored or stored under new names, depending on the schema in use	Changing the key field name breaks upsert matching for existing documents	A sample of records from each source version produces the documents you designed

Securing the MongoDB connection and credentials

The MongoDB connection in a NiFi flow is a privileged path into a database that other systems may depend on. Access is shaped by four layers: the database user, the transport, where credentials are stored, and who can change the shared controller service. Each card below covers one of these layers and what to verify before the flow goes live.

Least-privilege database users

Create a dedicated MongoDB user for each NiFi flow or group of flows, and grant read rights only on the collections GetMongoRecord needs and write rights only on the collections PutMongoRecord targets. Because PutMongoRecord does not delete, do not grant delete rights for that flow even if the database role would allow them. Check the granted roles against the flow design, and review them again whenever a processor is added or repointed to a new collection.

TLS on the database connection

Require TLS between NiFi and the MongoDB deployment through the connection settings in the MongoDBControllerService, and confirm that the server certificate chains to a CA that the NiFi nodes trust. Verify the TLS settings on every node, since each cluster member opens its own connections. Ask your DBA to confirm whether plaintext connections are rejected at the server, so a misconfigured client cannot quietly connect without encryption.

Sensitive property storage

Store the MongoDB password and any connection string secrets as sensitive values so they are encrypted in the flow definition and masked in the UI. Where the same credentials are used across environments, place them in a parameter context and reference them by parameter, so the flow JSON carries no secret values. Confirm the sensitive properties key is managed and backed up separately from the flow files, because losing it makes stored secrets unreadable.

Controller service access policies

Set NiFi policies so that only the operators who maintain a flow can view or modify the MongoDBControllerService and the record readers and writers it depends on. Use process group level policies rather than broad global grants, and keep read access separate from write access. Test the policies with a non-administrator account, because an operator with read access alone should not be able to change connection details.

Shared service change control

A single MongoDBControllerService may serve several process groups, so changing its URI, credentials or TLS settings affects every flow that references it. Treat edits to the shared service as a controlled change: record who approves it, test it in a lower environment first, and schedule it when dependent flows can be paused. Confirm which flows reference the service before any edit, using the controller service references view in NiFi.

Separate service accounts per environment

Use distinct MongoDB users and NiFi parameter values for development, test and production, so a credential exposed in a lower environment cannot reach production data. Ask the database team to confirm that each account is scoped to its own database and that the naming makes the environment obvious in audit logs. Review account inventory on a set schedule and remove accounts for flows that have been retired.

Common MongoDB flow failures and fixes

This table maps the failures teams most often hit when running PutMongoRecord and GetMongoRecord flows to their likely causes and fixes. Read each row as a first check: confirm the symptom in the processor's bulletin and logs, then work through the cause column before changing the flow.

Symptom	Likely cause	Fix
Processor fails with authentication or authorization errors	The MongoDBControllerService connection string, user or role does not grant the operation on the target database and collection	Confirm the account's roles on the target database with your MongoDB administrator, then update the shared controller service and retest a single FlowFile
Records fail with type or field errors on write	The schema in the Record Reader does not match the field types or structure the collection expects	Compare the reader schema with sample documents, correct field names and types, and test with a small batch before rerunning the full queue
Duplicate documents appear after a retry or restart	Insert mode re-sends FlowFiles that were already written before a failure, so the same records are inserted again	Switch to upsert with a stable key field where the data allows it, and make sure the key is unique in the collection
Writes are slow or the database shows high load during batches	Batch Size is set too high, so each request carries more documents than the database or network can absorb smoothly	Lower Batch Size, measure throughput at several values, and watch database metrics before increasing it again
Upserts fail or create unexpected documents	The key field used for matching is missing or null in some records, so the match cannot be made reliably	Validate the key field in the flow before PutMongoRecord, route incomplete records to a failure path, and fix the source extract
Records are expected to be removed but remain in the collection	PutMongoRecord does not support deletes, so removal requests are not applied	Plan deletes through a separate, explicitly designed step, such as a script or a database-side process, and confirm the approach with your data owner
Connection errors after changing the MongoDB URI	The deprecated Mongo URI property was still in use on older processors instead of the controller service	Move all processors to a shared MongoDBControllerService and remove the deprecated URI property from each one
Changes to one flow break other MongoDB flows	Several processors reference the same controller service, so a single edit affects every consumer	Document which processors share the service, test changes in a non-production cluster, and schedule updates with the owners of each flow

Questions buyers ask before building MongoDB flows in NiFi

What drives the cost of a NiFi and MongoDB integration?

The main drivers are the number of collections, how complex each document schema is, and how much testing the flows need. Each collection usually means its own read or write flow, a record reader and writer that match the document shape, and checks on how the data lands. Schema complexity matters most for nested documents and arrays, because those need explicit record handling and more test cases. Work on deletes is excluded from these processors, so if removals must be replicated, that logic lives in a separate flow or another system and adds scope. Ask any partner for a quote that lists collections, schema variants, delete handling, test environments and the cutover window as separate line items. Fixed-scope engagements are agreed in writing before work starts, so the quote should state exactly what is included.

How long does a scoped MongoDB integration take?

Duration follows the agreed scope, not a standard number of days. A fixed scope covering a small number of collections with simple schemas and a defined test plan is shorter than one that covers many collections, several schema versions and a staged cutover. The timeline should break out schema mapping, flow build, load and batch sizing tests, failure handling tests and the production change window. DeployOpen does not publish durations, so ask for a written plan with milestones and the dependencies that could move them, such as access approvals or test data availability.

What does the client need to provide before work starts?

Before work starts, the client should provide:

Database access for a non-production environment, with a read account for GetMongoRecord and a write account for PutMongoRecord, scoped to the named databases and collections
Current schema documentation or sample documents, including fields that vary between records
Representative test data that includes edge cases such as missing fields, nulls and oversized documents
A change window for production cutover and a rollback decision owner

If the team cannot supply these, the first phase usually becomes discovery, which affects the schedule and the price. Confirm the MongoDB version, network path from the NiFi cluster and whether the database is a replica set with the access rules your security team requires.

Can an internal team build MongoDB flows with the documented processors?

Yes, a team with NiFi and MongoDB experience can build basic flows using the NiFi component documentation for GetMongoRecord and PutMongoRecord. The processors are record-based, so the team needs to be comfortable with record readers and writers, schemas and NiFi controller services. The harder parts are insert versus upsert logic, batch sizing against a production database, idempotent reruns and handling documents that fail schema validation. The team also has to decide where deletes are handled, since PutMongoRecord does not support them. A useful test is whether the team can explain how a failed batch is retried without duplicating or losing documents. If not, a short review of the design before build often pays back the time.

Why does the MongoDB controller service matter for maintenance?

NiFi deprecated the Mongo URI property on PutMongo in version 1.22.0 and moved connection details to MongoDBControllerService implementations. Keeping the connection in one shared controller service means credentials, connection settings and TLS options are changed in one place, not in every processor. It also means a rotation or cluster move touches one component. The trade-off is that every flow using the service depends on it, so changes need to be tested against all dependent flows. Teams should confirm the service configuration against the current NiFi documentation for their version before they copy older flow examples.

How should batch size be set to protect the database?

PutMongoRecord controls batch size with the Batch Size property, so it is the main lever for write load on MongoDB. Larger batches reduce round trips but produce bigger bursts of writes and can hold locks or cache pressure longer on the database side. Smaller batches are gentler but slower and can increase NiFi overhead. The right value depends on document size, indexes, replica set topology and the other workloads on the cluster. Set it through load testing in a non-production environment that matches production sizing, and watch database metrics during the production run. Agree with the database owner on a maximum write rate before go-live.

What happens to deleted records, and who should handle them?

PutMongoRecord does not support deletes, so records removed at the source are not removed in MongoDB by that processor. The usual options are a tombstone or soft-delete field written by the source and filtered later, a periodic reconciliation job that compares key sets, or a separate process owned by the application team. Each option has different cost and accuracy trade-offs, and the choice affects reporting and compliance evidence. Decide this before build, record the owner and test it with real deletion scenarios. For data retention or erasure obligations, confirm the approach with your compliance lead, since DeployOpen prepares controls and evidence but does not issue audit reports.

What testing is needed before a MongoDB flow goes live?

Testing should cover schema mapping across all sample variants, insert and upsert behavior with duplicate keys, batch sizing under realistic load, failure and retry paths, and rerun safety. Teams should also test network interruptions between NiFi and MongoDB, and check behavior on a multi-node NiFi cluster so no document is processed twice. Verification should compare record counts and sampled documents between source and target. The test scope is one of the largest cost drivers, so ask a partner to list the test cases and the environments they run in, and to state what evidence will be delivered at the end.

How DeployOpen scopes and delivers this integration

Every MongoDB integration starts with a written scope agreed before any work begins. For this flow, that scope names the collections read with GetMongoRecord and written with PutMongoRecord, the record reader and writer pairs and schemas for each, the insert or upsert behavior per collection, the batch size, the MongoDB controller service and its connection settings, and the deliverables and acceptance checks. Because the scope states that PutMongoRecord does not delete, it also records how removals will be handled in the source system or in a separate process. Most engagements are fixed-scope, so the quote follows from this document. Ask for the list of collections, expected document volumes and the MongoDB deployment topology, since these change the effort more than the number of processors does.

Testing runs against a non-production MongoDB that matches the production version, authentication model and network path as closely as your environment allows. DeployOpen checks upsert keys against duplicate and missing-key cases, measures how batch size affects load on the database during test runs, and confirms that the shared MongoDB controller service behaves correctly when other flows reuse it. Confirm with your database team which replica set or sharded topology the test environment must reproduce, and who approves moving the final configuration to production.

Handover covers the flow definitions, the controller service configuration with credentials referenced rather than embedded, the schema files, the test records used for acceptance, and a runbook describing how to change batch size and how to recover from a failed write. Optional retainer support can cover NiFi and MongoDB upgrades, including checks of processor behavior after each NiFi 2.x release. Ask whether the retainer includes re-testing against new MongoDB versions, since that work is separate from routine NiFi patching.