Apache NiFi and Salesforce integration: what the flow does and where it stops
NiFi can extract Salesforce records incrementally into a warehouse, lake or operational system and write records back, and the supported path for that is the Salesforce processor bundle in NiFi 2. The reads use QuerySalesforceObject, which tracks progress with an age field so each run picks up only records changed since the last run, and the writes use PutSalesforceObject. Teams that want extract and write-back under NiFi control, rather than in a custom script, get scheduling, retries, provenance and routing around those two processors. Before designing anything, confirm the processor names, property options and state behavior against the NiFi component documentation for the exact NiFi 2 version you run. Processor properties have changed between releases, and the migration guidance for NiFi 2 is the place to check for changes that affect existing flows. Treat this section as a map of what to verify, not as a list of settings to copy.
The boundary matters as much as the capability. NiFi moves and transforms records, but the Salesforce data model, object and field permissions, licensing, and API limits remain owned by the Salesforce org and its administrators. A flow that works in a sandbox can still fail in production if an integration user lacks field-level access, if the org's API allocation is consumed by other applications, or if a custom object's age field is not indexed for the query your team needs. Ask the Salesforce admin to confirm these limits before the flow is scoped. The practical decision is who owns each failure mode. NiFi owns the flow logic, the state it keeps between runs, and the handling of records that fail to write. Salesforce owns the authoritative data, the rules that reject or change a record, and the quotas that throttle the integration. Agree on that split in writing, including who is alerted when a run fails, before the first production schedule is enabled.
What you need before the first Salesforce query runs
The Salesforce integration depends on five items that sit outside the NiFi flow itself: the runtime, the Salesforce identity, the OAuth setup, network paths and the target schema. Each card names who owns the item and the check that shows it is ready before the first QuerySalesforceObject run.
NiFi version and Java runtime
Confirm the NiFi release and JVM before building anything. The administration guide lists Java 21 as the minimum runtime for NiFi 2. The platform engineering lead owns this choice. Check it by starting a single node on the target build, confirming the java version reported in the bootstrap log matches the required runtime, and verifying the UI loads. Also confirm whether any optional Python-based processors are needed, since the guide describes them as beta.
Salesforce integration user
Create a dedicated integration user in the Salesforce org with read access only to the sObjects in scope, such as Account or Opportunity. The Salesforce administrator owns the profile and permission sets. Verify access by logging in as that user and running the exact SOQL the flow will use, confirming that the expected fields and the age field used for incremental loads are visible. Add write permissions only if PutSalesforceObject is in scope.
Connected app and OAuth configuration
The connected app, its OAuth scopes and the credential flow must be set up and confirmed inside your own Salesforce org. DeployOpen does not configure your org on your behalf without written scope. The Salesforce administrator and security owner should review the consumer key handling, the callback settings and the IP restrictions. Check the setup by obtaining a token with the integration user and recording the granted scopes against the list in the design document.
Secrets stored in a controller service
Consumer secrets, passwords and any private key material should live in a sensitive parameter or a secrets provider, not in processor properties written to the flow definition. The NiFi security owner decides where these values are held and who can read them. Check that the controller service shows the credential as protected and that a flow export does not reveal the value.
Network egress from every node
Every cluster node that may run the processor must reach the Salesforce API endpoints over HTTPS. Network and firewall teams own the egress rules, and proxy requirements must be captured in the design. Test from each node individually with a simple authenticated request, because a single blocked node will fail only when the primary node changes or is restarted.
Target schema agreed in advance
Before the first write-back or load, the data owner and the integration lead should agree the field mapping, record types, key columns and handling of deleted records in the target system. Confirm the mapping against a sample extract, including the age field used for incremental loads and how soft-deleted rows are represented. Without an agreed schema, the flow will pass records that the target silently rejects or reshapes.
Configuring the incremental Salesforce extract, step by step
This sequence builds a read path that pulls new and changed Salesforce sObject records on a schedule, proves the stored age-field state advances correctly, and only then adds the write path. Each step ends with a check so the team knows the flow is safe to promote.
Create the Salesforce credential and controller service
The NiFi administrator creates a dedicated Salesforce integration user with read access to the target sObjects and a connected app or other authentication method your Salesforce org supports, then configures the Salesforce controller service in NiFi with the login URL, API version and credentials. Store secrets as sensitive parameters rather than plain text in processor properties. Confirm the exact authentication options and API version support for your NiFi 2 release in the component documentation before you start. The step is done when the controller service reports enabled and no validation errors appear on the service.
Confirm the connection with a test query
Using a single sObject that the integration user can read, such as a small custom object in a sandbox, run a bounded SOQL query from a throwaway processor or the Salesforce tooling your team already uses to check field names, record counts and the age field you plan to use. This confirms that the user has field-level security on the age field, not only object-level access. The step is done when the test returns the expected record count and the age field values are populated and in the timezone you expect.
Add QuerySalesforceObject with the sObject and age field
Add QuerySalesforceObject to the canvas, point it at the Salesforce controller service, and set the sObject name and the age field that drives incremental loads. Leave the custom SOQL property empty for the first run so the processor builds the query from the age field, and record the field name in your flow documentation. The step is done when the processor validates without errors and a manual run against the sandbox object produces FlowFiles that carry the expected records.
Set primary node scheduling for the extract
In the processor scheduling settings, set execution to primary node only so that a clustered NiFi polls Salesforce from one node. Running the same extract on every node would duplicate pulls and race on the stored state. Confirm the scheduling on the processor and check that the cluster view shows the primary node, because primary node assignment can move during restarts. The step is done when the scheduling shows primary node execution and the processor runs on exactly one node in a multi-node test cluster.
Configure the record writer and output routing
Attach a record writer that matches the format your downstream consumer expects, such as JSON or Avro, and connect the success and failure relationships to separate destinations so bad records do not block good ones. Decide how the output should be partitioned or named, and add a way to inspect a sample FlowFile in the provenance view. The step is done when a sample FlowFile contains the expected fields, the record count matches a Salesforce query for the same time window, and failures route to the failure path without loss.
Verify the stored age-field state advances
Run the processor twice with new test records created between runs, then check the processor state in NiFi to confirm the stored age-field value moved forward to the latest record seen. Compare the record count from the second run with a Salesforce query bounded to the same time window to confirm that no records were skipped and none were repeated. Also test a soft-deleted record in the sandbox to confirm how your org's configuration surfaces it. The step is done when the state value advances as expected and the counts reconcile with the bounded query.
Add PutSalesforceObject after the read path is proven
Only after the extract reconciles should the team add PutSalesforceObject for write-back, using a sandbox sObject first and a separate integration user with the narrowest write permissions required. Map the record fields and the external ID or operation type deliberately, and confirm the behavior for each operation in the component documentation before you point the flow at production. The step is done when a test batch written from NiFi appears in the sandbox with the expected field values and the failure relationship receives any rejected rows with their error details.
How sObject fields map to record schema and target columns
This table compares each Salesforce concept with its NiFi record counterpart and the target column it should land in. Read it per object: confirm field types, nullability and picklist values against each org's metadata before fixing the record schema, because those details decide whether the writer succeeds.
| Salesforce concept | NiFi record representation | Target column or handling | What to confirm |
|---|---|---|---|
| sObject (for example Account or Opportunity) | One record type per sObject, defined by the record reader schema | One target table per sObject, named by your data model | Object API name and whether custom objects are included |
| Field API name | Record field keyed by the exact API name | Column name, mapped one to one or through a rename step | Custom fields end in __c and must be kept consistent across environments |
| Age field used for incremental loads | Read as the state key that QuerySalesforceObject stores between runs | Not written to the target unless you need an audit timestamp column | Which field the org uses for last modification and whether it is indexed |
| Salesforce record ID | 18-character ID string in the id field | Primary key of the target table, used as the upsert key | Whether your org exposes 15 or 18 character IDs in extracts |
| Soft-deleted records | Returned only when the query requests deleted rows, carried as their own records | Explicit boolean flag column, or a delete action in the write step | Whether your writer supports deletes for the target system |
| Picklist fields | Text value of the active picklist entry | Text column, or a lookup table keyed by picklist value | Active versus inactive values and any value changes over time |
| Relationship (lookup and master-detail) fields | Parent record ID or, with a relationship query, nested parent fields | Foreign key column, or flattened parent attributes in a separate table | Whether parents are loaded before children and how orphan IDs are handled |
| Date, datetime and currency fields | Typed values per the reader schema (date, timestamp, decimal) | Matching column types with a fixed decimal precision for currency | Time zone of datetime values and the currency fields in multicurrency orgs |
Choosing between full object reads, custom SOQL and write-back
Use this pattern when downstream systems need most fields of an sObject, such as Account or Opportunity, and a watermark on a date field is enough to pick up changes. QuerySalesforceObject reads the full object, tracks the last value of the age field in processor state, and emits only records newer than that value. It fits teams that want a simple, low-maintenance feed into a lakehouse or database and can accept the object's default field set.
- Watermark field: Confirm that the chosen age field, such as a last-modified timestamp, is populated and reliably updated for the object you extract.
- Soft-deleted records: Decide how deleted rows appear in the output and whether the target needs a tombstone process, because the watermark alone does not remove them downstream.
- Primary node scheduling: Run the extract on the primary node so that only one node holds the watermark and queries the org.
- API consumption: Check the org's remaining API allocation against the polling interval before setting the run schedule.
- Reset behavior: Test how the flow restarts after the state is cleared, since a full re-read can be large.
Before go-live, confirm the integration user's object and field permissions, and run a backfill in a sandbox to measure record volume and the time it takes to advance the watermark.
Use custom SOQL when the flow needs only some columns, filtered rows, or fields reached through parent or child relationships, which a full object read cannot express. The query is written by your team and passed to the processor, so the flow controls exactly what leaves Salesforce. This pattern suits teams that want narrow payloads for a warehouse table or need a status filter applied at the source.
- Selectivity: Confirm with Salesforce's documentation for your org edition that the WHERE clause is selective enough to avoid query limits and full table scans on large objects.
- Relationship fields: Test parent and child traversals against the expected record counts, since relationship queries can change the result shape.
- Watermark in the query: Make sure the age condition is part of the custom SOQL so incremental behavior is not lost when the query is edited.
- Field-level security: Verify that the integration user can read every field named in the SELECT clause, or the query will fail.
- Result mapping: Check that the downstream record schema matches the columns returned, including null handling for relationship fields.
Before go-live, confirm all custom SOQL limits and query behavior against Salesforce's own documentation for the org edition, and load-test the query with production-scale data volumes in a sandbox.
Use this pattern when an upstream system, such as a data quality process, a fulfillment system or an approval workflow, must create or update Salesforce records. PutSalesforceObject sends FlowFile content to the org as sObject records, so the flow must map each record to the correct object, identifier and operation. Writes change production business data, so the design needs stricter controls than a read-only extract.
- Operation and key: Decide whether each record is an insert or an update, and confirm which field identifies the target record in the org.
- Object permissions: Confirm that the integration user has create and edit rights on each target object and on the fields being written.
- Validation rules and triggers: Test writes in a sandbox, because validation rules, flows and triggers can reject or alter records and produce partial success.
- Error routing: Send rejected records to a failure path with the Salesforce error message attached, and avoid retrying blindly.
- Duplicate protection: Ensure the upstream source cannot resend the same change repeatedly, or the target will receive repeated updates.
Before go-live, test with representative records that include invalid values, duplicates and deleted targets, and confirm the rollback or correction plan with the business owner of each object.
How age-field incremental loads and soft-deleted records behave
The age field decides what QuerySalesforceObject treats as new or changed, so it is the first design decision in this integration. The processor stores the newest value it has seen and, on each run, asks Salesforce for records whose age field is later than that stored value. Pick a field that Salesforce sets or updates on every change that matters to your target, such as LastModifiedDate on the sObject you query, and confirm in the org that no automation, integration user or data load updates records without touching that field. If a change does not move the field, the record is silently skipped. If the field moves for changes you do not care about, the flow reprocesses records that did not need it.
Clock and time zone differences cause the quiet failures. Salesforce stores and compares timestamps in its own time handling, and the NiFi host has its own clock and time zone settings. Keep NiFi hosts on time synchronization, compare a record's timestamp in the org with the value NiFi stores after a test run, and use a small overlap window in your query logic so that records committed right at a boundary are picked up on the next run. The overlap produces some duplicates, so the target must be idempotent, for example by upserting on the Salesforce Id. Confirm the exact behavior of the age field and any overlap settings in the processor documentation for your NiFi 2 release before you rely on them.
Soft-deleted records need their own path. Deleted records are not returned as ordinary rows by a standard query, so a flow that only reads changed records will never tell the target that a record has gone. Decide whether the target should mark deletions, remove rows, or ignore them, then confirm with your Salesforce administrator how deleted and archived records are exposed through the API for the objects in scope, and whether the processor can retrieve them. Schedule a periodic reconciliation run that compares the set of Ids in the target with the set in Salesforce and corrects drift. For any state reset, write a documented procedure that records the stored age value, the reason for the reset, the expected reload window and the person who approves it.
Securing the Salesforce connection and the sObject data it carries
The Salesforce connection gives a NiFi flow standing access to customer records, so its security depends on several controls working together. Credential storage, the Salesforce user's permissions, NiFi's own authorization model, transport encryption, logging discipline and version control each produce different evidence, and an auditor will ask for each one.
Credential storage for Salesforce secrets
The Salesforce controller service holds the credentials or connected-app details used to authenticate. Store these as sensitive property values or source them from a parameter provider so they are not visible in exported flow definitions or plain text in configuration. Check that the flow definition export contains no secret values, and ask for the parameter provider configuration and its access list as evidence. Rotation steps should be written down and tested before the first production run.
Dedicated integration user with least privilege
Create a Salesforce user for the integration only, with read access limited to the sObjects and fields the flow needs and no access to unrelated objects or administrative permissions. For PutSalesforceObject, grant create or update rights only on the target objects. Evidence to request includes the profile and permission set assignments, field-level security exports and a record of who approved the grant. Confirm the exact permissions your Salesforce edition requires with Salesforce documentation before you finalize the role.
NiFi authorization on Salesforce components
Restrict who can view and change the Salesforce controller service and the QuerySalesforceObject and PutSalesforceObject properties, including the SOQL custom query and the age-field settings that determine which records are pulled. Use NiFi's policies to separate operators who run and monitor the flow from developers who edit it. Evidence includes the exported policy list, group membership and a sample of audit events showing component modifications.
TLS on NiFi and the Salesforce connection
Serve the NiFi UI and API over TLS with certificates issued by your certificate authority, and confirm that the Salesforce endpoints are reached over HTTPS with certificate validation enabled. Avoid disabling hostname or chain checks in proxy or trust store settings, since that removes the protection the connection depends on. Request the NiFi TLS configuration, the trust store inventory and a test showing a failed connection when the certificate is invalid.
Logs and bulletins without personal data
Processor bulletins, logs and provenance events can display field values from failed records, including names, email addresses and contact details. Configure failure routes to avoid logging full record content at warning or error levels, and limit what error messages copy from the response body. Review provenance retention and who can view record content. Evidence to request includes a sample bulletin and log extract from a failed test run, and the logging configuration with its retention settings.
Flow definition review under version control
Store the flow definition in a Git repository through a Flow Registry Client, and require review before a changed Salesforce query, field list or target object reaches production. Check that each change carries an author, an approver and a reason. Auditors typically ask for the commit history, the approval records for changes to the Salesforce components and proof that production matches the approved version.
Common Salesforce failures in NiFi and how to fix them
This table lists the failures most often seen when NiFi extracts or writes Salesforce records, with the likely cause, the first diagnostic step and the fix for each. Read it as a triage sheet: start with the symptom that matches what your flow is doing, then confirm the cause in NiFi's processor status and bulletins before changing configuration.
| Failure | Likely cause and diagnostic step | Fix |
|---|---|---|
| Authentication errors at startup or after a period of success | The connected app's OAuth credentials or the integration user's certificate or password have expired or been rotated, or the login URL points to the wrong org. Check the processor bulletin for the authentication error text and confirm the Salesforce Connected App settings and the Salesforce controller service properties. | Rotate or reissue the connected app credentials, update the controller service, and confirm the login URL matches the target org (production or sandbox). Re-test the controller service before restarting the flow. |
| Insufficient access on objects or fields | The integration user's profile or permission set lacks read access to the sObject or specific fields, so queries fail or fields come back empty. Run the same SOQL as that user in Workbench or Developer Console to confirm. | Grant read access on the object and the required fields through a permission set assigned to the integration user, and review field-level security after each new field is added to the query. |
| Incremental loads skip records after a state reset | The stored age-field high-water mark was cleared, or the flow was pointed at a new processor ID or cluster, so the stored state no longer matches the last loaded timestamp. Compare the state value with the newest record already in the target. | Restore a known-good timestamp in the processor state or set a deliberate starting date, and document the reset in the change record. Avoid clearing state on a live flow without a reconciliation query. |
| Incremental loads repeat or skip records because of clock drift | Records stamped by Salesforce are compared with a timestamp on the NiFi side, or NiFi nodes disagree on time, so records near the boundary are read twice or missed. Check NTP status on every node and compare the lag between the newest record and the stored value. | Synchronize clocks on all cluster nodes, keep the age field as the single source of truth, and add a deduplication step keyed on the Salesforce record ID in the downstream write. |
| Duplicate processing after a primary node change | Scheduling is set to run on the primary node, and a new primary picks up the query before the previous primary's state is committed. Look for repeated record IDs in the target around the time of the election event in the cluster logs. | Keep primary node scheduling, use idempotent writes keyed on record ID, and confirm the state is stored in cluster-wide state before relying on it. Test a forced primary change in a non-production cluster. |
| SOQL exceeds limits or times out on large objects | Large sObjects return too many rows per query, the query selects many fields, or a non-indexed filter forces a full scan. Check the processor bulletin for limit or timeout messages and the number of records per run. | Narrow the field list to what the flow needs, add a selective filter on an indexed field such as the age field, and lower the batch size. Confirm current governor limits for the org edition in Salesforce documentation before setting the batch. |
| Write-back rejected because of required fields | PutSalesforceObject sends records that omit a field required by the object or by a page layout validation, and Salesforce returns a REQUIRED_FIELD_MISSING style error. Check the failed FlowFile's error attribute and the object's field metadata. | Map every required field in the record path or reader schema, add a default or validation route for missing values, and send failed records to a retry or error queue with their error text. |
| Write-back rejected by validation rules or trigger logic | Validation rules, duplicate rules or Apex triggers reject values that are valid in NiFi, such as a picklist value outside the allowed list or a field that a trigger locks after creation. Reproduce one rejected record manually in the org sandbox to read the rule message. | Coordinate with the Salesforce administrator to align the source values with picklists, adjust the record transformation, and agree which rule overrides, if any, the integration may use. Do not disable rules in production to make the flow pass. |
| Wrong record type or lookup ID on write | The external values are mapped to Salesforce IDs that do not exist in the target org, usually because sandbox and production IDs were mixed. Check that lookup fields resolve to IDs from the same org before the write step. | Maintain a versioned mapping table per environment, validate lookup IDs before PutSalesforceObject, and route unmatched records to a review queue rather than writing partial data. |
What a Salesforce NiFi integration costs and what it takes
What drives the cost of a Salesforce NiFi integration?
The biggest drivers are how many sObjects are in scope, how much data changes each day, and how complex the target schema is. Each additional object adds its own incremental field, mapping, testing and monitoring work. Custom SOQL queries add effort because they need validation against Salesforce API behavior and tuning so they do not hit org limits. Write-back to Salesforce with PutSalesforceObject adds validation, duplicate handling and error routing, and governance requirements such as field-level access review, logging and change approval add further time. Ask for the object list, expected record volumes and target schema before comparing any numbers.
Which parts of a Salesforce NiFi integration are one-time and which recur?
The build is one-time: designing extraction and write-back flows, configuring the Salesforce and target connections, setting incremental state, and testing against a sandbox. Recurring work includes rotating credentials and certificates for the connected app, reviewing flows after Salesforce releases change objects or fields, adjusting queries as data grows, and upgrading NiFi itself. NiFi 2 upgrades and any move to a newer NiFi line also recur over the life of the flow. Ask the provider to separate build work from a support or maintenance arrangement in the quote.
How long does a Salesforce NiFi integration take to deliver?
Duration depends on the number of objects, how clean the Salesforce data is, how many target systems are involved, and how quickly decisions are made. A single object with a stable age field and one target is much shorter than a multi-object program with write-back and soft-delete handling. Sandbox access, the time needed for a Salesforce administrator to confirm field meanings, and the target owner's review cycle often set the critical path more than NiFi configuration does. Request a written timeline tied to the agreed scope and the dependencies it lists.
What does the client need to provide for this work?
You need an org administrator who can create or review a connected app, grant an integration user the right object and field permissions, and provide sandbox and production access. A Salesforce owner should make decisions about which objects, fields, record types and deleted-record behavior matter to the business. Target system owners must confirm the schema, keys, write rules and acceptance tests for their side. You also need a place to host NiFi that meets your network and security rules, and a named person who approves production changes.
Can our own team build and run this integration without a partner?
Yes, a team with NiFi, Salesforce API and integration experience can build it internally, and the NiFi documentation covers the processors involved. The risks are less about the first build and more about the details: incremental loads that miss records, soft-deleted records that never reach the target, write-back that creates duplicates, and state that is lost during a cluster change. Consider whether your team has the time to test these cases and keep them working through Salesforce releases and NiFi upgrades. A partner can take on part of the build or the test plan while your staff keep ownership.
What should we ask for in a written quote?
Ask for a scope statement that lists each sObject, the incremental field used for each, how soft-deleted records are handled, and whether custom SOQL is included. Request the deliverables, including flow definitions, documentation, test evidence and handover steps, and the acceptance criteria your team will sign off. Ask which items are fixed scope and which are assumptions, what the client must provide and by when, and how changes to scope are priced. Also ask whether ongoing support, credential rotation and upgrades are quoted separately and what they cover. DeployOpen plans and delivers work from a written scope and does not publish a price list.
Which Salesforce and NiFi details should engineers verify before build?
Engineers should confirm the Salesforce API version the processors use, the integration user's object and field permissions, and whether the age field used for incremental loads is reliably updated by every process that changes a record. They should test how QuerySalesforceObject behaves with soft-deleted records, confirm that primary node scheduling is set so that only one node polls Salesforce, and check that custom SOQL stays within the org's limits. Confirm the NiFi version and its supported Java runtime against the NiFi administration guide before deployment.
How DeployOpen scopes and delivers the Salesforce integration
DeployOpen plans, builds, documents and supports Apache NiFi flows that extract Salesforce sObject records and write records back, inside infrastructure you control. Engagements are fixed-scope, so the Salesforce objects, incremental fields, write rules and handover material are agreed in writing before build work starts.
How we scope
- Inventory the Salesforce objects, fields and volumes that each flow must read or write.
- Confirm the age field, soft-delete handling and expected change patterns for every incremental load.
- Agree in writing which flows run on the primary node and which NiFi cluster and network paths they use.
How we build
- Configure Salesforce credentials, connection services and NiFi parameters so secrets stay out of flow definitions.
- Build QuerySalesforceObject and PutSalesforceObject flows with tested SOQL limits and record-level error routing.
- Test incremental state, restarts and duplicate handling against representative Salesforce sandbox data before go-live.
How we hand over
- Deliver flow documentation covering object mappings, state behavior, schedules and known Salesforce API limits.
- Version the flows in a Git repository and document the promotion steps between environments you operate.
- Train your operators on monitoring, replaying failed records and reviewing provenance for each Salesforce flow.
How we support
- Offer ongoing support, NiFi upgrades and flow changes as a retainer, scoped separately from the build.
- Review Salesforce API version changes and NiFi release notes for impact on running flows.
- Run security testing of the NiFi deployment and its Salesforce connection when your organization requests it.
