Apache NiFi and Google BigQuery integration

How NiFi loads records into BigQuery with PutBigQuery

A team that wants NiFi-sourced records in BigQuery tables within a known latency, and without duplicates, needs to make two decisions before building the flow: how rows reach the table and how the processor proves each row was written once. PutBigQuery is the NiFi 2 processor that handles both. It writes FlowFile contents to a BigQuery table through the gRPC-based BigQuery Write API, using protocol buffers, and it supports both a streaming approach and a batching approach. Streaming makes appended rows readable in the table almost immediately, which suits latency-sensitive dashboards and alerting. Batching suits high-volume loads where a short delay is acceptable. Confirm the current throughput and quota limits for each approach against Google's BigQuery documentation for your project and region, since those limits govern how you size batches and concurrent tasks.

Delivery guarantees come from stream offsets rather than from NiFi's own retry behavior. The processor is designed to deliver exactly once through these offsets, so a retried write does not append the same rows twice. That guarantee holds only when the flow feeds the processor consistently, so plan how FlowFiles are split, how failed writes are routed, and how the processor behaves after a node restart in a cluster. Test these paths on a non-production dataset by forcing a failure mid-batch and counting rows afterward. Ask your BigQuery administrator which service account and dataset permissions the processor needs before the test, because a permission error mid-run can look like a delivery problem.

The schema for every write comes from the configured Record Reader, which is the component that decides how incoming records are parsed and typed. The processor skips attributes that do not match the target table schema, and the documentation for PutBigQuery describes this as a silent skip, so a misnamed or mistyped field can disappear without an error. Treat the Record Reader and the target table as one contract. Compare the reader's inferred or declared schema with the table definition during design, and add a row count or sample-field check after each load. Those checks are the practical way to catch the drops before downstream reports do.

What you need before the first PutBigQuery run

A PutBigQuery flow depends on four layers agreed before any processor is configured: the NiFi runtime, the Google Cloud project and identity, the network path to Google APIs, and the destination table. Confirm each layer with the teams that own it, because a gap in any one of them tends to surface as a failed run rather than a clear error.

NiFi 2 runtime with Java 21

The NiFi administration guide lists Java 21 as the minimum runtime for NiFi 2. Check the JDK on every cluster node, not only the one you use to build flows, and confirm that the NiFi version you deploy is a 2.x release. Flows built on 1.x need the upgrade steps described in the migration guidance before they can use the 2.x processor bundles.

Google Cloud project with BigQuery enabled

Create or select a Google Cloud project that will hold the target dataset and confirm that the BigQuery API and the BigQuery Storage Write API are enabled for it. Ask your cloud team which project is billed for streaming and batch loads, since ingestion volume will show up on that project's bill and its quotas will limit how fast NiFi can write.

Service account with dataset write access

Create a dedicated service account for the NiFi cluster and grant it only the write permissions needed on the target dataset. Confirm the exact IAM roles against Google's current BigQuery documentation before you grant them, because role names and their permissions change over time. Store the key material through a credential mechanism your security team approves, and avoid sharing one account across unrelated flows.

Egress to Google APIs

Every node that runs PutBigQuery must reach Google's BigQuery endpoints over gRPC on the network path your firewall and proxy allow. Test this from each node, not from a workstation, and check whether outbound traffic passes through a proxy or NAT gateway with its own limits. In restricted or air-gapped segments, this is the item most likely to block the project.

Target table created in advance

PutBigQuery takes its schema from the configured Record Reader, but the destination table must already exist in BigQuery. Create it with column names, types and modes that match the records your reader will produce, and agree on who owns schema changes. Fields that do not match the table are skipped without an error, so review the table definition against a sample record before the first production run.

Configuring PutBigQuery step by step

This sequence takes a NiFi flow from a credentials controller service to a verified load of records into a BigQuery table, with a check at each stage so schema and delivery problems surface before production traffic arrives.

Create the GCP credentials controller service
An administrator adds a Google Cloud credentials controller service to the NiFi canvas and supplies the authentication method your security team has approved for the service account. Use a dedicated service account that holds only the BigQuery roles needed to write to the target dataset, and confirm with your cloud team which credential source is permitted in your environment. The step is complete when the service enables without error and the flow can reference it from PutBigQuery.
Choose and configure the record reader
Select a Record Reader that matches the format of your FlowFile contents, such as JSON or Avro, and attach it to PutBigQuery. The reader is the source of the schema, so the field names and types it produces are what PutBigQuery compares against the target table. Review the schema access strategy in the reader and confirm that it reads the incoming data as you expect. The check is a provenance or data viewer inspection of a sample FlowFile showing every field and type the reader parses.
Set the project, dataset and table
Enter the Google Cloud project ID, dataset and target table name in the PutBigQuery properties, using parameters or variables if the same flow will run across environments. Confirm that the table already exists with the column names and types you intend, because this processor writes into existing tables rather than designing them for you. The step is done when the processor validates with no errors on these properties and the table reference resolves in your test project.
Select streaming or batch writes
Choose the write approach that fits your latency and volume needs: streaming, where appended rows are readable in the table immediately, or batching, which groups writes through the same Storage Write API. Review the processor's additional details before deciding and note which mode you chose in the flow documentation, since the two behave differently for freshness and for how stream offsets are tracked. The check is a short test run in each mode you are considering, with row visibility compared against the expected timing.
Set failure and success relationships
Route the success relationship to the next stage or to a terminal step, and route failure to a dedicated error path with a queue and alert rather than letting FlowFiles back up silently. Add a retry or dead-letter path for transient Google Cloud errors so that a temporary outage does not discard records. The step is complete when every relationship is connected or auto-terminated on purpose, and the flow passes NiFi validation with no warnings about unconnected relationships.
Run a test FlowFile with matching data
Send a small, known FlowFile whose fields match the target table exactly through the processor. Query the BigQuery table to confirm the rows appear, count them against the input, and compare the values for types and precision. The check passes when the row count and values match the source record and the FlowFile reaches the success relationship.
Verify that mismatched fields are reported
Deliberately send a test FlowFile that includes one field the table does not define, and another with a type that conflicts with the table column. Because PutBigQuery skips attributes that do not match the target schema, confirm in the processor's logs, provenance and the table contents that the unmatched field was dropped and not written, and decide whether your process should treat that as a failure. Record the result so that the team knows which mismatches your flow will tolerate and which should stop the load.
Document the validated configuration
Capture the credential method, reader choice, table reference, write mode, relationship handling and the results of the match and mismatch tests in the flow documentation or version control. Store the flow in your Git-backed registry if your team uses one, so that later changes to the schema or write mode are reviewed. The step is complete when another engineer can reproduce the test results from the written record alone.

Streaming or batch writes: choosing the mode for your latency and cost needs

The streaming approach appends rows to the table as they arrive, and those rows are readable immediately after they are written. This suits dashboards, alerting queries and operational lookups that need recent data without waiting for a load cycle. PutBigQuery uses the gRPC-based BigQuery Write API with protocol buffers for this path, so the cost and quota profile follows the Write API rather than a file-based load job.

Read latency: Rows become queryable soon after the append completes, which is the main reason to choose this mode.
Load pattern: Many small, frequent appends. Flow design should keep batches of FlowFiles at a size the table can absorb without excess per-request overhead.
Failed writes: Retries and stream offsets determine whether a failed append can be replayed without duplicates. Test replay on a non-production table first.
Cost behavior: Charges and quotas for streaming inserts depend on the current BigQuery pricing and limits, which change over time.
Data freshness trade-off: Rows written this way may not be available for every operation immediately, so confirm the consistency behavior your queries rely on.

Before committing to this mode, confirm current BigQuery quotas, pricing and write behavior with Google, since these values are not fixed by NiFi.

How the record reader decides your BigQuery schema

PutBigQuery takes its schema from the Record Reader configured on the processor, not from the target table. The reader parses each FlowFile into records, and the field names and types it produces are what PutBigQuery compares against the destination table. Field names and types in the reader must match the table exactly, so the reader's schema is the first thing to check when you design a flow. Attributes on the FlowFile are not part of this comparison. The NiFi PutBigQuery documentation states that attributes that do not match the target schema are skipped, so a value that lives only in an attribute never reaches the table. Put any value you need into the record content before the write.

Nested and repeated fields need the same care. A BigQuery RECORD column corresponds to a nested structure in the reader schema, and a REPEATED column corresponds to an array. If the reader infers a flat string where the table expects a nested record, or a single value where it expects a list, the mismatch is handled by skipping rather than by an error that stops the flow, which is why data can disappear quietly. Timestamp and numeric alignment is the other frequent source of drift. Confirm that source timestamps are parsed into the type your table uses, and that decimal or integer widths in the reader do not truncate values the table stores with greater precision. Confirm the exact type mapping for your table definition against the BigQuery documentation and your own sample data.

Validate the reader before connecting the flow to production tables. Take a representative sample file, including edge cases such as nulls, empty arrays, nested objects, and maximum-length timestamps, and run it through PutBigQuery into a scratch table with the same schema. Then compare row counts and column values in the scratch table against the source, and check the FlowFile provenance and processor logs for skipped fields. Repeat this check whenever the source system adds a column or the table schema changes. A write that succeeds is not evidence that every field arrived, so the comparison against the sample is the control that catches silent drops.

Mapping record fields to BigQuery columns

This table compares how each kind of source field behaves when PutBigQuery writes it to a table, from the record reader through to the column. Read it before you build the schema check, because the processor skips attributes that do not match the target schema and does not raise an error for them. The main takeaway is that a clean run does not prove every field arrived, so confirm column counts and sample values against the source.

Source field	Reader and schema behavior	BigQuery column and nulls	On mismatch
Matching top-level field	Record Reader supplies the name and type, which PutBigQuery uses for the write	Column of the same name and compatible type is populated	Written as expected if types are compatible
Field absent from the target table	Read by the Record Reader but has no column to receive it	No column is created or populated	Silently skipped, so confirm the table contains every field you expect
Field name differing only by case or spelling	Name must match the target schema as the reader presents it	Matches only when names align exactly	Treat as absent and skipped, so align names upstream
Nested record or repeated field	Reader must present the structure in the shape the table expects	Maps to a RECORD or REPEATED column when the schema agrees	Structure mismatches need testing with sample records before go-live
Numeric string sent to an integer or numeric column	Reader type is string, while the column is numeric	Coercion must be verified with sample values, including decimals and large integers	Confirm behavior with a test load rather than assuming conversion
Timestamp or date text	Reader format determines how the value is parsed	Column type TIMESTAMP or DATE needs a consistent format and time zone	Confirm format handling with representative values
Null or missing value	Reader may return null, or the key may be absent in some records	Nullable columns accept nulls, required columns reject them	Confirm how the target treats required columns for your records

Exactly-once delivery and stream offsets explained

PutBigQuery achieves exactly-once delivery by writing through the Storage Write API with stream offsets, so each append carries a position in the stream that BigQuery can use to recognize a row it has already accepted. The offset is the guard against duplicates, not the NiFi FlowFile itself. When a request times out, NiFi does not know whether BigQuery committed the rows, so the processor retries the append at the same offset. If BigQuery already holds rows at that offset, the retry is treated as a repeat rather than a second insert. This only works if the stream is written in order and the offset sequence is not reset between attempts, which is why the processor's stream handling matters more than the flow around it.

Duplicates can still reach the table in several situations. A FlowFile that is routed to a success relationship and then re-sent by an upstream process, such as a replayed queue or a manual restart of an earlier flow, creates a new stream position and a new write. Changing the target table, the stream type, or the record schema between runs can also break the offset chain. Operators who restart NiFi nodes during a write, or who copy FlowFiles from provenance and resubmit them, should expect the same records to arrive twice unless the flow has its own key-based deduplication. Confirm with Google's Storage Write API documentation and the PutBigQuery component documentation exactly which error codes trigger retries in your NiFi version.

Test restart and failure behavior in your own environment before go-live. Use a non-production dataset and a known record set, then force a timeout, stop a node mid-batch, restart NiFi during a write, and replay a queued FlowFile. After each test, count rows per key and compare them with the source. Record the processor's retry settings, the stream type in use, and the failure relationship behavior so the same test can be repeated after upgrades. Ask your DeployOpen team or your own platform owners to document these results as part of the acceptance evidence for the integration.

Securing the BigQuery connection from NiFi

A PutBigQuery flow is only as safe as the identity it writes with and the places that identity can be seen or used. The cards below cover the controls that sit around the processor: the service account, where its key lives, who can read it inside NiFi, and where the cluster is allowed to connect.

Scope the service account to one dataset

Create a dedicated Google Cloud service account for this flow and grant it only the roles needed to write to the target dataset, rather than project-wide data roles. Review whether it needs table creation rights or should only append to tables that already exist and match the schema. If the flow also reads from BigQuery for checks, use a separate account with read-only access. Confirm the exact roles with your Google Cloud administrator, since the Storage Write API needs specific permissions that should be verified in a test project before production.

Keep the key out of flow definitions

The service account key is a long-lived credential, so it should be entered into a sensitive property on the GCP credentials controller service rather than pasted into a processor or parameter with plain-text visibility. Where your organization already runs a secrets manager, decide whether NiFi reads the value through a parameter provider or whether the key is rotated by an operator outside NiFi. Ask who holds the original key file and whether it is deleted from workstations after upload.

Limit who can view controller services

Controller services hold the BigQuery credentials, so NiFi access policies matter as much as the key itself. Restrict the view and modify policies on the controller service and its process group to a small operator group, and give developers read access to flow structure without access to sensitive values. Confirm which authentication provider your deployment uses, such as LDAP, OpenID Connect or SAML 2.0, and that group membership maps cleanly to these policies before the flow goes live.

Authenticate NiFi users strongly

Access to a NiFi canvas can expose the flow and its credentials, so the login path needs to be strong. NiFi supports single-user, LDAP, OpenID Connect, SAML 2.0 and X.509 client certificate authentication, and the Kerberos provider is deprecated. Choose the provider that matches your identity system, enforce TLS on the UI, and confirm that single-user mode is not left in place for shared environments.

Control network egress from the cluster

Every NiFi node that runs PutBigQuery needs outbound access to Google APIs, so egress should be allowed only to the endpoints the flow needs and only from the nodes that need them. Decide whether traffic goes through a proxy or firewall with allow lists, and whether private Google access is available in your network design. Test the path from each node, since a failed node will show up as delivery errors rather than a clear network alert.

Verify what the flow is allowed to write

Credential scope limits who can write, but it does not stop a flow from sending data it should not hold. Review which attributes and record fields reach PutBigQuery, using a record reader that selects only the columns the table is meant to hold. Ask your security team to confirm the dataset location, retention, and any data classification rules that apply before the first load, and record that review as evidence for later audits.

Why rows are missing or loads fail

This table maps common PutBigQuery failures to their likely causes and fixes, so you can tell a silent data loss from a loud error. Read it before go-live and again whenever a load stops matching the source. The main takeaway is that PutBigQuery skips attributes that do not match the target schema without failing the FlowFile, so schema alignment must be verified on purpose.

Failure	Likely cause	Fix
Fields silently missing in BigQuery	Attributes or record fields do not match a column in the target table, and PutBigQuery skips them rather than failing the FlowFile	Compare the Record Reader schema with the table schema, rename or cast fields, and check row counts after a test load
Schema mismatch on write	The schema comes from the configured Record Reader, so a reader schema that differs from the table (types, nested fields, mode) produces unexpected results	Generate the reader schema from the target table definition and pin it, rather than inferring it from sample data
Permission denied	The service account used by the GCP credentials lacks write access to the dataset or table, or the target project is wrong	Grant the write role on the specific dataset or table to the service account and confirm the project ID in the processor properties
Rows not visible after a write	Rows written through the streaming approach are readable immediately, but a batch stream has not been finalized and committed yet	Confirm the write mode and finalize and commit the batch stream before querying, then retest with a known row count
Quota or throttling errors	The project hits Storage Write API limits when batch sizes are too small or concurrency is high across cluster nodes	Review current BigQuery quotas in the project, enlarge FlowFile batches, and limit concurrent tasks (confirm limits with Google)
Duplicate rows after a manual replay	A FlowFile is resent outside the stream offset sequence, so the same records are appended again	Replay only through the failure path that preserves stream offsets, and check row counts against the source before any manual rerun

Testing the pipeline before production traffic

Run a fixed set of test cases against a non-production BigQuery dataset before any live source is connected to PutBigQuery. Start with a schema-matched file that carries every column the target table expects, so you can confirm the record reader maps types and field names the way you intend. Then send a file with an extra field that the table does not define. PutBigQuery skips attributes that do not match the target schema, so this test shows you whether the dropped field is acceptable or whether it signals a mapping error upstream. Next, send a file with a wrong type in one column, such as a string where the table expects an integer. Record the exact FlowFile route and any failure relationship output, and confirm the bad record does not land silently in the table.

Two failure tests come next. Force an error partway through a batch, for example by revoking write permission on the table or by stopping network access for a short window, and confirm that the FlowFile is routed to failure and that retries do not create duplicate rows. Then restart the NiFi node and replay the same input, checking that stream offset handling produces the same row set as an uninterrupted run. Ask your team to confirm the exact retry and offset behavior for the NiFi version you deploy, since the component documentation is the reference for those settings.

Keep the evidence in a form change management can review: the flow version from your registry, the parameter values used, the input files with checksums, the NiFi bulletin and provenance exports for each test, and the BigQuery job or table metadata captured before and after each run. Compare row counts between the source and the table for every test, and for production loads, reconcile counts per batch rather than only in total. Store the comparison query and its output with the test record so an auditor can rerun it.

Questions buyers ask about NiFi and BigQuery

What drives the cost of a NiFi to BigQuery integration?

The cost follows the number of source feeds, the number of target tables, how much schema work each table needs, and how many error paths you want handled before go-live. A single append-only table fed from one source is a small scope. Several tables with nested records, type conversions and reconciliation checks are a larger one. Ongoing costs are your cloud usage for BigQuery and the NiFi hosting, which you pay to your own cloud provider, plus any support retainer you choose. Ask for a quote that lists each table, each source, the write mode for each, and who owns testing and the cutover.

How long does it take to get the first load into BigQuery?

The first load depends on how quickly the required access and a representative sample of data arrive, more than on the NiFi work itself. Timing is set in the written scope once the target dataset, table schema and sample records are confirmed. The biggest delays usually come from waiting for dataset owners to approve IAM roles, agreeing on how source types map to table columns, and getting realistic sample files for testing. Ask for milestones that separate a working test load from a production load with monitoring.

What do we need to provide before work starts?

You need a Google Cloud project where the BigQuery datasets live, a named owner for each dataset who can approve schema changes, and a service identity NiFi can use with the permissions your security team accepts. You also need sample data that reflects production, including edge cases such as nulls, nested fields and unexpected types. Provide the target table definitions if they exist, and the network path from the NiFi cluster to Google APIs. Confirm the exact roles and the credential method with your cloud team before the build starts.

Can our own team build this without a partner?

Yes, a team with NiFi and BigQuery experience can build this alone. The parts that usually need the most care are the record reader and schema setup, the choice between streaming and batch writes, and the testing of fields that silently fail to match the table. Your team should plan time for those checks and for operating the flow after launch. Many teams bring in a partner for the first flow so the patterns are set correctly and their own engineers can extend them later.

How does a fixed-scope engagement handle this work?

DeployOpen agrees scope, deliverables and timelines in writing before work starts, and most engagements for this kind of integration are fixed-scope. The written scope lists the sources, target tables, write mode, error handling, test evidence and handover items. Changes such as an extra table or a new source are documented as a scope change with their own effort. Ongoing monitoring, upgrades and tuning can be covered by a retainer if you want continued support after handover.

Why can fields disappear from BigQuery rows without an error?

PutBigQuery skips attributes that do not match the target schema, so a field name or type that differs from the table column will not be written and may not raise a failure. The record reader defines the schema the processor uses, so a mismatch between that schema and the table is the usual cause. Test with records that include every expected field, compare row counts and column contents in BigQuery after each run, and alert on unexpected drops. Confirm the behavior against the PutBigQuery documentation for your NiFi version.

Does the streaming or batch write mode change the cost and the behavior?

PutBigQuery uses the BigQuery Write API and supports a streaming approach, where appended rows are readable immediately, and a batching approach. Streaming gives faster visibility for downstream queries, while batching is often chosen when freshness is less important. The mode also changes how exactly-once delivery is reasoned about through stream offsets, and how retries behave on failure. Ask your engineers to test both modes against your data volumes and confirm the BigQuery pricing that applies to each in the Google Cloud documentation.

Which NiFi version should we use for this integration?

Use NiFi 2, because the project states that 1.x cannot be maintained further and recommends upgrading to 2. The NiFi downloads page lists NiFi 2.12.0 as released on September 13, 2026, and the project's 1.28.1 end-of-support date was December 8, 2024. Confirm the current release and the processor behavior in the NiFi components documentation before you pin a version, and plan a test cluster run before production.

How DeployOpen scopes and delivers BigQuery loads

DeployOpen plans and builds Apache NiFi flows that load records into Google BigQuery tables in infrastructure you control, starting from a written scope. We design the record reader, the write mode and the table mapping together, then test them against your real table schemas before handover.

How we scope

We confirm which source systems feed BigQuery, the target datasets and tables, and the expected record shapes.
We document whether each load needs immediate row visibility or can use batched writes through the Storage Write API.
We agree in writing how schema drift, rejected fields and duplicate handling are monitored and who owns each decision.

How we build

We build the PutBigQuery flow with a record reader matched to the source format and check the schema it derives.
We test attribute and field mapping against the table definition so unmatched columns are caught before production.
We configure service account credentials, stream offset handling and error routing, and test restarts and retries in a non-production environment.

How we hand over

We deliver the flow, parameters and controller service configuration with version-controlled flow definitions.
We document field mappings, expected drop behavior, reconciliation checks and the runbook for failed or rejected loads.
We walk your operators through verification queries and the steps to confirm row counts after a restart.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Retainer work can cover NiFi version upgrades, schema changes in BigQuery tables and periodic reconciliation reviews.
Any change to scope, tables or delivery mode is agreed in writing before the work starts.