DuckDB vs Apache Spark: when one server is enough
Choose DuckDB when one large server can hold the working set and the analytics are run by a small group, reading Parquet, Iceberg or Delta data directly from object storage or a file share. DuckDB is embedded in the host process, so there is no database server to install, patch or back up separately, and its core extensions include the iceberg and delta table format readers. Choose Apache Spark when the work outgrows one machine, when you need distributed batch jobs, Structured Streaming, or a shared cluster that several teams submit to through its supported cluster managers: standalone, Hadoop YARN or Kubernetes.
The decision turns on four questions. First, how much data does a typical job scan and how much memory does its largest join or aggregation need on the biggest server you are willing to run? Second, how many people or services must write concurrently: DuckDB allows one read-write process per database file, while several processes can read a file in READ_ONLY mode, so multi-writer pipelines need a different design. Third, do you have streaming requirements, which DuckDB does not address. Fourth, who will operate the platform? Spark needs cluster administration, and the Apache Spark cluster overview lists only the cluster managers, so notebooks, scheduling, catalog and access control must be assembled from other tools.
Before committing, run your heaviest real query on a server sized to your budget and measure memory use and runtime against the data you expect in twelve months. Confirm with your platform team which Iceberg or Delta features your tables depend on, because extension support tiers differ and must be checked against the version you deploy. If the answer is unclear, a fixed-scope proof of concept on your own infrastructure is a lower-risk way to settle it.
Which scenarios favor DuckDB and which favor Spark
This table walks through common platform team scenarios and names the better fit for DuckDB on one server or Apache Spark on a cluster, with the reason and the condition that would reverse the call. Read it as a starting point for your own data volumes, concurrency needs and team skills, then confirm the reversal conditions in a proof of concept.
| Scenario | Better fit and reason | What would reverse the choice |
|---|---|---|
| Nightly reporting job over a Parquet lake that fits on one large server | DuckDB. It reads Parquet directly by naming files in the FROM clause, runs as an embedded library with no server to operate, and suits complex queries over large portions of the data. | Data growth or peak memory and disk needs outgrow one machine, or the job must be split across many nodes within its window. |
| Analysts querying shared Iceberg tables with many concurrent users | Spark or a shared SQL engine. DuckDB's file-based concurrency allows one read-write writer process, so a shared multi-user query layer is a poor match. | Analysts query read-only snapshots through a single well-governed process, or a multi-writer protocol such as Quack reaches the maturity your team needs. |
| Continuous streaming ingestion with event-time processing | Spark Structured Streaming, since Spark offers unified batch and streaming processing. DuckDB is built for analytical queries and expects large-scale changes such as bulk appends. | Ingestion is micro-batched into periodic bulk loads, in which case DuckDB can handle the loads and the stream stays in a broker or ingestion tool. |
| Small team with no cluster operations skills | DuckDB. Nothing runs as a separate service, so there is no cluster to size, patch or monitor, which keeps recurring operating effort low. | Workloads grow past one server, or your team already runs a Spark platform and the skills are in place. |
| Data science team using Python and SQL on a few hundred gigabytes of local files | DuckDB, with its Python client running inside the notebook or job process. Spark adds JVM and cluster overhead that the data size does not justify. | Feature pipelines need distributed machine learning training or must run across many workers on a shared schedule. |
| Platform with heavy Spark investment, Kubernetes operators and multiple engineering teams | Spark. Its standalone, YARN and Kubernetes cluster managers and the Kubernetes operator fit an existing shared compute platform. | The platform is consolidated onto a single analytics workload that fits one server, in which case the cluster may cost more to run than it returns. |
How DuckDB and Spark are built and what that means for operations
DuckDB and Spark sit at opposite ends of the operating spectrum. One is a library that disappears into your application process, while the other is a distributed system with coordinators, workers and a cluster manager that someone must keep running. The cards below walk through each component and name what your team installs, patches, monitors and loses when it fails.
DuckDB as an embedded library
DuckDB runs inside a host process rather than as a separate server, so there is no database daemon to install, upgrade or back up on its own. Your team still owns the host application, the Python, R, Java or CLI client version and the database file itself. Check how writers coordinate: in read-write mode one process can write a database file, and writing from several processes needs the Quack remote protocol (beta as of v1.5.2) or DuckLake with PostgreSQL as the catalog. Failure modes center on the host process crashing mid-write and on concurrent writers from separate processes.
Spark driver and executors
A Spark application has a driver that plans work and coordinates executors running on worker nodes. Your team must provision those nodes, size executor memory and cores, and monitor shuffle spill, failed tasks and driver memory. The driver is a single point of failure for a job, so a driver crash usually means the job restarts. Failure modes include executor loss, out-of-memory errors during shuffles and skewed partitions that leave most executors idle while one runs long.
Spark cluster managers
Spark ships with three cluster managers: its own standalone manager, Hadoop YARN and Kubernetes. Each one is a separate system your team must install, secure, patch and monitor, and the choice shapes your skills requirement. Standalone is the simplest to stand up but offers the least surrounding tooling. YARN fits organizations that already run Hadoop. Kubernetes fits teams that already operate clusters. Failure modes include resource contention between jobs and misconfigured queues that starve batch work.
Spark Connect client-server mode
Spark Connect separates the client from the Spark driver by sending unresolved logical plans over gRPC and returning Apache Arrow batches. This lets notebooks and applications run on machines that do not hold the Spark runtime, which reduces dependency drift on user laptops. Your team must run and monitor the Connect server, manage gRPC endpoints and confirm client and server versions match. PySpark clients have been supported since 3.4 and Scala clients since 3.5, so confirm your client language before you commit.
Spark Kubernetes Operator
The Apache Spark Kubernetes Operator manages Spark applications on Kubernetes using the operator pattern, so Spark jobs are declared as Kubernetes resources. Your team must install and upgrade the operator, maintain its custom resource definitions and manage RBAC, image builds and the Kubernetes cluster beneath it. Confirm whether your platform team already supports operators in production. Failure modes include operator version drift against the Spark image, stuck application states and pod scheduling failures under resource pressure.
DuckDB vs Apache Spark on Parquet, Iceberg, Delta and operations
This table compares DuckDB and Apache Spark on license, execution model, data sources, writes, streaming, clients, cluster managers and the layers each one leaves to you. Read it as a checklist: where a row names an extension support tier or a component Spark does not ship, confirm that item in your own environment before committing. The main takeaway is that DuckDB removes the server and cluster, while Spark buys distributed execution and streaming at the cost of assembling a platform around it.
| Capability | DuckDB | Apache Spark |
|---|---|---|
| License | MIT License, with intellectual property held by the DuckDB Foundation | Apache License 2.0, by the Apache Software Foundation |
| Current release | DuckDB v1.5.6, published September 28, 2026 | Spark 4.2.0, released July 14, 2026; 3.5.9 maintenance release on July 16, 2026 |
| Execution model | Embedded in the host process on one machine, columnar-vectorized engine for analytical queries | Driver coordinating executors on worker nodes, or a single node; distributed batch, SQL and streaming |
| Parquet, CSV and table formats | Reads CSV and Parquet by naming files in FROM; iceberg, delta and ducklake extensions are mostly secondary (best-effort) support tier, so confirm the tier for your formats | Reads and writes Parquet and table formats through Spark's data sources; confirm Iceberg, Delta and catalog versions against the Spark release you deploy |
| Concurrent writes | One read-write process per file; multiple writer threads in one process, with conflicts only when two threads change the same row; multi-process writes need the Quack protocol (beta as of v1.5.2) or DuckLake with PostgreSQL | Designed for many concurrent executors writing to shared storage; coordination depends on the table format and catalog you choose |
| Streaming | Not a streaming engine; suited to bulk appends and batch analytics | Unified batch and streaming processing within the same engine |
| Language clients | CLI plus clients for Python, R, Java, Wasm and other languages | Python, SQL, Scala, Java and R; Spark Connect provides decoupled gRPC clients, PySpark since 3.4 and Scala since 3.5 |
| Cluster managers | None; runs on Linux, macOS and Windows on x86 and ARM | Standalone, Hadoop YARN and Kubernetes; the Apache Spark Kubernetes Operator is an Apache-2.0 subproject for Kubernetes |
| Notebook, scheduler and catalog | No separate server; notebooks, scheduling and catalog come from the host application or tools you add | Ships no notebook workspace, job scheduler UI, catalog or access control; typically paired with JupyterHub, Apache Airflow and a table catalog |
| Security and access control | Access is governed by the host process and file system permissions; confirm what your deployment needs for multi-user access | No built-in access control layer; confirm the authorization model of your catalog, cluster manager and Spark Connect endpoint |
What each platform's license and editions cover
DuckDB is released under the MIT License, a permissive license that lets your team use, modify and redistribute the code, including inside internal tools and commercial products, as long as the copyright and license notice are kept. The project's intellectual property is held by the DuckDB Foundation, so stewardship of the code sits with that foundation. The Why DuckDB page is the natural starting point for the project's own description of its licensing and platform support.
- Embedded deployment: DuckDB runs inside a host process, so there is no database server to license, install, patch or back up as a separate product.
- Release cadence: v1.5.6 was published on September 28, 2026, so plan upgrades as a recurring task and review release notes before each change.
- Extension tiers: most core extensions, including iceberg, delta and ducklake, sit at a secondary, best-effort support tier, which matters for audited pipelines.
- Third-party service terms: the motherduck extension is maintained by a third party for connecting to the MotherDuck service, and that service has terms separate from the MIT License.
Confirm with the DuckDB Foundation and the project's maintainers how the MIT License applies to every bundled component in your build, and record any extension you depend on along with its support tier.
Apache Spark is licensed under the Apache License 2.0 and is governed by the Apache Software Foundation. The license permits commercial use, modification and redistribution, and it includes an express patent grant from contributors, a point that legal and procurement teams usually review. Spark's cluster managers, including its standalone manager, Hadoop YARN and Kubernetes, are part of the same project, so choosing a cluster manager does not change the license of the engine itself.
- Operator subproject: the Apache Spark Kubernetes Operator is an Apache-2.0 subproject in the apache/spark-kubernetes-operator repository, and its maturity should be checked before you build on it.
- Client packaging: Spark Connect separates clients from the server, with PySpark clients since 3.4 and Scala clients since 3.5, which affects how client libraries are versioned and shipped.
- Scala 2.13 baseline: Spark 4 is pre-built with Scala 2.13 and has dropped Scala 2.12 support, so libraries built for 2.12 need a migration plan.
- Surrounding tools: Spark ships no notebook workspace, job scheduler UI, catalog or access control layer, so those components arrive with their own licenses.
Build a bill of materials that lists every component in the stack with its license and version, and confirm the license of each add-on before it reaches production.
Spark's versioning policy sets the lifecycle for its releases. Feature releases occur quarterly starting with 4.3.0, each major and feature release branch is maintained for six months, and the last release in each major line is an LTS maintained for 18 months. The 3.5.x extended LTS ends in November 2027 and receives security fixes only. On the downloads page, 4.2.0 was released on July 14, 2026, maintenance releases 4.1.3 and 4.0.4 on July 15, 2026, and 3.5.9 on July 16, 2026.
- Quarterly feature releases: from 4.3.0 onward, expect a new feature line about every quarter, so upgrade planning is recurring work.
- Six-month branch maintenance: a feature or major branch receives maintenance for six months, which sets a short window for non-LTS builds.
- 18-month LTS: the last release in each major line is supported for 18 months, which is the usual anchor for a stable production baseline.
- 3.5.x extended LTS: security fixes only, with support ending in November 2027, so a 3.5 estate has a fixed end date and no feature fixes.
- DuckDB cadence: the supplied sources do not state a DuckDB support window, so ask the maintainers for their policy before committing to a version.
Map each date against your change calendar and audit cycle, and request the support policy for DuckDB in writing before setting an internal upgrade schedule.
The supplied sources do not describe a paid edition of either project, so neither Apache Spark nor DuckDB appears with a commercial tier in the material reviewed. Any paid support has to come from a vendor or partner your team selects and contracts with directly, and the response times, patch commitments and escalation paths are set in that contract. Confirm the absence of paid editions with each project and with any vendor you consider, because licensing terms can change between releases.
- No license fee in the sources: MIT and Apache 2.0 do not carry per-core or per-user license charges, so your budget goes to infrastructure, staff time and any support contract.
- Who is paid for what: you pay for compute, storage, engineers and any support partner, while the project maintainers are not invoicing your organization in these sources.
- Recurring versus one-time: the initial build and data migration are one-time costs, while upgrades, regression testing and security patching recur with each release cycle.
- Support contract scope: any SLA, named engineer or patch window comes from the signed agreement, not from the license.
- Quote inputs: ask for the written scope, the versions covered, the upgrade cadence and the exit terms before comparing offers.
Request written scope, supported versions and exit terms from every option you shortlist before you compare costs, and confirm whether the support contract covers both DuckDB and Spark or only one of them.
What drives the running cost of a self-hosted DuckDB or Spark platform
The running cost of a DuckDB platform sits mostly in one server, the people who operate it and the data work itself. DuckDB is embedded in a host process, so there is no database server to install, patch or keep available as a separate service, which removes a line item that a cluster carries by default. Your recurring infrastructure cost is the machine or cloud instance, its storage and the egress from object storage if Parquet, Iceberg or Delta files live there. Spark's bill has more parts: driver and executor nodes, the cluster manager (standalone, Hadoop YARN or Kubernetes), shuffle and temporary storage, and network traffic between nodes. Ask the implementer to price each of these as a separate line so you can see which one grows with data volume and which grows with the number of jobs.
The larger difference is engineering time. A DuckDB platform is mainly maintained through SQL and Python, plus whatever scheduler and catalog you choose. A Spark platform adds cluster operations, JVM tuning, executor sizing and, for Kubernetes deployments, operator management through the Apache Spark Kubernetes Operator. Spark also does not ship a notebook workspace, job scheduler UI, catalog or access control layer, so you must assemble those yourself, for example with JupyterHub, Apache Airflow and a table catalog. Each of those components is another service to upgrade, secure, back up and document. Before you accept a quote, confirm who owns each tool after handover, who is on call for failed jobs, and whether the scope names the surrounding tools or leaves them to your team.
Separate one-time work from recurring work in every quote. One-time items usually include architecture and sizing, the initial build, identity and access integration, migration of existing pipelines and the first hardening pass. Recurring items include patching the engine and its dependencies, upgrades between release lines, monitoring, backups, restore tests and support hours. Spark's release calendar makes the recurring part concrete: the Spark versioning policy says feature releases are quarterly from 4.3.0, and branches are maintained for 6 months, while the last release in each major line is an LTS maintained for 18 months. DuckDB releases move faster, so budget for regular minor upgrades of the embedded library. Request in writing the scope, the list of deliverables, the upgrade cadence and which versions are covered, the support hours and response expectations, and who pays for the Spark or DuckDB version changes that break your jobs.
Moving Spark jobs to DuckDB or DuckDB work to Spark
This sequence shows which Spark workloads can move to a single DuckDB host, which must stay on a cluster engine, and what each move rebuilds. The output is a justified keep-or-move decision for each job and a tested write pattern before any production cutover.
Inventory jobs by size, concurrency and streaming
The data engineering lead lists every Spark job with its input volume, peak concurrency, schedule, and whether it runs as a streaming query or a batch job. Record the owner, the downstream consumers and the current runtime for each job. Mark any job that needs distributed execution across many nodes, streaming micro-batches or many simultaneous writers as a likely Spark keeper. The step is done when every job has a row in the inventory and each row carries a provisional move or keep tag that the owner has reviewed. Treat the tags as hypotheses until the later steps test them against real data.
Confirm table formats and catalog dependencies
The platform engineer records which table formats each job reads and writes, including Parquet files, Iceberg tables, Delta tables and any DuckLake catalogs. Note which catalog serves each table, how schemas are resolved, and whether any consumer depends on Spark-specific behavior such as particular SQL functions, UDFs or Spark-managed partitions. Confirm with the owners of the DuckDB core extensions for iceberg, delta and ducklake that each format and operation used in production is supported in the version you plan to run, and that the extension support tier meets your risk tolerance. The step is done when a written compatibility matrix exists for every table and query pattern, with open questions assigned to a named owner.
Move read-only analytical queries first
Start with jobs that only read tables and produce reports, dashboards or extracts. The analytics engineer reruns a representative set of these queries in DuckDB against the same Parquet, Iceberg or Delta data, then compares row counts, aggregates and sampled values with the Spark output. Measure runtime and peak memory on the target server and keep a rollback path that leaves the Spark job in place. Because DuckDB runs embedded in the host process, there is no separate database server to build for this stage, but the team still owns the host's patching, storage and access controls. The step is done when read-only outputs match Spark on agreed test data and the owners sign off on the results.
Test write patterns against concurrency rules
The data engineer tests every write path that will move, using the concurrency rules in DuckDB's documentation. In read-write mode, one process can read and write a DuckDB database file, while READ_ONLY mode allows several reader processes but no writers. Within a single process, multiple writer threads can update separate tables or separate subsets of one table, and a conflict error occurs only when two threads change the same row at the same time. If several independent processes must write, the design must use the Quack remote protocol (in beta as of DuckDB v1.5.2) or DuckLake with PostgreSQL as the catalog, and the team must confirm that choice is acceptable for production. The step is done when a load test shows the expected write throughput and conflict behavior, and the team has documented the single-writer design constraint.
Rebuild orchestration, code and access control
The platform team rebuilds what does not transfer. Job code moves from Spark APIs to DuckDB SQL or Python clients, and scheduling moves to the existing orchestrator or to a new one, since Spark has no job scheduler UI of its own. Access control must be designed separately because Spark does not include a catalog or access control layer, so teams usually rely on the table catalog and the platform's identity system. Define how credentials to object storage reach the DuckDB host, how secrets are rotated, and how audit records are kept. The step is done when each rebuilt job has a reviewed code path, a schedule, a named owner and an access model that the security team has approved.
Cut over and monitor parallel runs
The data operations lead runs the moved jobs in parallel with Spark for a period agreed in the change plan, comparing outputs and failure rates each run. Keep the Spark jobs and their cluster capacity available until the parallel results match and no unplanned fallback has been needed. Set alerts on job duration, memory use on the DuckDB host and write conflict errors, and record the operational runbook for the single-host design, including what happens if the host fails and how the Parquet, Iceberg or Delta data is restored from backup. The step is done when the moved jobs run on DuckDB without manual intervention for the agreed period and the runbook has been tested once.
Retire or keep Spark for distributed work
The engineering lead makes the final decision on each remaining Spark job. Keep Spark for workloads that need distributed execution across a cluster, streaming, or large parallel transformations that outgrow one server, and confirm the cluster manager choice among Spark standalone, Hadoop YARN and Kubernetes. Where Spark stays, decide whether clients connect through Spark Connect, which sends logical plans over gRPC and returns Arrow batches. Review the Spark release line in use against the versioning policy, since Spark 3.5.x extended LTS support ends in November 2027. The step is done when every Spark job has a documented reason to remain, the cluster's operating cost is in the budget, and the retired jobs are removed from schedules and access lists.
Where DuckDB and Spark implementation projects usually run long
DuckDB projects run long when the team has not decided who writes to the database file and how everyone else reaches it. Because DuckDB is embedded, there is no server to size or patch, so the effort moves to design: how Parquet, Iceberg or Delta files are laid out on object storage or local disk, how partitions are named and compacted, and which process owns each database file. In read-write mode one process can write a DuckDB file while others cannot, and several processes can only read in READ_ONLY mode. A team that needs several writers across processes must choose between the Quack remote protocol, which was in beta as of DuckDB v1.5.2, and DuckLake with PostgreSQL as the catalog database. Deferring that choice until dashboards are in production is the most common reason these projects stall. Read the DuckDB concurrency documentation before the design workshop and record the answer in writing.
Spark projects run long on the work around the engine. Spark ships three cluster managers, its standalone manager, Hadoop YARN and Kubernetes, and each needs its own setup for executor sizing, storage access and network tuning between the driver and executors. Spark also has no notebook workspace, job scheduler UI, catalog or access control layer, so the team must assemble JupyterHub, Apache Airflow, a table catalog and an authorization model, then test how they behave together under load and failure. Access control gaps are the usual surprise: a notebook user can reach tables that the catalog was meant to restrict, because nobody mapped permissions across the layers. Confirm which cluster manager and catalog the team will standardize on before scoping any integration work.
DuckDB vs Apache Spark: questions buyers ask before hiring help
Can DuckDB replace Spark for our data size?
Often for analytical queries over data that fits on one large server, but the answer depends on your data volume, query shape and where the data lives, not on a single size threshold. DuckDB is embedded in a host process, targets analytical workloads over large portions of the data, and reads Parquet, CSV and table formats such as Iceberg and Delta through its extensions. Spark earns its place when work must spread across many machines, when you run streaming pipelines, or when existing jobs already depend on Spark APIs. Test your heaviest real queries on a representative server copy and compare run time, memory and failure behavior before deciding.
What skills does our team need to run DuckDB or Spark?
Spark needs people who can operate a distributed engine: cluster managers such as standalone, Hadoop YARN or Kubernetes, executor sizing, job tuning and the surrounding tools, because Spark does not include a notebook workspace, scheduler UI, catalog or access control layer of its own. DuckDB asks for SQL and Python or R skills and basic server operations, since there is no database server to install or patch. Most teams still need data engineering skills for pipelines, data quality and access control either way. Confirm the skills you have in your own staff before you decide which path carries less operating load.
Can our own engineers migrate from Spark to DuckDB without outside help?
Yes, for simple batch SQL jobs that read Parquet files, a small team can often port queries directly, and DuckDB's CLI and client libraries make trials quick. Migration gets harder when jobs use Spark-specific DataFrame code, streaming, custom UDFs or many cluster-level dependencies. Inventory your jobs, classify them by complexity, and pilot a few representative pipelines first. Outside help is most useful for the classification, the parity testing and the cutover plan, which are the parts that fail quietly when rushed.
How long does a DuckDB or Spark project take, and what drives it?
No supplied fact gives a duration, so treat any number a vendor or consultant quotes as unverified until it is tied to a written scope. The main drivers are the number of pipelines, how many use Spark-specific features, the state of your table formats and catalog, how much data validation is required, and how many systems consume the output. Scheduling and access control also add time if your platform has none today. Ask for milestones that match your inventory and acceptance tests, not a single end date.
How many concurrent writers can one DuckDB server handle?
DuckDB supports multiple writer threads inside one process, so separate tables or separate subsets of a table can be updated in parallel, and a conflict error occurs only when two threads change the same row at the same time. Across processes, read-write mode allows only one process to write a database file, while READ_ONLY mode lets several processes read but none write. Multi-process writing needs the Quack remote protocol, which was in beta as of DuckDB v1.5.2, or DuckLake with PostgreSQL as the catalog database. Confirm which pattern your ingestion and reporting jobs need and test it under your real concurrency.
What does a single server handle, and how should we test it?
Capacity depends on your server's cores, memory, storage speed and the shape of your queries, so no general limit applies. DuckDB's columnar-vectorized engine suits complex, long-running queries over large parts of the data and bulk appends, which are the cases to measure first. Build a test plan with your largest tables in Parquet, your heaviest joins and aggregations, your expected peak concurrency, and failure tests such as out-of-memory conditions. Record run time, peak memory and recovery steps, then compare them with the Spark cluster cost you would otherwise pay.
How do upgrades differ between DuckDB and Spark?
DuckDB is a library-style dependency embedded in your application, so upgrades change the version your code and pipelines ship with, and you retest the query path and file format compatibility. The DuckDB v1.5.6 release was published on September 28, 2026, and core extensions carry different support tiers, with most at the secondary best-effort tier, so check each extension you rely on. Spark upgrades touch the cluster, and its versioning policy says feature releases move quarterly starting with 4.3.0, with major and feature branches maintained for 6 months and the last release in each major line maintained as an LTS for 18 months. Spark 4 also dropped Scala 2.12 support, so check your code and dependencies before planning an upgrade. Ask any implementation partner how they will pin versions and test each release.
What should we ask for in a quote for this work?
Ask for a written scope that lists the pipelines in and out of scope, the target platform and versions, the data sources and table formats, the acceptance tests, and the handover documents. Request the cost components separately: discovery, migration or build, testing, cutover, and any retained support or upgrades, with the one-time and recurring parts marked. Ask who performs each task, what your team must provide, and which infrastructure costs are yours, such as servers, storage and cloud accounts. DeployOpen works in fixed-scope engagements agreed in writing, and it does not publish a price list, so a quote follows from the written scope. Confirm the tools and licenses that apply to your environment before you sign.
How DeployOpen helps choose, implement or migrate between DuckDB and Spark
DeployOpen plans, implements and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For DuckDB vs Apache Spark, the work starts with your data volumes, query patterns and table formats, then turns into a written scope before any build begins.
How we scope
- Review which workloads run on one large server and which need cluster execution or streaming.
- Inventory Parquet, Iceberg and Delta sources, plus the concurrency pattern of writers and readers.
- Agree deliverables, timelines and responsibilities in writing before any work starts, since most engagements are fixed-scope.
How we build
- Install and configure DuckDB or Spark inside your environment, including the table format extensions or connectors your data requires.
- Build the query, ingestion or job layer, and wire it to your existing orchestrator, catalog and storage.
- Test query correctness, load behavior and failure handling against representative data before cutover.
How we hand over
- Document the architecture, configuration, access model and the reasons behind the platform choice.
- Provide runbooks for backups, upgrades and the write-concurrency limits your team must respect.
- Walk your engineers through operation so they can run the platform without us.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Retainers can cover version upgrades when the platform or its extensions change, scoped in writing.
- Ask for the retainer terms, response expectations and included services in your quote.
