Data pipeline RFP questions

A good partner names one orchestrator for scheduling and dependencies, explains why it fits your team's skills and your source mix, and assigns each tool a distinct role. Apache Airflow (Apache-2.0, with Airflow 3.3.2 as the current release) is a common choice for scheduled batch DAGs. Dagster (Apache-2.0, with 1.13.26 published on October 9, 2026) organizes work around assets and includes lineage and observability in its open-source tooling. Airbyte's platform repository is under the Elastic License 2.0, so confirm its terms for your use. Apache Kafka carries events, and Apache NiFi moves files and routes data between zones. Ask for a written mapping of each component to a responsibility, and for the connector licenses reviewed against your distribution model. (Illustrative: a team with mostly batch SaaS sources may need a different mix than a team moving plant telemetry.)

A strong answer describes code-level tests for every transformation, run in continuous integration before any change reaches production. Ask the partner to show a DAG integrity test in the pipeline, asset tests for Dagster, a shared test fixture for source and warehouse connections, and a pull request template that requires a reviewer. Deployment should move through named environments with the same container image promoted between them, and each promotion should be recorded. Ask who approves production changes, how that approval is evidenced in your change system, and whether rollback is a redeploy of the previous tagged image. Confirm that the partner's CI runners and artifact registry sit in your own infrastructure.

The direct answer is that failures are contained by design: each task is retryable, each write is idempotent, and backfills are scoped and reviewed. Ask for the retry policy per task type, the rule for which failures page someone versus retry silently, and how a partial load is rolled back. For idempotency, look for merge or partition-overwrite patterns instead of blind appends, plus deduplication keys for event streams. Backfills should run in a bounded date window with a dry run, a row-count reconciliation and a named approver. Ask how Kafka consumer offsets are reset during a replay and who signs off. The partner should describe the runbook, not only the feature.

A good answer starts from the version lifecycle and names the current supported releases. For Airflow, the supported-versions table shows Airflow 2 reached end of life on April 22, 2026, after which it gets no fixes, so any Airflow 2 estate needs a migration plan. The Airflow 3.3.2 prerequisites list Python 3.10 to 3.14, PostgreSQL 14 to 18 and Kubernetes 1.30 to 1.35, and the upgrade guide for Airflow 3 should be used as the checklist. For Kafka, the downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported, and the upgrade notes say Kafka 4.0 supports only KRaft and requires Java 17 for brokers and Kafka Connect. Ask for a staging rehearsal, a rolling-upgrade plan and a written rollback point before each change window.

Lineage should be captured automatically from the orchestrator and the transformation layer, not drawn by hand. Ask whether asset lineage comes from the orchestrator's metadata, whether SQL transformations are parsed for table-level lineage, and whether column-level lineage is in scope. Dagster describes integrated lineage in its open-source project, while some lineage features in Dagster+ are paid tiers, so confirm which features your deployment will use. Metadata catalogs such as DataHub or OpenMetadata (both Apache-2.0) can receive lineage from several tools. Ask where lineage records are stored, how long they are kept, and how they are exported for an audit or a supervisor request.

A good answer separates three layers: infrastructure health, pipeline run status and data freshness or quality. Ask for specific alerts on failed runs, missed schedules, consumer lag on Kafka topics, scheduler heartbeat and data that has not arrived by its deadline. Airflow supports running more than one scheduler against the same metadata database, and the partner should say how that is monitored. Ask which metrics feed which dashboard, who receives each alert, and how alerts are routed to on-call. Confirm that logs and metrics stay in your own storage. Ask for a written alert catalog with thresholds you can review.

The answer should name where credentials live and who can read them. Ask whether source and warehouse credentials are held in a dedicated secrets store, rotated on a schedule, and injected at run time rather than stored in DAG code or environment files. Kafka authentication options include SSL and SASL mechanisms such as SCRAM-SHA-512, GSSAPI and OAUTHBEARER, and authorization in KRaft clusters uses ACLs stored in the cluster metadata log. Ask whether the default of denying resources without ACLs is kept, and whether the Airflow auth manager is connected to your identity provider. Request a list of service accounts, their privileges and the review cycle.

A complete answer assigns each duty in writing: who monitors, who responds to a page, who approves a change, and who performs upgrades. Ask for a RACI table covering on-call, incident severity, change approval and the handover from the build team. Support can be a fixed-scope rollout followed by a separate retainer for upgrades, patching and continuous monitoring. DeployOpen's engagements are typically fixed-scope with scope and deliverables agreed in writing, and ongoing support, upgrades and testing are offered as retainers. Ask for the response expectations in writing, what is included in each retainer and what your own team must keep doing.

The cost is driven by the number of sources, how hard each one is to connect, the transformation logic, and the environments the pipelines must run in. You pay for scoping and design, connector and DAG build, testing, deployment into your infrastructure, and handover documentation. Recurring costs sit with you: compute and storage for the orchestrator, Kafka brokers or other components, backups, monitoring, and the on-call effort to keep jobs running. When you request a quote, ask for the cost of each source and destination separately, what is one-time versus recurring, which infrastructure you provide, and which upgrade and support work is included. DeployOpen issues a quote from the written scope, not a standard price list.

Duration depends on the number of sources, their access and documentation quality, the volume and latency targets, whether streaming is required, and how many environments and approval gates your change process has. Access requests and data owner reviews often take longer than the build itself, so they should be planned in the first weeks. Ask any partner for a phased plan with milestones tied to working pipelines in a non-production environment, a named acceptance test for each source, and the dependencies that could move dates. DeployOpen agrees scope, deliverables and timelines in writing before work starts, so the schedule reflects the environment you actually have.

Three things determine how fast the work moves: access, owners and environments. You need service accounts or read access to each source system and a defined path for credentials, ideally held in your secrets manager. You need named data owners who can confirm schemas, definitions, refresh expectations and which fields are sensitive. You also need the target environments, such as a development, staging and production cluster, the network routes between them, and the change approval process that will govern DAG or flow releases. Confirm in writing who signs off on each source and who receives alerts after handover.

Yes, if the handover is designed for it. Your engineers should receive the orchestrator configuration, deployment pipelines, runbooks for common failures and backfills, upgrade notes, and a walkthrough of each pipeline. The practical question is whether your team has people who know Python, SQL, Kubernetes or Docker, and whether they have time for on-call work. Ask the partner which knowledge transfer activities are included in the fixed scope, how the team is trained on the specific orchestrator, and what documentation exists for pipeline ownership. Plan the handover around a period where your engineers operate the pipelines while the partner observes.

Ongoing support is usually a retainer that covers the work a live pipeline platform needs over time: monitoring and triage, patching, planned upgrades of components such as Airflow or Kafka, connector updates, backup verification and changes to pipelines when source schemas move. Retainers are scoped in writing, so ask what response paths are included, which upgrades are covered, how changes are approved, and what is excluded. DeployOpen offers ongoing support, upgrades and continuous testing as retainers, separate from the initial fixed-scope implementation.

A SaaS ELT tool runs extraction and loading in the vendor's cloud and bills by usage, rows or connectors, while a self-hosted pipeline runs in your own cloud accounts, data center or air-gapped network under your control. With self-hosting, you own the data path, the credentials, the logs and the upgrade schedule, and you can keep sensitive data inside your boundary. The trade-off is that your organization carries the infrastructure, patching and on-call work that the vendor would otherwise handle. Compare total cost over several years, including licensing terms of the connectors you intend to use, which the Airbyte repository shows is governed by the Elastic License 2.0 for the platform repository. DeployOpen implements the self-hosted option; it does not sell a managed ELT service.