Data pipeline requirements checklist

The first requirement to settle is how quickly each consumer needs data and whether anyone will need to read the same events again later. A scheduled batch load runs at fixed intervals and suits daily finance extracts or warehouse models that nobody reads before the next business day. Micro-batch shortens the interval to minutes. Event streaming moves each change as it happens. The choice follows from the latency each downstream use case tolerates, and from whether a failed consumer must be able to rewind and reprocess a window of history. Write these answers down per source before any tool is named, because a 15-minute requirement and a sub-second requirement lead to very different platforms.

In a self-hosted estate, Apache Kafka usually fills the transport role. Its topics keep events for a configured retention period, which lets new consumers start from an earlier offset and lets a failed consumer replay after a fix. Confirm the retention settings, the replication factor and the authentication mechanism (SSL or SASL, as described in the Kafka security documentation) with your platform team, because they determine how much history is recoverable and who can read it. Kafka moves and holds events; it does not schedule transformations or track which model depends on which table. That work belongs to an orchestrator such as Apache Airflow for scheduled runs or Dagster for asset-based runs, which records dependencies, retries and backfills across batch jobs.

Many estates need both. A typical pattern (illustrative) lands Kafka events in object storage or a staging table, then lets Dagster or Airflow run the batch transformations that build reporting models every hour, while a small number of operational alerts consume the stream directly. The decision that matters is which path is the system of record for each dataset. Record that decision explicitly, including who may replay a topic, how orchestrator runs are reconciled against the events they consumed, and which team owns each side of the boundary. Ask any implementation partner to show how replay and backfill were tested on your data volumes before committing to either pattern.

Start by mapping every personal, regulated or confidential field from source to staging to warehouse before any connector is built. For each field, record the source system, the transformation that touches it, the stage where it lands, the people and service accounts that can read it, and the retention period. Then decide, per stage, whether the field may exist there at all. A common decision is that raw identifiers may sit only in the source-adjacent landing zone, while the warehouse receives pseudonymized or tokenized values. Write these decisions down as a data flow register that the platform team and the data owner both sign, because the register becomes the baseline for every later review.

Intermediate files and logs often leak more than the tables do. Set explicit rules for where staging files, dead-letter queues and temporary extracts may be written, how long they persist, and who can read them. For Airflow, check that task logs, XCom values and rendered templates do not echo payloads, and confirm which fields the scheduler and web UI store in the metadata database. For Kafka, confirm whether topic payloads are encrypted at rest in your storage layer, since TLS protects data only in transit. For Dagster, confirm where run and event log storage lives and whether asset metadata captures sample rows. Test these settings with synthetic records that contain dummy sensitive values, then search the metadata store, logs and object storage for those values.

If any pipeline feeds an AI system, the record-keeping duties in Article 12(1) and the six-month log retention in Article 19(1) of the EU AI Act may apply, but only when the system is high-risk and in scope for that regulation. Confirm applicability with legal counsel before designing log retention, because the answer determines whether pipeline logs must be kept as AI system evidence. Whichever way that question resolves, ask your implementation partner to show how the data flow register, retention rules and metadata checks will be kept as evidence, and who approves changes to them.

Decide during requirements which evidence the pipeline must produce, because it is far harder to add later. The core set is run history (every scheduled and manual run, its start and end time, its outcome and the retries behind it), access records (who can view, trigger, edit or read credentials for each DAG, asset or flow), change approvals (who reviewed and merged each change to a DAG, asset definition, connector configuration or NiFi flow, and which version ran in production), and lineage and retention settings (where each dataset came from, what transformed it, and how long raw, staging and warehouse copies are kept). Ask your auditor which of these they will sample, over what period, and in what format. Confirm whether they expect exports from the orchestrator itself or from your ticketing and source control systems, since that choice determines which tools you need to retain and where.

Map each evidence type to a concrete source in your design. Run history usually comes from the orchestrator metadata database, so confirm how long it is retained and whether it is backed up with the rest of the platform. For Airflow, check the authentication and authorization model in the auth manager documentation and confirm which role-based access settings your version supports. For Dagster, the open-source edition does not include the role-based access control, SAML SSO, column lineage or audit log features listed on the Dagster pricing page as Dagster+ features, so decide whether you need them, and if so, whether the managed tier or another control provides them. For Kafka, document the authentication mechanism, the ACL model stored in the KRaft metadata log and whether unauthenticated access is disabled. Record these settings as configuration exports rather than screenshots so they can be re-checked.

DeployOpen prepares controls, configurations and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies decide what is sufficient, and your own compliance function owns the control framework. If the pipeline feeds AI systems within scope of the EU AI Act, the record-keeping duty in Article 12 of the AI Act and the six-month minimum log retention in Article 19 apply to the logs you keep, so confirm with counsel whether your use is in scope before setting retention periods. Agree the evidence list, its owner and its retention in the written scope before build starts, so the implementation partner can design the logging and export jobs to produce it.