How the KubernetesPodOperator setup works in Airflow 3
The KubernetesPodOperator launches each task as a pod in a Kubernetes cluster that you control, so a team that needs task-level isolation, its own dependency set, or specific compute for a few heavy steps can get that without changing the rest of the DAG. The operator comes from the Kubernetes provider package, and it is separate from the choice of executor. You can run it under LocalExecutor, CeleryExecutor or KubernetesExecutor, because the executor decides where the operator task itself runs while the operator decides where the workload runs. Confirm the provider version against the Airflow 3.3.2 prerequisites, which list Kubernetes 1.30 to 1.35 as supported cluster versions, and confirm that your cluster version falls inside that range before you design around it.
Setup starts with three decisions. First, decide which namespace the task pods run in and which service account they use, since that account determines what the pod can reach inside the cluster. Second, decide how the Airflow scheduler and workers authenticate to the cluster, whether through an in-cluster service account or a kubeconfig stored outside DAG code. Third, decide what each pod image contains. Pin the image by tag or digest, keep the Airflow worker image separate from task images, and define resource requests and limits in the pod specification so one heavy task cannot starve the scheduler on the same cluster. Ask your platform team to confirm network policy, because task pods must reach the metadata database only if the task needs it, and in Airflow 3 task code cannot open Airflow database sessions directly and must use the REST API or Task SDK instead.
Operational behavior matters as much as the first run. Deferrable mode lets a waiting pod release the worker slot while the triggerer watches the pod, which keeps long jobs from holding capacity. Cleanup settings control whether finished pods are deleted, which affects log retention and cluster capacity, so set them deliberately and test that failed pods remain inspectable long enough for an on-call engineer. Secrets belong in Kubernetes Secrets mounted into the pod or in Airflow connections fetched through a secrets backend, not in plain environment variables written into the pod spec, because environment values appear in pod descriptions and are easy to leak through logs. Verify this with a test DAG that fails on purpose and check where each value appears.
Prerequisites: versions, cluster access and network
Before writing any pod specs, confirm that the Airflow, Python, Kubernetes and metadata database versions line up, and that the scheduler and workers can reach a cluster with narrowly scoped rights. Each card below is a check to close before the first task pod launches.
Airflow and Python versions
Airflow 3.3.2 supports Python 3.10 to 3.14, so pin the Python version in the image that runs the scheduler, workers and the DAG processor. Keep the Kubernetes provider package version compatible with that Airflow release, and record both in your build metadata. Confirm the provider version against the Airflow provider documentation before upgrading either side, because a mismatch shows up as operator import errors rather than a clear failure.
Kubernetes cluster version
Airflow 3.3.2 lists support for Kubernetes 1.30 to 1.35. Check the version of the cluster that will run task pods, not only the cluster hosting Airflow, because the operator talks to the API of the cluster where pods are created. Ask your platform team which cluster is the target, whether it is a managed service, and how often it is upgraded, since an upgrade outside the supported range can break pod creation or watch behavior.
Metadata database choice
Use PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release for the metadata database. MariaDB is not supported, and SQLite is for testing only. Deferrable operators and task state depend on this database being reliable, so plan backups and a sizing review before production. Confirm the exact MySQL release with the prerequisites page when you choose a version.
Scoped service account and RBAC
Give the Airflow workers a Kubernetes service account that can create, read, watch and delete pods only in the namespace used for task workloads. Avoid cluster-wide roles and do not let task pods reuse that account's token. Ask the cluster team to confirm the exact verbs needed for log retrieval and cleanup, since missing rights often surface only when a pod finishes or a deferred task resumes.
Registry access for task images
Each task runs in a container image, so the cluster must be able to pull those images from a registry you control. Set an image pull secret in the task namespace, pin images by digest where your process allows, and confirm the registry is reachable from every node pool that runs tasks. Use container images for Airflow itself as well, since the production guide recommends them for Kubernetes deployments.
Network paths to the API and data stores
Plan the network routes before deployment: the scheduler and triggerer must reach the Kubernetes API, task pods must reach the metadata-adjacent services they need, and the metadata database must be reachable from the scheduler, workers and API server. Ask the network team to document firewall rules and egress policies for air-gapped or restricted segments, because a blocked API path usually appears as pods stuck in pending or deferred tasks that never resume.
Configuration sequence from first pod to production
This sequence takes a KubernetesPodOperator task from a connection test to a DAG promoted through staging, with a named check closing each step. Each check confirms that the cluster, permissions, pod spec, resources, deferral and cleanup behave as intended before the next layer is added.
Install the Kubernetes provider and test the cluster connection
The platform engineer installs the Kubernetes provider package in the Airflow environment, using the same version constraints the rest of the Airflow image uses. They then create a Kubernetes connection in the Airflow UI or secrets backend that points at the target cluster, with the credential type your cluster policy allows. The step is done when a test of that connection succeeds and the cluster is reachable from the component that will launch pods, which in a Kubernetes deployment is typically the Airflow worker or the task pod launcher. Confirm the exact provider version and connection fields against the provider documentation for your Airflow release before you start.
Create a namespace and least-privilege service account
The cluster administrator creates a dedicated namespace for task pods, separate from the Airflow control plane, and a service account bound to a Role that grants only the verbs the operator needs in that namespace, such as creating, reading and deleting pods and reading their logs. The Airflow connection then authenticates as that service account rather than as a cluster administrator. The check is an access review: the service account can create and inspect pods in the task namespace and is denied in other namespaces and on cluster-scoped resources. Ask your security team to review the Role against the operator's required permissions before it is bound.
Run a minimal pod spec manually
The DAG author writes a single KubernetesPodOperator task with the smallest working pod: a known container image, a short command that prints a line and exits, and the target namespace. The task is triggered by hand from the Airflow UI with no schedule attached. The check is that the pod reaches the Succeeded phase in Kubernetes, and that its log lines appear in the task log in the Airflow UI. If the pod stays Pending or fails to pull the image, fix the image reference, registry credentials or node selection before adding anything else.
Add resource requests and limits
The platform engineer and DAG author set CPU and memory requests and limits on the container, and confirm the namespace has a ResourceQuota or LimitRange if your cluster policy requires one. Requests determine scheduling, and limits protect neighboring workloads from a runaway task. The check is a repeat run of the same task with the new settings: the pod is scheduled onto a node, the Kubernetes event log shows no eviction or OOMKilled status, and the pod reaches Succeeded. Size the values from measured runs of the real workload, not from guesses, and record them with the DAG so reviewers can see the reasoning.
Enable deferrable mode for long-running pods
For tasks that run for more than a few moments, the DAG author enables deferrable mode so the operator hands the wait over to the triggerer instead of holding a worker slot while the pod runs. The triggerer process must be running in the Airflow deployment, and it must have network access to the Kubernetes API. The check is a run where the task instance moves into a deferred state while the pod is running, the worker slot is released, and the task resumes and completes when the pod finishes. Confirm the triggerer is sized and monitored, since a stopped triggerer will leave deferred tasks waiting.
Configure cleanup for failed and successful pods
The DAG author sets the cleanup behavior so finished pods do not accumulate in the namespace. Decide separately what happens to pods that succeed and to pods that fail, because failed pods are usually kept longer for diagnosis. The platform team confirms who can read those retained pods and their logs, since they can contain data and environment details. The check is two test runs, one forced to succeed and one forced to fail, after which the successful pod is removed as configured and the failed pod remains available for inspection for the period you set. Confirm the exact cleanup options available in your provider version.
Promote the DAG through staging to production
The team moves the DAG, its pod spec and its connection references through a staging Airflow environment before production, using the same image tag, namespace pattern and service account names with environment-specific values. Staging runs the scheduled version against a representative dataset and a non-production cluster or namespace. The check is a clean scheduled run in staging, with pods succeeding, logs visible, cleanup behaving as configured and no secrets appearing in pod specs or task logs. Only then is the DAG deployed to production, with the same review record that the staging run produced.
Passing secrets: Kubernetes Secrets or Airflow connections
Environment variables are the most common first attempt, and they are the weakest option for credentials. A value set as an environment entry in the pod spec is readable by anyone who can read that spec, so it appears in manifests, rendered templates and cluster API responses. Once the process starts, the value can also surface in process listings, crash dumps and any log line where the application prints its configuration. Rotating an environment variable means changing the spec and restarting the pod, and nothing in the platform records who read the value. Teams that need to pass a secret to a container should treat environment entries as a last resort, reserved for non-sensitive settings such as a region name or a feature flag.
- Visibility in specs: Anyone with read access to pod or template objects in the namespace can see the value in plain form.
- Visibility at runtime: Process listings, debugging output and error reports from the application can expose the value.
- Log leakage: A single verbose logging statement in the task code can write the credential to Airflow task logs, which many teams keep for a long time.
- Weak rotation: Changing the value requires editing the DAG or template and redeploying the task definition.
- Weak audit trail: Reading an environment entry leaves no event that a security team can review.
Before accepting any environment-based design, confirm with the security team which logging and retention rules apply to Airflow task logs in your environment.
Mounting a Kubernetes Secret moves the credential out of the DAG code and into a cluster object that the platform team manages. The pod receives the value as a file in a volume, or as an environment entry sourced from the Secret, and the DAG refers only to the Secret name and key. Access is controlled by Kubernetes role-based access control, so the cluster administrators decide which service accounts may read which Secrets. This approach fits teams that already run their secret lifecycle inside the cluster, and it works regardless of which Airflow executor schedules the task. The trade-off is that the platform team, not the DAG author, owns rotation, and the Secret itself is only base64-encoded unless the cluster adds encryption at rest and a restricted read policy.
- Volume mounts over environment entries: Mounting the Secret as read-only files keeps the value out of the container's environment and out of most process listings.
- Scoped service accounts: Give each DAG or team a dedicated service account that can read only the Secrets that task needs.
- Encryption at rest: Confirm with the cluster team that Secrets are encrypted in the datastore, and that backups of the datastore are protected to the same standard.
- Rotation procedure: Define who updates the Secret, how running pods pick up the new value (file mounts refresh, environment entries do not), and how the change is recorded.
- Audit evidence: Enable Kubernetes audit logging for Secret reads and writes, and confirm the retention period with the security team.
Before relying on this approach, check that the cluster's audit policy records Secret access, because the default level of logging varies by distribution and configuration.
Airflow connections hold host, login and password values for external systems, and a secrets backend can supply them from an external store at runtime. The DAG asks for a connection by its ID, and the value is resolved when the task runs rather than being written into the pod spec. This keeps credentials in one governed location that the data or platform team administers, and it lets a single rotation update every DAG that uses the connection. The pod still needs a way to reach the Airflow connection or the backend, so the design has to decide whether the container fetches the value itself or whether the operator resolves it and passes it on. That choice determines which identity needs access, and it changes what a container can see once it is running.
- Connections as the source of truth: Store each credential once as a connection, and reference it by ID in the task definition.
- Secrets backend integration: When the value lives in an external store, confirm the backend setting and the path layout with the team that owns that store before writing DAGs.
- Minimal exposure in the container: Pass only the values the container needs, and confirm whether the operator writes them into the pod spec, which would bring back the visibility problems described above.
- Identity for the fetch: Decide which Airflow identity or Kubernetes service account performs the lookup, and restrict it to the connections that DAG uses.
- Audit trail: Confirm how connection changes are recorded in your Airflow deployment and in the external backend, and which of those records the audit team can access.
Before committing to this design, verify with your Airflow administrators how the operator resolves connections in your version, since the exact behavior depends on the provider package in use.
Most platforms end up with both approaches, and the choice depends on who owns the secret and who needs to change it. Credentials that belong to the cluster, such as a token the container uses to call a service inside the same cluster, often fit a Kubernetes Secret with a tight role binding. Credentials for external systems that several DAGs share, such as a warehouse login, usually fit an Airflow connection or a secrets backend, because one rotation can serve every consumer. The decision should be written down per credential, with its owner, its rotation trigger and the log that proves each read. An illustrative pattern is a team that keeps warehouse logins in connections and stores a short-lived API token for an internal service in a Secret that the platform team rotates on a schedule.
- Owner per credential: Name the team that rotates each secret and the team that consumes it.
- Rotation trigger: Decide whether rotation runs on a calendar, on staff changes or on a security event, and record the procedure.
- Blast radius: Keep shared credentials in one place so a compromise touches a known set of DAGs.
- Evidence map: List which logs prove access for each approach, such as cluster audit logs for Secret reads and Airflow activity records for connection changes.
- Restricted scope: Avoid giving one service account or one DAG access to every secret in the namespace.
Before finalizing the split, ask your security and compliance leads which evidence they need for each credential class, because the answer decides which logs must be retained and where.
Mapping Airflow tasks, connections and pod fields
This table maps each Airflow concept to the Kubernetes pod field it feeds, so a team can see where a task's identity, credentials, arguments, namespace and resources end up. Read each row as a design decision, and confirm the exact parameter names against the Kubernetes provider documentation for the version you install.
| Airflow concept | Pod field or Kubernetes object | Design note |
|---|---|---|
| Task ID | Pod name prefix | Task IDs are sanitized and suffixed to form the pod name, so long or unusual IDs can change how pods are found in logs. Confirm the naming behavior in your provider version. |
| Connection ID | Kubernetes Secret reference | Keep the connection in Airflow and reference a Secret by name, so the pod receives credentials at runtime instead of values written into the DAG file. |
| Airflow variable | Container argument or mounted config | Pass non-sensitive variables as arguments or mounted config values. Avoid putting sensitive variables in arguments, which can appear in pod specs and events. |
| Params | Container command or arguments | Params are rendered into the command, so validate them and treat any value that reaches a shell as untrusted input. |
| Team ownership | Namespace and service account | Give each team its own namespace and service account with narrow RBAC, so one team's pods cannot read another team's Secrets. |
| Resource requests | CPU and memory requests and limits | Requests drive where the scheduler places the pod. Set them from measured task usage, and confirm how your cluster's quota and limit ranges apply. |
| Image and pull secret | Container image and image pull secret | Pin images to tags or digests from your registry. Pull secrets must exist in the same namespace as the pod. |
Securing pods, images and secrets
Task pods run arbitrary code chosen by DAG authors, so the namespace, service account permissions, pod settings, images and network paths all need deliberate limits. The cards below cover each boundary and the checks to run before a DAG goes to production.
Dedicated namespace for task pods
Run task pods in a namespace used only by Airflow task execution, separate from the Airflow components, other applications and cluster system services. Keep the namespace name in configuration, not in DAG code, so DAG authors cannot point tasks at a namespace they should not reach. Ask your platform team to confirm how namespace quotas, limit ranges and labels will be applied to every task pod. Check that deleting a namespace or pod never removes Airflow metadata or logs.
RBAC limited to pod lifecycle
The Kubernetes service account used by the operator should hold only the permissions needed to create, read, watch, and delete pods in the task namespace, plus any pod log and event reads the operator needs. Avoid cluster-wide roles, secret read access in other namespaces, and permissions to modify deployments or role bindings. Ask your cluster team to test the role with kubectl auth can-i for each verb before go-live. Review the role whenever you upgrade the provider package or change how tasks fetch logs.
Pod security context and resource limits
Set a restrictive security context on each task pod: run as a non-root user, disable privilege escalation, drop Linux capabilities, and use a read-only root filesystem where the task allows it. Apply CPU and memory requests and limits so a runaway task cannot starve the node. Enforce these through Pod Security admission or a policy engine in the task namespace, so a DAG cannot opt out with a custom pod spec. Confirm the policy level with your security team before enabling it, since some task images need writable paths.
Image provenance and pull secrets
Pin task images to a specific digest or immutable tag, pull them only from a registry your organization controls, and keep the approved image list in version control. Store registry credentials as Kubernetes image pull secrets scoped to the task namespace, not inside the DAG file. Ask whether your registry can scan images and enforce signature checks before a pod starts. Treat any image that was not built through your pipeline as untrusted until it passes review.
Secrets outside DAG code and logs
Keep passwords, tokens and keys out of the DAG file, the env_vars argument and rendered templates. Pass them as Kubernetes Secrets mounted into the pod, or resolve Airflow connections at runtime, so they never appear in the DAG source. Configure Airflow's secrets masking and check task logs and the rendered template view for any value that leaked. Ask your security team to test a sample run with a canary secret and search the logs before production use.
Network policy for task pods
Task pods usually need to reach only their data source, the object store or API they call, and the logging endpoint. Apply a Kubernetes NetworkPolicy that denies ingress and limits egress to those destinations, and block access to the metadata database, the API server and other namespaces. Confirm with your network team that the policy is enforced by your CNI plugin, since some plugins ignore these rules. Document each allowed destination so an auditor can match it to a documented task purpose.
Deferrable mode and cleanup actions in practice
Deferrable mode changes where the waiting happens. Without it, a KubernetesPodOperator task holds a worker slot from pod launch until the container exits, even though the worker does little more than poll. In deferrable mode, the operator hands the wait to the triggerer, a separate Airflow component that watches many pods asynchronously, and the worker slot is released until the pod finishes. Confirm that the triggerer is running and healthy in your deployment before enabling the option, because a deferred task that cannot be resumed will sit in a waiting state. Treat the triggerer as a sizing decision of its own: the number of concurrent deferred pods, the polling load against the Kubernetes API, and the memory each trigger uses all grow with the number of long-running pods. Ask the platform team for the API server request budget and check it against your expected concurrency.
Cleanup behavior controls what remains in the cluster after a pod ends. The operator's cleanup settings decide whether successful pods are deleted, whether failed pods are kept, and for how long. Deleting every pod immediately keeps the namespace tidy, but it also removes the container state an engineer may need when a failure appears only under production data. Keeping failed pods gives investigators something to inspect with kubectl describe and logs, but it consumes scheduler objects, storage and quota, and a long-running pipeline that fails often can fill a namespace. Choose a retention period for failed pods, enforce it with a scheduled cleanup job or equivalent, and make sure task logs are shipped off the pod before deletion so the investigation does not depend on the pod surviving. Record the choice in your runbook.
Test both paths before production. Run a successful task and a deliberately failing task in each mode, confirm that the deferred task resumes after the triggerer restarts, and verify that the failed pod and its logs exist for the agreed retention window and are removed afterward. Include a triggerer restart and a node eviction in the test plan, since those are the events that expose gaps in deferrable handling. The Airflow executor documentation describes how executors interact with task state, which helps when you trace a stuck task. Check the exact operator parameters against the provider version you pin, because option names and defaults can change between provider releases.
Common KubernetesPodOperator failures and fixes
This table pairs the failures teams most often hit with KubernetesPodOperator with their usual causes and the first fix to try. Read each row by matching the symptom you see in the Airflow UI or cluster events to the cause, then apply the fix in the DAG code or the cluster configuration. Most of these issues come from the pod spec, RBAC, or how secrets and the triggerer are configured, so check those three areas before changing task logic.
| Symptom | Likely cause | Fix |
|---|---|---|
| Pod stays in Pending | The cluster has no node with enough free CPU or memory for the requested resources, or the node selector, affinity or toleration does not match any node | Compare the task's resource requests and node selector with node capacity and labels. Confirm which node pool the workload should use, then correct the selector or add a toleration for tainted nodes |
| ImagePullBackOff or ErrImagePull | The image is in a private registry and the pod has no image pull secret, or the secret is in another namespace, or the image tag does not exist | Create the registry secret in the namespace where the task pods run, reference it in the pod spec, and confirm the image tag exists in the registry |
| Forbidden errors from the Kubernetes API | The service account used by the Airflow worker or the pod lacks RBAC permissions to create, watch or delete pods, read logs, or manage secrets | Review the error for the verb and resource named, then grant only that permission to the service account in the task namespace. Ask your cluster administrator to confirm the role binding |
| Task logs missing after cleanup | The pod was deleted before Airflow collected its logs, often because on_finish_action is set to delete the pod, or the cluster removed it on completion | Keep the pod until logs are read, or set the cleanup action to keep pods on failure. Confirm that your remote logging backend captures container output |
| Secret not mounted or environment variable empty | The secret name, key or mount path in the pod spec does not match the Kubernetes Secret, or the Secret is in a different namespace than the pod | Check the Secret name and key against the spec in the same namespace, then restart the task. Prefer a mounted file or an Airflow connection over plain environment variables for credentials |
| Deferred task never resumes | The triggerer is undersized, has crashed, or is not running, so the trigger that watches the pod never fires | Confirm a triggerer process is running and healthy, check its logs and resource use, and size it for the number of deferred tasks. Set up alerts on its health so a stopped triggerer is visible |
| Task succeeds but the pod is still running | A deferrable or polling setting does not match the pod lifecycle, or the pod exits while Airflow is still waiting on a stale status | Review the operator's wait and polling settings and the pod's exit behavior. Check the pod status directly with kubectl and align the task's completion check with what the pod reports |
Questions teams ask before building KubernetesPodOperator tasks
How much effort does a first working KubernetesPodOperator task take compared with a production setup?
A first working pod is usually a small part of the work. A single task that launches one image into a test namespace can be proven quickly, but production needs much more: a pod spec per image with resource requests and limits, a service account with narrow permissions, log collection, cleanup behavior for failed or orphaned pods, secrets delivery, and a tested path for upgrading images. Duration depends on how many images, namespaces and teams are involved, and on how quickly your cluster team approves changes. Ask any partner for a written scope that lists each of these items as a deliverable, since that list is what determines the effort. The published duration for any given scope is not supplied here, so confirm it against your own environment during scoping.
What drives the cost of running isolated task containers on Kubernetes?
Cost comes from four areas. The first is cluster capacity: every pod consumes CPU and memory, so the number of concurrent task pods and their resource requests set how much node capacity you need. The second is team count, because each team that owns images adds specs, namespaces, review and support. The third is the number of isolated images, since each one needs its own build, pinned version and test run. The fourth is security review, which takes time from your security team and sometimes from an external tester. Recurring costs include node capacity, image rebuilds, patching and upgrades. In a quote, ask for each of these as a separate line so you can see which ones change if your scope grows.
Can an in-house platform team build and run KubernetesPodOperator tasks by itself?
Yes, a team that already runs Kubernetes and Airflow can build this, and the Airflow documentation and the Helm chart published by the Airflow community give a solid starting point. The harder parts are usually not the operator itself. They are the decisions around namespaces, RBAC, image promotion, secrets handling and cleanup, which need agreement from security and data owners. Teams that lack recent Kubernetes or Airflow 3 experience often spend more time on the upgrade path, since Airflow 3 changed the component layout and removed some older features. A useful test is whether someone on your team can own the pod specs, the RBAC model and the runbook for failed pods after the project ends.
What does DeployOpen need from our side before it starts work on pod-based tasks?
DeployOpen works inside your infrastructure, so the engagement depends on access and decisions from your team. Expect to provide:
- Cluster access for the namespaces Airflow will use, with a named owner for approvals.
- An image registry where task images can be built, scanned and pulled, along with credentials for the pull.
- RBAC approval for the service accounts and roles the operator needs to create and clean up pods.
- Sign-off on how secrets are delivered, whether through Kubernetes Secrets, Airflow connections or an external store.
- A decision on which Airflow version and executor setup the tasks will run under.
Schedule time for these reviews early, because they often set the timeline more than the build does.
Does the KubernetesPodOperator depend on which Airflow executor I choose?
No. The KubernetesPodOperator is a task operator that launches a pod for one task, and it can be used with several executors, including CeleryExecutor and KubernetesExecutor. Since Airflow 2.10.0, several executors can also run in one deployment, so a task-level pod can coexist with other execution paths. Executor selection is a separate decision that affects scaling, worker isolation and operations, and it should be made on its own merits. Keep in mind that the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0, so do not design around them. Confirm the combination you plan to use against the executor documentation before you commit to it.
How should secrets reach a task pod without ending up in environment variables?
Environment variables are easy to read from the pod spec, logs and process listings, so many teams prefer other paths. Two common options are Kubernetes Secrets mounted as files with tight RBAC on who can read them, and Airflow connections that the task fetches at run time, so the credential stays in Airflow's secret handling. Each option needs an owner, a rotation process and a review of who can read the Secret object. Decide this before writing pod specs, because changing it later means editing every task. Your security team should sign off on the choice, and the test plan should confirm that secrets do not appear in task logs or the pod definition.
What goes wrong most often with pod cleanup and deferrable tasks?
Two problems show up most often. The first is leftover pods: a failed task, a killed worker or a changed image can leave pods running or in error state, which consumes capacity and can hide the real failure. Set cleanup behavior on purpose and test it with failing runs. The second is worker slots held by tasks that wait for a pod to finish. Deferrable mode releases the worker while the triggerer watches the pod, but it needs a triggerer that is sized and running. Test both paths in staging, and check the Airflow logs and Kubernetes events together when a run hangs. Operational detail such as retention and alerting is worth agreeing before go-live.
Do pod-based tasks help with compliance evidence for scheduled work?
They can help, because each task runs in a defined container with a pinned image, a known service account and recorded logs, which gives auditors something concrete to review. Evidence still has to be produced and kept. Ask whether you need records of image versions per run, RBAC reviews, secrets rotation and change approvals, and confirm where those records will be stored. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports or certifications. Independent auditors and certification bodies make those determinations, so involve them early in the design.
How DeployOpen scopes and delivers this setup
DeployOpen treats the KubernetesPodOperator as part of a wider Airflow 3 platform that runs in your own clusters. We agree the pod specs, secrets approach, cleanup behavior and verification in writing before build work starts, then hand over a setup your platform and data teams can run.
How we scope
- Inventory the task types that will run as pods, their images, resource needs and runtime limits
- Agree the namespace layout, service accounts and secret sources in the written scope
- Confirm the executor in use and the Kubernetes versions your clusters run, since both shape the pod setup
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts
How we build
- Create reusable pod templates and operator defaults for the approved images and resource classes
- Configure deferrable mode so waiting pods do not hold worker slots, then test it on your triggerer setup
- Set up cleanup actions and pod retention so failed and completed pods are handled predictably
- Move credentials into Kubernetes Secrets or Airflow connections and check that none appear in pod environment variables
How we hand over
- Deliver documented pod specs, operator conventions and secret-handling rules for your DAG authors
- Provide test evidence for task success, failure, retry, cancellation and cleanup behavior
- Walk your platform and data engineers through troubleshooting for stuck, evicted and orphaned pods
- Record configuration and ownership so your team can change images and resource limits safely
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover
- Review the setup when Airflow or Kubernetes versions change, including the Kubernetes versions your Airflow release supports
- Revisit secret handling and access reviews as new teams and DAGs adopt pod-based tasks
- Scope any follow-on changes in writing so cost and timing stay visible before work begins
