Keeping a self-hosted Airflow fast and observable: scheduler and parser tuning, concurrency limits, worker autoscaling, metrics, database cleanup, backups and troubleshooting stuck or missing runs.
Scaling Celery workers on queued and running tasks with the Helm chart KEDA integration: how the query sizes workers, worker_concurrency, scale to zero, cooldown versus grace period, and why KEDA and HPA must not compete.
Reducing scheduling delay in large deployments: running multiple schedulers, parallelism and per-DAG limits, database load from scheduling loops, scheduler metrics to watch, and when the database is the real bottleneck.
Fixing slow or timing-out DAG files: top-level code to remove, dagbag_import_timeout, parsing order and intervals for thousands of files, the parse time metrics, and splitting generated DAGs.
Protecting databases and APIs from too many parallel tasks: global, DAG and task-level limits, pools and their slot metrics, priority weights, Celery queues, and limits for dynamically mapped tasks.
Controlling metadata database growth: what airflow db clean removes, retention for audit needs, the Helm chart cleanup CronJob, running cleanup before upgrades and checking the impact on history in the UI.
Running the Airflow 3 API server under real UI and API load: the one-worker default from 3.1, gunicorn with zero-downtime recycling from 3.2, horizontal replicas, the shared secret key and load balancer settings.
Recovering Airflow after losing a cluster or region: what to back up (metadata database, DAG repository, secrets, logs), consistent database backups, restore order, avoiding duplicate runs after recovery and testing the plan.
Choosing how Airflow emits metrics: StatsD versus OpenTelemetry settings, telling schedulers apart, allow and block lists, custom metrics from tasks, and the metrics that matter for alerting.
Getting Airflow metrics into Prometheus without a native endpoint: the OpenTelemetry Collector route with native histograms, statsd_exporter, PgBouncer and Flower metrics from the Helm chart, and dashboards and alerts to build.
Diagnosing tasks that sit in queued or fail without logs: task_queued_timeout, executor and worker capacity, pool slots, Celery broker and KEDA scaling issues, and the metrics that show where tasks wait.
Finding why a DAG does not appear or shows import errors: the DAG processor, import timeouts, missing dependencies, duplicate DAG IDs, reserializing DAGs and the import error metric to alert on.
Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.