You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Today, when a Dag is at max_active_runs, there is very little signal anywhere that explains it, across three different surfaces:
Logs: the only existing log message covering this lives in the scheduler's periodic bookkeeping (_set_exceeds_max_active_runs), and as of Log blocking run/task IDs when a DAG hits max_active_runs #72403 it includes the blocking run/task IDs — but only at the moment a Dag newly becomes blocked. A Dag Run created after the Dag is already at capacity (e.g. a second manual trigger while the first run is still active) produces no log output anywhere explaining why it never starts.
API: DagModel.exceeds_max_non_backfill (added in 3.2.0 by migration 0099_3_2_0_add_exceeds_max_runs_flag_to_dag_model) is persisted to the dag table but is never exposed on DAGResponse (GET /dags/{dag_id}) or anywhere else in the public API. There is no way for a client to ask "is this Dag currently blocked on max_active_runs" without independently computing active-run counts.
UI: the Dag header's "Active Runs" stat (airflow-core/src/airflow/ui/src/pages/Dag/Header.tsx) shows X of Y once a Dag has active runs, but nothing explains what happens to a run that's queued up behind the limit.
Related: #72387 / #72403 — that work enhances the "Dag newly became blocked" log message with blocking run/task IDs. It is complementary to, not overlapping with, the gaps above (different trigger point, different code path).
Historical note: the original motivating example for this issue was seeing X of Y with X > Y (e.g. "2 of 1") in the 3.2.2 UI, which reads as an outright illogical number. On current main that specific presentation can no longer happen — active_runs_count (the X) is RUNNING-only and capped by the scheduler's promotion gate — thanks to two already-shipped, independent upstream fixes: #69769 (narrowed the count to RUNNING-only; previously counted RUNNING+QUEUED) and #72371 (relabeled the stat from "Max Active Runs" to "Active Runs" to match). Older Airflow/Astro Runtime builds cut before both of those will still show the old, confusing behavior. The underlying problem this issue tracks is still real on main — a Dag can have runs genuinely stuck queued behind the limit with only an inferred indication in the UI (X = Y) — it just no longer surfaces as a strange-looking number.
Follow-up work
One PR per item, so each can be reviewed independently:
Expose active-run-related fields on DAGDetailsResponse: is_at_max_active_runs (aliased to exceeds_max_non_backfill) and queued_runs_count (live-computed, same pattern as the existing active_runs_count). Also fixes two real bugs found while building this: the dag-processor's active-run-count calculation hardcoded 0 for any Dag whose timetable can't be scheduled (e.g. schedule=None, triggered only manually/via the API), which made exceeds_max_non_backfill permanently unreliable for exactly the Dags that most need it; and a pre-existing N+1 query in the same code path (one active_runs_of_dags query per Dag per parse cycle instead of one batched query per parse). → Expose exceeds_max_active_runs on the Dag details API response #73692
Add an info icon + tooltip next to the "Active Runs" stat in the Dag header UI, shown whenever queued_runs_count > 0, and display the queued count inline (1 of 1 (2 queued)) so the information doesn't require a hover. Keyed off the live queued_runs_count rather than the periodically-recomputed is_at_max_active_runs flag, so the UI's correctness doesn't depend on that flag's refresh cadence. Went with an info icon rather than a warning, since (per the historical note above) the number itself never looks alarming on main — the queued state is invisible, not alarming-looking, so "info" fits better than "warning". → UI: Info tooltip when a Dag's active runs exceed max_active_runs #73693 (builds on Expose exceeds_max_active_runs on the Dag details API response #73692)
Drafted-by: Claude Sonnet 5 (reviewed by @seanmuth)
Today, when a Dag is at
max_active_runs, there is very little signal anywhere that explains it, across three different surfaces:_set_exceeds_max_active_runs), and as of Log blocking run/task IDs when a DAG hits max_active_runs #72403 it includes the blocking run/task IDs — but only at the moment a Dag newly becomes blocked. A Dag Run created after the Dag is already at capacity (e.g. a second manual trigger while the first run is still active) produces no log output anywhere explaining why it never starts.DagModel.exceeds_max_non_backfill(added in 3.2.0 by migration0099_3_2_0_add_exceeds_max_runs_flag_to_dag_model) is persisted to thedagtable but is never exposed onDAGResponse(GET /dags/{dag_id}) or anywhere else in the public API. There is no way for a client to ask "is this Dag currently blocked on max_active_runs" without independently computing active-run counts.airflow-core/src/airflow/ui/src/pages/Dag/Header.tsx) showsX of Yonce a Dag has active runs, but nothing explains what happens to a run that's queued up behind the limit.Related: #72387 / #72403 — that work enhances the "Dag newly became blocked" log message with blocking run/task IDs. It is complementary to, not overlapping with, the gaps above (different trigger point, different code path).
Historical note: the original motivating example for this issue was seeing
X of YwithX > Y(e.g. "2 of 1") in the 3.2.2 UI, which reads as an outright illogical number. On currentmainthat specific presentation can no longer happen —active_runs_count(theX) is RUNNING-only and capped by the scheduler's promotion gate — thanks to two already-shipped, independent upstream fixes: #69769 (narrowed the count to RUNNING-only; previously counted RUNNING+QUEUED) and #72371 (relabeled the stat from "Max Active Runs" to "Active Runs" to match). Older Airflow/Astro Runtime builds cut before both of those will still show the old, confusing behavior. The underlying problem this issue tracks is still real onmain— a Dag can have runs genuinely stuck queued behind the limit with only an inferred indication in the UI (X = Y) — it just no longer surfaces as a strange-looking number.Follow-up work
One PR per item, so each can be reviewed independently:
max_active_runs(covers the manual-trigger-while-blocked case that Log blocking run/task IDs when a DAG hits max_active_runs #72403 does not). → Log when a newly created Dag Run will not be scheduled due to max_active_runs #73689 (merged/open)DAGDetailsResponse:is_at_max_active_runs(aliased toexceeds_max_non_backfill) andqueued_runs_count(live-computed, same pattern as the existingactive_runs_count). Also fixes two real bugs found while building this: the dag-processor's active-run-count calculation hardcoded0for any Dag whose timetable can't be scheduled (e.g.schedule=None, triggered only manually/via the API), which madeexceeds_max_non_backfillpermanently unreliable for exactly the Dags that most need it; and a pre-existing N+1 query in the same code path (oneactive_runs_of_dagsquery per Dag per parse cycle instead of one batched query per parse). → Expose exceeds_max_active_runs on the Dag details API response #73692queued_runs_count > 0, and display the queued count inline (1 of 1 (2 queued)) so the information doesn't require a hover. Keyed off the livequeued_runs_countrather than the periodically-recomputedis_at_max_active_runsflag, so the UI's correctness doesn't depend on that flag's refresh cadence. Went with an info icon rather than a warning, since (per the historical note above) the number itself never looks alarming onmain— the queued state is invisible, not alarming-looking, so "info" fits better than "warning". → UI: Info tooltip when a Dag's active runs exceed max_active_runs #73693 (builds on Expose exceeds_max_active_runs on the Dag details API response #73692)Drafted-by: Claude Sonnet 5 (reviewed by @seanmuth)