Skip to content

Improve max_active_runs visibility across logs, API, and UI #73686

Description

@seanmuth

Today, when a Dag is at max_active_runs, there is very little signal anywhere that explains it, across three different surfaces:

  • Logs: the only existing log message covering this lives in the scheduler's periodic bookkeeping (_set_exceeds_max_active_runs), and as of Log blocking run/task IDs when a DAG hits max_active_runs #72403 it includes the blocking run/task IDs — but only at the moment a Dag newly becomes blocked. A Dag Run created after the Dag is already at capacity (e.g. a second manual trigger while the first run is still active) produces no log output anywhere explaining why it never starts.
  • API: DagModel.exceeds_max_non_backfill (added in 3.2.0 by migration 0099_3_2_0_add_exceeds_max_runs_flag_to_dag_model) is persisted to the dag table but is never exposed on DAGResponse (GET /dags/{dag_id}) or anywhere else in the public API. There is no way for a client to ask "is this Dag currently blocked on max_active_runs" without independently computing active-run counts.
  • UI: the Dag header's "Active Runs" stat (airflow-core/src/airflow/ui/src/pages/Dag/Header.tsx) shows X of Y once a Dag has active runs, but nothing explains what happens to a run that's queued up behind the limit.

Related: #72387 / #72403 — that work enhances the "Dag newly became blocked" log message with blocking run/task IDs. It is complementary to, not overlapping with, the gaps above (different trigger point, different code path).

Historical note: the original motivating example for this issue was seeing X of Y with X > Y (e.g. "2 of 1") in the 3.2.2 UI, which reads as an outright illogical number. On current main that specific presentation can no longer happen — active_runs_count (the X) is RUNNING-only and capped by the scheduler's promotion gate — thanks to two already-shipped, independent upstream fixes: #69769 (narrowed the count to RUNNING-only; previously counted RUNNING+QUEUED) and #72371 (relabeled the stat from "Max Active Runs" to "Active Runs" to match). Older Airflow/Astro Runtime builds cut before both of those will still show the old, confusing behavior. The underlying problem this issue tracks is still real on main — a Dag can have runs genuinely stuck queued behind the limit with only an inferred indication in the UI (X = Y) — it just no longer surfaces as a strange-looking number.

Follow-up work

One PR per item, so each can be reviewed independently:

  1. Log when a newly-created Dag Run will not be scheduled yet because the Dag is already at max_active_runs (covers the manual-trigger-while-blocked case that Log blocking run/task IDs when a DAG hits max_active_runs #72403 does not). → Log when a newly created Dag Run will not be scheduled due to max_active_runs #73689 (merged/open)
  2. Expose active-run-related fields on DAGDetailsResponse: is_at_max_active_runs (aliased to exceeds_max_non_backfill) and queued_runs_count (live-computed, same pattern as the existing active_runs_count). Also fixes two real bugs found while building this: the dag-processor's active-run-count calculation hardcoded 0 for any Dag whose timetable can't be scheduled (e.g. schedule=None, triggered only manually/via the API), which made exceeds_max_non_backfill permanently unreliable for exactly the Dags that most need it; and a pre-existing N+1 query in the same code path (one active_runs_of_dags query per Dag per parse cycle instead of one batched query per parse). → Expose exceeds_max_active_runs on the Dag details API response #73692
  3. Add an info icon + tooltip next to the "Active Runs" stat in the Dag header UI, shown whenever queued_runs_count > 0, and display the queued count inline (1 of 1 (2 queued)) so the information doesn't require a hover. Keyed off the live queued_runs_count rather than the periodically-recomputed is_at_max_active_runs flag, so the UI's correctness doesn't depend on that flag's refresh cadence. Went with an info icon rather than a warning, since (per the historical note above) the number itself never looks alarming on main — the queued state is invisible, not alarming-looking, so "info" fits better than "warning". → UI: Info tooltip when a Dag's active runs exceed max_active_runs #73693 (builds on Expose exceeds_max_active_runs on the Dag details API response #73692)

Drafted-by: Claude Sonnet 5 (reviewed by @seanmuth)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions