Skip to content

[CRCR] Add additional info that helps triage the issue when health is degraded in PR and Nightly #8804

Description

@jewelkm89

[CRCR] Add additional info that helps triage the issue when health is degraded in PR and Nightly

Current Behavior

The health cards show only a summary:

  • PR tab: "Degraded - 4/5 recent PRs passed · pytorch/crcr-test"
  • Nightly tab: "Degraded - 3/5 recent nightlies passed"

No details about which PR/SHA failed or which jobs caused the failure.

Data Currently Available

PR Health (crcr_health_last_prs):

SELECT pr_number, last_run, successes, total, pass_rate
-- Returns aggregate per PR, no job-level details

Nightly Health (crcr_nightly_dashboard):

SELECT pytorch_head_sha, job_name, status, conclusion, started_at
-- Has job-level details but not surfaced in UI

Information Needed for Triage

Field PR Tab Nightly Tab
Which PR/SHA failed? Need pr_number Need pytorch_head_sha
Which job(s) failed? Not available Available but not shown
Job conclusion Not available Available but not shown
Link to failing run Not available Not available
Failure timestamp Not available Available but not shown

Proposed UI Changes

Option A: Expandable Card

┌─────────────────────────────────────────────┐
│ CRCR Relay Health                           │
│ Degraded ▼                                  │
│ 4/5 recent PRs passed                       │
├─────────────────────────────────────────────┤
│ ❌ PR #12345 - 3/4 jobs passed              │
│    └─ l2-critical: failure (link)          │
│ ✅ PR #12344 - 4/4 jobs passed              │
│ ✅ PR #12343 - 4/4 jobs passed              │
│ ...                                         │
└─────────────────────────────────────────────┘

Option B: Popup/Modal on Click

Click "Degraded" → opens modal with:

  • List of recent PRs/SHAs with pass/fail status
  • Expandable rows showing failed jobs
  • Direct links to GitHub Actions runs

Technical Changes Needed

1. New/Modified Query for PR Job Details

-- crcr_health_pr_job_details
SELECT 
  pr_number,
  job_name,
  conclusion,
  workflow_run_url,
  started_at
FROM default.crcr_workflow_job
WHERE downstream_repo = 'pytorch/crcr-test'
  AND pr_number IN (SELECT pr_number FROM recent_failing_prs)
  AND status = 'completed'
ORDER BY pr_number DESC, started_at DESC

2. Frontend Components

File Changes
torchci/pages/crcr/index.tsx Add state for expanded view, fetch job details
CrcrTestHealthCard Add expandable/clickable UI
CrcrNightlyHealthCard Surface existing job details from nightlyJobs

3. API Endpoint

New endpoint: /api/clickhouse/crcr_health_pr_job_details

Acceptance Criteria

  • Clicking health card shows which PRs/SHAs failed
  • Failed PRs/SHAs show which jobs caused failure
  • Each failed job has a link to the GitHub Actions run
  • Works for both PR and Nightly tabs
  • No ClickHouse query required for basic triage

Related

  • The nightly card already fetches job-level data via crcr_nightly_dashboard but only uses it for aggregate pass/fail calculation, not display

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions