Skip to content

fix(api): resolve a vanished alert group with its values - #3507

Merged
kodiakhq[bot] merged 4 commits into
ClickHouse:mainfrom
andrewdunndev:ai/alert-resolve-vanished-group-values
Nov 3, 2026
Merged

kodiakhq[bot] merged 4 commits into
ClickHouse:mainfrom
andrewdunndev:ai/alert-resolve-vanished-group-values

Conversation

@andrewdunndev

Copy link
Copy Markdown
Contributor

Summary

Closes #3470

Stores a grouped alert's group-by values on its history row and reads them back at resolve time, so {{attributes.*}} and {{var.*}} are filled in when a group resolves after it has disappeared from the data. Until now they rendered empty on that resolve: the group had no rows left to read values from, and the joined group string on the history row cannot be split back into values.

  • groupValues is a list of { key, value } pairs rather than an object, since group-by column names can contain dots and Mongo would read them as nested paths.
  • Live values still win. The history fallback applies only when the group has no current rows.
  • Empty values are stored too. History rows written before this change have no groupValues; they resolve as they do today, and the next evaluation of a live group stores the values.
  • The docs no longer say {{var}} is empty on the resolve of a vanished group.

How to reproduce

Starting point: yarn dev, a webhook destination whose body template uses {{var.service}}, and a group-by alert on service.

On main:

  1. Let the alert fire for one service value, then stop sending data for that value and wait out the alert window.
  2. The next evaluation resolves the alert, and the webhook body has an empty {{var.service}}.

On this branch:

  1. Repeat the steps above.
  2. The resolve body carries the group's service value.

make dev-int FILE=checkAlerts runs the new cases: plain and dotted aliases, a generic webhook body, an empty value, and a legacy row.

How to test on Vercel preview

N/A, API-only change

References

  • Linear Issue:
  • Related PRs:

Store a group-by alert group's values on its history row and fall back to
them when the group has no rows left at resolve time, so
{{attributes.x}} and {{var.x}} render instead of coming out empty.
The joined group string cannot be parsed back into values.
@andrewdunndev
andrewdunndev requested a review from a team as a code owner October 30, 2026 23:04
@vercel

vercel Bot commented Oct 30, 2026

Copy link
Copy Markdown

@andrewdunndev is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@changeset-bot

changeset-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c8d13d2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

✅ No issues found.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a clean, backward-compatible additive change. The new groupValues field is optional (required: false, default: undefined), legacy rows without it resolve exactly as before via groupValuesToAttributes(undefined) => undefined, group-by column names are stored as string values of a generic key field (sidestepping Mongo's dotted-key nested-path interpretation), and the live-values-win precedence is preserved by the groupAttributes.get(groupKey) ?? groupValuesToAttributes(...) fallback ordering. The aggregation only materializes one small extra field per grouped row, well below the full-document hydration the existing $first optimization was guarding against.

🟡 P2 — recommended

  • packages/api/src/tasks/checkAlerts/index.ts:1480 — The new resolve fallback has a precedence branch (groupAttributes.get(groupKey) live values winning over stored previousHistory.groupValues), but every new test vanishes the group before resolve, so only the stored-fallback arm is exercised; the live-wins arm is asserted nowhere.
    • Fix: Add a resolve test where the group still reports rows at resolve time and assert the notification renders the current live values rather than the persisted groupValues.
    • testing, correctness
🔵 P3 nitpicks (2)
  • docs/alert-webhook-template-variables.md:24 — The {{var}} row now reads "Empty when not grouped," dropping the vanished-group clause, but a grouped alert whose history predates this change (no stored groupValues) still renders {{var}} empty on resolve, so the behavior is narrower than the new wording implies.

    • Fix: Note that {{var}} can still be empty when resolving a group whose fired history row was written before this change.
  • packages/api/src/models/alertHistory.ts:66 — IAlertHistory.groupValues is typed GroupValue[] (so | undefined) while AggregatedAlertHistory.groupValues is GroupValue[] | null; the asymmetry is intentional (Mongo $first yields null) but undocumented at the declaration.

    • Fix: Add a short comment on AggregatedAlertHistory.groupValues explaining the | null comes from the aggregation $first, so the two declarations don't read as an accidental mismatch.

Reviewers (11): correctness, adversarial, data-migrations, api-contract, reliability, performance, testing, maintainability, kieran-typescript, project-standards, previous-comments.

Testing gaps:

  • No direct assertion that live current values take precedence over stored groupValues at resolve time (only the vanished-group fallback is covered).
  • DocumentDB $group/$first parity for the new groupValues projection is assumed rather than verified in the integration suite.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@andrewdunndev

Copy link
Copy Markdown
Contributor Author

Thanks, all three are in 3b2a6d9: attributesToGroupValues and groupValuesToAttributes sit next to the model and are used at both sites, with the empty case handled in the helpers; GroupValue is exported and used for both declarations; and AggregatedAlertHistory.groupValues is typed GroupValue[] | null.

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@kodiakhq
kodiakhq Bot merged commit 0453aa1 into ClickHouse:main Nov 3, 2026
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Grouped alert resolve renders empty group values when the group vanishes from the data

2 participants