Metric charts read data points flagged "no recorded value" (Prometheus staleness markers) as zeros #3511

What happens

When a scraped series goes away (its pod is deleted, or the target stops exporting it), the OpenTelemetry Collector's prometheus receiver sends one last data point with the DATA_POINT_FLAGS_NO_RECORDED_VALUE_MASK flag. This is the OTLP form of a Prometheus staleness marker. The ClickHouse exporter stores it as a row with Value = 0 and Flags = 1.

HyperDX's metric queries don't look at Flags, so they read that row as a real value of 0. At the moment a series ends:

• min drops to 0,
• avg drops (one series at 10 plus a marker gives 5),
• count still includes the series that went away.

Alerts use the same queries, so a gauge alert on min or avg with a below threshold sees the same dip.

The proto defines the flag as missing data, not as a value:

This DataPoint is valid but has no recorded value. This value SHOULD be used to reflect explicitly missing data in a series, as for an equivalent to the Prometheus "staleness marker".

(metrics.proto, DataPointFlags)

Reproduction

Integration test against ClickHouse on main (7a9fe57), renderChartConfig with a gauge:

• Series a and b of one gauge report 10 every minute.
• Series b goes away after minute 4. At minute 5 there is its marker: Value = 0, Flags = 1.
• Line chart, 1-minute buckets.

All other minutes are correct.

How often this happens: in a small Kubernetes cluster scraped by the collector, I see about 1,800 flagged rows per hour in the gauge and sum tables, one for every series of every pod that is deleted.

Suggested fix

Leave out flagged rows wherever the metric tables are read: bitAnd(Flags, 1) = 0 in the WHERE clause of the gauge, sum, histogram, exponential histogram and summary queries in renderChartConfig. A flagged row never carries a value, so this doesn't change results for data without markers.

I can open a PR with the fix and an integration test, if that sounds right.