outputs.graphite: TestIntegration fails intermittently in the integration CI job

[skartikey]

Relevant telegraf.conf

# Not applicable. This is a flaky test in CI, not a runtime issue.

Logs from Telegraf

--- FAIL: TestIntegration (45.01s)
    graphite_test.go:NNN:
        Error:      Condition never satisfied
FAIL    github.com/influxdata/telegraf/plugins/outputs/graphite  45.076s

System info

Telegraf master, ci/circleci: test-integration, machine: ubuntu-2204:current on large.gen2

Docker

graphiteapp/graphite-statsd, started by the test through testcontainers.

Steps to reproduce

1. Open any pull request, or push to master.
2. Wait for ci/circleci: test-integration.
3. Roughly half the time plugins/outputs/graphite fails on TestIntegration.

Expected behavior

test-integration passes or fails based on the change under test.

Actual behavior

TestIntegration in plugins/outputs/graphite fails intermittently with Condition never satisfied, on changes that cannot affect it. Four of the last seven runs failed, across three unrelated pull requests:

Job	Branch	Result
563250	pull/19778	failed, 45.08s
563140	pull/19753	passed
563134	pull/19779	failed, 44.06s
563108	master	passed
563099	pull/19778	failed, 42.57s
563080	pull/19753	failed, 43.03s
563031	pull/19747	passed

The diffs on those branches touch .circleci/config.yml and the vsock code, so none of them reach this plugin.

Additional info

The test writes two metrics to carbon on port 2003, then polls /metrics/index.json for both series with require.Eventually and a 10 second window:

require.Eventually(t, func() bool {
    ...
}, 10*time.Second, 100*time.Millisecond)

The container wait condition covers ports 8080, 2003 and 2004 plus the run: statsd: log line, so the container is up before the write. What is not covered is the gap between carbon accepting the metric and the series becoming visible in the index, and 10 seconds appears to be too close to that on a loaded runner.

I have not confirmed which half of that gap is responsible. The two candidates are carbon's cache flush, since the whisper file does not exist until it flushes, and whether /metrics/index.json in this image serves a periodically rebuilt index rather than walking the tree live. Worth checking before picking a fix.

The cheap fix is a longer window. The failing runs take 42 to 45 seconds against a 10 second wait, so the budget is there. A better fix would be waiting on something that actually signals the write landed.

Raising this mainly because a test that fails half the time on unrelated changes trains everyone to ignore a red test-integration.