[Logstash - Investigation] Concurrent `PipelineAction::Create` at boot can raise `MetricStore` `KeyError: key not found`

[alexcams]

When several pipelines start concurrently, one pipeline's Create intermittently fails with KeyError: key not found while registering flow metrics. Evidence points to a race condition in metric registration, needs investigation to confirm and to implement the proper fix.

Evidence

• Backtrace (key frames):

  KeyError: key not found
    concurrent/map.rb:in `raise_fetch_no_key' / `fetch' / `fetch_or_store'
    logstash/instrument/metric_store.rb:in `block in fetch_or_store'
    logstash/instrument/collector.rb:in `register?'
    org/logstash/execution/AbstractPipelineExt.java:in `initialize_flow_metrics'
    logstash/agent.rb:in `block in converge_state'
  

• Intermittent (a different pipeline each time). Signature of a timing race, not a config/compile error.
• At boot each Create runs on its pwm, so many threads hit the single process-wide collector at once.

Code context

The random, intermittent pipeline error looks like a race condition, but the raised exception comes from a call protected by @structured_lookup_mutex, so I think every structured-store write is serialized. A call serialized that way would behave single-threaded and normally never raise. Resolving that contradiction is the goal of this issue.

• The fetch_or_store call in Logstash that raises — metric_store.rb#L85
• Where it throws in concurrent-ruby (~> 1.3) — raise_fetch_no_key map.rb#L327