The payload attestation duty always stops: the request for the payload attestation data uses the wrong URL format #1291

Summary

Anchor puts the slot in the URL path when it asks the beacon node for the payload attestation data.
Lodestar gives this endpoint with the slot as a query parameter. Lodestar answers with HTTP 404.
Anchor then writes No block received for slot, skipping payload attestation and stops the duty.

As a result, Anchor never sends a partial signature for the payload attestation duty. In a committee
that contains two Anchor operators, this duty never gets a quorum.

Test environment

• Network: ssv-mini, a local Kurtosis testnet with 4 SSV operators. The quorum is 3 of 4.
• Operators 1 and 2 run Anchor. Operators 3 and 4 run the Go SSV node.
• Anchor: branch epbs, commit 52ba589, version string Anchor/v1.3.1-52ba589.
  It pins sigp/lighthouse at rev e58ec88fe.
• Go SSV node: branch epbs-gloas, commit 613dd52c9.
• Consensus client: Lodestar v1.46.0/aa8303e. All 4 operators use the same beacon node.
• Execution client: go-ethereum, glamsterdam-devnet-8 image.
• Gloas fork epoch: 4 (slot 128). The Boole fork is not scheduled.
• 10 validators, indices 64 to 73.

What we saw

We ran the network for 3 epochs after the Gloas fork.

• Anchor, on each of the two nodes: 36 x Producing payload attestations, and then
  36 x No block received for slot, skipping payload attestation.
  0 x Successfully published payload attestations.
• Lodestar: 73 x warn: Route GET:/eth/v1/validator/payload_attestation_data/<slot> not found.
  72 of these agree with the 72 Anchor skips. The times agree to the millisecond. The last one is
  our own manual test request.
• Go SSV node: 30 payload attestation duties. Each duty got only 2 partial signatures, from the two
  Go SSV operators. The quorum is 3. All 30 duties failed with
  duty did not reach signature quorum before slot end.
• The committee published 0 payload attestations.

The two URL formats

GET /eth/v1/validator/payload_attestation_data/240        -> HTTP 404
GET /eth/v1/validator/payload_attestation_data?slot=240   -> HTTP 200
{"data":{"beacon_block_root":"0x8bff1f43...","slot":"240","payload_present":false,
 "blob_data_available":false},"version":"gloas"}

The Go SSV node uses the query parameter format (beacon/goclient/ptc.go), and it gets the data from
the same beacon node. The query parameter format is also the format of the other validator endpoints,
for example attestation_data?slot=&committee_index=.

The log message shows the wrong cause

The chain had a block at each of these slots. We read the block header of every slot in the window:
each slot has a canonical block. The blocks contain payload attestations with payload_present: true.

At slot 134 Anchor was the block proposer. It wrote New head from beacon node ... slot=134 8.8
seconds before it wrote No block received for slot.

The message tells the operator that the block is missing. The true cause is the failed request.

What we expect

1. Send the slot as a query parameter.
2. Write a different message when the request fails. Keep No block received for slot only for a
   slot that has no block.

Where the code is

We did not find these strings in the anchor repository at commit 52ba589. They come from the
pinned sigp/lighthouse validator_services crate at rev e58ec88fe. A change in Lighthouse can be
necessary.

Related

This defect stops all our tests of the payload attestation duty on a mixed committee. We could not
test the abstention rules, the behaviour when one slot has many payload attestation duties, or the
submission time.