feat(e2e): add two-chain anvil environment

[arnaubennassar]

Changes Summary

• Adds anvil-2chains, a hermetic two-L2 Anvil E2E environment assembled from the published kurtosis-cdk snapshot bundle.
• Migrates every Go E2E CI group, including the isolated force_ger_update test, from op-pp / op-pp-2chains to anvil-2chains.
• Makes anvil-2chains the local Go E2E default while retaining AGGKIT_E2E_ENV as an override.
• Generalizes test helpers and updates AutoClaim, proxy/tracker, remove-GER, and post-test bridge-health assumptions for the Anvil topology.

Config Updates

• Supplies separate rendered Aggkit configs for L2s 001 and 002, plus Agglayer and proxy config.
• Uses digest-pinned snapshot images for Anvil L1/L2 and Agglayer; Aggkit services continue to run aggkit:local.
• Explicitly enables the admin REST server on port 5579 and publishes it as 14579 / 15579 for AutoClaim approval tests.
• Uses TriggerCertMode = "ASAP", one-second AggOracle polling, and the current Agglayer pessimistic-proof settlement schema.

Testing

• go test -short ./test/e2e/...
• YAML parse/lint for .github/workflows/test-go-e2e.yml
• docker compose config --quiet for anvil-2chains
• Anvil lifecycle smoke, including startup, environment checks, post-test L1↔L2 bridge health check, and teardown
• TestAutoClaimL1ToL2APIApprove on anvil-2chains, including its post-test bridge health check

make lint could not run locally because golangci-lint is not installed in the development environment.

Related PRs

• kurtosis-cdk #929 publishes the snapshot bundle consumed by this environment.
• agglayer-dev-ui #24 consumes the same snapshot bundle.

The snapshot provenance, immutable image digests, port map, and regeneration procedure are documented in test/e2e/envs/anvil-2chains/README.md.

[arnaubennassar]

@claude

[claude]

Claude finished @arnaubennassar's task in 2m 12s —— View job

---

Code Review Complete ✅

---

🚀 What's New

This PR introduces anvil-2chains, a hermetic two-chain Anvil E2E environment sourced from kurtosis-cdk snapshot bundles, and migrates all Go E2E tests from op-pp environments to this new Anvil-based setup. The new environment uses digest-pinned snapshot images for L1/L2 execution (Anvil) and Agglayer, while continuing to test locally-built aggkit binaries.

🐛 Bug Fixes

None.

📋 Config Updates

• New Environment Default: anvil-2chains becomes the default for Go E2E tests (overridable via AGGKIT_E2E_ENV)
• Admin REST Server: Explicitly enabled on port 5579, published as 14579/15579 for AutoClaim approval tests
• Cert Triggering: Uses TriggerCertMode = "ASAP" with one-second AggOracle polling
• Settlement Schema: Uses current Agglayer pessimistic-proof settlement schema
• Port Mapping: Updated to avoid collisions with existing op-pp environments (L1: 13545, L2-001: 14545/14576/14577/14579, L2-002: 15545/15576/15577/15579)

⚠️ Breaking Changes

None. The existing op-pp environments remain available and AGGKIT_E2E_ENV can still override the default.

---

Review Feedback

🔴 Required Changes

1. Security: Private Keys in Summary JSON (test/e2e/envs/anvil-2chains/summary.json)

Severity: 🔴 Critical
Lines: 25-72, 125-174, 203-254

The summary.json file contains plaintext private keys for test accounts. While these appear to be well-known test mnemonics ("test test test..." and "giant issue aisle..."), this pattern is risky:

Issues:

• Private keys are stored in plaintext in version control
• If someone inadvertently funds these addresses on a real network, funds could be stolen
• Sets a dangerous precedent for handling sensitive data

Recommendation:

• Document prominently in the README that these are test-only keys that must NEVER be funded on real networks
• Consider adding a warning comment at the top of summary.json itself
• Add validation in the test setup to fail if these addresses have non-zero balance on any public network (mainnet, testnets)

Example for README:

⚠️ **SECURITY WARNING**: The private keys in summary.json are public test keys.
NEVER fund these addresses on real networks (mainnet or public testnets).

---

🟡 Suggested Improvements

2. Docker Healthcheck Robustness (test/e2e/envs/anvil-2chains/docker-compose.yml:94-102)

Severity: 🟡 Medium
Lines: docker-compose.yml:94-102

The Agglayer healthcheck uses bash TCP probe:

test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/4443"]

Potential Issues:

• Assumes bash is available in the agglayer image
• No verification that the gRPC service is actually responding correctly
• Could pass while gRPC is still initializing

Suggestion:

• Consider using grpc-health-probe or grpcurl for a proper gRPC health check
• Add a comment explaining why TCP-only is sufficient if that's intentional
• Document the race condition this fixes (from the extensive comment in the file)

---

3. Test Isolation Concerns (.github/workflows/test-go-e2e.yml)

Severity: 🟡 Medium
Lines: 72-98

Great job isolating stateful tests! However:

Current approach:

• Multiple test groups share the same anvil-2chains environment but run in separate CI jobs
• Each job pulls and loads the same Docker images independently

Potential optimization:

• Consider if tests within the same group could interfere with each other
• Document why autoclaim and bridge can't run in the same job (restart/state mutation mentioned in comments)

Minor nit: The comment on line 73-75 is excellent - this kind of documentation is valuable!

---

4. Error Handling in Bridge Utilities (test/e2e/bridge_utils.go)

Severity: 🟡 Low
Lines: 276-330

The AutoClaim race detection logic is sophisticated but complex:

if !strings.Contains(err.Error(), alreadyClaimedErrorSelector) {
    return fmt.Errorf("failed to send claim transaction: %w", err)
}
// Then polls IsClaimed for 30 seconds...

Observations:

• Good: Handles the race condition where AutoClaim claims before the test
• Good: Detailed logging helps debug failures
• Concern: String matching on error messages is fragile
• Concern: Hardcoded 30-iteration poll loop could be extracted to a constant

Suggestions:

• Consider checking IsClaimed BEFORE attempting claim, not just on error
• Extract magic numbers (30 iterations, 1-second sleep) to named constants
• Add a helper comment explaining the AutoClaim race scenario

---

5. Port Hardcoding in Test Files (test/e2e/autoclaim_test.go:33-57)

Severity: 🟡 Low
Lines: autoclaim_test.go:33-57

Multiple hardcoded URLs and constants:

const (
    autoClaimAPIBaseURL = "http://127.0.0.1:14579"
    bridgeServiceBaseURL = "http://127.0.0.1:14577"
    // ... many more
)

Issues:

• Tightly couples tests to specific port numbers
• Difficult to run tests against different environments
• Port collisions aren't detected until runtime

Suggestions:

• Read these from summary.json or environment variables where possible
• At minimum, add a comment explaining why these are hardcoded for anvil-2chains

---

🟢 Nice Work

6. Excellent Documentation (test/e2e/envs/anvil-2chains/README.md)

The README is outstanding:

• ✅ Full provenance trail with commit hashes and run IDs
• ✅ Regeneration procedure
• ✅ Image digest verification process
• ✅ Clear explanation of deviations from design
• ✅ Port mapping documentation

This is a model for infrastructure documentation!

---

7. Thoughtful Test Migration (bridge_utils.go, testmain_test.go)

test/e2e/bridge_utils.go:

• ✅ Added waitMinedL1WithDiagnostics with excellent logging for debugging stuck transactions
• ✅ Generalized bridge helpers to work across different environments
• ✅ Proper context handling and timeout management

test/e2e/testmain_test.go:

• ✅ Added post-test bridge health check with smart skip logic (E2E_SKIP_POSTTEST_BRIDGE_CHECK)
• ✅ Parallel L1→L2 and L2→L1 flows in health check
• ✅ Clear comments explaining the mint/approve dance for ERC20 tokens

---

8. CI/CD Improvements (.github/workflows/test-go-e2e.yml)

• ✅ Efficient artifact sharing (Docker images pulled once, shared via artifacts)
• ✅ Clear matrix strategy with descriptive group names
• ✅ Proper timeout settings (60min for test jobs, 45min for force-ger)
• ✅ Conditional cast installation only for tests that need it

---

🟢 Minor Observations

9. Keystore Files

The .keystore files appear to be properly encrypted (Scrypt KDF, AES-128-CTR). Good security practice for local test environments.

10. Config Consistency

Both config/001/aggkit-config.toml and config/002/aggkit-config.toml use identical structure, which is good for maintainability. The differences are appropriately limited to URLs and network-specific values.

---

Summary

This is a well-executed migration with thorough documentation and thoughtful test design. The main concern is the presence of private keys in summary.json, which should be clearly documented as test-only keys. The other suggestions are optimizations and improvements rather than blockers.

Recommendation: ✅ Approve with minor changes

Required before merge:

1. Add security warning about test keys to README and/or summary.json

Nice to have:
2. Consider the healthcheck improvement
3. Extract magic numbers to constants in bridge_utils.go

---

Priority Summary

• 🔴 1 Required: Document test key security warning
• 🟡 5 Suggested: Healthcheck robustness, test isolation docs, error handling refinement, port constant extraction, diagnostic improvements
• 🟢 5 Commendations: Documentation quality, test migration, CI efficiency, config consistency, security practices

---

| Branch: feat/anvil-2chains-e2e-env