Problem
The container workflow can fail before building Palace when Launchpad has a transient problem serving the Apptainer PPA signing key.
In PR #890, both container matrix legs failed in the Install Podman step of run 32844332239:
The failing command was:
sudo add-apt-repository -y ppa:apptainer/ppa
Launchpad returned HTTP 500 while add-apt-repository requested the PPA signing key:
lazr.restfulclient.errors.ServerError: HTTP Error 500: Internal Server Error
b'GPGKeyTemporarilyNotFoundError'
The two architecture jobs received separate Launchpad request/OOPS IDs and failed at essentially the same time. No Palace source, Dockerfile, OCI image, or SIF build was reached. A manual rerun was required.
GPGKeyTemporarilyNotFoundError is specifically a temporary Launchpad/key-retrieval failure, so failing the complete job after one request makes container CI unnecessarily flaky.
Location
.github/actions/build-container/action.yml, step Install Podman.
Proposed direction
Add bounded retry with backoff around the idempotent external package-repository setup, at minimum:
sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y podman apptainer
The implementation should:
- retry transient network/HTTP failures within the same job;
- log the attempt number and delay;
- preserve the final command output and exit status after the retry budget is exhausted;
- remain bounded so permanent configuration/package errors still fail promptly;
- avoid retrying Palace build or test failures.
A small shared shell retry helper in the runner/container setup actions may be preferable to duplicating loops. As an alternative, investigate configuring the official PPA through a pinned keyring/source entry without depending on Launchpad's API during every job, while retaining support for both amd64 and arm64.
Acceptance criteria
- A transient Launchpad 5xx/signing-key retrieval error is retried automatically.
- Container CI no longer requires a manual workflow rerun for this failure mode.
- Both x64 and arm64 use the hardened setup path.
- Permanent setup failures still terminate after a bounded number of attempts with useful logs.
Problem
The container workflow can fail before building Palace when Launchpad has a transient problem serving the Apptainer PPA signing key.
In PR #890, both container matrix legs failed in the
Install Podmanstep of run 32844332239:The failing command was:
Launchpad returned HTTP 500 while
add-apt-repositoryrequested the PPA signing key:The two architecture jobs received separate Launchpad request/OOPS IDs and failed at essentially the same time. No Palace source, Dockerfile, OCI image, or SIF build was reached. A manual rerun was required.
GPGKeyTemporarilyNotFoundErroris specifically a temporary Launchpad/key-retrieval failure, so failing the complete job after one request makes container CI unnecessarily flaky.Location
.github/actions/build-container/action.yml, stepInstall Podman.Proposed direction
Add bounded retry with backoff around the idempotent external package-repository setup, at minimum:
The implementation should:
A small shared shell retry helper in the runner/container setup actions may be preferable to duplicating loops. As an alternative, investigate configuring the official PPA through a pinned keyring/source entry without depending on Launchpad's API during every job, while retaining support for both amd64 and arm64.
Acceptance criteria