Skip to content

Retry transient Apptainer PPA failures in container CI #899

Description

@hughcars

Problem

The container workflow can fail before building Palace when Launchpad has a transient problem serving the Apptainer PPA signing key.

In PR #890, both container matrix legs failed in the Install Podman step of run 32844332239:

The failing command was:

sudo add-apt-repository -y ppa:apptainer/ppa

Launchpad returned HTTP 500 while add-apt-repository requested the PPA signing key:

lazr.restfulclient.errors.ServerError: HTTP Error 500: Internal Server Error
b'GPGKeyTemporarilyNotFoundError'

The two architecture jobs received separate Launchpad request/OOPS IDs and failed at essentially the same time. No Palace source, Dockerfile, OCI image, or SIF build was reached. A manual rerun was required.

GPGKeyTemporarilyNotFoundError is specifically a temporary Launchpad/key-retrieval failure, so failing the complete job after one request makes container CI unnecessarily flaky.

Location

.github/actions/build-container/action.yml, step Install Podman.

Proposed direction

Add bounded retry with backoff around the idempotent external package-repository setup, at minimum:

sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y podman apptainer

The implementation should:

  • retry transient network/HTTP failures within the same job;
  • log the attempt number and delay;
  • preserve the final command output and exit status after the retry budget is exhausted;
  • remain bounded so permanent configuration/package errors still fail promptly;
  • avoid retrying Palace build or test failures.

A small shared shell retry helper in the runner/container setup actions may be preferable to duplicating loops. As an alternative, investigate configuring the official PPA through a pinned keyring/source entry without depending on Launchpad's API during every job, while retaining support for both amd64 and arm64.

Acceptance criteria

  • A transient Launchpad 5xx/signing-key retrieval error is retried automatically.
  • Container CI no longer requires a manual workflow rerun for this failure mode.
  • Both x64 and arm64 use the hardened setup path.
  • Permanent setup failures still terminate after a bounded number of attempts with useful logs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingciRelated to continuous integration (CI) and/or GitHub Actions

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions