Support using PodmanProvisioner against a remote podman host

[comps]

Currently, PodmanProvisioner and SystemdPodmanProvisioner run podman and crun directly.

However there are valid use cases for wanting to run these on a remote system - Podman has some HTTP REST (?) remote API, but that wouldn't work for pipe semantics of crun.

The simplest way is to just install podman (and its ecosystem) on the remote end via external scripting, and give the user a way to abstract away command execution, e.g.

def run_podman_command(*args, **subprocess_kwargs):
    subprocess.run(*args, **subprocess_kwargs)

This lets the user subclass it as MyRemotePodmanExecutor, overriding subprocess.run with ie. ManagedSSHConnection-based .run() commands, effectively executing all podman commands via an arbitrary Connection's .cmd().

[comps]

Well, this isn't as straightforward.

We could add a host= argument to PodmanProvisioner, kind of like SharedVirtProvisioner has, and it could pass that Connection to a PodmanRemote / PodmanConnection, but the real problem is rsync.

Because the expectation would be that .rsync() on the newly-created Remote (Connection) would be probably tunnelled through the user-provided Connection, leaving it unclear how and where to actually run the rsync command.

• We can't run it locally, since a Connection has no proxy-like method similar to OpenSSH's ProxyCommand, so there's nothing to point a local rsync to as a remote end.
• We can't run it remotely via .cmd() because then that breaks the expectations of .rsync() local/remote sides, because "local" would be the user-provided Connection's remote end.

We would have to do two calls, e.g. when uploading tests from local dirs to the remote container:

1. rsync via the user-provided Connection to some temporary location on the remote host, likely using SSH
2. rsync from the remote host to the podman container running on the remote host, likely using podman exec / crun

and that seems like it could break some expectations of not touching the remote system's disk.

So the best approach, for now, is probably to run the ATEX-using python script directly on the remote system.

---

FTR: It's a solvable problem, but the solution would likely be a dedicated RemotePodmanConnection that uses either /var/tmp or ~/.cache or something and the behavior is fully documented, so users aren't surprised. It would also be a bit more invasive change to the PodmanProvisioner + other container functions.

edit: Actually, there probably is a way.

• remote side is just user_conn.cmd(... podman exec rsync ..., func=Popen) - we simply run the executed rsync via .cmd() of the user-provided Connection
• local side is much more clever

We can extract the real command used for .cmd() of the user-provided Connection with .cmd(None, func=tuple). This effectively calls tuple([real_cmd, arg1, arg2]) giving us a "reproducer" for which LOCAL commands would cause the desired remote .cmd(). We can then just give this to rsync's -e and job done.

[comps]

Putting it here since it's relevant;

It's possible to fully isolate multiple podman storages within a single unprivileged user too,

$BASE/storage.conf:

[storage]
driver = "overlay"
graphroot = "$BASE/storage"
runroot = "$BASE/run"

$BASE/containers.conf:

[engine]
lock_type = "file"
tmp_dir = "$BASE/tmp"

And

CONTAINERS_STORAGE_CONF="$BASE/storage.conf" CONTAINERS_CONF="$BASE/containers.conf" podman <command>

or

podman --root "$BASE/storage" --runroot "$BASE/run" --tmpdir "$BASE/tmp" <command>

Probably have some __init__(..., isolated=False) flag that would inspect pre-existing podman graphroot/runroot (in case the OS/user changed them globally) and create random subdirs inside them for its own graphroot/runroot/tmp_dir.

---

The storage.conf ones (no --module equivalent):

  - --root $BASE/storage
  - --runroot $BASE/run
  - --storage-driver <inherited>
  - --storage-opt additionalimagestore=<default graphroot> (repeat --storage-opt for any inherited driver opts like overlay.mount_program)

--tmpdir is optional — either pass it, or set tmp_dir in the --module file (it's an [engine] key). Everything else (lock_type, and tmp_dir if you choose) goes in the module.

One podman info call covers it all — no inspect needed:

podman info --format '{{.Store.GraphRoot}} {{.Store.RunRoot}} {{.Store.GraphDriverName}} {{json .Store.GraphOptions}}'

• .Store.GraphRoot → the additionalimagestore= value (and the dir to nest $BASE/storage under)

• .Store.RunRoot → dir to nest $BASE/run (+ tmp) under

• .Store.GraphDriverName → --storage-driver

• .Store.GraphOptions → re-pass each as --storage-opt

  Run it once (without any isolation flags) at start().

[comps]

This issue should be now workaroundable by:

1. .cmd(None) (or so) of the outer Connection, to get the exact command - not universal, but works for SSH Connections
2. Overriding podman_command and crun_command, from 646bc7a

This is still hacky (requires modifying classes in-place because PodmanProvisioner doesn't accept a Remote factory via an argument), but doable.