Skip to content

Fail fast on an unmapped port, and report the mapping back #26

Description

@andrewmusselman

A vLLM server on an unmapped Vast port starts cleanly, passes its local health
check, and is unreachable from outside. Nothing in the logs says so — the
failure only appears when something tries to route to it.

Background

Vast maps container ports to public ones at instance creation. A port not
declared on the template cannot be added afterwards; the instance has to be
recreated. The external port is assigned from the instance's range and is not
the same number as the internal one:

$ env | grep VAST_TCP_PORT
VAST_TCP_PORT_10100=12711
$ echo "$PUBLIC_IPADDR"
80.188.223.202

So the server is reachable at 80.188.223.202:12711, and 10100 appears
nowhere outside the box.

Three changes

1. Refuse to launch on an unmapped port.

install_set.py should read VAST_TCP_PORT_<port> before writing the
Supervisor unit. If it is absent, fail with a message naming the port rather
than starting a server nobody can reach.

This is the highest-value part. The current failure is silent and looks like
success.

2. Force --host 0.0.0.0 in the generated unit.

The Vast template's own vLLM launches with --host 127.0.0.1:

vllm serve Qwen/Qwen3.5-9B ... --host 127.0.0.1 --port 18000

Bound to loopback, it is unreachable from outside regardless of how the port is
mapped — and it looks identical to a mapping problem from the outside. Worth
setting explicitly rather than inheriting whatever the template does.

3. Report the public address back to llmao.

The box knows PUBLIC_IPADDR and its VAST_TCP_PORT_* map at exactly the
moment it fetches GET /vllm/config. Having install_set.py POST them back
means llmao never has to be told:

POST /vllm/observed
{ "public_ip": "80.188.223.202",
  "port_map": { "10100": 12711 },
  "name": "gemma4-26b" }

llmao can then build api_base for the route itself. This is an extension of
the observed-state reporting already planned, not a new mechanism.

Also worth doing

Declare ports on the template, not per instance. Every box then gets them
without anyone remembering, and a missing port becomes a template fix rather
than a rebuild.

Note for anything pinning the address

The external port differs per instance, so a CDN backend or hiera value pinned
to IP:port needs updating on every rebuild. Worth asking whether the CDN can
point at a hostname we control instead, so a rebuild becomes a DNS update
rather than a ticket.

Reproducing

# on the box -- is vLLM listening?
ss -tlnp | grep 10100
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:10100/v1/models
 
# from outside -- is the mapping routing?
curl -sS -m 10 -o /dev/null -w '%{http_code}\n' http://$PUBLIC_IPADDR:<external>/v1/models

Local works and external times out → the port was not declared at creation.
Both fail → vLLM is not up, or it is bound to loopback.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions