feat(activevln): serve recurrent Go2 navigation requests

docs/en/installation.md

@@ -124,3 +124,6 @@ python benchmarks/benchmark.py --preset small --sweep   # hf vs embodiinfer-eage
 
 The benchmark compares batch-one eager, batched eager, and batched CUDA-graph
 execution. It selects CUDA when available and otherwise runs on CPU.
+
+The ActiveVLN profile includes Accelerate for memory-efficient checkpoint loading.
+Its eager HTTP adapter requires `--no-cuda-graph --max-batch 1`; see [Serving](serving.md).

docs/en/serving.md

@@ -194,3 +194,21 @@ and refit API for model construction and weight updates. The integrations cover
 π0.5, GR00T, and OpenVLA-OFT on LIBERO, and LingBot-VLA on RoboTwin.
 
 See [refit integration patterns](api.md#zero-copy-refit) for the weight-update lifecycle.
+
+## ActiveVLN eager serving
+
+With the ActiveVLN dependency group installed and a local checkpoint:
+
+```bash
+embodiinfer-http-serve --policy activevln --checkpoint /models/activevln-bf16 \
+  --device cuda:0 --dtype auto --no-cuda-graph --max-batch 1 \
+  --host 127.0.0.1 --port 8000
+```
+
+Send one JPEG/PNG named `observation.images.rgb`, an instruction and empty state.
+Keep one session per episode and increment step IDs from zero; reset before
+changing the instruction. Responses contain `discrete_chunk` rows and
+`parsed_valid`; consumers must reject invalid parsed output. This eager path
+uses the policy's original decoding defaults. Optional `policy_kwargs` in an
+adapter JSON can select decoding parameters explicitly. It does not enable the
+benchmark graph/Triton profile or execute robot actions.

docs/proposals/0004-vln-activevln-adapter.md

@@ -757,3 +757,57 @@ remains faster in all five completed paired conditions. Neither engine achieves
 40 ms E2E; near-40 ms R2R B=4 numbers describe amortized forward only. No native
 vLLM closed-loop SR/SPL claim follows. The canonical benchmark README retains
 the exact conditions, comparative tables, validation limits and evidence paths.
+
+## 21. Opt-in asynchronous execution prefix
+
+For onboard R2R deployment, EmbodiRun may observe during the preceding chunk
+and overlap the next inference with its immutable remaining actions. The B=1
+HTTP adapter advertises `execution_prefix: activevln.execution_prefix.v1` only
+when its loaded processor supports the extension. Request metadata carries
+`activevln_execution = {schema, previous_step_id, remaining_actions}`; rows use
+R2R action IDs and remaining cm/degrees, including fractional progress. At most
+three forward/left/right rows are allowed; an empty tail means already finished.
+The previous step must be exactly `step_id - 1` with committed session memory.
+Unsupported profiles, malformed tails and missing history fail before execution.
+
+The owning processor inserts continuation context in the user turn, shared by
+CPU and GPU image preprocessing. It asks for actions AFTER the committed tail.
+The response echoes `execution_prefix_sha256`, SHA-256 of canonical JSON
+(sorted keys, compact separators, float magnitudes), so clients reject silently
+ignored context. No generic engine scheduling or flow-RTC capability is added.
+EmbodiRun owns observation timing, partial-action estimates, heartbeat renewal,
+STOP/cancellation and draining outstanding requests before session close.
+
+Without metadata, serialized prompts remain byte-identical to the synchronous
+reference; model math, dtype, attention and sampling are unchanged. With metadata,
+this is experimental AR prompt conditioning, not a numerical-parity optimization
+or hard flow-matching prefix guidance. Existing synchronous benchmark accuracy
+must not be attributed to this mode. CPU tests validate prompts, protocol failures,
+context forwarding and acknowledgement; Go2/NX closed-loop accuracy, timing and
+physical calibration remain unverified. No GPU/checkpoint validation is claimed.
+
+## 22. Eager HTTP serving for constrained CUDA environments
+
+The standard serving CLI now obtains an ActiveVLN adapter from the policy.
+It delegates recurrent checkout, commit and rollback to EngineCore, preserving
+explicit HTTP SessionKeys. The adapter validates one RGB image, consecutive
+steps, an unchanged instruction and execution-prefix references before model
+execution. It returns the existing R2R discrete-chunk contract, including parser
+validity and an acknowledgement of any consumed continuation prefix. Reset
+clears engine state and adapter metadata. It owns no robot execution.
+
+This path uses eager B=1 execution with `--no-cuda-graph`; it does not depend on
+Triton, frozen benchmark datasets or benchmark startup capture. Model math,
+sampling, image processing and token/context budgets remain policy settings.
+The HTTP factory's load_device option uses Accelerate-assisted CPU safetensors
+mappings followed by parameter transfers, preserving checkpoint dtype and
+avoiding Transformers' model-sized CUDA allocator warmup. BF16 snapshots must
+be prepared explicitly; this adapter does not silently quantize weights.
+
+CPU tests cover RGB/action/session mapping, invalid images, instruction changes,
+failed calls, resets and execution-prefix acknowledgements. They do not certify
+GPU compatibility, constrained-memory loading, real-camera inference latency or
+physical navigation. A separate JetPack 5.1.1 / CUDA 11.4 target check has now
+completed one stationary camera-to-model HTTP turn using BF16 weights and
+prebuilt PyTorch 2.2.0 / torchvision 0.17.2. This does not certify sustained
+latency, long histories or closed-loop navigation.

docs/zh/installation.md

@@ -91,3 +91,6 @@ python benchmarks/benchmark.py --preset small --sweep   # hf vs embodiinfer-eage
 
 该测试比较单请求 eager、批处理 eager 和批处理 CUDA graph 的执行性能。
 有 CUDA 时使用 CUDA，否则在 CPU 上运行。
+
+ActiveVLN 依赖组包含 Accelerate，用于降低加载权重时的内存占用。
+其 eager HTTP 适配器使用 `--no-cuda-graph --max-batch 1`，见[服务部署](serving.md)。

docs/zh/serving.md

@@ -170,3 +170,20 @@ HTTP 与 WirelessComm 遵循相同的会话和推理步骤规则。
 这些集成覆盖 LIBERO 上的 π0.5、GR00T 和 OpenVLA-OFT，以及 RoboTwin 上的 LingBot-VLA。
 
 权重更新流程见 [refit 接口用法](api.md#zero-copy-refit)。
+
+## ActiveVLN eager 服务
+
+安装 ActiveVLN 依赖组并准备本地权重后：
+
+```bash
+embodiinfer-http-serve --policy activevln --checkpoint /models/activevln-bf16 \
+  --device cuda:0 --dtype auto --no-cuda-graph --max-batch 1 \
+  --host 127.0.0.1 --port 8000
+```
+
+提交一张名为 `observation.images.rgb` 的 JPEG/PNG、任务指令和空 state。
+每个 episode 保持同一 session，step ID 从零递增，修改指令前先 reset。
+返回 `discrete_chunk` 动作行和 `parsed_valid`；客户端必须拒绝解析失败的输出。
+此 eager 路径保留 policy 原始解码默认值，可通过适配器 JSON 中的
+`policy_kwargs` 显式选择解码参数。该服务不启用 benchmark 的 graph/Triton
+配置，也不执行机器人动作。

embodiinfer/policies/activevln/execution_activevln.py

@@ -0,0 +1,91 @@
+"""Opt-in continuation context for observations taken during committed actions.
+
+This is prompt conditioning for the autoregressive policy, not flow RTC guidance.
+The deployment runtime owns execution, timing and cancellation.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import math
+from collections.abc import Mapping
+from dataclasses import dataclass
+
+EXECUTION_PREFIX_SCHEMA = "activevln.execution_prefix.v1"
+
+
+@dataclass(frozen=True)
+class ExecutionPrefix:
+    """Remaining R2R cm/degree actions from the immediately preceding model turn."""
+
+    previous_step_id: int
+    remaining_actions: tuple[tuple[int, float], ...]
+
+    @classmethod
+    def parse(cls, value: object, *, step_id: int | None = None) -> ExecutionPrefix:
+        """Reject unsupported fields, invalid magnitudes and out-of-order prefixes."""
+        if not isinstance(value, Mapping) or set(value) != {
+            "schema",
+            "previous_step_id",
+            "remaining_actions",
+        }:
+            raise ValueError("invalid ActiveVLN execution prefix fields")
+        if value["schema"] != EXECUTION_PREFIX_SCHEMA:
+            raise ValueError("unsupported ActiveVLN execution prefix schema")
+        previous = value["previous_step_id"]
+        if type(previous) is not int or previous < 0:
+            raise ValueError("previous_step_id must be non-negative")
+        if step_id is not None and previous != step_id - 1:
+            raise ValueError("execution prefix must reference the previous step")
+        rows = value["remaining_actions"]
+        if not isinstance(rows, (list, tuple)) or len(rows) > 3:
+            raise ValueError("execution prefix must contain at most three actions")
+        actions = []
+        for row in rows:
+            if not isinstance(row, (list, tuple)) or len(row) != 2:
+                raise ValueError("execution prefix actions must be [id, magnitude]")
+            kind, magnitude = row
+            if type(kind) is not int or kind not in (1, 2, 3):
+                raise ValueError("execution prefix allows only forward/left/right")
+            if (
+                isinstance(magnitude, bool)
+                or not isinstance(magnitude, (int, float))
+                or not math.isfinite(magnitude)
+                or not 0 < magnitude <= (75 if kind == 1 else 45)
+            ):
+                raise ValueError("execution prefix magnitude is outside R2R bounds")
+            actions.append((kind, float(magnitude)))
+        return cls(previous, tuple(actions))
+
+    def to_wire(self) -> dict[str, object]:
+        """Return the canonical JSON form used for the server acknowledgement."""
+        return {
+            "schema": EXECUTION_PREFIX_SCHEMA,
+            "previous_step_id": self.previous_step_id,
+            "remaining_actions": [[kind, value] for kind, value in self.remaining_actions],
+        }
+
+    @property
+    def digest(self) -> str:
+        """Bind the response to the exact continuation context it consumed."""
+        wire = json.dumps(self.to_wire(), sort_keys=True, separators=(",", ":"), allow_nan=False)
+        return hashlib.sha256(wire.encode()).hexdigest()
+
+    def prompt_context(self) -> str:
+        """Describe the immutable tail and request only actions after its completion."""
+        names = {1: "move forward", 2: "turn left", 3: "turn right"}
+        tail = (
+            ", ".join(
+                f"{names[kind]} {value:.3f} {'cm' if kind == 1 else 'degrees'}"
+                for kind, value in self.remaining_actions
+            )
+            or "none (the previous actions have finished)"
+        )
+        return (
+            "\nExecution context: this image was captured during the previous action chunk, "
+            "not necessarily after it. At image capture, the following remaining actions "
+            f"were already committed: {tail}. They will finish before your next actions start. "
+            "Plan only the continuation AFTER those actions; do not repeat the committed tail. "
+            "A stop response ends navigation after that tail.\n"
+        )

embodiinfer/policies/activevln/modeling_activevln.py

@@ -2,11 +2,11 @@
 
 from __future__ import annotations
 
-from collections.abc import Callable, Iterable, Iterator, Sequence
+from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
 from contextlib import contextmanager, nullcontext
 from dataclasses import dataclass, replace
 from pathlib import Path
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any
 
 import torch
 
@@ -27,8 +27,10 @@
 )
 
 if TYPE_CHECKING:
+    from ...engine.core import EngineCore
     from .batching_activevln import ActiveVLNBatchedRuntime
     from .cuda_graph import ActiveVLNGraphRuntime
+    from .serving import ActiveVLNServingAdapter
 
 
 ACTIVEVLN_REPO = "https://github.com/arvillion/ActiveVLN"
@@ -796,6 +798,18 @@ def decoder(self) -> _ActiveVLNDecoder:
     def collate(self, observations: list[Observation], request_ids: list[str]) -> ActiveVLNBatch:
         return self._processor.collate(observations, request_ids)
 
+    def build_serving_adapter(
+        self,
+        *,
+        core: EngineCore,
+        checkpoint: str | None = None,
+        config: Mapping[str, Any] | None = None,
+    ) -> ActiveVLNServingAdapter:
+        """Expose eager recurrent inference through the shared serving CLI."""
+        from .serving import ActiveVLNServingAdapter
+
+        return ActiveVLNServingAdapter(core=core, checkpoint=checkpoint, config=config)
+
     def pad(self, batch: ActiveVLNBatch, target_batch_size: int) -> ActiveVLNBatch:
         if target_batch_size != 1:
             raise UnsupportedRecurrentModeError("ActiveVLN does not support padded batching")
@@ -1103,6 +1117,7 @@ def _build_activevln(
     repetition_penalty: float = 1.05,
     do_sample: bool = True,
     action_space: str = "r2r",
+    load_device: str | None = None,
     **overrides,
 ) -> VLAPolicy:
     if checkpoint is None:
@@ -1121,14 +1136,23 @@ def _build_activevln(
 
     from transformers import Qwen2_5_VLForConditionalGeneration
 
+    loading = {}
+    if load_device is not None:
+        # CPU safetensors mappings remain reclaimable while Module.to transfers
+        # each parameter. Avoid Transformers' model-sized CUDA allocator warmup
+        # on devices where CPU and GPU share the same physical memory.
+        loading.update(low_cpu_mem_usage=True, device_map={"": "cpu"})
     qwen = Qwen2_5_VLForConditionalGeneration.from_pretrained(
         checkpoint,
         revision=revision,
         trust_remote_code=False,
         local_files_only=not allow_download,
         torch_dtype="auto",
+        **loading,
     )
     _validate_qwen(qwen)
+    if load_device is not None:
+        qwen.to(torch.device(load_device))
     processor = ActiveVLNProcessor(
         checkpoint, revision, allow_download=allow_download, action_space=action_space
     )

embodiinfer/policies/activevln/processor_activevln.py

@@ -10,6 +10,7 @@
 import torch
 
 from ...types import Observation
+from .execution_activevln import EXECUTION_PREFIX_SCHEMA, ExecutionPrefix
 from .prompt_activevln import SYSTEM_PROMPTS, render_turn_text
 
 
@@ -49,6 +50,16 @@ def to(self, device: torch.device | str, dtype: torch.dtype) -> ProcessedTurn:
 
 
 class ActiveVLNProcessor:
+    @staticmethod
+    def parse_execution_prefix(value: object, *, step_id: int | None = None) -> ExecutionPrefix:
+        """Validate a deployment continuation before running model execution."""
+        return ExecutionPrefix.parse(value, step_id=step_id)
+
+    @property
+    def execution_prefix_schema(self) -> str | None:
+        """Advertise continuation context only for the implemented R2R profile."""
+        return EXECUTION_PREFIX_SCHEMA if self.action_space == "r2r" else None
+
     def __init__(
         self,
         checkpoint: str,
@@ -115,9 +126,13 @@ def _pil_image(cls, observation: Observation):
     def turn_text(self, observation: Observation, *, initial: bool) -> str:
         if not observation.instruction:
             raise ValueError("ActiveVLN requires Observation.instruction")
-        return render_turn_text(
-            self._processor, observation.instruction, initial=initial, action_space=self.action_space
-        )
+        instruction = observation.instruction
+        context = observation.metadata.get("activevln_execution")
+        if context is not None:
+            if initial or self.action_space != "r2r":
+                raise ValueError("execution prefix requires a subsequent R2R turn")
+            instruction += self.parse_execution_prefix(context).prompt_context()
+        return render_turn_text(self._processor, instruction, initial=initial, action_space=self.action_space)
 
     def process_turn(self, observation: Observation, *, initial: bool) -> ProcessedTurn:
         text = self.turn_text(observation, initial=initial)