diff --git a/.agents/skills/build-from-issue/SKILL.md b/.agents/skills/build-from-issue/SKILL.md index bdcf02e8a2..d2c7ec07ea 100644 --- a/.agents/skills/build-from-issue/SKILL.md +++ b/.agents/skills/build-from-issue/SKILL.md @@ -11,7 +11,7 @@ Use a specific GitHub issue to plan and implement a scoped change. Direct user i ## Inspect the issue -1. Run `gh issue view --json number,title,body,state,labels,comments,assignees` and inspect the repository's current `state:*` labels and descriptions with `gh label list`. Infer whether triage, validation, and human acceptance have happened. Do not hard-code label names or change disposition as part of building. +1. Run `gh issue view --json number,title,body,state,labels,comments,assignees` and follow Label Discovery in `CONTRIBUTING.md` to retrieve every page of current `state:*` label definitions and resolve unclear meanings. Infer whether triage, validation, and human acceptance have happened. Do not hard-code label names or change disposition as part of building. 2. Read the issue, comments, linked PRs, and current code. Check for an active owner or implementation. If the issue concerns a vulnerability, use `review-security-issue` and `fix-security-issue` instead. 3. Confirm that the User Story attests to the human operator's first-hand OpenShell use and gives a specific use case. If the issue lacks this, ask the operator before proceeding with planning or implementation. For a bug, require reproduction using only an OpenShell deployment; do not install third-party tools solely to demonstrate the problem. 4. If the user's direct request starts before the normal issue disposition, briefly report the discrepancy and continue with the authorized phase. Stop only when information needed to do the work is actually unavailable or a conflicting owner needs resolution. @@ -24,11 +24,11 @@ Use a single issue comment beginning with `> **πŸ—οΈ build-plan**` when a plan ## Implement -1. Check the current branch and working tree. Preserve unrelated work. Create a branch or worktree as needed, with the branch named `/-/`. Use a Conventional Commits type for ``. +1. Check the current branch and working tree. Preserve unrelated work. Create an issue-specific branch or worktree as needed, following Branch Names in `CONTRIBUTING.md`. 2. Implement the smallest coherent change that fulfills the acceptance criteria. Update relevant skills when behavior or commands change. Keep published documentation minimal: explain exactly what users need, avoid duplication across pages, and omit internal details with no user impact. 3. Add meaningful tests for changed behavior and follow the verification guidance in `CONTRIBUTING.md`. Select format, lint, compile or type checks, and tests for affected components and their dependencies. Run the relevant E2E lane for infrastructure, sandbox, or policy changes. Guidance and template edits need applicable Markdown, YAML, link, and consistency checks. Do not require full Rust, SDK, or repository CI solely because a commit or PR is being created; broaden checks only for a concrete remaining risk or failed check. 4. Review the diff, use a signed-off Conventional Commit, and prepare a PR following `create-github-pr`. -Every PR must have its own existing issue and use `Closes #` in its Related Issue section. For work needing multiple PRs, split the scope into a closable issue per PR. A high-level issue may track those issues but should not be closed by an incomplete PR. Report the implementation, verification, and any remaining limitation in the PR description, rather than copying earlier issue diagnostics. +Every PR from this issue-backed workflow must have its own existing issue and use `Closes #` in its Related Issue section. For work needing multiple PRs, split the scope into a closable issue per PR. A high-level issue may track those issues but should not be closed by an incomplete PR. Report the implementation, verification, and any remaining limitation in the PR description, rather than copying earlier issue diagnostics. Do not apply acceptance or roadmap decisions on behalf of a maintainer. Do not introduce `agent:*` workflow labels. diff --git a/.agents/skills/create-github-issue/SKILL.md b/.agents/skills/create-github-issue/SKILL.md index f47ace151b..25a9cbc4c9 100644 --- a/.agents/skills/create-github-issue/SKILL.md +++ b/.agents/skills/create-github-issue/SKILL.md @@ -135,7 +135,7 @@ EOF GitHub built-in issue types (`Bug`, `Feature`, `Task`) should come from the matching issue template when possible, or be set manually afterward. Do not try to emulate them through labels. -Creating an issue does not accept it. Inspect the repository’s current `state:*` labels and follow its triage β†’ validation β†’ human acceptance process. Agents may assess facts, but only humans decide whether to accept work or place it on the roadmap. A direct user request authorizes the requested planning or implementation phase without changing issue disposition. +Creating an issue does not accept it. Follow Label Discovery in `CONTRIBUTING.md` to retrieve every page of current `state:*` label definitions and resolve unclear meanings, then follow its triage β†’ validation β†’ human acceptance process. Agents may assess facts, but only humans decide whether to accept work or place it on the roadmap. A direct user request authorizes the requested planning or implementation phase without changing issue disposition. ## Useful Options @@ -161,4 +161,4 @@ Created issue [#123](https://github.com/OWNER/REPO/issues/123) Use the issue number to: - Reference in signed-off Conventional Commits: `git commit --signoff -m "fix(cli): validate empty requests (fixes #123)"` -- Create a branch following project convention: `/-/`, where `` is a Conventional Commits type. +- Create a branch following Branch Names in `CONTRIBUTING.md`. diff --git a/.agents/skills/create-github-pr/SKILL.md b/.agents/skills/create-github-pr/SKILL.md index b1c73aa5a2..4f06a0818c 100644 --- a/.agents/skills/create-github-pr/SKILL.md +++ b/.agents/skills/create-github-pr/SKILL.md @@ -13,7 +13,7 @@ Create pull requests on GitHub using the `gh` CLI. - The `gh` CLI must be authenticated (`gh auth status`) - You must have commits on a branch that's pushed to the remote -- Every PR must close an existing issue. The branch should follow `/-/`. +- Every PR must close an existing issue, except automated dependency updates as described in `CONTRIBUTING.md`. Follow Branch Names in `CONTRIBUTING.md` for contributor branch names. ## Before Creating a PR @@ -47,10 +47,10 @@ Before creating a PR, verify: git branch --show-current ``` -2. **Branch follows naming convention** - Use `/-/`, where `` is a Conventional Commits type. +2. **Branch follows naming convention** - Follow Branch Names in `CONTRIBUTING.md`, including the exceptions for generated branches and private security work. ```bash - # Example: feat/1234-add-pagination/johntmyers + # Example: feat/1234-add-pagination/octocat git branch --show-current ``` @@ -106,7 +106,7 @@ gh pr create --title "PR title" --body "PR description" ### Link to an Issue -Every PR must close its own issue. Verify that the issue exists, remains open, and covers the PR scope. Use `Closes #` in the body so merge closes it: +Every PR except an automated dependency update must close its own issue. Verify that the issue exists, remains open, and covers the PR scope. Automated dependency updates follow the exception in `CONTRIBUTING.md`. Use `Closes #` in the body so merge closes it: ```bash gh pr create \ diff --git a/.agents/skills/create-spike/SKILL.md b/.agents/skills/create-spike/SKILL.md index d5e0878fc4..7fdc98c7d6 100644 --- a/.agents/skills/create-spike/SKILL.md +++ b/.agents/skills/create-spike/SKILL.md @@ -22,6 +22,6 @@ Ask the human operator to attest that they personally use OpenShell and directly ## Record the result -Create an issue with User Story, Problem Statement, Impact / Why This Matters, Proposed Design when relevant, Acceptance Criteria, Alternatives Considered, and concise Agent Investigation. Include OpenShell-only reproduction and environment details for bugs. Inspect current GitHub `state:*` labels and descriptions before applying the one that matches the evidence; do not hard-code label names. Do not apply an acceptance state or add the issue to the roadmap. +Create an issue with User Story, Problem Statement, Impact / Why This Matters, Proposed Design when relevant, Acceptance Criteria, Alternatives Considered, and concise Agent Investigation. Include OpenShell-only reproduction and environment details for bugs. Follow Label Discovery in `CONTRIBUTING.md` before applying the assessment state that matches the evidence; retrieve every page and resolve unclear meanings rather than hard-coding label names. Do not apply an acceptance state or add the issue to the roadmap. -Report the issue URL, technical findings, uncertainties, and the human disposition needed. For subsequent authorized implementation, use `build-from-issue`. Every eventual PR must close an issue covering its own scope; split multi-PR efforts into separate closable issues and use a high-level issue only for tracking. +Report the issue URL, technical findings, uncertainties, and the human disposition needed. For subsequent authorized implementation, use `build-from-issue`. Every eventual PR from this issue-backed workflow must close an issue covering its own scope; split multi-PR efforts into separate closable issues and use a high-level issue only for tracking. diff --git a/.agents/skills/fix-security-issue/SKILL.md b/.agents/skills/fix-security-issue/SKILL.md index 30e6fa15bd..83a145c216 100644 --- a/.agents/skills/fix-security-issue/SKILL.md +++ b/.agents/skills/fix-security-issue/SKILL.md @@ -9,10 +9,10 @@ metadata: Use this skill after an authorized `review-security-issue` review identifies an actionable concern. Follow `SECURITY.md`; do not disclose vulnerability details in a public issue. A direct user request to fix a specific reviewed issue authorizes implementation. For unattended work, inspect current `state:*` label descriptions, maintainer assignments, and comments to verify that remediation is authorized. Do not infer approval from a state that only records technical validation. -1. Fetch the issue and its comments with `gh issue view --json number,title,body,state,labels,comments`. Inspect current repository labels and confirm this is a security issue. Find the review marked `> **πŸ”’ security-review-agent**` and its remediation plan. If the review is missing or found the issue not actionable, stop and report that result. +1. Fetch the issue and its comments with `gh issue view --json number,title,body,state,labels,comments`. Follow Label Discovery in `CONTRIBUTING.md` and confirm this is a security issue; resolve unclear meanings before interpreting authorization. Find the review marked `> **πŸ”’ security-review-agent**` and its remediation plan. If the review is missing or found the issue not actionable, stop and report that result. 2. Verify the review against current code. Adapt the plan when code has changed, and record material deviations. Check for an existing owner, branch, or PR. -3. Create a branch or worktree using `fix/-/`, preserving unrelated changes. Implement the smallest safe fix and add regression tests for the security boundary. Avoid logging secrets or adding public exploit detail. +3. Create a `fix` branch or worktree following Branch Names in `CONTRIBUTING.md`, preserving unrelated changes and disclosure boundaries. Implement the smallest safe fix and add regression tests for the security boundary. Avoid logging secrets or adding public exploit detail. 4. Follow the verification guidance in `CONTRIBUTING.md`. Run format, lint, compile or type checks, and regression tests for the affected security boundary and dependent components, plus the relevant E2E lane for sandbox or policy changes. Broaden verification when the fix spans components or a concrete risk remains; do not require unaffected Rust or SDK suites solely to create a signed-off commit or PR. -5. Follow `create-github-pr` and use `Closes #` for the reviewed issue. Every PR must close its own issue; split multi-PR remediations into separate issues in the authorized security workflow. Keep the PR description appropriately scoped to its disclosure venue. +5. Follow `create-github-pr` and use `Closes #` for the reviewed issue. Every PR from this issue-backed remediation workflow must close its own issue; split multi-PR remediations into separate issues in the authorized security workflow. Keep the PR description appropriately scoped to its disclosure venue. Begin any fix comments with `> **πŸ”§ security-fix-agent**`. Do not change human disposition or introduce `agent:*` workflow labels. diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index c0d1e68071..450d23094d 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -38,8 +38,8 @@ and preloads the default sandbox image into k3d so the first sandbox create does not wait on a large registry pull. Traefik is disabled at cluster creation time. **Multi-worktree support:** the cluster name is derived from the last component of the -current git branch (e.g. branch `kube-support/local-dev/tmutch` β†’ cluster -`openshell-dev-tmutch`). Each worktree therefore gets its own isolated cluster and its +current git branch (e.g. branch `chore/1234-local-dev/octocat` β†’ cluster +`openshell-dev-octocat`). Each worktree therefore gets its own isolated cluster and its own `kubeconfig` file. Override with `HELM_K3S_CLUSTER_NAME` to force a specific name or share one cluster across worktrees. diff --git a/.agents/skills/review-github-pr/SKILL.md b/.agents/skills/review-github-pr/SKILL.md index de56cb4770..3903e03271 100644 --- a/.agents/skills/review-github-pr/SKILL.md +++ b/.agents/skills/review-github-pr/SKILL.md @@ -173,9 +173,9 @@ User says: "Review PR #456" ### Review by branch name -User says: "Review branch `feature/add-pagination`" +User says: "Review branch `feat/1234-add-pagination/octocat`" -1. Look up PR with `gh pr list --head "feature/add-pagination"` +1. Look up PR with `gh pr list --head "feat/1234-add-pagination/octocat"` 2. If found, fetch PR metadata and diff 3. If not found, diff against main locally 4. Produce summary diff --git a/.agents/skills/review-security-issue/SKILL.md b/.agents/skills/review-security-issue/SKILL.md index 0a014a62b5..cc0a3627dc 100644 --- a/.agents/skills/review-security-issue/SKILL.md +++ b/.agents/skills/review-security-issue/SKILL.md @@ -11,7 +11,7 @@ Review a security concern through its authorized private workflow. Do not file o ## Assess -1. Fetch the issue and comments with `gh issue view --json title,body,state,labels,comments`. Inspect current repository labels rather than assuming exact names. Verify that this is an authorized security issue and that a prior review does not already answer the request. +1. Fetch the issue and comments with `gh issue view --json title,body,state,labels,comments`. Follow Label Discovery in `CONTRIBUTING.md` rather than assuming exact label names; resolve unclear meanings before interpreting authorization. Verify that this is an authorized security issue and that a prior review does not already answer the request. 2. Inspect affected code and verify the claim. Assess impact, exploitability, prerequisites, affected surface, and a concrete attack scenario. Separate evidence from assumptions and give a severity with rationale. 3. If actionable, propose a remediation plan with code areas, safe rollout, and focused tests. If not actionable, explain the evidence and recommended disposition. Do not decide product acceptance or silently close the issue. 4. Post the review only when the request authorizes posting. Begin the comment with `> **πŸ”’ security-review-agent**` so later reviews can detect it. Keep sensitive details in the authorized private venue. diff --git a/.agents/skills/sync-agent-infra/SKILL.md b/.agents/skills/sync-agent-infra/SKILL.md index f16c171bea..8a07921e62 100644 --- a/.agents/skills/sync-agent-infra/SKILL.md +++ b/.agents/skills/sync-agent-infra/SKILL.md @@ -38,11 +38,11 @@ Search both `skills/` and `.agents/skills/` for affected commands, fields, and c 1. Compare `skills/*/SKILL.md` and `.agents/skills/*/SKILL.md` with the inventories in `CONTRIBUTING.md`. Public skills must work outside a checkout and use installed CLI help and published documentation. Contributor skills must set `metadata.internal: true`. Confirm names are unique and local links resolve. 2. Compare `crates/` with the architecture table in `AGENTS.md`. Check the public and contributor skill rows. -3. Read the current GitHub labels and their descriptions, then check issue guidance for the general triage β†’ validation β†’ human acceptance process. Skills should mention the `state:*` namespace without enumerating exact labels, and should not depend on `agent:*` labels. A direct request authorizes only the requested phase; unattended work uses current state descriptions, maintainer assignments, and comments to establish the authorized phase. +3. Follow Label Discovery in `CONTRIBUTING.md` to retrieve every page of current GitHub labels and resolve unclear descriptions, then check issue guidance for the general triage β†’ validation β†’ human acceptance process. Skills should mention the `state:*` namespace without enumerating exact labels, and should not depend on `agent:*` labels. A direct request authorizes only the requested phase; unattended work uses current state descriptions, maintainer assignments, and comments to establish the authorized phase. 4. Check issue templates, `create-github-issue`, `create-spike`, and `triage-issue` for the first-hand OpenShell User Story, OpenShell-only bug reproduction, notional UX examples, and consideration of applicable extension points. -5. Check the PR template, `create-github-pr`, `build-from-issue`, `fix-security-issue`, and `CONTRIBUTING.md`: every PR must close an existing issue covering its scope. Multi-PR work needs an issue per PR; a separate issue may track the overall effort. -6. Check `README.md`, `.github/ISSUE_TEMPLATE/`, `.github/workflows/`, `.agents/agents/`, and related skill cross references for stale workflow statements. Keep user-facing documentation changes minimal and avoid duplicated internal explanations. -7. Check that PR and build skills follow the scoped verification guidance in `CONTRIBUTING.md`; they must not require full Rust, SDK, or repository CI for changes that cannot affect those areas. +5. Check the PR template, `create-github-pr`, `build-from-issue`, `fix-security-issue`, and `CONTRIBUTING.md`: every PR must close an existing issue covering its scope, except automated dependency updates as described in `CONTRIBUTING.md`. Multi-PR work needs an issue per PR; a separate issue may track the overall effort. +6. Check contributor branch instructions and examples against Branch Names in `CONTRIBUTING.md`; AGENTS.md and skills should refer to that convention. Check `README.md`, `.github/ISSUE_TEMPLATE/`, `.github/workflows/`, `.agents/agents/`, and related skill cross references for stale workflow statements. Keep user-facing documentation changes minimal and avoid duplicated internal explanations. +7. Check that PR, build, TUI, and docs contribution guidance follow the scoped verification guidance in `CONTRIBUTING.md`; they must not require full Rust, SDK, or repository CI for changes that cannot affect those areas. 8. Use `npx -y skills add . --list` from a disposable clean copy when skill discovery changes. It should expose public skills only; clean generated files afterward. Fix contradictions, then repeat the affected checks. Report files changed and any remaining drift. Do not treat an old document's label list as a source of truth over current GitHub metadata. diff --git a/.agents/skills/triage-issue/SKILL.md b/.agents/skills/triage-issue/SKILL.md index 1c6a2b94ea..a341a56540 100644 --- a/.agents/skills/triage-issue/SKILL.md +++ b/.agents/skills/triage-issue/SKILL.md @@ -17,11 +17,11 @@ Do not apply an acceptance state, place issues on the roadmap, or make a product ## Assess one issue -1. Run `gh issue view --json number,title,body,state,labels,author,comments`. Inspect the current `state:*` label definitions with `gh label list`. Check for previous triage, new human evidence, an existing owner, and duplicate issues. +1. Run `gh issue view --json number,title,body,state,labels,author,comments`. Follow Label Discovery in `CONTRIBUTING.md` to retrieve every page of current `state:*` label definitions and resolve unclear meanings. Check for previous triage, new human evidence, an existing owner, and duplicate issues. 2. Confirm the User Story attests that the human operator personally uses OpenShell and directly encountered the problem or needs the feature for a specific use case. If this is absent, request that first-hand context; do not invent it. Check the Problem Statement, impact, workaround and its limits, and observable acceptance criteria. 3. For bugs, verify reproduction steps using only OpenShell deployments. Do not install third-party tools solely to demonstrate reproducibility. Check the OpenShell version and relevant deployment details. For features, check the user-facing workflow and alternatives. Consider applicable middleware, interceptors, providers, and other extension points; prefer an applicable extension when it satisfies the use case, and do not dismiss it merely because it runs another service. 4. For proposed configuration, CLI, SDK, or other UX changes, request notional examples of the suggested interaction if missing. Keep the issue framed entirely in terms of OpenShell. 5. Reproduce or investigate enough to distinguish a confirmed bug, feasible feature, missing evidence, duplicate, already-fixed behavior, or expected behavior. State what was observed and what remains uncertain. Do not turn a failed reproduction attempt alone into a dismissal. -6. Post a concise comment beginning with `> **πŸ“‹ triage-agent**`. Explain the evidence, any exact missing information, and the next human decision. Apply the appropriate current `state:*` label for the assessment outcome only after checking repository definitions. Leave acceptance and roadmap decisions to a maintainer. +6. Post a concise comment beginning with `> **πŸ“‹ triage-agent**`. Explain the evidence, any exact missing information, and the next human decision. Replace the previous assessment state with the appropriate current assessment state only after checking repository definitions. Preserve maintainer acceptance, execution state, roadmap decisions, and unrelated labels; do not accumulate assessment states. If substantial technical uncertainty remains, recommend `create-spike` with specific questions to answer. A direct user request can separately authorize planning or implementation through `build-from-issue`; it does not change the issue's recorded disposition. diff --git a/.agents/skills/tui-development/SKILL.md b/.agents/skills/tui-development/SKILL.md index ee946adf22..cae3b3ef5e 100644 --- a/.agents/skills/tui-development/SKILL.md +++ b/.agents/skills/tui-development/SKILL.md @@ -607,13 +607,9 @@ cargo fmt -p openshell-tui cargo clippy -p openshell-tui ``` -### Pre-commit +### Verification -Always run before committing: - -```bash -mise run pre-commit -``` +Follow Choose Verification for the Change in `CONTRIBUTING.md`. Select format, lint, and tests for the affected TUI behavior and its dependencies. Use `mise run pre-commit` when its broader scope is warranted. ### Gateway changes diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 6cf1898bf5..d479fd6aca 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -2,7 +2,7 @@ ## Related Issue - + ## Changes diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 2dd366d9f5..85097eea56 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -305,9 +305,9 @@ jobs: integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }} test-matrix: >- [ - {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"driver-podman"}, - {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"e2e-podman"}, - {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"driver-podman"} + {"environment":"fedora-podman-rootful","installer":"binaries","testsuite":"driver-podman"}, + {"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"e2e-podman"}, + {"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"driver-podman"} ] docker-e2e: diff --git a/.github/workflows/odh-cargo-deny-advisories.yml b/.github/workflows/odh-cargo-deny-advisories.yml index 8a620551fa..fba9f29f32 100644 --- a/.github/workflows/odh-cargo-deny-advisories.yml +++ b/.github/workflows/odh-cargo-deny-advisories.yml @@ -57,3 +57,35 @@ jobs: CARGO="$cargo_bin" \ PATH="$(dirname "$cargo_bin"):$(dirname "$deny_bin"):$PATH" \ "$deny_bin" check advisories + notify-slack: + name: Notify Slack on failure + needs: advisories + if: >- + ${{ always() && + needs.advisories.result == 'failure' && + github.repository == 'red-hat-data-services/openshell' }} + runs-on: ubuntu-latest + permissions: {} + steps: + - name: Post failure notification + env: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} + WORKFLOW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + if [ -z "${SLACK_WEBHOOK_URL:-}" ]; then + echo "::warning::SLACK_WEBHOOK_URL is not set; skipping Slack notification." + exit 0 + fi + + message="$(printf '%s\n%s' \ + ':x: ODH Cargo Deny Advisories' \ + "Run: $WORKFLOW_RUN_URL")" + + jq -nc \ + --arg text "$message" \ + '{text: $text}' | + curl --fail --silent --show-error \ + -X POST \ + -H 'Content-Type: application/json' \ + --data @- \ + "$SLACK_WEBHOOK_URL" diff --git a/.github/workflows/odh-trivy-images.yml b/.github/workflows/odh-trivy-images.yml index 8e28c535a1..9694b8de22 100644 --- a/.github/workflows/odh-trivy-images.yml +++ b/.github/workflows/odh-trivy-images.yml @@ -1,5 +1,5 @@ -# Midstream image CVE scan for the ODH Quay builds. Catch MEDIUM/HIGH/CRITICAL -# image vulnerabilities early, before a build is released on a GA branch. +# Midstream image CVE scan for the ODH Quay builds. Catch Critical RPM and +# Medium/High/Critical Rust crate vulnerabilities before a GA release. # Kept separate from trivy-scan.yml so we do not inherit NVIDIA GHCR login # or Helm config scans. # Rust crate rows appear only after Konflux images are built with @@ -77,58 +77,213 @@ jobs: refs+=("${image}:${tag}") done <<<"$DEFAULT_IMAGES" - severity=MEDIUM,HIGH,CRITICAL + rpm_severity=MEDIUM,HIGH,CRITICAL + crate_severity=MEDIUM,HIGH,CRITICAL { echo "### ODH Trivy image scan" echo - echo "Severity gate: \`$severity\`" + echo "RPM gate: \`CRITICAL\`" + echo "Rust crate gate: \`$crate_severity\`" echo - echo "Each image prints two tables: RHEL RPMs, then Rust crates. An empty crate table (\`language-specific files num=0\`) means the image has no cargo-auditable metadata; check \`Dockerfile.konflux.*\`." + echo "RPM Medium/High findings are reported but do not fail this workflow. They originate in the base image; ProdSec handles remediation on a best-effort basis." + echo + echo "Missing cargo-auditable crate metadata is a coverage warning only. Check \`Dockerfile.konflux.*\`." echo echo "Images:" for ref in "${refs[@]}"; do echo "- \`$ref\`" done + echo + echo "| Image | Scan | Result | Findings | Notes |" + echo "| --- | --- | :---: | --- | --- |" } >> "$GITHUB_STEP_SUMMARY" + reports_dir="$(mktemp -d)" + trap 'rm -rf "$reports_dir"' EXIT + status=0 + + record_result() { + local ref="$1" + local scan="$2" + local result="$3" + local findings="$4" + local notes="$5" + + echo "$result $scan - $ref: $findings. $notes" + printf '| `%s` | %s | %s | %s | %s |\n' \ + "$ref" "$scan" "$result" "$findings" "$notes" \ + >> "$GITHUB_STEP_SUMMARY" + } + for ref in "${refs[@]}"; do echo "::group::RPMs $ref" + rpm_report="$reports_dir/rpm-${ref##*/}.json" if trivy image \ --scanners vuln \ --pkg-types os \ - --severity "$severity" \ - --format table \ - --exit-code 1 \ + --severity "$rpm_severity" \ + --format json \ + --output "$rpm_report" \ "$ref"; then - echo "No $severity RPM vulnerabilities in $ref" + if ! trivy convert \ + --scanners vuln \ + --format table \ + --severity "$rpm_severity" \ + "$rpm_report"; then + echo "::error::Trivy could not render the RPM report for $ref" + record_result \ + "$ref" "RPMs" "❌" "scan error" \ + "Trivy could not render the RPM report." + status=1 + else + critical_count=$( + jq '[.Results[]?.Vulnerabilities[]? | + select(.Severity == "CRITICAL")] | length' \ + "$rpm_report" + ) + medium_high_count=$( + jq '[.Results[]?.Vulnerabilities[]? | + select(.Severity == "MEDIUM" or .Severity == "HIGH")] | length' \ + "$rpm_report" + ) + if [ "$critical_count" -gt 0 ]; then + echo "::error::Trivy found Critical RPM vulnerabilities in $ref" + record_result \ + "$ref" "RPMs" "❌" \ + "$critical_count Critical, $medium_high_count Medium/High" \ + "Critical RPM findings fail the workflow." + status=1 + elif [ "$medium_high_count" -gt 0 ]; then + record_result \ + "$ref" "RPMs" "⚠️" \ + "0 Critical, $medium_high_count Medium/High" \ + "Medium/High are reported only; base-image RPM remediation is ProdSec-owned and best-effort." + else + record_result \ + "$ref" "RPMs" "βœ…" \ + "0 at MEDIUM/HIGH/CRITICAL" \ + "No gated RPM findings." + fi + fi else - echo "::error::Trivy found $severity RPM vulnerabilities in $ref" + echo "::error::Trivy could not scan RPMs in $ref" + record_result \ + "$ref" "RPMs" "❌" "scan error" \ + "Trivy could not complete the RPM scan." status=1 fi - echo "::endgroup::" + echo "::endgroup::" echo "::group::Rust crates $ref" - crate_log=$(mktemp) + + crate_report="$reports_dir/crates-${ref##*/}.json" if trivy image \ --scanners vuln \ --pkg-types library \ - --severity "$severity" \ - --format table \ - --exit-code 1 \ - "$ref" 2>&1 | tee "$crate_log"; then - if grep -Eq 'language-specific files[[:space:]]+num=0' "$crate_log"; then - echo "No Rust crate results for $ref. Check that Dockerfile.konflux.* uses cargo auditable." + --severity "$crate_severity" \ + --list-all-pkgs \ + --format json \ + --output "$crate_report" \ + "$ref"; then + if ! trivy convert \ + --scanners vuln \ + --format table \ + --severity "$crate_severity" \ + "$crate_report"; then + echo "::error::Trivy could not render the crate report for $ref" + record_result \ + "$ref" "Rust crates" "❌" "scan error" \ + "Trivy could not render the crate report." + status=1 else - echo "No $severity crate vulnerabilities in $ref" + crate_count=$( + jq '[.Results[]?.Vulnerabilities[]? | + select( + .Severity == "MEDIUM" or + .Severity == "HIGH" or + .Severity == "CRITICAL" + )] | length' \ + "$crate_report" + ) + + rustbinary_count=$( + jq '[.Results[]? | + select( + .Type == "rustbinary" and + ((.Packages // []) | length > 0) + )] | length' \ + "$crate_report" + ) + + metadata_note="" + if [ "$rustbinary_count" -eq 0 ]; then + metadata_note=" No auditable crate metadata detected; check Dockerfile.konflux.* uses cargo auditable." + echo "::warning::No auditable crate metadata detected in $ref. Check Dockerfile.konflux.* uses cargo auditable." + fi + + if [ "$crate_count" -gt 0 ]; then + echo "::error::Trivy found Medium, High, or Critical crate vulnerabilities in $ref" + record_result \ + "$ref" "Rust crates" "❌" \ + "$crate_count at MEDIUM/HIGH/CRITICAL" \ + "Crate findings fail the workflow.$metadata_note" + status=1 + elif [ -n "$metadata_note" ]; then + record_result \ + "$ref" "Rust crates" "βœ…" \ + "0 at MEDIUM/HIGH/CRITICAL; metadata missing" \ + "Coverage warning:$metadata_note" + else + record_result \ + "$ref" "Rust crates" "βœ…" \ + "0 at MEDIUM/HIGH/CRITICAL" \ + "No gated crate findings." + fi fi else - echo "::error::Trivy found $severity crate vulnerabilities in $ref" + echo "::error::Trivy could not scan Rust crates in $ref" + record_result \ + "$ref" "Rust crates" "❌" "scan error" \ + "Trivy could not complete the crate scan." status=1 fi - rm -f "$crate_log" + echo "::endgroup::" done exit "$status" + notify-slack: + name: Notify Slack on failure + needs: scan + if: >- + ${{ always() && + needs.scan.result == 'failure' && + github.repository == 'red-hat-data-services/openshell' }} + runs-on: ubuntu-latest + permissions: {} + steps: + - name: Post failure notification + env: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} + WORKFLOW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + if [ -z "${SLACK_WEBHOOK_URL:-}" ]; then + echo "::warning::SLACK_WEBHOOK_URL is not set; skipping Slack notification." + exit 0 + fi + + message="$(printf '%s\n%s\n%s' \ + ':x: ODH Trivy Image Scan' \ + "Run: $WORKFLOW_RUN_URL" \ + "See the run's Step Summary for the full RPM and Rust-crate result table.")" + + jq -nc \ + --arg text "$message" \ + '{text: $text}' | + curl --fail --silent --show-error \ + -X POST \ + -H 'Content-Type: application/json' \ + --data @- \ + "$SLACK_WEBHOOK_URL" diff --git a/AGENTS.md b/AGENTS.md index b049c28f50..083785987f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -171,6 +171,10 @@ ocsf_emit!(event); test-only connections, though using it on any non-UDS TCP stream β€” tests included β€” is fine and preferred. +## Branch Names + +Follow the [branch naming convention in CONTRIBUTING.md](CONTRIBUTING.md#branch-names). + ## Commits - Always use [Conventional Commits](https://www.conventionalcommits.org/) format for commit messages diff --git a/CI.md b/CI.md index 600d2d8a54..1a1a74280f 100644 --- a/CI.md +++ b/CI.md @@ -77,10 +77,9 @@ runs without optional E2E labels. Core integration qualification builds and inst the DEB on Ubuntu with Docker and installs the CLI and gateway RPMs on Fedora with rootful and rootless Podman. These lanes run conformance using the matching runtime images. Release Dev and Release Tag use the same package installers. -Fedora provider-refresh tests also use RPMs. The Podman driver-specific branch -lanes use RPMs for rootful and rootless user-namespace comparisons and rootless -Podman E2E. Their fixtures use the installed gateway's registration, active -configuration, and service context. The manual Integration Tests workflow defaults +Fedora provider-refresh tests also use RPMs. The Podman driver-specific suites +retain the binary installer because their fixtures configure its system service, +local HTTP gateway, and CLI path. The manual Integration Tests workflow defaults to the package installers and downloads the packages selected by its matrix. Three opt-in labels enable the long-running E2E suites: @@ -418,7 +417,7 @@ Flow: Prerequisites: - DCO sign-off (`git commit -s`) on every commit. Manual admission does not require cryptographic commit signing. -- First-time external contributors must be vouched. See the [Vouch System](AGENTS.md#vouch-system). +- First-time external contributors must be vouched. See the [First-Time Contributors](CONTRIBUTING.md#first-time-contributors). Flow: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 95c9d43ffb..7da8be04b3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -112,12 +112,24 @@ Contributor and maintainer skills live in `.agents/skills/`. They are marked int ### Issue Workflow -Community issues move through triage, technical validation, and human acceptance. The repository's `state:*` labels record these stages. Inspect the current GitHub labels and their descriptions before applying or interpreting them; do not assume a fixed label list. An agent may assess evidence and request missing information. A maintainer decides whether to accept valid work and where it belongs on the roadmap. +Community issues move through triage, technical validation, and human acceptance. The repository's `state:*` labels record these stages. Follow [Label Discovery](#label-discovery) before applying or interpreting them; do not assume a fixed label list. An agent may assess evidence and request missing information. A maintainer decides whether to accept valid work and where it belongs on the roadmap. A direct request to an agent authorizes the requested planning or implementation. A request to plan alone does not authorize implementation. For unattended work, inspect current state descriptions, maintainer assignments, and comments to determine the authorized phase. Technical validation alone does not authorize implementation. Check for an existing owner, branch, or PR before starting. Do not file suspected vulnerabilities as public issues. Follow [SECURITY.md](SECURITY.md). Use the specialized security skills for authorized review or remediation. +### Label Discovery + +Retrieve every page of repository labels and their descriptions: + +```shell +gh api --paginate 'repos/{owner}/{repo}/labels?per_page=100' --jq '.[] | {name, description}' +``` + +The default `gh label list` output is limited to 30 labels. Do not treat that first page as the complete workflow definition. If a description is missing or ambiguous, check maintainer instructions and issue comments for the intended meaning. If it remains unclear, ask a maintainer to define it before changing the state or starting unattended work. Do not infer authorization from a label name alone. + +When recording a triage outcome, replace the previous assessment state rather than accumulating assessment labels. Preserve maintainer acceptance, execution state, roadmap decisions, and unrelated labels. + ## Prerequisites Install [mise](https://mise.jdx.dev/). This is used to set up the development environment. @@ -353,12 +365,22 @@ See [docs/CONTRIBUTING.mdx](docs/CONTRIBUTING.mdx) for the current docs authorin ## Pull Requests -1. Create a branch from `main` named `/-/`, using a Conventional Commits type such as `feat`, `fix`, `docs`, or `chore`. +1. Create a branch from `main`, or from the parent PR branch for dependent work, following [Branch Names](#branch-names). 2. Make your changes with tests. 3. Run the checks appropriate to the affected code and behavior, as described below. 4. Open a PR using the `create-github-pr` skill or manually following the [PR template](.github/PULL_REQUEST_TEMPLATE.md). -Every PR must close an existing issue. In the PR's **Related Issue** section, use `Closes #NNN` for the issue covering that PR's scope. Split multi-PR work into a closable issue for each PR; a tracking issue can link them. Security fixes follow the private disclosure process in [SECURITY.md](SECURITY.md). +Every PR except an automated dependency update must close an existing issue. In the PR's **Related Issue** section, use `Closes #NNN` for the issue covering that PR's scope. Split multi-PR work into a closable issue for each PR; a tracking issue can link them. Security fixes follow the private disclosure process in [SECURITY.md](SECURITY.md). + +Automated dependency update PRs, including Dependabot PRs, are exempt from the separate issue and closing-reference requirement. Human-authored dependency updates follow the normal issue requirement. The exception does not change applicable verification, review, or private vulnerability disclosure requirements. + +### Branch Names + +Use `/-/` for contributor branches. Choose the Conventional Commit type that describes the work, such as `feat`, `fix`, `refactor`, `chore`, `perf`, `docs`, `test`, or `ci`. The description is a short lowercase slug separated by hyphens; the final component is your GitHub login, not initials. + +For example, issue 1234 with contributor `octocat` uses `feat/1234-add-pagination/octocat`. Dependent PRs follow the same naming rule and branch from their parent PR's branch. + +Branches created by automation, such as Dependabot and CI mirror branches, keep their generated names. For private security work, follow the authorized disclosure workflow and use a neutral description that does not expose vulnerability details. ### Choose Verification for the Change diff --git a/Cargo.lock b/Cargo.lock index a36b8cd4a5..3a34f0cefa 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4330,6 +4330,7 @@ dependencies = [ "miette", "nix 0.29.0", "openshell-extension-core", + "openshell-otel", "openshell-policy-schema", "prost", "prost-types", @@ -4680,6 +4681,7 @@ dependencies = [ "serde", "serde_json", "sha2 0.10.9", + "socket2", "tokio", ] @@ -4980,6 +4982,7 @@ dependencies = [ "tower-http", "tracing", "tracing-appender", + "tracing-core", "tracing-opentelemetry", "tracing-subscriber", "url", @@ -5009,12 +5012,14 @@ dependencies = [ "openshell-extension-core", "openshell-isolation-interface", "openshell-ocsf", + "openshell-otel", "openshell-policy", "openshell-sandbox-backend", "openshell-supervisor-middleware", "openshell-supervisor-middleware-builtins", "openshell-supervisor-network", "openshell-supervisor-process", + "opentelemetry", "prost", "prost-types", "rustix 1.1.4", @@ -5090,10 +5095,12 @@ dependencies = [ "openshell-core", "openshell-isolation-interface", "openshell-ocsf", + "openshell-otel", "openshell-policy", "openshell-policy-schema", "openshell-supervisor-middleware", "openshell-supervisor-middleware-builtins", + "opentelemetry_sdk", "prost-types", "rcgen", "regorus", diff --git a/TESTING.md b/TESTING.md index bf94a3ebc0..a1c2f9476e 100644 --- a/TESTING.md +++ b/TESTING.md @@ -290,27 +290,10 @@ Run the portable subset in a disposable rootless Podman guest: ```shell nix run .#build-artifacts -nix run .#tmachine -- test fedora-podman-rootless rpm e2e-podman -nix run .#tmachine -- test fedora-podman-rootless rpm driver-podman +nix run .#tmachine -- test fedora-podman-rootless binaries e2e-podman +nix run .#tmachine -- test fedora-podman-rootless binaries driver-podman ``` -The driver suites also support the `binaries` installer. The shared installer -roles save the active gateway configuration, registration name, service scope, -service owner, and network name in `/var/lib/openshell-test/gateway.yaml`. -Suites resolve `openshell` from PATH and use that registration, including the -packaged gateway's HTTPS client credentials. Namespace fixtures modify the -active qualification configuration and restart its system or user service. -Failure diagnostics select the matching journal unit and user ID. Missing -metadata or credentials fail the run; suites do not replace package setup with -an HTTP gateway. Ansible sources participate in tmachine's installation cache -hash, so older cached installations are rebuilt with this metadata. - -The `driver-podman` suite supports rootful and rootless Podman. The -`e2e-podman` archive requires rootless Podman for its host workload fixtures. -DEB and RPM installers share the gateway role; available environments pair -DEB with Ubuntu/Docker and RPM with Fedora/Podman. A DEB/Podman run requires an -Ubuntu Podman environment. - Print the exact tmachine archive selection as a shell `PODMAN_CI_TESTS` array: ```shell diff --git a/crates/openshell-cli/Cargo.toml b/crates/openshell-cli/Cargo.toml index f097bb856f..5c92eefe5b 100644 --- a/crates/openshell-cli/Cargo.toml +++ b/crates/openshell-cli/Cargo.toml @@ -88,7 +88,7 @@ tracing-subscriber = { workspace = true } workspace = true [target.'cfg(unix)'.dependencies] -nix = { workspace = true } +nix = { workspace = true, features = ["poll"] } [dev-dependencies] # Tests import the example profiles from providers/ the way an operator diff --git a/crates/openshell-cli/src/run.rs b/crates/openshell-cli/src/run.rs index afd5872a45..f4728ecfea 100644 --- a/crates/openshell-cli/src/run.rs +++ b/crates/openshell-cli/src/run.rs @@ -1884,7 +1884,7 @@ enum PipedStdin { /// never waits on a thread parked in `read(2)`. The thread exits at EOF, on a /// read error, or when the receiver is dropped. fn spawn_piped_stdin_reader( - mut reader: impl Read + Send + 'static, + mut reader: impl PipedInput, ) -> tokio::sync::mpsc::Receiver>> { let (tx, rx) = tokio::sync::mpsc::channel::>>(64); std::thread::spawn(move || { @@ -1893,6 +1893,16 @@ fn spawn_piped_stdin_reader( match reader.read(&mut buf) { Ok(0) => return, Err(error) if error.kind() == ErrorKind::Interrupted => {} + // Processes that inherit the same stdin share its open file + // description, so another process may have made it + // nonblocking. Wait for input instead of failing. + #[cfg(unix)] + Err(error) if error.kind() == ErrorKind::WouldBlock => { + if let Err(error) = wait_until_readable(&reader) { + let _ = tx.blocking_send(Err(error)); + return; + } + } Err(error) => { let _ = tx.blocking_send(Err(error)); return; @@ -1908,6 +1918,30 @@ fn spawn_piped_stdin_reader( rx } +/// Input the piped-stdin reader accepts. On Unix it must expose a descriptor +/// so a nonblocking stream can be waited on. +#[cfg(unix)] +trait PipedInput: Read + std::os::fd::AsFd + Send + 'static {} +#[cfg(unix)] +impl PipedInput for T {} +#[cfg(not(unix))] +trait PipedInput: Read + Send + 'static {} +#[cfg(not(unix))] +impl PipedInput for T {} + +/// Block until `reader` has input or reaches end of file. +#[cfg(unix)] +fn wait_until_readable(reader: &impl std::os::fd::AsFd) -> std::io::Result<()> { + use nix::poll::{PollFd, PollFlags, PollTimeout, poll}; + let mut fds = [PollFd::new(reader.as_fd(), PollFlags::POLLIN)]; + loop { + match poll(&mut fds, PollTimeout::NONE) { + Err(nix::errno::Errno::EINTR) => {} + result => return result.map(drop).map_err(std::io::Error::from), + } + } +} + /// Collect piped stdin until EOF or until `grace` elapses, whichever comes /// first. Input beyond `limit` bytes is rejected with the upload hint. async fn collect_piped_stdin( @@ -8811,6 +8845,42 @@ mod tests { ); } + #[cfg(unix)] + #[test] + fn piped_stdin_left_nonblocking_by_another_process_still_streams() { + // Processes that inherit the same stdin share one open file + // description, so any of them can make it nonblocking for all. A + // read with no input yet then fails with EAGAIN instead of waiting. + use std::os::fd::AsRawFd as _; + let (reader, mut writer) = std::io::pipe().expect("pipe"); + nix::fcntl::fcntl( + reader.as_raw_fd(), + nix::fcntl::FcntlArg::F_SETFL(nix::fcntl::OFlag::O_NONBLOCK), + ) + .expect("make the shared pipe nonblocking"); + let runtime = exec_stdin_runtime(); + let collected = runtime.block_on(super::collect_piped_stdin( + super::spawn_piped_stdin_reader(reader), + Duration::from_millis(100), + super::MAX_EXEC_STDIN_BYTES, + )); + let super::PipedStdin::Open { prefix, mut rest } = collected.expect("collect") else { + panic!("an open pipe must start the command before EOF"); + }; + assert!(prefix.is_empty()); + writer.write_all(b"late").unwrap(); + drop(writer); + let next = runtime + .block_on(rest.recv()) + .expect("late chunk") + .expect("read"); + assert_eq!(next, b"late"); + assert!( + runtime.block_on(rest.recv()).is_none(), + "EOF closes the channel" + ); + } + #[test] fn piped_stdin_over_the_limit_is_rejected_with_the_upload_hint() { let (reader, mut writer) = std::io::pipe().expect("pipe"); diff --git a/crates/openshell-core/Cargo.toml b/crates/openshell-core/Cargo.toml index a246a40442..bcaf213f51 100644 --- a/crates/openshell-core/Cargo.toml +++ b/crates/openshell-core/Cargo.toml @@ -40,6 +40,7 @@ chrono = { version = "0.4", default-features = false, features = ["clock", "std" reqwest = { workspace = true, features = ["blocking", "rustls-tls-native-roots-no-provider"], optional = true } tar = { version = "0.4", optional = true } tempfile = { version = "3", optional = true } +openshell-otel = { path = "../openshell-otel", optional = true } [target.'cfg(unix)'.dependencies] nix = { workspace = true } @@ -56,6 +57,8 @@ telemetry = ["dep:reqwest", "dep:chrono", "reqwest?/blocking"] oauth = ["dep:reqwest"] ## Strict Ed25519 JWT issuance and verification for authenticated sandbox sessions. jwt = ["dep:jsonwebtoken", "dep:zeroize"] +## Propagate the active W3C trace context on supervisor gateway calls. +trace-context = ["dep:openshell-otel"] [build-dependencies] tonic-prost-build = { workspace = true } diff --git a/crates/openshell-core/src/grpc_client.rs b/crates/openshell-core/src/grpc_client.rs index f95314fece..be0aed5e49 100644 --- a/crates/openshell-core/src/grpc_client.rs +++ b/crates/openshell-core/src/grpc_client.rs @@ -195,6 +195,9 @@ impl tonic::service::Interceptor for AuthInterceptor { .expect("auth interceptor token slot poisoned") .clone(); req.metadata_mut().insert("authorization", bearer); + #[cfg(feature = "trace-context")] + let req = + tonic::service::Interceptor::call(&mut openshell_otel::TraceContextInterceptor, req)?; Ok(req) } } @@ -263,6 +266,34 @@ async fn build_plain_channel(endpoint: &str) -> Result { .wrap_err("failed to connect to OpenShell server") } +/// Marks the current client span failed unless the call finishes successfully. +struct ClientSpanStatus { + span: tracing::Span, + finished: bool, +} + +impl ClientSpanStatus { + fn current() -> Self { + Self { + span: tracing::Span::current(), + finished: false, + } + } + + fn finish(mut self, result: Result) -> Result { + self.finished = result.is_ok(); + result + } +} + +impl Drop for ClientSpanStatus { + fn drop(&mut self) { + if !self.finished { + self.span.record("otel.status_code", "ERROR"); + } + } +} + /// Build a Bearer-authenticated channel to the gateway. /// /// First call per process resolves the sandbox JWT via the three-step @@ -901,13 +932,19 @@ pub async fn fetch_policy( /// this snapshot instead of re-fetching metadata after policy construction. /// The snapshot also carries the external middleware registrations required /// by the policy. +#[tracing::instrument( + name = "supervisor.gateway.fetch_settings_snapshot", + skip_all, + fields(otel.kind = "client", otel.status_code = tracing::field::Empty) +)] pub async fn fetch_settings_snapshot( endpoint: &str, sandbox_name: &str, ) -> Result { + let status = ClientSpanStatus::current(); debug!(endpoint = %endpoint, sandbox_name = %sandbox_name, "Connecting to fetch OpenShell settings snapshot"); let mut client = connect(endpoint).await?; - fetch_settings_snapshot_with_client(&mut client, sandbox_name, None).await + status.finish(fetch_settings_snapshot_with_client(&mut client, sandbox_name, None).await) } async fn fetch_settings_snapshot_with_client( @@ -1014,19 +1051,30 @@ pub async fn sync_policy( } /// Sync an enriched policy and return the authoritative revision snapshot. +#[tracing::instrument( + name = "supervisor.gateway.sync_policy_and_fetch_snapshot", + skip_all, + fields(otel.kind = "client", otel.status_code = tracing::field::Empty) +)] pub async fn sync_policy_and_fetch_snapshot( endpoint: &str, sandbox: &str, policy: &ProtoSandboxPolicy, workspace: &str, ) -> Result { + let status = ClientSpanStatus::current(); let mut client = connect(endpoint).await?; sync_policy_with_client(&mut client, sandbox, policy, workspace).await?; - fetch_settings_snapshot_with_client(&mut client, sandbox, Some(workspace)).await + status.finish(fetch_settings_snapshot_with_client(&mut client, sandbox, Some(workspace)).await) } /// Report an exact runtime configuration generation. Pending registration uses /// the snapshot's instance fence; retain that snapshot across registration retries. +#[tracing::instrument( + name = "supervisor.gateway.report_sandbox_configuration", + skip_all, + fields(otel.kind = "client", otel.status_code = tracing::field::Empty) +)] pub async fn report_sandbox_configuration( endpoint: &str, sandbox_id: &str, @@ -1035,6 +1083,7 @@ pub async fn report_sandbox_configuration( state: crate::proto::ConfigurationAdmissionState, error: &str, ) -> Result<()> { + let status = ClientSpanStatus::current(); let mut client = connect(endpoint).await?; client .report_sandbox_configuration(crate::proto::ReportSandboxConfigurationRequest { @@ -1056,7 +1105,7 @@ pub async fn report_sandbox_configuration( }) .await .map_err(grpc_status_error)?; - Ok(()) + status.finish(Ok(())) } /// Fetch provider environment variables for a sandbox from `OpenShell` server via gRPC. @@ -1064,10 +1113,16 @@ pub async fn report_sandbox_configuration( /// Returns the credential snapshot and its exact readiness identity. An empty /// environment represents a sandbox without provider credentials. Transport /// failure returns an error so callers can revoke credentials and retry. +#[tracing::instrument( + name = "supervisor.gateway.fetch_provider_environment", + skip_all, + fields(otel.kind = "client", otel.status_code = tracing::field::Empty) +)] pub async fn fetch_provider_environment( endpoint: &str, sandbox_id: &str, ) -> Result { + let status = ClientSpanStatus::current(); debug!(endpoint = %endpoint, sandbox_id = %sandbox_id, "Fetching provider environment"); let mut client = connect(endpoint).await?; @@ -1080,7 +1135,7 @@ pub async fn fetch_provider_environment( .await .map_err(grpc_status_error)?; - provider_environment_result(response.into_inner()) + status.finish(provider_environment_result(response.into_inner())) } /// Preserve snapshot authority and reject invalid credential expiration times. @@ -1181,6 +1236,11 @@ mod provider_environment_tests { } } +#[tracing::instrument( + name = "supervisor.gateway.exchange_provider_subject_token", + skip_all, + fields(otel.kind = "client", otel.status_code = tracing::field::Empty) +)] pub async fn exchange_provider_subject_token( endpoint: &str, sandbox_id: &str, @@ -1188,6 +1248,7 @@ pub async fn exchange_provider_subject_token( credential_key: &str, supervisor_jwt_svid: &str, ) -> Result { + let status = ClientSpanStatus::current(); debug!( endpoint = %endpoint, sandbox_id = %sandbox_id, @@ -1216,11 +1277,11 @@ pub async fn exchange_provider_subject_token( .map_or(0, |value| { i64::try_from(value.as_secs()).unwrap_or(i64::MAX) }); - Ok(ProviderSubjectTokenExchangeResult { + status.finish(Ok(ProviderSubjectTokenExchangeResult { access_token: inner.access_token, expires_in, token_type: inner.token_type, - }) + })) } fn provider_subject_token_exchange_status(status: Status) -> miette::Report { diff --git a/crates/openshell-core/src/sandbox_env.rs b/crates/openshell-core/src/sandbox_env.rs index 9ac843cecd..5c3f981f57 100644 --- a/crates/openshell-core/src/sandbox_env.rs +++ b/crates/openshell-core/src/sandbox_env.rs @@ -26,6 +26,9 @@ pub const SSH_SOCKET_PATH: &str = "OPENSHELL_SSH_SOCKET_PATH"; /// Log level for the sandbox supervisor (e.g. `"debug"`, `"info"`, `"warn"`). pub const LOG_LEVEL: &str = "OPENSHELL_LOG_LEVEL"; +/// OTLP/gRPC collector endpoint for supervisor trace export. +pub const OTLP_ENDPOINT: &str = "OPENSHELL_OTLP_ENDPOINT"; + /// Versioned specification for the exact canonical main process. /// /// Most drivers use JSON directly. Transports that cannot preserve spaces in diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index dad13d74a4..70c44bcdea 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -178,6 +178,11 @@ pub struct DockerComputeConfig { /// Gateway gRPC endpoint the sandbox connects back to. pub grpc_endpoint: String, + /// OTLP/gRPC collector endpoint passed to supervisors. The gateway + /// supplies its own export endpoint; driver TOML cannot set it. + #[serde(skip)] + pub supervisor_otlp_endpoint: Option, + /// Image containing the trusted `openshell-sandbox` binary. pub sandbox_runtime_image: Option, @@ -276,6 +281,7 @@ impl Default for DockerComputeConfig { image_pull_policy: ImagePullPolicy::default(), sandbox_label: "default".to_string(), grpc_endpoint: String::new(), + supervisor_otlp_endpoint: None, sandbox_runtime_image: None, supervisor_bin: None, supervisor_image: None, @@ -310,6 +316,7 @@ struct DockerDriverRuntimeConfig { sandbox_binary: Arc>, supervisor_image_id: String, supervisor_grpc_endpoint: String, + supervisor_otlp_endpoint: Option, ssh_socket_path: String, guest_tls: Option, gpu: DockerGpuRuntimeCapabilities, @@ -939,6 +946,7 @@ impl DockerComputeDriver { sandbox_binary, supervisor_image_id, supervisor_grpc_endpoint, + supervisor_otlp_endpoint: docker_config.supervisor_otlp_endpoint.clone(), ssh_socket_path: docker_config.ssh_socket_path.clone(), guest_tls, gpu, @@ -5093,6 +5101,7 @@ async fn spawn_docker_control_process( openshell_core::telemetry::enabled_env_value() ), ]; + environment.extend(supervisor_tracing_environment(config)); if config.guest_tls.is_some() { environment.push(format!( "{}={SUPERVISOR_STATE_MOUNT_PATH}/tls/ca.pem", @@ -5534,6 +5543,17 @@ fn docker_child_environment(sandbox: &DriverSandbox) -> HashMap environment } +/// Environment that lets the supervisor export spans and join the current trace. +fn supervisor_tracing_environment(config: &DockerDriverRuntimeConfig) -> Vec { + let Some(endpoint) = &config.supervisor_otlp_endpoint else { + return Vec::new(); + }; + std::iter::once((openshell_core::sandbox_env::OTLP_ENDPOINT, endpoint.clone())) + .chain(openshell_otel::current_trace_context_environment()) + .map(|(name, value)| format!("{name}={value}")) + .collect() +} + fn build_boundary_environment( sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, diff --git a/crates/openshell-driver-docker/src/main.rs b/crates/openshell-driver-docker/src/main.rs index 247b9d9376..8134df4dc5 100644 --- a/crates/openshell-driver-docker/src/main.rs +++ b/crates/openshell-driver-docker/src/main.rs @@ -77,6 +77,7 @@ async fn main() -> Result<()> { if let Some(image) = args.supervisor_image { docker_config.supervisor_image = Some(image); } + docker_config.supervisor_otlp_endpoint = args.otlp_endpoint.clone(); let driver = DockerComputeDriver::new(args.gateway_bind, &args.log_level, &docker_config) .await .into_diagnostic()?; diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index 848be6b135..075300ec26 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -210,6 +210,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig { sandbox_binary: Arc::new(b"\x7fELFtest".to_vec()), supervisor_image_id: "sha256:supervisor-test".to_string(), supervisor_grpc_endpoint: "https://host.openshell.internal:8443".to_string(), + supervisor_otlp_endpoint: None, ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(), guest_tls: Some(DockerGuestTlsPaths { ca: PathBuf::from("/tmp/ca.crt"), @@ -1389,6 +1390,21 @@ fn docker_child_environment_strips_supervisor_control_keys() { assert!(!env.values().any(|value| value == "spoofed")); } +#[test] +fn supervisor_tracing_environment_requires_an_endpoint() { + let mut config = runtime_config(); + assert!(supervisor_tracing_environment(&config).is_empty()); + + config.supervisor_otlp_endpoint = Some("http://127.0.0.1:4317".to_string()); + assert_eq!( + supervisor_tracing_environment(&config), + [format!( + "{}=http://127.0.0.1:4317", + openshell_core::sandbox_env::OTLP_ENDPOINT + )] + ); +} + #[test] fn boundary_environment_contains_only_driver_owned_values() { let env = build_boundary_environment(&test_sandbox(), &runtime_config()); diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 6dd2f764b8..29f278ff23 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -17,8 +17,12 @@ workspace namespace modes via `workspace_mode`: - **Shared** (default): All sandboxes render into a single static namespace. Resource names use `{workspace}--{name}` for collision avoidance. - **Managed**: The driver auto-creates/deletes a K8s namespace per workspace - (`openshell-{gateway_id}-{workspace_name}`), creates a ServiceAccount in each, - and copies OpenShift SCC annotations from the gateway namespace when present. + (`openshell-{gateway_id}-{workspace_name}`) and creates a ServiceAccount in + each. On OpenShift, it leaves SCC annotations to the namespace allocator and + waits for the namespace's own MCS, UID-range, and supplemental-group + annotations before provisioning sandbox resources. An existing namespace with + a UID range but no MCS must be recreated so OpenShift can allocate a complete + set of SCC annotations. - **Operator**: Workspace names map 1:1 to pre-provisioned namespaces discovered through exactly one source: either a label selector (`operator_namespace_label`) or a drop-in allowlist file @@ -75,6 +79,12 @@ The supervisor Pod has a direct, non-controller owner reference to the Sandbox resource. This links its garbage-collection lifecycle to the sandbox without competing with the Agent Sandbox controller for workload-Pod ownership. +When the gateway exports OTLP traces, the driver sets +`OPENSHELL_OTLP_ENDPOINT` on the supervisor Pod to the gateway's endpoint and +`TRACEPARENT` to the trace context of the operation that created the Pod. The +supervisor exports its spans there and parents its startup span on that +context. The endpoint is not configurable in driver TOML. + The driver creates one namespace-wide `NetworkPolicy` before it releases any workload Pod. It selects every OpenShell workload, denies all workload egress, and permits OpenShell supervisor Pods to reach the sandbox TLS port. The diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index 992c9741ab..6bd7d93af3 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -244,6 +244,10 @@ pub struct KubernetesComputeConfig { /// contain at least one usable trust anchor. pub proxy_ca_bundle: Option, pub grpc_endpoint: String, + /// OTLP/gRPC collector endpoint passed to supervisor pods. The gateway + /// supplies its own export endpoint; driver TOML cannot set it. + #[serde(skip)] + pub supervisor_otlp_endpoint: Option, pub ssh_socket_path: String, pub client_tls_secret_name: String, pub host_gateway_ip: String, @@ -312,6 +316,9 @@ pub(crate) const DEFAULT_SANDBOX_UID: u32 = 10001; /// Format: `/` (e.g. `1000000000/10000`). pub const ANNOTATION_SCC_UID_RANGE: &str = "openshift.io/sa.scc.uid-range"; +/// The annotation key for the `OpenShift` MCS label allocated to a namespace. +pub const ANNOTATION_SCC_MCS: &str = "openshift.io/sa.scc.mcs"; + /// The annotation key for the `OpenShift` `ServiceAccount` supplemental groups. /// Format: `/` (e.g. `1000000000/10000`). pub const ANNOTATION_SCC_SUPPLEMENTAL_GROUPS: &str = "openshift.io/sa.scc.supplemental-groups"; @@ -349,6 +356,7 @@ impl Default for KubernetesComputeConfig { proxy_connect_by_hostname: None, proxy_ca_bundle: None, grpc_endpoint: String::new(), + supervisor_otlp_endpoint: None, ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(), client_tls_secret_name: String::new(), host_gateway_ip: String::new(), diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index b0cc8078e6..42f75abaa6 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -267,6 +267,7 @@ impl From for openshell_core::ComputeDriverError { /// This prevents gRPC handlers from blocking indefinitely when the k8s /// API server is unreachable or slow. const KUBE_API_TIMEOUT: Duration = Duration::from_secs(30); +const OPENSHIFT_SCC_ALLOCATOR_POLL_INTERVAL: Duration = Duration::from_millis(100); fn admission_error(error: tonic::Status) -> KubernetesDriverError { match error.code() { tonic::Code::InvalidArgument => { @@ -703,6 +704,19 @@ impl std::fmt::Debug for KubernetesComputeDriver { } impl KubernetesComputeDriver { + /// Environment that lets a supervisor export spans and join the current trace. + fn supervisor_tracing_environment(&self) -> Vec<(&'static str, String)> { + let Some(endpoint) = self.config.supervisor_otlp_endpoint.as_deref() else { + return Vec::new(); + }; + let mut environment = vec![( + openshell_core::sandbox_env::OTLP_ENDPOINT, + endpoint.to_string(), + )]; + environment.extend(openshell_otel::current_trace_context_environment()); + environment + } + #[cfg(test)] pub(crate) fn new_for_test(config: KubernetesComputeConfig) -> Self { let service = tower::service_fn(|_request: http::Request| async { @@ -1063,6 +1077,13 @@ impl KubernetesComputeDriver { ))); } }; + let openshift_scc_allocator_enabled = [ + crate::config::ANNOTATION_SCC_MCS, + crate::config::ANNOTATION_SCC_UID_RANGE, + crate::config::ANNOTATION_SCC_SUPPLEMENTAL_GROUPS, + ] + .iter() + .any(|key| gateway_ns_annotations.contains_key(*key)); let mut labels = BTreeMap::new(); labels.insert( @@ -1072,35 +1093,24 @@ impl KubernetesComputeDriver { labels.insert(LABEL_GATEWAY_ID.to_string(), self.config.gateway_id.clone()); labels.insert(LABEL_SANDBOX_WORKSPACE.to_string(), workspace.to_string()); - let mut annotations = BTreeMap::new(); - for key in [ - crate::config::ANNOTATION_SCC_UID_RANGE, - crate::config::ANNOTATION_SCC_SUPPLEMENTAL_GROUPS, - ] { - if let Some(val) = gateway_ns_annotations.get(key) { - annotations.insert(key.to_string(), val.clone()); - } - } - let ns = Namespace { metadata: ObjectMeta { name: Some(ns_name.clone()), labels: Some(labels), - annotations: if annotations.is_empty() { - None - } else { - Some(annotations) - }, ..Default::default() }, ..Default::default() }; - match tokio::time::timeout(KUBE_API_TIMEOUT, ns_api.create(&PostParams::default(), &ns)) - .await + let initial_namespace = match tokio::time::timeout( + KUBE_API_TIMEOUT, + ns_api.create(&PostParams::default(), &ns), + ) + .await { - Ok(Ok(_)) => { + Ok(Ok(created)) => { info!(namespace = %ns_name, workspace = %workspace, "created managed namespace"); + created } Ok(Err(KubeError::Api(api))) if api.code == 409 => { let existing = @@ -1122,6 +1132,7 @@ impl KubernetesComputeDriver { ))); } debug!(namespace = %ns_name, "managed namespace already exists"); + existing } Ok(Err(e)) => return Err(KubernetesDriverError::from_kube(e)), Err(_) => { @@ -1129,6 +1140,11 @@ impl KubernetesComputeDriver { "timeout creating namespace {ns_name}" ))); } + }; + + if openshift_scc_allocator_enabled { + self.wait_for_openshift_scc_annotations(&ns_api, &ns_name, initial_namespace) + .await?; } self.ensure_service_account(&ns_name).await?; @@ -1137,6 +1153,62 @@ impl KubernetesComputeDriver { Ok(ns_name) } + async fn wait_for_openshift_scc_annotations( + &self, + ns_api: &Api, + namespace: &str, + mut current: Namespace, + ) -> Result<(), KubernetesDriverError> { + let operation = async { + loop { + let annotations = current.metadata.annotations.unwrap_or_default(); + if [ + crate::config::ANNOTATION_SCC_MCS, + crate::config::ANNOTATION_SCC_UID_RANGE, + crate::config::ANNOTATION_SCC_SUPPLEMENTAL_GROUPS, + ] + .into_iter() + .all(|key| { + annotations + .get(key) + .is_some_and(|value| !value.trim().is_empty()) + }) { + return Ok(()); + } + + let has_uid_range = annotations + .get(crate::config::ANNOTATION_SCC_UID_RANGE) + .is_some_and(|value| !value.trim().is_empty()); + let has_mcs = annotations + .get(crate::config::ANNOTATION_SCC_MCS) + .is_some_and(|value| !value.trim().is_empty()); + if has_uid_range && !has_mcs { + return Err(KubernetesDriverError::Precondition(format!( + "managed namespace {namespace} has an OpenShift UID range but no MCS annotation; the SCC allocator skips namespaces with an existing UID range, so delete and recreate this workspace namespace" + ))); + } + + tokio::time::sleep(OPENSHIFT_SCC_ALLOCATOR_POLL_INTERVAL).await; + current = match ns_api.get(namespace).await { + Ok(namespace) => namespace, + Err(KubeError::Api(api)) if api.code == 404 => { + return Err(KubernetesDriverError::Message(format!( + "managed namespace {namespace} disappeared while waiting for OpenShift SCC allocator annotations" + ))); + } + Err(error) => return Err(KubernetesDriverError::from_kube(error)), + }; + } + }; + tokio::time::timeout(KUBE_API_TIMEOUT, operation) + .await + .map_err(|_| { + KubernetesDriverError::Message(format!( + "timed out waiting for OpenShift SCC allocator annotations on managed namespace {namespace}" + )) + })? + } + async fn ensure_managed_ssh_network_policy( &self, namespace: &str, @@ -2383,6 +2455,7 @@ impl KubernetesComputeDriver { .as_str(), ), dependent_owner.clone(), + &self.supervisor_tracing_environment(), ) .map_err(KubernetesDriverError::Message)?, ) @@ -3189,6 +3262,7 @@ impl KubernetesComputeDriver { &sandbox_api.resource.api_version, false, ), + &self.supervisor_tracing_environment(), ) .map_err(KubernetesDriverError::Message)?, ) @@ -11117,6 +11191,302 @@ mod tests { } } + async fn call_ensure_workspace( + service: &crate::ComputeDriverService, + ) -> Result< + tonic::Response, + tonic::Status, + > { + openshell_core::proto::compute::v1::compute_driver_server::ComputeDriver::ensure_workspace( + service, + tonic::Request::new(openshell_core::proto::compute::v1::EnsureWorkspaceRequest { + workspace: "team-a".into(), + }), + ) + .await + } + + #[tokio::test] + async fn managed_openshift_namespace_waits_for_allocator_without_copying_annotations() { + let namespace = "openshell-gateway-a-team-a"; + let gateway_namespace = serde_json::json!({ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": "openshell", + "annotations": { + "openshift.io/sa.scc.mcs": "s0:c1,c2", + "openshift.io/sa.scc.uid-range": "1000000000/10000", + "openshift.io/sa.scc.supplemental-groups": "1000000000/10000" + } + } + }); + let pending_namespace = serde_json::json!({ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": {"name": namespace} + }); + let allocated_namespace = serde_json::json!({ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": namespace, + "annotations": { + "openshift.io/sa.scc.mcs": "s0:c3,c4", + "openshift.io/sa.scc.uid-range": "1000000001/10000", + "openshift.io/sa.scc.supplemental-groups": "1000000001/10000" + } + } + }); + for allocated_on_create in [true, false] { + let mut steps = vec![ + ( + http::Method::GET, + "/api/v1/namespaces/openshell", + kube_test_response(http::StatusCode::OK, gateway_namespace.clone()), + ), + ( + http::Method::POST, + "/api/v1/namespaces", + kube_test_response( + http::StatusCode::CREATED, + if allocated_on_create { + allocated_namespace.clone() + } else { + pending_namespace.clone() + }, + ), + ), + ]; + if !allocated_on_create { + steps.push(( + http::Method::GET, + "/api/v1/namespaces/openshell-gateway-a-team-a", + kube_test_response(http::StatusCode::OK, allocated_namespace.clone()), + )); + } + steps.push(( + http::Method::POST, + "/api/v1/namespaces/openshell-gateway-a-team-a/serviceaccounts", + kube_test_response( + http::StatusCode::CREATED, + serde_json::json!({ + "apiVersion": "v1", + "kind": "ServiceAccount", + "metadata": {"name": "default", "namespace": namespace} + }), + ), + )); + let (driver, steps, bodies) = scripted_driver(managed_config(&[]), steps); + + assert_eq!(driver.ensure_namespace("team-a").await.unwrap(), namespace); + assert!(steps.lock().unwrap().is_empty()); + assert_eq!( + bodies.lock().unwrap()[0]["metadata"].get("annotations"), + None + ); + } + } + + #[tokio::test] + async fn ensure_workspace_rejects_existing_uid_range_without_mcs() { + let namespace = "openshell-gateway-a-team-a"; + let gateway_namespace = serde_json::json!({ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": "openshell", + "annotations": { + "openshift.io/sa.scc.mcs": "s0:c1,c2", + "openshift.io/sa.scc.uid-range": "1000000000/10000", + "openshift.io/sa.scc.supplemental-groups": "1000000000/10000" + } + } + }); + let existing_namespace = serde_json::json!({ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": namespace, + "labels": { + "openshell.ai/managed-by": "openshell", + "openshell.ai/gateway-id": "gateway-a", + "openshell.ai/sandbox-workspace": "team-a" + }, + "annotations": { + "openshift.io/sa.scc.uid-range": "1000000001/10000", + "openshift.io/sa.scc.supplemental-groups": "1000000001/10000" + } + } + }); + let (driver, steps, _) = scripted_driver( + managed_config(&[]), + vec![ + ( + http::Method::GET, + "/api/v1/namespaces/openshell", + kube_test_response(http::StatusCode::OK, gateway_namespace), + ), + ( + http::Method::POST, + "/api/v1/namespaces", + kube_test_response( + http::StatusCode::CONFLICT, + serde_json::json!({ + "apiVersion": "v1", + "kind": "Status", + "status": "Failure", + "reason": "AlreadyExists", + "message": format!("namespaces {namespace:?} already exists"), + "code": 409 + }), + ), + ), + ( + http::Method::GET, + "/api/v1/namespaces/openshell-gateway-a-team-a", + kube_test_response(http::StatusCode::OK, existing_namespace), + ), + ], + ); + + let service = crate::ComputeDriverService::new(driver); + + let error = call_ensure_workspace(&service) + .await + .expect_err("a stale OpenShift namespace should be a permanent precondition failure"); + + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert!(error.message().contains("MCS") && error.message().contains("recreate")); + assert!(steps.lock().unwrap().is_empty()); + } + + #[tokio::test] + async fn ensure_workspace_keeps_namespace_setup_failures_internal() { + let gateway_namespace = serde_json::json!({ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": "openshell", + "annotations": {"openshift.io/sa.scc.uid-range": "1000000000/10000"} + } + }); + for steps in [ + vec![( + http::Method::GET, + "/api/v1/namespaces/openshell", + kube_test_not_found("namespaces", "openshell"), + )], + vec![ + ( + http::Method::GET, + "/api/v1/namespaces/openshell", + kube_test_response(http::StatusCode::OK, gateway_namespace), + ), + ( + http::Method::POST, + "/api/v1/namespaces", + kube_test_response( + http::StatusCode::CREATED, + serde_json::json!({ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": {"name": "openshell-gateway-a-team-a"} + }), + ), + ), + ( + http::Method::GET, + "/api/v1/namespaces/openshell-gateway-a-team-a", + kube_test_not_found("namespaces", "openshell-gateway-a-team-a"), + ), + ], + ] { + let (driver, steps, _) = scripted_driver(managed_config(&[]), steps); + let service = crate::ComputeDriverService::new(driver); + + let error = call_ensure_workspace(&service) + .await + .expect_err("namespace setup failures should remain internal"); + + assert_eq!(error.code(), tonic::Code::Internal); + assert!(steps.lock().unwrap().is_empty()); + } + } + + #[tokio::test(start_paused = true)] + async fn ensure_workspace_reports_scc_allocator_timeout_as_internal() { + let namespace = "openshell-gateway-a-team-a"; + let gateway_namespace = serde_json::json!({ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": "openshell", + "annotations": { + "openshift.io/sa.scc.uid-range": "1000000000/10000" + } + } + }); + let pending_namespace = serde_json::json!({ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": {"name": namespace} + }); + for stalled_get in [false, true] { + let gateway_namespace = gateway_namespace.clone(); + let pending_namespace = pending_namespace.clone(); + let reads = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let service_reads = reads.clone(); + let service = tower::service_fn(move |request: http::Request| { + let gateway_namespace = gateway_namespace.clone(); + let pending_namespace = pending_namespace.clone(); + let reads = service_reads.clone(); + async move { + let response = match (request.method(), request.uri().path()) { + (&http::Method::GET, "/api/v1/namespaces/openshell") => { + kube_test_response(http::StatusCode::OK, gateway_namespace) + } + (&http::Method::POST, "/api/v1/namespaces") => { + kube_test_response(http::StatusCode::CREATED, pending_namespace) + } + (&http::Method::GET, "/api/v1/namespaces/openshell-gateway-a-team-a") => { + reads.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + if stalled_get { + std::future::pending::<()>().await; + } + kube_test_response(http::StatusCode::OK, pending_namespace) + } + unexpected => panic!("unexpected Kubernetes API request: {unexpected:?}"), + }; + Ok::<_, std::convert::Infallible>(response) + } + }); + let client = Client::new(service, "openshell"); + let driver = KubernetesComputeDriver { + client: client.clone(), + watch_client: client, + sandbox_api_version: Arc::new(OnceCell::new()), + lifecycle_gates: Arc::default(), + config: managed_config(&[]), + operator_allowlist: None, + }; + let service = crate::ComputeDriverService::new(driver); + + let error = call_ensure_workspace(&service) + .await + .expect_err("allocator timeout should remain a retryable setup failure"); + + assert_eq!(error.code(), tonic::Code::Internal); + assert!(error.message().contains("timed out")); + let reads = reads.load(std::sync::atomic::Ordering::Relaxed); + if stalled_get { + assert_eq!(reads, 1); + } else { + assert!(reads > 1); + } + } + } + #[tokio::test] async fn managed_pods_reference_generation_image_pull_secrets() { let names = SandboxRuntimeNames::for_generation("sandbox-1", "gen7"); diff --git a/crates/openshell-driver-kubernetes/src/grpc.rs b/crates/openshell-driver-kubernetes/src/grpc.rs index 12cd0ebfff..ecab24cbe4 100644 --- a/crates/openshell-driver-kubernetes/src/grpc.rs +++ b/crates/openshell-driver-kubernetes/src/grpc.rs @@ -275,10 +275,14 @@ impl ComputeDriver for ComputeDriverService { })?; match self.driver.workspace_mode() { WorkspaceMode::Managed => { - self.driver - .ensure_namespace(&workspace) - .await - .map_err(|e| Status::internal(e.to_string()))?; + self.driver.ensure_namespace(&workspace).await.map_err( + |error| match error { + crate::KubernetesDriverError::Precondition(message) => { + Status::failed_precondition(message) + } + error => Status::internal(error.to_string()), + }, + )?; } WorkspaceMode::Operator => { if let Some(allowlist) = self.driver.operator_allowlist() diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index dc09dc1710..d70f706f5f 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -273,6 +273,7 @@ async fn main() -> Result<()> { proxy_connect_by_hostname: args.proxy_connect_by_hostname.then_some(true), proxy_ca_bundle: args.proxy_ca_bundle, grpc_endpoint: args.grpc_endpoint.unwrap_or_default(), + supervisor_otlp_endpoint: args.otlp_endpoint.clone(), ssh_socket_path: args.sandbox_ssh_socket_path, client_tls_secret_name: args.client_tls_secret_name.unwrap_or_default(), host_gateway_ip: args.host_gateway_ip.unwrap_or_default(), diff --git a/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs index 642809aff5..b6d75be77c 100644 --- a/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs +++ b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs @@ -233,6 +233,7 @@ pub fn supervisor_pod( upstream_proxy_ca_bundle_staged: bool, provider_spiffe_socket_path: Option<&str>, owner: OwnerReference, + tracing_environment: &[(&str, String)], ) -> Result { let labels = control_labels(sandbox_id, gateway_id); let mut environment = vec![ @@ -269,6 +270,11 @@ pub fn supervisor_pod( "", ), ]; + environment.extend( + tracing_environment + .iter() + .map(|(name, value)| env_var(name, value)), + ); let mut volume_mounts = vec![ volume_mount("bootstrap", "/.openshell/supervisor", true), volume_mount("sa-token", "/var/run/secrets/openshell", true), @@ -811,6 +817,7 @@ mod tests { false, None, owner(), + &[], ) .expect("render supervisor Pod") } @@ -840,6 +847,58 @@ mod tests { (env, volumes) } + #[test] + fn supervisor_pod_carries_tracing_environment() { + const TRACEPARENT: &str = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; + let client_tls = SupervisorClientTls::Disabled; + let pod = supervisor_pod( + "sandbox", + &SandboxRuntimeNames::new("pair"), + "pair", + "demo", + "gateway", + "supervisor:latest", + None, + "sandbox-sa", + 1000, + 1000, + &[], + "https://gateway:8080", + client_tls, + "{}", + "info", + 600, + None, + None, + None, + false, + false, + false, + None, + owner(), + &[ + ( + "OPENSHELL_OTLP_ENDPOINT", + "http://collector:4317".to_string(), + ), + ("TRACEPARENT", TRACEPARENT.to_string()), + ], + ) + .expect("render supervisor Pod"); + let environment = pod.spec.expect("Pod spec").containers[0] + .env + .clone() + .unwrap_or_default() + .into_iter() + .map(|variable| (variable.name, variable.value.unwrap_or_default())) + .collect::>(); + assert_eq!( + environment["OPENSHELL_OTLP_ENDPOINT"], + "http://collector:4317" + ); + assert_eq!(environment["TRACEPARENT"], TRACEPARENT); + } + #[test] fn supervisor_pod_reads_client_tls_from_the_bootstrap_secret() { let (env, volumes) = tls_env_and_volumes(&supervisor_pod_with_client_tls( @@ -920,6 +979,7 @@ mod tests { false, None, owner(), + &[], ) .expect("render supervisor Pod"); let pod_spec = pod.spec.as_ref().expect("Pod spec"); @@ -1177,6 +1237,7 @@ mod tests { staged, None, owner(), + &[], ) .expect("render supervisor Pod") } diff --git a/crates/openshell-driver-podman/README.md b/crates/openshell-driver-podman/README.md index abf2147513..85a31f2cf8 100644 --- a/crates/openshell-driver-podman/README.md +++ b/crates/openshell-driver-podman/README.md @@ -43,7 +43,8 @@ stopped Podman container does not populate nested named volumes. Restart restore only the channel bootstrap into the existing channel volume, preserving the workspace. The workload starts before the supervisor so its user namespace exists when the supervisor joins it; a stopped supervisor resolves that namespace again -on its next start. +on its next start. The driver creates the managed workspace volume owned by +the workload's final UID and GID, so the workload never starts as root. The runtime must pass the sandbox's unprivileged enforcement probe, including nested seccomp notification and Landlock. Unsupported runtime defaults fail diff --git a/crates/openshell-driver-podman/src/client.rs b/crates/openshell-driver-podman/src/client.rs index 80e24513a3..3d6b79f622 100644 --- a/crates/openshell-driver-podman/src/client.rs +++ b/crates/openshell-driver-podman/src/client.rs @@ -166,6 +166,8 @@ pub struct PortBinding { pub struct ContainerConfig { #[serde(default)] pub labels: HashMap, + #[serde(default)] + pub user: String, } /// Immutable image metadata needed to bind OCI identity inspection to launch. @@ -238,6 +240,38 @@ pub struct VolumeInspect { } impl VolumeInspect { + /// Whether metadata matches a managed local volume with the exact labels + /// and requested ownership options. This does not inspect filesystem ownership. + pub(crate) fn matches_managed_volume( + &self, + labels: &HashMap, + requested_owner: Option<(u32, u32)>, + ) -> bool { + self.driver == "local" + && self.labels.as_ref() == Some(labels) + && self.options_match_requested_owner(requested_owner) + } + + /// Whether option metadata matches the requested owner. `None` means no + /// ownership options were requested and requires an empty options map. + /// This does not inspect filesystem ownership. Podman records the parsed + /// `UID` and `GID` next to the raw `o` option. + pub(crate) fn options_match_requested_owner( + &self, + requested_owner: Option<(u32, u32)>, + ) -> bool { + let Some((uid, gid)) = requested_owner else { + return self.options.is_empty(); + }; + self.options.get("o").map(String::as_str) == Some(format!("uid={uid},gid={gid}").as_str()) + && self.options.iter().all(|(key, value)| match key.as_str() { + "o" => true, + "UID" => *value == uid.to_string(), + "GID" => *value == gid.to_string(), + _ => false, + }) + } + pub(crate) fn admission_identity(&self) -> Value { serde_json::json!({"name": self.name, "driver": self.driver, "options": self.options, "created_at": self.created_at}) } @@ -700,12 +734,38 @@ impl PodmanClient { // ── Volume operations ──────────────────────────────────────────────── + /// Create and inspect a local volume. HTTP 409 conflicts also proceed to + /// inspection; callers must verify the returned labels and options. + async fn create_volume( + &self, + name: &str, + labels: &HashMap, + options: &HashMap, + ) -> Result { + validate_name(name)?; + let mut body = serde_json::json!({ + "Name": name, + "Driver": "local", + "Labels": labels, + }); + if !options.is_empty() { + body["Options"] = serde_json::json!(options); + } + self.create_ignore_conflict("/libpod/volumes/create", &body) + .await?; + self.inspect_volume(name).await + } + /// Never adopt an unrelated existing volume on a private provisioning path. + /// + /// With `owner`, Podman creates the volume root owned by that UID and GID, + /// so a non-root workload can use it without a privileged chown. pub(crate) async fn create_owned_volume( &self, name: &str, sandbox_id: &str, workspace: &str, + owner: Option<(u32, u32)>, ) -> Result<(), PodmanApiError> { let labels = HashMap::from([ ( @@ -719,10 +779,7 @@ impl PodmanClient { ]); match self.inspect_volume(name).await { Ok(existing) => { - if existing.driver != "local" - || !existing.options.is_empty() - || existing.labels.as_ref() != Some(&labels) - { + if !existing.matches_managed_volume(&labels, owner) { return Err(PodmanApiError::InvalidInput( "private volume name collides with an unrelated resource".into(), )); @@ -732,16 +789,11 @@ impl PodmanClient { Err(PodmanApiError::NotFound(_)) => {} Err(error) => return Err(error), } - self.create_ignore_conflict( - "/libpod/volumes/create", - &serde_json::json!({"Name":name,"Driver":"local","Labels":labels}), - ) - .await?; - let created = self.inspect_volume(name).await?; - if created.driver != "local" - || !created.options.is_empty() - || created.labels.as_ref() != Some(&labels) - { + let options = owner.map_or_else(HashMap::new, |(uid, gid)| { + HashMap::from([("o".to_string(), format!("uid={uid},gid={gid}"))]) + }); + let created = self.create_volume(name, &labels, &options).await?; + if !created.matches_managed_volume(&labels, owner) { return Err(PodmanApiError::InvalidInput( "private volume ownership verification failed".into(), )); @@ -1157,6 +1209,81 @@ mod tests { let _ = std::fs::remove_file(socket_path); } + #[tokio::test] + async fn create_owned_volume_verifies_requested_options() { + let labels = + r#"{"openshell.ai/sandbox-id":"sandbox-1","openshell.ai/sandbox-workspace":"team-a"}"#; + for (owner, options, accepted) in [ + ( + Some((1234, 1235)), + r#"{"o":"uid=1234,gid=1235","UID":"1234","GID":"1235"}"#, + true, + ), + (Some((1234, 1235)), r#"{"o":"uid=1234,gid=1235"}"#, true), + // Podman accepts either order, but OpenShell always requests uid first. + (Some((1234, 1235)), r#"{"o":"gid=1235,uid=1234"}"#, false), + ( + Some((1234, 1235)), + r#"{"o":"uid=1234,gid=1235","UID":"0","GID":"1235"}"#, + false, + ), + ( + Some((1234, 1235)), + r#"{"o":"uid=1234,gid=1235","device":"/srv/work"}"#, + false, + ), + (Some((1234, 1235)), "{}", false), + (None, "{}", true), + (None, r#"{"o":"uid=1234,gid=1235"}"#, false), + (None, r#"{"o":"bind","device":"/srv/work"}"#, false), + ] { + for existing in [false, true] { + let inspected = || { + StubResponse::new( + StatusCode::OK, + format!( + r#"{{"Name":"work","Driver":"local","Options":{options},"Labels":{labels}}}"# + ), + ) + }; + let responses = if existing { + vec![inspected()] + } else { + vec![ + StubResponse::new(StatusCode::NOT_FOUND, ""), + StubResponse::new(StatusCode::CREATED, "{}"), + inspected(), + ] + }; + let (socket_path, request_log, handle) = + spawn_podman_stub("owned-volume", responses); + let result = PodmanClient::new(socket_path.clone()) + .create_owned_volume("work", "sandbox-1", "team-a", owner) + .await; + assert_eq!( + result.is_ok(), + accepted, + "owner {owner:?}, options {options}, existing {existing}: {result:?}" + ); + handle.await.expect("stub task should finish"); + let expected_requests = if existing { + vec!["GET /v5.0.0/libpod/volumes/work/json"] + } else { + vec![ + "GET /v5.0.0/libpod/volumes/work/json", + "POST /v5.0.0/libpod/volumes/create", + "GET /v5.0.0/libpod/volumes/work/json", + ] + }; + assert_eq!( + request_log.lock().expect("request log lock").as_slice(), + expected_requests, + ); + let _ = std::fs::remove_file(socket_path); + } + } + } + #[tokio::test] async fn inspect_image_reads_immutable_id_and_oci_user() { let (socket_path, request_log, handle) = spawn_podman_stub( diff --git a/crates/openshell-driver-podman/src/config.rs b/crates/openshell-driver-podman/src/config.rs index 4559cdd65e..9e4e9b571b 100644 --- a/crates/openshell-driver-podman/src/config.rs +++ b/crates/openshell-driver-podman/src/config.rs @@ -43,6 +43,10 @@ pub struct PodmanComputeConfig { /// When empty, the driver selects loopback on Linux or /// `host.containers.internal` with Podman Machine, using `gateway_port`. pub grpc_endpoint: String, + /// OTLP/gRPC collector endpoint passed to supervisors. The gateway + /// supplies its own export endpoint; driver TOML cannot set it. + #[serde(skip)] + pub supervisor_otlp_endpoint: Option, /// Port the gateway server is actually listening on. /// /// Used by the driver's auto-detection fallback when `grpc_endpoint` @@ -493,6 +497,7 @@ impl Default for PodmanComputeConfig { default_image: openshell_core::image::default_sandbox_image(), image_pull_policy: ImagePullPolicy::default(), grpc_endpoint: String::new(), + supervisor_otlp_endpoint: None, gateway_port: openshell_core::config::DEFAULT_SERVER_PORT, ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(), network_name: DEFAULT_NETWORK_NAME.to_string(), diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs index cc42e9e2f0..f6b1c71b59 100644 --- a/crates/openshell-driver-podman/src/container.rs +++ b/crates/openshell-driver-podman/src/container.rs @@ -579,6 +579,17 @@ fn build_env( openshell_core::sandbox_env::TELEMETRY_ENABLED.into(), openshell_core::telemetry::enabled_env_value().into(), ); + if let Some(endpoint) = &config.supervisor_otlp_endpoint { + env.insert( + openshell_core::sandbox_env::OTLP_ENDPOINT.into(), + endpoint.clone(), + ); + env.extend( + openshell_otel::current_trace_context_environment() + .into_iter() + .map(|(name, value)| (name.to_string(), value)), + ); + } // Runtime capabilities are driver-owned. Override image/user input with // only the substrate that this driver configures for the supervisor. env.insert( @@ -1406,8 +1417,6 @@ pub struct IsolationSpecInput<'a> { pub supervisor_bin: Option<&'a Path>, pub tls_secrets: Option<&'a [String; 1]>, pub identity: &'a openshell_isolation_interface::contract::ResolvedWorkloadIdentity, - /// Whether this workload is created by a rootless Podman service. - pub rootless: bool, } pub struct IsolationSpecs { @@ -1459,44 +1468,19 @@ pub fn build_isolation_specs( .iter() .filter_map(|entry| entry.split_once('=').map(|(key, _)| key.to_string())) .collect(); - if input.rootless || input.identity.source == "default" { - // Podman's archive endpoint leaves named-volume contents owned by - // container root for rootless services and for a rootful USER-less - // image's newly-created workspace. Start the trusted runtime as root - // only long enough to chown the workspace, then irreversibly drop to - // the resolved workload identity before reading bootstrap material or - // accepting a control connection. - workload.command = vec![ - "launch-capability-free".into(), - input.identity.uid.to_string(), - input.identity.gid.to_string(), - crate::isolation::BOOTSTRAP_PATH.into(), - driver_mounts::DEFAULT_WORKSPACE_ROOT.into(), - ]; - workload.user = "0:0".into(); - workload.groups.clear(); - workload.cap_drop = vec!["ALL".into()]; - workload.cap_add = vec![ - "CHOWN".into(), - "SETGID".into(), - "SETUID".into(), - "SETPCAP".into(), - ]; - } else { - workload.command = vec![ - "--bootstrap".into(), - crate::isolation::BOOTSTRAP_PATH.into(), - ]; - workload.user.clone_from(&user); - workload.groups = input - .identity - .supplementary_gids - .iter() - .map(ToString::to_string) - .collect(); - workload.cap_drop = vec!["ALL".into()]; - workload.cap_add.clear(); - } + workload.command = vec![ + "--bootstrap".into(), + crate::isolation::BOOTSTRAP_PATH.into(), + ]; + workload.user.clone_from(&user); + workload.groups = input + .identity + .supplementary_gids + .iter() + .map(ToString::to_string) + .collect(); + workload.cap_drop = vec!["ALL".into()]; + workload.cap_add.clear(); workload.apparmor_profile = input .config .app_armor_profile @@ -1790,7 +1774,6 @@ mod tests { supervisor_bin: None, tls_secrets: None, identity: &identity, - rootless: true, }) .unwrap(); for spec in [&specs.workload, &specs.supervisor] { @@ -1798,21 +1781,14 @@ mod tests { assert!(spec.seccomp_profile_path.is_empty()); assert!(spec.no_new_privileges); } - assert_eq!(specs.workload.user, "0:0"); - assert!(specs.workload.groups.is_empty()); - assert_eq!( - specs.workload.cap_add, - vec!["CHOWN", "SETGID", "SETUID", "SETPCAP"] - ); + // The driver creates the managed workspace volume owned by the + // workload identity, so the workload never starts as root. + assert_eq!(specs.workload.user, "1000:1001"); + assert_eq!(specs.workload.groups, vec!["2000"]); + assert!(specs.workload.cap_add.is_empty()); assert_eq!( specs.workload.command, - vec![ - "launch-capability-free", - "1000", - "1001", - crate::isolation::BOOTSTRAP_PATH, - driver_mounts::DEFAULT_WORKSPACE_ROOT, - ] + vec!["--bootstrap", crate::isolation::BOOTSTRAP_PATH] ); assert_eq!(specs.supervisor.user, "1000:1001"); assert_eq!(specs.supervisor.groups, vec!["2000"]); @@ -1841,7 +1817,7 @@ mod tests { "sha256:image".into(), ) .unwrap(); - let rootful_specs = build_isolation_specs(IsolationSpecInput { + let default_specs = build_isolation_specs(IsolationSpecInput { sandbox: &sandbox, config: &config, token_secret: Some("jwt"), @@ -1854,19 +1830,13 @@ mod tests { supervisor_bin: None, tls_secrets: None, identity: &default_identity, - rootless: false, }) .unwrap(); - assert_eq!(rootful_specs.workload.user, "0:0"); + assert_eq!(default_specs.workload.user, "1000:1000"); + assert!(default_specs.workload.cap_add.is_empty()); assert_eq!( - rootful_specs.workload.command, - vec![ - "launch-capability-free", - "1000", - "1000", - crate::isolation::BOOTSTRAP_PATH, - driver_mounts::DEFAULT_WORKSPACE_ROOT, - ] + default_specs.workload.command, + vec!["--bootstrap", crate::isolation::BOOTSTRAP_PATH] ); let workload_json = serde_json::to_string(&specs.workload).unwrap(); assert!(workload_json.contains("\"apparmor_profile\":\"openshell-sandbox\"")); @@ -2461,6 +2431,27 @@ mod tests { ); } + #[test] + fn container_spec_passes_the_gateway_otlp_endpoint_to_the_supervisor() { + let sandbox = test_sandbox("test-id", "legit-name"); + let spec = build_container_spec(&sandbox, &test_config()); + assert!( + spec["env"] + .get(openshell_core::sandbox_env::OTLP_ENDPOINT) + .is_none() + ); + + let config = PodmanComputeConfig { + supervisor_otlp_endpoint: Some("http://127.0.0.1:4317".to_string()), + ..test_config() + }; + let spec = build_container_spec(&sandbox, &config); + assert_eq!( + spec["env"][openshell_core::sandbox_env::OTLP_ENDPOINT], + "http://127.0.0.1:4317" + ); + } + #[test] fn container_spec_telemetry_toggle_comes_from_driver_env() { use openshell_core::proto::compute::v1::{DriverSandboxSpec, DriverSandboxTemplate}; diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index d3023cddd8..0b5d0ddcd4 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -77,6 +77,36 @@ fn select_grpc_endpoint( ) } +/// Point a loopback collector endpoint at the desktop host on Podman Machine, +/// where supervisor loopback is the VM's. +fn select_supervisor_otlp_endpoint( + config: &PodmanComputeConfig, + environment: PodmanEndpointEnvironment, +) -> Option { + let endpoint = config.supervisor_otlp_endpoint.as_deref()?; + if environment == PodmanEndpointEnvironment::LinuxHost { + return Some(endpoint.to_string()); + } + let Ok(mut url) = url::Url::parse(endpoint) else { + return Some(endpoint.to_string()); + }; + let loopback = match url.host() { + Some(url::Host::Domain(domain)) => domain.eq_ignore_ascii_case("localhost"), + Some(url::Host::Ipv4(ip)) => ip.is_loopback(), + Some(url::Host::Ipv6(ip)) => ip.is_loopback(), + None => false, + }; + if !loopback || url.set_host(Some(environment.gateway_host())).is_err() { + return Some(endpoint.to_string()); + } + let rewritten = String::from(url); + if endpoint.ends_with('/') { + Some(rewritten) + } else { + Some(rewritten.trim_end_matches('/').to_string()) + } +} + fn decode_launch_authentication( encoded: &[u8], ) -> Result { @@ -110,8 +140,6 @@ impl From for ComputeDriverError { pub struct PodmanComputeDriver { client: PodmanClient, config: PodmanComputeConfig, - /// Whether Podman's service is running without root privileges. - rootless: bool, gpu_selector: Arc, gpu_inventory_refresh: Arc (CdiGpuInventory, bool) + Send + Sync>, lifecycle_event_fences: LifecycleEventFences, @@ -123,7 +151,6 @@ impl std::fmt::Debug for PodmanComputeDriver { .field("socket_path", &self.config.socket_path) .field("default_image", &self.config.default_image) .field("network_name", &self.config.network_name) - .field("rootless", &self.rootless) .field("gpu_inventory", &self.gpu_selector.device_ids()) .finish() } @@ -145,6 +172,12 @@ fn validated_container_name(sandbox: &DriverSandbox) -> Result Option<(u32, u32)> { + let (uid, gid) = user.split_once(':')?; + Some((uid.parse().ok()?, gid.parse().ok()?)) +} + fn podman_volume_is_bind_backed(volume: &VolumeInspect) -> bool { (volume.driver.is_empty() || volume.driver == "local") && volume.options.get("o").is_some_and(|options| { @@ -464,7 +497,7 @@ impl PodmanComputeDriver { } // Verify cgroups v2, detect rootless mode, and log system info. - let rootless = match client.system_info().await { + match client.system_info().await { Ok(info) => { if info.host.cgroup_version != "v2" { return Err(PodmanApiError::Connection(format!( @@ -486,14 +519,13 @@ impl PodmanComputeDriver { apparmor_enabled = info.host.security.apparmor_enabled, "Connected to Podman" ); - info.host.security.rootless } Err(e) => { return Err(PodmanApiError::Connection(format!( "failed to query Podman system info: {e}" ))); } - }; + } // Rootless pre-flight: warn if subuid/subgid ranges look missing. // Not a hard error because some systems configure these via LDAP or @@ -507,6 +539,8 @@ impl PodmanComputeDriver { // standard desktop-host alias. let endpoint_was_selected = config.grpc_endpoint.is_empty(); config.grpc_endpoint = select_grpc_endpoint(&config, PodmanEndpointEnvironment::current()); + config.supervisor_otlp_endpoint = + select_supervisor_otlp_endpoint(&config, PodmanEndpointEnvironment::current()); if endpoint_was_selected { info!( grpc_endpoint = %config.grpc_endpoint, @@ -529,7 +563,6 @@ impl PodmanComputeDriver { let driver = Self { client, config, - rootless, gpu_selector: Arc::new(CdiGpuDefaultSelector::new( gpu_inventory, allow_all_default_gpu, @@ -775,14 +808,22 @@ impl PodmanComputeDriver { if volume.name != name { return Err(missing()); } - if name == container::volume_name(sandbox_id) - || name == crate::isolation::channel_volume_name(sandbox_id) + let workspace_volume = name == container::volume_name(sandbox_id); + if workspace_volume || name == crate::isolation::channel_volume_name(sandbox_id) { let owned = volume.labels.as_ref().is_some_and(|labels| { labels.get(LABEL_SANDBOX_ID) == Some(sandbox_id) && labels.get(container::LABEL_SANDBOX_WORKSPACE) == Some(workspace) }); - if !owned || volume.driver != "local" || !volume.options.is_empty() { + // The channel volume has no options. The managed + // workspace is owned by the container's final identity, + // or has no options when an older gateway created it. + let options_ok = volume.options.is_empty() + || (workspace_volume + && numeric_user(&inspect.config.user).is_some_and(|owner| { + volume.options_match_requested_owner(Some(owner)) + })); + if !owned || volume.driver != "local" || !options_ok { return Err(missing()); } } else { @@ -1032,7 +1073,12 @@ impl PodmanComputeDriver { let phase_status = openshell_otel::ErrorStatusGuard::current(); let result = async { self.client - .create_owned_volume(&vol_name, &sandbox.id, &sandbox.workspace) + .create_owned_volume( + &vol_name, + &sandbox.id, + &sandbox.workspace, + Some((identity.uid, identity.gid)), + ) .await .map_err(ComputeDriverError::from)?; let resolver_secret_name = @@ -1160,7 +1206,6 @@ impl PodmanComputeDriver { supervisor_bin: supervisor_bin_path.as_deref(), tls_secrets: tls_secret_names.as_ref(), identity: &identity, - rootless: self.rootless, }); let mut specs = match specs { Ok(spec) => spec, @@ -1175,7 +1220,7 @@ impl PodmanComputeDriver { let identities = self.validate_user_volume_mounts_available(sandbox).await?; specs.record_resource_identities(&identities)?; self.client - .create_owned_volume(&channel_volume, &sandbox.id, &sandbox.workspace) + .create_owned_volume(&channel_volume, &sandbox.id, &sandbox.workspace, None) .await?; channel_owned.store(true, std::sync::atomic::Ordering::Relaxed); let workload_id = self.client.create_typed_container(&specs.workload).await?; @@ -1806,7 +1851,6 @@ impl PodmanComputeDriver { Self { client, config, - rootless: false, gpu_selector: Arc::new(CdiGpuDefaultSelector::new( gpu_inventory, allow_all_default_gpu, @@ -2691,6 +2735,67 @@ mod tests { ); } + fn otlp_config(endpoint: Option<&str>) -> PodmanComputeConfig { + PodmanComputeConfig { + supervisor_otlp_endpoint: endpoint.map(str::to_string), + ..PodmanComputeConfig::default() + } + } + + #[test] + fn supervisor_otlp_endpoint_stays_unset_without_gateway_export() { + assert_eq!( + select_supervisor_otlp_endpoint( + &otlp_config(None), + PodmanEndpointEnvironment::PodmanMachine + ), + None + ); + } + + #[test] + fn supervisor_otlp_endpoint_keeps_loopback_on_linux() { + assert_eq!( + select_supervisor_otlp_endpoint( + &otlp_config(Some("http://127.0.0.1:4317")), + PodmanEndpointEnvironment::LinuxHost + ) + .as_deref(), + Some("http://127.0.0.1:4317") + ); + } + + #[test] + fn supervisor_otlp_endpoint_uses_host_alias_for_loopback_on_podman_machine() { + for endpoint in [ + "http://127.0.0.1:4317", + "http://localhost:4317", + "http://[::1]:4317", + ] { + assert_eq!( + select_supervisor_otlp_endpoint( + &otlp_config(Some(endpoint)), + PodmanEndpointEnvironment::PodmanMachine + ) + .as_deref(), + Some("http://host.containers.internal:4317"), + "{endpoint}" + ); + } + } + + #[test] + fn supervisor_otlp_endpoint_keeps_routable_hosts_on_podman_machine() { + assert_eq!( + select_supervisor_otlp_endpoint( + &otlp_config(Some("https://collector.example.com:4317")), + PodmanEndpointEnvironment::PodmanMachine + ) + .as_deref(), + Some("https://collector.example.com:4317") + ); + } + #[test] fn ca_only_tls_config_is_enabled() { let cfg = PodmanComputeConfig { @@ -3022,7 +3127,7 @@ mod tests { assert!( driver .client - .create_owned_volume("private-collision", "sandbox-123", "team-a") + .create_owned_volume("private-collision", "sandbox-123", "team-a", None) .await .is_err() ); @@ -3079,6 +3184,80 @@ mod tests { } } + #[tokio::test] + async fn admission_accepts_workspace_volume_owned_by_workload_identity() { + let owned = serde_json::json!({"o":"uid=1234,gid=1235","UID":"1234","GID":"1235"}); + for (workspace_options, channel_options, allowed) in [ + (owned.clone(), serde_json::json!({}), true), + // Created by a gateway that did not set volume ownership. + (serde_json::json!({}), serde_json::json!({}), true), + ( + serde_json::json!({"o":"uid=0,gid=0","UID":"0","GID":"0"}), + serde_json::json!({}), + false, + ), + (owned.clone(), owned.clone(), false), + ] { + let sandbox_id = "sandbox-owned"; + let workspace_volume = container::volume_name(sandbox_id); + let channel_volume = crate::isolation::channel_volume_name(sandbox_id); + let volume = |name: &str, options: &serde_json::Value| { + StubResponse::new( + StatusCode::OK, + serde_json::json!({ + "Name": name, "Driver": "local", "Options": options, + "Labels": {LABEL_SANDBOX_ID: sandbox_id, container::LABEL_SANDBOX_WORKSPACE: "team-a"} + }) + .to_string(), + ) + }; + let container = serde_json::json!({ + "Id": "workload", "Name": "workload", "State": {"Status": "created", "Running": false}, + "Config": { + "User": "1234:1235", + "Labels": { + LABEL_SANDBOX_ID: sandbox_id, + container::LABEL_SANDBOX_WORKSPACE: "team-a", + openshell_core::resource_admission::CONFIG_USED_LABEL: "false", + openshell_core::resource_admission::IDENTITIES_LABEL: "{}", + } + }, + "Mounts": [ + {"Type": "volume", "Name": workspace_volume}, + {"Type": "volume", "Name": channel_volume}, + ] + }); + let (socket, _, handle) = spawn_podman_stub( + "admission-owned", + vec![ + StubResponse::new(StatusCode::OK, container.to_string()), + volume(&workspace_volume, &workspace_options), + volume(&channel_volume, &channel_options), + ], + ); + let driver = PodmanComputeDriver::for_tests(PodmanComputeConfig { + socket_path: Some(socket.clone()), + resource_admission: openshell_core::resource_admission::ResourceAdmissionConfig { + enabled: true, + ..Default::default() + }, + ..Default::default() + }); + let result = driver.admit_container_resources("workload").await; + assert_eq!( + result.is_ok(), + allowed, + "workspace {workspace_options}, channel {channel_options}: {result:?}" + ); + if allowed { + handle.await.unwrap(); + } else { + handle.abort(); + } + let _ = fs::remove_file(socket); + } + } + #[tokio::test] async fn admission_driver_config_denial_does_not_contact_podman() { for enabled in [true, false] { @@ -3538,7 +3717,11 @@ mod tests { image_response("sha256:supervisor"), StubResponse::new(StatusCode::NOT_FOUND, ""), // no existing private workspace StubResponse::new(StatusCode::CREATED, "{}"), // workspace volume - owned_volume_response(&container::volume_name(sandbox_id), sandbox_id), + owned_volume_response( + &container::volume_name(sandbox_id), + sandbox_id, + Some((1234, 1235)), // the stub image's OCI user + ), StubResponse::new(StatusCode::CREATED, "{}"), // resolver secret ]; if proxy_secret { @@ -3555,15 +3738,30 @@ mod tests { responses.push(owned_volume_response( &crate::isolation::channel_volume_name(sandbox_id), sandbox_id, + None, )); responses } - fn owned_volume_response(name: &str, sandbox_id: &str) -> StubResponse { + fn owned_volume_response( + name: &str, + sandbox_id: &str, + owner: Option<(u32, u32)>, + ) -> StubResponse { + let options = owner.map_or_else( + || serde_json::json!({}), + |(uid, gid)| { + serde_json::json!({ + "o": format!("uid={uid},gid={gid}"), + "UID": uid.to_string(), + "GID": gid.to_string(), + }) + }, + ); StubResponse::new( StatusCode::OK, serde_json::json!({ - "Name": name, "Driver": "local", "Options": {}, + "Name": name, "Driver": "local", "Options": options, "Labels": {LABEL_SANDBOX_ID: sandbox_id, container::LABEL_SANDBOX_WORKSPACE: ""} }) .to_string(), diff --git a/crates/openshell-driver-podman/src/main.rs b/crates/openshell-driver-podman/src/main.rs index 7468ae89a2..4fe83e0c8a 100644 --- a/crates/openshell-driver-podman/src/main.rs +++ b/crates/openshell-driver-podman/src/main.rs @@ -214,6 +214,7 @@ async fn main() -> Result<()> { default_image: args.sandbox_image.unwrap_or_default(), image_pull_policy: args.sandbox_image_pull_policy, grpc_endpoint: args.grpc_endpoint.unwrap_or_default(), + supervisor_otlp_endpoint: args.otlp_endpoint.clone(), gateway_port: args.gateway_port, host_gateway_ip: args .host_gateway_ip diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index 799cb0b6e8..0893fe16d3 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -254,6 +254,10 @@ pub struct VmDriverConfig { #[serde(default)] pub resource_admission: openshell_core::resource_admission::ResourceAdmissionConfig, pub grpc_endpoint: String, + /// OTLP/gRPC collector endpoint passed to supervisors. The gateway + /// supplies its own export endpoint; driver TOML cannot set it. + #[serde(skip)] + pub supervisor_otlp_endpoint: Option, pub state_dir: PathBuf, pub launcher_bin: Option, pub default_image: String, @@ -307,6 +311,7 @@ impl std::fmt::Debug for VmDriverConfig { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("VmDriverConfig") .field("grpc_endpoint", &self.grpc_endpoint) + .field("supervisor_otlp_endpoint", &self.supervisor_otlp_endpoint) .field("state_dir", &self.state_dir) .field("launcher_bin", &self.launcher_bin) .field("default_image", &self.default_image) @@ -370,6 +375,7 @@ impl Default for VmDriverConfig { fn default() -> Self { Self { grpc_endpoint: String::new(), + supervisor_otlp_endpoint: None, allow_driver_config: false, resource_admission: openshell_core::resource_admission::ResourceAdmissionConfig::default(), @@ -972,6 +978,11 @@ impl VmDriver { openshell_core::sandbox_env::TELEMETRY_ENABLED, openshell_core::telemetry::enabled_env_value(), ); + if let Some(endpoint) = &self.config.supervisor_otlp_endpoint { + command + .env(openshell_core::sandbox_env::OTLP_ENDPOINT, endpoint) + .envs(openshell_otel::current_trace_context_environment()); + } if let Some(server_name) = gateway_tls_server_name { command.env( openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME, diff --git a/crates/openshell-driver-vm/src/main.rs b/crates/openshell-driver-vm/src/main.rs index 4b4bcf3f6a..0fbaf7b2fe 100644 --- a/crates/openshell-driver-vm/src/main.rs +++ b/crates/openshell-driver-vm/src/main.rs @@ -297,6 +297,7 @@ async fn main() -> Result<()> { grpc_endpoint: args .grpc_endpoint .ok_or_else(|| miette::miette!("OPENSHELL_GRPC_ENDPOINT is required"))?, + supervisor_otlp_endpoint: args.otlp_endpoint.clone(), state_dir: args.state_dir.clone(), launcher_bin: None, default_image: args.default_image.clone(), diff --git a/crates/openshell-gateway/src/lib.rs b/crates/openshell-gateway/src/lib.rs index 03c043e36c..ad6d20a9a0 100644 --- a/crates/openshell-gateway/src/lib.rs +++ b/crates/openshell-gateway/src/lib.rs @@ -264,7 +264,8 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory { &self, context: openshell_server::ComputeDriverBuildContext<'_>, ) -> openshell_core::Result { - let config = kubernetes_config(context.config_context())?; + let mut config = kubernetes_config(context.config_context())?; + config.supervisor_otlp_endpoint = context.otlp_config().map(|otlp| otlp.endpoint.clone()); let driver = openshell_driver_kubernetes::KubernetesComputeDriver::new( config, context.shutdown_receiver(), @@ -325,6 +326,7 @@ impl openshell_server::ComputeDriverFactory for DockerFactory { context: openshell_server::ComputeDriverBuildContext<'_>, ) -> openshell_core::Result { let mut config = docker_config(context.config_context())?; + config.supervisor_otlp_endpoint = context.otlp_config().map(|otlp| otlp.endpoint.clone()); require_guest_tls_for_local_driver(&context, "docker")?; apply_guest_tls(&mut config.guest_tls_ca, context.guest_tls_ca()); let driver = openshell_driver_docker::DockerComputeDriver::new( @@ -386,6 +388,7 @@ impl openshell_server::ComputeDriverFactory for PodmanFactory { context: openshell_server::ComputeDriverBuildContext<'_>, ) -> openshell_core::Result { let mut config = podman_config(context.config_context())?; + config.supervisor_otlp_endpoint = context.otlp_config().map(|otlp| otlp.endpoint.clone()); require_guest_tls_for_local_driver(&context, "podman")?; apply_guest_tls(&mut config.guest_tls_ca, context.guest_tls_ca()); let driver = openshell_driver_podman::PodmanComputeDriver::new(config) diff --git a/crates/openshell-isolation-interface/Cargo.toml b/crates/openshell-isolation-interface/Cargo.toml index 0426a9011f..fe3e6f9551 100644 --- a/crates/openshell-isolation-interface/Cargo.toml +++ b/crates/openshell-isolation-interface/Cargo.toml @@ -22,7 +22,8 @@ tokio = { workspace = true } libc = "0.2" [target.'cfg(target_os = "linux")'.dependencies] -rustix = { workspace = true, features = ["fs", "process"] } +rustix = { workspace = true, features = ["fs", "net", "process"] } +socket2 = { workspace = true, features = ["all"] } [dev-dependencies] tokio = { workspace = true } diff --git a/crates/openshell-isolation-interface/src/linux/child_seccomp.rs b/crates/openshell-isolation-interface/src/linux/child_seccomp.rs index 4ba4eefafb..6fc9f67f73 100644 --- a/crates/openshell-isolation-interface/src/linux/child_seccomp.rs +++ b/crates/openshell-isolation-interface/src/linux/child_seccomp.rs @@ -29,6 +29,7 @@ const SECCOMP_DATA_ARGS_OFFSET: u32 = 16; const X32_SYSCALL_BIT: u32 = 0x4000_0000; const CLOSE_RANGE_UNSHARE_FLAG: u32 = 1 << 1; +const CLOSE_RANGE_CLOEXEC_FLAG: u32 = 1 << 2; const F_SETOWN_COMMAND: u32 = 8; const F_SETSIG_COMMAND: u32 = 10; const F_SETOWN_EX_COMMAND: u32 = 15; @@ -53,6 +54,7 @@ impl ChildHardeningProgram { /// The caller must invoke this from the post-fork child after all /// sandbox-wide TSYNC work and the launcher's `NEW_LISTENER` filter. pub fn install(&mut self) -> io::Result<()> { + mark_inherited_descriptors_close_on_exec()?; set_no_new_privileges()?; let len = u16::try_from(self.instructions.len()).map_err(|_| { io::Error::new( @@ -88,13 +90,43 @@ impl ChildHardeningProgram { } } +/// Mark every descriptor above stdio close-on-exec in the post-fork child. +/// +/// Workloads receive INET sockets only through broker injection, which binds +/// them to loopback first. A descriptor the sandbox process inherited from its +/// container runtime, or opened without `O_CLOEXEC`, must never cross `exec` +/// as an unconfined socket. The command's stdio is already installed on 0-2 +/// when `pre_exec` hooks run. This is a single async-signal-safe syscall. +/// +/// # Errors +/// +/// Returns the kernel error; kernels without `CLOSE_RANGE_CLOEXEC` (before +/// Linux 5.11) fail closed. +pub fn mark_inherited_descriptors_close_on_exec() -> io::Result<()> { + // SAFETY: close_range takes scalar arguments and only sets FD_CLOEXEC. + let result = unsafe { + libc::syscall( + libc::SYS_close_range, + 3_u32, + u32::MAX, + CLOSE_RANGE_CLOEXEC_FLAG, + ) + }; + if result < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(()) + } +} + /// Build the same-UID workload self-protection program before `fork`. /// /// `sandbox_tgid` is the sandbox PID as visible from its workload namespace. /// The filter blocks thread-targeting operations that name the trusted sandbox /// leader and blocks process-directed operations with the same target. The -/// ordinary workload listener additionally mediates `kill`, `tkill`, and -/// `rt_sigqueueinfo`: Linux accepts nonleader TIDs for these operations, so a +/// ordinary workload listener additionally mediates `kill`, `tkill`, +/// `rt_sigqueueinfo`, and `SIGCONT` sent with `tgkill` or +/// `rt_tgsigqueueinfo`: Linux accepts nonleader TIDs for these operations, so a /// static TGID comparison alone cannot protect future sandbox worker threads. pub fn prepare(sandbox_tgid: u32) -> io::Result { if sandbox_tgid == 0 { @@ -352,6 +384,48 @@ fn set_no_new_privileges() -> io::Result<()> { mod tests { use super::*; + #[test] + fn inherited_sockets_are_marked_close_on_exec_but_stdio_is_not() { + // The sweep changes every descriptor in the calling process, so run + // it in a fresh copy of this test binary rather than the harness. + const CHILD_MARKER: &str = "OPENSHELL_CLOEXEC_SWEEP_CHILD"; + if std::env::var_os(CHILD_MARKER).is_some() { + let socket = rustix::net::socket( + rustix::net::AddressFamily::INET, + rustix::net::SocketType::STREAM, + None, + ) + .expect("inheritable socket"); + assert!( + !rustix::io::fcntl_getfd(&socket) + .unwrap() + .contains(rustix::io::FdFlags::CLOEXEC) + ); + mark_inherited_descriptors_close_on_exec().expect("sweep descriptors"); + assert!( + rustix::io::fcntl_getfd(&socket) + .unwrap() + .contains(rustix::io::FdFlags::CLOEXEC) + ); + assert!( + !rustix::io::fcntl_getfd(io::stderr()) + .unwrap() + .contains(rustix::io::FdFlags::CLOEXEC) + ); + return; + } + let status = std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "linux::child_seccomp::tests::inherited_sockets_are_marked_close_on_exec_but_stdio_is_not", + "--nocapture", + ]) + .env(CHILD_MARKER, "1") + .status() + .expect("run isolated sweep test"); + assert!(status.success(), "isolated sweep test failed"); + } + #[test] fn rejects_zero_sandbox_tgid() { assert_eq!( diff --git a/crates/openshell-isolation-interface/src/linux/mod.rs b/crates/openshell-isolation-interface/src/linux/mod.rs index bad3d329dc..a15b8b6f43 100644 --- a/crates/openshell-isolation-interface/src/linux/mod.rs +++ b/crates/openshell-isolation-interface/src/linux/mod.rs @@ -11,6 +11,7 @@ pub mod landlock; pub mod proc_fd; pub mod process_signal; pub mod seccomp_notify; +pub mod socket_confinement; pub mod socket_registry; pub mod task_memory; pub mod workload_launcher; diff --git a/crates/openshell-isolation-interface/src/linux/process_signal.rs b/crates/openshell-isolation-interface/src/linux/process_signal.rs index 796f548037..f2addd54dc 100644 --- a/crates/openshell-isolation-interface/src/linux/process_signal.rs +++ b/crates/openshell-isolation-interface/src/linux/process_signal.rs @@ -12,6 +12,7 @@ use std::io; use std::os::fd::{AsRawFd, FromRawFd, OwnedFd}; +use std::sync::atomic::{AtomicBool, Ordering}; use crate::linux::seccomp_notify::{Notification, NotificationListener}; use crate::linux::task_memory; @@ -27,6 +28,7 @@ pub fn mediate_process_signal( listener: &NotificationListener, notification: Notification, sandbox_tgid: u32, + workload_frozen: &AtomicBool, ) -> io::Result<()> { listener.validate_id(notification.id)?; let target = scalar_int(notification.args[0]); @@ -38,6 +40,7 @@ pub fn mediate_process_signal( libc::EINVAL })); } + refuse_resume_while_frozen(signal, workload_frozen)?; let target = u32::try_from(target).map_err(|_| io::Error::from_raw_os_error(libc::ESRCH))?; let retained = retain_signal_target(target, sandbox_tgid)?; // SAFETY: all-zero siginfo consists of valid integer/pointer fields. A @@ -64,6 +67,7 @@ pub fn mediate_process_signal( _ => return Err(io::Error::from_raw_os_error(libc::ENOSYS)), }; listener.validate_id(notification.id)?; + refuse_resume_while_frozen(signal, workload_frozen)?; // SAFETY: retained owns a live pidfd; info is null or a complete trusted // copy. The kernel targets that process object, never a reused numeric PID. let result = unsafe { @@ -81,37 +85,62 @@ pub fn mediate_process_signal( listener.respond_value(notification.id, 0) } -/// Continue a positive-target `tkill` only when the target thread belongs to -/// an untrusted workload process rather than the sandbox runtime itself. +/// Continue a thread-directed signal (`tkill`, `tgkill`, +/// `rt_tgsigqueueinfo`) aimed at an untrusted workload thread. /// /// Continuing preserves Linux's thread-directed signal semantics, including -/// the cancellation signal used by musl. A target that exits between the -/// ownership check and continuation can only be reused inside the same PID -/// namespace; the static child filter still rejects the sandbox leader. +/// the cancellation signal used by musl. The static child filter rejects the +/// sandbox leader as a `tgkill`/`rt_tgsigqueueinfo` group, and the kernel +/// rejects a thread outside the named group, so those two are notified only +/// for `SIGCONT`. A `tkill` names a bare thread, so its group is resolved here; +/// a target reused between this check and continuation stays inside the same +/// PID namespace. pub fn mediate_thread_signal( listener: &NotificationListener, notification: Notification, sandbox_tgid: u32, + workload_frozen: &AtomicBool, ) -> io::Result<()> { listener.validate_id(notification.id)?; - let target = scalar_int(notification.args[0]); - let signal = scalar_int(notification.args[1]); - if target <= 0 || !(0..=64).contains(&signal) { - return Err(io::Error::from_raw_os_error(if target <= 0 { - libc::EPERM - } else { - libc::EINVAL - })); - } - let target = u32::try_from(target).map_err(|_| io::Error::from_raw_os_error(libc::ESRCH))?; - let target_group = thread_group_id(target)?; - if target_group == sandbox_tgid || target_group == 0 { - return Err(io::Error::from_raw_os_error(libc::EPERM)); + let signal = match i64::from(notification.syscall) { + libc::SYS_tkill => { + let target = scalar_int(notification.args[0]); + let signal = scalar_int(notification.args[1]); + if target <= 0 { + return Err(io::Error::from_raw_os_error(libc::EPERM)); + } + let target = + u32::try_from(target).map_err(|_| io::Error::from_raw_os_error(libc::ESRCH))?; + let target_group = thread_group_id(target)?; + if target_group == sandbox_tgid || target_group == 0 { + return Err(io::Error::from_raw_os_error(libc::EPERM)); + } + signal + } + libc::SYS_tgkill | libc::SYS_rt_tgsigqueueinfo => scalar_int(notification.args[2]), + _ => return Err(io::Error::from_raw_os_error(libc::ENOSYS)), + }; + if !(0..=64).contains(&signal) { + return Err(io::Error::from_raw_os_error(libc::EINVAL)); } listener.validate_id(notification.id)?; + refuse_resume_while_frozen(signal, workload_frozen)?; listener.respond_continue(notification.id) } +/// While the boundary has stopped the workload for supervisor recovery, a +/// workload process that was not yet stopped must not resume the others. +/// +/// The flag is read immediately before delivery. The freezer does not wait +/// for in-flight notifications, so a signal already past this check when the +/// freeze begins can still be delivered. +fn refuse_resume_while_frozen(signal: i32, workload_frozen: &AtomicBool) -> io::Result<()> { + if signal == libc::SIGCONT && workload_frozen.load(Ordering::Acquire) { + return Err(io::Error::from_raw_os_error(libc::EPERM)); + } + Ok(()) +} + fn scalar_int(value: u64) -> i32 { let bytes = value.to_ne_bytes(); #[cfg(target_endian = "little")] @@ -176,8 +205,13 @@ mod tests { .recv_timeout(std::time::Duration::from_secs(5)) .unwrap(); let notification = listener.receive().unwrap(); - let error = - mediate_process_signal(&listener, notification, std::process::id()).unwrap_err(); + let error = mediate_process_signal( + &listener, + notification, + std::process::id(), + &AtomicBool::new(false), + ) + .unwrap_err(); assert_eq!(error.raw_os_error(), Some(libc::EPERM)); listener .respond_errno(notification.id, libc::EPERM) diff --git a/crates/openshell-isolation-interface/src/linux/seccomp_notify.rs b/crates/openshell-isolation-interface/src/linux/seccomp_notify.rs index a87dc080d5..5ae24cd27d 100644 --- a/crates/openshell-isolation-interface/src/linux/seccomp_notify.rs +++ b/crates/openshell-isolation-interface/src/linux/seccomp_notify.rs @@ -20,7 +20,6 @@ use std::time::Duration; const SECCOMP_SET_MODE_FILTER: libc::c_uint = 1; const SECCOMP_GET_NOTIF_SIZES: libc::c_uint = 3; const SECCOMP_FILTER_FLAG_NEW_LISTENER: libc::c_ulong = 1 << 3; -const SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV: libc::c_ulong = 1 << 5; const SECCOMP_RET_KILL_PROCESS: u32 = 0x8000_0000; const SECCOMP_RET_USER_NOTIF: u32 = 0x7fc0_0000; @@ -141,8 +140,6 @@ pub struct Notification { /// kernel, outer seccomp profile, and LSM posture. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct NotificationProbeReport { - /// Whether `SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` was accepted. - pub wait_killable_recv: bool, features: NotificationProbeFeatures, } @@ -170,81 +167,24 @@ impl NotificationProbeReport { } } -/// Cancellation posture a listener was installed with. -/// -/// `WAIT_KILLABLE_RECV` (Linux 5.19+) keeps the notified workload thread in a -/// kill-only wait so a non-fatal signal cannot resume the mediated syscall -/// after the broker has validated the notification. A plain listener has no -/// such guarantee, so it runs read-only: the broker must refuse every -/// task-memory *output* write to stay cancellation-safe. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum ListenerMode { - /// Modern kernel: `WAIT_KILLABLE_RECV` active; full mediation including - /// task-memory output writes. - Killable, - /// Legacy kernel (< 5.19): plain listener; task-memory output writes are - /// disabled so a resumed syscall cannot race a broker write. - LegacyReadOnly, -} - -impl ListenerMode { - /// Stable identifier for qualification output and diagnostics. - #[must_use] - pub fn as_str(self) -> &'static str { - match self { - Self::Killable => "killable", - Self::LegacyReadOnly => "legacy_read_only", - } - } -} - /// Owned listener returned by `SECCOMP_FILTER_FLAG_NEW_LISTENER`. +/// +/// The listener is installed without `WAIT_KILLABLE_RECV`, so mediation is the +/// same on every kernel. A signal can interrupt a notified syscall, which the +/// kernel then restarts or fails with `EINTR`; broker handlers check the +/// notification is still live before acting and answer a repeated operation +/// as the kernel would. pub struct NotificationListener { fd: OwnedFd, - wait_killable_recv: bool, } impl NotificationListener { - /// Construct a listener from an already-owned notification descriptor in a - /// specific mode. Intended for tests that must exercise the legacy - /// read-only fail-closed paths without a `< 5.19` kernel. - #[must_use] - pub fn from_fd_with_mode(fd: OwnedFd, mode: ListenerMode) -> Self { - Self { - fd, - wait_killable_recv: matches!(mode, ListenerMode::Killable), - } - } - /// Raw listener descriptor for readiness integration and diagnostics. #[must_use] pub fn as_raw_fd(&self) -> RawFd { self.fd.as_raw_fd() } - /// Whether the listener was installed with killable receive waits. - #[must_use] - pub fn wait_killable_recv(&self) -> bool { - self.wait_killable_recv - } - - /// The cancellation mode this listener was installed with. - #[must_use] - pub fn mode(&self) -> ListenerMode { - if self.wait_killable_recv { - ListenerMode::Killable - } else { - ListenerMode::LegacyReadOnly - } - } - - /// Whether broker task-memory output writes are disabled for this listener. - /// True exactly in `LegacyReadOnly` mode (no `WAIT_KILLABLE_RECV`). - #[must_use] - pub fn writes_disabled(&self) -> bool { - matches!(self.mode(), ListenerMode::LegacyReadOnly) - } - /// Receive the next kernel notification. pub fn receive(&self) -> io::Result { let mut raw = RawNotification::default(); @@ -272,34 +212,6 @@ impl NotificationListener { Ok(()) } - /// Write broker-produced output into the notified task's memory, closing - /// the validation-to-write race that a plain listener cannot. - /// - /// In `Killable` mode `WAIT_KILLABLE_RECV` keeps the notified workload - /// thread in a kill-only wait, so a non-fatal signal cannot resume the - /// mediated syscall between `validate_id` and this write. In - /// `LegacyReadOnly` mode (kernels < 5.19) there is no such guarantee: a - /// resumed syscall could repurpose the target buffer while the privileged - /// broker writes through the captured tid and pointer β€” via `/proc//mem` - /// even into pages the workload has since made read-only. There is no way to - /// close that window without the flag, so this fails closed (`EOPNOTSUPP`) - /// rather than racing. Callers must route every task-memory *output* write - /// through this method; input reads never write workload memory and are - /// unaffected. - pub fn write_task_output( - &self, - id: u64, - tid: u32, - address: u64, - data: &[u8], - ) -> io::Result<()> { - if self.writes_disabled() { - return Err(io::Error::from_raw_os_error(libc::EOPNOTSUPP)); - } - self.validate_id(id)?; - crate::linux::task_memory::write_exact(tid, address, data) - } - /// Return a successful scalar result to the notifying syscall. pub fn respond_value(&self, id: u64, value: i64) -> io::Result<()> { self.validate_id(id)?; @@ -407,22 +319,7 @@ pub fn install_listener(syscalls: &[i64]) -> io::Result { verify_notification_sizes()?; set_no_new_privileges()?; - // WAIT_KILLABLE_RECV (Linux 5.19+) keeps the *notified workload thread* in - // a kill-only wait while the broker services its syscall, so a non-fatal - // signal cannot resume the syscall and repurpose its buffers underneath a - // pending broker write. Kernels older than 5.19 (for example RHEL 9.x / - // 5.14 nodes) reject the flag with EINVAL. Rather than refusing to start - // there, fall back to a plain listener so the sandbox boots; the resulting - // listener records `wait_killable_recv = false`, and the broker then fails - // closed on every task-memory output write (see `write_task_output`) - // instead of racing them. Input mediation is unaffected. - match install_listener_with_flags(syscalls, true) { - Ok(listener) => Ok(listener), - Err(error) if error.raw_os_error() == Some(libc::EINVAL) => { - install_listener_with_flags(syscalls, false) - } - Err(error) => Err(error), - } + install_notification_filter(syscalls) } /// Install the capability-free workload networking listener on the calling @@ -430,7 +327,9 @@ pub fn install_listener(syscalls: &[i64]) -> io::Result { /// /// The filter mediates every syscall that can create, select, or materially /// reconfigure an INET endpoint. Connected `send()`/null-destination -/// `sendto()` retains the audited cBPF fast path. +/// `sendto()` retains the audited cBPF fast path. `accept`/`accept4` run +/// natively: the broker binds every INET socket to loopback before injecting +/// it, and accepted sockets inherit their listener's binding. pub fn install_workload_listener() -> io::Result { #[allow(unused_mut)] // SYS_open is unavailable on some architectures. let mut syscalls = vec![ @@ -438,15 +337,14 @@ pub fn install_workload_listener() -> io::Result { libc::SYS_connect, libc::SYS_bind, libc::SYS_listen, - libc::SYS_accept, - libc::SYS_accept4, libc::SYS_sendto, libc::SYS_sendmsg, libc::SYS_sendmmsg, - libc::SYS_getpeername, libc::SYS_setsockopt, libc::SYS_kill, libc::SYS_tkill, + libc::SYS_tgkill, + libc::SYS_rt_tgsigqueueinfo, libc::SYS_rt_sigqueueinfo, libc::SYS_openat, libc::SYS_openat2, @@ -462,30 +360,28 @@ pub fn install_workload_listener() -> io::Result { /// one dedicated thread and moved to an unfiltered broker thread through an /// in-process channel. pub fn probe_notification_api() -> io::Result { - let wait_killable_recv = probe_scalar_round_trip()?; + probe_scalar_round_trip()?; probe_addfd_send()?; probe_connected_sendto_fast_path()?; Ok(NotificationProbeReport { - wait_killable_recv, features: NotificationProbeFeatures(1 | 2 | 4), }) } -fn probe_scalar_round_trip() -> io::Result { +fn probe_scalar_round_trip() -> io::Result<()> { const PROBE_VALUE: libc::c_long = 0x5a17; let (sender, receiver) = mpsc::sync_channel(1); let launcher = thread::spawn(move || -> io::Result { let listener = install_listener(&[libc::SYS_getppid])?; - let wait_killable = listener.wait_killable_recv(); sender - .send((listener, wait_killable)) + .send(listener) .map_err(|_| io::Error::other("notification broker disappeared"))?; // SAFETY: getppid has no pointer arguments. The installed filter causes // the kernel to block here until the broker validates and responds. Ok(unsafe { libc::syscall(libc::SYS_getppid) }) }); - let (listener, wait_killable) = receiver + let listener = receiver .recv() .map_err(|_| io::Error::other("notification launcher disappeared"))?; let notification = match receive_probe_notification(&listener) { @@ -505,7 +401,7 @@ fn probe_scalar_round_trip() -> io::Result { if observed != PROBE_VALUE { return Err(io::Error::other("seccomp response value was not delivered")); } - Ok(wait_killable) + Ok(()) } fn probe_addfd_send() -> io::Result<()> { @@ -759,10 +655,7 @@ fn receive_probe_notification(listener: &NotificationListener) -> io::Result io::Result { +fn install_notification_filter(syscalls: &[i64]) -> io::Result { let mut program = build_filter(syscalls)?; let length = u16::try_from(program.len()) .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "seccomp filter is too large"))?; @@ -770,12 +663,7 @@ fn install_listener_with_flags( len: length, filter: program.as_mut_ptr(), }; - let flags = SECCOMP_FILTER_FLAG_NEW_LISTENER - | if wait_killable_recv { - SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV - } else { - 0 - }; + let flags = SECCOMP_FILTER_FLAG_NEW_LISTENER; // SAFETY: `fprog` points to a live classic-BPF program for the duration of // the syscall. The returned nonnegative value is a newly owned FD. let result = unsafe { @@ -793,10 +681,7 @@ fn install_listener_with_flags( .map_err(|_| io::Error::other("seccomp listener FD does not fit RawFd"))?; // SAFETY: successful NEW_LISTENER returns one newly owned descriptor. let fd = unsafe { OwnedFd::from_raw_fd(fd) }; - Ok(NotificationListener { - fd, - wait_killable_recv, - }) + Ok(NotificationListener { fd }) } fn build_filter(syscalls: &[i64]) -> io::Result> { @@ -821,6 +706,10 @@ fn build_filter(syscalls: &[i64]) -> io::Result> { append_sendto_filter(&mut program)?; continue; } + if matches!(syscall, libc::SYS_tgkill | libc::SYS_rt_tgsigqueueinfo) { + append_resume_signal_filter(&mut program, syscall)?; + continue; + } let syscall = u32::try_from(syscall) .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "negative syscall number"))?; program.extend([ @@ -861,6 +750,27 @@ fn append_sendto_filter(program: &mut Vec) -> io::Result<()> Ok(()) } +/// Notify a thread-group signal only when it sends `SIGCONT`, which the broker +/// refuses while the workload is frozen. Every other signal (for example Go's +/// preemption signal) stays in the kernel. +fn append_resume_signal_filter( + program: &mut Vec, + syscall: i64, +) -> io::Result<()> { + let syscall = u32::try_from(syscall) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "negative syscall number"))?; + let resume = u32::try_from(libc::SIGCONT) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "negative signal number"))?; + program.extend([ + jump(BPF_JMP_JEQ_K, syscall, 0, 4), + stmt(BPF_LD_W_ABS, argument_word_offset(2, 0)), + jump(BPF_JMP_JEQ_K, resume, 0, 1), + stmt(BPF_RET_K, SECCOMP_RET_USER_NOTIF), + stmt(BPF_RET_K, SECCOMP_RET_ALLOW), + ]); + Ok(()) +} + const fn argument_word_offset(argument: u32, word: u32) -> u32 { SECCOMP_DATA_ARGS_OFFSET + argument * 8 + word * 4 } @@ -1077,59 +987,10 @@ mod tests { let listener = NotificationListener { // SAFETY: successful dup returned a new owned descriptor. fd: unsafe { OwnedFd::from_raw_fd(duplicated) }, - wait_killable_recv: false, }; let error = listener .respond_errno(1, 0) .expect_err("zero errno must fail"); assert_eq!(error.kind(), io::ErrorKind::InvalidInput); } - - #[test] - fn plain_listener_fails_closed_on_output_write() { - // A LegacyReadOnly listener (kernels < 5.19) must refuse every - // task-memory output write rather than race a resumed syscall. The - // guard short-circuits before touching the descriptor or workload - // memory, so a dup of stderr is a sufficient stand-in. - // SAFETY: dup takes one valid descriptor and returns a new descriptor - // or a negative error without modifying memory. - let duplicated = unsafe { libc::dup(libc::STDERR_FILENO) }; - assert!(duplicated >= 0, "duplicate stderr for validation test"); - // SAFETY: successful dup returned a new owned descriptor. - let listener = NotificationListener::from_fd_with_mode( - unsafe { OwnedFd::from_raw_fd(duplicated) }, - ListenerMode::LegacyReadOnly, - ); - assert!(listener.writes_disabled()); - assert_eq!(listener.mode(), ListenerMode::LegacyReadOnly); - let error = listener - .write_task_output(1, 0, 0, &[0_u8; 4]) - .expect_err("plain listener must reject output writes"); - assert_eq!(error.raw_os_error(), Some(libc::EOPNOTSUPP)); - } - - #[test] - fn real_plain_listener_disables_output_writes() { - // Deliberately install a plain NEW_LISTENER (no WAIT_KILLABLE_RECV) - // even on a modern CI kernel and prove the broker write path fails - // closed on the real listener object. - set_no_new_privileges().expect("no_new_privs for listener install"); - let listener = install_listener_with_flags(&[libc::SYS_getppid], false) - .expect("install plain listener"); - assert_eq!(listener.mode(), ListenerMode::LegacyReadOnly); - assert!(listener.writes_disabled()); - let error = listener - .write_task_output(1, 0, 0, &[0_u8; 4]) - .expect_err("plain listener must reject output writes"); - assert_eq!(error.raw_os_error(), Some(libc::EOPNOTSUPP)); - } - - #[test] - fn killable_listener_enables_output_writes() { - set_no_new_privileges().expect("no_new_privs for listener install"); - let listener = install_listener_with_flags(&[libc::SYS_getppid], true) - .expect("install killable listener"); - assert_eq!(listener.mode(), ListenerMode::Killable); - assert!(!listener.writes_disabled()); - } } diff --git a/crates/openshell-isolation-interface/src/linux/socket_confinement.rs b/crates/openshell-isolation-interface/src/linux/socket_confinement.rs new file mode 100644 index 0000000000..715ce4ca9b --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/socket_confinement.rs @@ -0,0 +1,293 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Standing kernel confinement for workload INET sockets. +//! +//! Every workload INET socket is bound to the loopback device before its +//! descriptor is injected. The binding is kernel state on the socket itself, +//! so it survives `dup`, `fork`, `exec`, `AF_UNSPEC` disconnect, and is +//! inherited by sockets accepted from a confined listener. It restricts both +//! directions: route lookups are pinned to `lo`, and listener/UDP lookup only +//! matches packets that arrive on `lo`. Clearing or changing an existing +//! binding requires `CAP_NET_RAW` in the network namespace's owning user +//! namespace, which the capability-free sandbox and workload do not hold. + +use std::io; +use std::os::fd::AsFd; + +use socket2::{Domain, SockFilter, SockRef, Socket, Type}; + +const LOOPBACK_DEVICE: &[u8] = b"lo"; + +/// Bind `fd` to the loopback device. +/// +/// # Errors +/// +/// Returns the kernel error, including `EPERM` when the socket is already +/// bound to a device. +pub fn confine_to_loopback(fd: impl AsFd) -> io::Result<()> { + SockRef::from(&fd).bind_device(Some(LOOPBACK_DEVICE)) +} + +/// Return the device name `fd` is bound to, or `None` when unbound. +/// +/// # Errors +/// +/// Returns the kernel error from `getsockopt(SO_BINDTODEVICE)`. +pub fn bound_device(fd: impl AsFd) -> io::Result>> { + SockRef::from(&fd).device() +} + +/// Drop TCP/UDP ingress that arrives on the loopback interface. +/// +/// Attach this to a trusted listener whose legitimate clients are never in the +/// same network namespace. Matching the ingress interface rather than the +/// source address also rejects connections to the host's own non-loopback +/// address, which the kernel delivers through loopback. The filter is not +/// locked: the listener descriptor never leaves the trusted sandbox process, +/// which marks every descriptor above stdio close-on-exec before running +/// workload code. +/// +/// # Errors +/// +/// Returns the kernel error when the interface index cannot be resolved or +/// the filter cannot be attached. +pub fn reject_loopback_ingress(fd: impl AsFd) -> io::Result<()> { + let index = rustix::net::netdevice::name_to_index(&fd, "lo")?; + reject_ingress_interface(fd, index) +} + +fn reject_ingress_interface(fd: impl AsFd, index: u32) -> io::Result<()> { + // Ancillary loads use the documented negative offset encoding. + let ifindex_offset = (libc::SKF_AD_OFF + libc::SKF_AD_IFINDEX).cast_unsigned(); + let program = [ + filter( + libc::BPF_LD | libc::BPF_W | libc::BPF_ABS, + 0, + 0, + ifindex_offset, + ), + filter(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 0, 1, index), + filter(libc::BPF_RET | libc::BPF_K, 0, 0, 0), + filter(libc::BPF_RET | libc::BPF_K, 0, 0, u32::MAX), + ]; + SockRef::from(&fd).attach_filter(&program) +} + +#[allow( + clippy::cast_possible_truncation, + reason = "classic BPF opcodes are 16-bit by definition" +)] +const fn filter(code: u32, jt: u8, jf: u8, k: u32) -> SockFilter { + SockFilter::new(code as u16, jt, jf, k) +} + +/// Actively prove loopback confinement under the current runtime profile. +/// +/// For each supported workload socket type this installs the binding and +/// proves that the sandbox credentials cannot clear or replace it. For IPv4 +/// and IPv6 it proves that a stream accepted from a confined listener inherits +/// the binding and keeps it after an `AF_UNSPEC` disconnect. IPv6 is skipped +/// only when the kernel or namespace does not provide it. +/// +/// # Errors +/// +/// Returns an error describing the first failed property. +pub fn probe_loopback_confinement() -> io::Result<()> { + for (domain, kind) in [ + (Domain::IPV4, Type::STREAM), + (Domain::IPV4, Type::DGRAM), + (Domain::IPV6, Type::STREAM), + (Domain::IPV6, Type::DGRAM), + ] { + let socket = match Socket::new(domain, kind, None) { + Ok(socket) => socket, + Err(error) + if domain == Domain::IPV6 && error.raw_os_error() == Some(libc::EAFNOSUPPORT) => + { + continue; + } + Err(error) => return Err(error), + }; + confine_to_loopback(&socket) + .map_err(|error| probe_error("install loopback binding", &error))?; + probe_binding_is_immutable(&socket)?; + } + probe_accept_inherits_binding() +} + +fn probe_binding_is_immutable(socket: &Socket) -> io::Result<()> { + // `None` requests an unbind; replacing the device takes the same path. + if socket.bind_device(None).is_ok() { + return Err(io::Error::other( + "sandbox credentials can clear a socket device binding", + )); + } + if socket.device()?.as_deref() != Some(LOOPBACK_DEVICE) { + return Err(io::Error::other("socket device binding changed")); + } + Ok(()) +} + +fn probe_accept_inherits_binding() -> io::Result<()> { + for loopback in [ + std::net::IpAddr::V4(std::net::Ipv4Addr::LOCALHOST), + std::net::IpAddr::V6(std::net::Ipv6Addr::LOCALHOST), + ] { + let listener = match std::net::TcpListener::bind((loopback, 0)) { + Ok(listener) => listener, + // Kernels or namespaces without IPv6 have no ::1 to bind. + Err(error) + if loopback.is_ipv6() + && matches!( + error.raw_os_error(), + Some(libc::EAFNOSUPPORT | libc::EADDRNOTAVAIL) + ) => + { + continue; + } + Err(error) => return Err(error), + }; + confine_to_loopback(&listener) + .map_err(|error| probe_error("confine probe listener", &error))?; + let _client = std::net::TcpStream::connect(listener.local_addr()?)?; + let (accepted, _) = listener.accept()?; + if bound_device(&accepted)?.as_deref() != Some(LOOPBACK_DEVICE) { + return Err(io::Error::other( + "accepted socket did not inherit the loopback binding", + )); + } + // Natively accepted sockets are not tracked by the broker, so a + // workload can disconnect and reconnect them. The binding must + // survive that transition. + rustix::net::connect_unspec(&accepted) + .map_err(|error| probe_error("disconnect accepted probe socket", &error.into()))?; + if bound_device(&accepted)?.as_deref() != Some(LOOPBACK_DEVICE) { + return Err(io::Error::other( + "accepted socket lost the loopback binding after disconnect", + )); + } + } + Ok(()) +} + +fn probe_error(context: &str, error: &io::Error) -> io::Error { + io::Error::new(error.kind(), format!("{context}: {error}")) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{Read as _, Write as _}; + use std::net::{Ipv4Addr, SocketAddr, TcpListener, TcpStream}; + use std::time::Duration; + + fn new_socket(domain: Domain, kind: Type) -> Socket { + Socket::new(domain, kind, None).unwrap() + } + + #[test] + fn active_probe_passes_without_capabilities() { + // Root holds CAP_NET_RAW, which may change a device binding. + if rustix::process::geteuid().is_root() { + return; + } + probe_loopback_confinement().expect("loopback confinement probe"); + } + + #[test] + fn unbound_socket_reports_no_device() { + let socket = new_socket(Domain::IPV4, Type::STREAM); + assert_eq!(bound_device(&socket).unwrap(), None); + } + + #[test] + fn confined_socket_cannot_be_rebound() { + if rustix::process::geteuid().is_root() { + return; + } + let socket = new_socket(Domain::IPV4, Type::DGRAM); + confine_to_loopback(&socket).unwrap(); + assert!(confine_to_loopback(&socket).is_err()); + assert_eq!(bound_device(&socket).unwrap().as_deref(), Some(&b"lo"[..])); + } + + fn connect_with_timeout(address: SocketAddr) -> io::Result { + TcpStream::connect_timeout(&address, Duration::from_millis(300)) + } + + #[test] + fn loopback_ingress_filter_rejects_loopback_connections() { + let listener = TcpListener::bind((Ipv4Addr::UNSPECIFIED, 0)).unwrap(); + reject_loopback_ingress(&listener).unwrap(); + listener.set_nonblocking(true).unwrap(); + let port = listener.local_addr().unwrap().port(); + // Dropped SYNs never complete the handshake. + assert!(connect_with_timeout(SocketAddr::from((Ipv4Addr::LOCALHOST, port))).is_err()); + assert_eq!( + listener.accept().unwrap_err().kind(), + io::ErrorKind::WouldBlock + ); + } + + #[test] + fn ingress_filter_admits_other_interfaces() { + // Positive control: the same program keyed to an absent interface + // index must leave loopback traffic untouched. + let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).unwrap(); + reject_ingress_interface(&listener, u32::MAX).unwrap(); + let mut client = connect_with_timeout(listener.local_addr().unwrap()).unwrap(); + let (mut accepted, _) = listener.accept().unwrap(); + client.write_all(b"ping").unwrap(); + let mut buffer = [0_u8; 4]; + accepted.read_exact(&mut buffer).unwrap(); + assert_eq!(&buffer, b"ping"); + } + + /// Return a local non-loopback address, if this namespace has one. + fn local_non_loopback_address() -> Option { + let probe = std::net::UdpSocket::bind((Ipv4Addr::UNSPECIFIED, 0)).ok()?; + probe.connect((Ipv4Addr::new(192, 0, 2, 1), 9)).ok()?; + match probe.local_addr().ok()?.ip() { + std::net::IpAddr::V4(address) if !address.is_loopback() => Some(address), + _ => None, + } + } + + #[test] + fn confined_listener_peers_are_limited_to_this_network_namespace() { + // Only a same-namespace client can reach a loopback-bound listener; + // connecting to the host's own address is refused. + let Some(local) = local_non_loopback_address() else { + eprintln!("skipping: no non-loopback IPv4 address in this namespace"); + return; + }; + let listener = TcpListener::bind((Ipv4Addr::UNSPECIFIED, 0)).unwrap(); + confine_to_loopback(&listener).unwrap(); + listener.set_nonblocking(true).unwrap(); + let port = listener.local_addr().unwrap().port(); + + let connect_from = |source: Ipv4Addr, destination: Ipv4Addr| { + let client = new_socket(Domain::IPV4, Type::STREAM); + client.bind(&SocketAddr::from((source, 0)).into()).unwrap(); + client + .connect_timeout( + &SocketAddr::from((destination, port)).into(), + Duration::from_millis(300), + ) + .map(|()| client) + }; + + // The host's own address is matched against its real interface. + assert!(connect_from(local, local).is_err()); + assert_eq!( + listener.accept().unwrap_err().kind(), + io::ErrorKind::WouldBlock + ); + + let client = connect_from(local, Ipv4Addr::LOCALHOST).expect("same-namespace client"); + let (_, peer) = listener.accept().unwrap(); + assert_eq!(peer, client.local_addr().unwrap().as_socket().unwrap()); + assert_eq!(peer.ip(), std::net::IpAddr::V4(local)); + } +} diff --git a/crates/openshell-isolation-interface/src/linux/socket_registry.rs b/crates/openshell-isolation-interface/src/linux/socket_registry.rs index 7a245d1eac..9754720792 100644 --- a/crates/openshell-isolation-interface/src/linux/socket_registry.rs +++ b/crates/openshell-isolation-interface/src/linux/socket_registry.rs @@ -76,8 +76,6 @@ pub enum SocketState { DnsTcp { relay: SocketAddr }, /// Workload-owned listening socket. Listening { local: SocketAddr }, - /// Stream accepted from a verified local peer. - AcceptedLocal { peer: SocketAddr }, /// A committed relay failed after connection. Failed { errno: i32 }, } @@ -254,9 +252,8 @@ impl SocketRegistry { /// Publish a tentative socket in a caller-proven initial state. /// - /// Accepted sockets are created and classified by the trusted broker, so - /// they enter the registry directly as [`SocketState::AcceptedLocal`] - /// rather than pretending to be unconnected. + /// Used when the trusted broker has already established the socket's + /// state before publication, so the entry never appears unconnected. pub fn commit_with_state( &mut self, tentative: TentativeSocket, diff --git a/crates/openshell-isolation-interface/src/linux/task_memory.rs b/crates/openshell-isolation-interface/src/linux/task_memory.rs index 1b37685b3b..ed34f234e8 100644 --- a/crates/openshell-isolation-interface/src/linux/task_memory.rs +++ b/crates/openshell-isolation-interface/src/linux/task_memory.rs @@ -64,53 +64,6 @@ pub fn read_exact(tid: u32, address: u64, destination: &mut [u8]) -> io::Result< } } -/// Write exactly all of `source` to `address` in `tid`. -/// -/// This is used only for syscall outputs such as `getpeername` and -/// `sendmmsg.msg_len`. Revalidate the notification, task generation, and -/// destination layout immediately before calling it. -pub fn write_exact(tid: u32, address: u64, source: &[u8]) -> io::Result<()> { - validate_request(tid, address, source.len())?; - let pid = libc::pid_t::try_from(tid) - .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "TID does not fit pid_t"))?; - let remote_address = usize::try_from(address).map_err(|_| { - io::Error::new( - io::ErrorKind::InvalidInput, - "remote address does not fit usize", - ) - })?; - let local = libc::iovec { - iov_base: source.as_ptr().cast_mut().cast(), - iov_len: source.len(), - }; - let remote = libc::iovec { - iov_base: remote_address as *mut libc::c_void, - iov_len: source.len(), - }; - - // SAFETY: the local iovec spans the caller-provided live buffer. The - // remote address is untrusted but bounded; the kernel validates that it is - // writable in the target process. - let copied = retry_eintr(|| unsafe { - libc::process_vm_writev( - pid, - std::ptr::addr_of!(local), - 1, - std::ptr::addr_of!(remote), - 1, - 0, - ) - }); - match copied { - Ok(copied) => require_exact(copied, source.len(), "task-memory write"), - Err(error) if syscall_profile_denied(&error) => { - write_exact_to_proc_mem(tid, address, source) - .map_err(|fallback| fallback_error("write", &error, fallback)) - } - Err(error) => Err(error), - } -} - fn syscall_profile_denied(error: &io::Error) -> bool { matches!( error.raw_os_error(), @@ -145,22 +98,14 @@ fn read_exact_from_proc_mem(tid: u32, address: u64, destination: &mut [u8]) -> i require_exact(copied, destination.len(), "proc task-memory read") } -fn write_exact_to_proc_mem(tid: u32, address: u64, source: &[u8]) -> io::Result<()> { - let file = std::fs::OpenOptions::new() - .write(true) - .open(format!("/proc/{tid}/mem"))?; - let copied = file.write_at(source, address)?; - require_exact(copied, source.len(), "proc task-memory write") -} - -/// Prove same-UID parent-to-child read and write access under the active Yama, -/// LSM, and outer seccomp posture. +/// Prove same-UID parent-to-child read access under the active Yama, LSM, and +/// outer seccomp posture. The broker only reads workload memory; it never +/// writes it. /// /// Call this only from a single-threaded probe process. The child executes /// raw, allocation-free syscalls between `fork` and `_exit`. pub fn probe_child_access() -> io::Result<()> { const INITIAL: u64 = 0x1122_3344_5566_7788; - const REPLACEMENT: u64 = 0xaabb_ccdd_eeff_0011; // SAFETY: mmap creates one private anonymous page owned by this process. let mapping = unsafe { libc::mmap( @@ -216,7 +161,6 @@ pub fn probe_child_access() -> io::Result<()> { if libc::prctl(libc::PR_SET_DUMPABLE, 1, 0, 0, 0) < 0 || write_eventfd(ready.as_raw_fd()).is_err() || read_eventfd(proceed.as_raw_fd()).is_err() - || mapping.cast::().read() != REPLACEMENT { libc::_exit(1); } @@ -237,11 +181,6 @@ pub fn probe_child_access() -> io::Result<()> { "cross-child memory read returned wrong data", )); } - write_exact( - u32::try_from(child).map_err(|_| io::Error::other("child PID does not fit u32"))?, - mapping_address, - &REPLACEMENT.to_ne_bytes(), - )?; write_eventfd(proceed.as_raw_fd())?; let mut status = 0; // SAFETY: child is a live direct child and status points to storage. @@ -354,9 +293,8 @@ mod tests { use super::*; #[test] - fn reads_and_writes_exact_same_process_memory() { + fn reads_exact_same_process_memory() { let source = 0x1122_3344_5566_7788_u64; - let mut destination = 0_u64; let mut bytes = [0_u8; size_of::()]; read_exact( @@ -366,21 +304,11 @@ mod tests { ) .expect("read source"); assert_eq!(u64::from_ne_bytes(bytes), source); - - let replacement = 0xaabb_ccdd_eeff_0011_u64; - write_exact( - std::process::id(), - std::ptr::addr_of_mut!(destination) as u64, - &replacement.to_ne_bytes(), - ) - .expect("write destination"); - assert_eq!(destination, replacement); } #[test] - fn proc_mem_fallback_reads_and_writes_exact_memory() { + fn proc_mem_fallback_reads_exact_memory() { let source = 0x0102_0304_0506_0708_u64; - let mut destination = 0_u64; let mut bytes = [0_u8; size_of::()]; read_exact_from_proc_mem( std::process::id(), @@ -389,14 +317,6 @@ mod tests { ) .expect("read through proc mem"); assert_eq!(u64::from_ne_bytes(bytes), source); - - write_exact_to_proc_mem( - std::process::id(), - std::ptr::addr_of_mut!(destination) as u64, - &source.to_ne_bytes(), - ) - .expect("write through proc mem"); - assert_eq!(destination, source); } #[test] diff --git a/crates/openshell-otel/src/lib.rs b/crates/openshell-otel/src/lib.rs index 7a9162ab92..182a1e0837 100644 --- a/crates/openshell-otel/src/lib.rs +++ b/crates/openshell-otel/src/lib.rs @@ -18,7 +18,9 @@ pub use grpc::{ grpc_status_code_name, record_grpc_status, rpc, }; pub use propagation::{ - HeaderMapExtractor, MetadataMapInjector, TraceContextInterceptor, current_trace_context_carrier, + HeaderMapExtractor, MetadataMapInjector, TRACEPARENT_ENV, TRACESTATE_ENV, + TraceContextInterceptor, current_trace_context_carrier, current_trace_context_environment, + set_parent_from_environment, trace_context_environment, }; use opentelemetry::KeyValue; diff --git a/crates/openshell-otel/src/propagation.rs b/crates/openshell-otel/src/propagation.rs index c6e51a0a6b..764e343146 100644 --- a/crates/openshell-otel/src/propagation.rs +++ b/crates/openshell-otel/src/propagation.rs @@ -75,6 +75,80 @@ pub fn current_trace_context_carrier() -> Option> { carrier.contains_key("traceparent").then_some(carrier) } +/// Environment variable carrying a W3C `traceparent` into a child process. +pub const TRACEPARENT_ENV: &str = "TRACEPARENT"; + +/// Environment variable carrying a W3C `tracestate` into a child process. +pub const TRACESTATE_ENV: &str = "TRACESTATE"; + +/// Map W3C propagation fields to child-process environment variables. +#[must_use] +pub fn trace_context_environment( + carrier: &BTreeMap, +) -> Vec<(&'static str, String)> { + [ + ("traceparent", TRACEPARENT_ENV), + ("tracestate", TRACESTATE_ENV), + ] + .into_iter() + .filter_map(|(field, name)| { + carrier + .get(field) + .filter(|value| !value.is_empty()) + .map(|value| (name, value.clone())) + }) + .collect() +} + +/// Environment carrying the active span's trace context to a child process. +/// +/// Empty when the current span has no valid OpenTelemetry context. +#[must_use] +pub fn current_trace_context_environment() -> Vec<(&'static str, String)> { + current_trace_context_carrier() + .map(|carrier| trace_context_environment(&carrier)) + .unwrap_or_default() +} + +/// Parent `span` under the trace context in `TRACEPARENT` and `TRACESTATE`. +/// +/// Leaves `span` unchanged when the environment carries no valid context. +pub fn set_parent_from_environment(span: &tracing::Span) { + let carrier = [ + ("traceparent", TRACEPARENT_ENV), + ("tracestate", TRACESTATE_ENV), + ] + .into_iter() + .filter_map(|(field, name)| { + std::env::var(name) + .ok() + .map(|value| (field.to_string(), value)) + }) + .collect::>(); + set_parent_from_carrier(span, &carrier); +} + +fn set_parent_from_carrier(span: &tracing::Span, carrier: &BTreeMap) { + use opentelemetry::trace::TraceContextExt as _; + + let parent = TraceContextPropagator::new().extract(&TraceContextMapExtractor(carrier)); + if parent.span().span_context().is_valid() { + let _ = span.set_parent(parent); + } +} + +struct TraceContextMapExtractor<'a>(&'a BTreeMap); + +impl Extractor for TraceContextMapExtractor<'_> { + fn get(&self, key: &str) -> Option<&str> { + self.0.get(key).map(String::as_str) + } + + fn keys(&self) -> Vec<&str> { + self.0.keys().map(String::as_str).collect() + } +} + /// Injects the active W3C trace context into an outbound tonic request. #[derive(Debug, Clone, Copy)] pub struct TraceContextInterceptor; @@ -97,6 +171,133 @@ impl tonic::service::Interceptor for TraceContextInterceptor { mod tests { use super::*; + fn env_test_exporter() -> ( + opentelemetry_sdk::trace::SdkTracerProvider, + opentelemetry_sdk::trace::InMemorySpanExporter, + impl tracing::Subscriber + Send + Sync, + ) { + use tracing_subscriber::layer::SubscriberExt as _; + + let exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); + let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry().with(crate::layer(&provider, "env-test")); + (provider, exporter, subscriber) + } + + #[test] + fn environment_carrier_parents_a_span_in_the_same_trace() { + use opentelemetry::trace::TraceContextExt as _; + + let _tracing_lock = crate::test_lock(); + let (_provider, exporter, subscriber) = env_test_exporter(); + + let (parent, environment) = tracing::subscriber::with_default(subscriber, || { + let parent = tracing::info_span!("parent"); + let environment = parent + .in_scope(|| trace_context_environment(¤t_trace_context_carrier().unwrap())); + let child = tracing::info_span!("child"); + let carrier = environment + .iter() + .map(|(name, value)| { + let field = if *name == TRACEPARENT_ENV { + "traceparent" + } else { + "tracestate" + }; + (field.to_string(), value.clone()) + }) + .collect(); + set_parent_from_carrier(&child, &carrier); + drop(child); + (parent.context().span().span_context().clone(), environment) + }); + + assert_eq!(environment.len(), 1, "empty tracestate is omitted"); + assert_eq!(environment[0].0, TRACEPARENT_ENV); + let spans = exporter.get_finished_spans().unwrap(); + let child = spans.iter().find(|span| span.name == "child").unwrap(); + assert_eq!(child.span_context.trace_id(), parent.trace_id()); + assert_eq!(child.parent_span_id, parent.span_id()); + } + + #[test] + #[allow(unsafe_code)] + fn environment_traceparent_parents_a_span() { + let _tracing_lock = crate::test_lock(); + let (_provider, exporter, subscriber) = env_test_exporter(); + let trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"; + let span_id = "00f067aa0ba902b7"; + let original = std::env::var(TRACEPARENT_ENV).ok(); + unsafe { + std::env::set_var(TRACEPARENT_ENV, format!("00-{trace_id}-{span_id}-01")); + } + + tracing::subscriber::with_default(subscriber, || { + let span = tracing::info_span!("child"); + set_parent_from_environment(&span); + }); + + unsafe { + match original { + Some(value) => std::env::set_var(TRACEPARENT_ENV, value), + None => std::env::remove_var(TRACEPARENT_ENV), + } + } + let spans = exporter.get_finished_spans().unwrap(); + let child = spans.iter().find(|span| span.name == "child").unwrap(); + assert_eq!(child.span_context.trace_id().to_string(), trace_id); + assert_eq!(child.parent_span_id.to_string(), span_id); + } + + #[test] + fn invalid_environment_carrier_leaves_the_span_unparented() { + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = crate::test_lock(); + let exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); + let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry().with(crate::layer(&provider, "env-test")); + + tracing::subscriber::with_default(subscriber, || { + let span = tracing::info_span!("orphan"); + let carrier = BTreeMap::from([("traceparent".to_string(), "invalid".to_string())]); + set_parent_from_carrier(&span, &carrier); + }); + + let spans = exporter.get_finished_spans().unwrap(); + let orphan = spans.iter().find(|span| span.name == "orphan").unwrap(); + assert_eq!(orphan.parent_span_id, opentelemetry::trace::SpanId::INVALID); + } + + #[test] + fn interceptor_adds_traceparent_only_inside_an_exported_span() { + use tonic::service::Interceptor as _; + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = crate::test_lock(); + let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder().build(); + let subscriber = tracing_subscriber::registry().with(crate::layer(&provider, "rpc-test")); + + let (outside, inside) = tracing::subscriber::with_default(subscriber, || { + let outside = TraceContextInterceptor + .call(tonic::Request::new(())) + .unwrap(); + let span = tracing::info_span!("client"); + let _entered = span.enter(); + let inside = TraceContextInterceptor + .call(tonic::Request::new(())) + .unwrap(); + (outside, inside) + }); + + assert!(outside.metadata().get("traceparent").is_none()); + assert!(inside.metadata().get("traceparent").is_some()); + } + #[test] fn header_map_extractor_reads_valid_headers() { let mut headers = HeaderMap::new(); diff --git a/crates/openshell-sandbox-backend/src/boundary_protocol.rs b/crates/openshell-sandbox-backend/src/boundary_protocol.rs index b7b32f72bd..70d491d318 100644 --- a/crates/openshell-sandbox-backend/src/boundary_protocol.rs +++ b/crates/openshell-sandbox-backend/src/boundary_protocol.rs @@ -97,9 +97,6 @@ pub struct SeccompEvidence { pub retained_socket_operation: bool, pub proc_fd_identity: bool, pub task_memory_read: bool, - pub task_memory_write: bool, - pub cancellation: bool, - pub task_memory_writes_disabled: bool, } /// Mechanism-specific audit evidence for the native Linux sandbox adapter. @@ -127,6 +124,10 @@ pub struct NativeLinuxSandboxAuditEvidence { pub tcp_dns_round_trip: bool, pub tcp_allow_round_trip: bool, pub tcp_deny_round_trip: bool, + /// Workload INET sockets are bound to loopback before injection, the + /// binding cannot be changed from sandbox credentials, and accepted + /// sockets inherit it. Native local `accept` depends on this property. + pub socket_loopback_confinement: bool, } impl NativeLinuxSandboxAuditEvidence { @@ -146,14 +147,13 @@ impl NativeLinuxSandboxAuditEvidence { && self.seccomp.retained_socket_operation && self.seccomp.proc_fd_identity && self.seccomp.task_memory_read - && self.seccomp.task_memory_write - && (self.seccomp.cancellation || self.seccomp.task_memory_writes_disabled) && self.landlock_abi >= 3 && self.landlock_allow_deny && self.udp_dns_round_trip && self.tcp_dns_round_trip && self.tcp_allow_round_trip - && self.tcp_deny_round_trip; + && self.tcp_deny_round_trip + && self.socket_loopback_confinement; if complete { Ok(()) } else { @@ -178,14 +178,14 @@ impl NativeLinuxSandboxAuditEvidence { && self.udp_dns_round_trip && self.tcp_dns_round_trip && self.tcp_allow_round_trip - && self.tcp_deny_round_trip, + && self.tcp_deny_round_trip + && self.socket_loopback_confinement, "seccomp-notify", ), request_attribution: EnforcedProperty::new( self.seccomp.id_validation && self.seccomp.proc_fd_identity - && self.seccomp.task_memory_read - && self.seccomp.task_memory_write, + && self.seccomp.task_memory_read, "seccomp-notify-procfs", ), privilege_floor: EnforcedProperty::new( @@ -1484,9 +1484,6 @@ mod tests { retained_socket_operation: true, proc_fd_identity: true, task_memory_read: true, - task_memory_write: true, - cancellation: true, - task_memory_writes_disabled: false, }, landlock_abi: 6, landlock_allow_deny: true, @@ -1494,6 +1491,7 @@ mod tests { tcp_dns_round_trip: true, tcp_allow_round_trip: true, tcp_deny_round_trip: true, + socket_loopback_confinement: true, } } @@ -1518,19 +1516,11 @@ mod tests { } #[test] - fn audit_evidence_accepts_legacy_read_only_listener() { + fn audit_evidence_requires_socket_loopback_confinement() { let mut audit = complete_audit_evidence(); - audit.seccomp.cancellation = false; - audit.seccomp.task_memory_writes_disabled = true; - assert!(audit.validate().is_ok()); - } - - #[test] - fn audit_evidence_rejects_plain_listener_with_writes_enabled() { - let mut audit = complete_audit_evidence(); - audit.seccomp.cancellation = false; - audit.seccomp.task_memory_writes_disabled = false; + audit.socket_loopback_confinement = false; assert!(audit.validate().is_err()); + assert!(!audit.properties().egress_interception.enforced); } #[test] diff --git a/crates/openshell-sandbox-backend/src/runtime.rs b/crates/openshell-sandbox-backend/src/runtime.rs index b4a644ebb6..55b84ca431 100644 --- a/crates/openshell-sandbox-backend/src/runtime.rs +++ b/crates/openshell-sandbox-backend/src/runtime.rs @@ -2909,9 +2909,6 @@ mod tests { retained_socket_operation: true, proc_fd_identity: true, task_memory_read: true, - task_memory_write: true, - cancellation: true, - task_memory_writes_disabled: false, }, landlock_abi: 3, landlock_allow_deny: true, @@ -2919,6 +2916,7 @@ mod tests { tcp_dns_round_trip: true, tcp_allow_round_trip: true, tcp_deny_round_trip: true, + socket_loopback_confinement: true, }; openshell_isolation_interface::contract::BoundaryConfirmation { generation: "test-generation".to_string(), diff --git a/crates/openshell-sandbox/src/accept_interrupt.rs b/crates/openshell-sandbox/src/accept_interrupt.rs deleted file mode 100644 index 86bbf39218..0000000000 --- a/crates/openshell-sandbox/src/accept_interrupt.rs +++ /dev/null @@ -1,297 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Cancellation for broker-owned blocking accepts without changing workload OFDs. -//! -//! SIGUSR2 is reserved by the sandbox binary. Its process-global disposition is -//! necessarily kernel state, not a global application context. All registration, -//! cancellation and thread ownership state belongs to one broker instance. - -#![allow(unsafe_code)] - -use std::collections::HashMap; -use std::io; -use std::marker::PhantomData; -use std::rc::Rc; -use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::{Arc, Condvar, Mutex}; -use std::time::Duration; - -const INTERRUPT_SIGNAL: libc::c_int = libc::SIGUSR2; -const INTERRUPT_INTERVAL: Duration = Duration::from_millis(10); - -extern "C" fn interrupt_accept(_: libc::c_int) {} - -fn reserve_signal() -> io::Result<()> { - // SAFETY: both actions are initialized storage. The no-op handler is - // async-signal-safe and deliberately omits SA_RESTART so accept returns EINTR. - unsafe { - let mut previous: libc::sigaction = std::mem::zeroed(); - if libc::sigaction(INTERRUPT_SIGNAL, std::ptr::null(), &raw mut previous) < 0 { - return Err(io::Error::last_os_error()); - } - if previous.sa_sigaction != libc::SIG_DFL - && previous.sa_sigaction != interrupt_accept as *const () as usize - { - return Err(io::Error::other( - "sandbox SIGUSR2 is already reserved by another handler", - )); - } - let mut action: libc::sigaction = std::mem::zeroed(); - action.sa_sigaction = interrupt_accept as *const () as usize; - libc::sigemptyset(&raw mut action.sa_mask); - if libc::sigaction(INTERRUPT_SIGNAL, &raw const action, std::ptr::null_mut()) < 0 { - return Err(io::Error::last_os_error()); - } - } - Ok(()) -} - -#[derive(Default)] -struct State { - workers: Mutex>, - changed: Condvar, - stopped: AtomicBool, -} - -// musl represents pthread_t as an opaque pointer, unlike glibc's integer. It -// is only passed back to pthread_kill, never dereferenced by this module. -struct RegisteredThread(libc::pthread_t); - -// SAFETY: POSIX permits signaling a live pthread from another thread. The -// handle is accessed only under State::workers, and the owning worker removes -// its registration under that same mutex before returning. AcceptRegistration -// cannot move to another thread, so its Drop cannot outlive the owning worker. -unsafe impl Send for RegisteredThread {} - -pub struct AcceptMonitor { - state: Arc, - thread: Option>, -} - -impl AcceptMonitor { - pub(crate) fn start(valid: impl Fn(u64) -> bool + Send + 'static) -> io::Result { - reserve_signal()?; - let state = Arc::new(State::default()); - let worker_state = state.clone(); - let thread = std::thread::Builder::new() - .name("openshell-accept-cancellation".into()) - .spawn(move || monitor(&worker_state, valid))?; - Ok(Self { - state, - thread: Some(thread), - }) - } - - pub(crate) fn registrar(&self) -> AcceptRegistrar { - AcceptRegistrar(self.state.clone()) - } -} - -impl Drop for AcceptMonitor { - fn drop(&mut self) { - let workers = lock(&self.state.workers); - self.state.stopped.store(true, Ordering::Release); - self.state.changed.notify_all(); - drop(workers); - // The monitor keeps interrupting registered workers during shutdown. - // Registrations are removed before their threads can exit/reuse IDs. - if let Some(thread) = self.thread.take() { - let _ = thread.join(); - } - } -} - -#[derive(Clone)] -pub struct AcceptRegistrar(Arc); - -impl AcceptRegistrar { - pub(crate) fn register(&self, notification_id: u64) -> io::Result { - // SAFETY: this changes only the current broker worker's signal mask. - // Workload launchers do not inherit this mask; exec resets the handler. - let thread = unsafe { - let mut mask: libc::sigset_t = std::mem::zeroed(); - libc::sigemptyset(&raw mut mask); - libc::sigaddset(&raw mut mask, INTERRUPT_SIGNAL); - let error = - libc::pthread_sigmask(libc::SIG_UNBLOCK, &raw const mask, std::ptr::null_mut()); - if error != 0 { - return Err(io::Error::from_raw_os_error(error)); - } - libc::pthread_self() - }; - let mut workers = lock(&self.0.workers); - if self.0.stopped.load(Ordering::Acquire) { - return Err(io::Error::from_raw_os_error(libc::ECANCELED)); - } - if workers.contains_key(¬ification_id) { - return Err(io::Error::other( - "duplicate accept notification registration", - )); - } - workers.insert(notification_id, RegisteredThread(thread)); - self.0.changed.notify_one(); - Ok(AcceptRegistration { - state: self.0.clone(), - notification_id, - owning_thread: PhantomData, - }) - } -} - -pub struct AcceptRegistration { - state: Arc, - notification_id: u64, - // Drop must run on the registering thread before its pthread_t can expire. - // No Rc is allocated; this marker makes the guard neither Send nor Sync. - owning_thread: PhantomData>, -} - -impl AcceptRegistration { - pub(crate) fn ensure_running(&self) -> io::Result<()> { - if self.state.stopped.load(Ordering::Acquire) { - Err(io::Error::from_raw_os_error(libc::ECANCELED)) - } else { - Ok(()) - } - } -} - -impl Drop for AcceptRegistration { - fn drop(&mut self) { - lock(&self.state.workers).remove(&self.notification_id); - self.state.changed.notify_one(); - } -} - -fn monitor(state: &State, valid: impl Fn(u64) -> bool) { - let mut workers = lock(&state.workers); - loop { - let stopped = state.stopped.load(Ordering::Acquire); - if stopped && workers.is_empty() { - return; - } - for (¬ification_id, thread) in &*workers { - if stopped || !valid(notification_id) { - // SAFETY: the registration lock pins this live pthread_t. - // Repeated interrupts close the check-to-accept race: a signal - // received before accept cannot leave a later accept stranded. - let _ = unsafe { libc::pthread_kill(thread.0, INTERRUPT_SIGNAL) }; - } - } - workers = if workers.is_empty() { - state - .changed - .wait(workers) - .unwrap_or_else(std::sync::PoisonError::into_inner) - } else { - state - .changed - .wait_timeout(workers, INTERRUPT_INTERVAL) - .unwrap_or_else(std::sync::PoisonError::into_inner) - .0 - }; - } -} - -fn lock(mutex: &Mutex) -> std::sync::MutexGuard<'_, T> { - mutex - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner) -} - -#[cfg(test)] -mod tests { - use super::*; - use std::net::{TcpListener, TcpStream}; - use std::os::fd::AsRawFd; - - #[test] - fn registrar_crosses_threads_but_registration_ends_before_worker_exit() { - fn assert_send_sync() {} - assert_send_sync::(); - - let monitor = AcceptMonitor::start(|_| true).unwrap(); - let registrar = monitor.registrar(); - std::thread::spawn(move || { - let registration = registrar.register(3).unwrap(); - assert!(registrar.register(3).is_err()); - assert!(lock(®istrar.0.workers).contains_key(&3)); - drop(registration); - assert!(lock(®istrar.0.workers).is_empty()); - }) - .join() - .unwrap(); - assert!(lock(&monitor.state.workers).is_empty()); - } - - #[test] - fn cancellation_interrupts_competing_accept_after_readiness_was_consumed() { - let valid = Arc::new(AtomicBool::new(true)); - let monitored = valid.clone(); - let monitor = AcceptMonitor::start(move |_| monitored.load(Ordering::Acquire)).unwrap(); - let listener = TcpListener::bind("127.0.0.1:0").unwrap(); - let client = TcpStream::connect(listener.local_addr().unwrap()).unwrap(); - // Both contenders could observe this same readable listener. Consume - // its only connection before the second contender actually accepts. - let accepted = listener.accept().unwrap(); - let registrar = monitor.registrar(); - let (ready_tx, ready_rx) = std::sync::mpsc::channel(); - let (done_tx, done_rx) = std::sync::mpsc::channel(); - let worker = std::thread::spawn(move || { - let registration = registrar.register(1).unwrap(); - ready_tx.send(()).unwrap(); - // SAFETY: the listener is live and null address outputs are valid. - // Use the syscall directly: std::net retries EINTR internally. - let result = unsafe { - libc::accept4( - listener.as_raw_fd(), - std::ptr::null_mut(), - std::ptr::null_mut(), - libc::SOCK_CLOEXEC, - ) - }; - assert_eq!(result, -1); - let error = io::Error::last_os_error(); - assert_eq!(error.kind(), io::ErrorKind::Interrupted); - drop(registration); - done_tx.send(()).unwrap(); - }); - ready_rx.recv_timeout(Duration::from_secs(2)).unwrap(); - valid.store(false, Ordering::Release); - done_rx.recv_timeout(Duration::from_secs(2)).unwrap(); - worker.join().unwrap(); - drop((accepted, client, monitor)); - } - - #[test] - fn shutdown_interrupts_registered_accepts_and_reclaims_the_monitor() { - let monitor = AcceptMonitor::start(|_| true).unwrap(); - let registrar = monitor.registrar(); - let listener = TcpListener::bind("127.0.0.1:0").unwrap(); - let (ready_tx, ready_rx) = std::sync::mpsc::channel(); - let (done_tx, done_rx) = std::sync::mpsc::channel(); - let worker = std::thread::spawn(move || { - let registration = registrar.register(2).unwrap(); - ready_tx.send(()).unwrap(); - // SAFETY: owned listener and optional null address outputs. - let result = unsafe { - libc::accept4( - listener.as_raw_fd(), - std::ptr::null_mut(), - std::ptr::null_mut(), - libc::SOCK_CLOEXEC, - ) - }; - assert_eq!(result, -1); - assert!(registration.ensure_running().is_err()); - drop(registration); - done_tx.send(()).unwrap(); - }); - ready_rx.recv_timeout(Duration::from_secs(2)).unwrap(); - let shutdown = std::thread::spawn(move || drop(monitor)); - done_rx.recv_timeout(Duration::from_secs(2)).unwrap(); - worker.join().unwrap(); - shutdown.join().unwrap(); - } -} diff --git a/crates/openshell-sandbox/src/boundary_exec.rs b/crates/openshell-sandbox/src/boundary_exec.rs index da35294c06..5482629069 100644 --- a/crates/openshell-sandbox/src/boundary_exec.rs +++ b/crates/openshell-sandbox/src/boundary_exec.rs @@ -168,7 +168,7 @@ impl LocalBoundaryExec { command.env("SHELL", shell); } for (key, value) in &self.user_environment { - if !key.starts_with("OPENSHELL_") { + if !key.starts_with(crate::process::RESERVED_ENV_PREFIX) { command.env(key, value); } } @@ -184,7 +184,7 @@ impl LocalBoundaryExec { } crate::process::strip_proxy_env_std(&mut command); for (key, value) in &spec.env { - if !key.starts_with("OPENSHELL_") { + if !key.starts_with(crate::process::RESERVED_ENV_PREFIX) { command.env(key, value); } } diff --git a/crates/openshell-sandbox/src/boundary_io.rs b/crates/openshell-sandbox/src/boundary_io.rs index d404739af0..1889052421 100644 --- a/crates/openshell-sandbox/src/boundary_io.rs +++ b/crates/openshell-sandbox/src/boundary_io.rs @@ -220,6 +220,25 @@ impl BoundaryRuntimeState { .is_ok_and(|groups| !groups.is_empty()) } + /// Whether any workload process remains, registered or not. + /// + /// A registered root is unregistered once it is reaped, but descendants + /// that ignored `SIGTERM` may outlive it. When the sandbox owns the + /// process tree (PID 1 or a child subreaper), every live descendant is + /// counted so termination is not reported complete while one survives. + #[must_use] + pub fn has_owned_processes(&self) -> bool { + if self.has_registered_processes() { + return true; + } + // An unreadable /proc fails closed: processes may remain. + #[cfg(target_os = "linux")] + return owned_processes(&[], self.exclusive_pid_namespace) + .map_or(true, |owned| !owned.is_empty()); + #[cfg(not(target_os = "linux"))] + false + } + /// End the boundary because required standing enforcement was lost. /// /// Returns `true` only to the caller that won the active-to-terminated @@ -265,13 +284,11 @@ impl BoundaryRuntimeState { // requiring ptrace or a capability. let mut previous = Vec::new(); for _ in 0..4 { - let owned = owned_process_ids(&roots, self.exclusive_pid_namespace); - for pid in &owned { - if roots.contains(pid) { - continue; - } - if let Ok(pid) = i32::try_from(*pid) { - let _ = nix::sys::signal::kill(nix::unistd::Pid::from_raw(pid), signal); + let owned = + owned_processes(&roots, self.exclusive_pid_namespace).unwrap_or_default(); + for process in &owned { + if !roots.contains(&process.pid) { + signal_owned_process(*process, signal); } } if owned == previous { @@ -283,13 +300,54 @@ impl BoundaryRuntimeState { } } +/// One scanned workload process, identified by PID and kernel start time so a +/// reused PID is never mistaken for it. #[cfg(target_os = "linux")] -fn owned_process_ids(roots: &[u32], exclusive_pid_namespace: bool) -> Vec { - let mut parents = HashMap::new(); - let Ok(entries) = std::fs::read_dir("/proc") else { - return roots.to_vec(); - }; - for entry in entries.flatten() { +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +struct OwnedProcess { + pid: u32, + start_time: u64, +} + +#[cfg(target_os = "linux")] +struct ProcStat { + parent: u32, + start_time: u64, + live: bool, +} + +#[cfg(target_os = "linux")] +fn read_proc_stat(pid: u32) -> Option { + let stat = std::fs::read_to_string(format!("/proc/{pid}/stat")).ok()?; + // The command name may contain spaces or parentheses; fields resume after + // the final ") ". Field 3 is the state, 4 the parent, 22 the start time. + let fields = stat + .rsplit_once(") ")? + .1 + .split_whitespace() + .collect::>(); + Some(ProcStat { + parent: fields.get(1)?.parse().ok()?, + start_time: fields.get(19)?.parse().ok()?, + live: !matches!(*fields.first()?, "Z" | "X" | "x"), + }) +} + +/// Whether orphaned descendants are reparented to this sandbox process. +#[cfg(target_os = "linux")] +fn sandbox_owns_process_tree() -> bool { + std::process::id() == 1 + || rustix::process::child_subreaper().is_ok_and(|subreaper| subreaper.is_some()) +} + +#[cfg(target_os = "linux")] +fn owned_processes( + roots: &[u32], + exclusive_pid_namespace: bool, +) -> std::io::Result> { + let mut stats = HashMap::new(); + let mut children: HashMap> = HashMap::new(); + for entry in std::fs::read_dir("/proc")?.flatten() { let Some(pid) = entry .file_name() .to_str() @@ -297,52 +355,65 @@ fn owned_process_ids(roots: &[u32], exclusive_pid_namespace: bool) -> Vec { else { continue; }; - let Ok(stat) = std::fs::read_to_string(entry.path().join("stat")) else { - continue; - }; - let Some(after_name) = stat.rsplit_once(") ").map(|(_, fields)| fields) else { - continue; - }; - let Some(parent) = after_name - .split_whitespace() - .nth(1) - .and_then(|field| field.parse::().ok()) - else { + if let Some(stat) = read_proc_stat(pid) { + children.entry(stat.parent).or_default().push(pid); + stats.insert(pid, stat); + } + } + + // When the sandbox is PID 1 of its exclusive namespace or a child + // subreaper, orphans are reparented to it, so its descendants are exactly + // the workload tree. Otherwise walk only from registered roots so unit + // tests and development runs cannot affect sibling tasks. + let sandbox = std::process::id(); + let mut pending = if exclusive_pid_namespace && sandbox_owns_process_tree() { + vec![sandbox] + } else { + roots.to_vec() + }; + let mut visited = std::collections::HashSet::new(); + let mut owned = Vec::new(); + while let Some(pid) = pending.pop() { + if !visited.insert(pid) { continue; - }; - parents.insert(pid, parent); - } - - // When openshell-sandbox is PID 1, every other process in its exclusive - // namespace is workload-owned, including an orphan reparented during the - // scan. Outside that deployment shape, restrict the walk to registered - // roots so unit tests and development runs cannot affect sibling tasks. - if exclusive_pid_namespace && std::process::id() == 1 { - let mut owned = parents - .keys() - .copied() - .filter(|pid| *pid != 1) - .collect::>(); - owned.sort_unstable(); - return owned; - } - - let mut owned = roots.to_vec(); - loop { - let mut changed = false; - for (&pid, &parent) in &parents { - if !owned.contains(&pid) && owned.contains(&parent) { - owned.push(pid); - changed = true; - } } - if !changed { - break; + if let Some(descendants) = children.get(&pid) { + pending.extend(descendants); + } + if let Some(stat) = stats.get(&pid) + && pid != sandbox + && stat.live + { + owned.push(OwnedProcess { + pid, + start_time: stat.start_time, + }); } } owned.sort_unstable(); - owned.dedup(); - owned + Ok(owned) +} + +/// Signal one scanned process through a pidfd, after confirming the pidfd +/// refers to the scanned process rather than a later process with its PID. +#[cfg(target_os = "linux")] +fn signal_owned_process(process: OwnedProcess, signal: nix::sys::signal::Signal) { + let Some(pid) = i32::try_from(process.pid) + .ok() + .and_then(rustix::process::Pid::from_raw) + else { + return; + }; + let Some(signal) = rustix::process::Signal::from_named_raw(signal as i32) else { + return; + }; + let Ok(pidfd) = rustix::process::pidfd_open(pid, rustix::process::PidfdFlags::empty()) else { + return; + }; + if read_proc_stat(process.pid).map(|stat| stat.start_time) != Some(process.start_time) { + return; + } + let _ = rustix::process::pidfd_send_signal(&pidfd, signal); } #[derive(Clone)] @@ -509,6 +580,73 @@ mod tests { )); } + #[cfg(target_os = "linux")] + #[test] + fn termination_waits_for_descendants_that_outlive_their_root() { + use std::io::BufRead as _; + use std::os::unix::process::CommandExt as _; + + // Becoming a subreaper changes this whole process, so run the + // scenario in a fresh copy of the test binary. + const CHILD_MARKER: &str = "OPENSHELL_SUBREAPER_TEARDOWN_CHILD"; + if std::env::var_os(CHILD_MARKER).is_none() { + let status = std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "boundary_io::tests::termination_waits_for_descendants_that_outlive_their_root", + "--nocapture", + ]) + .env(CHILD_MARKER, "1") + .status() + .expect("run isolated teardown test"); + assert!(status.success(), "isolated teardown test failed"); + return; + } + rustix::process::set_child_subreaper(Some(rustix::process::getpid())) + .expect("become child subreaper"); + let runtime = BoundaryRuntimeState::new_exclusive_pid_namespace(); + // The grandchild inherits an ignored SIGTERM; the root restores the + // default disposition and exits on SIGTERM. + let mut root = std::process::Command::new("/bin/sh") + .args([ + "-c", + "trap '' TERM; sleep 600 & trap - TERM; echo ready; wait", + ]) + .process_group(0) + .stdout(std::process::Stdio::piped()) + .spawn() + .expect("spawn root"); + let mut line = String::new(); + std::io::BufReader::new(root.stdout.take().unwrap()) + .read_line(&mut line) + .unwrap(); + assert_eq!(line.trim(), "ready"); + let terminal = Arc::new(std::sync::atomic::AtomicBool::new(false)); + runtime + .register_process_group(root.id(), terminal.clone(), Arc::new(Mutex::new(()))) + .expect("register root"); + + assert!(runtime.begin_termination()); + root.wait().expect("root exits on SIGTERM"); + terminal.store(true, Ordering::Release); + runtime.unregister_process_group(root.id(), &terminal); + assert!(!runtime.has_registered_processes()); + assert!( + runtime.has_owned_processes(), + "a SIGTERM-ignoring grandchild must keep termination incomplete" + ); + + runtime.force_kill(); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while runtime.has_owned_processes() { + assert!( + std::time::Instant::now() < deadline, + "forced termination left a descendant alive" + ); + std::thread::sleep(std::time::Duration::from_millis(20)); + } + } + #[test] fn freeze_blocks_new_operations_until_explicit_resume() { let runtime = BoundaryRuntimeState::new_exclusive_pid_namespace(); diff --git a/crates/openshell-sandbox/src/boundary_server.rs b/crates/openshell-sandbox/src/boundary_server.rs index 80ca6f32cd..7098c37230 100644 --- a/crates/openshell-sandbox/src/boundary_server.rs +++ b/crates/openshell-sandbox/src/boundary_server.rs @@ -86,16 +86,15 @@ mod linux { // NVML may traverse the persistenced socket directory during initialization; // WSL2 supplies GPU libraries under /usr/lib/wsl and the /dev/dxg device. const GPU_BASELINE_READ_ONLY: &[&str] = &["/run/nvidia-persistenced", "/usr/lib/wsl"]; - // CUDA opens device nodes read-write and writes thread names through - // /proc//task//comm during cuInit(). A /proc/self rule would bind - // to the launcher's inodes, not those of its workload children. + // CUDA opens device nodes read-write. Its thread-name writes through + // /proc//task//comm are served by open mediation, so /proc + // stays read-only. const GPU_BASELINE_READ_WRITE: &[&str] = &[ "/dev/nvidiactl", "/dev/nvidia-uvm", "/dev/nvidia-uvm-tools", "/dev/nvidia-modeset", "/dev/dxg", - "/proc", ]; fn duration_micros(duration: Duration) -> u64 { @@ -154,13 +153,7 @@ mod linux { continue; } if policy.filesystem.read_only.contains(&path) { - if path != Path::new("/proc") { - continue; - } - policy - .filesystem - .read_only - .retain(|allowed| allowed != &path); + continue; } policy.filesystem.read_write.push(path); modified = true; @@ -223,10 +216,15 @@ mod linux { } crate::sandbox::apply_supervisor_startup_hardening() .map_err(|error| format!("install sandbox process prelude: {error}"))?; - if nix::unistd::getpid().as_raw() == 1 { - crate::managed_children::start_orphan_reaper() - .map_err(|error| format!("start sandbox orphan reaper: {error}"))?; - } + // Keep orphaned workload descendants in this process tree so + // termination can find and kill them, then reap the adopted ones. + // PID 1 already receives orphans; elsewhere become a child subreaper. + if nix::unistd::getpid().as_raw() != 1 { + rustix::process::set_child_subreaper(Some(rustix::process::getpid())) + .map_err(|error| format!("become child subreaper: {error}"))?; + } + crate::managed_children::start_orphan_reaper() + .map_err(|error| format!("start sandbox orphan reaper: {error}"))?; let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() .map_err(|error| format!("start sandbox workload launcher: {error}"))?; let protected_control_port = match &config.listener { @@ -343,6 +341,16 @@ mod linux { .to_string(), ); } + // Workload sockets share the loopback interface with a loopback + // listener. + BoundaryListenerConfig::TlsTcp { address, .. } + if address.ip().to_canonical().is_loopback() => + { + return Err( + "boundary TLS listener must not bind a loopback address; workloads share the loopback interface" + .to_string(), + ); + } BoundaryListenerConfig::Vsock { control_port: 0, .. } => { @@ -1764,6 +1772,7 @@ mod linux { "frozen workload could not be resumed".to_string(), )); } + self.network_broker.set_workload_frozen(false); tracing::info!( connection_id = ?principal.connection_id(), "Sandbox Protocol connection recovered; workload resumed" @@ -1820,6 +1829,7 @@ mod linux { return; } if let Some(process) = &process { + self.network_broker.set_workload_frozen(true); let _ = process.boundary_runtime.freeze(); } *connection = SupervisorConnectionState::Frozen { recovery_id }; @@ -1946,12 +1956,12 @@ mod linux { async fn wait_for_process_tree_exit(process: &ManagedProcess, timeout: Duration) -> bool { let deadline = tokio::time::Instant::now() + timeout; - while process.boundary_runtime.has_registered_processes() + while process.boundary_runtime.has_owned_processes() && tokio::time::Instant::now() < deadline { tokio::time::sleep(Duration::from_millis(25)).await; } - !process.boundary_runtime.has_registered_processes() + !process.boundary_runtime.has_owned_processes() } fn shutdown(&self) { @@ -2479,6 +2489,7 @@ mod linux { tcp_dns_round_trip: self.qualification.tcp_dns_round_trip, tcp_allow_round_trip: self.qualification.tcp_allow_round_trip, tcp_deny_round_trip: self.qualification.tcp_deny_round_trip, + socket_loopback_confinement: self.qualification.socket_loopback_confinement, }; // The boundary reports mechanism evidence; the authenticated host // backend validates it before constructing a ConfirmedBoundary. @@ -3294,7 +3305,7 @@ mod linux { }) } BoundaryListenerConfig::TlsTcp { address, tls } => { - let listener = std::net::TcpListener::bind(address)?; + let listener = Self::bind_tcp(*address)?; listener.set_nonblocking(true)?; let server_config = Arc::new(load_tls_server_config(tls)?); Ok(Self::Tcp { @@ -3305,6 +3316,28 @@ mod linux { } } + /// Bind the TCP control listener, dropping loopback-interface ingress + /// before it listens so workload sockets cannot reach it through + /// loopback or the pod's own address. Configuration rejects loopback + /// addresses; tests bind them without the filter. + fn bind_tcp(address: std::net::SocketAddr) -> io::Result { + let socket = socket2::Socket::new( + socket2::Domain::for_address(address), + socket2::Type::STREAM, + Some(socket2::Protocol::TCP), + )?; + socket.set_cloexec(true)?; + socket.set_reuse_address(true)?; + if !address.ip().is_loopback() { + openshell_isolation_interface::linux::socket_confinement::reject_loopback_ingress( + &socket, + )?; + } + socket.bind(&address.into())?; + socket.listen(128)?; + Ok(socket.into()) + } + fn bind_vsock(port: u32) -> io::Result { let family = libc::sa_family_t::try_from(libc::AF_VSOCK).map_err(|_| { io::Error::new(io::ErrorKind::InvalidInput, "AF_VSOCK exceeds sa_family_t") @@ -4606,9 +4639,6 @@ mod linux { retained_socket_operation: true, proc_fd_identity: true, task_memory_read: true, - task_memory_write: true, - cancellation: true, - task_memory_writes_disabled: false, }, landlock_abi: 6, landlock_allow_deny: true, @@ -4616,6 +4646,7 @@ mod linux { tcp_dns_round_trip: true, tcp_allow_round_trip: true, tcp_deny_round_trip: true, + socket_loopback_confinement: true, } } @@ -4944,6 +4975,46 @@ mod linux { } } + #[test] + fn tcp_control_listener_rejects_loopback_addresses() { + let directory = tempfile::tempdir().expect("temporary directory"); + let (server_tls, _client_tls) = stage_test_tls(directory.path(), "validate"); + let config = |address: &str| BoundaryConfig { + boundary_id: "sandbox-1".to_string(), + generation: "generation-1".to_string(), + session_id: test_session_id(), + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .expect("session rotation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + gateway_id: "test-gateway".to_string(), + verification_keys: vec![test_verification_key()], + listener: BoundaryListenerConfig::TlsTcp { + address: address.parse().expect("valid address"), + tls: server_tls.clone(), + }, + resource_claims: std::collections::BTreeMap::new(), + resource_claim_files: std::collections::BTreeMap::new(), + workload_identity: test_workload_identity(), + outer_fence: test_outer_fence(), + child_env: std::collections::HashMap::new(), + }; + for address in [ + "127.0.0.1:5500", + "127.0.0.2:5500", + "[::1]:5500", + "[::ffff:127.0.0.1]:5500", + ] { + assert!( + validate_config(&config(address)).is_err(), + "{address} must be rejected" + ); + } + for address in ["0.0.0.0:5500", "[::]:5500", "10.42.0.7:5500"] { + validate_config(&config(address)) + .unwrap_or_else(|error| panic!("{address} must be accepted: {error}")); + } + } + #[test] fn runtime_resource_claim_file_must_match_admitted_claim() { let directory = tempfile::tempdir().expect("temporary directory"); @@ -5066,6 +5137,35 @@ mod linux { server.abort(); } + #[test] + fn pod_control_listener_rejects_loopback_ingress() { + let directory = tempfile::tempdir().expect("temporary directory"); + let (server_tls, _client_tls) = stage_test_tls(directory.path(), "loopback"); + let listener = ControlListener::bind(&BoundaryListenerConfig::TlsTcp { + address: "0.0.0.0:0".parse().expect("valid address"), + tls: server_tls, + }) + .expect("bind TLS listener"); + let port = listener + .tcp_local_addr() + .expect("TLS listener address") + .port(); + // Loopback and the host's own address both arrive on `lo`; the + // dropped SYN never completes a handshake. + let result = std::net::TcpStream::connect_timeout( + &std::net::SocketAddr::from(([127, 0, 0, 1], port)), + Duration::from_millis(300), + ); + assert!( + result.is_err(), + "loopback client reached the control listener" + ); + assert!(matches!( + listener.accept().map(|_| ()), + Err(error) if error.kind() == io::ErrorKind::WouldBlock + )); + } + #[test] fn tls_listener_preserves_session_when_control_switches_to_async_streaming() { let directory = tempfile::tempdir().expect("temporary directory"); diff --git a/crates/openshell-sandbox/src/lib.rs b/crates/openshell-sandbox/src/lib.rs index a8d31fbfe9..110a395842 100644 --- a/crates/openshell-sandbox/src/lib.rs +++ b/crates/openshell-sandbox/src/lib.rs @@ -3,8 +3,6 @@ //! Capability-free in-workload sandbox boundary. -#[cfg(target_os = "linux")] -mod accept_interrupt; pub mod boundary_exec; pub mod boundary_io; mod boundary_server; @@ -44,6 +42,7 @@ pub struct RuntimeQualification { pub tcp_dns_round_trip: bool, pub tcp_allow_round_trip: bool, pub tcp_deny_round_trip: bool, + pub socket_loopback_confinement: bool, } /// Placeholder used when compiling the package on a non-Linux host. diff --git a/crates/openshell-sandbox/src/main.rs b/crates/openshell-sandbox/src/main.rs index 99f37b202e..514f2c1901 100644 --- a/crates/openshell-sandbox/src/main.rs +++ b/crates/openshell-sandbox/src/main.rs @@ -95,17 +95,14 @@ struct QualificationReport { task_memory_copy: bool, connected_send_fast_path: bool, socket_virtualization: bool, + /// Workload INET sockets are bound to loopback, the binding cannot be + /// changed from sandbox credentials, and accepted sockets inherit it. + socket_loopback_confinement: bool, dns_relay_bind: bool, udp_dns_round_trip: bool, tcp_dns_round_trip: bool, tcp_allow_round_trip: bool, tcp_deny_round_trip: bool, - wait_killable_recv: bool, - /// Selected seccomp listener cancellation mode: `killable` (>= 5.19) or - /// `legacy_read_only` (< 5.19, broker output writes disabled). - seccomp_listener_mode: &'static str, - /// Whether the broker disables task-memory output writes (legacy mode). - task_memory_writes_disabled: bool, } #[cfg(target_os = "linux")] @@ -163,6 +160,9 @@ fn qualify_runtime() -> Result<(openshell_sandbox::RuntimeQualification, Qualifi .into_diagnostic() .wrap_err("seccomp notification probe")?; probe_socket_virtualization().wrap_err("socket virtualization probe")?; + openshell_isolation_interface::linux::socket_confinement::probe_loopback_confinement() + .into_diagnostic() + .wrap_err("socket loopback confinement probe")?; probe_dns_relay_bind().wrap_err("DNS relay bind probe")?; let landlock_abi = openshell_isolation_interface::linux::landlock::abi_version() .into_diagnostic() @@ -196,18 +196,12 @@ fn qualify_runtime() -> Result<(openshell_sandbox::RuntimeQualification, Qualifi task_memory_copy, connected_send_fast_path: notification.connected_send_fast_path(), socket_virtualization: true, + socket_loopback_confinement: true, dns_relay_bind: true, udp_dns_round_trip: true, tcp_dns_round_trip: true, tcp_allow_round_trip: true, tcp_deny_round_trip: true, - wait_killable_recv: notification.wait_killable_recv, - seccomp_listener_mode: if notification.wait_killable_recv { - "killable" - } else { - "legacy_read_only" - }, - task_memory_writes_disabled: !notification.wait_killable_recv, }; let qualification = openshell_sandbox::RuntimeQualification { seccomp: openshell_sandbox_backend::boundary_protocol::SeccompEvidence { @@ -218,11 +212,6 @@ fn qualify_runtime() -> Result<(openshell_sandbox::RuntimeQualification, Qualifi retained_socket_operation: true, proc_fd_identity: true, task_memory_read: task_memory_copy, - task_memory_write: task_memory_copy, - cancellation: notification.wait_killable_recv, - // Legacy plain listener (< 5.19) disables broker output writes; - // satisfies the `cancellation || writes_disabled` launch invariant. - task_memory_writes_disabled: !notification.wait_killable_recv, }, landlock_abi, landlock_allow_deny: true, @@ -230,6 +219,7 @@ fn qualify_runtime() -> Result<(openshell_sandbox::RuntimeQualification, Qualifi tcp_dns_round_trip: true, tcp_allow_round_trip: true, tcp_deny_round_trip: true, + socket_loopback_confinement: true, }; Ok((qualification, report)) } @@ -356,8 +346,8 @@ fn run_capability_landlock_child(_args: &[String]) -> Result<()> { /// One dedicated launcher thread installs the non-TSYNC listener, moves the /// listener FD to this unfiltered broker through an in-process channel, then /// execs the child. The child proves that the injected open-file description -/// survives dup and epoll registration before connect and that the broker can -/// return the original peer rather than the local relay endpoint. +/// survives dup and epoll registration before connect and that native +/// getpeername reports the endpoint used by the probe connection. #[cfg(target_os = "linux")] #[allow(unsafe_code)] fn probe_socket_virtualization() -> Result<()> { @@ -461,7 +451,6 @@ fn probe_socket_virtualization() -> Result<()> { openshell_isolation_interface::linux::seccomp_notify::install_listener(&[ libc::SYS_socket, libc::SYS_connect, - libc::SYS_getpeername, libc::SYS_sendto, ]); let Ok(listener) = listener else { @@ -516,7 +505,6 @@ fn probe_socket_virtualization() -> Result<()> { let mut observed_connect = false; let mut observed_dns_tcp_connect = false; let mut observed_denied_connect = false; - let mut observed_peer = false; let mut observed_dns_send = false; while !(observed_tcp_sockets == 3 @@ -524,7 +512,6 @@ fn probe_socket_virtualization() -> Result<()> { && observed_connect && observed_dns_tcp_connect && observed_denied_connect - && observed_peer && observed_dns_send) { let notification = listener @@ -650,27 +637,6 @@ fn probe_socket_virtualization() -> Result<()> { .respond_value(notification.id, 0) .into_diagnostic()?; } - libc::SYS_getpeername => { - let fd = i32::try_from(notification.args[0]) - .map_err(|_| miette::miette!("peer FD does not fit i32"))?; - let entry = registry.resolve(notification.tid, fd).into_diagnostic()?; - let SocketState::Connected { original_peer } = entry.state() else { - listener - .respond_errno(notification.id, libc::ENOTCONN) - .into_diagnostic()?; - return Err(miette::miette!("peer query preceded mediated connect")); - }; - write_probe_sockaddr( - notification.tid, - notification.args[1], - notification.args[2], - *original_peer, - )?; - listener - .respond_value(notification.id, 0) - .into_diagnostic()?; - observed_peer = true; - } libc::SYS_sendto => { let fd = i32::try_from(notification.args[0]) .map_err(|_| miette::miette!("sendto FD does not fit i32"))?; @@ -944,38 +910,6 @@ fn decode_probe_sockaddr(bytes: &[u8]) -> Result { ))) } -#[cfg(target_os = "linux")] -fn write_probe_sockaddr( - tid: u32, - address: u64, - length_address: u64, - peer: std::net::SocketAddr, -) -> Result<()> { - use std::mem::size_of; - - let (sockaddr, sockaddr_length) = encode_probe_sockaddr(peer)?; - let mut requested_length = [0_u8; size_of::()]; - openshell_isolation_interface::linux::task_memory::read_exact( - tid, - length_address, - &mut requested_length, - ) - .into_diagnostic()?; - let requested_length = libc::socklen_t::from_ne_bytes(requested_length); - if requested_length < sockaddr_length { - return Err(miette::miette!("peer sockaddr buffer is too small")); - } - openshell_isolation_interface::linux::task_memory::write_exact(tid, address, &sockaddr) - .into_diagnostic()?; - openshell_isolation_interface::linux::task_memory::write_exact( - tid, - length_address, - &sockaddr_length.to_ne_bytes(), - ) - .into_diagnostic()?; - Ok(()) -} - #[cfg(target_os = "linux")] #[allow(unsafe_code)] fn run_capability_socket_child(args: &[String]) -> Result<()> { diff --git a/crates/openshell-sandbox/src/network_broker.rs b/crates/openshell-sandbox/src/network_broker.rs index f2f196c925..8988ae3328 100644 --- a/crates/openshell-sandbox/src/network_broker.rs +++ b/crates/openshell-sandbox/src/network_broker.rs @@ -30,11 +30,9 @@ use tokio::sync::{mpsc, oneshot}; const SOCKET_CAPACITY: usize = 4_096; const SOCKET_FD_HEADROOM: usize = 64; const OPEN_QUEUE_CAPACITY: usize = 256; -const ACCEPT_WORKER_CAPACITY: usize = 64; const DNS_QUEUE_CAPACITY: usize = 256; const DNS_WORKER_CAPACITY: usize = 256; const DNS_QUERY_TIMEOUT: Duration = Duration::from_secs(10); -const ACCEPT_POLL_INTERVAL: Duration = Duration::from_millis(250); const DNS_RELAY_ADDRESS: SocketAddr = SocketAddr::V4(std::net::SocketAddrV4::new( Ipv4Addr::new(127, 0, 0, 53), 53, @@ -73,27 +71,6 @@ fn acquire_pending_dns_slot(active: &Arc) -> io::Result, -} - -impl Drop for PendingAcceptSlot { - fn drop(&mut self) { - self.active.fetch_sub(1, Ordering::AcqRel); - } -} - -fn acquire_pending_accept_slot(active: &Arc) -> io::Result { - active - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| { - (current < ACCEPT_WORKER_CAPACITY).then_some(current + 1) - }) - .map_err(|_| io::Error::from_raw_os_error(libc::EAGAIN))?; - Ok(PendingAcceptSlot { - active: Arc::clone(active), - }) -} - fn acquire_pending_open_slot(active: &Arc) -> io::Result { active .fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| { @@ -191,13 +168,12 @@ fn register_dns_socket( #[derive(Clone)] struct NotificationQueues { provider_files: crate::provider_files::ProviderFiles, + workload_frozen: Arc, protected_control_port: Option, - accept_registrar: crate::accept_interrupt::AcceptRegistrar, identity_resolver: ProcfsIdentityResolver, pending: mpsc::Sender, dns_relay: DnsRelay, active_opens: Arc, - active_accepts: Arc, retained_socket_capacity: usize, decision_timeout: Duration, } @@ -206,7 +182,7 @@ struct NotificationQueues { #[derive(Clone)] pub struct NetworkBroker { provider_files: crate::provider_files::ProviderFiles, - _accept_monitor: Arc, + workload_frozen: Arc, pending: Arc>>, pending_dns: Arc>>, dns_address: SocketAddr, @@ -250,28 +226,23 @@ impl NetworkBroker { decision_timeout: Duration, ) -> io::Result { let listener = Arc::new(listener); - let monitor_listener = listener.clone(); - let accept_monitor = Arc::new(crate::accept_interrupt::AcceptMonitor::start(move |id| { - monitor_listener.validate_id(id).is_ok() - })?); let (pending_tx, pending_rx) = mpsc::channel(OPEN_QUEUE_CAPACITY); let (pending_dns_tx, pending_dns_rx) = mpsc::channel(DNS_QUEUE_CAPACITY); let active_opens = Arc::new(AtomicUsize::new(0)); - let active_accepts = Arc::new(AtomicUsize::new(0)); let dns_relay = start_dns_relay(dns_address, pending_dns_tx)?; let dns_address = dns_relay.address; let retained_socket_capacity = retained_socket_capacity()?; let registry = Arc::new(Mutex::new(SocketRegistry::new(1, SOCKET_CAPACITY)?)); let provider_files = crate::provider_files::ProviderFiles::default(); + let workload_frozen = Arc::new(AtomicBool::new(false)); let queues = NotificationQueues { provider_files: provider_files.clone(), + workload_frozen: workload_frozen.clone(), protected_control_port, - accept_registrar: accept_monitor.registrar(), identity_resolver: ProcfsIdentityResolver::for_pid_namespace(), pending: pending_tx, dns_relay, active_opens, - active_accepts, retained_socket_capacity, decision_timeout, }; @@ -295,28 +266,52 @@ impl NetworkBroker { break; } }; - if let Err(error) = dispatch_notification( - Arc::clone(®istry), - Arc::clone(&listener), - notification, - queues.clone(), - ) { - tracing::warn!( - tid = notification.tid, - syscall = notification.syscall, - %error, - "sandbox network notification denied (tid={}, syscall={}): {error}", - notification.tid, - notification.syscall - ); - let _ = listener.respond_errno(notification.id, error_to_errno(&error)); + // Contain a handler panic so one faulty notification + // cannot silently kill the broker and hang every blocked + // workload syscall. The failing syscall gets an error; the + // broker keeps mediating the rest. + let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + dispatch_notification( + Arc::clone(®istry), + Arc::clone(&listener), + notification, + queues.clone(), + ) + })); + match outcome { + Ok(Ok(())) => {} + Ok(Err(error)) => { + tracing::warn!( + tid = notification.tid, + syscall = notification.syscall, + %error, + "sandbox network notification denied (tid={}, syscall={}): {error}", + notification.tid, + notification.syscall + ); + let _ = + listener.respond_errno(notification.id, error_to_errno(&error)); + } + Err(_) => { + tracing::error!( + tid = notification.tid, + syscall = notification.syscall, + "sandbox network notification handler panicked (tid={}, syscall={})", + notification.tid, + notification.syscall + ); + let _ = listener.respond_errno(notification.id, libc::EIO); + } } } + // The broker thread is exiting; dependent operations must fail + // closed rather than block on a listener no one services. + broker_healthy.store(false, Ordering::Release); }) .map_err(|error| io::Error::other(format!("start network broker: {error}")))?; Ok(Self { provider_files, - _accept_monitor: accept_monitor, + workload_frozen, pending: Arc::new(tokio::sync::Mutex::new(pending_rx)), pending_dns: Arc::new(tokio::sync::Mutex::new(pending_dns_rx)), dns_address, @@ -324,6 +319,14 @@ impl NetworkBroker { }) } + /// Record whether the boundary has stopped the workload for supervisor + /// recovery. While frozen, workload requests to send `SIGCONT` are refused + /// so a process that was not yet stopped cannot resume the others. Set + /// this before stopping the workload and clear it after resuming it. + pub(crate) fn set_workload_frozen(&self, frozen: bool) { + self.workload_frozen.store(frozen, Ordering::Release); + } + pub(crate) async fn accept(&self) -> io::Result { self.pending .lock() @@ -549,13 +552,18 @@ fn dispatch_notification( &listener, notification, std::process::id(), + &queues.workload_frozen, ); } - if syscall == libc::SYS_tkill { + if matches!( + syscall, + libc::SYS_tkill | libc::SYS_tgkill | libc::SYS_rt_tgsigqueueinfo + ) { return openshell_isolation_interface::linux::process_signal::mediate_thread_signal( &listener, notification, std::process::id(), + &queues.workload_frozen, ); } if syscall == libc::SYS_socket { @@ -575,32 +583,18 @@ fn dispatch_notification( if syscall == libc::SYS_listen { return listen_socket(®istry, &listener, notification); } - if matches!(syscall, libc::SYS_accept | libc::SYS_accept4) { - return accept_socket( - registry, - listener, - notification, - queues.active_accepts, - queues.accept_registrar, - ); - } if matches!( syscall, libc::SYS_sendto | libc::SYS_sendmsg | libc::SYS_sendmmsg ) { return classify_send(®istry, &listener, notification, &queues.dns_relay); } - if syscall == libc::SYS_getpeername { - return get_peer_name(®istry, &listener, notification); - } if syscall == libc::SYS_setsockopt { let level = i32::try_from(notification.args[1]) .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; let option = i32::try_from(notification.args[2]) .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; - if (level == libc::IPPROTO_TCP && option == libc::TCP_FASTOPEN_CONNECT) - || (level == libc::IPPROTO_IPV6 && option == libc::IPV6_ADDRFORM) - { + if socket_option_is_denied(level, option) { return Err(io::Error::from_raw_os_error(libc::EPERM)); } return listener.respond_continue(notification.id); @@ -608,6 +602,32 @@ fn dispatch_notification( Err(io::Error::from_raw_os_error(libc::EPERM)) } +/// Options the workload may never set, decided from scalar syscall arguments +/// that another thread cannot replace before the kernel reads them. +/// +/// Interface-selection options could redirect or unpin a socket's loopback +/// device binding; the others enable Fast Open or change the address family. The kernel already refuses to change an existing binding +/// without `CAP_NET_RAW`; denying them here keeps confinement independent of +/// the capability state of the namespace that owns the network namespace. +fn socket_option_is_denied(level: i32, option: i32) -> bool { + matches!( + (level, option), + (libc::IPPROTO_TCP, libc::TCP_FASTOPEN_CONNECT) + | ( + libc::IPPROTO_IPV6, + libc::IPV6_ADDRFORM | libc::IPV6_UNICAST_IF | libc::IPV6_MULTICAST_IF + ) + | ( + libc::SOL_SOCKET, + libc::SO_BINDTODEVICE | libc::SO_BINDTOIFINDEX + ) + | ( + libc::IPPROTO_IP, + libc::IP_UNICAST_IF | libc::IP_MULTICAST_IF + ) + ) +} + fn create_socket( registry: &Mutex, listener: &NotificationListener, @@ -617,7 +637,13 @@ fn create_socket( let domain = i32::try_from(notification.args[0]) .map_err(|_| io::Error::from_raw_os_error(libc::EAFNOSUPPORT))?; if !matches!(domain, libc::AF_INET | libc::AF_INET6) { - return listener.respond_continue(notification.id); + // The workload filter already refuses other families. Repeat the + // decision here so a filter change cannot let a kernel transport + // socket bypass loopback confinement; the domain is a scalar argument. + if matches!(domain, libc::AF_UNIX | libc::AF_NETLINK) { + return listener.respond_continue(notification.id); + } + return Err(io::Error::from_raw_os_error(libc::EAFNOSUPPORT)); } let raw_kind = i32::try_from(notification.args[1]) .map_err(|_| io::Error::from_raw_os_error(libc::EPROTONOSUPPORT))?; @@ -654,6 +680,10 @@ fn create_socket( } // SAFETY: successful socket returned one owned descriptor. let source = unsafe { OwnedFd::from_raw_fd(source) }; + // Confinement is standing kernel state that must exist before the workload + // can observe the descriptor. Natively accepted children inherit it, so + // local accept needs no per-connection broker inspection. + openshell_isolation_interface::linux::socket_confinement::confine_to_loopback(&source)?; let metadata = SocketMetadata { family, kind, @@ -773,15 +803,21 @@ fn connect_socket( let destination = read_socket_addr(notification.tid, notification.args[1], notification.args[2])?; reject_protected_control_destination(destination, protected_control_port)?; - let (kind, socket_identity, nonblocking) = { + let (kind, socket_identity, nonblocking, repeated) = { let registry = lock(®istry); let entry = registry.resolve(notification.tid, fd)?; ( entry.metadata().kind, entry.identity(), entry.metadata().nonblocking, + repeated_connect_outcome(entry.state(), entry.metadata().kind, destination), ) }; + match repeated { + Some(0) => return listener.respond_value(notification.id, 0), + Some(errno) => return Err(io::Error::from_raw_os_error(errno)), + None => {} + } if kind == InetKind::DnsUdp && destination.port() == 0 { let mut registry = lock(®istry); let entry = registry.resolve_mut(notification.tid, fd)?; @@ -798,6 +834,7 @@ fn connect_socket( if entry.metadata().family != destination_family { return Err(io::Error::from_raw_os_error(libc::EAFNOSUPPORT)); } + listener.validate_id(notification.id)?; // glibc and uv use UDP connect(..., port 0), getsockname(), and an // AF_UNSPEC disconnect to rank resolved addresses. Bind only to the // matching loopback family and report success; never connect the @@ -820,6 +857,7 @@ fn connect_socket( return Err(io::Error::from_raw_os_error(libc::EISCONN)); } let source_fd = entry.retained_preconnect()?.as_raw_fd(); + listener.validate_id(notification.id)?; let peer = ensure_dns_source_bound(source_fd, entry.metadata().family)?; let admissions = match kind { InetKind::Tcp => &dns_relay.tcp_admissions, @@ -842,8 +880,21 @@ fn connect_socket( if destination.ip().is_loopback() && !openshell_core::google_cloud::is_metadata_destination(destination) { + if kind == InetKind::Tcp { + return connect_local_tcp( + ®istry, + &listener, + notification, + fd, + socket_identity, + destination, + &active_opens, + ); + } + // A UDP connect completes immediately. let mut registry = lock(®istry); let entry = registry.resolve_mut(notification.tid, fd)?; + listener.validate_id(notification.id)?; connect_exact(entry.retained_preconnect()?.as_raw_fd(), destination)?; entry.set_state(SocketState::Local { peer: destination }); entry.release_preconnect(); @@ -928,6 +979,136 @@ fn connect_socket( Ok(()) } +/// Connect a workload TCP socket to a loopback endpoint without blocking the +/// notification dispatcher. +/// +/// The broker connects a duplicate of its retained socket, which shares the +/// workload's open file, and never holds the registry lock while waiting. A +/// nonblocking socket gets the native `EINPROGRESS` and the kernel completes +/// the handshake on the shared socket; a blocking socket waits on a bounded +/// worker thread. A slow or full local listener therefore cannot stall +/// mediation of unrelated syscalls. A nonblocking socket is recorded as +/// connected once the handshake starts, so a repeated `connect` reports +/// `EISCONN` even while the handshake is still in progress. +fn connect_local_tcp( + registry: &Arc>, + listener: &Arc, + notification: Notification, + fd: RawFd, + socket_identity: SocketIdentity, + destination: SocketAddr, + active_opens: &Arc, +) -> io::Result<()> { + let connector = { + let registry = lock(registry); + let entry = registry.resolve(notification.tid, fd)?; + rustix::io::fcntl_dupfd_cloexec(entry.retained_preconnect()?, 3)? + }; + listener.validate_id(notification.id)?; + // SAFETY: F_GETFL reads the flags of the live shared open file. + let flags = unsafe { libc::fcntl(connector.as_raw_fd(), libc::F_GETFL) }; + if flags < 0 { + return Err(io::Error::last_os_error()); + } + if flags & libc::O_NONBLOCK != 0 { + let started = with_sockaddr(destination, |pointer, length| { + // SAFETY: pointer/length describe a live sockaddr; the connector + // is a live duplicate of the workload's socket. + if unsafe { libc::connect(connector.as_raw_fd(), pointer, length) } == 0 { + Ok(()) + } else { + Err(io::Error::last_os_error()) + } + }); + let in_progress = started + .as_ref() + .is_err_and(|error| error.raw_os_error() == Some(libc::EINPROGRESS)); + if started.is_ok() || in_progress { + commit_local_connect(registry, notification.tid, fd, socket_identity, destination); + } + return match started { + Ok(()) => listener.respond_value(notification.id, 0), + Err(error) => Err(error), + }; + } + let slot = acquire_pending_open_slot(active_opens)?; + let registry = Arc::clone(registry); + let worker_listener = Arc::clone(listener); + std::thread::Builder::new() + .name("openshell-local-connect".to_string()) + .spawn(move || { + let _slot = slot; + // The duplicate shares the workload's blocking open file; wait + // natively without changing its flags. + let result = with_sockaddr(destination, |pointer, length| { + // SAFETY: pointer/length describe a live sockaddr and the + // connector is a live duplicate of the workload's socket. + if unsafe { libc::connect(connector.as_raw_fd(), pointer, length) } == 0 { + Ok(()) + } else { + Err(io::Error::last_os_error()) + } + }); + if result.is_ok() { + commit_local_connect( + ®istry, + notification.tid, + fd, + socket_identity, + destination, + ); + } + let _ = match result { + Ok(()) => worker_listener.respond_value(notification.id, 0), + Err(error) => { + worker_listener.respond_errno(notification.id, error_to_errno(&error)) + } + }; + }) + .map_err(|error| io::Error::other(format!("start local-connect worker: {error}")))?; + Ok(()) +} + +/// Record a completed or in-progress loopback TCP connect, unless the +/// descriptor now names a different socket. +fn commit_local_connect( + registry: &Mutex, + tid: u32, + fd: RawFd, + socket_identity: SocketIdentity, + destination: SocketAddr, +) { + let mut registry = lock(registry); + if let Ok(entry) = registry.resolve_mut(tid, fd) + && entry.identity() == socket_identity + { + entry.set_state(SocketState::Local { peer: destination }); + entry.release_preconnect(); + } +} + +/// Result for a `connect` on a socket the broker already connected, as the +/// kernel would report it: `Some(0)` for success, `Some(errno)` for an error, +/// `None` when the socket is not yet connected. +/// +/// A notified syscall interrupted by a signal is restarted after the broker +/// may already have completed it, so a repeat must not depend on the broker's +/// released pre-connect descriptor. +fn repeated_connect_outcome( + state: &SocketState, + kind: InetKind, + destination: SocketAddr, +) -> Option { + match state { + SocketState::Created | SocketState::Bound { .. } | SocketState::Listening { .. } => None, + // UDP connect replaces the association; repeating the same one succeeds. + SocketState::DnsUdp { relay } if *relay == destination => Some(0), + SocketState::Local { peer } if kind == InetKind::DnsUdp && *peer == destination => Some(0), + _ if kind == InetKind::Tcp => Some(libc::EISCONN), + _ => None, + } +} + const fn tcp_denial_errno(reason: TcpOpenDenial) -> i32 { match reason { TcpOpenDenial::PolicyDenied @@ -1037,6 +1218,12 @@ fn bind_socket( let bind_result = { let mut registry = lock(registry); let entry = registry.resolve_mut(notification.tid, fd)?; + // A native bind is never restarted, but a notified one can be after a + // signal. Report a repeat of the bind the broker completed as success. + if entry.state() == &(SocketState::Bound { local }) { + return listener.respond_value(notification.id, 0); + } + listener.validate_id(notification.id)?; bind_exact(entry.retained_preconnect()?.as_raw_fd(), local) }; if bind_result @@ -1082,6 +1269,8 @@ fn listen_socket( let Ok(entry) = registry.resolve_mut(notification.tid, fd) else { return listener.respond_continue(notification.id); }; + // listen(2) may be repeated natively, so a restart needs no special case. + listener.validate_id(notification.id)?; // SAFETY: retained descriptor is the exact registered socket OFD. if unsafe { libc::listen(entry.retained_preconnect()?.as_raw_fd(), backlog) } < 0 { return Err(io::Error::last_os_error()); @@ -1091,218 +1280,6 @@ fn listen_socket( listener.respond_value(notification.id, 0) } -fn accept_socket( - registry: Arc>, - listener: Arc, - notification: Notification, - active_accepts: Arc, - accept_registrar: crate::accept_interrupt::AcceptRegistrar, -) -> io::Result<()> { - let fd = raw_fd(notification.args[0])?; - let flags = if i64::from(notification.syscall) == libc::SYS_accept4 { - i32::try_from(notification.args[3]) - .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))? - } else { - 0 - }; - if flags & !(libc::SOCK_CLOEXEC | libc::SOCK_NONBLOCK) != 0 { - return Err(io::Error::from_raw_os_error(libc::EINVAL)); - } - if (notification.args[1] == 0) != (notification.args[2] == 0) { - return Err(io::Error::from_raw_os_error(libc::EFAULT)); - } - let (listener_inode, metadata, source) = { - let registry = lock(®istry); - let Ok(entry) = registry.resolve(notification.tid, fd) else { - return listener.respond_continue(notification.id); - }; - if !matches!(entry.state(), SocketState::Listening { .. }) - || entry.metadata().kind != InetKind::Tcp - { - return Err(io::Error::from_raw_os_error(libc::EINVAL)); - } - let source = duplicate_close_on_exec(entry.retained_preconnect()?.as_raw_fd())?; - (entry.identity().inode, entry.metadata(), source) - }; - let slot = acquire_pending_accept_slot(&active_accepts)?; - let worker_listener = Arc::clone(&listener); - std::thread::Builder::new() - .name("openshell-local-accept".to_string()) - .spawn(move || { - let _slot = slot; - let registration = match accept_registrar.register(notification.id) { - Ok(registration) => registration, - Err(error) => { - let _ = worker_listener.respond_errno(notification.id, error_to_errno(&error)); - return; - } - }; - if let Err(error) = accept_and_inject( - ®istry, - &worker_listener, - notification, - AcceptOperation { - flags, - listener_inode, - metadata, - source, - registration, - }, - ) { - let _ = worker_listener.respond_errno(notification.id, error_to_errno(&error)); - } - }) - .map_err(|error| io::Error::other(format!("start local-accept worker: {error}")))?; - Ok(()) -} - -struct AcceptOperation { - flags: i32, - listener_inode: u64, - metadata: SocketMetadata, - source: OwnedFd, - registration: crate::accept_interrupt::AcceptRegistration, -} - -fn accept_and_inject( - registry: &Mutex, - listener: &NotificationListener, - notification: Notification, - operation: AcceptOperation, -) -> io::Result<()> { - let AcceptOperation { - flags, - listener_inode, - metadata, - source, - registration, - } = operation; - let mut poll = libc::pollfd { - fd: source.as_raw_fd(), - events: libc::POLLIN, - revents: 0, - }; - // SAFETY: F_GETFL reads the live listener OFD flags. - let current_flags = unsafe { libc::fcntl(source.as_raw_fd(), libc::F_GETFL) }; - if current_flags < 0 { - return Err(io::Error::last_os_error()); - } - let nonblocking = current_flags & libc::O_NONBLOCK != 0; - let timeout = if nonblocking { - 0 - } else { - i32::try_from(ACCEPT_POLL_INTERVAL.as_millis()).map_err(io::Error::other)? - }; - // Readiness may disappear before accept (another accept or an aborted - // connection). The registered watchdog interrupts a blocked syscall when - // its notification dies or the broker shuts down. No workload OFD flags - // are changed, and no worker can outlive its cancellation registration. - loop { - registration.ensure_running()?; - listener.validate_id(notification.id)?; - // SAFETY: poll references one live pollfd for this call. - let ready = unsafe { libc::poll(&raw mut poll, 1, timeout) }; - if ready < 0 { - let error = io::Error::last_os_error(); - if error.kind() == io::ErrorKind::Interrupted { - continue; - } - return Err(error); - } - if ready == 0 { - if nonblocking { - return Err(io::Error::from_raw_os_error(libc::EAGAIN)); - } - continue; - } - break; - } - - let mut storage = std::mem::MaybeUninit::::zeroed(); - let mut length = - libc::socklen_t::try_from(size_of::()).map_err(io::Error::other)?; - // Always keep the broker-side descriptor close-on-exec. ADDFD separately - // applies the workload's requested descriptor flag. - let accepted_flags = flags | libc::SOCK_CLOEXEC; - // SAFETY: storage and length are live outputs and source is a listening - // socket proven by the registry. - let accepted = unsafe { - libc::accept4( - source.as_raw_fd(), - storage.as_mut_ptr().cast(), - &raw mut length, - accepted_flags, - ) - }; - if accepted < 0 { - return Err(io::Error::last_os_error()); - } - // Only the blocking accept phase needs asynchronous interruption. Stop - // monitoring before ADDFD completes the notification, otherwise a normal - // successful response could be mistaken for cancellation during commit. - drop(registration); - // SAFETY: successful accept4 returned one newly owned descriptor. - let accepted = unsafe { OwnedFd::from_raw_fd(accepted) }; - // SAFETY: accept4 initialized the reported prefix of storage. - let peer = decode_sockaddr( - unsafe { storage.assume_init() }, - usize::try_from(length).unwrap_or(0), - )?; - if !peer.ip().is_loopback() { - return Err(io::Error::from_raw_os_error(libc::EACCES)); - } - if notification.args[1] != 0 { - write_socket_addr( - listener, - notification.id, - notification.tid, - notification.args[1], - notification.args[2], - peer, - )?; - } - - let accepted_metadata = SocketMetadata { - family: metadata.family, - kind: InetKind::Tcp, - close_on_exec: flags & libc::SOCK_CLOEXEC != 0, - nonblocking: flags & libc::SOCK_NONBLOCK != 0, - creator_generation: u64::from(notification.tid), - }; - let mut registry = lock(registry); - let notifying_fd = raw_fd(notification.args[0])?; - if registry - .resolve(notification.tid, notifying_fd)? - .identity() - .inode - != listener_inode - { - return Err(io::Error::from_raw_os_error(libc::EBADF)); - } - if registry.is_full() { - collect_closed_socket_entries_locked(&mut registry)?; - } - let tentative = registry.stage(accepted, accepted_metadata)?; - listener.add_fd_and_send( - notification.id, - tentative.source_fd(), - accepted_metadata.close_on_exec, - )?; - registry.commit_with_state(tentative, SocketState::AcceptedLocal { peer })?; - Ok(()) -} - -fn duplicate_close_on_exec(fd: RawFd) -> io::Result { - // SAFETY: F_DUPFD_CLOEXEC returns an independent owned descriptor for the - // same open-file description. - let duplicate = unsafe { libc::fcntl(fd, libc::F_DUPFD_CLOEXEC, 3) }; - if duplicate < 0 { - return Err(io::Error::last_os_error()); - } - // SAFETY: successful fcntl returned one newly owned descriptor. - Ok(unsafe { OwnedFd::from_raw_fd(duplicate) }) -} - fn classify_send( registry: &Mutex, listener: &NotificationListener, @@ -1311,6 +1288,12 @@ fn classify_send( ) -> io::Result<()> { let fd = raw_fd(notification.args[0])?; let syscall = i64::from(notification.syscall); + // Fast Open turns a send into a connect. Decide from the scalar flags + // argument, which another thread cannot replace, so the denial also + // covers natively accepted and other unregistered descriptors. + if send_flags(syscall, notification.args) & libc::MSG_FASTOPEN != 0 { + return Err(io::Error::from_raw_os_error(libc::EPERM)); + } let (state, metadata) = { let registry = lock(registry); let Ok(entry) = registry.resolve(notification.tid, fd) else { @@ -1320,10 +1303,8 @@ fn classify_send( }; (entry.state().clone(), entry.metadata()) }; - if matches!( - &state, - SocketState::Connected { .. } | SocketState::AcceptedLocal { .. } - ) || (metadata.kind == InetKind::Tcp && matches!(&state, SocketState::Local { .. })) + if matches!(&state, SocketState::Connected { .. }) + || (metadata.kind == InetKind::Tcp && matches!(&state, SocketState::Local { .. })) { return listener.respond_continue(notification.id); } @@ -1332,9 +1313,6 @@ fn classify_send( libc::SYS_sendmsg => vec![read_sendmsg_message( notification.tid, notification.args[1], - i32::try_from(notification.args[2]) - .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?, - None, )?], libc::SYS_sendmmsg => read_sendmmsg_messages(notification)?, _ => return Err(io::Error::from_raw_os_error(libc::ENOSYS)), @@ -1388,58 +1366,52 @@ fn classify_send( { let entry = registry.resolve_mut(notification.tid, fd)?; let source_fd = entry.retained_preconnect()?.as_raw_fd(); + listener.validate_id(notification.id)?; let peer = ensure_dns_source_bound(source_fd, entry.metadata().family)?; register_dns_socket(&dns_relay.udp_admissions, peer, entry.identity())?; if let Err(error) = connect_exact(source_fd, dns_relay.address) { lock(&dns_relay.udp_admissions).remove(&peer); return Err(error); } - for message in &messages { - send_dns_message(source_fd, message)?; - if let Some(length_address) = message.result_length_address { - let length = u32::try_from(message.data.len()) - .map_err(|_| io::Error::from_raw_os_error(libc::EMSGSIZE))?; - listener.write_task_output( - notification.id, - notification.tid, - length_address, - &length.to_ne_bytes(), - )?; - } - } + // The socket is pinned to the relay and bound to loopback. The + // kernel performs the send; ancillary data was rejected at read + // time and a loopback destination contains any per-message + // routing override that races the check. entry.set_state(SocketState::DnsUdp { relay: dns_relay.address, }); entry.release_preconnect(); - let result = if syscall == libc::SYS_sendmmsg { - i64::try_from(messages.len()).unwrap_or(i64::MAX) - } else { - i64::try_from(messages[0].data.len()).unwrap_or(i64::MAX) - }; - listener.respond_value(notification.id, result) + listener.respond_continue(notification.id) } Ok(_) => Err(io::Error::from_raw_os_error(libc::EDESTADDRREQ)), - // Non-INET sockets and accepted local sockets were never registered. - // The mandatory outer fence still prevents an external kernel route. + // Non-INET sockets and natively accepted sockets were never + // registered. Accepted sockets inherit their listener's loopback + // binding, and the mandatory outer fence remains an independent + // backstop against an external kernel route. Err(_) => listener.respond_continue(notification.id), } } +fn send_flags(syscall: i64, args: [u64; 6]) -> i32 { + let flags = match syscall { + libc::SYS_sendmsg => args[2], + libc::SYS_sendto | libc::SYS_sendmmsg => args[3], + _ => 0, + }; + // Syscall flag arguments are C ints; the kernel ignores the upper word. + #[allow( + clippy::cast_possible_truncation, + reason = "the kernel reads only the low 32 bits of the flags argument" + )] + let flags = flags as u32; + flags.cast_signed() +} + struct SendMessage { - data: Vec, destination: Option, - flags: i32, - result_length_address: Option, } fn read_sendto_message(notification: Notification) -> io::Result { - let length = usize::try_from(notification.args[2]) - .map_err(|_| io::Error::from_raw_os_error(libc::EMSGSIZE))?; - if u16::try_from(length).is_err() { - return Err(io::Error::from_raw_os_error(libc::EMSGSIZE)); - } - let mut data = vec![0_u8; length]; - task_memory::read_exact(notification.tid, notification.args[1], &mut data)?; let destination = if notification.args[4] == 0 { None } else { @@ -1449,22 +1421,15 @@ fn read_sendto_message(notification: Notification) -> io::Result { notification.args[5], )?) }; - Ok(SendMessage { - data, - destination, - flags: i32::try_from(notification.args[3]) - .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?, - result_length_address: None, - }) + Ok(SendMessage { destination }) } -fn read_sendmsg_message( - tid: u32, - address: u64, - flags: i32, - result_length_address: Option, -) -> io::Result { +fn read_sendmsg_message(tid: u32, address: u64) -> io::Result { let header = read_task_value::(tid, address)?; + // Ancillary data can carry a per-message routing override (IP_PKTINFO). + // Refuse it rather than continue a send the broker did not inspect; a + // loopback destination additionally contains an override that races this + // check. if header.msg_controllen != 0 { return Err(io::Error::from_raw_os_error(libc::EOPNOTSUPP)); } @@ -1477,39 +1442,7 @@ fn read_sendmsg_message( u64::from(header.msg_namelen), )?) }; - #[cfg(target_env = "musl")] - let iov_count = usize::try_from(header.msg_iovlen) - .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; - #[cfg(not(target_env = "musl"))] - let iov_count = header.msg_iovlen; - if iov_count > 32 { - return Err(io::Error::from_raw_os_error(libc::EMSGSIZE)); - } - let mut data = Vec::new(); - for index in 0..iov_count { - let offset = index - .checked_mul(size_of::()) - .ok_or_else(|| io::Error::from_raw_os_error(libc::EOVERFLOW))?; - let iov = read_task_value::( - tid, - (header.msg_iov as u64) - .checked_add(u64::try_from(offset).unwrap_or(u64::MAX)) - .ok_or_else(|| io::Error::from_raw_os_error(libc::EOVERFLOW))?, - )?; - let start = data.len(); - let end = start - .checked_add(iov.iov_len) - .filter(|length| u16::try_from(*length).is_ok()) - .ok_or_else(|| io::Error::from_raw_os_error(libc::EMSGSIZE))?; - data.resize(end, 0); - task_memory::read_exact(tid, iov.iov_base as u64, &mut data[start..end])?; - } - Ok(SendMessage { - data, - destination, - flags, - result_length_address, - }) + Ok(SendMessage { destination }) } fn read_sendmmsg_messages(notification: Notification) -> io::Result> { @@ -1518,8 +1451,6 @@ fn read_sendmmsg_messages(notification: Notification) -> io::Result 32 { return Err(io::Error::from_raw_os_error(libc::EMSGSIZE)); } - let flags = i32::try_from(notification.args[3]) - .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; (0..count) .map(|index| { let offset = index @@ -1528,18 +1459,7 @@ fn read_sendmmsg_messages(notification: Notification) -> io::Result(tid: u32, address: u64) -> io::Result { Ok(unsafe { std::ptr::read_unaligned(bytes.as_ptr().cast::()) }) } -fn send_dns_message(fd: RawFd, message: &SendMessage) -> io::Result<()> { - // SAFETY: `fd` is the retained exact UDP socket and the buffer remains - // valid for the duration of the syscall. - let sent = unsafe { - libc::send( - fd, - message.data.as_ptr().cast(), - message.data.len(), - message.flags, - ) - }; - if sent < 0 { - return Err(io::Error::last_os_error()); - } - if usize::try_from(sent).ok() == Some(message.data.len()) { - Ok(()) - } else { - Err(io::Error::from_raw_os_error(libc::EIO)) - } -} - -fn get_peer_name( - registry: &Mutex, - listener: &NotificationListener, - notification: Notification, -) -> io::Result<()> { - let fd = raw_fd(notification.args[0])?; - let registry = lock(registry); - let Ok(entry) = registry.resolve(notification.tid, fd) else { - return listener.respond_continue(notification.id); - }; - let peer = match entry.state() { - SocketState::Connected { original_peer } => *original_peer, - SocketState::Local { peer } | SocketState::AcceptedLocal { peer } => *peer, - _ => return Err(io::Error::from_raw_os_error(libc::ENOTCONN)), - }; - write_socket_addr( - listener, - notification.id, - notification.tid, - notification.args[1], - notification.args[2], - peer, - )?; - listener.respond_value(notification.id, 0) -} - fn connect_exact(fd: RawFd, address: SocketAddr) -> io::Result<()> { // Never let a blocking connect pin the single notification dispatcher. // O_NONBLOCK is an OFD flag, so restore the workload's original setting @@ -1745,50 +1618,6 @@ fn decode_sockaddr(storage: libc::sockaddr_storage, length: usize) -> io::Result } } -fn write_socket_addr( - listener: &NotificationListener, - notification_id: u64, - tid: u32, - address: u64, - length_address: u64, - value: SocketAddr, -) -> io::Result<()> { - // A LegacyReadOnly listener (kernels < 5.19) cannot safely write into - // workload memory: without WAIT_KILLABLE_RECV the notified accept/ - // getpeername could resume and repurpose these buffers between validation - // and the broker write. Fail closed before reading or writing anything, so - // this address-writing path is inert in legacy mode. Callers that pass a - // null address argument (accept with a null peer address) never reach here. - if listener.writes_disabled() { - return Err(io::Error::from_raw_os_error(libc::EOPNOTSUPP)); - } - let mut supplied_length = [0_u8; size_of::()]; - task_memory::read_exact(tid, length_address, &mut supplied_length)?; - let supplied_length = libc::socklen_t::from_ne_bytes(supplied_length); - let (bytes, actual_length) = sockaddr_bytes(value)?; - let copied = usize::try_from(supplied_length) - .unwrap_or(0) - .min(bytes.len()); - if copied != 0 { - listener.write_task_output(notification_id, tid, address, &bytes[..copied])?; - } - listener.write_task_output( - notification_id, - tid, - length_address, - &actual_length.to_ne_bytes(), - ) -} - -fn sockaddr_bytes(address: SocketAddr) -> io::Result<(Vec, libc::socklen_t)> { - with_sockaddr(address, |native, length| { - let length_usize = usize::try_from(length).map_err(io::Error::other)?; - // SAFETY: with_sockaddr lends fully initialized storage for this call. - let bytes = unsafe { std::slice::from_raw_parts(native.cast::(), length_usize) }; - Ok((bytes.to_vec(), length)) - }) -} - fn with_sockaddr( address: SocketAddr, operation: impl FnOnce(*const libc::sockaddr, libc::socklen_t) -> io::Result, @@ -1846,7 +1675,7 @@ fn error_to_errno(error: &io::Error) -> i32 { #[cfg(test)] mod tests { use super::*; - use openshell_isolation_interface::linux::seccomp_notify::ListenerMode; + use openshell_isolation_interface::linux::socket_confinement; #[test] fn provider_files_are_opened_on_demand_and_replaced() { @@ -2026,25 +1855,6 @@ mod tests { ))); } - #[test] - fn legacy_listener_rejects_socket_addr_write() { - // accept-with-address and getpeername both route through - // write_socket_addr; on a LegacyReadOnly listener the path must fail - // closed (EOPNOTSUPP) before any task-memory access. - // SAFETY: dup returns a new descriptor or a negative error. - let dup = unsafe { libc::dup(libc::STDERR_FILENO) }; - assert!(dup >= 0, "dup stderr"); - let listener = NotificationListener::from_fd_with_mode( - // SAFETY: successful dup returned a new owned descriptor. - unsafe { OwnedFd::from_raw_fd(dup) }, - ListenerMode::LegacyReadOnly, - ); - let peer: SocketAddr = "127.0.0.1:8080".parse().unwrap(); - let error = write_socket_addr(&listener, 1, 0, 0, 0, peer) - .expect_err("legacy listener must reject socket-address writes"); - assert_eq!(error.raw_os_error(), Some(libc::EOPNOTSUPP)); - } - #[test] fn relay_rejects_descriptor_replaced_after_policy_decision() { let metadata = SocketMetadata { @@ -2056,11 +1866,10 @@ mod tests { }; let mut registry = SocketRegistry::new(1, 2).unwrap(); let mut create = || { - // SAFETY: a successful socket call returns a new owned descriptor. - let fd = unsafe { libc::socket(libc::AF_INET, libc::SOCK_STREAM, 0) }; - assert!(fd >= 0); - let socket = unsafe { OwnedFd::from_raw_fd(fd) }; - let installed = duplicate_close_on_exec(fd).unwrap(); + let socket = OwnedFd::from( + socket2::Socket::new(socket2::Domain::IPV4, socket2::Type::STREAM, None).unwrap(), + ); + let installed = rustix::io::fcntl_dupfd_cloexec(&socket, 3).unwrap(); let tentative = registry.stage(socket, metadata).unwrap(); let identity = registry.commit(tentative).unwrap(); (installed, identity) @@ -2383,42 +2192,39 @@ mod tests { } #[test] - fn accepted_loopback_stream_is_registered_for_notified_operations() { + fn native_accept_inherits_loopback_confinement() { let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() .expect("start workload launcher"); let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); let (ready_tx, ready_rx) = std::sync::mpsc::sync_channel(1); let workload = std::thread::spawn(move || { launcher - .execute(move || -> io::Result { - let listener = TcpListener::bind("127.0.0.1:0")?; - ready_tx - .send(listener.local_addr()?) - .map_err(|_| io::Error::other("test client disappeared"))?; - let (stream, _) = listener.accept()?; - let peer = stream.peer_addr()?; - let payload = b"accepted"; - let iov = libc::iovec { - iov_base: payload.as_ptr().cast_mut().cast(), - iov_len: payload.len(), - }; - let message = libc::msghdr { - msg_name: std::ptr::null_mut(), - msg_namelen: 0, - msg_iov: (&raw const iov).cast_mut(), - msg_iovlen: 1, - msg_control: std::ptr::null_mut(), - msg_controllen: 0, - msg_flags: 0, - }; - // SAFETY: message references one live immutable payload; - // the accepted stream remains open for the call. - let sent = unsafe { libc::sendmsg(stream.as_raw_fd(), &raw const message, 0) }; - if sent != isize::try_from(payload.len()).expect("payload fits isize") { - return Err(io::Error::last_os_error()); - } - Ok(peer) - }) + .execute( + move || -> io::Result<(SocketAddr, SocketAddr, Option>)> { + let listener = TcpListener::bind("127.0.0.1:0")?; + ready_tx + .send(listener.local_addr()?) + .map_err(|_| io::Error::other("test client disappeared"))?; + // std passes a peer-address buffer; native accept + // fills it directly from the kernel. + let (stream, accepted_peer) = listener.accept()?; + let peer = stream.peer_addr()?; + let device = socket_confinement::bound_device(&stream)?; + // sendmsg with no destination is notified and must + // continue for an untracked accepted stream. + let payload = b"accepted"; + let sent = rustix::net::sendmsg( + &stream, + &[io::IoSlice::new(payload)], + &mut rustix::net::SendAncillaryBuffer::default(), + rustix::net::SendFlags::empty(), + )?; + if sent != payload.len() { + return Err(io::Error::from_raw_os_error(libc::EIO)); + } + Ok((accepted_peer, peer, device)) + }, + ) .expect("launcher result") }); @@ -2435,14 +2241,360 @@ mod tests { .read_exact(&mut payload) .expect("read accepted stream"); assert_eq!(&payload, b"accepted"); + let (accepted_peer, peer, device) = workload + .join() + .expect("join workload") + .expect("accepted workload"); + let client_address = client.local_addr().unwrap(); + assert_eq!(accepted_peer, client_address); + assert_eq!(peer, client_address); + assert_eq!(device.as_deref(), Some(&b"lo"[..])); + } + + /// Bound device name and the errno from an attempted rebind. + type ConfinementObservation = (Option>, Option); + + #[test] + fn workload_sockets_are_bound_to_loopback_and_cannot_be_rebound() { + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let results = launcher + .execute(|| -> io::Result> { + let mut results = Vec::new(); + for (domain, kind) in [ + (socket2::Domain::IPV4, socket2::Type::STREAM), + (socket2::Domain::IPV4, socket2::Type::DGRAM), + (socket2::Domain::IPV6, socket2::Type::STREAM), + ] { + let socket = socket2::Socket::new(domain, kind, None)?; + let device = socket_confinement::bound_device(&socket)?; + let rebind_error = socket + .bind_device(Some(b"eth0")) + .err() + .and_then(|error| error.raw_os_error()); + results.push((device, rebind_error)); + } + Ok(results) + }) + .expect("launcher result") + .expect("workload sockets"); + for (device, rebind_error) in results { + assert_eq!(device.as_deref(), Some(&b"lo"[..])); + assert_eq!(rebind_error, Some(libc::EPERM)); + } + } + + #[test] + fn fast_open_sends_are_denied_for_every_descriptor() { + let local = TcpListener::bind("127.0.0.1:0").unwrap(); + local.set_nonblocking(true).unwrap(); + let address = local.local_addr().unwrap(); + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let error = launcher + .execute(move || -> io::Result<()> { + let socket = + socket2::Socket::new(socket2::Domain::IPV4, socket2::Type::STREAM, None)?; + socket + .send_to_with_flags(b"x", &address.into(), libc::MSG_FASTOPEN) + .map(drop) + }) + .unwrap() + .unwrap_err(); + assert_eq!(error.raw_os_error(), Some(libc::EPERM)); + assert_eq!( + local.accept().unwrap_err().kind(), + io::ErrorKind::WouldBlock + ); + } + + #[test] + fn send_flags_read_the_scalar_argument_for_each_syscall() { + let flags = u64::try_from(libc::MSG_FASTOPEN).unwrap(); + assert_eq!( + send_flags(libc::SYS_sendto, [0, 0, 0, flags, 0, 0]), + libc::MSG_FASTOPEN + ); + assert_eq!( + send_flags(libc::SYS_sendmsg, [0, 0, flags, 0, 0, 0]), + libc::MSG_FASTOPEN + ); + assert_eq!( + send_flags(libc::SYS_sendmmsg, [0, 0, 0, flags | (1 << 32), 0, 0]), + libc::MSG_FASTOPEN + ); + } + + #[test] + fn broker_refuses_socket_families_it_cannot_confine() { + // Independent of the static workload filter: the broker continues + // only Unix and netlink sockets and creates INET sockets itself. + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let results = launcher + .execute(|| { + [ + (libc::AF_UNIX, socket2::Type::STREAM), + (libc::AF_RXRPC, socket2::Type::DGRAM), + (libc::AF_ALG, socket2::Type::SEQPACKET), + ] + .map(|(domain, kind)| { + socket2::Socket::new(socket2::Domain::from(domain), kind, None) + .map(drop) + .map_err(|error| error.raw_os_error()) + }) + }) + .expect("launcher result"); + assert_eq!( + results, + [ + Ok(()), + Err(Some(libc::EAFNOSUPPORT)), + Err(Some(libc::EAFNOSUPPORT)) + ] + ); + } + + #[test] + fn repeated_connects_report_what_the_kernel_would() { + let relay: SocketAddr = "127.0.0.53:53".parse().unwrap(); + let peer: SocketAddr = "127.0.0.1:8080".parse().unwrap(); + let other: SocketAddr = "127.0.0.1:9090".parse().unwrap(); + for (state, kind, destination, expected) in [ + (SocketState::Created, InetKind::Tcp, peer, None), + ( + SocketState::Bound { local: peer }, + InetKind::Tcp, + peer, + None, + ), + ( + SocketState::Local { peer }, + InetKind::Tcp, + peer, + Some(libc::EISCONN), + ), + ( + SocketState::Connected { + original_peer: "203.0.113.7:443".parse().unwrap(), + }, + InetKind::Tcp, + peer, + Some(libc::EISCONN), + ), + ( + SocketState::DnsTcp { relay }, + InetKind::Tcp, + relay, + Some(libc::EISCONN), + ), + ( + SocketState::DnsUdp { relay }, + InetKind::DnsUdp, + relay, + Some(0), + ), + (SocketState::DnsUdp { relay }, InetKind::DnsUdp, other, None), + (SocketState::Local { peer }, InetKind::DnsUdp, peer, Some(0)), + ] { + assert_eq!( + repeated_connect_outcome(&state, kind, destination), + expected, + "{state:?} {kind:?} {destination}" + ); + } + } + + #[test] + fn repeated_bind_and_connect_after_completion() { + // A signal can restart a syscall the broker already completed. A + // repeated connect reports EISCONN; a repeated bind of the same + // address succeeds, unlike a native EINVAL, so a restart is safe. + let service = TcpListener::bind("127.0.0.1:0").unwrap(); + let address = service.local_addr().unwrap(); + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let (bind_again, connect_again) = launcher + .execute(move || -> io::Result<(io::Result<()>, Option)> { + let socket = + socket2::Socket::new(socket2::Domain::IPV4, socket2::Type::STREAM, None)?; + let local: SocketAddr = "127.0.0.1:0".parse().unwrap(); + socket.bind(&local.into())?; + let bind_again = socket.bind(&local.into()); + socket.connect(&address.into())?; + let connect_again = socket + .connect(&address.into()) + .err() + .and_then(|error| error.raw_os_error()); + Ok((bind_again, connect_again)) + }) + .expect("launcher result") + .expect("workload socket"); + bind_again.expect("repeated bind of the same address"); + assert_eq!(connect_again, Some(libc::EISCONN)); + } + + #[test] + fn frozen_workload_cannot_resume_processes_with_sigcont() { + // A workload process that is not yet stopped when the boundary + // freezes must not resume the others, through process-directed + // (kill) or thread-directed (tgkill) signals. + const CHILD_MARKER: &str = "OPENSHELL_FROZEN_SIGCONT_CHILD"; + if std::env::var_os(CHILD_MARKER).is_some() { + let errno = |result: nix::Result<()>| result.err().map_or(0, |error| error as i32); + let kill = errno(nix::sys::signal::kill( + nix::unistd::getpid(), + nix::sys::signal::Signal::SIGCONT, + )); + // SAFETY: tgkill takes scalar arguments naming this thread. + let tgkill = unsafe { + libc::syscall( + libc::SYS_tgkill, + libc::getpid(), + libc::syscall(libc::SYS_gettid), + libc::SIGCONT, + ) + }; + let tgkill = if tgkill < 0 { + io::Error::last_os_error().raw_os_error().unwrap_or(-1) + } else { + 0 + }; + println!("kill={kill} tgkill={tgkill}"); + return; + } + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let run_workload = |launcher: &openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher| { + let output = launcher + .execute(|| { + std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "network_broker::tests::frozen_workload_cannot_resume_processes_with_sigcont", + "--nocapture", + "--quiet", + ]) + .env(CHILD_MARKER, "1") + .output() + }) + .unwrap() + .expect("run workload child"); + String::from_utf8_lossy(&output.stdout) + .lines() + .find(|line| line.starts_with("kill=")) + .expect("workload child result") + .to_string() + }; + broker.set_workload_frozen(true); + assert_eq!( + run_workload(&launcher), + format!("kill={} tgkill={}", libc::EPERM, libc::EPERM) + ); + broker.set_workload_frozen(false); + assert_eq!(run_workload(&launcher), "kill=0 tgkill=0"); + } + + #[test] + fn slow_loopback_connect_does_not_stall_other_mediation() { + // A listener that never accepts, with a full backlog, makes further + // connects wait. Other mediated syscalls must not wait behind them, + // whether the pending connect is blocking or nonblocking. + let saturated = + socket2::Socket::new(socket2::Domain::IPV4, socket2::Type::STREAM, None).unwrap(); + saturated + .bind(&"127.0.0.1:0".parse::().unwrap().into()) + .unwrap(); + saturated.listen(0).unwrap(); + let address = saturated.local_addr().unwrap().as_socket().unwrap(); + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let (blocking_wait, nonblocking_result) = launcher + .execute(move || { + // Fill the accept queue; later connects to it stall. + let filled = TcpStream::connect(address).expect("fill accept queue"); + let pending = std::thread::spawn(move || TcpStream::connect(address)); + std::thread::sleep(Duration::from_millis(500)); + let started = Instant::now(); + drop(UdpSocket::bind("127.0.0.1:0")); + let blocking_wait = started.elapsed(); + // A nonblocking connect reports progress immediately. + let nonblocking = socket2::Socket::new( + socket2::Domain::IPV4, + socket2::Type::STREAM.nonblocking(), + None, + ) + .unwrap(); + let nonblocking_result = nonblocking + .connect(&address.into()) + .err() + .and_then(|error| error.raw_os_error()); + // Leave the pending connect running; closing the listener when + // the test ends resets it. + drop(pending); + drop(filled); + (blocking_wait, nonblocking_result) + }) + .expect("launcher result"); assert!( - workload - .join() - .expect("join workload") - .expect("accepted workload") - .ip() - .is_loopback() + blocking_wait < Duration::from_secs(2), + "mediated socket creation waited {blocking_wait:?} behind a slow connect" ); + assert_eq!(nonblocking_result, Some(libc::EINPROGRESS)); + } + + #[test] + fn workload_cannot_bind_a_non_loopback_source_address() { + // Workload sockets can only present loopback source addresses. A + // non-loopback peer on a loopback-bound workload listener is therefore + // a non-workload process in the same network namespace. + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let errors = launcher + .execute(|| { + ["192.0.2.10:0", "[2001:db8::10]:0"].map(|address| { + UdpSocket::bind(address) + .err() + .and_then(|error| error.raw_os_error()) + }) + }) + .expect("launcher result"); + assert_eq!(errors, [Some(libc::EACCES), Some(libc::EACCES)]); + } + + #[test] + fn interface_selection_options_are_denied() { + for (level, option) in [ + (libc::SOL_SOCKET, libc::SO_BINDTODEVICE), + (libc::SOL_SOCKET, libc::SO_BINDTOIFINDEX), + (libc::IPPROTO_IP, libc::IP_UNICAST_IF), + (libc::IPPROTO_IP, libc::IP_MULTICAST_IF), + (libc::IPPROTO_IPV6, libc::IPV6_UNICAST_IF), + (libc::IPPROTO_IPV6, libc::IPV6_MULTICAST_IF), + (libc::IPPROTO_IPV6, libc::IPV6_ADDRFORM), + (libc::IPPROTO_TCP, libc::TCP_FASTOPEN_CONNECT), + ] { + assert!(socket_option_is_denied(level, option), "{level}/{option}"); + } + assert!(!socket_option_is_denied( + libc::SOL_SOCKET, + libc::SO_REUSEADDR + )); + assert!(!socket_option_is_denied( + libc::IPPROTO_TCP, + libc::TCP_NODELAY + )); + assert!(!socket_option_is_denied( + libc::IPPROTO_IPV6, + libc::IPV6_V6ONLY + )); } #[test] @@ -2554,6 +2706,110 @@ mod tests { ); } + #[test] + fn udp_dns_after_connect_sends_with_sendmmsg() { + // glibc connects the resolver socket to the nameserver, then sends A + // and AAAA together with sendmmsg and no destination. + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let dns_address = broker.dns_address(); + let client = std::thread::spawn(move || { + launcher + .execute(move || -> io::Result>> { + let socket = UdpSocket::bind("0.0.0.0:0")?; + socket.set_read_timeout(Some(Duration::from_secs(5)))?; + socket.connect(dns_address)?; + let queries = [&b"dns-query-a"[..], &b"dns-query-aaaa"[..]]; + let iovecs = queries.map(|query| [io::IoSlice::new(query)]); + let mut controls = [ + rustix::net::SendAncillaryBuffer::default(), + rustix::net::SendAncillaryBuffer::default(), + ]; + let [first, second] = &mut controls; + let mut messages = [ + rustix::net::MMsgHdr::new(&iovecs[0], first), + rustix::net::MMsgHdr::new(&iovecs[1], second), + ]; + let sent = rustix::net::sendmmsg( + &socket, + &mut messages, + rustix::net::SendFlags::empty(), + )?; + if sent != 2 { + return Err(io::Error::other("sendmmsg sent too few")); + } + let mut responses = Vec::new(); + for _ in 0..2 { + let mut response = [0_u8; 32]; + let length = socket.recv(&mut response)?; + responses.push(response[..length].to_vec()); + } + responses.sort(); + Ok(responses) + }) + .expect("launcher result") + }); + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + for _ in 0..2 { + let query = runtime.block_on(broker.accept_dns()).expect("DNS query"); + let response = if query.request == b"dns-query-a" { + b"dns-response-a".to_vec() + } else if query.request == b"dns-query-aaaa" { + b"dns-response-aaaa".to_vec() + } else { + panic!("unexpected DNS query: {:?}", query.request); + }; + query.complete(Ok(response)).unwrap(); + } + assert_eq!( + client.join().expect("join client").expect("DNS client"), + vec![b"dns-response-a".to_vec(), b"dns-response-aaaa".to_vec()] + ); + } + + #[test] + fn dns_send_with_ancillary_data_is_refused() { + // Ancillary data can carry a per-message routing override such as + // IP_PKTINFO. The kernel performs mediated DNS sends, so control data + // is refused. + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let dns_address = broker.dns_address(); + let errno = launcher + .execute(move || { + let socket = UdpSocket::bind("0.0.0.0:0").unwrap(); + let native = socket2::SockAddr::from(dns_address); + let payload = *b"dns-query"; + let mut iov = libc::iovec { + iov_base: payload.as_ptr().cast_mut().cast(), + iov_len: payload.len(), + }; + // Any non-empty control buffer is refused. + let mut control = [0_u8; 32]; + let header = libc::msghdr { + msg_name: native.as_ptr().cast_mut().cast(), + msg_namelen: native.len(), + msg_iov: &raw mut iov, + msg_iovlen: 1, + msg_control: control.as_mut_ptr().cast(), + msg_controllen: control.len(), + msg_flags: 0, + }; + // SAFETY: the header references live local buffers for the call. + let sent = unsafe { libc::sendmsg(socket.as_raw_fd(), &raw const header, 0) }; + (sent < 0) + .then(|| io::Error::last_os_error().raw_os_error()) + .flatten() + }) + .expect("launcher result"); + assert_eq!(errno, Some(libc::EOPNOTSUPP)); + } + #[test] fn udp_dns_allows_repeated_destination_sends_to_the_pinned_relay() { let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() diff --git a/crates/openshell-sandbox/src/process.rs b/crates/openshell-sandbox/src/process.rs index 595ceac29e..20812ce315 100644 --- a/crates/openshell-sandbox/src/process.rs +++ b/crates/openshell-sandbox/src/process.rs @@ -105,6 +105,9 @@ pub(crate) fn ca_runtime_read_only_paths(ca_paths: Option<&(PathBuf, PathBuf)>) paths } +/// Prefix of environment variable names reserved for `OpenShell`. +pub(crate) const RESERVED_ENV_PREFIX: &str = "OPENSHELL_"; + const SUPERVISOR_ONLY_ENV_VARS: &[&str] = &[ openshell_core::sandbox_env::OCI_IMAGE_USER, openshell_core::sandbox_env::SANDBOX_UID, @@ -205,6 +208,21 @@ fn apply_canonical_process_environment( interactive: bool, user_environment: &HashMap, ) { + // The canonical process inherits the sandbox's environment so the image's + // own ENV (PATH, LANG, JAVA_HOME, ...) reaches the workload. Remove the + // reserved OPENSHELL_ namespace inherited from the sandbox itself, which + // carries its own control state (for example the serialized user + // environment and log level), then restore the one marker the workload is + // meant to see. The gateway rejects declared variables in this namespace. + for (key, _) in std::env::vars_os() { + if key + .to_str() + .is_some_and(|key| key.starts_with(RESERVED_ENV_PREFIX)) + { + cmd.env_remove(key); + } + } + cmd.env(openshell_core::sandbox_env::SANDBOX, "1"); cmd.envs(user_environment); let (session_user, session_home) = session_user_and_home(policy, workspace.home()); // Resolve a shell present in the workload image. This code runs inside the @@ -450,9 +468,7 @@ impl ProcessHandle { provider_env: &HashMap, ) -> Result { let mut cmd = Command::new(program); - cmd.args(args) - .kill_on_drop(true) - .env(openshell_core::sandbox_env::SANDBOX, "1"); + cmd.args(args).kill_on_drop(true); let mut pty_master = None; let mut terminal_slave_fd = None; @@ -615,9 +631,7 @@ impl ProcessHandle { provider_env: &HashMap, ) -> Result { let mut cmd = Command::new(program); - cmd.args(args) - .kill_on_drop(true) - .env(openshell_core::sandbox_env::SANDBOX, "1"); + cmd.args(args).kill_on_drop(true); let mut pty_master = None; let mut terminal_slave_fd = None; @@ -1122,6 +1136,73 @@ mod tests { assert_eq!(variables.get("TERM"), Some(&"xterm-256color")); } + #[cfg(unix)] + #[test] + fn canonical_process_drops_inherited_reserved_environment() { + // The sandbox's own control variables live in the reserved + // OPENSHELL_ namespace and must not reach the workload, while the + // image's ordinary ENV must. Run in a fresh copy of the test binary + // so the test harness environment is untouched. + const CHILD_MARKER: &str = "OPENSHELL_TEST_RESERVED_ENV_CHILD"; + if std::env::var_os(CHILD_MARKER).is_none() { + let status = std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "process::tests::canonical_process_drops_inherited_reserved_environment", + "--nocapture", + ]) + .env(CHILD_MARKER, "1") + .env(openshell_core::sandbox_env::LOG_LEVEL, "debug") + .env(openshell_core::sandbox_env::USER_ENVIRONMENT, "{}") + .env("IMAGE_LANG", "keep") + .status() + .expect("run isolated environment test"); + assert!(status.success(), "isolated environment test failed"); + return; + } + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let current_user = User::from_uid(nix::unistd::geteuid()).unwrap().unwrap(); + let policy = policy_with_process(ProcessPolicy { + run_as_user: Some(current_user.name), + run_as_group: None, + }); + // Mirror production: inherit the sandbox environment, no env_clear. + let mut cmd = Command::new("/usr/bin/env"); + cmd.stdout(StdStdio::piped()); + apply_canonical_process_environment( + &mut cmd, + &policy, + &ResolvedWorkspace::default(), + false, + &HashMap::from([("DECLARED".into(), "yes".into())]), + ); + let output = runtime + .block_on(async { cmd.output().await }) + .expect("run environment probe"); + assert!(output.status.success()); + let environment = String::from_utf8(output.stdout).unwrap(); + let variables: HashMap<_, _> = environment + .lines() + .filter_map(|line| line.split_once('=')) + .collect(); + assert!( + !variables + .keys() + .any(|key| key.starts_with(RESERVED_ENV_PREFIX) + && *key != openshell_core::sandbox_env::SANDBOX), + "reserved variables reached the workload: {variables:?}" + ); + assert_eq!( + variables.get(openshell_core::sandbox_env::SANDBOX), + Some(&"1") + ); + assert_eq!(variables.get("IMAGE_LANG"), Some(&"keep")); + assert_eq!(variables.get("DECLARED"), Some(&"yes")); + } + #[cfg(unix)] #[tokio::test] async fn canonical_process_receives_declared_environment_and_home() { diff --git a/crates/openshell-sandbox/src/provider_files.rs b/crates/openshell-sandbox/src/provider_files.rs index ab719cd355..384eb6b31f 100644 --- a/crates/openshell-sandbox/src/provider_files.rs +++ b/crates/openshell-sandbox/src/provider_files.rs @@ -17,6 +17,7 @@ use openshell_isolation_interface::linux::seccomp_notify::{Notification, Notific use openshell_isolation_interface::linux::task_memory; const PREFIX: &str = "/run/openshell/providers/"; +const PROC_PREFIX: &str = "/proc/"; const MAX_FILE_BYTES: usize = 65_536; const MAX_TOTAL_BYTES: usize = 262_144; const MAX_PATH_BYTES: usize = 4_096; @@ -75,6 +76,9 @@ impl ProviderFiles { } else { notification.args[0] }; + if handle_thread_comm_open(listener, notification, path_address)? { + return Ok(()); + } // Every workload open reaches the listener. Copy only the reserved // prefix for ordinary paths; full path reads are rare. let mut prefix = [0_u8; PREFIX.len()]; @@ -160,6 +164,97 @@ fn validate_path(path: &str) -> io::Result<()> { Ok(()) } +/// Serve a write open of the caller's own thread name file. +/// +/// `pthread_setname_np` and CUDA's `cuInit` rename threads by writing +/// `/proc//task//comm`. Landlock keeps `/proc` read-only, so the +/// broker opens the caller's own `comm` file and injects the descriptor; no +/// syscall is continued. The kernel's `comm_write` accepts a write only from +/// the target's own thread group, so a substituted path or reused thread ID +/// cannot rename another process's thread through the descriptor. Returns +/// `false` when the open is not such a request and normal mediation applies. +fn handle_thread_comm_open( + listener: &NotificationListener, + notification: Notification, + path_address: u64, +) -> io::Result { + let Ok(flags) = open_flags(¬ification) else { + return Ok(false); + }; + let access = flags & libc::O_ACCMODE; + // Reads are already allowed by the read-only /proc rule. + if access == libc::O_RDONLY { + return Ok(false); + } + let mut prefix = [0_u8; PROC_PREFIX.len()]; + if task_memory::read_exact(notification.tid, path_address, &mut prefix).is_err() + || prefix != PROC_PREFIX.as_bytes() + { + return Ok(false); + } + let Ok(path) = read_path(notification.tid, path_address) else { + return Ok(false); + }; + let Some(caller_group) = thread_group_of(notification.tid) else { + return Ok(false); + }; + let Some(target) = comm_target(&path, notification.tid, caller_group) else { + return Ok(false); + }; + // Anything else, including another process's thread, is left to + // Landlock, which denies the write. + if thread_group_of(target) != Some(caller_group) { + return Ok(false); + } + // A shell redirect opens with O_CREAT|O_TRUNC; both are no-ops on an + // existing comm file. O_EXCL fails as it would natively. + if flags & libc::O_EXCL != 0 { + listener.respond_errno(notification.id, libc::EEXIST)?; + return Ok(true); + } + if flags & (libc::O_TMPFILE | libc::O_DIRECTORY | libc::O_PATH) != 0 { + listener.respond_errno(notification.id, libc::EINVAL)?; + return Ok(true); + } + let file = std::fs::OpenOptions::new() + .read(access == libc::O_RDWR) + .write(true) + .open(format!("/proc/{caller_group}/task/{target}/comm"))?; + listener.add_fd_and_send( + notification.id, + file.as_raw_fd(), + flags & libc::O_CLOEXEC != 0, + )?; + Ok(true) +} + +/// Resolve the thread whose `comm` file `path` names, if it is the caller's +/// own thread group. +fn comm_target(path: &str, caller_tid: u32, caller_group: u32) -> Option { + let parts = path + .strip_prefix(PROC_PREFIX)? + .split('/') + .collect::>(); + let own_group = |part: &str| part == "self" || part.parse::().ok() == Some(caller_group); + match parts.as_slice() { + ["thread-self", "comm"] => Some(caller_tid), + [group, "comm"] if own_group(group) => Some(caller_group), + [group, "task", tid, "comm"] if own_group(group) => tid.parse().ok(), + _ => None, + } +} + +/// Thread group (process) ID of a thread, from its procfs status. +fn thread_group_of(tid: u32) -> Option { + std::fs::read_to_string(format!("/proc/{tid}/status")) + .ok()? + .lines() + .find_map(|line| line.strip_prefix("Tgid:"))? + .trim() + .parse() + .ok() +} + fn read_path(tid: u32, mut address: u64) -> io::Result { if address == 0 { return Err(io::Error::from_raw_os_error(libc::EFAULT)); @@ -228,12 +323,32 @@ fn sealed_memfd(content: &[u8]) -> io::Result { #[cfg(test)] mod tests { - use super::{ProviderFiles, sealed_memfd}; + use super::{ProviderFiles, comm_target, sealed_memfd}; use std::collections::HashMap; use std::io::Read as _; use std::os::fd::AsRawFd as _; use std::os::unix::fs::PermissionsExt as _; + #[test] + fn comm_target_accepts_only_the_callers_own_thread_names() { + let (caller_tid, group) = (4242, 4200); + for (path, expected) in [ + ("/proc/thread-self/comm", Some(caller_tid)), + ("/proc/self/comm", Some(group)), + ("/proc/4200/comm", Some(group)), + ("/proc/self/task/4243/comm", Some(4243)), + ("/proc/4200/task/4243/comm", Some(4243)), + // Another process's thread, or not a comm file. + ("/proc/1/task/1/comm", None), + ("/proc/9999/comm", None), + ("/proc/self/task/4243/environ", None), + ("/proc/self/task/4243/comm/extra", None), + ("/proc/self/mem", None), + ] { + assert_eq!(comm_target(path, caller_tid, group), expected, "{path}"); + } + } + #[test] fn paths_cannot_escape_the_managed_tree() { let valid = "/run/openshell/providers/acme/client.toml"; diff --git a/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs b/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs index 81d0b1630f..e9fa55cc1b 100644 --- a/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs +++ b/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs @@ -5,7 +5,11 @@ //! //! The filter uses a default-allow policy with targeted blocks: //! -//! 1. **Socket domain blocks** -- prevent raw/kernel sockets that bypass the proxy +//! 1. **Socket domain allowlist** -- only `AF_UNIX`, `AF_NETLINK`, and (when +//! networking is enabled) the brokered `AF_INET`/`AF_INET6` families can be +//! created. Every other family is refused, because protocol families such as +//! `AF_RXRPC`, `AF_SMC`, and `AF_KCM` carry traffic over kernel-owned +//! sockets that the broker never creates or confines to loopback. //! 2. **Unconditional syscall blocks** -- block syscalls that enable sandbox escape //! (fileless exec, ptrace, BPF, cross-process memory access, `io_uring`, mount) //! 3. **Conditional syscall blocks** -- block dangerous flag combinations on otherwise @@ -184,23 +188,18 @@ fn apply_runtime_filters( fn build_filter_rules(allow_inet: bool) -> Result>> { let mut rules: BTreeMap> = BTreeMap::new(); - // --- Socket domain blocks --- - let mut blocked_domains = vec![ - libc::AF_PACKET, - libc::AF_BLUETOOTH, - libc::AF_VSOCK, - // AF_NETLINK is handled separately below: NETLINK_ROUTE (protocol 0) - // is allowed for getifaddrs(3); all other netlink protocols are blocked. - ]; - if !allow_inet { - blocked_domains.push(libc::AF_INET); - blocked_domains.push(libc::AF_INET6); - } - - for domain in blocked_domains { - debug!(domain, "Blocking socket domain via seccomp"); - add_socket_domain_rule(&mut rules, domain)?; + // --- Socket domain allowlist --- + // AF_NETLINK is narrowed further below: only NETLINK_ROUTE (protocol 0) + // is allowed, for getifaddrs(3). + let mut allowed_domains = vec![libc::AF_UNIX, libc::AF_NETLINK]; + if allow_inet { + allowed_domains.extend([libc::AF_INET, libc::AF_INET6]); } + debug!(?allowed_domains, "Restricting socket domains via seccomp"); + add_socket_domain_allowlist(&mut rules, libc::SYS_socket, &allowed_domains)?; + // socketpair(2) is only meaningful for AF_UNIX here; other families either + // reject it or create kernel transport sockets. + add_socket_domain_allowlist(&mut rules, libc::SYS_socketpair, &[libc::AF_UNIX])?; // Allow AF_NETLINK only for NETLINK_ROUTE (protocol 0). // @@ -296,14 +295,27 @@ fn build_filter_rules(allow_inet: bool) -> Result Ok(rules) } +/// Refuse `syscall` unless its domain argument is one of `allowed`. +/// +/// A seccomp rule matches only when all of its conditions hold, so one rule +/// with a `!=` condition per allowed domain matches exactly the domains +/// outside the allowlist. The domain is a scalar argument that another thread +/// cannot replace before the kernel reads it. #[allow(clippy::cast_sign_loss)] -fn add_socket_domain_rule(rules: &mut BTreeMap>, domain: i32) -> Result<()> { - let condition = - SeccompCondition::new(0, SeccompCmpArgLen::Dword, SeccompCmpOp::Eq, domain as u64) - .into_diagnostic()?; - - let rule = SeccompRule::new(vec![condition]).into_diagnostic()?; - rules.entry(libc::SYS_socket).or_default().push(rule); +fn add_socket_domain_allowlist( + rules: &mut BTreeMap>, + syscall: i64, + allowed: &[i32], +) -> Result<()> { + let conditions = allowed + .iter() + .map(|domain| { + SeccompCondition::new(0, SeccompCmpArgLen::Dword, SeccompCmpOp::Ne, *domain as u64) + .into_diagnostic() + }) + .collect::>>()?; + let rule = SeccompRule::new(conditions).into_diagnostic()?; + rules.entry(syscall).or_default().push(rule); Ok(()) } @@ -851,6 +863,78 @@ mod tests { ); } + #[test] + fn behavioral_socket_families_are_allowlisted() { + // Applying a filter is irreversible, so run the probe in a fresh copy + // of this test binary rather than in the harness process. + const CHILD_MARKER: &str = "OPENSHELL_SOCKET_FAMILY_ALLOWLIST_CHILD"; + // libc does not export these family numbers. + const AF_KCM: i32 = 41; + const AF_SMC: i32 = 43; + if std::env::var_os(CHILD_MARKER).is_some() { + set_no_new_privs().expect("set no_new_privs"); + apply_filter(&build_filter(true).unwrap()).expect("apply proxy-mode filter"); + let create = |domain: i32, kind: socket2::Type| { + socket2::Socket::new(socket2::Domain::from(domain), kind, None) + .map(drop) + .map_err(|error| error.raw_os_error()) + }; + for (domain, kind) in [ + (libc::AF_UNIX, socket2::Type::STREAM), + (libc::AF_NETLINK, socket2::Type::RAW), + (libc::AF_INET, socket2::Type::STREAM), + (libc::AF_INET6, socket2::Type::DGRAM), + ] { + assert_eq!( + create(domain, kind), + Ok(()), + "domain {domain} must be allowed" + ); + } + // Families whose kernel transport sockets the broker cannot + // confine, plus previously denylisted ones. + for (domain, kind) in [ + (libc::AF_RXRPC, socket2::Type::DGRAM), + (AF_SMC, socket2::Type::STREAM), + (AF_KCM, socket2::Type::DGRAM), + (libc::AF_ALG, socket2::Type::SEQPACKET), + (libc::AF_TIPC, socket2::Type::from(libc::SOCK_RDM)), + (libc::AF_PACKET, socket2::Type::RAW), + (libc::AF_VSOCK, socket2::Type::STREAM), + ] { + assert_eq!( + create(domain, kind), + Err(Some(libc::EPERM)), + "domain {domain} must be refused by the filter" + ); + } + assert!( + socket2::Socket::pair(socket2::Domain::UNIX, socket2::Type::STREAM, None).is_ok() + ); + assert_eq!( + socket2::Socket::pair( + socket2::Domain::from(libc::AF_TIPC), + socket2::Type::STREAM, + None + ) + .map(drop) + .map_err(|error| error.raw_os_error()), + Err(Some(libc::EPERM)) + ); + return; + } + let status = std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "sandbox::linux::seccomp::tests::behavioral_socket_families_are_allowlisted", + "--nocapture", + ]) + .env(CHILD_MARKER, "1") + .status() + .expect("run isolated socket family test"); + assert!(status.success(), "isolated socket family test failed"); + } + #[test] fn behavioral_block_mode_denies_inet_and_packet_sockets() { let filter = build_filter(false).unwrap(); diff --git a/crates/openshell-server/Cargo.toml b/crates/openshell-server/Cargo.toml index 74ff40d1ee..173e212fe5 100644 --- a/crates/openshell-server/Cargo.toml +++ b/crates/openshell-server/Cargo.toml @@ -129,6 +129,7 @@ tonic-prost-build = { workspace = true } protoc-bin-vendored = { workspace = true } [dev-dependencies] +tracing-core = "0.1" tokio = { workspace = true, features = ["test-util"] } # Tests import the example profiles from providers/ the way an operator # would; the feature is test-only and never reaches a release binary. diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index 2c84c147c3..ba7d2d7dfa 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -14118,7 +14118,6 @@ mod tests { /// Driver watch events arrive on a background stream, so the store writes /// they trigger land outside the request that caused them. #[tokio::test] - #[ignore = "flaky under concurrent test execution"] async fn driver_watch_events_are_roots_and_store_operations_have_parents() { use crate::otel_tracing::test_exporter; @@ -14133,20 +14132,12 @@ mod tests { .await .unwrap(); + let root = traced.wait_for_span("driver_watch.sandbox_deleted").await; let spans = traced.finished_spans(); - let root = spans - .iter() - .find(|s| s.name == "driver_watch.sandbox_deleted") - .unwrap_or_else(|| { - panic!( - "the event records a span of its own, got {:?}", - spans.iter().map(|s| &s.name).collect::>() - ) - }); - test_exporter::assert_is_root(root); + test_exporter::assert_is_root(&root); assert_eq!( - test_exporter::attribute(root, "sandbox.id").as_deref(), + test_exporter::attribute(&root, "sandbox.id").as_deref(), Some("sb-1"), "the span names which sandbox the driver reported on" ); @@ -14164,7 +14155,6 @@ mod tests { /// The reconciler runs on a timer with no inbound request, so without a /// span of its own each store call becomes its own anonymous trace. #[tokio::test] - #[ignore = "flaky under concurrent test execution"] async fn reconcile_sweeps_are_roots_and_operations_have_parents() { use crate::otel_tracing::test_exporter; @@ -14179,9 +14169,8 @@ mod tests { .await .unwrap(); - // Other tests drive their own reconcile loops into the shared - // exporter, so match on the shape of a sweep rather than assuming - // there is exactly one. + // A closed sweep has no remaining worker-held child spans. + traced.wait_for_span("reconcile.sandboxes").await; let spans = traced.finished_spans(); let roots = traced.spans_named("reconcile.sandboxes"); assert!( diff --git a/crates/openshell-server/src/grpc/provider.rs b/crates/openshell-server/src/grpc/provider.rs index cc2395fe7d..fb22dc698d 100644 --- a/crates/openshell-server/src/grpc/provider.rs +++ b/crates/openshell-server/src/grpc/provider.rs @@ -2127,6 +2127,7 @@ fn provider_credential_config_key_collision( struct DynamicTokenGrantBinding { provider_name: String, credential_name: String, + header_name: String, host: String, port: u32, path: String, @@ -2182,7 +2183,7 @@ fn push_dynamic_token_grant_bindings_for_endpoint( push_dynamic_token_grant_binding( bindings, provider_name, - &credential.name, + credential, endpoint_host, endpoint_port, endpoint_path, @@ -2214,7 +2215,7 @@ fn push_dynamic_token_grant_bindings_for_endpoint( push_dynamic_token_grant_binding( bindings, provider_name, - &credential.name, + credential, override_host, override_port, override_path, @@ -2225,14 +2226,21 @@ fn push_dynamic_token_grant_bindings_for_endpoint( fn push_dynamic_token_grant_binding( bindings: &mut Vec, provider_name: &str, - credential_name: &str, + credential: &ProviderProfileCredential, host: &str, port: u32, path: &str, ) { let candidate = DynamicTokenGrantBinding { provider_name: provider_name.to_string(), - credential_name: credential_name.to_string(), + credential_name: credential.name.clone(), + // The supervisor selects one grant per case-insensitive header, using + // Authorization when bearer placement omits an explicit destination. + header_name: if credential.header_name.trim().is_empty() { + "authorization".to_string() + } else { + credential.header_name.trim().to_ascii_lowercase() + }, host: host.to_ascii_lowercase(), port, path: path.to_string(), @@ -2253,7 +2261,8 @@ fn validate_dynamic_token_grant_bindings_unambiguous( { continue; } - if first.port == second.port + if first.header_name == second.header_name + && first.port == second.port && first.score == second.score && host_patterns_can_overlap(&first.host, &second.host) && path_patterns_can_overlap(&first.path, &second.path) @@ -5456,9 +5465,28 @@ mod tests { host: &str, port: u32, path: &str, + ) { + import_token_grant_profile_with_credentials( + state, + id, + host, + port, + path, + vec![token_grant_credential("access_token")], + ) + .await; + } + + async fn import_token_grant_profile_with_credentials( + state: &Arc, + id: &str, + host: &str, + port: u32, + path: &str, + credentials: Vec, ) { let mut profile = custom_profile(id); - profile.credentials = vec![token_grant_credential("access_token")]; + profile.credentials = credentials; profile.endpoints = vec![NetworkEndpoint { host: host.to_string(), port, @@ -5515,6 +5543,102 @@ mod tests { .unwrap() } + #[tokio::test] + async fn dynamic_token_grants_allow_distinct_headers_in_one_profile() { + let state = test_server_state().await; + let store = state.store.as_ref(); + let service = token_grant_credential("service"); + let mut identity = token_grant_credential("identity"); + identity.auth_style = "header".into(); + identity.header_name = "X-Workload-Jwt".into(); + let grant = identity.token_grant.as_mut().unwrap(); + grant.token_endpoint = "https://identity.example.com/token".into(); + grant.jwt_svid_audience = "identity-proxy".into(); + grant.audience = "workload".into(); + grant.scopes = vec!["identity.read".into()]; + grant.cache_ttl = Some(prost_types::Duration { + seconds: 45, + nanos: 0, + }); + import_token_grant_profile_with_credentials( + &state, + "grant-pair", + "api.example.com", + 443, + "/v1/**", + vec![service.clone(), identity.clone()], + ) + .await; + create_empty_token_grant_provider(store, "provider", "grant-pair").await; + validate_provider_environment_keys_unique(store, "default", &["provider".into()]) + .await + .expect("distinct headers must compose in one provider"); + + let catalog = ProviderProfileSources::with_default_sources() + .snapshot_catalog(store, "default") + .await + .unwrap(); + let profile = get_provider_type_profile_for_scope(&catalog, "grant-pair", "default") + .unwrap() + .to_proto(); + let mut credentials = HashMap::new(); + insert_dynamic_credentials_for_profile(&mut credentials, &profile, "provider"); + assert_eq!(credentials.len(), 2); + for expected in [service, identity] { + let key = dynamic_credential_key( + "api.example.com", + 443, + "/v1/**", + "provider", + &expected.name, + ); + assert_eq!(credentials[&key].token_grant, expected.token_grant); + assert_eq!(credentials[&key].header_name, expected.header_name); + assert!(credentials[&key].env_vars.is_empty()); + } + } + + #[test] + fn dynamic_token_grants_reject_normalized_header_collisions() { + for header in ["Authorization", " authorization ", ""] { + let mut profile = custom_profile("grant-pair"); + let mut second = token_grant_credential("second"); + second.header_name = header.into(); + profile.credentials = vec![token_grant_credential("first"), second]; + profile.endpoints = vec![NetworkEndpoint { + host: "api.example.com".into(), + port: 443, + path: "/v1/**".into(), + ..Default::default() + }]; + let bindings = dynamic_token_grant_bindings_for_profile("provider", &profile); + assert_eq!(bindings.len(), 2); + assert!(validate_dynamic_token_grant_bindings_unambiguous(&bindings).is_err()); + } + } + + #[test] + fn dynamic_token_grants_allow_distinct_headers_across_providers() { + let mut first = custom_profile("grant-a"); + first.credentials = vec![token_grant_credential("service")]; + first.endpoints = vec![NetworkEndpoint { + host: "api.example.com".into(), + port: 443, + path: "/v1/**".into(), + ..Default::default() + }]; + let mut second = first.clone(); + second.credentials[0].auth_style = "header".into(); + second.credentials[0].header_name = "X-Workload-Jwt".into(); + let mut bindings = dynamic_token_grant_bindings_for_profile("provider-a", &first); + bindings.extend(dynamic_token_grant_bindings_for_profile( + "provider-b", + &second, + )); + validate_dynamic_token_grant_bindings_unambiguous(&bindings) + .expect("different headers must not be treated as alternatives"); + } + #[tokio::test] async fn dynamic_token_grants_reject_equal_specificity_overlap() { let state = test_server_state().await; diff --git a/crates/openshell-server/src/grpc/sandbox.rs b/crates/openshell-server/src/grpc/sandbox.rs index 8f2bf9a784..8e0e175f42 100644 --- a/crates/openshell-server/src/grpc/sandbox.rs +++ b/crates/openshell-server/src/grpc/sandbox.rs @@ -4543,7 +4543,6 @@ mod tests { } #[tokio::test] - #[ignore = "flaky under concurrent test execution"] async fn watch_producer_releases_request_span_when_client_disconnects() { use crate::otel_tracing::test_exporter; use tokio_stream::StreamExt as _; @@ -4582,6 +4581,7 @@ mod tests { drop(request_span); stream.disconnect_and_wait().await; + traced.wait_for_span("disconnected_watch_request").await; assert_eq!( traced.spans_named("disconnected_watch_request").len(), diff --git a/crates/openshell-server/src/otel_tracing.rs b/crates/openshell-server/src/otel_tracing.rs index cdae552871..fd2cb05e33 100644 --- a/crates/openshell-server/src/otel_tracing.rs +++ b/crates/openshell-server/src/otel_tracing.rs @@ -146,6 +146,113 @@ where /// Isolated in-memory span exporters for tracing tests. #[cfg(test)] pub mod test_exporter { + use std::sync::{Arc, OnceLock}; + + use tracing::{Dispatch, Subscriber, dispatcher::WeakDispatch}; + + /// Keep parent-span cleanup on the registry that created the span. + /// + /// `SQLx` moves spans onto its `SQLite` worker without installing the test's + /// thread-local dispatcher. `tracing-subscriber` closes a child's parent + /// through the current dispatcher, so the worker can otherwise look up the + /// parent in the unrelated global registry when it drops the last reference. + struct CloseWithDispatch { + inner: S, + dispatch: OnceLock, + closed: Arc, + } + + impl Subscriber for CloseWithDispatch { + fn on_register_dispatch(&self, dispatch: &Dispatch) { + self.dispatch + .set(dispatch.downgrade()) + .expect("test subscriber is registered once"); + self.inner.on_register_dispatch(dispatch); + } + + fn register_callsite( + &self, + metadata: &'static tracing::Metadata<'static>, + ) -> tracing::subscriber::Interest { + self.inner.register_callsite(metadata) + } + + fn enabled(&self, metadata: &tracing::Metadata<'_>) -> bool { + self.inner.enabled(metadata) + } + + fn max_level_hint(&self) -> Option { + self.inner.max_level_hint() + } + + fn new_span(&self, attributes: &tracing::span::Attributes<'_>) -> tracing::span::Id { + self.inner.new_span(attributes) + } + + fn record(&self, id: &tracing::span::Id, values: &tracing::span::Record<'_>) { + self.inner.record(id, values); + } + + fn record_follows_from(&self, id: &tracing::span::Id, follows: &tracing::span::Id) { + self.inner.record_follows_from(id, follows); + } + + fn event_enabled(&self, event: &tracing::Event<'_>) -> bool { + self.inner.event_enabled(event) + } + + fn event(&self, event: &tracing::Event<'_>) { + self.inner.event(event); + } + + fn enter(&self, id: &tracing::span::Id) { + self.inner.enter(id); + } + + fn exit(&self, id: &tracing::span::Id) { + self.inner.exit(id); + } + + fn clone_span(&self, id: &tracing::span::Id) -> tracing::span::Id { + self.inner.clone_span(id) + } + + fn try_close(&self, id: tracing::span::Id) -> bool { + // The span being closed owns a strong dispatcher reference. Store + // only a weak reference here to avoid a subscriber/dispatcher cycle. + let dispatch = self + .dispatch + .get() + .and_then(WeakDispatch::upgrade) + .expect("a live span keeps its test dispatcher alive"); + let closed = tracing::dispatcher::with_default(&dispatch, || self.inner.try_close(id)); + if closed { + // The simple exporter has finished before try_close returns. + // Wake assertions only after the owning registry and layers + // have completed cleanup, including recursive parent closure. + self.closed.notify_waiters(); + } + closed + } + + fn current_span(&self) -> tracing_core::span::Current { + self.inner.current_span() + } + + // OpenTelemetrySpanExt downcasts through the subscriber to its layer. + // SAFETY: Forward the unchanged TypeId to the inner subscriber, which + // owns the returned pointer for exactly as long as this wrapper lives. + #[allow(unsafe_code)] + unsafe fn downcast_raw(&self, id: std::any::TypeId) -> Option<*const ()> { + if id == std::any::TypeId::of::() { + Some(std::ptr::from_ref(self).cast()) + } else { + // SAFETY: The inner subscriber owns and validates this downcast. + unsafe { self.inner.downcast_raw(id) } + } + } + } + /// Installs a process-wide registry before any scoped test subscriber is /// used. /// @@ -174,11 +281,17 @@ pub mod test_exporter { .with_simple_exporter(exporter.clone()) .build(); let subscriber = tracing_subscriber::registry().with(super::layer(&provider, None)); - let dispatch = tracing::Dispatch::new(subscriber); + let closed = Arc::new(tokio::sync::Notify::new()); + let dispatch = Dispatch::new(CloseWithDispatch { + inner: subscriber, + dispatch: OnceLock::new(), + closed: Arc::clone(&closed), + }); TracingTestGuard { _default: tracing::dispatcher::set_default(&dispatch), _provider: provider, exporter, + closed, _lock: lock, } } @@ -189,6 +302,52 @@ pub mod test_exporter { self.exporter.get_finished_spans().expect("in-memory spans") } + /// Wait for expected spans to finish before taking an assertion snapshot. + /// + /// A completed `SQLx` query may still have its span held by a `SQLite` + /// worker. Export is synchronous once the span closes, but flushing + /// cannot close that live span. Await closure notifications instead of + /// assuming the query result also means tracing cleanup has completed. + pub async fn wait_for_spans( + &self, + predicate: impl Fn(&[opentelemetry_sdk::trace::SpanData]) -> bool + Send + Sync, + ) -> Vec { + tokio::time::timeout(std::time::Duration::from_secs(5), async { + loop { + // notify_waiters wakes futures created before notification, + // even before polling. Subscribe before reading so closure + // between the snapshot and await cannot lose a wakeup. + let notified = self.closed.notified(); + let spans = self.finished_spans(); + if predicate(&spans) { + return spans; + } + notified.await; + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "timed out waiting for expected spans, got {:?}", + self.finished_spans() + .iter() + .map(|span| &span.name) + .collect::>() + ) + }) + } + + /// Wait for the completed span named `name`. + pub async fn wait_for_span(&self, name: &str) -> opentelemetry_sdk::trace::SpanData { + let spans = self + .wait_for_spans(|spans| spans.iter().any(|span| span.name == name)) + .await; + spans + .into_iter() + .find(|span| span.name == name) + .expect("the awaited snapshot contains the expected span") + } + /// Spans named `name`. pub fn spans_named(&self, name: &str) -> Vec { self.finished_spans() @@ -255,7 +414,7 @@ pub mod test_exporter { /// Forces the global subscriber up first so callsite interest is decided /// by a registry that records, not by the no-op default. #[must_use] - pub fn install_scoped(subscriber: impl Into) -> ScopedTracingTestGuard { + pub fn install_scoped(subscriber: impl Into) -> ScopedTracingTestGuard { let lock = crate::TEST_TRACING_LOCK .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); @@ -276,6 +435,7 @@ pub mod test_exporter { _default: tracing::dispatcher::DefaultGuard, _provider: opentelemetry_sdk::trace::SdkTracerProvider, exporter: opentelemetry_sdk::trace::InMemorySpanExporter, + closed: Arc, _lock: std::sync::MutexGuard<'static, ()>, } @@ -518,6 +678,109 @@ mod tests { assert!(err.is_some(), "the failure is reportable, not swallowed"); } + #[tokio::test] + async fn tracing_child_closed_on_worker_keeps_its_parent_and_exporter() { + use opentelemetry::trace::TraceContextExt as _; + use tracing_opentelemetry::OpenTelemetrySpanExt as _; + + let traced = test_exporter::install_traced(); + let parent = tracing::info_span!("worker_parent"); + let child = tracing::info_span!(parent: &parent, "worker_child"); + // Also exercise layer downcasting through the fixture's subscriber. + let parent_context = parent.context(); + let child_context = child.context(); + drop(parent); + + let (release, released) = std::sync::mpsc::channel(); + let worker = std::thread::spawn(move || { + released.recv().expect("test releases the worker's span"); + // Like SQLx, enter the carried span without installing its dispatcher. + let entered = child.enter(); + drop(entered); + // This is deliberately the child's last reference. Its parent has + // no remaining references either, so both must close on this worker. + drop(child); + // Closing the carried span must also restore the worker's default, + // so unrelated work cannot leak into this test's private exporter. + drop(tracing::info_span!("unrelated_worker_span")); + }); + + assert!(traced.finished_spans().is_empty()); + let waiting = traced.wait_for_span("worker_parent"); + tokio::pin!(waiting); + assert!(futures::poll!(waiting.as_mut()).is_pending()); + + release.send(()).unwrap(); + let parent = waiting.await; + worker + .join() + .expect("worker closes spans without consulting the global registry"); + let child = traced.span_named("worker_child"); + test_exporter::assert_is_root(&parent); + assert_eq!(child.parent_span_id, parent.span_context.span_id()); + assert_eq!( + child.span_context.trace_id(), + parent.span_context.trace_id() + ); + assert_eq!(parent_context.span().span_context(), &parent.span_context); + assert_eq!(child_context.span().span_context(), &child.span_context); + assert_eq!(traced.finished_spans().len(), 2); + } + + #[tokio::test] + async fn tracing_exporters_isolate_unrelated_threads_and_successive_tests() { + // Use the same callsite under all dispatchers to exercise the global + // interest cache without sharing their captured spans. + fn emit_span() { + drop(tracing::info_span!("isolated_test_span")); + } + + let traced = test_exporter::install_traced(); + emit_span(); + std::thread::spawn(emit_span) + .join() + .expect("unrelated worker records only into the global registry"); + let first = traced.span_named("isolated_test_span"); + test_exporter::assert_is_root(&first); + assert_eq!(traced.finished_spans().len(), 1); + drop(traced); + + let traced = test_exporter::install_traced(); + assert!(traced.finished_spans().is_empty()); + emit_span(); + let second = traced.span_named("isolated_test_span"); + test_exporter::assert_is_root(&second); + assert_eq!(traced.finished_spans().len(), 1); + assert_ne!( + first.span_context.trace_id(), + second.span_context.trace_id() + ); + } + + #[tokio::test] + async fn tracing_wait_does_not_lose_closure_between_snapshot_and_await() { + let traced = test_exporter::install_traced(); + let span = std::sync::Mutex::new(Some(tracing::info_span!("close_before_await"))); + let spans = traced + .wait_for_spans(|spans| { + // The first snapshot is empty. Close its span before polling + // the notification future, as a worker could do concurrently. + drop(span.lock().unwrap().take()); + spans.iter().any(|span| span.name == "close_before_await") + }) + .await; + assert_eq!(spans.len(), 1); + test_exporter::assert_is_root(&spans[0]); + } + + #[tokio::test(start_paused = true)] + #[should_panic(expected = "timed out waiting for expected spans")] + async fn tracing_wait_times_out_when_a_span_never_closes() { + let traced = test_exporter::install_traced(); + let _span = tracing::info_span!("still_open"); + traced.wait_for_span("still_open").await; + } + #[tokio::test] async fn tracing_events_are_not_exported() { let traced = test_exporter::install_traced(); diff --git a/crates/openshell-server/src/persistence/tests.rs b/crates/openshell-server/src/persistence/tests.rs index 7ba0bd1245..a62cbd2b42 100644 --- a/crates/openshell-server/src/persistence/tests.rs +++ b/crates/openshell-server/src/persistence/tests.rs @@ -35,7 +35,6 @@ async fn failed_store_calls_are_marked_on_the_span() { /// the span must stay clean β€” otherwise every lease a replica does not win, and /// every gateway restart, exports as a failure. #[tokio::test] -#[ignore = "flaky under concurrent test execution"] async fn expected_conflicts_leave_the_span_unmarked() { use crate::otel_tracing::test_exporter; @@ -66,6 +65,7 @@ async fn expected_conflicts_leave_the_span_unmarked() { .await .expect_err("the name is already taken"); + traced.wait_for_span("store.put_if").await; let span = traced.span_with("store.put_if", "object.id", "expected-conflict-second"); assert_eq!( @@ -79,7 +79,6 @@ async fn expected_conflicts_leave_the_span_unmarked() { /// Span names stay low-cardinality so they group across object types; what /// each call touched is carried as attributes. #[tokio::test] -#[ignore = "flaky under concurrent test execution"] async fn store_spans_record_what_they_touched_as_attributes() { use crate::otel_tracing::test_exporter; @@ -97,6 +96,13 @@ async fn store_spans_record_what_they_touched_as_attributes() { .unwrap(); store.list("sandbox", "default", 10, 0).await.unwrap(); + traced + .wait_for_spans(|spans| { + ["store.get", "store.get_by_name", "store.list"] + .iter() + .all(|name| spans.iter().any(|span| &span.name == name)) + }) + .await; let by_name = traced.span_with("store.get_by_name", "object.name", "my-sandbox"); assert_eq!( test_exporter::attribute(&by_name, "object_type").as_deref(), @@ -2807,7 +2813,6 @@ async fn membership_selector_escapes_adversarial_label_key() { /// so a trace decomposes an RPC into the storage work it did rather than /// bottoming out at the request boundary. #[tokio::test] -#[ignore = "flaky under concurrent test execution"] async fn store_operations_export_spans_with_parents() { use tracing::Instrument as _; @@ -2827,7 +2832,7 @@ async fn store_operations_export_spans_with_parents() { .await; drop(request_span); - let root = traced.span_named("request"); + let root = traced.wait_for_span("request").await; let spans = traced.finished_spans(); let child = spans .iter() diff --git a/crates/openshell-server/src/provider_refresh.rs b/crates/openshell-server/src/provider_refresh.rs index c7d98c9917..c1b5184d18 100644 --- a/crates/openshell-server/src/provider_refresh.rs +++ b/crates/openshell-server/src/provider_refresh.rs @@ -3580,7 +3580,6 @@ mod tests { } #[tokio::test] - #[ignore = "flaky under concurrent test execution"] async fn refresh_worker_records_a_root_span_only_when_a_state_has_work() { use crate::otel_tracing::test_exporter; @@ -3621,7 +3620,7 @@ mod tests { Box::pin(run_refresh_worker_tick(&store, None, None)) .await .unwrap(); - test_exporter::assert_is_root(&traced.span_named("refresh.provider_credentials")); + test_exporter::assert_is_root(&traced.wait_for_span("refresh.provider_credentials").await); } #[test] diff --git a/crates/openshell-supervisor-network/Cargo.toml b/crates/openshell-supervisor-network/Cargo.toml index 300db03acd..c093c46ae9 100644 --- a/crates/openshell-supervisor-network/Cargo.toml +++ b/crates/openshell-supervisor-network/Cargo.toml @@ -63,7 +63,9 @@ bundled-ca-roots = ["dep:webpki-roots"] [dev-dependencies] openshell-ocsf = { path = "../openshell-ocsf", features = ["test-support"] } +openshell-otel = { path = "../openshell-otel" } openshell-supervisor-middleware-builtins = { path = "../openshell-supervisor-middleware-builtins" } +opentelemetry_sdk = { workspace = true, features = ["testing"] } tonic = { workspace = true } temp-env = "0.3" tokio = { workspace = true, features = ["test-util"] } diff --git a/crates/openshell-supervisor-network/src/l7/relay.rs b/crates/openshell-supervisor-network/src/l7/relay.rs index d0351e2153..bc45964137 100644 --- a/crates/openshell-supervisor-network/src/l7/relay.rs +++ b/crates/openshell-supervisor-network/src/l7/relay.rs @@ -5021,6 +5021,130 @@ network_policies: ); } + #[tokio::test] + async fn l7_rest_tls_relay_injects_multiple_grants() { + assert_multiple_grants_tls_relay(Ok("identity-token")).await; + } + + #[tokio::test] + async fn l7_rest_tls_relay_second_grant_failure_forwards_nothing() { + assert_multiple_grants_tls_relay(Err("issuer echoed identity-secret")).await; + } + + async fn token_grant_tls_pair() -> ( + tokio_rustls::client::TlsStream, + tokio_rustls::server::TlsStream, + ) { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + let key = rcgen::KeyPair::generate().unwrap(); + let cert = rcgen::CertificateParams::new(vec!["api.example.test".into()]) + .unwrap() + .self_signed(&key) + .unwrap(); + let server_config = rustls::ServerConfig::builder() + .with_no_client_auth() + .with_single_cert( + vec![cert.der().clone()], + rustls::pki_types::PrivatePkcs8KeyDer::from(key.serialize_der()).into(), + ) + .unwrap(); + let mut roots = rustls::RootCertStore::empty(); + roots.add(cert.der().clone()).unwrap(); + let client_config = rustls::ClientConfig::builder() + .with_root_certificates(roots) + .with_no_client_auth(); + let connector = tokio_rustls::TlsConnector::from(Arc::new(client_config)); + let acceptor = tokio_rustls::TlsAcceptor::from(Arc::new(server_config)); + let (client, server) = tokio::io::duplex(16384); + let (client, server) = tokio::join!( + connector.connect("api.example.test".try_into().unwrap(), client), + acceptor.accept(server), + ); + (client.unwrap(), server.unwrap()) + } + + async fn assert_multiple_grants_tls_relay(identity_result: std::result::Result<&str, &str>) { + let (config, tunnel_engine, ctx, fixture) = + rest_token_grant_relay_context(Ok("service-token")); + let service_key = "api.example.test\t8080\t/v1/**\tprovider:access_token"; + let identity_key = "api.example.test\t8080\t/v1/**\tprovider:identity"; + let mut identity = fixture.dynamic_credentials().read().unwrap()[service_key].clone(); + identity.name = "identity".into(); + identity.auth_style = "header".into(); + identity.header_name = "X-Workload-Jwt".into(); + fixture.add_credential(identity_key, identity, identity_result); + // Both sides verify a synthetic certificate: the test exercises encrypted + // application traffic, inspection and credential injection, then upstream TLS. + let (mut app, mut relay_client) = token_grant_tls_pair().await; + let (mut relay_upstream, mut upstream) = token_grant_tls_pair().await; + let relay = tokio::spawn(async move { + relay_with_inspection( + &config, + tunnel_engine, + &mut relay_client, + &mut relay_upstream, + &ctx, + ) + .await + }); + app.write_all(b"GET /v1/projects HTTP/1.1\r\nHost: api.example.test\r\nAuthorization: Bearer agent-token\r\nX-Workload-Jwt: agent-identity\r\nConnection: close\r\n\r\n") + .await.unwrap(); + if identity_result.is_ok() { + let mut request = [0u8; 2048]; + let n = tokio::time::timeout( + std::time::Duration::from_secs(2), + upstream.read(&mut request), + ) + .await + .unwrap() + .unwrap(); + let request = String::from_utf8_lossy(&request[..n]); + assert!(request.contains("Authorization: Bearer service-token\r\n")); + assert!(request.contains("X-Workload-Jwt: identity-token\r\n")); + assert!(!request.contains("agent-token")); + assert!(!request.contains("agent-identity")); + upstream + .write_all( + b"HTTP/1.1 204 No Content\r\nContent-Length: 0\r\nConnection: close\r\n\r\n", + ) + .await + .unwrap(); + } + let mut response = [0u8; 1024]; + let n = tokio::time::timeout(std::time::Duration::from_secs(2), app.read(&mut response)) + .await + .unwrap() + .unwrap(); + let response = String::from_utf8_lossy(&response[..n]); + assert!(response.contains(if identity_result.is_ok() { + "204 No Content" + } else { + "502 Bad Gateway" + })); + assert!(!response.contains("service-token")); + assert!(!response.contains("identity-secret")); + drop(app); + tokio::time::timeout(std::time::Duration::from_secs(2), relay) + .await + .unwrap() + .unwrap() + .unwrap(); + if identity_result.is_err() { + let mut request = [0u8; 128]; + match tokio::time::timeout( + std::time::Duration::from_secs(2), + upstream.read(&mut request), + ) + .await + .unwrap() + { + Ok(n) => assert_eq!(n, 0, "failed grant must send no request bytes"), + Err(error) => assert_eq!(error.kind(), std::io::ErrorKind::UnexpectedEof), + } + } + fixture.assert_requested_keys(&[service_key, identity_key]); + } + #[tokio::test] async fn l7_rest_relay_injects_token_grant_authorization_header() { let (config, tunnel_engine, ctx, fixture) = diff --git a/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs b/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs index 1813ddcf4b..98d22c4854 100644 --- a/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs +++ b/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs @@ -3,6 +3,10 @@ //! Endpoint-bound dynamic token grant injection for HTTP relay paths. +#[cfg(test)] +mod multiple_grants; + +use std::collections::BTreeMap; use std::future::Future; use std::pin::Pin; use std::sync::Arc; @@ -68,13 +72,18 @@ pub fn default_resolver() -> Arc { Arc::new(SpiffeTokenGrantResolver) } -/// Checks for endpoint-bound token grant credentials and injects an -/// Authorization header before forwarding the request upstream. +/// Resolves one endpoint-bound grant per protected header before rewriting a request. +/// +/// Each header independently uses its most-specific matching binding. Every selected +/// grant must succeed; callers must not forward the request when this returns an error. pub async fn inject_if_needed(req: L7Request, ctx: &L7EvalContext) -> Result { let request_path = req.target.split('?').next().unwrap_or(req.target.as_str()); - let token_grant_credential = ctx.dynamic_credentials.as_ref().and_then(|dyn_creds| { - dyn_creds.read().map_or(None, |creds_guard| { - creds_guard + let credentials = match ctx.dynamic_credentials.as_ref() { + Some(dynamic_credentials) => { + let credentials = dynamic_credentials + .read() + .map_err(|_| miette!("dynamic credential snapshot unavailable"))?; + let candidates = credentials .iter() .filter_map(|(key, cred)| { let score = @@ -83,83 +92,117 @@ pub async fn inject_if_needed(req: L7Request, ctx: &L7EvalContext) -> Result Vec::new(), + }; + if credentials.is_empty() { + return Ok(req); + } + let resolver = ctx + .token_grant_resolver + .as_ref() + .ok_or_else(|| miette!("token grant resolver unavailable"))?; + + // Release the credential snapshot lock before acquisition. Keep all token values + // local until every selected grant and header has been validated, so a later + // failure cannot expose a partially authenticated request to a relay caller. + let mut headers = Vec::with_capacity(credentials.len()); + for (provider_key, cred) in &credentials { + let token_grant = cred + .token_grant .as_ref() - .ok_or_else(|| miette!("token grant resolver unavailable"))?; - let request = token_grant_request(&provider_key, token_grant)?; - - match resolver.obtain(request).await { - Ok(access_token) => { - let modified_raw_header = - inject_token_grant_header(&req.raw_header, &cred, &access_token)?; - let provider_key = ocsf_message_field(&provider_key); - ocsf_emit!( - HttpActivityBuilder::new(ocsf_ctx()) - .activity(ActivityId::Other) - .action(ActionId::Allowed) - .disposition(DispositionId::Allowed) - .severity(SeverityId::Informational) - .http_request(HttpRequest::new( - &req.action, - OcsfUrl::new("http", &ctx.host, request_path, ctx.port), - )) - .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) - .message(format!( - "Token grant successful for {} to {}:{}", - provider_key, ctx.host, ctx.port - )) - .build() - ); - return Ok(L7Request { - action: req.action, - target: req.target, - query_params: req.query_params, - raw_header: modified_raw_header, - body_length: req.body_length, - }); - } - Err(e) => { - warn!( - host = %ctx.host, - port = ctx.port, - provider = %provider_key, - error = %e, - "Token grant failed: {e}" - ); - let provider_key = ocsf_message_field(&provider_key); - ocsf_emit!( - HttpActivityBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .http_request(HttpRequest::new( - &req.action, - OcsfUrl::new("http", &ctx.host, request_path, ctx.port), - )) - .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) - .message(format!( - "Token grant failed for {} to {}:{}: {}", - provider_key, ctx.host, ctx.port, e - )) - .build() - ); - return Err(miette!("Token grant failed: {}", e)); - } + .ok_or_else(|| miette!("selected credential has no token grant"))?; + let request = token_grant_request(provider_key, token_grant)?; + if let Ok(access_token) = resolver.obtain(request).await { + crate::token_grant::validate_access_token(&access_token)?; + headers.push(token_grant_header(cred, &access_token)?); + } else { + // An issuer may echo credentials in its error description. Only the + // binding identity is safe to include in diagnostics or relay errors. + let provider_key = ocsf_message_field(provider_key); + warn!( + host = %ctx.host, + port = ctx.port, + provider = %provider_key, + "Token grant failed" + ); + ocsf_emit!( + HttpActivityBuilder::new(ocsf_ctx()) + .activity(ActivityId::Fail) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .http_request(HttpRequest::new( + &req.action, + OcsfUrl::new("http", &ctx.host, request_path, ctx.port), + )) + .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) + .message(format!( + "Token grant failed for {} to {}:{}", + provider_key, ctx.host, ctx.port + )) + .build() + ); + return Err(miette!("Token grant failed")); } } - Ok(req) + let mut raw_header = req.raw_header; + for (name, value) in headers { + raw_header = inject_header(&raw_header, &name, &value)?; + } + for (provider_key, _) in credentials { + ocsf_emit!( + HttpActivityBuilder::new(ocsf_ctx()) + .activity(ActivityId::Other) + .action(ActionId::Allowed) + .disposition(DispositionId::Allowed) + .severity(SeverityId::Informational) + .http_request(HttpRequest::new( + &req.action, + OcsfUrl::new("http", &ctx.host, request_path, ctx.port), + )) + .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) + .message(format!( + "Token grant successful for {} to {}:{}", + ocsf_message_field(&provider_key), + ctx.host, + ctx.port + )) + .build() + ); + } + Ok(L7Request { raw_header, ..req }) +} + +fn select_token_grants( + mut candidates: Vec<(u32, String, ProviderProfileCredential)>, +) -> Result> { + // Examine the strongest bindings first. The complete binding key stays with + // its grant because the resolver uses it to separate cache entries and revisions. + candidates.sort_unstable_by(|a, b| b.0.cmp(&a.0).then_with(|| b.1.cmp(&a.1))); + let mut selected = BTreeMap::::new(); + for (score, key, credential) in candidates { + let header = token_grant_header_name(&credential)?.to_ascii_lowercase(); + if let Some((selected_score, selected_key, _)) = selected.get(&header) { + // Equal-specificity selectors of the same credential can overlap. A + // different credential cannot win a tie for the same protected header. + if score == *selected_score + && key.rsplit('\t').next() != selected_key.rsplit('\t').next() + { + return Err(miette!("ambiguous dynamic token grants for one header")); + } + continue; + } + selected.insert(header, (score, key, credential)); + } + Ok(selected + .into_values() + .map(|(_, key, cred)| (key, cred)) + .collect()) } fn ocsf_message_field(value: &str) -> String { @@ -246,6 +289,7 @@ fn count_as_u32(count: usize) -> u32 { u32::try_from(count).unwrap_or(u32::MAX) } +#[cfg(test)] fn inject_token_grant_header( raw_header: &[u8], credential: &ProviderProfileCredential, @@ -260,6 +304,16 @@ fn token_grant_header( credential: &ProviderProfileCredential, access_token: &str, ) -> Result<(String, String)> { + let header_name = token_grant_header_name(credential)?; + let header_value = if credential.auth_style.trim().eq_ignore_ascii_case("header") { + access_token.to_string() + } else { + format!("Bearer {access_token}") + }; + Ok((header_name.to_string(), header_value)) +} + +fn token_grant_header_name(credential: &ProviderProfileCredential) -> Result<&str> { match credential.auth_style.trim().to_ascii_lowercase().as_str() { "" | "bearer" => { let header_name = if credential.header_name.trim().is_empty() { @@ -268,7 +322,7 @@ fn token_grant_header( credential.header_name.trim() }; validate_header_name(header_name)?; - Ok((header_name.to_string(), format!("Bearer {access_token}"))) + Ok(header_name) } "header" => { let header_name = credential.header_name.trim(); @@ -278,7 +332,7 @@ fn token_grant_header( )); } validate_header_name(header_name)?; - Ok((header_name.to_string(), access_token.to_string())) + Ok(header_name) } other => Err(miette!( "token grant auth_style '{other}' is not supported; use bearer or header" @@ -371,7 +425,7 @@ pub mod test_support { struct FakeTokenGrantResolver { requests: Arc>>, - response: std::result::Result, + responses: Arc>>>, } #[derive(Debug, Clone, PartialEq, Eq)] @@ -391,6 +445,7 @@ pub mod test_support { dynamic_credentials: Arc>>, resolver: Arc, requests: Arc>>, + responses: Arc>>>, } impl TokenGrantTestFixture { @@ -420,9 +475,13 @@ pub mod test_support { token_grant: ProviderCredentialTokenGrant, ) -> Self { let requests = Arc::new(Mutex::new(Vec::new())); + let responses = Arc::new(Mutex::new(HashMap::from([( + key.to_string(), + response.map(str::to_string).map_err(str::to_string), + )]))); let resolver = Arc::new(FakeTokenGrantResolver { requests: requests.clone(), - response: response.map(str::to_string).map_err(str::to_string), + responses: responses.clone(), }); let mut dynamic_credentials = HashMap::new(); @@ -441,9 +500,65 @@ pub mod test_support { dynamic_credentials: Arc::new(std::sync::RwLock::new(dynamic_credentials)), resolver, requests, + responses, } } + pub fn add_credential( + &self, + key: &str, + credential: ProviderProfileCredential, + response: std::result::Result<&str, &str>, + ) { + self.dynamic_credentials + .write() + .unwrap() + .insert(key.to_string(), credential); + self.responses.lock().unwrap().insert( + key.to_string(), + response.map(str::to_string).map_err(str::to_string), + ); + } + + pub fn assert_requested_keys(&self, expected: &[&str]) { + let requests = self.requests.lock().unwrap(); + assert_eq!( + requests + .iter() + .map(|r| r.provider_key.as_str()) + .collect::>(), + expected + ); + } + + pub fn assert_request_configuration( + &self, + key: &str, + grant: &ProviderCredentialTokenGrant, + ) { + let requests = self.requests.lock().unwrap(); + let request = requests + .iter() + .find(|r| r.provider_key == key) + .expect("grant was requested"); + assert_eq!(request.token_endpoint, grant.token_endpoint); + assert_eq!(request.jwt_svid_audience, grant.jwt_svid_audience); + assert_eq!(request.client_assertion_type, grant.client_assertion_type); + assert_eq!(request.audience, grant.audience); + assert_eq!(request.scopes, grant.scopes); + assert_eq!( + request.cache_ttl, + grant + .cache_ttl + .as_ref() + .map(openshell_core::time::duration_to_std) + .transpose() + .unwrap() + ); + assert_eq!(request.grant_type, grant.grant_type); + assert_eq!(request.requested_token_type, grant.requested_token_type); + } + pub fn dynamic_credentials( &self, ) -> Arc>> { @@ -571,7 +686,16 @@ pub mod test_support { .lock() .expect("fake token grant requests lock poisoned") .push(owned); - self.response.clone().map_err(|err| miette!("{err}")) + // Keep acquisition pending once so joined request tests exercise + // overlapping grants instead of completing each branch in one poll. + tokio::task::yield_now().await; + self.responses + .lock() + .unwrap() + .get(request.provider_key) + .expect("fake grant response configured") + .clone() + .map_err(|err| miette!("{err}")) }) } } diff --git a/crates/openshell-supervisor-network/src/l7/token_grant_injection/multiple_grants.rs b/crates/openshell-supervisor-network/src/l7/token_grant_injection/multiple_grants.rs new file mode 100644 index 0000000000..cb46fa883a --- /dev/null +++ b/crates/openshell-supervisor-network/src/l7/token_grant_injection/multiple_grants.rs @@ -0,0 +1,257 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use super::test_support::TokenGrantTestFixture; +use super::*; +use crate::l7::provider::BodyLength; +use std::collections::HashMap; +use std::sync::Mutex; +use tracing::instrument::WithSubscriber; +use tracing_subscriber::layer::SubscriberExt; + +const SERVICE: &str = "api.example.com\t443\t/v1/**\trev:7\tprovider:service"; +const IDENTITY: &str = "api.example.com\t443\t/v1/**\trev:7\tprovider:identity"; + +fn request() -> L7Request { + L7Request { + action: "POST".into(), + target: "/v1/projects?view=full".into(), + query_params: HashMap::default(), + raw_header: b"POST /v1/projects?view=full HTTP/1.1\r\nHost: api.example.com\r\nAuthorization: Bearer agent-token\r\nauthorization : duplicate\r\nX-Workload-Jwt: agent-identity\r\nx-workload-jwt: duplicate\r\nX-Static: openshell:placeholder\r\nContent-Length: 4\r\n\r\nbody".to_vec(), + body_length: BodyLength::ContentLength(4), + } +} + +fn fixture( + identity_result: std::result::Result<&str, &str>, +) -> (TokenGrantTestFixture, L7EvalContext) { + let fixture = TokenGrantTestFixture::success(SERVICE, "service-token"); + let mut identity = fixture.dynamic_credentials().read().unwrap()[SERVICE].clone(); + identity.name = "identity".into(); + identity.auth_style = "header".into(); + identity.header_name = "X-Workload-Jwt".into(); + let grant = identity.token_grant.as_mut().unwrap(); + grant.token_endpoint = "https://identity.example.com/token".into(); + grant.jwt_svid_audience = "identity-proxy".into(); + grant.audience = "workload".into(); + grant.scopes = vec!["identity.read".into()]; + grant.cache_ttl = Some(prost_types::Duration { + seconds: 45, + nanos: 0, + }); + fixture.add_credential(IDENTITY, identity, identity_result); + let ctx = L7EvalContext { + host: "api.example.com".into(), + port: 443, + dynamic_credentials: Some(fixture.dynamic_credentials()), + token_grant_resolver: Some(fixture.resolver()), + ..Default::default() + }; + (fixture, ctx) +} + +#[tokio::test] +async fn injects_independent_grants_and_replaces_all_protected_headers() { + let (fixture, ctx) = fixture(Ok("identity-token")); + let rewritten = inject_if_needed(request(), &ctx).await.unwrap(); + let bytes = String::from_utf8(rewritten.raw_header).unwrap(); + assert_eq!( + bytes + .matches("Authorization: Bearer service-token\r\n") + .count(), + 1 + ); + assert_eq!( + bytes.matches("X-Workload-Jwt: identity-token\r\n").count(), + 1 + ); + assert!(!bytes.contains("agent-token")); + assert!(!bytes.contains("agent-identity")); + assert!(!bytes.contains("duplicate")); + assert!(bytes.contains("X-Static: openshell:placeholder\r\n")); + assert!(bytes.ends_with("\r\n\r\nbody")); + fixture.assert_requested_keys(&[SERVICE, IDENTITY]); + for key in [SERVICE, IDENTITY] { + let credentials = fixture.dynamic_credentials(); + let credentials = credentials.read().unwrap(); + fixture.assert_request_configuration(key, credentials[key].token_grant.as_ref().unwrap()); + } +} + +#[tokio::test] +async fn chooses_specific_binding_independently_for_each_header() { + let (fixture, ctx) = fixture(Ok("identity-token")); + let specific_key = "api.example.com\t443\t/v1/projects\trev:7\tother:service"; + let mut specific = fixture.dynamic_credentials().read().unwrap()[SERVICE].clone(); + specific.token_grant.as_mut().unwrap().audience = "projects-only".into(); + fixture.add_credential(specific_key, specific, Ok("specific-service")); + let rewritten = inject_if_needed(request(), &ctx).await.unwrap(); + let bytes = String::from_utf8(rewritten.raw_header).unwrap(); + assert!(bytes.contains("Authorization: Bearer specific-service\r\n")); + assert!(bytes.contains("X-Workload-Jwt: identity-token\r\n")); + fixture.assert_requested_keys(&[specific_key, IDENTITY]); +} + +#[tokio::test] +async fn audience_override_changes_only_its_own_credential() { + let (fixture, ctx) = fixture(Ok("identity-token")); + let override_key = "api.example.com\t443\t/v1/projects\trev:7\tprovider:identity"; + let mut identity = fixture.dynamic_credentials().read().unwrap()[IDENTITY].clone(); + identity.token_grant.as_mut().unwrap().audience = "project-identity".into(); + fixture.add_credential(override_key, identity.clone(), Ok("project-identity-token")); + let rewritten = inject_if_needed(request(), &ctx).await.unwrap(); + let bytes = String::from_utf8(rewritten.raw_header).unwrap(); + assert!(bytes.contains("Authorization: Bearer service-token\r\n")); + assert!(bytes.contains("X-Workload-Jwt: project-identity-token\r\n")); + fixture.assert_requested_keys(&[SERVICE, override_key]); + fixture.assert_request_configuration(override_key, identity.token_grant.as_ref().unwrap()); +} + +#[tokio::test] +async fn rejects_tied_credentials_for_same_header_before_acquisition() { + for header in ["Authorization", " authorization ", ""] { + let (fixture, ctx) = fixture(Ok("identity-token")); + let mut collision = fixture.dynamic_credentials().read().unwrap()[SERVICE].clone(); + collision.name = "collision".into(); + collision.header_name = header.into(); + fixture.add_credential( + "api.example.com\t443\t/v1/**\trev:7\tother:collision", + collision, + Ok("collision-token"), + ); + let error = inject_if_needed(request(), &ctx).await.unwrap_err(); + assert_eq!( + error.to_string(), + "ambiguous dynamic token grants for one header" + ); + fixture.assert_no_requests(); + } +} + +#[tokio::test] +async fn rejects_invalid_second_header_before_acquisition() { + let (fixture, ctx) = fixture(Ok("identity-token")); + let mut identity = fixture.dynamic_credentials().read().unwrap()[IDENTITY].clone(); + identity.header_name = "Content-Length".into(); + fixture.add_credential(IDENTITY, identity, Ok("identity-token")); + assert!(inject_if_needed(request(), &ctx).await.is_err()); + fixture.assert_no_requests(); +} + +#[tokio::test] +async fn rejects_malformed_second_token_after_first_grant_succeeds() { + let (fixture, ctx) = fixture(Ok("identity-token\r\nInjected: yes")); + let error = inject_if_needed(request(), &ctx).await.unwrap_err(); + assert_eq!( + error.to_string(), + "token grant returned a malformed access token" + ); + fixture.assert_requested_keys(&[SERVICE, IDENTITY]); +} + +#[derive(Clone)] +struct Capture(Arc>>); + +impl std::io::Write for Capture { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + self.0.lock().unwrap().extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +#[tokio::test] +async fn second_grant_failure_redacts_errors_and_emits_no_success() { + // OCSF callsite interest is process-global. Run capture alone so unrelated + // parallel tests cannot disable the event while its subscriber is installed. + const CAPTURE_CHILD: &str = "OPENSHELL_MULTIPLE_GRANTS_CAPTURE_CHILD"; + if std::env::var_os(CAPTURE_CHILD).is_none() { + let output = std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "l7::token_grant_injection::multiple_grants::second_grant_failure_redacts_errors_and_emits_no_success", + "--nocapture", + ]) + .env(CAPTURE_CHILD, "1") + .output() + .unwrap(); + assert!( + output.status.success(), + "isolated capture failed: {}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + return; + } + let (fixture, ctx) = fixture(Err("issuer echoed service-token and identity-secret")); + let logs = Arc::new(Mutex::new(Vec::new())); + let events = Arc::new(Mutex::new(Vec::new())); + let log_writer = Capture(logs.clone()); + let subscriber = tracing_subscriber::registry() + .with( + tracing_subscriber::fmt::layer() + .with_ansi(false) + .without_time() + .with_writer(move || log_writer.clone()), + ) + .with(openshell_ocsf::OcsfJsonlLayer::new(Capture(events.clone()))); + let error = inject_if_needed(request(), &ctx) + .with_subscriber(subscriber) + .await + .unwrap_err(); + assert_eq!(error.to_string(), "Token grant failed"); + let logs = String::from_utf8(logs.lock().unwrap().clone()).unwrap(); + let events = String::from_utf8(events.lock().unwrap().clone()).unwrap(); + assert!(logs.contains("Token grant failed")); + assert!(events.contains("Token grant failed")); + for output in [&logs, &events] { + assert!(!output.contains("service-token")); + assert!(!output.contains("identity-secret")); + assert!(!output.contains("Token grant successful")); + } + fixture.assert_requested_keys(&[SERVICE, IDENTITY]); +} + +#[tokio::test] +async fn does_not_acquire_grants_outside_endpoint_bindings() { + for (host, port, path) in [ + ("other.example.com", 443, "/v1/projects"), + ("api.example.com", 8443, "/v1/projects"), + ("api.example.com", 443, "/private/projects"), + ] { + let (fixture, mut ctx) = fixture(Ok("identity-token")); + ctx.host = host.into(); + ctx.port = port; + let mut req = request(); + req.target = path.into(); + let original = req.raw_header.clone(); + assert_eq!( + inject_if_needed(req, &ctx).await.unwrap().raw_header, + original + ); + fixture.assert_no_requests(); + } +} + +#[tokio::test] +async fn concurrent_requests_keep_credential_snapshots_separate() { + let (first, first_ctx) = fixture(Ok("first-identity")); + let (second, second_ctx) = fixture(Ok("second-identity")); + let (first_result, second_result) = tokio::join!( + inject_if_needed(request(), &first_ctx), + inject_if_needed(request(), &second_ctx), + ); + for (result, expected, absent) in [ + (first_result, "first-identity", "second-identity"), + (second_result, "second-identity", "first-identity"), + ] { + let bytes = String::from_utf8(result.unwrap().raw_header).unwrap(); + assert!(bytes.contains(expected)); + assert!(!bytes.contains(absent)); + } + first.assert_requested_keys(&[SERVICE, IDENTITY]); + second.assert_requested_keys(&[SERVICE, IDENTITY]); +} diff --git a/crates/openshell-supervisor-network/src/proxy.rs b/crates/openshell-supervisor-network/src/proxy.rs index 7064cda008..df5175acef 100644 --- a/crates/openshell-supervisor-network/src/proxy.rs +++ b/crates/openshell-supervisor-network/src/proxy.rs @@ -56,7 +56,7 @@ use tokio::net::TcpListener; use tokio::net::TcpStream; use tokio::sync::mpsc; use tokio::task::JoinHandle; -use tracing::{debug, warn}; +use tracing::{Instrument as _, debug, warn}; type ProxyClient = tokio::io::BufReader; type AcceptedProxyConnection = ( @@ -69,6 +69,7 @@ type AcceptedProxyConnection = ( struct TransparentOpen { destination: SocketAddr, authorization: Option<(EgressDecision, destination::UpstreamConnector)>, + connect_span: Option, } enum ProxyAcceptError { @@ -636,6 +637,7 @@ async fn preauthorize_transparent_open( Some(TransparentOpen { destination, authorization: None, + connect_span: None, }), )); } @@ -685,6 +687,7 @@ async fn preauthorize_transparent_open( Some(TransparentOpen { destination, authorization: None, + connect_span: None, }), )); } @@ -705,12 +708,16 @@ async fn preauthorize_transparent_open( } }; let mapped_host = intent.is_some().then_some(host.as_str()); - let supplied_authorization = authorize_supplied_identity_with_denial( - opa_engine, - identity_cache, - EgressIntent::connect(host.clone(), destination.port()), - &binary_identity, - ); + let connect_intent = EgressIntent::connect(host.clone(), destination.port()); + let connect_span = egress::connect_span(&connect_intent); + let supplied_authorization = connect_span.in_scope(|| { + authorize_supplied_identity_with_denial( + opa_engine, + identity_cache, + connect_intent, + &binary_identity, + ) + }); let mut decision = supplied_authorization.decision; if let NetworkAction::Deny { reason } = &decision.action { let (denial, status_detail) = supplied_authorization.denial.map_or( @@ -819,6 +826,7 @@ async fn preauthorize_transparent_open( sandbox_entrypoint_pid: 0, plan, }) + .instrument(egress::resolve_span(&connect_span)) .await { Ok(connector) => connector, @@ -845,6 +853,7 @@ async fn preauthorize_transparent_open( Some(TransparentOpen { destination, authorization: Some((decision, connector)), + connect_span: Some(connect_span), }), )) } @@ -1123,11 +1132,13 @@ async fn handle_transparent_tcp_connection( let port = original.port(); let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, original); let intent = EgressIntent::transparent_tcp(host.clone(), port); + let connect_span = egress::connect_span(&intent); let engine = opa_engine.clone(); let cache = identity_cache.clone(); let pid = entrypoint_pid.clone(); + let parent = connect_span.clone(); let decision = tokio::task::spawn_blocking(move || { - authorize_egress_intent(connection, &engine, &cache, &pid, intent) + parent.in_scope(|| authorize_egress_intent(connection, &engine, &cache, &pid, intent)) }) .await .map_err(|error| miette::miette!("identity resolution task panicked: {error}"))?; @@ -1220,9 +1231,13 @@ async fn handle_transparent_tcp_connection( return Ok(()); }; - let connector = mapping.connector_for(&endpoint_id).await.map_err(|error| { - miette::miette!("transparent TCP pinned destination is invalid: {error}") - })?; + let connector = mapping + .connector_for(&endpoint_id) + .instrument(egress::resolve_span(&connect_span)) + .await + .map_err(|error| { + miette::miette!("transparent TCP pinned destination is invalid: {error}") + })?; let mut ctx = relay::http_context( &decision, None, @@ -1248,10 +1263,15 @@ async fn handle_transparent_tcp_connection( } let approved_real_ip_candidates = connector.addrs().to_vec(); generation_guard.ensure_current()?; + let dial_span = egress::dial_span(&connect_span); let mut upstream = dial_transparent_upstream(&upstream_proxy, &host, port, &approved_real_ip_candidates) + .instrument(dial_span.clone()) .await + .inspect_err(|_| egress::mark_error(&dial_span)) .into_diagnostic()?; + drop(dial_span); + drop(connect_span); let upstream_socket_peer = upstream.peer_addr().into_diagnostic()?; let (connected_real_destination, dial_mode) = match upstream.connect_target() { Some(upstream_proxy::ConnectTarget::Ip(ip)) => ( @@ -2446,35 +2466,39 @@ async fn handle_mediated_connection( .and_then(EndpointObservationSender::capture); let mut policy_local_transparent = false; let mut metadata_transparent = false; - let (mut preauthorized_decision, prevalidated_connector) = if let Some(transparent) = - transparent_open - { - let destination = transparent.destination; - if openshell_core::google_cloud::is_metadata_destination(destination) { - metadata_transparent = true; - (None, None) - } else if destination.ip() == IpAddr::V4(crate::policy_dns::POLICY_LOCAL_ADDRESS) - && destination.port() == 80 - { - policy_local_transparent = true; - (None, None) + let (mut preauthorized_decision, prevalidated_connector, preauthorized_span) = + if let Some(transparent) = transparent_open { + let destination = transparent.destination; + if openshell_core::google_cloud::is_metadata_destination(destination) { + metadata_transparent = true; + (None, None, None) + } else if destination.ip() == IpAddr::V4(crate::policy_dns::POLICY_LOCAL_ADDRESS) + && destination.port() == 80 + { + policy_local_transparent = true; + (None, None, None) + } else { + let host = resolve_transparent_target( + destination, + policy_dns_store.as_ref(), + &opa_engine, + )? + .host; + let (decision, connector) = transparent + .authorization + .map_or((None, None), |(decision, connector)| { + (Some(decision), Some(connector)) + }); + let authority = format!("{host}:{}", destination.port()); + client = tokio::io::BufReader::new(virtual_connect_stream( + client.into_inner(), + authority, + )); + (decision, connector, transparent.connect_span) + } } else { - let host = - resolve_transparent_target(destination, policy_dns_store.as_ref(), &opa_engine)? - .host; - let (decision, connector) = transparent - .authorization - .map_or((None, None), |(decision, connector)| { - (Some(decision), Some(connector)) - }); - let authority = format!("{host}:{}", destination.port()); - client = - tokio::io::BufReader::new(virtual_connect_stream(client.into_inner(), authority)); - (decision, connector) - } - } else { - (None, None) - }; + (None, None, None) + }; let metadata_deadline = metadata_transparent .then(|| tokio::time::Instant::now() + std::time::Duration::from_secs(5)); let mut buf = vec![0u8; MAX_HEADER_BYTES]; @@ -2617,12 +2641,15 @@ async fn handle_mediated_connection( // Wrapped in spawn_blocking because identity resolution does heavy sync I/O: // /proc scanning + SHA256 hashing of binaries (e.g. node at 124MB). let intent = EgressIntent::connect(host_lc.clone(), port); + let connect_span = preauthorized_span.unwrap_or_else(|| egress::connect_span(&intent)); let mut decision = if let Some(decision) = preauthorized_decision.take() { decision } else if let Some(identity) = supplied_identity.as_ref() { - authorize_supplied_identity(&opa_engine, &identity_cache, intent, identity) + connect_span.in_scope(|| { + authorize_supplied_identity(&opa_engine, &identity_cache, intent, identity) + }) } else if !opa_engine.binary_identity_required() { - evaluate_endpoint_only_opa(&opa_engine, intent) + connect_span.in_scope(|| evaluate_endpoint_only_opa(&opa_engine, intent)) } else { let (workload_addr, proxy_addr) = socket_addrs.ok_or_else(|| { miette::miette!("legacy proxy connection is missing socket addresses") @@ -2631,8 +2658,11 @@ async fn handle_mediated_connection( let opa_clone = opa_engine.clone(); let cache_clone = identity_cache.clone(); let pid_clone = entrypoint_pid.clone(); + let parent = connect_span.clone(); tokio::task::spawn_blocking(move || { - authorize_egress_intent(connection, &opa_clone, &cache_clone, &pid_clone, intent) + parent.in_scope(|| { + authorize_egress_intent(connection, &opa_clone, &cache_clone, &pid_clone, intent) + }) }) .await .map_err(|e| miette::miette!("identity resolution task panicked: {e}"))? @@ -2797,7 +2827,6 @@ async fn handle_mediated_connection( .expect("destination plan hydrated"); // Defense-in-depth: resolve DNS and reject connections to internal IPs. - let dns_connect_start = std::time::Instant::now(); let connector = if let Some(connector) = prevalidated_connector { connector } else { @@ -2807,6 +2836,7 @@ async fn handle_mediated_connection( sandbox_entrypoint_pid, plan: destination_plan, }) + .instrument(egress::resolve_span(&connect_span)) .await { Ok(connector) => connector, @@ -2938,8 +2968,10 @@ async fn handle_mediated_connection( return Ok(()); } + let dial_span = egress::dial_span(&connect_span); let upstream_result = tokio::select! { - result = dial_upstream(&upstream_proxy, &host_lc, &raw_host_lc, port, connector.addrs()) => Some(result), + result = dial_upstream(&upstream_proxy, &host_lc, &raw_host_lc, port, connector.addrs()) + .instrument(dial_span.clone()) => Some(result), () = connect_generation_guard.wait_until_stale() => None, }; let Some(upstream_result) = upstream_result else { @@ -2961,6 +2993,7 @@ async fn handle_mediated_connection( let mut upstream = match upstream_result { Ok(upstream) => upstream, Err(error) => { + egress::mark_error(&dial_span); if let Some(observer) = connect_endpoint_observer.as_ref() { observer.observe(EndpointResult::TransportFailed); } @@ -2973,10 +3006,8 @@ async fn handle_mediated_connection( return Ok(()); } - debug!( - "handle_tcp_connection dns_resolve_and_tcp_connect: {}ms host={host_lc}", - dns_connect_start.elapsed().as_millis() - ); + drop(dial_span); + drop(connect_span); respond(&mut client, b"HTTP/1.1 200 Connection Established\r\n\r\n").await?; @@ -3491,9 +3522,32 @@ fn resolve_process_identity( Ok(identity) } +/// Authorize a proxied TCP connection inside an egress decision span. +fn authorize_egress_intent( + connection: crate::procfs::WorkloadProxyTcpConnection, + engine: &OpaEngine, + identity_cache: &BinaryIdentityCache, + entrypoint_pid: &AtomicU32, + intent: EgressIntent, +) -> EgressDecision { + egress::traced_authorization( + intent, + |intent| { + authorize_egress_intent_inner( + connection, + engine, + identity_cache, + entrypoint_pid, + intent, + ) + }, + |decision| decision, + ) +} + /// Evaluate OPA policy for a TCP connection with identity binding via /proc/net/tcp. #[cfg(target_os = "linux")] -fn authorize_egress_intent( +fn authorize_egress_intent_inner( connection: crate::procfs::WorkloadProxyTcpConnection, engine: &OpaEngine, identity_cache: &BinaryIdentityCache, @@ -3535,7 +3589,6 @@ fn authorize_egress_intent( ); }; - let total_start = std::time::Instant::now(); let identity = match resolve_process_identity(proc_net_anchor_pid, connection, identity_cache) { Ok(id) => id, Err(err) => { @@ -3567,7 +3620,7 @@ fn authorize_egress_intent( cmdline_paths: cmdline_paths.clone(), }; - let result = match engine.authorize_egress(&input) { + match engine.authorize_egress(&input) { Ok(authorization) => EgressDecision { intent: intent.clone(), action: authorization.action.clone(), @@ -3587,15 +3640,7 @@ fn authorize_egress_intent( ancestors, cmdline_paths, ), - }; - debug!( - "authorize_egress_intent TOTAL: {}ms host={} port={} transport={:?}", - total_start.elapsed().as_millis(), - intent.destination.host, - intent.destination.port, - intent.transport, - ); - result + } } #[cfg(target_os = "linux")] @@ -3604,6 +3649,14 @@ fn proc_net_anchor_pid(entrypoint_pid: u32) -> Option { } fn evaluate_endpoint_only_opa(engine: &OpaEngine, intent: EgressIntent) -> EgressDecision { + egress::traced_authorization( + intent, + |intent| evaluate_endpoint_only_opa_inner(engine, intent), + |decision| decision, + ) +} + +fn evaluate_endpoint_only_opa_inner(engine: &OpaEngine, intent: EgressIntent) -> EgressDecision { let input = crate::opa::NetworkInput { host: intent.destination.host.clone(), port: intent.destination.port, @@ -3673,6 +3726,21 @@ fn authorize_supplied_identity_with_denial( identity_cache: &BinaryIdentityCache, intent: EgressIntent, identity: &Result, +) -> SuppliedIdentityAuthorization { + egress::traced_authorization( + intent, + |intent| { + authorize_supplied_identity_with_denial_inner(engine, identity_cache, intent, identity) + }, + |authorization| &authorization.decision, + ) +} + +fn authorize_supplied_identity_with_denial_inner( + engine: &OpaEngine, + identity_cache: &BinaryIdentityCache, + intent: EgressIntent, + identity: &Result, ) -> SuppliedIdentityAuthorization { let deny = |reason: String, binary: Option, @@ -3762,7 +3830,7 @@ fn authorize_supplied_identity_with_denial( /// Non-Linux stub: OPA identity binding requires /proc. #[cfg(not(target_os = "linux"))] -fn authorize_egress_intent( +fn authorize_egress_intent_inner( _connection: crate::procfs::WorkloadProxyTcpConnection, engine: &OpaEngine, _identity_cache: &BinaryIdentityCache, @@ -5371,10 +5439,13 @@ async fn handle_forward_proxy( // 2. Evaluate OPA policy (same identity binding as CONNECT) let intent = EgressIntent::forward_http(host_lc.clone(), port); + let connect_span = egress::connect_span(&intent); let mut decision = if let Some(identity) = supplied_identity { - authorize_supplied_identity(&opa_engine, &identity_cache, intent, identity) + connect_span.in_scope(|| { + authorize_supplied_identity(&opa_engine, &identity_cache, intent, identity) + }) } else if !opa_engine.binary_identity_required() { - evaluate_endpoint_only_opa(&opa_engine, intent) + connect_span.in_scope(|| evaluate_endpoint_only_opa(&opa_engine, intent)) } else { let (workload_addr, proxy_addr) = socket_addrs.ok_or_else(|| { miette::miette!("legacy proxy connection is missing socket addresses") @@ -5383,8 +5454,11 @@ async fn handle_forward_proxy( let opa_clone = opa_engine.clone(); let cache_clone = identity_cache.clone(); let pid_clone = entrypoint_pid.clone(); + let parent = connect_span.clone(); tokio::task::spawn_blocking(move || { - authorize_egress_intent(connection, &opa_clone, &cache_clone, &pid_clone, intent) + parent.in_scope(|| { + authorize_egress_intent(connection, &opa_clone, &cache_clone, &pid_clone, intent) + }) }) .await .map_err(|e| miette::miette!("identity resolution task panicked: {e}"))? @@ -6103,6 +6177,7 @@ async fn handle_forward_proxy( sandbox_entrypoint_pid, plan: destination_plan, }) + .instrument(egress::resolve_span(&connect_span)) .await { Ok(connector) => connector, @@ -6474,10 +6549,12 @@ async fn handle_forward_proxy( // would need absolute-form requests rather than a CONNECT tunnel. Dial // only after every local authorization and transformation step so a // rejected WebSocket preflight cannot contact the destination. - let dial_result = connector.connect().await; + let dial_span = egress::dial_span(&connect_span); + let dial_result = connector.connect().instrument(dial_span.clone()).await; let mut upstream = match dial_result { Ok(s) => s, Err(e) => { + egress::mark_error(&dial_span); let event = HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) .activity(ActivityId::Fail) .severity(SeverityId::Low) @@ -6515,6 +6592,8 @@ async fn handle_forward_proxy( return Ok(()); } }; + drop(dial_span); + drop(connect_span); if let Err(e) = forward_generation_guard.ensure_current() { warn!( @@ -7497,6 +7576,7 @@ process: { run_as_user: sandbox, run_as_group: sandbox } .parse() .unwrap(), authorization: None, + connect_span: None, }), None, engine, @@ -13170,8 +13250,7 @@ network_policies: .await .expect_err("forward token grant failure should stop request rewriting"); - assert!(err.to_string().contains("Token grant failed")); - assert!(err.to_string().contains("oauth unavailable")); + assert_eq!(err.to_string(), "Token grant failed"); fixture.assert_one_request("api.example.test\t8080\t/v1/**\tprovider:access_token"); } @@ -13184,8 +13263,7 @@ network_policies: .await .expect_err("forward token exchange failure should stop request rewriting"); - assert!(err.to_string().contains("Token grant failed")); - assert!(err.to_string().contains("oauth unavailable")); + assert_eq!(err.to_string(), "Token grant failed"); fixture.assert_one_token_exchange_request( "api.example.test\t8080\t/v1/**\tprovider:access_token", ); diff --git a/crates/openshell-supervisor-network/src/proxy/egress.rs b/crates/openshell-supervisor-network/src/proxy/egress.rs index 55c4dd9099..77b89a9b7e 100644 --- a/crates/openshell-supervisor-network/src/proxy/egress.rs +++ b/crates/openshell-supervisor-network/src/proxy/egress.rs @@ -80,6 +80,16 @@ pub(super) enum EgressTransport { TransparentTcp, } +impl EgressTransport { + fn as_str(self) -> &'static str { + match self { + Self::Connect => "connect", + Self::ForwardHttp => "forward_http", + Self::TransparentTcp => "transparent_tcp", + } + } +} + /// Destination requested by an explicit proxy adapter. #[derive(Debug, Clone, PartialEq, Eq)] pub(super) struct RequestedDestination { @@ -160,6 +170,58 @@ pub(super) struct EgressDecision { pub(super) cmdline_paths: Vec, } +pub(super) fn connect_span(intent: &EgressIntent) -> tracing::Span { + tracing::debug_span!( + "supervisor.egress.connect", + server.address = intent.destination.host.as_str(), + server.port = intent.destination.port, + openshell.egress.transport = intent.transport.as_str(), + ) +} + +pub(super) fn resolve_span(parent: &tracing::Span) -> tracing::Span { + tracing::debug_span!(parent: parent, "supervisor.egress.resolve") +} + +pub(super) fn dial_span(parent: &tracing::Span) -> tracing::Span { + tracing::debug_span!( + parent: parent, + "supervisor.egress.dial", + otel.status_code = tracing::field::Empty, + ) +} + +pub(super) fn mark_error(span: &tracing::Span) { + span.record("otel.status_code", "ERROR"); +} + +/// Run one egress authorization inside a `supervisor.egress.authorize` span +/// that records the policy outcome. +pub(super) fn traced_authorization( + intent: EgressIntent, + authorize: impl FnOnce(EgressIntent) -> T, + decision: impl FnOnce(&T) -> &EgressDecision, +) -> T { + let span = tracing::debug_span!( + "supervisor.egress.authorize", + openshell.policy.decision = tracing::field::Empty, + openshell.policy.name = tracing::field::Empty, + ); + let result = span.in_scope(|| authorize(intent)); + match &decision(&result).action { + NetworkAction::Allow { matched_policy } => { + span.record("openshell.policy.decision", "allow"); + if let Some(name) = matched_policy { + span.record("openshell.policy.name", name.as_str()); + } + } + NetworkAction::Deny { .. } => { + span.record("openshell.policy.decision", "deny"); + } + } + result +} + #[cfg(test)] mod tests { use super::*; @@ -180,4 +242,122 @@ mod tests { assert_eq!(transparent.destination.host, "db.example.com"); assert_eq!(transparent.destination.port, 5432); } + + fn decision(intent: EgressIntent, action: NetworkAction) -> EgressDecision { + EgressDecision { + intent, + action, + policy_generation: 1, + identity: ProcessIdentityEvidence::Available, + endpoint: EndpointDecision::default(), + binary: None, + binary_pid: None, + ancestors: vec![], + cmdline_paths: vec![], + } + } + + #[test] + fn egress_spans_are_debug_level() { + let subscriber = tracing_subscriber::registry(); + tracing::subscriber::with_default(subscriber, || { + let intent = EgressIntent::connect("api.example.com".to_string(), 443); + let connect = connect_span(&intent); + let level = |span: &tracing::Span| *span.metadata().expect("span enabled").level(); + assert_eq!(level(&connect), tracing::Level::DEBUG); + assert_eq!(level(&resolve_span(&connect)), tracing::Level::DEBUG); + assert_eq!(level(&dial_span(&connect)), tracing::Level::DEBUG); + let authorize = traced_authorization( + intent, + |intent| { + let level = level(&tracing::Span::current()); + let deny = NetworkAction::Deny { + reason: String::new(), + }; + (decision(intent, deny), level) + }, + |(d, _)| d, + ); + assert_eq!(authorize.1, tracing::Level::DEBUG); + }); + } + + #[test] + fn authorization_is_a_child_of_the_connect_span() { + use tracing_subscriber::layer::SubscriberExt as _; + + let exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); + let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = + tracing_subscriber::registry().with(openshell_otel::layer(&provider, "egress-test")); + + let authorize = |intent: EgressIntent, action: NetworkAction| { + let connect = connect_span(&intent); + connect.in_scope(|| { + traced_authorization(intent, |intent| decision(intent, action), |d| d); + }); + resolve_span(&connect).in_scope(|| {}); + dial_span(&connect).in_scope(|| {}); + }; + tracing::subscriber::with_default(subscriber, || { + authorize( + EgressIntent::connect("api.example.com".to_string(), 443), + NetworkAction::Allow { + matched_policy: Some("github".to_string()), + }, + ); + authorize( + EgressIntent::forward_http("blocked.example.com".to_string(), 80), + NetworkAction::Deny { + reason: "not allowed".to_string(), + }, + ); + }); + + let spans = exporter.get_finished_spans().unwrap(); + let attributes = |span: &opentelemetry_sdk::trace::SpanData| { + span.attributes + .iter() + .map(|kv| (kv.key.as_str().to_string(), kv.value.to_string())) + .collect::>() + }; + let child = |parent: &opentelemetry_sdk::trace::SpanData, name: &str| { + spans + .iter() + .find(|span| { + span.name == name && span.parent_span_id == parent.span_context.span_id() + }) + .unwrap_or_else(|| panic!("{name} is a child of the connect span")) + .clone() + }; + let connect = |host: &str| { + spans + .iter() + .find(|span| { + span.name == "supervisor.egress.connect" + && attributes(span)["server.address"] == host + }) + .unwrap() + .clone() + }; + + let allowed = connect("api.example.com"); + assert_eq!(attributes(&allowed)["server.port"], "443"); + assert_eq!( + attributes(&allowed)["openshell.egress.transport"], + "connect" + ); + let authorized = attributes(&child(&allowed, "supervisor.egress.authorize")); + assert_eq!(authorized["openshell.policy.decision"], "allow"); + assert_eq!(authorized["openshell.policy.name"], "github"); + child(&allowed, "supervisor.egress.resolve"); + child(&allowed, "supervisor.egress.dial"); + + let denied = connect("blocked.example.com"); + let authorized = attributes(&child(&denied, "supervisor.egress.authorize")); + assert_eq!(authorized["openshell.policy.decision"], "deny"); + assert!(!authorized.contains_key("openshell.policy.name")); + } } diff --git a/crates/openshell-supervisor-network/src/token_grant.rs b/crates/openshell-supervisor-network/src/token_grant.rs index 5d03012f61..f522b0a3d3 100644 --- a/crates/openshell-supervisor-network/src/token_grant.rs +++ b/crates/openshell-supervisor-network/src/token_grant.rs @@ -900,6 +900,69 @@ mod tests { ); } + #[tokio::test] + async fn multiple_credentials_cache_and_expire_independently() { + let cache = TokenCache::new(); + let calls = Arc::new(AtomicUsize::new(0)); + let scopes = vec!["read".to_string()]; + let service = "api.example.test\t443\t/v1/**\trev:1\tprovider:service"; + let identity = "api.example.test\t443\t/v1/**\trev:1\tprovider:identity"; + for (key, expected) in [(service, "token-1"), (identity, "token-2")] { + let token = obtain_counted_test_token(CountedTokenGrantInput { + cache: &cache, + provider_name: key, + token_endpoint: "https://auth.example.com/token", + jwt_svid_audience: "https://auth.example.com", + audience: "api://resource", + scopes: &scopes, + cache_ttl_override: None, + expires_in: 120, + grant_calls: calls.clone(), + }) + .await + .unwrap(); + assert_eq!(token, expected); + } + let identity_cache_key = token_cache_key(TokenCacheKeyInput { + provider_name: identity, + token_endpoint: "https://auth.example.com/token", + jwt_svid_audience: "https://auth.example.com", + client_assertion_type: DEFAULT_CLIENT_ASSERTION_TYPE, + audience: "api://resource", + scopes: &scopes, + grant_type: ProviderCredentialTokenGrantType::ClientCredentials, + requested_token_type: ACCESS_TOKEN_TYPE, + }); + cache.set(identity_cache_key, "expired".into(), current_time_ms() - 1); + let service_token = obtain_token_without_grant_call( + &cache, + service, + "https://auth.example.com/token", + "https://auth.example.com", + "api://resource", + &scopes, + None, + ) + .await + .unwrap(); + assert_eq!(service_token, "token-1"); + let identity_token = obtain_counted_test_token(CountedTokenGrantInput { + cache: &cache, + provider_name: identity, + token_endpoint: "https://auth.example.com/token", + jwt_svid_audience: "https://auth.example.com", + audience: "api://resource", + scopes: &scopes, + cache_ttl_override: None, + expires_in: 120, + grant_calls: calls.clone(), + }) + .await + .unwrap(); + assert_eq!(identity_token, "token-3"); + assert_eq!(calls.load(Ordering::SeqCst), 3); + } + #[tokio::test] async fn obtain_provider_token_uses_cache_for_same_key() { let cache = TokenCache::new(); diff --git a/crates/openshell-supervisor/Cargo.toml b/crates/openshell-supervisor/Cargo.toml index 483dd50bff..04fe45b8d1 100644 --- a/crates/openshell-supervisor/Cargo.toml +++ b/crates/openshell-supervisor/Cargo.toml @@ -15,11 +15,12 @@ name = "openshell-supervisor" path = "src/main.rs" [dependencies] -openshell-core = { path = "../openshell-core", default-features = false, features = ["jwt"] } +openshell-core = { path = "../openshell-core", default-features = false, features = ["jwt", "trace-context"] } openshell-extension-core = { path = "../openshell-extension-core" } openshell-isolation-interface = { path = "../openshell-isolation-interface" } openshell-sandbox-backend = { path = "../openshell-sandbox-backend" } openshell-ocsf = { path = "../openshell-ocsf" } +openshell-otel = { path = "../openshell-otel" } openshell-policy = { path = "../openshell-policy" } openshell-supervisor-middleware = { path = "../openshell-supervisor-middleware" } openshell-supervisor-middleware-builtins = { path = "../openshell-supervisor-middleware-builtins" } @@ -29,6 +30,7 @@ openshell-supervisor-process = { path = "../openshell-supervisor-process" } clap = { workspace = true } miette = { workspace = true } nix = { workspace = true } +opentelemetry = { workspace = true } prost = { workspace = true } prost-types = { workspace = true } rustls = { workspace = true } diff --git a/crates/openshell-supervisor/README.md b/crates/openshell-supervisor/README.md new file mode 100644 index 0000000000..6ecbfb6059 --- /dev/null +++ b/crates/openshell-supervisor/README.md @@ -0,0 +1,17 @@ +# OpenShell supervisor + +The supervisor loads and reconciles policy, maintains provider credentials, applies network and MCP inspection, and drives the admitted isolation backend through attachment, confirmation, and workload start. + +## Backend startup + +The public `run_sandbox` entry point selects the OpenShell Sandbox Protocol backend and collects its startup inputs into a private `SandboxRunConfig`. Shared startup receives that config and the trusted backend setup separately. The `backend_setup` module owns that backend's launch-data decoder, workload policy discovery, and client construction. Descriptor contents cannot select an implementation. + +Shared startup checks the admitted backend name before passing the opaque payload to its decoder. It then compares the decoded sandbox, session, and runtime generation with the trusted launch inputs before installing credentials or discovering workload policy. A mismatch stops startup. + +The built-in decoder also carries the VM driver's fixed workload identity into shared policy validation. Startup and later policy updates must reject selectors that conflict with that identity. Other launch descriptors do not enable this VM-specific check. + +The supervisor admits policy and prepares credentials before constructing and attaching the selected client. It uses the isolation contract's `BoundBoundary` and `ConfirmedBoundary` directly: confirm the attached boundary, prepare network mediation, then start the workload. Backend implementations remain responsible for validating their native enforcement evidence through the isolation contract. + +The client receives the supervisor's live provider state, bearer-token slot, and CA-path slot. Provider refresh, token rotation, and later CA publication must remain visible through those shared handles. Startup does not create independent copies of their current values. + +The setup interface stays private to the supervisor. It adds no runtime backend registration, endpoint configuration, or public factory API. The public `run_sandbox` signature and standard backend selection remain unchanged. diff --git a/crates/openshell-supervisor/src/backend_setup.rs b/crates/openshell-supervisor/src/backend_setup.rs new file mode 100644 index 0000000000..61fe5a71f0 --- /dev/null +++ b/crates/openshell-supervisor/src/backend_setup.rs @@ -0,0 +1,250 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Backend-owned launch decoding and client construction for supervisor startup. + +use std::path::PathBuf; +use std::sync::{Arc, Mutex}; + +use miette::Result; +use openshell_core::SandboxSessionId; +use openshell_core::jwt::{SessionBearerTokenSlot, SupervisorAuthBundle}; +use openshell_core::provider_credentials::ProviderCredentialState; +use openshell_isolation_interface::AgentSpec; +use openshell_isolation_interface::contract::{ + BackendDescriptor, BackendError, BackendRegistry, BoundBoundary, IsolationBackend, + ResolvedWorkloadIdentity, SandboxContext, SandboxPolicy, +}; + +/// Coordinates decoded by the selected trusted backend. Shared startup checks +/// these against admission before credentials or discovery reach that backend. +pub struct LaunchIdentity { + pub sandbox_id: String, + pub generation: String, + pub session_id: SandboxSessionId, + pub workload_identity: ResolvedWorkloadIdentity, + /// Preserve driver-fixed VM selectors during shared startup and reload. + pub vm_policy_identity: Option, +} + +/// Runtime state remains owned by the supervisor. Backends retain these same +/// handles so CA publication, provider reload, and bearer rotation stay visible +/// after attachment; copying their current contents would lose later updates. +pub struct BackendServices { + pub ca_file_paths: Arc>>, + pub provider_credentials: ProviderCredentialState, + pub sandbox_bearer: SessionBearerTokenSlot, +} + +/// Selected by trusted composition, never by payload contents. Decoding and +/// construction may prepare a client but must not launch workload code. +pub trait BackendSetup: Sync { + /// Name chosen by trusted composition and checked against admission before decoding. + fn backend_name(&self) -> &str; + + /// Decode native launch data without starting workload code. The returned + /// identity must describe the same resource retained by the prepared backend. + fn decode( + &self, + payload: &[u8], + ) -> std::result::Result<(LaunchIdentity, Box), BackendError>; +} + +/// Native launch data stays with its backend until client construction. Shared +/// startup admits policy and credentials between discovery and attachment. +#[tonic::async_trait] +pub trait PreparedBackend: Send + Sync { + /// Read image policy through the backend's authenticated client. An invalid + /// or unreadable policy must return the invalid flag or an error, never missing. + async fn discover_policy( + &self, + bearer: SessionBearerTokenSlot, + ) -> std::result::Result<(Option, bool), BackendError>; + + /// Consume prepared launch data and retain the supervisor's live handles. + /// Construction does not attach the resource or start workload code. + fn build( + self: Box, + services: BackendServices, + ) -> std::result::Result, BackendError>; +} + +/// Created only after name and launch identity checks. Consuming attachment +/// prevents reusing one prepared startup to build or attach a second client. +pub struct SelectedBackend { + descriptor: BackendDescriptor, + identity: LaunchIdentity, + prepared: Box, +} + +impl SelectedBackend { + /// Reject absent or mismatched admission before decoding, then bind the + /// decoded sandbox, session, and generation to trusted launch inputs. + pub fn select( + setup: &dyn BackendSetup, + descriptor: BackendDescriptor, + admitted_backend: Option<&str>, + sandbox_id: Option<&str>, + auth: &SupervisorAuthBundle, + ) -> Result { + let admitted_backend = admitted_backend.ok_or_else(|| { + miette::miette!("runtime descriptor supplied without an admitted isolation backend") + })?; + if descriptor.backend_name != admitted_backend { + return Err(miette::miette!( + "descriptor backend {:?} does not match admitted backend {admitted_backend:?}", + descriptor.backend_name + )); + } + if setup.backend_name() != admitted_backend { + return Err(miette::miette!( + "selected backend {:?} does not match admitted backend {admitted_backend:?}", + setup.backend_name() + )); + } + + let (identity, prepared) = setup + .decode(&descriptor.payload) + .map_err(|error| miette::miette!(error.to_string()))?; + if sandbox_id.is_none_or(|id| id.is_empty() || identity.sandbox_id != id) { + return Err(miette::miette!( + "runtime descriptor does not match admitted sandbox" + )); + } + if identity.session_id != auth.session_id { + return Err(miette::miette!( + "supervisor authentication bundle does not match runtime session" + )); + } + if identity.generation != auth.runtime_generation.as_str() { + return Err(miette::miette!( + "supervisor authentication bundle does not match runtime generation" + )); + } + Ok(Self { + descriptor, + identity, + prepared, + }) + } + + /// Return the backend name already matched to the admitted selection. + pub fn backend_name(&self) -> &str { + &self.descriptor.backend_name + } + + /// Carry the selected backend's VM identity constraint into policy handling. + pub fn vm_policy_identity(&self) -> Option { + self.identity.vm_policy_identity + } + + /// Delegate image discovery only after shared name and identity checks. + /// Callers may retry this read; discovery cannot authorize workload launch. + pub async fn discover_policy( + &self, + bearer: SessionBearerTokenSlot, + ) -> Result<(Option, bool)> { + self.prepared + .discover_policy(bearer) + .await + .map_err(|error| miette::miette!("discover workload image policy: {error}")) + } + + /// Construct and attach the selected client using the admitted policy and + /// shared services. Registry verification rejects a differently named client. + pub async fn attach( + self, + services: BackendServices, + policy: SandboxPolicy, + agent: AgentSpec, + ) -> Result> { + let backend = self + .prepared + .build(services) + .map_err(|error| miette::miette!(error.to_string()))?; + let mut registry = BackendRegistry::new(); + registry + .register(backend) + .map_err(|error| miette::miette!(error.to_string()))?; + let admitted_backend = self.descriptor.backend_name.clone(); + let (backend, verified) = registry + .resolve(self.descriptor, &admitted_backend) + .map_err(|error| miette::miette!(error.to_string()))?; + backend + .attach( + verified, + SandboxContext { + sandbox_id: self.identity.sandbox_id, + session_id: self.identity.session_id, + policy, + agent, + identity: self.identity.workload_identity, + }, + ) + .await + .map_err(|error| miette::miette!(error.to_string())) + } +} + +/// The standard binary selects the `OpenShell` Sandbox Protocol. Its wire schema +/// and concrete client stay here rather than in the shared startup sequence. +pub struct OpenShellBackendSetup; + +impl BackendSetup for OpenShellBackendSetup { + fn backend_name(&self) -> &str { + openshell_sandbox_backend::BACKEND_NAME + } + + fn decode( + &self, + payload: &[u8], + ) -> std::result::Result<(LaunchIdentity, Box), BackendError> { + let descriptor: openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor = + serde_json::from_slice(payload).map_err(|error| { + BackendError::Descriptor(format!("decode sandbox runtime descriptor: {error}")) + })?; + let identity = LaunchIdentity { + sandbox_id: descriptor.boundary_id.clone(), + generation: descriptor.generation.clone(), + session_id: descriptor.session_id, + workload_identity: descriptor.workload_identity.clone(), + vm_policy_identity: descriptor + .resource_claims + .contains_key("vm.generation") + .then_some(super::VmPolicyIdentity { + uid: descriptor.workload_identity.uid, + gid: descriptor.workload_identity.gid, + }), + }; + Ok((identity, Box::new(OpenShellLaunch(descriptor)))) + } +} + +struct OpenShellLaunch(openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor); + +#[tonic::async_trait] +impl PreparedBackend for OpenShellLaunch { + async fn discover_policy( + &self, + bearer: SessionBearerTokenSlot, + ) -> std::result::Result<(Option, bool), BackendError> { + openshell_sandbox_backend::OpenShellRuntimeBackend::discover_policy(self.0.clone(), bearer) + .await + } + + fn build( + self: Box, + services: BackendServices, + ) -> std::result::Result, BackendError> { + Ok(Arc::new( + openshell_sandbox_backend::OpenShellRuntimeBackend::new( + services.ca_file_paths, + services.provider_credentials, + services.sandbox_bearer, + ), + )) + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/openshell-supervisor/src/backend_setup/tests.rs b/crates/openshell-supervisor/src/backend_setup/tests.rs new file mode 100644 index 0000000000..fa4ab65ba5 --- /dev/null +++ b/crates/openshell-supervisor/src/backend_setup/tests.rs @@ -0,0 +1,797 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use super::*; +use std::collections::{BTreeMap, HashMap}; +use std::sync::Weak; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + +use openshell_core::jwt::{CredentialEpoch, SecretJwt, SessionRotation}; +use openshell_core::policy::{FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy}; +use openshell_core::sandbox_generation::SandboxGenerationId; +use openshell_isolation_interface::contract::{ + BoundaryConfirmation, BoundaryDuplexStream, BoundaryExec, BoundaryExitStatus, + BoundaryLoopbackConnector, BoundaryProcess, BoundaryProperties, BoundarySignal, + ConfirmedBoundary, EnforcedProperty, ExecSession, ExecSpec, LoopbackTarget, + NetworkMediationSource, OuterFenceGuarantee, OuterFenceGuarantees, PendingDnsQuery, + PendingTcpOpen, ReadyBoundary, RunningBoundary, VerifiedBackendDescriptor, +}; +use openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs; + +const TEST_BACKEND: &str = "in-process-test"; +// Deliberately not JSON or a SandboxRuntimeDescriptor. Only TestSetup accepts it. +const TEST_PAYLOAD: &[u8] = b"in-process-v1\0owned-launch"; + +#[derive(Default)] +struct Observed { + events: Mutex>, + services: Mutex>, + discovery_unavailable: AtomicBool, + deny_confirmation: AtomicBool, + require_networking: AtomicBool, + active: AtomicBool, + starts: AtomicUsize, + releases: AtomicUsize, +} + +impl Observed { + fn record(&self, event: &'static str) { + self.events.lock().unwrap().push(event); + } + + fn events(&self) -> Vec<&'static str> { + self.events.lock().unwrap().clone() + } +} + +struct TestSetup { + auth: SupervisorAuthBundle, + sandbox_id: String, + selected_name: &'static str, + built_name: &'static str, + observed: Arc, +} + +impl TestSetup { + fn new() -> Self { + Self { + auth: SupervisorAuthBundle { + session_id: SandboxSessionId::new(), + runtime_generation: SandboxGenerationId::parse("generation-1").unwrap(), + session_rotation: SessionRotation::new(1).unwrap(), + auth_epoch: CredentialEpoch::new(1).unwrap(), + gateway_token: SecretJwt::parse("test-gateway-token").unwrap(), + gateway_expires_at: 0, + sandbox_token: SecretJwt::parse("test-sandbox-token").unwrap(), + sandbox_expires_at: 0, + }, + sandbox_id: "sandbox-1".to_string(), + selected_name: TEST_BACKEND, + built_name: TEST_BACKEND, + observed: Arc::new(Observed::default()), + } + } + + fn descriptor() -> BackendDescriptor { + BackendDescriptor { + backend_name: TEST_BACKEND.to_string(), + payload: TEST_PAYLOAD.to_vec(), + } + } + + fn select(&self) -> SelectedBackend { + SelectedBackend::select( + self, + Self::descriptor(), + Some(TEST_BACKEND), + Some("sandbox-1"), + &self.auth, + ) + .unwrap() + } + + fn services(&self) -> BackendServices { + BackendServices { + ca_file_paths: Arc::new(Mutex::new(None)), + provider_credentials: ProviderCredentialState::from_child_env_snapshot( + 1, + HashMap::from([("TEST_CONFIG".into(), "initial".into())]), + ), + sandbox_bearer: self.auth.sandbox_bearer_slot().unwrap(), + } + } +} + +fn identity() -> ResolvedWorkloadIdentity { + ResolvedWorkloadIdentity::new(1000, 1000, vec![], "test".into(), "test-resource".into()) + .unwrap() +} + +fn policy() -> SandboxPolicy { + SandboxPolicy { + version: 1, + filesystem: FilesystemPolicy::default(), + network: NetworkPolicy::default(), + landlock: LandlockPolicy::default(), + process: ProcessPolicy::default(), + } +} + +fn agent() -> AgentSpec { + AgentSpec { + program: "test-agent".into(), + args: vec!["test-argument".into()], + workdir: Some("/test-workspace".into()), + timeout_secs: 15, + interactive: false, + } +} + +impl BackendSetup for TestSetup { + fn backend_name(&self) -> &str { + self.selected_name + } + + fn decode( + &self, + payload: &[u8], + ) -> std::result::Result<(LaunchIdentity, Box), BackendError> { + self.observed.record("decode"); + if payload != TEST_PAYLOAD { + return Err(BackendError::Descriptor( + "invalid in-process launch data".into(), + )); + } + Ok(( + LaunchIdentity { + sandbox_id: self.sandbox_id.clone(), + generation: self.auth.runtime_generation.to_string(), + session_id: self.auth.session_id, + workload_identity: identity(), + vm_policy_identity: None, + }, + Box::new(TestLaunch { + name: self.built_name, + observed: self.observed.clone(), + generation: self.auth.runtime_generation.to_string(), + expected_session: self.auth.session_id, + }), + )) + } +} + +struct TestLaunch { + name: &'static str, + observed: Arc, + generation: String, + expected_session: SandboxSessionId, +} + +#[tonic::async_trait] +impl PreparedBackend for TestLaunch { + async fn discover_policy( + &self, + bearer: SessionBearerTokenSlot, + ) -> std::result::Result<(Option, bool), BackendError> { + self.observed.record("discover"); + bearer.authorization_metadata().unwrap(); + if self.observed.discovery_unavailable.load(Ordering::SeqCst) { + return Err(BackendError::Unavailable("discovery unavailable".into())); + } + Ok((None, false)) + } + + fn build( + self: Box, + services: BackendServices, + ) -> std::result::Result, BackendError> { + self.observed.record("build"); + let services = Arc::new(services); + *self.observed.services.lock().unwrap() = Arc::downgrade(&services); + Ok(Arc::new(TestBackend { + name: self.name, + observed: self.observed, + services, + generation: self.generation, + expected_session: self.expected_session, + })) + } +} + +struct TestBackend { + name: &'static str, + observed: Arc, + services: Arc, + generation: String, + expected_session: SandboxSessionId, +} + +#[tonic::async_trait] +impl IsolationBackend for TestBackend { + fn backend_name(&self) -> &str { + self.name + } + + async fn attach( + &self, + descriptor: VerifiedBackendDescriptor, + sandbox: SandboxContext, + ) -> std::result::Result, BackendError> { + self.observed.record("attach"); + assert_eq!(descriptor.payload(), TEST_PAYLOAD); + assert_eq!(descriptor.backend_name(), TEST_BACKEND); + assert_eq!(sandbox.sandbox_id, "sandbox-1"); + assert_eq!(sandbox.session_id, self.expected_session); + assert_eq!(sandbox.identity, identity()); + assert_eq!(sandbox.agent.program, "test-agent"); + assert_eq!(sandbox.agent.args, ["test-argument"]); + assert_eq!(sandbox.agent.workdir.as_deref(), Some("/test-workspace")); + assert_eq!(sandbox.agent.timeout_secs, 15); + assert_eq!(sandbox.policy.version, 1); + if self.observed.active.swap(true, Ordering::SeqCst) { + return Err(BackendError::Denied("resource already owned".into())); + } + Ok(Box::new(TestBound { + lease: Lease(self.observed.clone()), + services: self.services.clone(), + generation: self.generation.clone(), + session_id: sandbox.session_id, + })) + } +} + +// Models backend ownership only. Dropping a rejected or completed attempt +// releases this in-process resource; these tests make no Linux cleanup claim. +struct Lease(Arc); + +impl Drop for Lease { + fn drop(&mut self) { + self.0.active.store(false, Ordering::SeqCst); + self.0.releases.fetch_add(1, Ordering::SeqCst); + } +} + +struct TestBound { + lease: Lease, + services: Arc, + generation: String, + session_id: SandboxSessionId, +} + +#[tonic::async_trait] +impl BoundBoundary for TestBound { + fn network_mediation_source(&self) -> Arc { + Arc::new(TestIo) + } + + async fn confirm(self: Box) -> std::result::Result { + self.lease.0.record("confirm"); + let property = EnforcedProperty::new(true, "in-process-test"); + let confirmation = BoundaryConfirmation { + generation: self.generation.clone(), + identity: identity(), + properties: BoundaryProperties { + filesystem_confinement: property.clone(), + egress_interception: property.clone(), + request_attribution: property.clone(), + privilege_floor: property, + }, + authenticated_supervisor: !self.lease.0.deny_confirmation.load(Ordering::SeqCst), + session_id: self.session_id, + outer_fence: OuterFenceGuarantees::from_enforcement_evidence( + &self.generation, + [ + OuterFenceGuarantee::DefaultDenyEgress, + OuterFenceGuarantee::NoUnmanagedEgressPath, + OuterFenceGuarantee::RevocationVerified, + OuterFenceGuarantee::ControllerLossFailsClosed, + ], + b"in-process-enforcement-evidence", + )?, + runtime_exit_terminates_workload: true, + resource_claims: BTreeMap::new(), + backend_audit: serde_json::json!({"in_process": true}), + }; + ConfirmedBoundary::try_new(self, confirmation, &identity()) + } +} + +#[tonic::async_trait] +impl ReadyBoundary for TestBound { + async fn start_agent( + self: Box, + ) -> std::result::Result, BackendError> { + if self.lease.0.require_networking.load(Ordering::SeqCst) { + let paths = self.services.ca_file_paths.lock().unwrap(); + let (certificate, bundle) = paths.as_ref().expect("networking published CA paths"); + assert!(certificate.is_file()); + assert!(bundle.is_file()); + } + self.lease.0.record("start"); + self.lease.0.starts.fetch_add(1, Ordering::SeqCst); + Ok(Box::new(TestRunning { + _lease: self.lease, + _services: self.services, + })) + } +} + +struct TestRunning { + _lease: Lease, + _services: Arc, +} + +#[tonic::async_trait] +impl RunningBoundary for TestRunning { + fn agent(&self) -> Arc { + Arc::new(TestIo) + } + fn exec(&self) -> Arc { + Arc::new(TestIo) + } + fn loopback_connector(&self) -> Arc { + Arc::new(TestIo) + } + async fn terminate(&self) -> std::result::Result<(), BackendError> { + Ok(()) + } +} + +struct TestIo; + +#[tonic::async_trait] +impl NetworkMediationSource for TestIo { + async fn accept_tcp(&self) -> std::result::Result { + std::future::pending().await + } + async fn accept_dns(&self) -> std::result::Result { + std::future::pending().await + } +} + +#[tonic::async_trait] +impl BoundaryProcess for TestIo { + async fn wait(&self) -> std::result::Result { + Ok(BoundaryExitStatus::Exited(0)) + } + async fn signal(&self, _: BoundarySignal) -> std::result::Result<(), BackendError> { + Ok(()) + } + async fn terminate(&self) -> std::result::Result<(), BackendError> { + Ok(()) + } +} + +#[tonic::async_trait] +impl BoundaryExec for TestIo { + async fn exec(&self, _: ExecSpec) -> std::result::Result { + Err(BackendError::Unsupported("test backend has no exec".into())) + } +} + +#[tonic::async_trait] +impl BoundaryLoopbackConnector for TestIo { + async fn connect( + &self, + _: LoopbackTarget, + ) -> std::result::Result { + Err(BackendError::Unsupported( + "test backend has no connector".into(), + )) + } +} + +#[test] +fn admission_rejection_never_decodes_or_discovers() { + let mut setup = TestSetup::new(); + for (descriptor_name, admitted) in [ + (TEST_BACKEND, None), + ("foreign", Some(TEST_BACKEND)), + (TEST_BACKEND, Some("foreign")), + ] { + let mut descriptor = TestSetup::descriptor(); + descriptor.backend_name = descriptor_name.into(); + descriptor.payload = b"malformed".to_vec(); + assert!( + SelectedBackend::select(&setup, descriptor, admitted, Some("sandbox-1"), &setup.auth) + .is_err() + ); + } + // Valid data would decode successfully if selection skipped its name check. + setup.selected_name = "another-selected-backend"; + assert!( + SelectedBackend::select( + &setup, + TestSetup::descriptor(), + Some(TEST_BACKEND), + Some("sandbox-1"), + &setup.auth + ) + .is_err() + ); + assert!(setup.observed.events().is_empty()); +} + +#[test] +fn shared_identity_rejection_stops_before_discovery() { + let setup = TestSetup::new(); + for sandbox in [None, Some(""), Some("other-sandbox")] { + assert!( + SelectedBackend::select( + &setup, + TestSetup::descriptor(), + Some(TEST_BACKEND), + sandbox, + &setup.auth + ) + .is_err() + ); + } + let mut auth = setup.auth.clone(); + auth.session_id = SandboxSessionId::new(); + assert!( + SelectedBackend::select( + &setup, + TestSetup::descriptor(), + Some(TEST_BACKEND), + Some("sandbox-1"), + &auth + ) + .is_err() + ); + auth = setup.auth.clone(); + auth.runtime_generation = SandboxGenerationId::parse("other-generation").unwrap(); + assert!( + SelectedBackend::select( + &setup, + TestSetup::descriptor(), + Some(TEST_BACKEND), + Some("sandbox-1"), + &auth + ) + .is_err() + ); + assert_eq!(setup.observed.events(), vec!["decode"; 5]); + assert_eq!(setup.observed.starts.load(Ordering::SeqCst), 0); +} + +#[tokio::test] +async fn attachment_receives_live_supervisor_services() { + let setup = TestSetup::new(); + let services = setup.services(); + let ca_paths = services.ca_file_paths.clone(); + let providers = services.provider_credentials.clone(); + let bearer = services.sandbox_bearer.clone(); + let bound = setup + .select() + .attach(services, policy(), agent()) + .await + .unwrap(); + let backend_services = setup.observed.services.lock().unwrap().upgrade().unwrap(); + assert!(Arc::ptr_eq(&ca_paths, &backend_services.ca_file_paths)); + assert!(Arc::ptr_eq( + &providers.snapshot(), + &backend_services.provider_credentials.snapshot() + )); + + // Later updates must reach the handles retained during attachment. + *ca_paths.lock().unwrap() = Some(("/test/ca".into(), "/test/bundle".into())); + providers.install_child_env_snapshot( + 2, + HashMap::from([("TEST_CONFIG".into(), "repaired".into())]), + ); + bearer + .update( + SecretJwt::parse("rotated-test-token").unwrap(), + 0, + CredentialEpoch::new(2).unwrap(), + ) + .unwrap(); + assert_eq!( + *backend_services.ca_file_paths.lock().unwrap(), + Some(("/test/ca".into(), "/test/bundle".into())) + ); + assert!(Arc::ptr_eq( + &providers.snapshot(), + &backend_services.provider_credentials.snapshot() + )); + assert_eq!( + backend_services.provider_credentials.snapshot().child_env["TEST_CONFIG"], + "repaired" + ); + assert_eq!( + backend_services.sandbox_bearer.credential_epoch(), + Some(CredentialEpoch::new(2).unwrap()) + ); + bearer.clear(); + assert!( + backend_services + .sandbox_bearer + .authorization_metadata() + .is_err() + ); + drop(backend_services); + drop(bound); + assert!(setup.observed.services.lock().unwrap().upgrade().is_none()); +} + +#[tokio::test] +async fn constructed_backend_must_keep_selected_name() { + let mut setup = TestSetup::new(); + setup.built_name = "wrong-backend"; + assert!( + setup + .select() + .attach(setup.services(), policy(), agent()) + .await + .is_err() + ); + assert_eq!(setup.observed.events(), ["decode", "build"]); + assert_eq!(setup.observed.starts.load(Ordering::SeqCst), 0); + assert!(setup.observed.services.lock().unwrap().upgrade().is_none()); +} + +#[test] +fn standard_setup_decodes_native_descriptor_and_preserves_vm_identity() { + use openshell_sandbox_backend::boundary_protocol::{ + SandboxRuntimeDescriptor, SandboxTlsClientConfig, SandboxTransport, + }; + + assert!(OpenShellBackendSetup.decode(b"{}").is_err()); + let setup = TestSetup::new(); + let mut descriptor = SandboxRuntimeDescriptor { + boundary_id: setup.sandbox_id.clone(), + generation: setup.auth.runtime_generation.to_string(), + session_id: setup.auth.session_id, + workload_identity: ResolvedWorkloadIdentity::new( + 1000, + 1001, + vec![], + "test".into(), + "test-resource".into(), + ) + .unwrap(), + transport: SandboxTransport::Unix { + socket_path: "/unused-test-runtime".into(), + }, + tls: SandboxTlsClientConfig { + server_name: "unused-test-runtime".into(), + trust_anchor_pem: String::new(), + }, + host_gateway_ip: None, + resource_claims: BTreeMap::new(), + outer_fence: OuterFenceGuarantees::from_enforcement_evidence( + "generation-1", + [], + b"test-evidence", + ) + .unwrap(), + }; + for is_vm in [false, true] { + if is_vm { + descriptor + .resource_claims + .insert("vm.generation".into(), "generation-1".into()); + } + let selected = SelectedBackend::select( + &OpenShellBackendSetup, + descriptor.backend_descriptor().unwrap(), + Some(openshell_sandbox_backend::BACKEND_NAME), + Some(&setup.sandbox_id), + &setup.auth, + ) + .unwrap(); + let projected = selected.vm_policy_identity(); + assert_eq!(projected.is_some(), is_vm); + if let Some(projected) = projected { + assert_eq!((projected.uid, projected.gid), (1000, 1001)); + for (user, group, accepted) in [ + ("1000", "1001", true), + ("1001", "sandbox", false), + ("sandbox", "1000", false), + ] { + let policy = openshell_core::proto::SandboxPolicy { + process: Some(openshell_core::proto::ProcessPolicy { + run_as_user: user.into(), + run_as_group: group.into(), + }), + ..Default::default() + }; + assert_eq!( + projected.validate(&policy).is_ok(), + accepted, + "{user}:{group}" + ); + } + } + } +} + +/// Auth slots and shutdown signals are process-wide. Run the real startup in a +/// child so parallel tests cannot replace credentials or consume its SIGTERM. +#[cfg(unix)] +#[tokio::test] +async fn shared_startup_uses_selected_backend_through_readiness_and_shutdown() { + use std::time::Duration; + + const CHILD_ROOT: &str = "OPENSHELL_TEST_SELECTED_BACKEND_ROOT"; + const TEST: &str = + "backend_setup::tests::shared_startup_uses_selected_backend_through_readiness_and_shutdown"; + let Ok(root) = std::env::var(CHILD_ROOT) else { + // Keep Unix socket paths short, including on macOS. The parent owns + // every fixture file so timeout or child failure still removes them. + let root = tempfile::Builder::new() + .prefix("selected-backend-") + .tempdir_in("/tmp") + .unwrap(); + let mut child = tokio::process::Command::new(std::env::current_exe().unwrap()); + child.args(["--exact", TEST, "--nocapture", "--test-threads=1"]); + for (name, _) in std::env::vars_os() { + if name.to_string_lossy().starts_with("OPENSHELL_") { + child.env_remove(name); + } + } + child + .env(CHILD_ROOT, root.path()) + .env( + openshell_core::sandbox_env::PROXY_TLS_DIR, + root.path().join("tls"), + ) + .kill_on_drop(true); + let output = tokio::time::timeout(Duration::from_secs(30), child.output()) + .await + .expect("shared startup child exceeded its deadline") + .expect("run isolated shared startup test"); + assert!( + output.status.success(), + "shared startup child failed: {}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr), + ); + return; + }; + + // Match the binary's process setup before exercising the shared library. + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + let root = PathBuf::from(root); + let rules = root.join("policy.rego"); + let data = root.join("policy.yaml"); + let readiness = root.join("ready.sock"); + let marker = root.join("main-exit"); + std::fs::write( + &rules, + include_str!("../../../openshell-supervisor-network/data/sandbox-policy.rego"), + ) + .unwrap(); + std::fs::write(&data, "network_policies: {}\n").unwrap(); + let reservation = std::net::TcpListener::bind((std::net::Ipv4Addr::LOCALHOST, 0)).unwrap(); + let health_port = reservation.local_addr().unwrap().port(); + drop(reservation); + + let setup = TestSetup::new(); + setup + .observed + .require_networking + .store(true, Ordering::SeqCst); + let startup = || { + Box::pin(crate::run_sandbox_with_backend( + &setup, + crate::SandboxRunConfig { + command: vec!["test-agent".into(), "test-argument".into()], + workdir: Some("/test-workspace".into()), + timeout_secs: 15, + interactive: false, + await_main_process_attachment: false, + sandbox_id: Some(setup.sandbox_id.clone()), + sandbox: None, + openshell_endpoint: None, + policy_rules: Some(rules.to_string_lossy().into_owned()), + policy_data: Some(data.to_string_lossy().into_owned()), + ssh_socket_path: None, + health_socket_path: Some(readiness.clone()), + health_port: Some(health_port), + ocsf_enabled: Arc::new(AtomicBool::new(false)), + ocsf_schema_version: Arc::new(Mutex::new(String::new())), + upstream_proxy_args: UpstreamProxyArgs::default(), + backend_descriptor: TestSetup::descriptor(), + auth_bundle: setup.auth.clone(), + admitted_isolation_backend: Some(TEST_BACKEND.into()), + main_exit_marker: Some(marker.clone()), + }, + )) + }; + + setup + .observed + .discovery_unavailable + .store(true, Ordering::SeqCst); + let error = tokio::time::timeout(Duration::from_secs(5), startup()) + .await + .expect("discovery failure must end startup") + .expect_err("discovery failure must reject startup"); + assert!(error.to_string().contains("discovery unavailable")); + assert_eq!(setup.observed.events(), ["decode", "discover"]); + assert_eq!(setup.observed.starts.load(Ordering::SeqCst), 0); + assert!(!readiness.exists()); + assert!(!marker.exists()); + assert!( + tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST, health_port)) + .await + .is_err() + ); + setup + .observed + .discovery_unavailable + .store(false, Ordering::SeqCst); + setup + .observed + .deny_confirmation + .store(true, Ordering::SeqCst); + assert!(startup().await.is_err()); + assert_eq!( + setup.observed.events(), + [ + "decode", "discover", "decode", "discover", "build", "attach", "confirm" + ] + ); + assert_eq!(setup.observed.starts.load(Ordering::SeqCst), 0); + assert_eq!(setup.observed.releases.load(Ordering::SeqCst), 1); + assert!(!setup.observed.active.load(Ordering::SeqCst)); + assert!(setup.observed.services.lock().unwrap().upgrade().is_none()); + assert!(!readiness.exists()); + assert!(!marker.exists()); + assert!( + tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST, health_port)) + .await + .is_err() + ); + + setup + .observed + .deny_confirmation + .store(false, Ordering::SeqCst); + let observe_and_shutdown = async { + tokio::time::timeout(Duration::from_secs(10), async { + loop { + if marker.exists() + && tokio::net::UnixStream::connect(&readiness).await.is_ok() + && tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST, health_port)) + .await + .is_ok() + { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("selected backend did not reach process completion and readiness"); + assert_eq!(std::fs::read_to_string(&marker).unwrap(), "exit_code=0\n"); + assert_eq!(setup.observed.starts.load(Ordering::SeqCst), 1); + assert!(setup.observed.active.load(Ordering::SeqCst)); + // Exercise the production shutdown future while it retains access + // after main-process exit; only this isolated child receives the signal. + nix::sys::signal::kill(nix::unistd::Pid::this(), nix::sys::signal::Signal::SIGTERM) + .unwrap(); + }; + let (result, ()) = tokio::join!(startup(), observe_and_shutdown); + assert_eq!(result.unwrap(), 0); + assert_eq!( + setup.observed.events(), + [ + "decode", "discover", "decode", "discover", "build", "attach", "confirm", "decode", + "discover", "build", "attach", "confirm", "start" + ] + ); + assert_eq!(setup.observed.releases.load(Ordering::SeqCst), 2); + assert!(!setup.observed.active.load(Ordering::SeqCst)); + assert!(setup.observed.services.lock().unwrap().upgrade().is_none()); + assert!(!readiness.exists()); + // Listener tasks are aborted on drop; let the runtime reap their sockets. + tokio::task::yield_now().await; + assert!( + tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST, health_port)) + .await + .is_err() + ); +} diff --git a/crates/openshell-supervisor/src/lib.rs b/crates/openshell-supervisor/src/lib.rs index 5f2c95bd1f..e8c49e1bbc 100644 --- a/crates/openshell-supervisor/src/lib.rs +++ b/crates/openshell-supervisor/src/lib.rs @@ -16,6 +16,7 @@ compile_error!( ); mod activity_aggregator; +mod backend_setup; mod denial_aggregator; mod endpoint_status; mod mechanistic_mapper; @@ -28,7 +29,7 @@ use std::pin::Pin; use std::sync::Arc; use std::sync::atomic::{AtomicBool, AtomicU32}; use std::time::Duration; -use tracing::{debug, info, warn}; +use tracing::{Instrument as _, debug, info, warn}; use openshell_core::PolicyValidationFailureMode; @@ -637,6 +638,87 @@ pub async fn run_sandbox( admitted_isolation_backend: Option, main_exit_marker: Option, ) -> Result { + // Shared startup retains policy and networking state; box it to keep callers' futures small. + Box::pin(run_sandbox_with_backend( + &backend_setup::OpenShellBackendSetup, + SandboxRunConfig { + command, + workdir, + timeout_secs, + interactive, + await_main_process_attachment, + sandbox_id, + sandbox, + openshell_endpoint, + policy_rules, + policy_data, + ssh_socket_path, + health_socket_path, + health_port, + ocsf_enabled, + ocsf_schema_version, + upstream_proxy_args, + backend_descriptor, + auth_bundle, + admitted_isolation_backend, + main_exit_marker, + }, + )) + .await +} + +struct SandboxRunConfig { + command: Vec, + workdir: Option, + timeout_secs: u64, + interactive: bool, + await_main_process_attachment: bool, + sandbox_id: Option, + sandbox: Option, + openshell_endpoint: Option, + policy_rules: Option, + policy_data: Option, + ssh_socket_path: Option, + health_socket_path: Option, + health_port: Option, + ocsf_enabled: Arc, + ocsf_schema_version: Arc>, + upstream_proxy_args: openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs, + backend_descriptor: openshell_isolation_interface::contract::BackendDescriptor, + auth_bundle: openshell_core::jwt::SupervisorAuthBundle, + admitted_isolation_backend: Option, + main_exit_marker: Option, +} + +/// Trusted composition chooses the setup before shared admission, policy, and +/// lifecycle handling. Payload contents never select a backend implementation. +#[allow(clippy::similar_names)] +async fn run_sandbox_with_backend( + backend_setup: &dyn backend_setup::BackendSetup, + config: SandboxRunConfig, +) -> Result { + let SandboxRunConfig { + command, + workdir, + timeout_secs, + interactive, + await_main_process_attachment, + sandbox_id, + sandbox, + openshell_endpoint, + policy_rules, + policy_data, + ssh_socket_path, + health_socket_path, + health_port, + ocsf_enabled, + ocsf_schema_version, + upstream_proxy_args, + backend_descriptor, + auth_bundle, + admitted_isolation_backend, + main_exit_marker, + } = config; // An empty command is the versioned scratch-sandbox sentinel. The // external supervisor cannot inspect the workload filesystem, so preserve // it for openshell-sandbox to resolve against the agent image. @@ -674,22 +756,27 @@ pub async fn run_sandbox( // and the policy poll loop that rotates them stay the same objects. let extension_credentials = openshell_extension_core::ExtensionCredentialStore::new(); - let runtime_descriptor: openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor = - serde_json::from_slice(&backend_descriptor.payload) - .map_err(|error| miette::miette!("decode sandbox runtime descriptor: {error}"))?; - if auth_bundle.runtime_generation.as_str() != runtime_descriptor.generation { - return Err(miette::miette!( - "supervisor authentication bundle does not match runtime generation" - )); - } + let selected_backend = backend_setup::SelectedBackend::select( + backend_setup, + backend_descriptor, + admitted_isolation_backend.as_deref(), + sandbox_id.as_deref(), + &auth_bundle, + )?; let sandbox_bearer = openshell_core::grpc_client::install_supervisor_auth_bundle(&auth_bundle)?; - let (image_yaml, invalid_image) = - openshell_sandbox_backend::OpenShellRuntimeBackend::discover_policy( - runtime_descriptor.clone(), - sandbox_bearer.clone(), - ) - .await - .map_err(|error| miette::miette!("discover workload image policy: {error}"))?; + // Startup joins the trace that created the sandbox when the driver passes + // one, and ends once the access plane is up. + let startup = tracing::info_span!( + "supervisor.startup", + sandbox.id = sandbox_id.as_deref().unwrap_or_default(), + otel.status_code = tracing::field::Empty, + ); + openshell_otel::set_parent_from_environment(&startup); + let startup_status = startup.in_scope(openshell_otel::ErrorStatusGuard::current); + let (image_yaml, invalid_image) = selected_backend + .discover_policy(sandbox_bearer.clone()) + .instrument(tracing::info_span!(parent: &startup, "supervisor.boundary.discover_policy")) + .await?; let image_discovery = if invalid_image { ImagePolicyDiscovery::Invalid } else if let Some(yaml) = image_yaml { @@ -701,13 +788,7 @@ pub async fn run_sandbox( ImagePolicyDiscovery::Missing }; - let vm_policy_identity = runtime_descriptor - .resource_claims - .contains_key("vm.generation") - .then_some(VmPolicyIdentity { - uid: runtime_descriptor.workload_identity.uid, - gid: runtime_descriptor.workload_identity.gid, - }); + let vm_policy_identity = selected_backend.vm_policy_identity(); // Load policy and initialize OPA engine let openshell_endpoint_for_proxy = openshell_endpoint.clone(); @@ -735,6 +816,7 @@ pub async fn run_sandbox( endpoint: openshell_endpoint.clone().unwrap_or_default(), }, ) + .instrument(tracing::info_span!(parent: &startup, "supervisor.policy.load")) .await?; // Normalize the active driver's identity contract once, while both the @@ -753,7 +835,10 @@ pub async fn run_sandbox( // This is done after loading the policy so the sandbox can still start // even if provider env fetch fails (graceful degradation). let environment = if let (Some(id), Some(endpoint)) = (&sandbox_id, &openshell_endpoint) { - match openshell_core::grpc_client::fetch_provider_environment(endpoint, id).await { + match openshell_core::grpc_client::fetch_provider_environment(endpoint, id) + .instrument(startup.clone()) + .await + { Ok(result) => { ocsf_emit!( ConfigStateChangeBuilder::new(ocsf_ctx()) @@ -823,44 +908,29 @@ pub async fn run_sandbox( // the entrypoint process's /proc/net/tcp for identity binding. let entrypoint_pid = Arc::new(AtomicU32::new(0)); - // The sandbox runtime uses the shared authenticated boundary protocol. - // The admitted backend name is resolved independently of the protected - // descriptor, and generic supervisor code never imports a driver crate. - let admitted_backend_name = admitted_isolation_backend.ok_or_else(|| { - miette::miette!("runtime descriptor supplied without an admitted isolation backend") - })?; - let session_id = runtime_descriptor.session_id; + // Construct the selected client only after shared policy and credential + // setup has completed. Keep live credentials and later CA publication in + // supervisor-owned handles shared with the backend. + let admitted_backend_name = selected_backend.backend_name().to_string(); let ca_file_paths = Arc::new(std::sync::Mutex::new(None)); - let backend: Arc = - Arc::new(openshell_sandbox_backend::OpenShellRuntimeBackend::new( - ca_file_paths.clone(), - provider_credentials.clone(), - sandbox_bearer, - )); - let mut registry = openshell_isolation_interface::contract::BackendRegistry::new(); - registry - .register(backend) - .map_err(|error| miette::miette!(error.to_string()))?; - let (backend, verified) = registry - .resolve(backend_descriptor, &admitted_backend_name) - .map_err(|error| miette::miette!(error.to_string()))?; - let context = openshell_isolation_interface::contract::SandboxContext { - sandbox_id: sandbox_id.clone().unwrap_or_default(), - session_id, - policy: policy.clone(), - agent: openshell_isolation_interface::AgentSpec { - program, - args, - workdir: workspace, - timeout_secs, - interactive, - }, - identity: runtime_descriptor.workload_identity, - }; - let bound = backend - .attach(verified, context) - .await - .map_err(|error| miette::miette!(error.to_string()))?; + let bound = selected_backend + .attach( + backend_setup::BackendServices { + ca_file_paths: ca_file_paths.clone(), + provider_credentials: provider_credentials.clone(), + sandbox_bearer, + }, + policy.clone(), + openshell_isolation_interface::AgentSpec { + program, + args, + workdir: workspace, + timeout_secs, + interactive, + }, + ) + .instrument(tracing::info_span!(parent: &startup, "supervisor.boundary.attach")) + .await?; info!(backend = %admitted_backend_name, "Isolation boundary attached"); let remote_boundary = (bound, admitted_backend_name, ca_file_paths); @@ -933,6 +1003,7 @@ pub async fn run_sandbox( let (bound, backend_name, ca_file_paths) = remote_boundary; let ready = bound .confirm() + .instrument(tracing::info_span!(parent: &startup, "supervisor.boundary.confirm")) .await .map_err(|error| miette::miette!(error.to_string()))?; info!(backend = %backend_name, "Isolation boundary enforcement confirmed"); @@ -1161,6 +1232,7 @@ pub async fn run_sandbox( let running = confirmed .into_boundary() .start_agent() + .instrument(tracing::info_span!(parent: &startup, "supervisor.boundary.start_agent")) .await .map_err(|error| miette::miette!(error.to_string()))?; workload_started_tx.send_replace(true); @@ -1179,8 +1251,11 @@ pub async fn run_sandbox( agent.clone(), Some(supervisor_session_updates), ) + .instrument(tracing::info_span!(parent: &startup, "supervisor.access.start")) .await?; info!(backend = %backend_name, "Control-mode access plane started"); + startup_status.finish(Ok::<_, ()>(())).ok(); + drop(startup); let _provider_reporter = sandbox_id .as_ref() @@ -5000,6 +5075,60 @@ mod tests { use super::*; use std::sync::atomic::{AtomicUsize, Ordering}; + /// Admission must reject this input before any backend decodes its payload + /// or contacts a workload. Keep this fixture independent of setup helpers + /// so it can also reproduce the ordering defect on earlier revisions. + #[tokio::test] + async fn startup_rejects_missing_admission_before_decoding_payload() { + let auth_bundle = openshell_core::jwt::SupervisorAuthBundle { + session_id: openshell_core::SandboxSessionId::new(), + runtime_generation: openshell_core::sandbox_generation::SandboxGenerationId::parse( + "generation-1", + ) + .expect("generation"), + session_rotation: openshell_core::jwt::SessionRotation::new(1).expect("rotation"), + auth_epoch: openshell_core::jwt::CredentialEpoch::new(1).expect("epoch"), + gateway_token: openshell_core::jwt::SecretJwt::parse("test-gateway-token") + .expect("token"), + gateway_expires_at: 0, + sandbox_token: openshell_core::jwt::SecretJwt::parse("test-sandbox-token") + .expect("token"), + sandbox_expires_at: 0, + }; + let error = run_sandbox( + vec!["true".to_string()], + None, + 0, + false, + false, + Some("sandbox-1".to_string()), + None, + None, + None, + None, + None, + None, + None, + Arc::new(AtomicBool::new(false)), + Arc::new(std::sync::Mutex::new(String::new())), + openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs::default(), + openshell_isolation_interface::contract::BackendDescriptor { + backend_name: "in-process-test".to_string(), + payload: b"in-process-only-launch-data".to_vec(), + }, + auth_bundle, + None, + None, + ) + .await + .expect_err("missing admission cannot authorize startup"); + + assert_eq!( + error.to_string(), + "runtime descriptor supplied without an admitted isolation backend" + ); + } + fn effective_bool(value: bool) -> openshell_core::proto::EffectiveSetting { openshell_core::proto::EffectiveSetting { value: Some(openshell_core::proto::SettingValue { diff --git a/crates/openshell-supervisor/src/main.rs b/crates/openshell-supervisor/src/main.rs index 4bfb469891..0faccdbb63 100644 --- a/crates/openshell-supervisor/src/main.rs +++ b/crates/openshell-supervisor/src/main.rs @@ -18,6 +18,7 @@ use tracing_subscriber::{Layer as _, layer::SubscriberExt as _, util::Subscriber const DEBUG_RPC_SUBCOMMAND: &str = "debug-rpc"; const HEALTH_SUBCOMMAND: &str = "health"; +const SERVICE_NAME: &str = "openshell-supervisor"; #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, ValueEnum)] enum SupervisorRole { @@ -80,6 +81,10 @@ struct Args { #[arg(long, default_value = "warn", env = openshell_core::sandbox_env::LOG_LEVEL)] log_level: String, + /// OTLP/gRPC collector endpoint for supervisor trace export. + #[arg(long, env = openshell_core::sandbox_env::OTLP_ENDPOINT)] + otlp_endpoint: Option, + /// Create the private readiness socket after boundary and gateway attach. #[arg(long, env = "OPENSHELL_HEALTH_SOCKET_PATH")] health_socket_path: Option, @@ -327,6 +332,22 @@ fn main() -> Result<()> { .enable_all() .build() .into_diagnostic()?; + // The tonic exporter must be created inside the runtime that drives it. + let (otlp_provider, otlp_setup_error) = runtime.block_on(async { + openshell_otel::provider_for(args.otlp_endpoint.as_deref().map(|endpoint| { + openshell_otel::OtlpTraceConfig { + endpoint, + service_name: openshell_otel::ServiceName::Fixed(SERVICE_NAME), + service_version: Some(openshell_core::VERSION), + resource_attributes: args + .sandbox_id + .iter() + .map(|id| opentelemetry::KeyValue::new("openshell.sandbox.id", id.clone())) + .collect(), + } + })) + }); + let otlp_layer_provider = otlp_provider.clone(); let exit_code = runtime.block_on(async move { let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); @@ -378,6 +399,12 @@ fn main() -> Result<()> { ) .with(jsonl_layer.with_filter(LevelFilter::INFO)) .with(push_layer.clone()) + .with( + otlp_layer_provider + .as_ref() + .map(|provider| openshell_otel::layer(provider, SERVICE_NAME)) + .with_filter(otlp_span_filter(&args.log_level)), + ) .init(); (Some(file_guard), jsonl_guard) } else { @@ -388,10 +415,21 @@ fn main() -> Result<()> { .with_filter(console_filter), ) .with(push_layer) + .with( + otlp_layer_provider + .as_ref() + .map(|provider| openshell_otel::layer(provider, SERVICE_NAME)) + .with_filter(otlp_span_filter(&args.log_level)), + ) .init(); warn!("Could not open /var/log for log rotation; using stderr-only logging"); (None, None) }; + if let Some(error) = otlp_setup_error { + warn!(%error, "OTLP exporting could not be started; continuing without it"); + } else if let Some(endpoint) = &args.otlp_endpoint { + info!(endpoint, "OTLP exporting enabled"); + } let workdir = args.workdir.clone(); let (command, interactive, await_main_process_attachment) = if !args.command.is_empty() { @@ -473,15 +511,40 @@ fn main() -> Result<()> { .await } } - })?; + }); + // Flush spans before `process::exit` skips destructors. + if let Some(provider) = otlp_provider + && let Err(error) = provider.shutdown() + { + warn!(%error, "OTLP tracer provider shutdown failed"); + } + + std::process::exit(exit_code?); +} - std::process::exit(exit_code); +/// A more verbose log level raises `openshell*` spans; dependency spans stay at +/// INFO so they do not flood the exporter queue. +fn otlp_span_filter(log_level: &str) -> EnvFilter { + let level = log_level + .parse::() + .unwrap_or(LevelFilter::INFO) + .max(LevelFilter::INFO); + EnvFilter::new(format!("info,openshell={level}")) } #[cfg(test)] mod tests { use super::*; + #[test] + fn otlp_span_filter_raises_only_openshell_targets() { + let directives = |log_level: &str| otlp_span_filter(log_level).to_string(); + assert_eq!(directives("warn"), "openshell=info,info"); + assert_eq!(directives("info"), "openshell=info,info"); + assert_eq!(directives("debug"), "openshell=debug,info"); + assert_eq!(directives("info,h2=debug"), "openshell=info,info"); + } + #[test] fn isolation_backend_is_the_default_role() { let directory = tempfile::tempdir().expect("temporary runtime descriptor directory"); diff --git a/deploy/docker/docker-compose.yml b/deploy/docker/docker-compose.yml index e491b825c3..6c5681c93a 100644 --- a/deploy/docker/docker-compose.yml +++ b/deploy/docker/docker-compose.yml @@ -9,7 +9,7 @@ # # Quick start: # -# 1. Start the gateway: +# 1. Start the gateway (init first generates JWT keys in /var/lib/openshell/tls): # docker compose up -d # # 2. Register the gateway with the CLI (one-time): @@ -49,19 +49,16 @@ # bind-mount source when sandbox containers are created. Named volumes # cannot be used here because Docker resolves bind-mount sources against the # host filesystem, not the container filesystem. -# -# Linux note: -# host.docker.internal and host.openshell.internal are not automatically -# added on Linux Docker. Add the following under the gateway service: -# extra_hosts: -# - "host.docker.internal:host-gateway" -# - "host.openshell.internal:host-gateway" services: gateway: - image: ghcr.io/nvidia/openshell/gateway:${IMAGE_TAG:-latest} + image: &gateway-image ghcr.io/nvidia/openshell/gateway:${IMAGE_TAG:-latest} restart: unless-stopped + depends_on: + init: + condition: service_completed_successfully + # Clear the default CMD so gateway.toml owns all settings (see note above). command: [] @@ -96,7 +93,8 @@ services: # (e.g. /var/lib/openshell/gateway): the path must match exactly on # both the host and inside the container, and a single gateway per host # is the expected topology. - - type: bind + - &openshell-data + type: bind source: /var/lib/openshell target: /var/lib/openshell bind: @@ -126,3 +124,14 @@ services: # bind-mounted directory so its path is resolvable by the host Docker daemon. XDG_DATA_HOME: /var/lib/openshell HOME: /var/lib/openshell + + # One-shot: generates the sandbox JWT keys (kept if present). + init: + image: *gateway-image + user: "0" + restart: "no" + command: ["generate-certs", "--output-dir", "/var/lib/openshell/tls"] + environment: + HOME: /var/lib/openshell + volumes: + - *openshell-data diff --git a/deploy/docker/gateway.toml b/deploy/docker/gateway.toml index c0cbda4ef6..eea0faa78d 100644 --- a/deploy/docker/gateway.toml +++ b/deploy/docker/gateway.toml @@ -14,12 +14,8 @@ # clearing the CMD first. # # grpc_endpoint note: -# host.docker.internal is automatically resolvable from containers on -# Docker Desktop (Windows / macOS). On Linux, add extra_hosts to the -# gateway service: -# extra_hosts: -# - "host.docker.internal:host-gateway" -# - "host.openshell.internal:host-gateway" +# Omitted on purpose. The supervisor uses host networking and reaches the +# published port at http://127.0.0.1:8080 (keep OPENSHELL_PORT at 8080). [openshell] version = 2 @@ -33,6 +29,18 @@ log_level = "info" compute_driver = "docker" disable_tls = true +# No TLS and loopback only: user calls are unauthenticated. +# Supervisors still authenticate with the JWT keys below. +[openshell.gateway.auth] +allow_unauthenticated_users = true + +# Written by the compose init service. +[openshell.gateway.gateway_jwt] +signing_key_path = "/var/lib/openshell/tls/jwt/signing.pem" +public_key_path = "/var/lib/openshell/tls/jwt/public.pem" +kid_path = "/var/lib/openshell/tls/jwt/kid" +gateway_id = "openshell-docker" + [openshell.drivers.docker] # Default image pulled for `openshell sandbox create` without --from. default_image = "nvcr.io/nvidia/base/ubuntu:24.04" @@ -44,12 +52,6 @@ supervisor_image = "ghcr.io/nvidia/openshell/supervisor:latest" image_pull_policy = "if_not_present" # Value assigned to the openshell.sandbox_namespace label on sandbox containers. sandbox_label = "openshell" -# Address sandbox containers use to call back to the gateway. -# The Docker driver replaces the host with host.openshell.internal and the -# port with the gateway's own bind port (8080). Only the scheme survives. -# The gateway must be published on port 8080 on the Docker host so that -# host.openshell.internal:8080 resolves to the gateway container. -grpc_endpoint = "http://host.openshell.internal:8080" # Explicit supervisor-compatible Docker default. Set RuntimeDefault or # Localhost/ only when the daemon host has AppArmor available. app_armor_profile = "Unconfined" diff --git a/docs/CONTRIBUTING.mdx b/docs/CONTRIBUTING.mdx index 91a422bd45..e6676056a0 100644 --- a/docs/CONTRIBUTING.mdx +++ b/docs/CONTRIBUTING.mdx @@ -161,7 +161,7 @@ Use these consistently: 2. Make your changes. 3. Preview locally with `mise run docs:serve`. 4. Run `mise run docs`. -5. Run `mise run pre-commit` to catch formatting issues. +5. Run the relevant formatting, link, and content checks following [Choose Verification for the Change in CONTRIBUTING.md](https://github.com/NVIDIA/OpenShell/blob/main/CONTRIBUTING.md#choose-verification-for-the-change). Use `mise run pre-commit` when its broader scope is warranted. 6. Open a PR with `docs:` as the conventional commit type. ```text diff --git a/docs/about/support-matrix.mdx b/docs/about/support-matrix.mdx index f6b85765ca..3edbdafd0f 100644 --- a/docs/about/support-matrix.mdx +++ b/docs/about/support-matrix.mdx @@ -176,8 +176,8 @@ when it runs inside a container or microVM: | -------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Landlock LSM](https://docs.kernel.org/security/landlock.html) | Required | ABI 3 or newer, introduced in Linux 6.2, with Landlock enabled. The mandatory baseline protects private channel and bootstrap files, including against truncation. A filesystem policy's `best_effort` setting never disables this baseline. | | seccomp | Required | Nested user-notification filters and atomic `SECCOMP_IOCTL_NOTIF_ADDFD` with `SECCOMP_ADDFD_FLAG_SEND`, usable under the runtime's existing seccomp profile without added capabilities. The sandbox actively probes these operations before admitting the workload. | -| Task-memory access | Required | The non-dumpable broker must be able to read and write a same-UID, dumpable workload child's memory through `process_vm_readv` / `process_vm_writev` or `/proc//mem`. The sandbox actively probes the production parent-to-child topology before admitting the workload. | -| seccomp `WAIT_KILLABLE_RECV` | Recommended (Linux 5.19+) | Keeps a notified workload thread in a kill-only wait so the broker can safely write mediated results into workload memory. Without it (kernels < 5.19, for example RHEL 9.x / RHCOS 5.14) the sandbox still starts, in a reduced **legacy read-only** mode described below. | +| Task-memory access | Required | The non-dumpable broker must be able to read a same-UID, dumpable workload child's memory through `process_vm_readv` or `/proc//mem`. The broker never writes workload memory. The sandbox actively probes the production parent-to-child topology before admitting the workload. | +| Socket device binding | Required | The sandbox binds every workload TCP and UDP socket to the loopback interface with `SO_BINDTODEVICE`, without added capabilities. Accepted sockets inherit the binding, so workloads accept connections only from inside the sandbox network namespace. The sandbox probes this before admitting the workload. | A kernel version alone does not establish support. A disabled Landlock LSM or a runtime profile that blocks the required seccomp operations causes launch to @@ -185,35 +185,20 @@ fail closed. An upstream Linux 6.2 or newer kernel provides the required Landlock ABI; distribution backports must pass the same active qualification. The broker remains non-dumpable during qualification. A runtime may satisfy task-memory access through `/proc//mem` even when its kernel omits the -`process_vm_readv` and `process_vm_writev` system calls; OpenShell qualifies the +`process_vm_readv` system call; OpenShell qualifies the same parent-to-executed-child access shape used by mediated workloads. -### Legacy read-only mode (kernels before Linux 5.19) - -`SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` was added in Linux 5.19. On older -kernels β€” notably RHEL 9.x and RHCOS, which ship a 5.14 kernel β€” the sandbox -cannot install a kill-only listener, so it falls back to a plain listener and -runs in a **legacy read-only** cancellation mode. The sandbox starts and -enforces the full isolation boundary (Landlock, the outer NetworkPolicy fence, -DNS and TCP authorization); the only difference is that the broker refuses the -mediated operations that write results back into workload memory, failing them -closed with `EOPNOTSUPP`: - -- `getpeername`; -- `accept` / `accept4` **when a non-null peer-address argument is supplied** - (a null address argument still works); -- `sendmmsg` paths that write per-message lengths back to the caller. - -Socket creation, `connect`, `bind`, `listen`, `sendto`, and `sendmsg` are -unaffected. Outbound-oriented workloads generally run unchanged; server -workloads whose accept wrappers request the peer address will see `EOPNOTSUPP` -until the node runs a kernel that provides `WAIT_KILLABLE_RECV` (Linux 5.19+, or -a distribution backport). The selected mode is reported in the sandbox -qualification output as `seccomp_listener_mode` (`killable` or -`legacy_read_only`). - On macOS, these kernel modules run inside the Docker Desktop Linux VM, not on the host kernel. +### Socket peer addresses + +On outbound TCP connections relayed by OpenShell, `getpeername()` reports the +local relay's loopback address (`127.0.0.1` or `::1`) and port rather than the +upstream destination. Accepted sockets and directly connected local sockets +report their actual peer address; inbound peer-address reporting retains its +normal socket semantics. Connection authorization continues to use the requested +destination, independently of the address reported by `getpeername()`. + ## Agent Workloads OpenShell runs agents and tools that you install in a user-owned OCI image. The diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index c44101150b..a2b72381a1 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -375,6 +375,10 @@ Only OpenTelemetry traces are exported. Inbound gRPC and HTTP requests produce s The gateway forwards the OTLP configuration, configured gateway name, configured compute driver, and W3C trace context to managed external drivers. Built-in drivers also export their spans to the same collector through dedicated in-process providers. Driver spans retain the gateway trace context, use a distinct service name such as `openshell-driver-docker` or `openshell-driver-podman`, and carry the same `openshell.gateway.name` and `openshell.gateway.compute_driver` resource attributes as gateway spans. Compute-driver client and server spans use the same fully qualified protobuf operation name, such as `openshell.compute.v1.ComputeDriver/CreateSandbox`, in both the span name and `rpc.method`. The service name and span kind distinguish each side. Backend-prefixed child spans identify implementation work. A streaming watch records a terminal status when observed; consumer teardown without a terminal status leaves the span status unset. Operator-run external drivers own their own telemetry configuration. +Sandbox supervisors export spans to the same collector, so the endpoint must also be reachable from sandboxes. + +Spans for routine or high-volume operations use DEBUG level, and the default INFO level does not export them. Set the gateway or sandbox log level to `debug` to include them. + For Helm deployments, set `server.otlp.endpoint` to render this table. The optional `server.otlp.serviceName` value overrides the gateway service name; driver service names remain fixed. diff --git a/docs/how-it-works/gateways/container-deployment.mdx b/docs/how-it-works/gateways/container-deployment.mdx index e42ede6bc4..660eea5995 100644 --- a/docs/how-it-works/gateways/container-deployment.mdx +++ b/docs/how-it-works/gateways/container-deployment.mdx @@ -156,7 +156,7 @@ the repository contains a production-ready Compose setup with full inline docume | File | Purpose | |---|---| -| `docker-compose.yml` | Gateway service, volumes, and environment variables | +| `docker-compose.yml` | Gateway and init services, volumes, and environment variables | | `gateway.toml` | TOML configuration mounted into the container | Clone or copy those files, then start the gateway: @@ -165,6 +165,8 @@ Clone or copy those files, then start the gateway: docker compose -f deploy/docker/docker-compose.yml up -d ``` +The `init` service runs first and writes the sandbox JWT signing keys to `/var/lib/openshell/tls` on the host. The Docker driver requires them, and existing keys are kept. Do not commit them. This setup disables TLS and user authentication and publishes the port on loopback only. + Register the gateway with the CLI. If registering from the same machine: ```shell @@ -178,6 +180,30 @@ machine's LAN address: openshell gateway add http://HOST_IP:8080 --remote --name remote ``` +### Enable mTLS + +The `init` service already writes the server, client, and CA certificates to `/var/lib/openshell/tls`, so enabling mTLS only needs configuration changes: + +1. In `gateway.toml`, remove `disable_tls = true` and the `[openshell.gateway.auth]` table. +2. In `docker-compose.yml`, add `OPENSHELL_LOCAL_TLS_DIR: /var/lib/openshell/tls` under the `gateway` service `environment`. The gateway loads its server certificate, client CA, and sandbox client bundle from that directory, and turns on mTLS user authentication. +3. Apply the change: + + ```shell + docker compose -f deploy/docker/docker-compose.yml up -d + ``` + +Then copy the client bundle to `~/.config/openshell/gateways/local/mtls/`, where the CLI loads it (refer to [Gateway Authentication](/how-it-works/gateways/authentication#mtls)). The files are owned by root, so copy them with `sudo` and hand them to your user. If you registered the plaintext gateway as `local`, run `openshell gateway remove local` first. + +```shell +mtls=~/.config/openshell/gateways/local/mtls +mkdir -p "$mtls" +sudo cp /var/lib/openshell/tls/ca.crt \ + /var/lib/openshell/tls/client/tls.crt \ + /var/lib/openshell/tls/client/tls.key "$mtls"/ +sudo chown "$USER" "$mtls"/* +openshell gateway add https://127.0.0.1:8080 --local --name local +``` + ## Using Podman Replace `docker` with `podman` in the commands above. Mount the Podman socket instead of the Docker socket and set the driver to `podman`: diff --git a/docs/how-it-works/providers/profiles.mdx b/docs/how-it-works/providers/profiles.mdx index 11c5438d3c..7a68c6aec3 100644 --- a/docs/how-it-works/providers/profiles.mdx +++ b/docs/how-it-works/providers/profiles.mdx @@ -555,7 +555,45 @@ The refresh attaches to the primary credential (`access_key_id`). Each reference ### Dynamic Token Grants -`token_grant` belongs to one credential declaration. When a sandbox with the provider attached sends HTTP traffic to a matching profile endpoint, the supervisor resolves the dynamic credential, caches the returned access token, and injects it before forwarding the request upstream. Use `auth_style: bearer` to inject `Authorization: Bearer `, or `auth_style: header` with `header_name` to inject the raw access token into a custom header. Token grants do not support `query` or `path` placement. +`token_grant` belongs to one credential declaration. When a sandbox with the provider attached sends HTTP traffic to a matching profile endpoint, the supervisor resolves the dynamic credentials, caches each returned access token separately, and injects them before forwarding the request upstream. Use `auth_style: bearer` to inject `Authorization: Bearer `, or `auth_style: header` with `header_name` to inject the raw access token into a custom header. Token grants do not support `query` or `path` placement. + +Declare multiple credentials with different destination headers when one request needs independently issued tokens. Each credential keeps its own token endpoint, audience, scopes, and cache lifetime. For example, this profile sends a service bearer token and a separate workload token on the same inspected request: + +```yaml +id: protected-api +display_name: Protected API +category: data +credentials: + - name: service_access + required: true + auth_style: bearer + header_name: Authorization + token_grant: + token_endpoint: https://service-issuer.example.com/token + audience: https://api.example.com + scopes: [resource.read] + cache_ttl: 60s + - name: workload_identity + required: true + auth_style: header + header_name: x-workload-jwt + token_grant: + token_endpoint: https://identity-issuer.example.com/token + jwt_svid_audience: identity-proxy + cache_ttl: 45s +endpoints: + - host: api.example.com + port: 443 + path: /v1/** + protocol: rest + access: full +``` + +The custom header carries the identity issuer's returned access token. To carry a JWT-SVID there, that issuer must return the JWT-SVID as its access token. Neither granted value is placed in the sandbox environment. + +The supervisor chooses the most-specific matching binding independently for each destination header. Bindings that target the same header are alternatives; equal-specificity matches from different credentials are rejected. Header names are case-insensitive. Omitting both `auth_style` and `header_name` selects `Authorization`; an explicit `bearer` or `header` style requires `header_name`. A more-specific audience override for one credential does not suppress grants for other headers. + +All selected grants must succeed and return valid header values before the request is rewritten. A failed grant returns HTTP 502 without forwarding the request. An earlier successful grant may remain in its own cache for a later attempt. Injection replaces every agent-supplied instance of each selected header, including any static credential placeholder in that header, and preserves unrelated headers. Issuer error descriptions are excluded from grant-failure logs and client errors because they may contain credentials. OpenShell supports two dynamic grant types: @@ -1080,7 +1118,7 @@ For a static provider, the sequence is attach, wait, then launch client A; updat An acknowledged detachment removes its provider policy layer from the active effective policy, revokes future resolution for its existing placeholders, and removes its credential placeholders from future process environments. It does not remove strings from already-running process environments or undo requests already forwarded upstream. -OpenShell rejects provider updates and refresh configuration when they would make two providers attached to the same sandbox expose the same active credential environment key. It also rejects attached provider sets with ambiguous dynamic token grants at equal host/path specificity. Use provider-specific credential names and make one dynamic grant selector more specific when one sandbox needs multiple providers with overlapping upstream concepts. +OpenShell rejects provider updates and refresh configuration when they would make two providers attached to the same sandbox expose the same active credential environment key. It also rejects attached provider sets whose dynamic token grants target the same header at equal host/path specificity. Grants for distinct headers can share an endpoint. Use provider-specific credential names and make one same-header selector more specific when the bindings are intended as alternatives. ## Next Steps diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index c6189b1ebb..46a1b4812a 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -19,20 +19,11 @@ process to install a nested seccomp user-notification filter and use Landlock. OpenShell fails sandbox startup when either capability-free runtime probe fails. -## Node kernel and legacy read-only mode - -OpenShift nodes run RHCOS, which currently ships a RHEL 9.x kernel (5.14). That -kernel predates `SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` (Linux 5.19), so the -sandbox starts in a reduced **legacy read-only** cancellation mode. Isolation is -unchanged, but the broker fails closed with `EOPNOTSUPP` on the mediated -operations that write results back into workload memory β€” `getpeername`, -`accept`/`accept4` with a non-null peer-address argument, and `sendmmsg` -per-message length write-backs. Outbound-oriented workloads run unchanged; -server workloads that read the peer address on accept need a node kernel with -`WAIT_KILLABLE_RECV` (Linux 5.19+, or a distribution backport). See the -[support matrix](/about/support-matrix#legacy-read-only-mode-kernels-before-linux-519) -for the full behavior; the selected mode is reported as `seccomp_listener_mode` -in the sandbox qualification output. +## Node kernel + +OpenShell requires OpenShift 4.19 or later. The RHCOS kernels in OpenShift 4.16 +through 4.18 (RHEL 9.4, 5.14.0-427) are built without Landlock, so sandbox +startup fails its Landlock probe on those releases. ## Prerequisites @@ -73,6 +64,19 @@ oc -n openshell rollout status statefulset/openshell If you set `workload.kind=deployment`, wait for `deployment/openshell` instead. +## Managed workspace namespaces + +In managed workspace mode, the Kubernetes driver creates one namespace per +workspace. On OpenShift, the driver leaves SCC annotations unset so the cluster's +namespace allocator can assign each workspace its own MCS, UID range, and +supplemental-group range. The driver waits for those annotations before it +provisions the workspace ServiceAccount or sandbox Pods. + +Namespaces created by an earlier version may have a UID range but no MCS +annotation. OpenShift does not allocate a new MCS for a namespace that already +has a UID range, so preserve any needed workspace data and recreate that +workspace namespace before retrying sandbox creation. + ## Connect to the Gateway Forward the gateway port for local evaluation: diff --git a/docs/security/best-practices.mdx b/docs/security/best-practices.mdx index 2f5b346708..e70f38ecc4 100644 --- a/docs/security/best-practices.mdx +++ b/docs/security/best-practices.mdx @@ -220,7 +220,7 @@ OpenShell applies seccomp in two phases. A narrow supervisor-startup prelude run | Aspect | Detail | |---|---| | Startup prelude | After privileged bootstrap helpers complete, including network setup and provider-token SPIFFE child mount-namespace preparation, the supervisor sets `PR_SET_NO_NEW_PRIVS` and synchronizes a seccomp filter across all runtime threads that blocks `mount`, the new mount API syscalls, `pivot_root`, `umount2`, `bpf`, `perf_event_open`, `userfaultfd`, module-loading syscalls, and kexec. This closes the long-lived privileged remount and kernel-surface window while leaving required setup syscalls such as `setns` available. | -| Socket domains | The filter allows `AF_INET` and `AF_INET6` (for proxy communication) and blocks `AF_PACKET`, `AF_BLUETOOTH`, and `AF_VSOCK` with `EPERM`. `AF_NETLINK` is partially allowed: only `NETLINK_ROUTE` (protocol 0) is permitted so that `getifaddrs(3)` works; all other netlink protocols are blocked. Write operations via `NETLINK_ROUTE` still require `CAP_NET_ADMIN`, which the sandbox does not grant. | +| Socket domains | The filter allows only `AF_UNIX`, `AF_NETLINK`, and the brokered `AF_INET` and `AF_INET6` families. Every other family, including `AF_PACKET`, `AF_VSOCK`, `AF_RXRPC`, `AF_SMC`, `AF_KCM`, and `AF_ALG`, fails with `EPERM`, and `socketpair(2)` is limited to `AF_UNIX`. `AF_NETLINK` is partially allowed: only `NETLINK_ROUTE` (protocol 0) is permitted so that `getifaddrs(3)` works; all other netlink protocols are blocked. Write operations via `NETLINK_ROUTE` still require `CAP_NET_ADMIN`, which the sandbox does not grant. | | Runtime unconditional syscall blocks | `memfd_create`, `ptrace`, `bpf`, `process_vm_readv`, `process_vm_writev`, `pidfd_open`, `pidfd_getfd`, `pidfd_send_signal`, `io_uring_setup`, `mount`, `fsopen`, `fsconfig`, `fsmount`, `fspick`, `move_mount`, `open_tree`, `setns`, `umount2`, `pivot_root`, `userfaultfd`, `perf_event_open`. | | Conditional syscall blocks | `execveat` with `AT_EMPTY_PATH`, `unshare` and `clone` with `CLONE_NEWUSER`, and `seccomp(SECCOMP_SET_MODE_FILTER)` are denied with `EPERM`. | | What you can change | This is not a user-facing knob. OpenShell enforces it automatically. | diff --git a/e2e/rust/src/harness/gateway.rs b/e2e/rust/src/harness/gateway.rs index e68d88c2b7..edb6f4570a 100644 --- a/e2e/rust/src/harness/gateway.rs +++ b/e2e/rust/src/harness/gateway.rs @@ -21,6 +21,7 @@ pub struct ManagedGateway { args_file: PathBuf, log: PathBuf, pid_file: PathBuf, + supervisor_image: Option, } impl ManagedGateway { @@ -36,6 +37,7 @@ impl ManagedGateway { args_file: args_file.into(), log: log.into(), pid_file: pid_file.into(), + supervisor_image: None, } } @@ -53,9 +55,15 @@ impl ManagedGateway { args_file: env_path("OPENSHELL_E2E_GATEWAY_ARGS_FILE")?, log: env_path("OPENSHELL_E2E_GATEWAY_LOG")?, pid_file: env_path("OPENSHELL_E2E_GATEWAY_PID_FILE")?, + supervisor_image: None, })) } + /// Override the supervisor image for gateways started by this handle. + pub fn set_supervisor_image(&mut self, image: &str) { + self.supervisor_image = Some(image.to_owned()); + } + /// Start the gateway if it is not already running. pub fn start(&self) -> Result<(), String> { if let Some(pid) = self.current_pid()? { @@ -80,10 +88,15 @@ impl ManagedGateway { .try_clone() .map_err(|err| format!("clone gateway log handle: {err}"))?; - let child = Command::new(&self.bin) + let mut command = Command::new(&self.bin); + command .args(args) .stdout(Stdio::from(log)) - .stderr(Stdio::from(stderr)) + .stderr(Stdio::from(stderr)); + if let Some(image) = &self.supervisor_image { + command.env("OPENSHELL_SUPERVISOR_IMAGE", image); + } + let child = command .spawn() .map_err(|err| format!("start openshell-gateway '{}': {err}", self.bin.display()))?; let pid = child.id(); diff --git a/e2e/rust/src/harness/sandbox.rs b/e2e/rust/src/harness/sandbox.rs index 9922db02b4..0ebfde9081 100644 --- a/e2e/rust/src/harness/sandbox.rs +++ b/e2e/rust/src/harness/sandbox.rs @@ -618,7 +618,11 @@ impl SandboxGuard { for arg in argv { cmd.arg(arg); } - cmd.stdout(Stdio::piped()).stderr(Stdio::piped()); + // Never share the test runner's stdin: parallel test processes share + // its open file description, and the command needs no input. + cmd.stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); let output = cmd .output() diff --git a/e2e/rust/tests/podman_oci_identity.rs b/e2e/rust/tests/podman_oci_identity.rs index 3405d730c9..50aaba399f 100644 --- a/e2e/rust/tests/podman_oci_identity.rs +++ b/e2e/rust/tests/podman_oci_identity.rs @@ -244,6 +244,23 @@ async fn podman_uses_oci_identity_and_inspected_image_id() { "Podman sandbox must launch the immutable image ID inspected before creation" ); + let workspace_output = sandbox + .exec(&[ + "sh", + "-c", + "set -eu; stat -c 'workspace-owner=%u:%g' /sandbox; touch /sandbox/probe; rm /sandbox/probe; echo podman-workspace-write-ok", + ]) + .await + .expect("OCI workload should be able to write to the managed workspace"); + assert!( + workspace_output.contains(&format!("workspace-owner={OCI_UID}:{OCI_GID}")), + "expected workspace owner {OCI_UID}:{OCI_GID}:\n{workspace_output}" + ); + assert!( + workspace_output.contains("podman-workspace-write-ok"), + "expected workspace write marker:\n{workspace_output}" + ); + assert_isolated_pair(&image, &sandbox, &container_id).await; sandbox.cleanup().await; } @@ -258,8 +275,9 @@ async fn assert_isolated_pair(image: &ImageGuard, sandbox: &SandboxGuard, contai ) .unwrap(); assert_eq!( - workload_user, "0:0", - "the trusted rootless boundary starts as container root before dropping to the OCI identity" + workload_user, + format!("{OCI_UID}:{OCI_GID}"), + "the workload must start directly as the final OCI identity" ); let supervisor_user = run_engine( &image.engine, diff --git a/e2e/rust/tests/provider_readiness.rs b/e2e/rust/tests/provider_readiness.rs index 8ec6398a2b..b7d1256641 100644 --- a/e2e/rust/tests/provider_readiness.rs +++ b/e2e/rust/tests/provider_readiness.rs @@ -212,16 +212,14 @@ impl FixtureImage { // This fixture owns the only test in its binary. The wrapper's gateway is // private to this run, so replacing its supervisor image cannot affect another // test while the public fixture CA is installed in the supervisor trust store. -struct GatewayTrustConfig { - path: PathBuf, - original: String, - image_range: std::ops::Range, +struct GatewayTrustFixture { + directory: PathBuf, supervisor_image: String, health_port: u16, restore_required: bool, } -impl GatewayTrustConfig { +impl GatewayTrustFixture { fn load() -> Result { if std::env::var_os("OPENSHELL_GATEWAY_ENDPOINT").is_some() || std::env::var_os("OPENSHELL_E2E_GATEWAY_BIN").is_none() @@ -233,6 +231,10 @@ impl GatewayTrustConfig { } let args_file = std::env::var_os("OPENSHELL_E2E_GATEWAY_ARGS_FILE") .ok_or("managed gateway argument metadata is missing")?; + let directory = Path::new(&args_file) + .parent() + .ok_or("managed gateway arguments have no parent directory")? + .to_path_buf(); let raw = std::fs::read(args_file).map_err(|_| "could not read managed gateway arguments")?; let args = raw @@ -249,17 +251,13 @@ impl GatewayTrustConfig { } Ok(value) }; - let path = PathBuf::from(argument("--config")?); let health_port = argument("--health-port")? .parse::() .map_err(|_| "managed gateway health port is invalid")?; - let original = std::fs::read_to_string(&path) - .map_err(|_| "could not read managed gateway configuration")?; - let (image_range, supervisor_image) = docker_supervisor_image(&original)?; + let supervisor_image = std::env::var("OPENSHELL_SUPERVISOR_IMAGE") + .map_err(|_| "managed supervisor image is missing")?; Ok(Self { - path, - original, - image_range, + directory, supervisor_image, health_port, restore_required: false, @@ -267,94 +265,32 @@ impl GatewayTrustConfig { } async fn apply(&mut self, image: &str) -> Result<(), String> { - let mut updated = self.original.clone(); - updated.replace_range(self.image_range.clone(), image); - // Set the guard before the write: a failed write or restart must still - // flow through explicit restoration of the exact original bytes. self.restore_required = true; - std::fs::write(&self.path, updated) - .map_err(|_| "could not install fixture supervisor configuration")?; - restart_fixture_gateway(self.health_port).await + restart_fixture_gateway(self.health_port, Some(image)).await } async fn restore(&mut self) -> Result<(), String> { if !self.restore_required { return Ok(()); } - std::fs::write(&self.path, &self.original) - .map_err(|_| "could not restore original gateway configuration")?; - restart_fixture_gateway(self.health_port) + restart_fixture_gateway(self.health_port, None) .await - .map_err(|_| "original gateway configuration was restored but restart failed")?; + .map_err(|_| "could not restore original gateway runtime")?; self.restore_required = false; Ok(()) } } -impl Drop for GatewayTrustConfig { - fn drop(&mut self) { - if self.restore_required { - // Cancellation/panic fallback restores disk state only. Normal - // Result paths explicitly restart and verify health; Drop never - // launches a subprocess or hides a failed restart as success. - let _ = std::fs::write(&self.path, &self.original); - } - } -} - -fn docker_supervisor_image(config: &str) -> Result<(std::ops::Range, String), String> { - let mut in_docker = false; - let mut offset = 0; - let mut found = None; - for line in config.split_inclusive('\n') { - let trimmed = line.trim(); - if trimmed.starts_with('[') { - in_docker = trimmed == "[openshell.drivers.docker]"; - } else if in_docker && let Some((key, value)) = trimmed.split_once('=') { - if key.trim() == "socket_path" { - return Err( - "fixture cannot replace an external Docker driver configuration".to_string(), - ); - } - if key.trim() == "supervisor_image" { - // Accept only the wrapper's single-line quoted OCI reference. - // Reject escapes/comments instead of treating general TOML as - // text and accidentally changing a different configuration key. - let image = value - .trim() - .strip_prefix('"') - .and_then(|value| value.strip_suffix('"')) - .filter(|image| { - !image.is_empty() - && image.bytes().all(|byte| { - byte.is_ascii_alphanumeric() || b"/:@._-".contains(&byte) - }) - }) - .ok_or("managed supervisor image is not a simple quoted OCI reference")?; - let start = offset - + line - .find('"') - .ok_or("managed supervisor image is not quoted")? - + 1; - if found - .replace((start..start + image.len(), image.to_string())) - .is_some() - { - return Err("managed Docker supervisor image is duplicated".to_string()); - } - } - } - offset += line.len(); - } - found.ok_or_else(|| "managed Docker supervisor image is missing".to_string()) -} - -async fn restart_fixture_gateway(health_port: u16) -> Result<(), String> { - let gateway = ManagedGateway::from_env() +async fn restart_fixture_gateway( + health_port: u16, + supervisor_image: Option<&str>, +) -> Result<(), String> { + let mut gateway = ManagedGateway::from_env() .map_err(|_| "could not load managed gateway restart metadata")? .ok_or("managed gateway restart metadata disappeared")?; - // ManagedGateway bounds graceful shutdown before force-kill. Keep it local: - // its Drop can start a stopped gateway, but never owns configuration restore. + if let Some(image) = supervisor_image { + gateway.set_supervisor_image(image); + } gateway .stop() .map_err(|_| "could not stop fixture gateway")?; @@ -1298,19 +1234,15 @@ fn check( #[allow(clippy::too_many_lines)] async fn acknowledged_provider_changes_apply_to_fresh_clients_and_revoke_retained_references() -> Result<(), String> { - let mut gateway_config = GatewayTrustConfig::load()?; + let mut gateway_fixture = GatewayTrustFixture::load()?; // Sandbox names are limited to 19 characters. Retain all 64 random bits // within that limit so concurrent fixtures still own distinct resources. let name = format!("e2e{:016x}", rand::random::()); let mut backend = BackendPair::new(&name)?; // The wrapper's directory is shared with the host Docker daemon in CI; // a job-container-local temporary path cannot back the TLS bind mount. - let fixture_parent = gateway_config - .path - .parent() - .ok_or("managed gateway configuration has no parent directory")?; - let directory = - TempDir::new_in(fixture_parent).map_err(|_| "could not allocate fixture directory")?; + let directory = TempDir::new_in(&gateway_fixture.directory) + .map_err(|_| "could not allocate fixture directory")?; let context = directory.path().join("image"); std::fs::create_dir(&context).map_err(|_| "could not allocate public image context")?; let backend_tls = directory.path().join("backend-tls"); @@ -1382,9 +1314,9 @@ async fn acknowledged_provider_changes_apply_to_fresh_clients_and_revoke_retaine // user to the supervisor's private bootstrap files. std::fs::write(&supervisor_dockerfile, format!( "FROM {} AS supervisor\nFROM {} AS trust-bundle\nUSER 0\nCOPY --from=supervisor /etc/ssl/certs/ca-certificates.crt /tmp/ca-certificates.crt\nCOPY fixture-ca.crt /tmp/readiness-fixture-ca.crt\nRUN [\"/usr/bin/python3\", \"-c\", \"from pathlib import Path; bundle = Path('/tmp/ca-certificates.crt'); bundle.write_bytes(bundle.read_bytes() + Path('/tmp/readiness-fixture-ca.crt').read_bytes())\"]\nFROM {}\nCOPY --from=trust-bundle /tmp/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt\n", - gateway_config.supervisor_image, + gateway_fixture.supervisor_image, image.tag(), - gateway_config.supervisor_image, + gateway_fixture.supervisor_image, )).map_err(|_| "could not write fixture supervisor Dockerfile")?; supervisor_image .build( @@ -1393,7 +1325,7 @@ async fn acknowledged_provider_changes_apply_to_fresh_clients_and_revoke_retaine "build supervisor fixture image", ) .await?; - gateway_config.apply(supervisor_image.tag()).await?; + gateway_fixture.apply(supervisor_image.tag()).await?; let profile = directory.path().join("profile.json"); let policy = directory.path().join("policy.json"); write_profile(&profile, &name, &host, port, python)?; @@ -1639,7 +1571,7 @@ async fn acknowledged_provider_changes_apply_to_fresh_clients_and_revoke_retaine // Restore the original runtime before removing its replacement. Retain the // derived supervisor image if restoration fails, and report that failure // even when a lifecycle assertion already failed. - let gateway_restore = gateway_config.restore().await; + let gateway_restore = gateway_fixture.restore().await; let supervisor_cleanup = if gateway_restore.is_ok() { supervisor_image.remove().await } else { diff --git a/e2e/rust/tests/provider_token_exchange.rs b/e2e/rust/tests/provider_token_exchange.rs index e845938e70..97d8e2c587 100644 --- a/e2e/rust/tests/provider_token_exchange.rs +++ b/e2e/rust/tests/provider_token_exchange.rs @@ -11,20 +11,20 @@ use std::net::{Ipv4Addr, SocketAddr}; use std::os::unix::fs::PermissionsExt as _; use std::path::{Path, PathBuf}; use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; use std::task::{Context, Poll}; use std::time::{Duration, SystemTime, UNIX_EPOCH}; use base64::Engine as _; use futures_util::future::BoxFuture; -use jsonwebtoken::{Algorithm, EncodingKey, Header}; +use jsonwebtoken::{Algorithm, DecodingKey, EncodingKey, Header, Validation}; use openshell_e2e::harness::binary::openshell_cmd; use openshell_e2e::harness::port::find_free_port; use openshell_e2e::harness::sandbox::SandboxGuard; use serde_json::json; use tempfile::NamedTempFile; use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; -use tokio::net::{TcpListener, UnixListener}; -use tokio::process::Command; +use tokio::net::{TcpListener, TcpStream, UnixListener}; use tokio_stream::wrappers::{ReceiverStream, TcpListenerStream, UnixListenerStream}; use tonic::body::Body as TonicBody; use tonic::codegen::{Body, http}; @@ -38,6 +38,30 @@ const INTERMEDIATE_TOKEN: &str = "intermediate-token"; const FINAL_ACCESS_TOKEN: &str = "final-access-token"; const TOKEN_TYPE_ACCESS_TOKEN: &str = "urn:ietf:params:oauth:token-type:access_token"; const CLIENT_ASSERTION_TYPE: &str = "urn:ietf:params:oauth:client-assertion-type:jwt-spiffe"; +const IDENTITY_AUDIENCE: &str = "identity-proxy"; +const IDENTITY_JWT_AUDIENCE: &str = "https://identity.openshell-e2e.test"; +const IDENTITY_ASSERTION_TYPE: &str = "urn:ietf:params:oauth:client-assertion-type:jwt-bearer"; + +#[derive(Default)] +struct GrantObservations { + service_issued: [AtomicUsize; 2], + identity_issued: [AtomicUsize; 2], + service_denied: AtomicUsize, + identity_denied: AtomicUsize, + target_requests: AtomicUsize, + target_rejected: AtomicUsize, +} + +impl GrantObservations { + fn issued(&self) -> [(usize, usize); 2] { + std::array::from_fn(|index| { + ( + self.service_issued[index].load(Ordering::SeqCst), + self.identity_issued[index].load(Ordering::SeqCst), + ) + }) + } +} const TEST_RSA_PRIVATE_KEY: &str = r"-----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCvCoZ0mVHpCHsF @@ -118,7 +142,7 @@ struct SpiffeWorkloadApi { } impl SpiffeWorkloadApi { - fn jwt_svid(&self, audience: Vec) -> Result { + fn jwt_svid(&self, audience: &[String]) -> Result { let now = unix_timestamp(); let mut header = Header::new(Algorithm::RS256); header.kid = Some(KEY_ID.to_string()); @@ -173,7 +197,7 @@ where let inner = Arc::clone(&self.0); Box::pin(async move { let request = request.into_inner(); - let svid = inner.jwt_svid(request.audience)?; + let svid = inner.jwt_svid(&request.audience)?; Ok(Response::new(JwtsvidResponse { svids: vec![Jwtsvid { spiffe_id: inner.subject.to_string(), @@ -326,7 +350,81 @@ async fn start_spiffe_workload_api(path: &Path, subject: &str) -> FixtureHandle FixtureHandle { task } } -async fn start_gateway_token_endpoint(port: u16) -> FixtureHandle { +// Forms and JWT headers can span TCP reads. Read the bounded, Content-Length +// framed request completely so packet boundaries cannot change fixture results. +async fn read_http_request(stream: &mut TcpStream) -> Option { + tokio::time::timeout(Duration::from_secs(5), async { + let mut request = Vec::new(); + let mut buffer = [0_u8; 4096]; + loop { + let length = stream.read(&mut buffer).await.ok()?; + if length == 0 || request.len() + length > 32 * 1024 { + return None; + } + request.extend_from_slice(&buffer[..length]); + if let Some(end) = request.windows(4).position(|bytes| bytes == b"\r\n\r\n") { + let headers = std::str::from_utf8(&request[..end]).ok()?; + let content_length = headers + .lines() + .filter_map(|line| line.split_once(':')) + .find(|(name, _)| name.eq_ignore_ascii_case("content-length")) + .map_or(Some(0), |(_, value)| value.trim().parse::().ok())?; + if request.len() >= end.checked_add(4)?.checked_add(content_length)? { + return String::from_utf8(request).ok(); + } + } + } + }) + .await + .ok() + .flatten() +} + +fn verifies_jwt_svid(token: &str, instance: &str) -> bool { + let keys: jsonwebtoken::jwk::JwkSet = + serde_json::from_slice(&jwks()).expect("fixture JWKS should parse"); + let key = DecodingKey::from_jwk(keys.find(KEY_ID).expect("fixture signing key exists")) + .expect("fixture verification key should parse"); + let mut validation = Validation::new(Algorithm::RS256); + validation.set_issuer(&[ISSUER]); + validation.set_audience(&[format!("{IDENTITY_JWT_AUDIENCE}/{instance}")]); + validation.sub = Some(format!("spiffe://{TRUST_DOMAIN}/openshell/sandbox/e2e")); + validation.set_required_spec_claims(&["exp", "iat", "iss", "sub", "aud"]); + jsonwebtoken::decode::(token, &key, &validation) + .is_ok_and(|verified| verified.header.kid.as_deref() == Some(KEY_ID)) +} + +fn token_form(request: &str) -> HashMap { + let (_, body) = request.split_once("\r\n\r\n").unwrap_or_default(); + url::form_urlencoded::parse(body.as_bytes()) + .into_owned() + .collect() +} + +async fn write_token_response(stream: &mut TcpStream, access_token: Option<&str>, ttl: u64) { + let (status, body) = if let Some(access_token) = access_token { + ( + "HTTP/1.1 200 OK", + json!({"access_token": access_token, "token_type": "Bearer", "expires_in": ttl}) + .to_string(), + ) + } else { + ( + "HTTP/1.1 400 Bad Request", + json!({"error": "invalid_grant"}).to_string(), + ) + }; + let response = format!( + "{status}\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}", + body.len() + ); + let _ = stream.write_all(response.as_bytes()).await; +} + +async fn start_gateway_token_endpoint( + port: u16, + observations: Arc, +) -> FixtureHandle { let listener = TcpListener::bind(SocketAddr::from((Ipv4Addr::LOCALHOST, port))) .await .expect("bind gateway token endpoint"); @@ -335,51 +433,91 @@ async fn start_gateway_token_endpoint(port: u16) -> FixtureHandle { let Ok((mut stream, _peer)) = listener.accept().await else { break; }; + let observations = Arc::clone(&observations); tokio::spawn(async move { - let mut buf = vec![0_u8; 8192]; - let n = stream.read(&mut buf).await.unwrap_or(0); - let request = String::from_utf8_lossy(&buf[..n]); - let access_token = if request.starts_with("POST /token ") - && request.contains("subject_token=stored-user-token") - && request.contains("client_assertion=") - { - Some(INTERMEDIATE_TOKEN) - } else if request.starts_with("POST /token ") - && request.contains("subject_token=intermediate-token") - && request.contains("client_assertion=") - { - Some(FINAL_ACCESS_TOKEN) - } else { - None + let Some(request) = read_http_request(&mut stream).await else { + return; }; - let (status, body) = if let Some(access_token) = access_token { - ( - "HTTP/1.1 200 OK", - json!({ - "access_token": access_token, - "token_type": "Bearer", - "expires_in": 300 - }) - .to_string(), - ) - } else { - ( - "HTTP/1.1 400 Bad Request", - json!({"error": "unexpected_token_exchange"}).to_string(), - ) + let form = token_form(&request); + let field = |name: &str| form.get(name).map_or("", String::as_str); + let valid = request.starts_with("POST /token ") + && field("grant_type") == "urn:ietf:params:oauth:grant-type:token-exchange" + && field("client_assertion_type") == CLIENT_ASSERTION_TYPE + && field("subject_token_type") == TOKEN_TYPE_ACCESS_TOKEN + && field("requested_token_type") == TOKEN_TYPE_ACCESS_TOKEN + && !field("client_assertion").is_empty(); + let mut access_token = None; + for (index, instance) in ["a", "b"].into_iter().enumerate() { + if valid + && field("subject_token") == format!("{USER_SUBJECT_TOKEN}-{instance}") + && field("audience") + == format!("spiffe://{TRUST_DOMAIN}/openshell/sandbox/e2e") + && field("scope").is_empty() + { + access_token = Some(format!("{INTERMEDIATE_TOKEN}-{instance}")); + } else if valid + && field("subject_token") == format!("{INTERMEDIATE_TOKEN}-{instance}") + && field("scope") == "service.read" + { + if field("audience") == format!("service-{instance}") { + observations.service_issued[index].fetch_add(1, Ordering::SeqCst); + access_token = Some(format!("{FINAL_ACCESS_TOKEN}-{instance}")); + } else if field("audience") == format!("denied-service-{instance}") { + observations.service_denied.fetch_add(1, Ordering::SeqCst); + } + } + } + write_token_response(&mut stream, access_token.as_deref(), 600).await; + }); + } + }); + FixtureHandle { task } +} + +// This issuer returns the actual Workload API JWT-SVID after checking its +// signature and audience. No new production grant type or sandbox-visible +// credential is needed to exercise custom-header injection. +async fn start_identity_token_endpoint( + port: u16, + observations: Arc, +) -> FixtureHandle { + let listener = TcpListener::bind(SocketAddr::from((Ipv4Addr::LOCALHOST, port))) + .await + .expect("bind identity token endpoint"); + let task = tokio::spawn(async move { + loop { + let Ok((mut stream, _peer)) = listener.accept().await else { + break; + }; + let observations = Arc::clone(&observations); + tokio::spawn(async move { + let Some(request) = read_http_request(&mut stream).await else { + return; }; - let response = format!( - "{status}\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}", - body.len() - ); - let _ = stream.write_all(response.as_bytes()).await; + let form = token_form(&request); + let field = |name: &str| form.get(name).map_or("", String::as_str); + let mut access_token = None; + for (index, instance) in ["a", "b"].into_iter().enumerate() { + let valid = request.starts_with("POST /identity-token ") + && field("grant_type") == "client_credentials" + && field("client_assertion_type") == IDENTITY_ASSERTION_TYPE + && field("scope") == "identity.read" + && verifies_jwt_svid(field("client_assertion"), instance); + if valid && field("audience") == format!("{IDENTITY_AUDIENCE}-{instance}") { + observations.identity_issued[index].fetch_add(1, Ordering::SeqCst); + access_token = Some(field("client_assertion")); + } else if valid && field("audience") == format!("denied-identity-{instance}") { + observations.identity_denied.fetch_add(1, Ordering::SeqCst); + } + } + write_token_response(&mut stream, access_token, 900).await; }); } }); FixtureHandle { task } } -async fn start_protected_target(port: u16) -> FixtureHandle { +async fn start_protected_target(port: u16, observations: Arc) -> FixtureHandle { let listener = TcpListener::bind(SocketAddr::from((Ipv4Addr::UNSPECIFIED, port))) .await .expect("bind protected target"); @@ -388,19 +526,49 @@ async fn start_protected_target(port: u16) -> FixtureHandle { let Ok((mut stream, _peer)) = listener.accept().await else { break; }; + let observations = Arc::clone(&observations); tokio::spawn(async move { - let mut buf = vec![0_u8; 8192]; - let n = stream.read(&mut buf).await.unwrap_or(0); - let request = String::from_utf8_lossy(&buf[..n]); - let ok = request.lines().any(|line| { - line.eq_ignore_ascii_case(&format!( - "authorization: Bearer {FINAL_ACCESS_TOKEN}" - )) + // Count any request bytes, including an incomplete header. A + // failed grant may open a TCP stream but must send no request. + let mut byte = [0_u8; 1]; + if !matches!( + tokio::time::timeout(Duration::from_secs(5), stream.peek(&mut byte)).await, + Ok(Ok(1)) + ) { + return; + } + observations.target_requests.fetch_add(1, Ordering::SeqCst); + let Some(request) = read_http_request(&mut stream).await else { + return; + }; + let headers = request.split("\r\n\r\n").next().unwrap_or_default(); + let header_values = |name: &str| { + headers + .lines() + .filter_map(|line| line.split_once(':')) + .filter(|(key, _)| key.eq_ignore_ascii_case(name)) + .map(|(_, value)| value.trim()) + .collect::>() + }; + let bearer = header_values("authorization"); + let identity = header_values("x-workload-jwt"); + let ok = ["a", "b"].into_iter().any(|instance| { + request.starts_with(&format!("GET /resource/{instance} ")) + && bearer.len() == 1 + && bearer[0] == format!("Bearer {FINAL_ACCESS_TOKEN}-{instance}") + && identity.len() == 1 + && verifies_jwt_svid(identity[0], instance) }); let (status, body) = if ok { - ("HTTP/1.1 200 OK", "token-exchange-ok") + ("HTTP/1.1 200 OK", "independent-grants-ok") } else { - ("HTTP/1.1 401 Unauthorized", "missing-final-token") + // Keep rejection evidence even if the workload retries and + // a later request happens to receive the correct credentials. + observations.target_rejected.fetch_add(1, Ordering::SeqCst); + ( + "HTTP/1.1 401 Unauthorized", + "credential-verification-failed", + ) }; let response = format!( "{status}\r\ncontent-type: text/plain\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}", @@ -413,21 +581,19 @@ async fn start_protected_target(port: u16) -> FixtureHandle { FixtureHandle { task } } -async fn run_cli(args: &[&str]) -> Result { +async fn run_cli(args: &[&str]) -> Result<(), String> { let output = openshell_cmd() .args(args) .output() .await - .map_err(|err| format!("spawn openshell: {err}"))?; - let stdout = String::from_utf8_lossy(&output.stdout); - let stderr = String::from_utf8_lossy(&output.stderr); - let combined = format!("{stdout}{stderr}"); + .map_err(|_| "could not spawn openshell CLI".to_string())?; if output.status.success() { - Ok(combined) + Ok(()) } else { + // Provider creation arguments contain the stored subject token. + // Keep command arguments and raw diagnostics out of test failures. Err(format!( - "openshell {:?} failed with {:?}:\n{combined}", - args, + "openshell command failed; exit={:?}", output.status.code() )) } @@ -437,104 +603,13 @@ async fn run_cli_ignore_error(args: &[&str]) { let _ = openshell_cmd().args(args).output().await; } -async fn sandbox_logs(sandbox_name: &str) -> String { - run_cli(&["logs", sandbox_name]) - .await - .unwrap_or_else(|err| format!("failed to collect sandbox logs: {err}")) -} - -async fn podman_exec_capture(container_name: &str, args: &[&str]) -> String { - let Ok(socket) = std::env::var("OPENSHELL_PODMAN_SOCKET") else { - return "OPENSHELL_PODMAN_SOCKET is not set".to_string(); - }; - let mut cmd = Command::new("podman"); - cmd.arg("--url") - .arg(format!("unix://{socket}")) - .arg("exec") - .arg(container_name) - .args(args); - apply_podman_config_env(&mut cmd); - match cmd.output().await { - Ok(output) => format!( - "exit={:?}\nstdout:\n{}\nstderr:\n{}", - output.status.code(), - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - ), - Err(err) => format!("failed to run podman exec {:?}: {err}", args), - } -} - -async fn podman_logs_capture(container_name: &str) -> String { - let Ok(socket) = std::env::var("OPENSHELL_PODMAN_SOCKET") else { - return "OPENSHELL_PODMAN_SOCKET is not set".to_string(); - }; - let mut cmd = Command::new("podman"); - cmd.arg("--url").arg(format!("unix://{socket}")).args([ - "logs", - "--tail", - "200", - container_name, - ]); - apply_podman_config_env(&mut cmd); - match cmd.output().await { - Ok(output) => format!( - "exit={:?}\nstdout:\n{}\nstderr:\n{}", - output.status.code(), - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - ), - Err(err) => format!("failed to run podman logs: {err}"), - } -} - -async fn provider_token_debug(sandbox_name: &str, target_port: u16) -> String { - let sandbox_logs = sandbox_logs(sandbox_name).await; - let Ok(socket) = std::env::var("OPENSHELL_PODMAN_SOCKET") else { - return format!("Sandbox logs:\n{sandbox_logs}\nOPENSHELL_PODMAN_SOCKET is not set"); - }; - let container_name = match podman_container_name_for_sandbox(&socket, sandbox_name).await { - Ok(name) => name, - Err(err) => return format!("Sandbox logs:\n{sandbox_logs}\n{err}"), - }; - let env = podman_exec_capture(&container_name, &["env"]).await; - let hosts = podman_exec_capture(&container_name, &["cat", "/etc/hosts"]).await; - let processes = podman_exec_capture(&container_name, &["ps", "-ef"]).await; - let resolve_host = podman_exec_capture( - &container_name, - &[ - "python3", - "-c", - "import socket; print(socket.getaddrinfo('host.openshell.internal', 0, type=socket.SOCK_STREAM))", - ], - ) - .await; - let target_probe = podman_exec_capture( - &container_name, - &[ - "python3", - "-c", - &format!( - "import socket; s=socket.create_connection(('host.openshell.internal', {target_port}), 2); print('connected', s.getpeername()); s.close()" - ), - ], - ) - .await; - let container_logs = podman_logs_capture(&container_name).await; - - format!( - "Sandbox logs:\n{sandbox_logs}\n\ - Container: {container_name}\n\ - --- podman env ---\n{env}\n\ - --- /etc/hosts ---\n{hosts}\n\ - --- ps -ef ---\n{processes}\n\ - --- resolve host.openshell.internal ---\n{resolve_host}\n\ - --- protected target TCP probe ---\n{target_probe}\n\ - --- podman logs ---\n{container_logs}" - ) -} - -fn write_profile(profile_type: &str, token_port: u16, target_port: u16) -> NamedTempFile { +fn write_profile( + profile_type: &str, + token_port: u16, + identity_port: u16, + target_port: u16, + instance: &str, +) -> NamedTempFile { let token_endpoint = format!("http://127.0.0.1:{token_port}/token"); let mut file = tempfile::Builder::new() .suffix(".yaml") @@ -543,7 +618,7 @@ fn write_profile(profile_type: &str, token_port: u16, target_port: u16) -> Named let profile = format!( r"id: {profile_type} display_name: Podman token exchange e2e -description: Podman e2e provider profile for two-stage token exchange +description: Independent bearer and JWT-SVID grants for one request category: other credentials: - name: subject_token @@ -557,18 +632,39 @@ credentials: token_grant: grant_type: token_exchange token_endpoint: {token_endpoint} - audience: final-audience + audience: service-{instance} + scopes: [service.read] jwt_svid_audience: {token_endpoint} client_assertion_type: {CLIENT_ASSERTION_TYPE} requested_token_type: {TOKEN_TYPE_ACCESS_TOKEN} - cache_ttl_seconds: 30 + cache_ttl_seconds: 300 + audience_overrides: + - path: /deny-service/{instance} + audience: denied-service-{instance} subject_token: source: provider_credential credential: subject_token subject_token_type: {TOKEN_TYPE_ACCESS_TOKEN} + - name: workload_identity + description: Signed workload identity for the identity proxy + required: false + auth_style: header + header_name: X-Workload-Jwt + token_grant: + grant_type: client_credentials + token_endpoint: http://127.0.0.1:{identity_port}/identity-token + audience: {IDENTITY_AUDIENCE}-{instance} + scopes: [identity.read] + jwt_svid_audience: {IDENTITY_JWT_AUDIENCE}/{instance} + client_assertion_type: {IDENTITY_ASSERTION_TYPE} + cache_ttl_seconds: 600 + audience_overrides: + - path: /deny-identity/{instance} + audience: denied-identity-{instance} endpoints: - host: host.openshell.internal port: {target_port} + path: /** protocol: rest access: read-write enforcement: enforce @@ -595,64 +691,56 @@ while true; do sleep 60; done .to_string() } -async fn podman_container_name_for_sandbox( - socket: &str, +async fn sandbox_exec_http( sandbox_name: &str, + target_port: u16, + path: &str, + expect_denied: bool, ) -> Result { - let mut cmd = Command::new("podman"); - cmd.arg("--url") - .arg(format!("unix://{socket}")) - .arg("ps") - .arg("--filter") - .arg(format!("label=openshell.ai/sandbox-name={sandbox_name}")) - .arg("--filter") - .arg("label=openshell.io/isolation-role=sandbox") - .arg("--format") - .arg("{{.Names}}"); - apply_podman_config_env(&mut cmd); - let output = cmd - .output() - .await - .map_err(|err| format!("spawn podman ps for sandbox container: {err}"))?; - if !output.status.success() { - return Err(format!( - "podman ps for sandbox container failed: {}{}", - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - )); - } - let names = String::from_utf8_lossy(&output.stdout) - .lines() - .map(str::trim) - .filter(|line| !line.is_empty()) - .map(str::to_string) - .collect::>(); - match names.as_slice() { - [name] => Ok(name.clone()), - [] => Err(format!( - "no running Podman container found for sandbox '{sandbox_name}'" - )), - _ => Err(format!( - "multiple running Podman containers found for sandbox '{sandbox_name}': {}", - names.join(", ") - )), - } -} - -fn apply_podman_config_env(cmd: &mut Command) { - if std::env::var_os("OPENSHELL_E2E_CONTAINER_ENGINE_UNSET_XDG_CONFIG_HOME").is_some() { - cmd.env_remove("XDG_CONFIG_HOME"); - } else if let Some(value) = std::env::var_os("OPENSHELL_E2E_CONTAINER_ENGINE_XDG_CONFIG_HOME") { - cmd.env("XDG_CONFIG_HOME", value); - } -} - -async fn sandbox_exec_http(sandbox_name: &str, target_port: u16) -> Result { - let url = format!("http://host.openshell.internal:{target_port}/resource"); + let url = format!("http://host.openshell.internal:{target_port}{path}"); + let expect_denied_python = if expect_denied { "True" } else { "False" }; let script = format!( - "import urllib.request; print(urllib.request.urlopen({url:?}, timeout=5).read().decode())" + r#"import base64, json, os, re, urllib.error, urllib.request + +def contains_credential(value): + if any(marker in value for marker in ( + "stored-user-token", "intermediate-token", "final-access-token", "openshell:resolve:" + )): + return True + for token in re.findall(r"[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+", value): + try: + payload = token.split(".")[1] + claims = json.loads(base64.urlsafe_b64decode(payload + "=" * (-len(payload) % 4))) + if claims.get("iss") == "{ISSUER}": + return True + except (ValueError, UnicodeError): + pass + return False + +if any(contains_credential(value) for value in os.environ.values()): + raise RuntimeError("provider credential material reached the workload environment") +request = urllib.request.Request({url:?}, headers={{ + "Authorization": "Bearer agent-supplied", + "X-Workload-Jwt": "agent-supplied", +}}) +try: + with urllib.request.urlopen(request, timeout=10) as response: + body = response.read().decode() + if contains_credential(body): + raise RuntimeError("provider credential material reached the workload response") + if {expect_denied_python}: + raise RuntimeError("failed grant unexpectedly reached the target") + print(body) +except urllib.error.HTTPError as error: + body = error.read().decode() + if contains_credential(body): + raise RuntimeError("provider credential material reached the workload error") + if not {expect_denied_python} or error.code != 502: + raise RuntimeError("unexpected HTTP response") from None + print("grant-denied") +"# ); - let mut last_output = String::new(); + let mut last_status = None; for _ in 0..20 { let output = openshell_cmd() .args([ @@ -668,26 +756,61 @@ async fn sandbox_exec_http(sandbox_name: &str, target_port: u16) -> Result String { + let name = format!("podman-grants-{instance}-{}", std::process::id()); + run_cli_ignore_error(&["provider", "delete", &name, "--yes"]).await; + run_cli_ignore_error(&["profile", "delete", &name, "--yes"]).await; + let profile = write_profile(&name, token_port, identity_port, target_port, instance); + let profile_path = profile + .path() + .to_str() + .expect("profile path should be UTF-8"); + run_cli(&["profile", "import", "-f", profile_path]) + .await + .expect("import independent-grant profile"); + run_cli(&[ + "provider", + "create", + "--name", + &name, + "--type", + &name, + "--credential", + &format!("subject_token={USER_SUBJECT_TOKEN}-{instance}"), + ]) + .await + .expect("create provider instance"); + name } #[tokio::test] -async fn podman_provider_token_exchange_injects_bearer_header() { +async fn podman_provider_token_exchange_injects_independent_grants_across_sandboxes() { let gateway_socket = PathBuf::from( std::env::var("OPENSHELL_E2E_GATEWAY_SPIFFE_SOCKET") .expect("OPENSHELL_E2E_GATEWAY_SPIFFE_SOCKET must be set by e2e-podman.sh"), @@ -696,71 +819,102 @@ async fn podman_provider_token_exchange_injects_bearer_header() { std::env::var("OPENSHELL_E2E_PROVIDER_SPIFFE_SOCKET") .expect("OPENSHELL_E2E_PROVIDER_SPIFFE_SOCKET must be set by e2e-podman.sh"), ); - - let profile_type = format!("podman-token-exchange-e2e-{}", std::process::id()); - let provider_name = format!("podman-token-exchange-e2e-{}", std::process::id()); let token_port = find_free_port(); + let identity_port = find_free_port(); let target_port = find_free_port(); - let token_endpoint = format!("http://127.0.0.1:{token_port}/token"); let gateway_subject = format!("spiffe://{TRUST_DOMAIN}/openshell/gateway"); + // The mock shares a workload subject. Distinct providers and JWT audiences + // test cross-sandbox credential isolation, not production SPIRE attestation. let supervisor_subject = format!("spiffe://{TRUST_DOMAIN}/openshell/sandbox/e2e"); - + let observations = Arc::new(GrantObservations::default()); let _gateway_spiffe = start_spiffe_workload_api(&gateway_socket, &gateway_subject).await; let _provider_spiffe = start_spiffe_workload_api(&provider_socket, &supervisor_subject).await; - let _gateway_token = start_gateway_token_endpoint(token_port).await; - let _target = start_protected_target(target_port).await; - - run_cli_ignore_error(&["provider", "delete", &provider_name, "--yes"]).await; - run_cli_ignore_error(&["profile", "delete", &profile_type, "--yes"]).await; - - let profile = write_profile(&profile_type, token_port, target_port); - let profile_path = profile - .path() - .to_str() - .expect("profile path should be UTF-8"); - run_cli(&["profile", "import", "-f", profile_path]) - .await - .expect("import provider profile"); - run_cli(&[ - "provider", - "create", - "--name", - &provider_name, - "--type", - &profile_type, - "--credential", - &format!("subject_token={USER_SUBJECT_TOKEN}"), - ]) - .await - .expect("create provider"); + let _gateway_token = start_gateway_token_endpoint(token_port, Arc::clone(&observations)).await; + let _identity_token = + start_identity_token_endpoint(identity_port, Arc::clone(&observations)).await; + let _target = start_protected_target(target_port, Arc::clone(&observations)).await; + + let mut provider_names = Vec::new(); + for instance in ["a", "b"] { + provider_names + .push(create_provider_instance(token_port, identity_port, target_port, instance).await); + } let script = sandbox_script(); - let mut sandbox = SandboxGuard::create_keep_with_args( - &["--provider", &provider_name], + let mut sandbox_a = SandboxGuard::create_keep_with_args( + &["--provider", &provider_names[0]], &["sh", "-lc", &script], "token-server-ready", ) .await - .unwrap_or_else(|err| { - panic!( - "sandbox should complete token exchange against {token_endpoint} and protected target port {target_port}:\n{err}" - ) - }); - let request_output = match sandbox_exec_http(&sandbox.name, target_port).await { - Ok(output) => output, - Err(err) => { - let debug = provider_token_debug(&sandbox.name, target_port).await; - panic!("request protected target from kept sandbox: {err}\n{debug}"); - } - }; + .expect("create sandbox A"); + let mut sandbox_b = SandboxGuard::create_keep_with_args( + &["--provider", &provider_names[1]], + &["sh", "-lc", &script], + "token-server-ready", + ) + .await + .expect("create sandbox B"); + + // Both supervisors request the same host/port concurrently. The target + // verifies each path's distinct bearer and signed JWT audience together. + let initial = tokio::join!( + sandbox_exec_http(&sandbox_a.name, target_port, "/resource/a", false), + sandbox_exec_http(&sandbox_b.name, target_port, "/resource/b", false), + ); + let before_cache = observations.issued(); + let cached = tokio::join!( + sandbox_exec_http(&sandbox_a.name, target_port, "/resource/a", false), + sandbox_exec_http(&sandbox_b.name, target_port, "/resource/b", false), + ); + let after_cache = observations.issued(); + let target_requests = observations.target_requests.load(Ordering::SeqCst); + // An audience override forces acquisition of only the failing credential; + // the other credential remains cached from the successful request. + let denied = tokio::join!( + sandbox_exec_http(&sandbox_a.name, target_port, "/deny-service/a", true), + sandbox_exec_http(&sandbox_b.name, target_port, "/deny-identity/b", true), + ); - run_cli_ignore_error(&["provider", "delete", &provider_name, "--yes"]).await; - run_cli_ignore_error(&["profile", "delete", &profile_type, "--yes"]).await; - sandbox.cleanup().await; + sandbox_a.cleanup().await; + sandbox_b.cleanup().await; + for name in &provider_names { + run_cli_ignore_error(&["provider", "delete", name, "--yes"]).await; + run_cli_ignore_error(&["profile", "delete", name, "--yes"]).await; + } + for outcome in [initial.0, initial.1, cached.0, cached.1] { + assert_eq!( + outcome.expect("independent grant request should succeed"), + "independent-grants-ok" + ); + } + assert_eq!( + observations.target_rejected.load(Ordering::SeqCst), + 0, + "no request may present mismatched or untrusted credentials, even before a retry" + ); assert!( - request_output.contains("token-exchange-ok"), - "protected target should receive the final exchanged bearer token:\n{}", - request_output + before_cache + .iter() + .all(|(service, identity)| *service > 0 && *identity > 0), + "each provider instance must acquire both credentials" + ); + assert_eq!( + after_cache, before_cache, + "repeat requests should use both caches" + ); + for outcome in [denied.0, denied.1] { + assert_eq!( + outcome.expect("failed grant should return 502"), + "grant-denied" + ); + } + assert!(observations.service_denied.load(Ordering::SeqCst) > 0); + assert!(observations.identity_denied.load(Ordering::SeqCst) > 0); + assert_eq!( + observations.target_requests.load(Ordering::SeqCst), + target_requests, + "neither failed grant may forward any request bytes" ); } diff --git a/e2e/with-docker-gateway.sh b/e2e/with-docker-gateway.sh index 2298c2312c..b1a2559c20 100755 --- a/e2e/with-docker-gateway.sh +++ b/e2e/with-docker-gateway.sh @@ -526,6 +526,7 @@ if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then fi SUPERVISOR_IMAGE="$(resolve_docker_supervisor_image)" +export OPENSHELL_SUPERVISOR_IMAGE="${SUPERVISOR_IMAGE}" build_local_docker_supervisor_image_if_required "${SUPERVISOR_IMAGE}" ensure_docker_supervisor_image "${SUPERVISOR_IMAGE}" echo "Using Docker supervisor image: ${SUPERVISOR_IMAGE}" diff --git a/tests/ansible/playbooks/drivers/podman/default-userns-baseline.yaml b/tests/ansible/playbooks/drivers/podman/default-userns-baseline.yaml index 2542741dd8..7356ae3c5e 100644 --- a/tests/ansible/playbooks/drivers/podman/default-userns-baseline.yaml +++ b/tests/ansible/playbooks/drivers/podman/default-userns-baseline.yaml @@ -12,10 +12,6 @@ - name: Wait for SSH ansible.builtin.wait_for_connection: - - name: Resolve installed gateway context - ansible.builtin.include_role: - name: openshell_test_gateway - - name: Detect tmachine container runtime ansible.builtin.include_role: name: tmachine_container_runtime @@ -42,7 +38,7 @@ - name: Read OpenShell gateway configuration become: true ansible.builtin.slurp: - src: "{{ openshell_test_gateway.config_path }}" + src: /etc/openshell/gateway.toml register: openshell_gateway_config - name: Require unconfigured Podman user namespaces diff --git a/tests/ansible/playbooks/drivers/podman/e2e.yaml b/tests/ansible/playbooks/drivers/podman/e2e.yaml index e20d120d68..e839a2cfff 100644 --- a/tests/ansible/playbooks/drivers/podman/e2e.yaml +++ b/tests/ansible/playbooks/drivers/podman/e2e.yaml @@ -11,21 +11,6 @@ - name: Wait for SSH ansible.builtin.wait_for_connection: - - name: Resolve installed gateway context - ansible.builtin.include_role: - name: openshell_test_gateway - - - name: Detect tmachine container runtime - ansible.builtin.include_role: - name: tmachine_container_runtime - - - name: Require rootless Podman for the E2E workload fixtures - ansible.builtin.assert: - that: - - tmachine_container_runtime_name == 'podman' - - tmachine_container_runtime_is_rootless - fail_msg: The Podman E2E archive requires the rootless Podman environment - - name: Create Podman E2E test directory become: true ansible.builtin.file: @@ -61,7 +46,17 @@ - "{{ podman_e2e_test_root }}/openshell-e2e-python-dev.tar" environment: HOME: /home/tmachine - XDG_RUNTIME_DIR: "/run/user/{{ tmachine_container_runtime_tmachine_uid.stdout }}" + XDG_RUNTIME_DIR: /run/user/1000 + + - name: Remove any previous OpenShell gateway registration + ansible.builtin.command: + argv: [/usr/local/bin/openshell, gateway, remove, tmachine] + changed_when: false + failed_when: false + + - name: Register the configured OpenShell gateway + ansible.builtin.command: + argv: [/usr/local/bin/openshell, gateway, add, http://127.0.0.1:17670, --local, --name, tmachine] - name: Run Podman E2E archive tests ansible.builtin.command: @@ -78,16 +73,16 @@ - --no-fail-fast environment: CONTAINER_ENGINE: podman - CONTAINER_HOST: "unix://{{ tmachine_container_runtime_socket }}" + CONTAINER_HOST: unix:///run/user/1000/podman/podman.sock HOME: /home/tmachine - OPENSHELL_BIN: "{{ openshell_test_cli.stdout }}" + OPENSHELL_BIN: /usr/local/bin/openshell OPENSHELL_E2E_CONTAINER_ENGINE_UNSET_XDG_CONFIG_HOME: "1" OPENSHELL_E2E_DRIVER: podman - OPENSHELL_E2E_NETWORK_NAME: "{{ openshell_test_gateway.network_name }}" + OPENSHELL_E2E_NETWORK_NAME: tmachine OPENSHELL_E2E_SANDBOX_NAMESPACE: tmachine - OPENSHELL_GATEWAY: "{{ openshell_test_gateway.name }}" - OPENSHELL_PODMAN_SOCKET: "{{ tmachine_container_runtime_socket }}" - XDG_RUNTIME_DIR: "/run/user/{{ tmachine_container_runtime_tmachine_uid.stdout }}" + OPENSHELL_GATEWAY: tmachine + OPENSHELL_PODMAN_SOCKET: /run/user/1000/podman/podman.sock + XDG_RUNTIME_DIR: /run/user/1000 register: podman_e2e_test_result changed_when: false failed_when: false @@ -100,10 +95,18 @@ ansible.builtin.debug: var: podman_e2e_test_result.stdout_lines - - name: Collect installed gateway diagnostics after test failure - ansible.builtin.include_role: - name: openshell_test_gateway - tasks_from: journal.yaml + - name: Capture OpenShell gateway journal after Podman E2E test failure + become: true + ansible.builtin.command: + argv: [journalctl, --unit, openshell-gateway.service, --no-pager, --lines, "200"] + changed_when: false + failed_when: false + when: podman_e2e_test_result.rc != 0 + register: podman_e2e_gateway_journal + + - name: Show OpenShell gateway journal after Podman E2E test failure + ansible.builtin.debug: + var: podman_e2e_gateway_journal.stdout_lines when: podman_e2e_test_result.rc != 0 - name: Require Podman E2E archive success diff --git a/tests/ansible/playbooks/drivers/podman/tests.yaml b/tests/ansible/playbooks/drivers/podman/tests.yaml index 11390fb57f..abe2f6e553 100644 --- a/tests/ansible/playbooks/drivers/podman/tests.yaml +++ b/tests/ansible/playbooks/drivers/podman/tests.yaml @@ -12,10 +12,6 @@ - name: Wait for SSH ansible.builtin.wait_for_connection: - - name: Resolve installed gateway context - ansible.builtin.include_role: - name: openshell_test_gateway - - name: Detect tmachine container runtime ansible.builtin.include_role: name: tmachine_container_runtime @@ -43,12 +39,21 @@ owner: tmachine group: tmachine + - name: Remove any previous OpenShell gateway registration + ansible.builtin.command: + argv: [/usr/local/bin/openshell, gateway, remove, tmachine] + changed_when: false + failed_when: false + + - name: Register the configured OpenShell gateway + ansible.builtin.command: + argv: [/usr/local/bin/openshell, gateway, add, http://127.0.0.1:17670, --local, --name, tmachine] + - name: Run Podman archive tests ansible.builtin.command: argv: [cargo-nextest, nextest, run, --archive-file, "{{ podman_test_root }}/tests.tar.zst", --workspace-remap, "{{ podman_test_root }}", --no-capture] environment: - OPENSHELL_GATEWAY: "{{ openshell_test_gateway.name }}" - OPENSHELL_BIN: "{{ openshell_test_cli.stdout }}" + OPENSHELL_BIN: /usr/local/bin/openshell OPENSHELL_TEST_INPUT_DIR: "{{ podman_test_input_dir }}" OPENSHELL_PODMAN_TEST_IMAGE: "{{ openshell_podman_test_image | default('') }}" register: podman_test_result @@ -63,10 +68,24 @@ ansible.builtin.debug: var: podman_test_result.stdout_lines - - name: Collect installed gateway diagnostics after test failure - ansible.builtin.include_role: - name: openshell_test_gateway - tasks_from: journal.yaml + - name: Capture OpenShell gateway journal after Podman test failure + become: true + ansible.builtin.command: + argv: + - journalctl + - --unit + - openshell-gateway.service + - --no-pager + - --lines + - "200" + changed_when: false + failed_when: false + when: podman_test_result.rc != 0 + register: podman_gateway_journal + + - name: Show OpenShell gateway journal after Podman test failure + ansible.builtin.debug: + var: podman_gateway_journal.stdout_lines when: podman_test_result.rc != 0 - name: Discover Podman containers after test failure diff --git a/tests/ansible/playbooks/drivers/podman/userns-profile.yaml b/tests/ansible/playbooks/drivers/podman/userns-profile.yaml index 64880a0aa4..a3071c2833 100644 --- a/tests/ansible/playbooks/drivers/podman/userns-profile.yaml +++ b/tests/ansible/playbooks/drivers/podman/userns-profile.yaml @@ -12,10 +12,6 @@ - name: Wait for SSH ansible.builtin.wait_for_connection: - - name: Resolve installed gateway context - ansible.builtin.include_role: - name: openshell_test_gateway - - name: Detect tmachine container runtime ansible.builtin.include_role: name: tmachine_container_runtime @@ -42,15 +38,21 @@ - name: Apply the Podman user-namespace fixture become: true ansible.builtin.blockinfile: - path: "{{ openshell_test_gateway.config_path }}" - insertafter: '^\[openshell\.drivers\.podman\]$' + path: /etc/openshell/gateway.toml marker: "# {mark} OpenShell Podman userns test fixture" block: "{{ lookup('ansible.builtin.file', podman_userns_config) | trim }}" - - name: Restart installed gateway with the Podman user-namespace fixture - ansible.builtin.include_role: - name: openshell_test_gateway - tasks_from: restart.yaml + - name: Restart OpenShell gateway with the Podman user-namespace fixture + become: true + ansible.builtin.systemd_service: + name: openshell-gateway.service + state: restarted + + - name: Wait for the configured OpenShell gateway + ansible.builtin.wait_for: + host: 127.0.0.1 + port: 17670 + timeout: 60 - name: Capture direct Podman user-namespace mapping become: true diff --git a/tests/ansible/roles/openshell_client/tasks/main.yaml b/tests/ansible/roles/openshell_client/tasks/main.yaml index 8a60195fa1..6586f23ac2 100644 --- a/tests/ansible/roles/openshell_client/tasks/main.yaml +++ b/tests/ansible/roles/openshell_client/tasks/main.yaml @@ -12,17 +12,3 @@ - --local - --name - "{{ openshell_client_gateway_name | default('tmachine') }}" - -- name: Publish test gateway context - ansible.builtin.include_role: - name: openshell_test_gateway - tasks_from: publish.yaml - vars: - openshell_test_gateway_context: - name: "{{ openshell_client_gateway_name | default('tmachine') }}" - config_path: "{{ openshell_client_gateway_config_path | default('/etc/openshell/gateway.toml') }}" - service_scope: "{{ openshell_client_gateway_service_scope | default('system') }}" - service_user: "{{ openshell_gateway_user if openshell_client_gateway_service_scope | default('system') == 'user' else 'root' }}" - service_home: "{{ openshell_gateway_home if openshell_client_gateway_service_scope | default('system') == 'user' else '/root' }}" - service_uid: "{{ openshell_gateway_uid if openshell_client_gateway_service_scope | default('system') == 'user' else '0' }}" - network_name: "{{ openshell_client_gateway_network_name | default('tmachine') }}" diff --git a/tests/ansible/roles/openshell_packaged_gateway/tasks/main.yaml b/tests/ansible/roles/openshell_packaged_gateway/tasks/main.yaml index e74fa5edf6..4d1277f5f4 100644 --- a/tests/ansible/roles/openshell_packaged_gateway/tasks/main.yaml +++ b/tests/ansible/roles/openshell_packaged_gateway/tasks/main.yaml @@ -106,6 +106,3 @@ vars: openshell_client_gateway_endpoint: https://127.0.0.1:17670 openshell_client_gateway_name: openshell - openshell_client_gateway_config_path: /var/lib/openshell-qualification/gateway.toml - openshell_client_gateway_service_scope: user - openshell_client_gateway_network_name: openshell diff --git a/tests/ansible/roles/openshell_test_gateway/tasks/journal.yaml b/tests/ansible/roles/openshell_test_gateway/tasks/journal.yaml deleted file mode 100644 index 2e5d7874f9..0000000000 --- a/tests/ansible/roles/openshell_test_gateway/tasks/journal.yaml +++ /dev/null @@ -1,22 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - ---- -# Root's packaged user unit can log to the system journal. Match journal -# fields instead of --user so both root and tmachine user units are captured. -- name: Capture installed gateway journal - become: true - ansible.builtin.command: - argv: >- - {{ ['journalctl'] - + (['_SYSTEMD_USER_UNIT=openshell-gateway.service', '_UID=' ~ openshell_test_gateway.service_uid] - if openshell_test_gateway.service_scope == 'user' - else ['--unit', 'openshell-gateway.service']) - + ['--no-pager', '--lines', '200'] }} - register: openshell_test_gateway_journal - changed_when: false - failed_when: false - -- name: Show installed gateway journal - ansible.builtin.debug: - var: openshell_test_gateway_journal.stdout_lines diff --git a/tests/ansible/roles/openshell_test_gateway/tasks/main.yaml b/tests/ansible/roles/openshell_test_gateway/tasks/main.yaml deleted file mode 100644 index 55d18b310e..0000000000 --- a/tests/ansible/roles/openshell_test_gateway/tasks/main.yaml +++ /dev/null @@ -1,42 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - ---- -- name: Read installed gateway context - ansible.builtin.slurp: - src: /var/lib/openshell-test/gateway.yaml - register: openshell_test_gateway_metadata - -- name: Load installed gateway context - ansible.builtin.set_fact: - openshell_test_gateway: "{{ (openshell_test_gateway_metadata.content | b64decode | from_yaml).openshell_test_gateway }}" - -- name: Validate installed gateway context - ansible.builtin.assert: - that: - - openshell_test_gateway.service_scope in ['system', 'user'] - - openshell_test_gateway.config_path is match('^/') - - openshell_test_gateway.name | length > 0 - -- name: Resolve installed OpenShell CLI from PATH - ansible.builtin.command: - argv: [/bin/sh, -c, command -v openshell] - register: openshell_test_cli - changed_when: false - -- name: Check active gateway configuration - become: true - ansible.builtin.stat: - path: "{{ openshell_test_gateway.config_path }}" - register: openshell_test_gateway_config - -- name: Require active gateway configuration - ansible.builtin.assert: - that: [openshell_test_gateway_config.stat.isreg | default(false)] - -- name: Resolve gateway service environment - ansible.builtin.set_fact: - openshell_test_gateway_service_environment: - HOME: "{{ openshell_test_gateway.service_home }}" - XDG_RUNTIME_DIR: "/run/user/{{ openshell_test_gateway.service_uid }}" - DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ openshell_test_gateway.service_uid }}/bus" diff --git a/tests/ansible/roles/openshell_test_gateway/tasks/publish.yaml b/tests/ansible/roles/openshell_test_gateway/tasks/publish.yaml deleted file mode 100644 index 1e04d266c2..0000000000 --- a/tests/ansible/roles/openshell_test_gateway/tasks/publish.yaml +++ /dev/null @@ -1,23 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - ---- -# Installation and suite execution are separate Ansible processes. Persist only -# non-secret metadata; the CLI keeps registration and mTLS credentials itself. -- name: Create test gateway metadata directory - become: true - ansible.builtin.file: - path: /var/lib/openshell-test - state: directory - owner: root - group: root - mode: "0755" - -- name: Publish installed gateway context - become: true - ansible.builtin.copy: - dest: /var/lib/openshell-test/gateway.yaml - owner: root - group: root - mode: "0644" - content: "{{ {'openshell_test_gateway': openshell_test_gateway_context} | to_nice_yaml }}" diff --git a/tests/ansible/roles/openshell_test_gateway/tasks/restart.yaml b/tests/ansible/roles/openshell_test_gateway/tasks/restart.yaml deleted file mode 100644 index 56b56123f9..0000000000 --- a/tests/ansible/roles/openshell_test_gateway/tasks/restart.yaml +++ /dev/null @@ -1,35 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - ---- -- name: Restart installed gateway - block: - - name: Restart gateway service with the active configuration - become: true - become_user: "{{ openshell_test_gateway.service_user }}" - ansible.builtin.systemd_service: - name: openshell-gateway.service - scope: "{{ openshell_test_gateway.service_scope }}" - state: restarted - environment: "{{ openshell_test_gateway_service_environment }}" - - - name: Wait for authenticated gateway readiness - ansible.builtin.command: - argv: ["{{ openshell_test_cli.stdout }}", gateway, info, --output, json] - environment: - OPENSHELL_GATEWAY: "{{ openshell_test_gateway.name }}" - changed_when: false - register: openshell_test_gateway_ready - retries: 30 - delay: 2 - until: >- - openshell_test_gateway_ready.rc == 0 and - (openshell_test_gateway_ready.stdout | from_json).status == 'healthy' - - rescue: - - name: Collect gateway restart diagnostics - ansible.builtin.include_tasks: journal.yaml - - - name: Report gateway restart failure - ansible.builtin.fail: - msg: Installed gateway failed to restart with the driver fixture diff --git a/tests/suites/drivers/podman/tests/default_userns.rs b/tests/suites/drivers/podman/tests/default_userns.rs index f9aaf4354c..95cb8c814f 100644 --- a/tests/suites/drivers/podman/tests/default_userns.rs +++ b/tests/suites/drivers/podman/tests/default_userns.rs @@ -16,19 +16,33 @@ const SANDBOX_TIMEOUT: Duration = Duration::from_secs(300); const PODMAN_TEST_INPUT_DIR_ENV: &str = "OPENSHELL_TEST_INPUT_DIR"; const PODMAN_TEST_IMAGE_ENV: &str = "OPENSHELL_PODMAN_TEST_IMAGE"; +const WORKSPACE_AND_UID_MAP_PROBE: &str = r#"set -eu +workload_owner="$(id -u):$(id -g)" +workspace_owner="$(stat -c '%u:%g' /sandbox)" +printf 'workload-owner=%s\nworkspace-owner=%s\n' "$workload_owner" "$workspace_owner" +test "$(id -u)" -ne 0 +test "$workspace_owner" = "$workload_owner" +probe=$(mktemp /sandbox/userns-probe.XXXXXX) +printf 'workspace probe\n' > "$probe" +rm "$probe" +echo podman-userns-workspace-ok +cat /proc/self/uid_map +"#; + /// Verify that the gateway's user-namespace configuration matches Podman's -/// direct behavior for the same profile. +/// direct behavior for the same profile and preserves workspace access. /// /// The test runs a short-lived sandbox command and compares its user-namespace /// mapping with the direct-Podman reference stored at /// `OPENSHELL_TEST_INPUT_DIR/reference-uid-map`. The tmachine pre-test /// playbook creates that reference in the same gateway-user context. This deliberately /// avoids baking a particular Podman mapping into OpenShell's test contract. -/// +/// The workload also verifies that the managed workspace is owned by its +/// non-root UID/GID and that it can create, write, and remove a file there. #[tokio::test] async fn configured_userns_matches_podman_reference() { - let mut runner = OpenShellRunner::from_env("podman-userns") - .expect("candidate openshell CLI is available"); + let mut runner = + OpenShellRunner::from_env("podman-userns").expect("candidate openshell CLI is available"); let result = async { runner.check_gateway_status().await?; assert_podman_gateway(&runner).await?; @@ -59,15 +73,18 @@ async fn configured_userns_matches_podman_reference() { if let Some(image) = workload_image.as_deref() { create_args.extend(["--from", image]); } - create_args.extend(["--no-tty", "--", "cat", "/proc/self/uid_map"]); + create_args.extend(["--no-tty", "--", "sh", "-c", WORKSPACE_AND_UID_MAP_PROBE]); let run = runner - .step("userns/uid-map") - .description("sandbox exposes its UID map") + .step("userns/workspace-and-uid-map") + .description("sandbox can write to its owned workspace and exposes its UID map") .with_timeout(SANDBOX_TIMEOUT) .run(&create_args) .await .map_err(|error| error.to_string())?; run.require_success()?; + if !run.stdout().contains("podman-userns-workspace-ok") { + return Err(run.failure_diagnostic("non-root workload owns and can write to /sandbox")); + } let sandbox_uid_map = normalize_uid_map(run.stdout()).ok_or_else(|| { run.failure_diagnostic("sandbox returns a non-empty UID map") })?;