Skip to content

Commit 820046e

Browse files
gitcommitankitgitcommitankit
authored
Merge pull request #11 from gitcommitankit/feature/tenant-network-policy
feat(networking): add tenant agent isolation NetworkPolicy
2 parents 168e93d + 39bb184 commit 820046e

8 files changed

Lines changed: 148 additions & 31 deletions

File tree

‎.agents/skills/agentrax-context/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ description: Project context and settled architecture decisions for the Agentrax
2020

2121
- **Autoscaling**: native `HorizontalPodAutoscaler` pointed at Prometheus Adapter custom metrics (`queueDepth` or `gpuUtilization`). No custom scaling loop. During active canary, the stable HPA is paused (deleted) and no canary HPA is created — autoscaling resumes only after promotion or rollback.
2222
- **Traffic splitting**: Gateway API `HTTPRoute` weighted backends. Not Istio, not ingress annotations.
23+
- **Network Isolation**: Two-tier Kubernetes `NetworkPolicy` (`allow-metrics-traffic` in `agentrax-system` allowing operator metrics on TCP 8443; `tenant-agent-isolation` rendered into every `tenant-*` namespace selecting agent pods with `agentrax.io/agent: "true"` for scraping on TCP 8080 and egress to API server/CoreDNS). No service mesh.
2324
- **MCP registry**: embedded HTTP handler inside the operator process, backed by a `ConfigMap`. Not a separate Deployment, not a new database — HA storage is a v2 item.
2425
- **Non-goals**: no model training/fine-tuning, no general-purpose workload management, no service mesh, no UI in v1. Flag any drift toward these rather than quietly implementing them.
2526

‎config/default/kustomization.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ resources:
3131
# Only Pod(s) running a namespace labeled with 'metrics: enabled' will be able to gather the metrics.
3232
# Only CR(s) which requires webhooks and are applied on namespaces labeled with 'webhooks: enabled' will
3333
# be able to communicate with the Webhook Server.
34-
#- ../network-policy
34+
- ../network-policy
3535

3636
# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
3737
patches:

‎config/network-policy/kustomization.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
resources:
22
- allow-metrics-traffic.yaml
3+
- tenant-agent-isolation.yaml

‎config/network-policy/tenant-agent-isolation.yaml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
# Tenant Agent Isolation NetworkPolicy
3+
#
4+
# Purpose: Restricts agent pods across tenant namespaces, enforcing zero-trust
5+
# isolation between tenants and preventing unauthorized outbound traffic.
6+
#
7+
# Selector: Matches all pods labelled `agentrax.io/agent: "true"`.
8+
# The AgentDeployment reconciler sets this label on every pod template
9+
# it manages, so this policy applies to all managed agent pods.
10+
#
11+
# Ingress rules:
12+
# - Allow Prometheus to scrape metrics on port 8080 from namespaces
13+
# labelled `monitoring: enabled` (kube-prometheus-stack namespace).
14+
#
15+
# Egress rules:
16+
# - Allow egress to kube-apiserver on port 6443 (required for agent-to-API
17+
# communication and tool-calling via the Kubernetes API).
18+
# - Allow CoreDNS lookups on port 53 (UDP and TCP) for service discovery
19+
# within the cluster.
20+
# - All other egress (internet, cross-tenant) is denied by default.
21+
#
22+
# Usage: Apply this manifest to every tenant namespace:
23+
# kubectl apply -n tenant-<name> -f tenant-agent-isolation.yaml
24+
#
25+
# Note: This policy applies only to tenant-* namespaces (managed agent pods).
26+
# The operator namespace (agentrax-system) is protected by allow-metrics-traffic.yaml (TCP 8443).
27+
apiVersion: networking.k8s.io/v1
28+
kind: NetworkPolicy
29+
metadata:
30+
name: tenant-agent-isolation
31+
labels:
32+
app.kubernetes.io/name: agentrax
33+
app.kubernetes.io/managed-by: kustomize
34+
spec:
35+
# Select all pods carrying the agentrax.io/agent=true label.
36+
# This label is set by agentLabels() in the AgentDeployment reconciler.
37+
podSelector:
38+
matchLabels:
39+
agentrax.io/agent: "true"
40+
policyTypes:
41+
- Ingress
42+
- Egress
43+
ingress:
44+
# Allow Prometheus to scrape agent /metrics on port 8080.
45+
# Prometheus Operator runs in a namespace labelled `monitoring: enabled`.
46+
- from:
47+
- namespaceSelector:
48+
matchLabels:
49+
monitoring: enabled
50+
ports:
51+
- port: 8080
52+
protocol: TCP
53+
egress:
54+
# Allow outbound to kube-apiserver via ClusterIP (port 443) and direct endpoint (port 6443).
55+
# Agents may call the Kubernetes API to discover services or use cluster tools.
56+
- ports:
57+
- port: 443
58+
protocol: TCP
59+
- port: 6443
60+
protocol: TCP
61+
# Allow CoreDNS resolution on UDP and TCP port 53.
62+
# Matches cluster DNS pods in the kube-system namespace.
63+
- to:
64+
- namespaceSelector:
65+
matchLabels:
66+
kubernetes.io/metadata.name: kube-system
67+
podSelector:
68+
matchExpressions:
69+
- key: k8s-app
70+
operator: In
71+
values: ["kube-dns", "coredns"]
72+
ports:
73+
- port: 53
74+
protocol: UDP
75+
- port: 53
76+
protocol: TCP

‎docs/ARCHITECTURE.md‎

Lines changed: 34 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -89,16 +89,16 @@ flowchart TB
8989

9090
The repository enforces strict directional boundaries to prevent circular dependencies and isolate business logic from Kubernetes plumbing:
9191

92-
| Package | Scope & Responsibility | Key Invariants |
93-
| ---------------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
94-
| `api/v1alpha1/` | CRD type definitions, OpenAPI markers, schema validation rules, and status condition constants. | **Zero business logic**; only struct declarations and generated deep-copy methods. |
95-
| `internal/controller/` | Controller-runtime reconcile loops (`AgentDeployment`, `TenantQuota`). | Only layer that executes write calls against the Kubernetes API for core-owned resources (Deployments, Services, HPAs, HTTPRoutes). Consumes subsystems via interfaces. |
96-
| `internal/quota/` | Quota arithmetic and concurrency-safe in-flight reservation cache. | Pure arithmetic; mutex-guarded state map; zero direct API server network calls in calculation paths. |
97-
| `internal/webhook/` | Validating and Mutating admission webhooks. | Shared with `internal/quota` to enforce admission rules before objects are persisted. |
98-
| `internal/scaling/` | HPA synthesis, velocity rules, and dynamic quota ceiling headroom. | Calculates `QuotaHeadroom()` to cap HPA `maxReplicas` and applies stabilization windows. |
99-
| `internal/rollout/` | Canary state machine, PromQL query construction, and threshold evaluation. | Re-entrant state machine; sample-size gating; fail-safe timeout evaluation. |
100-
| `internal/registry/` | MCP registrar, JSON-RPC 2.0 handshake, TTL sweeper, and discovery REST API. | In-memory registry with ConfigMap write-through for persistence; background health probes and TTL sweep. Explicitly allowed to write the `agentrax-registry` ConfigMap for state recovery. |
101-
| `internal/metrics/` | Bounded HTTP Prometheus query client. | Wraps all responses with `io.LimitReader` (1 MiB ceiling) to prevent memory exhaustion. |
92+
| Package | Scope & Responsibility | Key Invariants |
93+
| ---------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
94+
| `api/v1alpha1/` | CRD type definitions, OpenAPI markers, schema validation rules, and status condition constants. | **Zero business logic**; only struct declarations and generated deep-copy methods. |
95+
| `internal/controller/` | Controller-runtime reconcile loops (`AgentDeployment`, `TenantQuota`). | Only layer that executes write calls against the Kubernetes API for core-owned resources (Deployments, Services, HPAs, HTTPRoutes, ServiceMonitors). Consumes subsystems via interfaces. |
96+
| `internal/quota/` | Quota arithmetic and concurrency-safe in-flight reservation cache. | Pure arithmetic; mutex-guarded state map; zero direct API server network calls in calculation paths. |
97+
| `internal/webhook/` | Validating and Mutating admission webhooks. | Shared with `internal/quota` to enforce admission rules before objects are persisted. |
98+
| `internal/scaling/` | HPA synthesis, velocity rules, and dynamic quota ceiling headroom. | Calculates `QuotaHeadroom()` to cap HPA `maxReplicas` and applies stabilization windows. |
99+
| `internal/rollout/` | Canary state machine, PromQL query construction, and threshold evaluation. | Re-entrant state machine; sample-size gating; fail-safe timeout evaluation. |
100+
| `internal/registry/` | MCP registrar, JSON-RPC 2.0 handshake, TTL sweeper, and discovery REST API. | In-memory registry with ConfigMap write-through for persistence; background health probes and TTL sweep. Explicitly allowed to write the `agentrax-registry` ConfigMap for state recovery. |
101+
| `internal/metrics/` | Bounded HTTP Prometheus query client. | Wraps all responses with `io.LimitReader` (1 MiB ceiling) to prevent memory exhaustion. |
102102

103103
---
104104

@@ -319,19 +319,35 @@ When an `AgentDeployment` is deleted, Kubernetes sets `metadata.deletionTimestam
319319
5. Kubernetes GC cascade deletes child resources (Deployment, Service, HPA, Route)
320320
```
321321

322-
**Invariant**: MCP deregistration MUST complete _before_ the child `Service` is garbage collected, ensuring external clients never encounter dead routing endpoints.
322+
### 4.6 Zero-Trust Multi-Tenant Network Isolation
323+
324+
Agentrax enforces a zero-trust network perimeter around all AI agent workloads running in `tenant-*` namespaces. Because autonomous agents dynamically execute tools via MCP and consume cluster resources, flat Kubernetes networking presents severe security risks (unauthorized inter-tenant access, data exfiltration, and lateral movement).
325+
326+
Agentrax maintains a **two-tier network policy model**:
327+
328+
| Policy Manifest | Target Namespace | Scope & Responsibility |
329+
| :---------------------------- | :---------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
330+
| `allow-metrics-traffic.yaml` | `agentrax-system` | Protects the operator process; allows Prometheus to scrape operator `/metrics` on port `:8443` (HTTPS). |
331+
| `tenant-agent-isolation.yaml` | Every `tenant-*` | Isolates agent pods; enforces default-deny on ingress/egress, strictly whitelisting only metrics scraping (`:8080`), Kubernetes API server (`:443`/`:6443`), and CoreDNS (`:53`). |
332+
333+
#### Ingress & Egress Invariants:
334+
335+
- **Ingress**: Only TCP port `8080` from namespaces labeled `monitoring: enabled` (Prometheus scraping tenant agent metrics).
336+
- **Egress**: Only to the Kubernetes API server (`kube-apiserver` on TCP ports `443`/`6443`) and cluster CoreDNS (`UDP/TCP :53` in `kube-system` DNS pods). All cross-tenant and arbitrary external internet egress destinations remain blocked at the CNI layer.
337+
- **Label Selector Binding**: The `tenant-agent-isolation` policy selects pods dynamically via `agentrax.io/agent: "true"`. The `AgentDeploymentReconciler` automatically stamps this label into the `PodTemplateSpec` of every managed `Deployment` via `agentLabels()`.
323338

324339
---
325340

326341
## 5. Architectural Decision Records (ADRs) & Trade-Offs
327342

328-
| Decision | Alternative Considered | Trade-Off & Rationale for Agentrax |
329-
| --------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
330-
| **Gateway API (`HTTPRoute`)** | Istio `VirtualService` / Ingress Annotations | Istio requires a heavy service-mesh control plane and sidecar injection. Ingress annotations lack standardized multi-backend weighted traffic splits. Gateway API provides a lightweight, vendor-neutral standard for traffic shifting. |
331-
| **Custom Canary Rollout Engine** | Argo Rollouts / Flagger | Generic rollout tools treat metric anomalies as pure percentages without low-traffic statistical gating (`minRequestSample`). Building an embedded, re-entrant state machine allowed us to guarantee sample-size gating and MCP tool re-registration upon promotion. |
332-
| **Native HPA via Custom Metrics** | KEDA (`ScaledObject`) | KEDA is powerful but adds external CRD dependencies. Generating native Kubernetes `HorizontalPodAutoscaler` objects tied to the Prometheus Adapter custom metrics pipeline minimized dependencies while giving full control over stabilization windows. |
333-
| **Embedded Registry + ConfigMap Store** | Dedicated etcd / Redis / Database | Adding a dedicated database for service discovery increases operator operational complexity. The in-operator HTTP server with ConfigMap write-through store provides simple, robust storage for hundreds of agent services with cold-restart recovery. |
334-
| **Go (`controller-runtime`)** | Python (`Kopf`) | Go provides native compile-time safety, seamless alignment with Kubernetes upstream libraries, and access to `setup-envtest` for isolated in-process integration testing. |
343+
| Decision | Alternative Considered | Trade-Off & Rationale for Agentrax |
344+
| --------------------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
345+
| **Gateway API (`HTTPRoute`)** | Istio `VirtualService` / Ingress Annotations | Istio requires a heavy service-mesh control plane and sidecar injection. Ingress annotations lack standardized multi-backend weighted traffic splits. Gateway API provides a lightweight, vendor-neutral standard for traffic shifting. |
346+
| **Custom Canary Rollout Engine** | Argo Rollouts / Flagger | Generic rollout tools treat metric anomalies as pure percentages without low-traffic statistical gating (`minRequestSample`). Building an embedded, re-entrant state machine allowed us to guarantee sample-size gating and MCP tool re-registration upon promotion. |
347+
| **Native HPA via Custom Metrics** | KEDA (`ScaledObject`) | KEDA is powerful but adds external CRD dependencies. Generating native Kubernetes `HorizontalPodAutoscaler` objects tied to the Prometheus Adapter custom metrics pipeline minimized dependencies while giving full control over stabilization windows. |
348+
| **Embedded Registry + ConfigMap Store** | Dedicated etcd / Redis / Database | Adding a dedicated database for service discovery increases operator operational complexity. The in-operator HTTP server with ConfigMap write-through store provides simple, robust storage for hundreds of agent services with cold-restart recovery. |
349+
| **Two-Tier NetworkPolicy** | Istio / Linkerd Service Mesh | Service mesh requires sidecar injection and significant control plane memory overhead. Native Kubernetes NetworkPolicy with label-selector binding (`agentrax.io/agent: "true"`) provides lightweight, CNI-enforced zero-trust tenant isolation with default-deny rules. |
350+
| **Go (`controller-runtime`)** | Python (`Kopf`) | Go provides native compile-time safety, seamless alignment with Kubernetes upstream libraries, and access to `setup-envtest` for isolated in-process integration testing. |
335351

336352
---
337353

‎internal/controller/agentdeployment_builder_test.go‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -240,7 +240,8 @@ func TestDesiredService_ClusterIPType(t *testing.T) {
240240

241241
// ── agentLabels ───────────────────────────────────────────────────────────────
242242

243-
// TestAgentLabels verifies standard label generation for an AgentDeployment.
243+
// TestAgentLabels verifies standard label generation for an AgentDeployment,
244+
// including the agentrax.io/agent selector key used by NetworkPolicies.
244245
func TestAgentLabels(t *testing.T) {
245246
ad := &agentraxv1alpha1.AgentDeployment{
246247
ObjectMeta: metav1.ObjectMeta{Name: "foo"},
@@ -252,7 +253,14 @@ func TestAgentLabels(t *testing.T) {
252253
"app.kubernetes.io/name": "foo",
253254
"app.kubernetes.io/managed-by": "agentrax",
254255
"agentrax.io/tenant": "bar",
256+
"agentrax.io/variant": "stable",
257+
"agentrax.io/agent": "true",
255258
}
259+
260+
if len(labels) != len(expected) {
261+
t.Errorf("expected %d labels, got %d: %v", len(expected), len(labels), labels)
262+
}
263+
256264
for k, v := range expected {
257265
if labels[k] != v {
258266
t.Errorf("agentLabels[%s] = %q, want %q", k, labels[k], v)

‎internal/controller/agentdeployment_controller.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -659,12 +659,15 @@ func (r *AgentDeploymentReconciler) reconcileMCPRegistration(ctx context.Context
659659

660660
// agentLabels returns the canonical label set applied to all resources owned by ad.
661661
// For stable resources (Deployment, Service), this includes variant=stable.
662+
// The agentrax.io/agent label is the NetworkPolicy selector key — all agent pod
663+
// templates carry it so the tenant-agent-isolation policy applies automatically.
662664
func agentLabels(ad *agentraxv1alpha1.AgentDeployment) map[string]string {
663665
return map[string]string{
664666
"app.kubernetes.io/name": ad.Name,
665667
"app.kubernetes.io/managed-by": "agentrax",
666668
"agentrax.io/tenant": ad.Spec.TenantRef,
667669
"agentrax.io/variant": "stable",
670+
"agentrax.io/agent": "true",
668671
}
669672
}
670673

0 commit comments

Comments
 (0)