Private local inference. Kubernetes-native operations. Evidence-driven observability.

Architecture · Projects · Reference platform · Get started · Contribute · Security · Support

We build an open, Go-first platform for running and observing private language models on resource-constrained Linux edge systems. The current reference path combines Ubuntu, single-node k3s, NVIDIA GPU acceleration, Ollama/GGUF models, Open WebUI, OpenTelemetry, Prometheus, and Grafana—without treating a laptop or small server like an unlimited cloud cluster.

flowchart LR
  O[Operator] --> CLI[edge-cli<br/>control plane]
  CLI --> L1[k3s-nvidia-edge<br/>Layer 1]
  CLI --> L2[llm-observability-stack<br/>Layer 2]
  L1 --> K[(Ubuntu + k3s<br/>NVIDIA runtime)]
  L2 --> A[(Ollama + WebUI<br/>telemetry + dashboards)]
  K --> Q[gguf-observability<br/>multi-model runtime evidence]
  A --> Q
  CLI --> T[edge-llm-tests<br/>validation evidence]
  L1 --> T
  L2 --> T
  Q --> T

• Edge-sized by design. CPU fallback and low-memory NVIDIA profiles make constrained hardware a first-class target.
• One owner per layer. Infrastructure, workloads, runtime evidence, and presentation have explicit repository boundaries.
• Observable end to end. GPU, Kubernetes, request, model, and service-path signals meet in the same operational story.
• Privacy is structural. Local inference stays local; published test evidence excludes credentials, Secrets, kubeconfig content, prompts, responses, logs, and model weights.
• Failures remain evidence. Validation records unhealthy live state instead of silently changing a host or cluster to make a test green.

The project family is exercised on Ubuntu 24.04 with k3s and an NVIDIA GeForce 940M using low-memory Qwen 1.8B, Gemma 3 1B, and Llama 3.2 1B profiles. Models run sequentially with partial CUDA layer offload, system-RAM fallback, and a 900 MiB observed VRAM ceiling. This is a constrained-edge reference, not a claim of fleet-scale production capacity.

Ubuntu host
└── k3s + containerd
    ├── NVIDIA runtime + GPU resource
    ├── Ollama + selectable Qwen, Gemma, or Llama GGUF model
    ├── Open WebUI
    └── OpenTelemetry → Prometheus → Grafana

Point-in-time claims belong in versioned evidence, not marketing copy. Browse edge-llm-tests for the cross-project matrix and gguf-observability for the model-runtime contract.

Use the unified control plane for new deployments:

git clone https://github.com/Edge-Computing-LLM/edge-cli.git
cd edge-cli
go build -o edge ./cmd/edge
./edge doctor
./edge install all --accelerator auto --yes
./edge validate infra
./edge validate observability

Review the command output and repository documentation before allowing host or cluster changes. For a source-only introduction, begin with the edge-cli README and then follow the Layer 1 and Layer 2 links above.

1. Prefer dependency-light Go control and diagnostic tooling.
2. Keep deployment ownership in Helm and Kubernetes-native APIs.
3. Make accelerator selection explicit and CPU fallback honest.
4. Publish reproducible, sanitized evidence with clear timestamps and limits.
5. Never commit credentials, model weights, private prompts, or responses.

Every repository has an explicit ownership boundary and repository-specific validation commands. Start with the shared CONTRIBUTING.md, use the structured issue and pull-request templates, and follow the Code of Conduct. Operational questions belong in the repository that owns the affected layer; security-sensitive reports must follow the private guidance in SECURITY.md.

Build small. Observe deeply. Keep the model close to the data.