English · 繁體中文

PresenceKit

Know when the person you are talking to has left.

PresenceKit combines a camera and a microphone to tell a conversational device (a robot, a kiosk, a smart speaker with a camera) whether the person, or the group, it is talking to is still there. It is a small, explainable pipeline (face and body detection, tracking, an engagement score, a probabilistic presence model and a five-state machine) plus a web demo that shows every step live.

Watch the concept film

Watch the full film with sound — 49 seconds · English subtitles

An animated introduction to quiet presence, group continuity, and how distance informs audio decisions. The waveforms illustrate the intended behavior; they are not recorded benchmark results.

Live web demo

Why

"No speech for 20 seconds" is a poor way to end a conversation. People pause, look at their phone, or listen quietly while a friend talks, and a TV in the background never stops talking. PresenceKit looks at who is actually there and engaged, uses audio only where it helps, and writes down the reason for every decision.

Features

• Five clear states: ACTIVE → IDLE → MAYBE_AWAY → AWAY → ENDED, with hysteresis and a written reason for every transition, plus NEW_GROUP when the next visitor steps up without the room ever being empty.
• Groups, not just faces: every person is tracked; the session lasts while any participant is still there.
• Robust tracking: greedy matching or a Kalman filter with Hungarian assignment, clothing colour and coasting through short occlusions.
• Engagement intent: passers-by and bystanders do not start conversations or keep them alive.
• Probabilistic presence: a per-person probability that shrugs off flickering detections.
• Audio that knows where the voice comes from (optional): PureSound voice isolation, cross-checked against vision so a TV or a distant voice does not count as the user.
• Pose and distance (optional): MoveNet skeletons, orientation, and a calibrated distance estimate.
• Light enough for small devices: motion gate, embedding throttling, adaptive fps and ROI cropping cut CPU by about 70%; TFLite / NPU embedding backend.
• Embeddable SDK on ONNX Runtime: presencekit.sdk.PresenceEngine (frames and VAD or audio in, state and events out) with no web framework; every model runs on ONNX Runtime with selectable execution providers (CPU, CUDA, DirectML, CoreML, QNN, OpenVINO…).
• Private by default: frames are processed in memory and never stored; face fingerprints are erased when a session ends. Cloud speech recognition and cross-session re-ID are optional and off by default.
• Bilingual web UI: English and Traditional Chinese, with live settings, presets, charts and an event log.

Quick start

Requires uv and Python ≥ 3.10.

git clone https://github.com/mcw519/presencekit.git
cd presencekit
uv sync
uv run presencekit            # open http://127.0.0.1:8000 and press Start

Browsers only allow the camera on localhost or HTTPS. Before exposing the server to others, set PRESENCE_TOKEN (see Getting started). In Settings, try the Recommended preset to switch on the newer algorithms.

Optional voice isolation:

uv sync --extra puresound
uv run python scripts/fetch_puresound_model.py --clone

Use it from your own code (no web server; pip install presencekit is enough):

from presencekit.sdk import PresenceEngine

engine = PresenceEngine()
step = engine.process_frame(bgr_frame, speaking=vad_flag)
print(step.snapshot["state"], [e.reason for e in step.events])

How it works

flowchart LR
  CAM[Camera] --> P1[Detection] --> P2[Tracking] --> P3[Intent] --> P4[Presence] --> SM[State machine]
  MIC[Microphone] --> AU[Audio cross-check] --> P4
  P2 -.-> AU

Loading

Each stage has its own page with intuition, formulas, figures and evaluation results. Start with How it works, then dive into detection, tracking, intent, presence, the state machine and audio.

Documentation

The full index is in docs/. Highlights:

Getting started	Install, run, expose safely, optional extras
Configuration	Every parameter, presets, environment variables
Tuning	Adapting to a new room or camera
Performance	CPU cost, latency, NPU backends
Privacy	What is processed, kept and (optionally) sent out
SDK and ONNX Runtime	Embedding PresenceEngine, execution providers, model files, porting
Architecture	Code layout and WebSocket protocol

Project layout

presencekit/     core logic, ONNX Runtime inference layer, SDK facade (sdk.py), FastAPI + WebSocket server
presencekit/static/  web UI (HTML, JS, string tables, vendored PureSound CSS)
scripts/           benchmark, evaluation, model export, figure and doc generators
tests/             unit, scenario, WebSocket, privacy, UI and i18n tests
docs/              documentation (English) and docs/zh-TW/ (Traditional Chinese)

Development

uv run ruff check . && uv run ruff format --check .
uv run pytest -q

Contributions are welcome. Please read CONTRIBUTING.md and the Code of Conduct. Report security issues as described in SECURITY.md. Changes are listed in CHANGELOG.md.

License

Apache License 2.0; see also NOTICE. Model weights and third-party packages keep their own licenses. In particular, the SFace face-embedding weights may carry non-commercial training-data restrictions; read docs/licenses.md before commercial use, or disable SFace with PRESENCE_FACE_EMBEDDER=hist.