Control your mouse cursor and trigger mouse and keyboard actions with hand gestures captured through your webcam.
Turns your webcam into a virtual mouse controller. Hand detection runs entirely through OpenCV's own cv2.dnn module against two small ONNX models (a palm detector and a 21-point hand landmark regressor, converted by the OpenCV Zoo from Google's MediaPipe hand models) — no mediapipe package required, keeping the dependency footprint to just OpenCV and PyAutoGUI.
Gesture reference:
| Gesture | Action |
|---|---|
| Move index finger | Move cursor |
| Quick thumb curl into the palm | Left click |
| Hold thumb curled (0.35s+) | Click and hold (drag) |
| Closed fist (all 5 fingers curled), held ~0.4s | Copy (Ctrl+C) |
| Open palm (all 5 fingers extended), held ~0.4s | Paste (Ctrl+V) |
The index finger never takes part in the click gesture — it only drives the cursor. Curling the thumb toward the palm doesn't move the index fingertip, so clicking never nudges the cursor off target. Copy/paste require the whole hand to commit to a fist or an open palm, which keeps them clearly distinct from pointing (index finger only) and clicking (thumb only) — there's no natural real-world gesture for copy/paste (per HCI gesture-elicitation research), so this leans on the closest physical metaphor available: grab to take a copy, open the hand to release it. Both fire once per sustained pose and re-arm only once the hand returns to a neutral shape, so holding the pose doesn't repeat-fire. All gesture states are derived purely from landmark geometry (fingertip-to-wrist vs. knuckle-to-wrist distance), no external classifier involved.
Cursor movement is smoothed with an exponential moving average to reduce jitter. The app displays handedness, detection confidence, and the current action state on screen to help with calibration (src/hand_control/config.py).
main.py— thin entrypoint that runs the capture loopsrc/hand_control/— camera setup, palm/hand detection (ONNX), gesture classification, action mapping, cursor smoothing, and config constantssrc/hand_control/assets/— small pre-computed data (SSD anchor grid) bundled with the sourcemodels/— palm detector and hand landmark ONNX models, downloaded automatically on first run (not committed to git)
pip install -r requirements.txt
python main.pyA webcam window will open. On first run, the two detection models (~8 MB combined) are downloaded automatically to models/. Place your hand in front of the camera to start controlling the mouse.
This project is licensed under the MIT License.
The bundled palm/hand detection code is adapted from OpenCV Zoo (Apache 2.0 License).
