Workflow for creating a supermarket security system.
Currently, supermarkets such as Sainsbury's use receipts to determine whether a person has paid. Not only is this finnicky and not very secure (since any receipt will do), it also wastes paper as it is made necessary to print the receipts, which are often thrown away immediately. One solution is to produce an online receipt of your purchases that is integrated within the smartshop app. This also further encourages users to have a nectar card. The barcode on the virtual receipt can be scanned by the gate (for which the functionality already exists). This virtual receipt can also allow users to easily keep track of shopping, query unexpected items, and simplify returns. However, this doesn't use Computer Vision. Therefore, another (less useful and less reliable) solution makes use of the cameras at the self-checkout tills. The person's face profile is stored within a database from the moment they enter the store and the system is alerted if they are seen on a camera exiting the supermarket without the payment tag being true. While visible on the camera at the self-checkout, when a payment is made, this is updated on the customer's profile. When the payment is made, the customer's profile is deleted from the database. This system can also be used to keep track of offenders as they can remain on the system and can be highlighted on cameras.
Initially, I would like to explore the capabilities of OpenCV in:
- Face detection
- Face recognition
... and (though unnecessary for this project) potentially
- Pose detection
- Pose classification
The aim is to produce suitably functioning programs for each of these key capabilities in order to explore the possibilities before designing the game mechanics.
For facial detection, I chose to work with YuNet.
YuNet is a Convolutional Neural Network (CNN)-based pretrained facial detection model. Its main advantages are:
- It is within the OpenCV ecosystem, simplifying the overall pipeline.
- It is open source.
- It provides efficient real-time face detection.
The model achieves:
- 0.834 AP_easy
- 0.824 AP_medium
- 0.708 AP_hard
on the WIDER Face validation set.
The pretrained model is available through the OpenCV Zoo repository on GitHub.
- Initialise webcam as the input source.
- Initialise the
VideoWriterobject:- Frame rate
- Frame size
- Codec
- Output filename
- Configure the output video window.
- Load the YuNet face detection model:
- Download pretrained model weights (
.onnxfile). - Initialise the YuNet detector.
- Download pretrained model weights (
- Set the model input size.
For each frame:
-
Read frame and validate input.
-
Model-specific operations
- Prepare frame input for the model.
- Run face detection.
Note: YuNet only requires the input size to be set. This is fixed for the webcam stream and is therefore configured outside the processing loop.
-
Application logic
- Process detection results.
- Draw annotations.
- Display frame.
- Save frame.
- Check for exit condition (e.g. key press).
Release resources:
- Release webcam input.
- Release
VideoWriter. - Close display windows.
For face recognition, I chose to work with SFace.
SFace is a pretrained deep learning face recognition model that works by extracting a feature embedding, which is a high-dimensional vector that encodes distinguishing features of the face. These face embeddings can then be compared using similarity metrics such as cosine similarity or L2 distance to recognise the same person between images. Its main advantages are:
- It is within the OpenCV ecosystem, simplifying the overall pipeline.
- It builds upon face detection using YuNet
- It is open source.
There are three models available, achieving:
- SFace 0.9940
- SFace block 0.9942
- SFace quant 0.9932
While the baseline model boasts the highest accuracy, since the model's weights are stored in a higher precision format (FP32) while the quantized counterparts use a less precise INT8 format, the quantized variants are said to be 2-4× faster. The greater speed is ideal for webcam face recognition, however, I have decided to test the capabilities of all three models.
The pretrained model is available through the OpenCV Zoo repository on GitHub.
- Initialise webcam as the input source.
- Configure the output video window.
- Initialise the
VideoWriterobject:- Frame rate
- Frame size
- Codec
- Output filename
- Load the YuNet face detection model:
- Download pretrained model weights (
.onnxfile). - Initialise the YuNet detector object.
- Download pretrained model weights (
- Load the SFace face recognition model:
- Download pretrained model weights (
.onnxfile). - Initialise the SFace recognition object.
- Download pretrained model weights (
- Set the model input size.
For each frame:
-
Read frame and validate input.
-
YuNet Model-specific operations
- Prepare frame input for the model.
- Run face detection.
Note: YuNet only requires the input size to be set. This is fixed for the webcam stream and is therefore configured outside the processing loop.
-
Run face recognition on each face
-
Application logic
- Process detection and recognition results.
- Draw annotations.
- Display frame.
- Save frame.
- Check for exit condition (e.g. key press).
Release resources:
- Release webcam input.
- Release
VideoWriter. - Close display windows.