Video Link: ShopkAIper YT Demo
- Anannyo Dey
- Debasmit Roy
- Kanko Ghosh
- Aditya Ganguly
- Kabir Raj Singh
- Linux (Ubuntu Latest Version)
- Python 3.9 or higher
- Clone the repository
- Run the following command in the terminal:
pip install python-opencv
pip install numpy
pip install requests
Follow the instructions in the link below to install Network Optix:
Follow the instructions in the link below to install Nx AI Plugin:
sudo chmod -R 777 /opt/networkoptix-metavms/mediaserver/bin/plugins/nxai_plugin
systemctl restart networkoptix-metavms-mediaserver.service
Follow the instructions in the link below to install NX Meta:
Register and login to the NX Meta application.
- Run the following command in the terminal:
cd camera_server
python3 flaskk.py
The camera server will start running on port 8081. Then setup the camera in the Nx Meta application. Camera1: http://localhost:8081/v2 Camera2: http://localhost:8081/v3
- Run the following command in the terminal:
cd analysis_server
python3 flask_anal_server.py
The analysis server will start running on port 8888. It is hosted on AWS as well. The link is: Analysis Server Backend Change it accordingly in the sclbl-integration-sdk-main/postprocessor-python-cereal/postprocessor-python-cereal.py and sclbl-integration-sdk-main/postprocessor-python-fish/postprocessor-python-fish.py files.
Run the following command in the terminal:
make
cmake .
cp postprocessor-python-cereal/postprocessor-python-cereal /opt/networkoptix-metavms/mediaserver/bin/plugins/nxai_plugin/nxai_manager/postprocessors
cp external_postprocessors.json /opt/networkoptix-metavms/mediaserver/bin/plugins/nxai_plugin/nxai_manager/postprocessors
chmod 777 /opt/networkoptix-metavms/mediaserver/bin/plugins/nxai_plugin/nxai_manager/postprocessors/postprocessor-python-cereal
cp postprocessor-python-fish/postprocessor-python-fish /opt/networkoptix-metavms/mediaserver/bin/plugins/nxai_plugin/nxai_manager/postprocessors
cp external_postprocessors.json /opt/networkoptix-metavms/mediaserver/bin/plugins/nxai_plugin/nxai_manager/postprocessors
chmod 777 /opt/networkoptix-metavms/mediaserver/bin/plugins/nxai_plugin/nxai_manager/postprocessors/postprocessor-python-fish
chmod 777 /opt/networkoptix-metavms/mediaserver/bin/plugins/nxai_plugin/nxai_manager/postprocessors/external_postprocessors.json
Check if the files are copied successfully:
ls -la /opt/networkoptix-metavms/mediaserver/bin/plugins/nxai_plugin/nxai_manager/postprocessors
sclbl-integration-sdk-main/postprocessor-python-cereal/postprocessor-python-cereal.py
sclbl-integration-sdk-main/postprocessor-python-fish/postprocessor-python-fish.py
The project is now ready to be used.
- Input: Real-time CCTV footage is processed as input.
- Product Placement Data: Admins assign bounding boxes for each product group on the shelves under each camera’s field of view.
- Pose-Detection And Object Detection: Using the YOLO-v8-pose (ONNX) model, we detect bounding boxes and body key points for each person, focusing on the 9th and 10th points (left and right wrist) for further processing.
- Post Processing: Detection results go to the NX-Postprocessors, which apply non-max suppression and confidence thresholding, then pass the data to the NX-client for visualizing the boxes and to the Analysis server for generating insights.
- Client Integration: The NX-Meta App receives the bounding box feed from NX-cloud and displays them.
- Analysis Server: The Analysis server generates insights such as Frequent Item Sets, Dwell Time Analysis, and Anomaly Detection in product handling patterns.
- Visualization: We visualize the number of interactions for each product group under each camera, along with product grouping visualizations.
- Input: All product placement and customer body-part bounding boxes for each frame.
- Handing Stream of Frames: The analysis server receives frames every two seconds and keeps a 20-second buffer for synchronization.
- Movement & Proximity Analysis: If hand-keypoints are static for a while, surrounding product bounding boxes are evaluated, and holding a product for over 8 seconds counts as an interaction.
- Product Bucket Creation: Product buckets are created for each user based on purchased product combinations. In 30-second windows, all interactions within a product section form a "Bucket" added to the frequently purchased items dataset.
- Apriori Algorithm: These product buckets are then fed into the Apriori algorithm, which generates association rules from sets of products to products. This helps improve product placement strategies.
- Output: Frequent item sets for each section and interaction dwell time for each product category.
Product group-wise dwell time is analyzed as time series data. We then search for sudden spikes in interaction levels, treating them as anomalies. Statistical analysis is used to examine each product section, and anomalies are reported based on the z-score. We also check if that customer is not carrying a cart, in which case these anomalies are notified on the client dashboard as potential thefts.
Dwell time is calculated for each product category by analyzing the time spent by customers in front of each product group. This helps in understanding customer preferences and product placement strategies.
Product buckets are created for each user based on purchased product combinations. In 30-second windows, all interactions within a product section form a "Bucket" added to the frequently purchased items dataset. This dataset is then fed into the Apriori algorithm, which generates association rules from sets of products to products. This helps improve product placement strategies.