Skip to content

Commit 1272352

Browse files
vijaykr458vijaykr338adrianboguszewskiCopilot
authored
Added Gesture Control with OpenVINO (#393)
* Added Gesture Control with OpenVINO * changed the directory * added a readme * fixed the utils files, added the models * removed QR * Update apps/gesture_control/README.md Co-authored-by: Adrian Boguszewski <[email protected]> * Update apps/gesture_control/src/openvino_models.py Co-authored-by: Copilot <[email protected]> * Update apps/README.md Co-authored-by: Adrian Boguszewski <[email protected]> * Update apps/gesture_control/src/benchmark_dialog.py Co-authored-by: Copilot <[email protected]> * Update apps/gesture_control/src/gesture_engine.py Co-authored-by: Copilot <[email protected]> * Update hand_landmark.py * changed dir_name to palm_pilot * Update apps/palm_pilot/README.md * Update requirements.txt * Update README.md --------- Co-authored-by: Vijay Kumar <[email protected]> Co-authored-by: Adrian Boguszewski <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Adrian Boguszewski <[email protected]>
1 parent 219efb9 commit 1272352

33 files changed

+35109
-0
lines changed

‎apps/README.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,18 @@ The app blurs the background behind a person on a webcam, and it's built using N
1818
| Platforms | Client (CPU, GPU, NPU), Edge (CPU) |
1919
| Owner | [@Roszczyk](https://github.com/Roszczyk), [@adrianboguszewski](https://github.com/adrianboguszewski) |
2020

21+
### 🖐️ PalmPilot - Gesture Control
22+
23+
Control your computer with hand gestures using real-time hand detection. Features multiple application modes including presentation control, media playback, gaming, and browser navigation.
24+
25+
[![PalmPilot - Gesture Control Demo](https://img.youtube.com/vi/U29NRoX4sjk/maxresdefault.jpg)](https://www.youtube.com/watch?v=U29NRoX4sjk)
26+
27+
| [PalmPilot - Gesture Control](palm_pilot) | |
28+
|--------------------------------------|------------------------------------------------------|
29+
| Related AI concepts | hand detection, gesture recognition, computer vision |
30+
| Platforms | Client (CPU, GPU), Edge (CPU) |
31+
| Owner | [@vijaykr338](https://github.com/vijaykr338) |
32+
2133
### 🚦 Smart Intersection Traffic Monitoring System
2234

2335
The Smart Intersection Traffic Monitoring System is an advanced AI-powered desktop application built with PySide6 and OpenVINO for real-time traffic monitoring, violation detection, and smart intersection analytics. The system combines computer vision, machine learning, and IoT technologies to provide comprehensive traffic management capabilities with enterprise-grade features.

‎apps/palm_pilot/README.md‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
# PalmPilot - Gesture Control with OpenVINO™
2+
3+
Control your computer with hand gestures using real-time hand detection and OpenVINO AI models. Perfect for presentations, media control, gaming, and system navigation.
4+
5+
[![PalmPilot Demo](https://img.youtube.com/vi/U29NRoX4sjk/maxresdefault.jpg)](https://www.youtube.com/watch?v=U29NRoX4sjk)
6+
7+
## Quick Launch using Setup Scripts
8+
9+
If you want a **quick setup** without manually installing dependencies, use the provided installer scripts. These scripts will **automatically configure** everything needed to run the PalmPilot Demo.
10+
11+
### **For Windows**
12+
1. Download the `install.bat` and `run.bat` files to your local directory.
13+
2. Double-click `install.bat` to install dependencies and set up the environment.
14+
3. After installation, double-click `run.bat` to start the demo.
15+
16+
### **For Linux and MacOS**
17+
1. Download the `install.sh` and `run.sh` files to your local directory.
18+
2. First, ensure the installer scripts have execute permissions:
19+
```bash
20+
chmod +x install.sh run.sh
21+
```
22+
3. Run the installer to set up everything:
23+
```bash
24+
./install.sh
25+
```
26+
4. After installation, start the demo by running:
27+
```bash
28+
./run.sh
29+
```
30+
31+
These scripts will handle cloning the repository, creating the virtual environment, and installing dependencies automatically. If you prefer a manual setup, follow Steps 1-3 below.
32+
33+
## Manual Environment Setup
34+
35+
Here are the steps involved in this demo:
36+
- Step 1: Install Python and prerequisites
37+
- Step 2: Set up the environment
38+
- Step 3: Run the Application
39+
40+
Now, let's dive into the steps starting with installing Python.
41+
42+
## Step 0
43+
Star the [repository](https://github.com/openvinotoolkit/openvino_build_deploy) (optional, but recommended)
44+
45+
## Step 1
46+
This project requires Python 3.11 and a few libraries. If you don't have Python installed on your machine, go to [https://www.python.org/downloads/](https://www.python.org/downloads/) and download the latest version for your operating system. Follow the prompts to install Python, making sure to check the option to add Python to your PATH environment variable.
47+
48+
Install libraries and tools:
49+
```bash
50+
sudo apt install git python3-venv python3-dev
51+
```
52+
_NOTE: If you are using Windows, you may need to install [Microsoft Visual C++ Redistributable](https://aka.ms/vs/16/release/vc_redist.x64.exe) also._
53+
54+
## Step 2
55+
56+
### 1. Clone the Repository
57+
To clone the repository, run the following command:
58+
```bash
59+
git clone https://github.com/openvinotoolkit/openvino_build_deploy.git
60+
```
61+
The above will clone the repository into a directory named "openvino_build_deploy" in the current directory. Then, navigate into the directory using the following command:
62+
```bash
63+
cd openvino_build_deploy/apps/palm_pilot
64+
```
65+
66+
### 2. Create a virtual environment
67+
To create a virtual environment, open your terminal or command prompt and navigate to the directory where you want to create the environment. Then, run the following command:
68+
```bash
69+
python3 -m venv venv
70+
```
71+
This will create a new virtual environment named "venv" in the current directory.
72+
73+
### 3. Activate the environment
74+
Activate the virtual environment using the following command:
75+
```bash
76+
source venv/bin/activate # For Unix-based operating system such as Linux or macOS
77+
```
78+
_NOTE: If you are using Windows, use `venv\Scripts\activate` command instead._
79+
80+
This will activate the virtual environment and change your shell's prompt to indicate that you are now working within that environment.
81+
82+
### 4. Install the Packages
83+
To install the required packages, run the following commands:
84+
```bash
85+
python -m pip install --upgrade pip
86+
pip install -r requirements.txt
87+
```
88+
89+
## Step 3
90+
91+
To run the application with webcam input:
92+
```bash
93+
python main.py --stream 0
94+
```
95+
96+
**Using the GUI:**
97+
1. **Launch the GUI** - The main dashboard will open with the "Gesture Engine Offline" status
98+
2. **Start the Engine** - Click the "Start Engine" button in the Engine Controls section
99+
3. **Select Mode** - Choose from the available gesture modes:
100+
- Browser Mode
101+
- Game Mode (Racing)
102+
- Media Player Mode
103+
- PowerPoint Mode
104+
- Volume Control Mode
105+
4. **Configure Gestures** - Use the "Edit Mode" button to customize gesture mappings for each mode
106+
5. **Dock Mode** - After starting the engine, a compact dock window will open showing:
107+
- Live camera feed with gesture detection
108+
- Current mode selector dropdown
109+
- Pause/Stop controls
110+
- QR code for mobile access
111+
112+
### Command Line Options
113+
The application currently supports the following option:
114+
```bash
115+
python main.py --stream SOURCE
116+
```
117+
118+
Where SOURCE can be:
119+
- `0` for default webcam
120+
- Path to a video file for video input
121+
122+
## Supported Gestures
123+
124+
- **Open Hand**: Cursor control
125+
- **Closed Fist**: Click actions
126+
- **Peace Sign**: Scroll mode
127+
- **Thumbs Up/Down**: Volume control
128+
- **Two-Hand Steering**: Racing game control
129+
- **Index Finger Bent**: Navigation controls
130+
- **I Love You Sign**: Special actions
131+
132+
## Application Modes
133+
134+
### Browser Mode
135+
**Left Hand Gestures:**
136+
- **Bending Index Finger**: Left mouse click
137+
- **Bending Index + Middle Finger**: Right mouse click
138+
- **Closed Fist**: Speech to Text (Windows + H)
139+
- **I Love You Sign**: Tab switcher (Ctrl + Shift + Tab)
140+
141+
**Right Hand Features:**
142+
- **Cursor Control**: Controlled by tip of index finger
143+
- **Scrolling**: Scroll up/down gestures
144+
- **I Love You Gesture**: Toggles cursor to scroll mode
145+
146+
### Media Player Mode
147+
**Left Hand Gestures:**
148+
- **Bending Index Finger**: Mute (M key)
149+
- **Bending Index + Middle Finger**: Skip 10 seconds backwards
150+
151+
**Right Hand Gestures:**
152+
- **Bending Index Finger**: Play/Pause (Spacebar)
153+
- **Bending Index + Middle Finger**: Skip 10 seconds forward
154+
155+
**Volume Control:**
156+
- **Pinch Out**: Increase volume
157+
- **Pinch In**: Decrease volume
158+
159+
### Racing Game Mode
160+
- **Two-hand steering control**: Uses hand detector model for faster responsiveness
161+
- **Steering Logic**:
162+
- Left hand lower than right hand = Turn left (φ <= -13)
163+
- Right hand lower than left hand = Turn right (φ >= 13)
164+
- **Natural steering feel**: Mimics real steering wheel interaction
165+
- **Accelerate/Brake zones**: Based on hand positioning
166+
167+
For detailed information about all available modes and their gesture mappings, visit our comprehensive documentation at: [https://palm-pilot-docs.vercel.app/docs/gesture-control-system/modes](https://palm-pilot-docs.vercel.app/docs/gesture-control-system/modes)
168+
169+
## Configuration
170+
171+
The application uses a JSON configuration file (`gesture_config.json`) that is automatically created on first run. You can customize:
172+
173+
- Gesture mappings and actions
174+
- Detection parameters
175+
- Application-specific modes
176+
- Device settings
177+
178+
## Requirements
179+
180+
- **Python**: 3.11
181+
- **OpenVINO**: 2025.2+
182+
- **OpenCV**: 4.10+
183+
- **Camera**: USB webcam or built-in camera
184+
- **OS**: Windows 10/11, Ubuntu 20.04+, macOS 10.15+
185+
186+
[//]: # (telemetry pixel)
187+
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=7003a37c-568d-40a5-9718-0d021d8589ca&project=apps/palm_pilot&file=README.md" />

0 commit comments

Comments
 (0)