Skip to content

Commit 35038a0

Browse files
91volt91volt
committed
add object hunting draft docs
1 parent 33f1cf0 commit 35038a0

File tree

2 files changed

+157
-1
lines changed

2 files changed

+157
-1
lines changed

‎examples/object-hunting/README.md‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Object Hunting
2+
3+
The **Object Hunting** is an interactive scavenger hunt that uses real-time object detection. Players must locate specific physical objects in their environment using a USB camera connected to the Arduino UNO Q to win the game.
4+
5+
![Object Hunting Game Example](assets/docs_assets/thumbnail.png)
6+
7+
## Description
8+
9+
This App creates an interactive game that recognizes real-world objects. It utilizes the `video_objectdetection` Brick to stream video from a USB webcam and perform continuous inference. The web interface challenges the user to find five specific items: **Book, Bottle, Chair, Cup, and Cell Phone**.
10+
11+
Key features include:
12+
13+
- Real-time video streaming and object recognition
14+
- Interactive checklist that updates automatically when items are found
15+
- Confidence threshold adjustment to tune detection sensitivity
16+
- "Win" state triggering upon locating all target objects
17+
18+
## Bricks Used
19+
20+
The object hunting game example uses the following Bricks:
21+
22+
- `web_ui`: Brick to create the interactive game interface and handle WebSocket communication.
23+
- `video_objectdetection`: Brick that manages the USB camera stream, runs the machine learning model, and provides real-time detection results.
24+
25+
## Hardware and Software Requirements
26+
27+
### Hardware
28+
29+
- Arduino UNO Q (x1)
30+
- **USB-C® hub with external power (x1)**
31+
- A power supply (5 V, 3 A) for the USB hub (x1)
32+
- **USB Webcam** (x1)
33+
34+
### Software
35+
36+
- Arduino App Lab
37+
38+
**Important:** A **USB-C® hub is mandatory** for this example to connect the USB Webcam. Consequently, this example must be run in **[Network Mode](learn/network-mode)** or **[SBC Mode](learn/single-board-computer)**.
39+
40+
**Note:** You must connect the USB camera **before** running the App. If the camera is not connected or not detected, the App will fail to start.
41+
42+
## How to Use the Example
43+
44+
1. **Hardware Setup**
45+
Connect your **USB Webcam** to a powered **USB-C® hub** attached to the UNO Q. Ensure the hub is powered to support the camera.
46+
47+
2. **Run the App**
48+
Launch the App from Arduino App Lab.
49+
*Note: If the App stops immediately after clicking Run, check your USB camera connection.*
50+
51+
3. **Access the Web Interface**
52+
Open the App in your browser at `<UNO-Q-IP-ADDRESS>:7000`. The interface will load, showing the game introduction and the video feed placeholder.
53+
54+
4. **Start the Game**
55+
Click the **Start Game** button. The interface will switch to the gameplay view, displaying the live video feed and the list of objects to find.
56+
57+
5. **Hunt for Objects**
58+
Point the camera at the required items (Book, Bottle, Chair, Cup, Cell Phone). When the system detects an object with sufficient confidence, it will automatically mark it as "Found" in the UI.
59+
60+
6. **Adjust Sensitivity**
61+
If the camera is not detecting objects easily, or is detecting them incorrectly, use the **Confidence Level** slider on the right.
62+
- **Lower value:** Detects objects more easily but may produce false positives.
63+
- **Higher value:** Requires a clearer view of the object to trigger a match.
64+
65+
7. **Win the Game**
66+
Once all five objects are checked off the list, a "You found them all!" screen appears. You can click **Play Again** to reset the list and restart.
67+
68+
## How it Works
69+
70+
Once the App is running, it performs the following operations:
71+
72+
- **Video Streaming**: The `video_objectdetection` Brick captures video from the USB camera and hosts a low-latency stream on port `4912`. The frontend embeds this stream via an `<iframe>`.
73+
- **Inference**: The backend continuously runs an object detection model on the video frames.
74+
- **Event Handling**: When objects are detected, the backend sends the labels to the frontend via WebSockets.
75+
- **Game Logic**: The frontend JavaScript compares the received labels against the target list (`['book', 'bottle', 'chair', 'cup', 'cell phone']`) and updates the game state.
76+
77+
## Understanding the Code
78+
79+
### 🔧 Backend
80+
81+
The Python® script initializes the detection engine and bridges the communication between the computer vision model and the web UI.
82+
83+
- **Initialization**: Sets up the WebUI and the Video Object Detection engine.
84+
- **Threshold Control**: Listens for `override_th` messages from the UI to adjust how strict the model is when identifying objects.
85+
86+
```python
87+
ui = WebUI()
88+
detection_stream = VideoObjectDetection()
89+
90+
# Allow the slider in the UI to change detection sensitivity
91+
ui.on_message("override_th", lambda sid, threshold: detection_stream.override_threshold(threshold))
92+
```
93+
94+
- **Reporting Detections**: The script registers a callback using `on_detect_all`. Whenever the model identifies objects, this function iterates through them and sends the labels to the frontend.
95+
96+
```python
97+
def send_detections_to_ui(detections: dict):
98+
for key, value in detections.items():
99+
entry = {
100+
"content": key,
101+
"timestamp": datetime.now(UTC).isoformat()
102+
}
103+
ui.send_message("detection", message=entry)
104+
105+
detection_stream.on_detect_all(send_detections_to_ui)
106+
```
107+
108+
### 🔧 Frontend
109+
110+
The web interface handles the game logic, video embedding, and user feedback.
111+
112+
- **Video Embedding**: The HTML loads the raw video stream provided by the backend on a specific port.
113+
114+
```javascript
115+
const targetPort = 4912;
116+
const streamUrl = `http://${currentHostname}:${targetPort}/embed`;
117+
```
118+
119+
- **Game State Management**: The JavaScript defines the targets and tracks progress.
120+
121+
```javascript
122+
const targetObjects = ['book', 'bottle', 'chair', 'cup', 'cell phone'];
123+
let foundObjects = [];
124+
```
125+
126+
- **Processing Detections**: When the backend sends a `detection` event, the script checks if the detected object is in the target list and hasn't been found yet. If it matches, it updates the UI and checks for a win condition.
127+
128+
```javascript
129+
function handleDetection(detection) {
130+
const detectedObject = detection.content.toLowerCase();
131+
132+
// Check if the detected item is a target and not yet found
133+
if (targetObjects.includes(detectedObject) && !foundObjects.includes(detectedObject)) {
134+
foundObjects.push(detectedObject);
135+
136+
// Update UI to show item as found
137+
const foundItem = document.getElementById(`obj-${detectedObject}`);
138+
foundItem.classList.add('found');
139+
140+
updateFoundCounter();
141+
checkWinCondition();
142+
}
143+
}
144+
```
145+
146+
- **Win Condition**:
147+
148+
```javascript
149+
function checkWinCondition() {
150+
if (foundObjects.length === targetObjects.length) {
151+
gameStarted = false;
152+
// Hide video, show win screen
153+
videoFeedContainer.classList.add('hidden');
154+
winScreen.classList.remove('hidden');
155+
}
156+
}
157+
```

‎examples/object-hunting/app.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
name: Object Hunting
22
icon: 🔍
33
description: Detect a list of object to win the game
4-
54
bricks:
65
- arduino:video_object_detection
76
- arduino:web_ui

0 commit comments

Comments
 (0)