|
| 1 | +# Object Hunting |
| 2 | + |
| 3 | +The **Object Hunting Game** is an interactive scavenger hunt that uses real-time object detection. Players must locate specific physical objects in their environment using a USB camera connected to the Arduino UNO Q to win the game. |
| 4 | + |
| 5 | +**Note:** This example requires to be run using **Network Mode** or **Single-Board Computer (SBC) Mode**, since it requires a **USB-C® hub** and a **USB webcam**. |
| 6 | + |
| 7 | + |
| 8 | + |
| 9 | +## Description |
| 10 | + |
| 11 | +This App creates an interactive game that recognizes real-world objects. It utilizes the `video_objectdetection` Brick to stream video from a USB webcam and perform continuous inference using the **YoloX Nano** model. The web interface challenges the user to find five specific items: **Book, Bottle, Chair, Cup, and Cell Phone**. |
| 12 | + |
| 13 | +**Key features include:** |
| 14 | + |
| 15 | +- Real-time video streaming and object recognition |
| 16 | +- Interactive checklist that updates automatically when items are found |
| 17 | +- Confidence threshold adjustment to tune detection sensitivity |
| 18 | +- "Win" state triggering upon locating all target objects |
| 19 | + |
| 20 | +## Bricks Used |
| 21 | + |
| 22 | +The object hunting game example uses the following Bricks: |
| 23 | + |
| 24 | +- `web_ui`: Brick to create the interactive game interface and handle WebSocket communication. |
| 25 | +- `video_objectdetection`: Brick that manages the USB camera stream, runs the machine learning model, and provides real-time detection results. |
| 26 | + |
| 27 | +## Hardware and Software Requirements |
| 28 | + |
| 29 | +### Hardware |
| 30 | + |
| 31 | +- Arduino UNO Q (x1) |
| 32 | +- **USB-C® hub with external power (x1)** |
| 33 | +- A power supply (5 V, 3 A) for the USB hub (x1) |
| 34 | +- **USB Webcam** (x1) |
| 35 | + |
| 36 | +### Software |
| 37 | + |
| 38 | +- Arduino App Lab |
| 39 | +**Important:** A **USB-C® hub is mandatory** for this example to connect the USB Webcam. |
| 40 | +**Note:** You must connect the USB camera **before** running the App. If the camera is not connected or not detected, the App will fail to start. |
| 41 | + |
| 42 | +## How to Use the Example |
| 43 | + |
| 44 | +1. **Hardware Setup** |
| 45 | + Connect your **USB Webcam** to a powered **USB-C® hub** attached to the UNO Q. Ensure the hub is powered to support the camera. |
| 46 | +  |
| 47 | + |
| 48 | +2. **Run the App** |
| 49 | + Launch the App from Arduino App Lab. |
| 50 | + *Note: If the App stops immediately after clicking Run, check your USB camera connection.* |
| 51 | +  |
| 52 | + |
| 53 | +3. **Access the Web Interface** |
| 54 | + Open the App in your browser at `<UNO-Q-IP-ADDRESS>:7000`. The interface will load, showing the game introduction and the video feed placeholder. |
| 55 | + |
| 56 | +4. **Start the Game** |
| 57 | + Click the **Start Game** button. The interface will switch to the gameplay view, displaying the live video feed and the list of objects to find. |
| 58 | + |
| 59 | +5. **Hunt for Objects** |
| 60 | + Point the camera at the required items (Book, Bottle, Chair, Cup, Cell Phone). When the system detects an object with sufficient confidence, it will automatically mark it as "Found" in the UI. |
| 61 | + |
| 62 | +6. **Adjust Sensitivity** |
| 63 | + If the camera is not detecting objects easily, or is detecting them incorrectly, use the **Confidence Level** slider on the right. |
| 64 | + - **Lower value:** Detects objects more easily but may produce false positives. |
| 65 | + - **Higher value:** Requires a clearer view of the object to trigger a match. |
| 66 | + |
| 67 | +7. **Win the Game** |
| 68 | + Once all five objects are checked off the list, a "You found them all!" screen appears. You can click **Play Again** to reset the list and restart. |
| 69 | + |
| 70 | +## How it Works |
| 71 | + |
| 72 | +The application relies on a continuous data pipeline between the hardware, the inference engine, and the web browser. |
| 73 | + |
| 74 | +**High-level data flow:** |
| 75 | + |
| 76 | +``` |
| 77 | + USB Camera ──► VideoObjectDetection ──► Inference Model (YoloX) |
| 78 | + │ │ |
| 79 | + │ (MJPEG Stream) │ (Detection Events) |
| 80 | + ▼ ▼ |
| 81 | + Frontend (Browser) ◄── WebUI Brick |
| 82 | + │ |
| 83 | + └──► WebSocket (Threshold Control) |
| 84 | +``` |
| 85 | + |
| 86 | +- **Video Streaming**: The `video_objectdetection` Brick captures video from the USB camera and hosts a low-latency stream on port `4912`. The frontend embeds this stream via an `<iframe>`. |
| 87 | +- **Inference**: The backend continuously runs the **YoloX Nano** object detection model on the video frames. |
| 88 | +- **Event Handling**: When objects are detected, the backend sends the labels to the frontend via WebSockets. |
| 89 | +- **Game Logic**: The frontend JavaScript compares the received labels against the target list and updates the game state. |
| 90 | + |
| 91 | +## Understanding the Code |
| 92 | + |
| 93 | +### 🔧 Backend (`main.py`) |
| 94 | + |
| 95 | +The Python script initializes the detection engine and bridges the communication between the computer vision model and the web UI. |
| 96 | + |
| 97 | +- **Initialization**: Sets up the WebUI and the Video Object Detection engine. |
| 98 | +- **Threshold Control**: Listens for `override_th` messages from the UI to adjust how strict the model is when identifying objects. |
| 99 | + |
| 100 | +```python |
| 101 | +ui = WebUI() |
| 102 | +detection_stream = VideoObjectDetection() |
| 103 | + |
| 104 | +# Allow the slider in the UI to change detection sensitivity |
| 105 | +ui.on_message("override_th", lambda sid, threshold: detection_stream.override_threshold(threshold)) |
| 106 | +``` |
| 107 | + |
| 108 | +- **Reporting Detections**: The script registers a callback using `on_detect_all`. Whenever the model identifies objects, this function iterates through them and sends the labels to the frontend. |
| 109 | + |
| 110 | +```python |
| 111 | +def send_detections_to_ui(detections: dict): |
| 112 | + for key, value in detections.items(): |
| 113 | + entry = { |
| 114 | + "content": key, |
| 115 | + "timestamp": datetime.now(UTC).isoformat() |
| 116 | + } |
| 117 | + ui.send_message("detection", message=entry) |
| 118 | + |
| 119 | +detection_stream.on_detect_all(send_detections_to_ui) |
| 120 | +``` |
| 121 | + |
| 122 | +### 🔧 Frontend (`app.js`) |
| 123 | + |
| 124 | +The web interface handles the game logic. It defines the specific objects required to win the game. |
| 125 | + |
| 126 | +```javascript |
| 127 | +const targetObjects = ['book', 'bottle', 'chair', 'cup', 'cell phone']; |
| 128 | +let foundObjects = []; |
| 129 | + |
| 130 | +function handleDetection(detection) { |
| 131 | + const detectedObject = detection.content.toLowerCase(); |
| 132 | + |
| 133 | + // Check if the detected item is a target and not yet found |
| 134 | + if (targetObjects.includes(detectedObject) && !foundObjects.includes(detectedObject)) { |
| 135 | + foundObjects.push(detectedObject); |
| 136 | + updateFoundCounter(); |
| 137 | + checkWinCondition(); |
| 138 | + } |
| 139 | +} |
| 140 | +``` |
| 141 | + |
| 142 | +### 🛠️ Customizing the Game |
| 143 | + |
| 144 | +The default model used by the `video_objectdetection` Brick is **YoloX Nano**, trained on the **COCO dataset**. This means the camera can detect approximately 80 different types of objects, not just the five used in this example. |
| 145 | + |
| 146 | +**To change the objects you want to hunt:** |
| 147 | + |
| 148 | +1. **Choose new targets**: You can select any object from the [standard COCO dataset list](https://github.com/amikelive/coco-labels/blob/master/coco-labels-2014_2017.txt) (e.g., `person`, `keyboard`, `mouse`, `backpack`, `banana`). |
| 149 | +2. **Update the code**: Open `assets/app.js` and locate the `targetObjects` array: |
| 150 | + ```javascript |
| 151 | + const targetObjects = ['book', 'bottle', 'chair', 'cup', 'cell phone']; |
| 152 | + ``` |
| 153 | +3. **Replace the items**: Substitute the strings with your chosen object names from the COCO list. |
| 154 | + ```javascript |
| 155 | + const targetObjects = ['person', 'keyboard', 'mouse', 'laptop', 'backpack']; |
| 156 | + ``` |
| 157 | +4. (Optional) Update `assets/index.html` to change the icons and text displayed in the game introduction to match your new targets. |
| 158 | + |
| 159 | +## Troubleshooting |
| 160 | + |
| 161 | +### App fails to start or stops immediately |
| 162 | +If the application crashes right after launching, it is likely because the **USB Camera** is not detected. |
| 163 | +**Fix:** |
| 164 | +1. Ensure the camera is connected to a **powered USB-C hub**. |
| 165 | +2. Verify the hub has its external power supply connected (5 V, 3 A). |
| 166 | +3. Reconnect the camera and try running the App again. |
| 167 | + |
| 168 | +### Video stream is black or not loading |
| 169 | +If the game interface loads but the video area remains black or shows "Searching Webcam...": |
| 170 | +- **Browser Security:** Some browsers block mixed content or insecure frames. Ensure you are not blocking the iframe loading from port `4912`. |
| 171 | +- **Network:** Ensure your computer and the UNO Q are on the same network. |
| 172 | +- **Camera Status:** If the camera was disconnected while the App was running, you must restart the App. |
| 173 | + |
| 174 | +### Objects are not being detected |
| 175 | +If you are pointing the camera at an object but it doesn't register: |
| 176 | +- **Check the list:** Ensure the object is one of the targets defined in `app.js`. |
| 177 | +- **Adjust Confidence:** Lower the **Confidence Level** slider. If set too high (e.g., > 0.80), the model requires a perfect angle to trigger a detection. |
| 178 | +- **Lighting:** Ensure the object is well-lit. Shadows or darkness can prevent detection. |
| 179 | +- **Distance:** Move the camera closer or further away. The object should occupy a significant portion of the frame. |
0 commit comments