Skip to content

Commit 3912709

Browse files
committed
Added the last use case, added some variables notes.
1 parent 946c162 commit 3912709

File tree

1 file changed

+72
-20
lines changed

1 file changed

+72
-20
lines changed

docs/models/models.md

Lines changed: 72 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Ths is a possible implementation of AI model support in AppLab.
1+
This is a possible implementation of AI model support in AppLab.
22

33
# Models
44
In AI, a model is a trained system able to recognize patterns and/or make predictions.
@@ -22,14 +22,15 @@ models/
2222
build#2
2323
```
2424

25-
*NOTE:* We need to discuss how to handle different builds of the same model. In this case, we must ensure we reference the specific build to be run and manage the models' and model definitions' state and deletion accordingly.
25+
*NOTE:* We need to eventually discuss how to handle different builds of the same model. In this case, we must ensure we reference the specific build to be run and manage the model state, model definition state, and deletion accordingly.
2626

2727
*NOTE:* To be discussed how to map Edge Impulse in an AppLab model. See Edge Impulse section below.
2828

29-
The ```conf.yaml``` file contains generic model information. The application expects this data to be available for all models.
29+
The ```conf.yaml``` file contains generic model information. The application expects this data to be available for all models. The ```model_categories``` is a list of the categories of the model.
3030
```
3131
runner: brick
3232
name : "Glass breaking classifier"
33+
model_categories: [audio, video]
3334
model_labels:
3435
- "Background"
3536
- "Glass_Breaking"
@@ -50,8 +51,8 @@ In AI, a category is a common way to classify AI models, indicating the type of
5051

5152
In AppLab, a category is defined as a keyword used to connect models to bricks. It defines which models can be attached to which bricks. Both a model and a brick have a list of categories.
5253

53-
### Use Case (Linking Models to Bricks)
54-
Use Case (link the Figma here): In the brick configuration, we can choose the model to be used with the brick. The provided list is a filtered list of all models where the intersection of the model's category list and the brick's category list is not empty.
54+
### Linking Models to Bricks
55+
In the brick configuration, we can choose the model to be used with the brick. The provided list is a filtered list of all models where the intersection of the model's category list and the brick's category list is not empty.
5556

5657
**We assume here that every model with a given category is able to work with any provider.** If there is a special model that is only able to work with a specific brick, we can always add a custom category to associate the model with the working brick.
5758

@@ -62,12 +63,11 @@ A brick is defined in the brick-list.yaml
6263
```
6364
- id: arduino:audio_classification
6465
name: Audio Classification
65-
category: audio, other_cat
66-
default_model_name: arduino:glass-breaking
66+
categories: audio, other_cat
6767
```
68-
The ```default_model_name``` field was added to specify a default model.
6968

70-
*NOTE:* What should be done if this is a custom model? The concept of a default custom model does not exist.
69+
*NOTE:* categories is a model specific variable, what to do for a brick that does not involve a model?
70+
Somewhere is null, somewhere is "storage"
7171

7272
## App.yaml
7373
This file contains the app's information and configuration.
@@ -221,7 +221,6 @@ source-model-url: "https://aihub.qualcomm.com/models/hey_ardino_spotting_lite"
221221
- microphone
222222
```
223223
*Note:* The following variables are defined in the current implementation but should be removed because they must be hidden from the final user.
224-
We need to discuss whether these variables should be defined here, or if we should leave them hardcoded instead (is this the correct alternative approach?).
225224
```
226225
variables:
227226
- name: CUSTOM_MODEL_PATH
@@ -309,7 +308,7 @@ bricks:
309308
- arduino:fast_object_detection: {}
310309
variables:
311310
CUSTOM_MODEL_PATH: /home/arduino/.arduino-bricks/ei-models
312-
OTHER_ENGINE_VARIABLE: /models/yolo_x_nano/builds/yolo-x-nano.eim
311+
QLC_OBJ_DET_ENGINE_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim
313312
SET_CUSTOM_PARAMETER: 5
314313
icon: 🏞️
315314
```
@@ -372,10 +371,12 @@ bricks:
372371
- arduino:keyword_spotting:
373372
EI_KEYWORD_SPOTTING_MODEL: "/models/hei_arduino_qlc/builds/qualcomm_impulse_hey_arduino.model"
374373
```
374+
## Case #5 Two distinct bricks using the same custom model
375+
This case is the same as Case #3, where the variables used to customize the model are configurable by the user, with an optional default value.
375376

376377
The following section explores whether this model is suitable for specific use cases.
377378

378-
## The user chooses a model for a brick (link Figma here)
379+
## Case #6 The user chooses a model for a brick (link Figma here)
379380
In the brick configuration, there is a list of compatible models and the user selects the ```dog-detector``` model.
380381
In this case, the model will be stored in ```App.yaml```. The brick section will look like:
381382
```
@@ -385,7 +386,7 @@ bricks:
385386
model: dog-detector
386387
- arduino:mood_detector: {}
387388
```
388-
## We need to know if a model build or a model definition can be deleted (link the Figma here):
389+
## Case #7 We need to know if a model build or a model definition can be deleted (link the Figma here):
389390
To delete a build oir a model we need to know the state of the model in the system:
390391
* **Downloaded** (present on the board and available for a brick to be used)
391392
* **Installed** At least one brick is referencing the model.
@@ -399,7 +400,7 @@ The Installed state can be inferred by checking all application ```App.yaml``` f
399400
We can provide a list of models, their states, and all the builds on the File System (FS) (if we implement different versions/builds).
400401
We can allow the deletion of only specific builds or the entire model structure.
401402

402-
## We want to support a different model/executor
403+
## Case #8 We want to support a different model/executor
403404
We aim to support a learning model, where the model is defined by a set of weights and generated by a framework like TensorFlow. We intend to deploy and run this model within a web browser using JavaScript (TensorFlow.js).
404405

405406
In this scenario, we can declare the model using its folder structure (API requests can be developed if required) and the necessary YAML files.
@@ -414,7 +415,7 @@ The question here is how to make the model variables available to the container.
414415

415416
2. Environment Variables: Alternatively, we can make the entire model configuration available to the container as environment variables.
416417

417-
# Questions:
418+
# Questions to be discussed:
418419
1. Should we keep "audio" as a generic category instead of using something more specific, such as ```ei-audio``` or ```qualcomm-audio```?
419420
At this level, this has no relevance.
420421

@@ -428,12 +429,63 @@ If this becomes technically impossible in the future, we can then add a more spe
428429
3. Why there is the
429430
```require model``` in the brick definition?
430431

431-
4. Check the use case of the Figma where we go to EI and come back with a model
432+
4. Variable Configuration: Should variables related to a model be placed in the model or in the brick?
432433

433-
# Proposed Changes to Bricks and Models Configuration
434-
This section details the proposed changes to the current implementation.
434+
Since different bricks may use different variables to enable the container engine to start the AI engine, it is better to do not declare these variables in the model definition.
435435

436+
When the user provides custom values, the variable definitions will be placed in the App.yaml file. See Use Case #3 App.yaml below, where one brick requires:
437+
```EI_OBJ_DETECTION_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim```
438+
and the other one requires:
439+
```QLC_OBJ_DET_ENGINE_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim
440+
SET_CUSTOM_PARAMETER: 5
441+
```
442+
**App.yaml definition**
443+
```
444+
name: Detect objects on images using two webcam, compare slow and fast engine over the same model
445+
ports: []
446+
bricks:
447+
- arduino:object_detection: {}
448+
variables:
449+
CUSTOM_MODEL_PATH: /home/arduino/.arduino-bricks/ei-models
450+
EI_OBJ_DETECTION_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim
451+
- arduino:fast_object_detection: {}
452+
variables:
453+
CUSTOM_MODEL_PATH: /home/arduino/.arduino-bricks/ei-models
454+
QLC_OBJ_DET_ENGINE_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim
455+
SET_CUSTOM_PARAMETER: 5
456+
icon: 🏞️
457+
```
458+
459+
5. Check the use case of the Figma where we go to EI and come back with a model
460+
461+
# Proposed Changes to Bricks and Models Configuration
436462
A Model definition contains a list of categories.
437463

438-
A Brick definition containes a list of supported categories supported and a default_mode_name.
439-
464+
A Brick definition containes a list of supported categories and a default_model_name.
465+
466+
# Variables
467+
This is a cross-cutting issue, not strictly related to models, but it's worth spending a moment here because it affects how models and bricks are managed.
468+
469+
In the current implementation, all variables originating from the following sources are collected into a single map:
470+
- Variables declared in the brick with a default model.
471+
- Variables defined in the model.
472+
- Variables defined or overridden by the user in the App.yaml file.
473+
This map is then passed directly to the ```environment:``` section of the Compose file.
474+
475+
If a variable can be overridden by the user, it must be declared in the brick definition, it is not required or mandatory to declare
476+
the other variables that can not be visible/overrided by the user.
477+
478+
*NOTE:* Due to a recent issue where using an empty string as a default value is possible, the proposal is to add the is_required flag to the brick definition file, as follows:
479+
```
480+
- name: MY_VARIABLE
481+
description: Variable with empty string value as default
482+
default: ""
483+
required: true
484+
```
485+
486+
*NOTE:* Another proposal is to add all required brick variables to the brick definition file. The drawback is that the brick file could potentially become large. The advantage is that all variables are visible, which reduces the risk of conflicts when adding new variables to existing projects.
487+
```
488+
- name: MY_VARIABLE
489+
description: Variable is not user-overridable
490+
private: true
491+
```

0 commit comments

Comments
 (0)