test: add native integration tests setup #754

chmjkb · 2026-01-24T11:33:29Z

Description

Introduces a breaking change?

Yes
No

Type of change

Bug fix (change which fixes an issue)
New feature (change which adds functionality)
Documentation update (improves or adds clarity to existing documentation)
Other (chores, tests, code style improvements etc.)

Tested on

iOS
Android

Testing instructions

Screenshots

Related issues

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly
My changes generate no new warnings

Additional notes

chmjkb · 2026-01-26T13:47:37Z

#758 - This is to be done as a separate PR.

msluszniak · 2026-01-26T13:52:56Z

This PR closes issue called: "Add CI tests", but there is no CI that runs tests. I guess such CI should be added in this PR.

packages/react-native-executorch/common/rnexecutorch/models/llm/LLM.cpp

packages/react-native-executorch/common/rnexecutorch/models/ocr/utils/RecognizerUtils.cpp

packages/react-native-executorch/common/rnexecutorch/tests/integration/BaseModelTest.cpp

packages/react-native-executorch/common/rnexecutorch/tests/integration/BaseModelTests.h

packages/react-native-executorch/common/rnexecutorch/tests/integration/StyleTransferTest.cpp

packages/react-native-executorch/common/rnexecutorch/tests/integration/utils/TestUtils.h

packages/react-native-executorch/common/rnexecutorch/tests/integration/TextEmbeddingsTest.cpp

packages/react-native-executorch/common/rnexecutorch/tests/integration/TextToImageTest.cpp

packages/react-native-executorch/common/rnexecutorch/tests/integration/VerticalOCRTest.cpp

msluszniak · 2026-01-27T11:12:41Z

packages/react-native-executorch/common/rnexecutorch/tests/README.md

+### Running tests
+To run tests navigate to tests directory, namely:
+* `cd packages/react-native-executorch/common/rnexecutorch/tests`
+To run tests, you need to make sure your Android emulator is running. This is because we're cross-compiling the test executables for Android, so we can easily run the tests using prebuilt Android third-party libs. To run the tests, you need to run the following command:


I would also add information about setting NDK. I have simulator running, but get error:

[ERROR] ANDROID_NDK is not set

Another thing, installing wget:

./run_tests.sh: line 123: wget: command not found

msluszniak · 2026-01-27T11:32:58Z

Also, when I run tests, in logs I got the following msg:
[ERROR] File not found: ../integration/assets/test_ocr_image.png

Additional, what are the simulator and its parameters, because when I run the test suite I run out of memory on device 😅 :

[12:26:03] Creating device test directory...
[12:26:03] Pushing test executables to device...
NumericalTests: 1 file pushed, 0 skipped. 342.7 MB/s (4438584 bytes in 0.012s)
LogTests: 1 file pushed, 0 skipped. 464.6 MB/s (8387640 bytes in 0.017s)
BaseModelTests: 1 file pushed, 0 skipped. 409.8 MB/s (6310296 bytes in 0.015s)
ClassificationTests: 1 file pushed, 0 skipped. 278.8 MB/s (14794728 bytes in 0.051s)
ObjectDetectionTests: 1 file pushed, 0 skipped. 303.5 MB/s (14748152 bytes in 0.046s)
ImageEmbeddingsTests: 1 file pushed, 0 skipped. 255.5 MB/s (14471032 bytes in 0.054s)
TextEmbeddingsTests: 1 file pushed, 0 skipped. 229.8 MB/s (19341392 bytes in 0.080s)
StyleTransferTests: 1 file pushed, 0 skipped. 252.8 MB/s (14395248 bytes in 0.054s)
VADTests: 1 file pushed, 0 skipped. 364.0 MB/s (6627248 bytes in 0.017s)
TokenizerModuleTests: 1 file pushed, 0 skipped. 155.5 MB/s (17474344 bytes in 0.107s)
SpeechToTextTests: 1 file pushed, 0 skipped. 113.4 MB/s (21240696 bytes in 0.179s)
LLMTests: 1 file pushed, 0 skipped. 156.5 MB/s (21151344 bytes in 0.129s)
TextToImageTests: 1 file pushed, 0 skipped. 285.6 MB/s (21191208 bytes in 0.071s)
OCRTests: 1 file pushed, 0 skipped. 266.1 MB/s (19066896 bytes in 0.068s)
VerticalOCRTests: 1 file pushed, 0 skipped. 217.6 MB/s (19273144 bytes in 0.084s)
[12:26:07] Pushing test assets to device...
../integration/assets/test_audio_float.raw: 1 file pushed, 0 skipped. 162.0 MB/s (1568184 bytes in 0.009s)
../integration/assets/we_are_software_mansion.jpg: 1 file pushed, 0 skipped. 66.1 MB/s (52710 bytes in 0.001s)
[ERROR] File not found: ../integration/assets/test_ocr_image.png
[12:26:07] Downloading models (use --refresh-models to force re-download)...
[12:26:07] Downloading style_transfer_candy_xnnpack.pte...
[12:26:08] Downloading efficientnet_v2_s_xnnpack.pte...
[12:26:11] Downloading ssdlite320-mobilenetv3-large.pte...
[12:26:12] Downloading test_image.jpg...
[12:26:12] Downloading clip-vit-base-patch32-vision_xnnpack.pte...
[12:26:23] Downloading all-MiniLM-L6-v2_xnnpack.pte...
[12:26:26] Downloading tokenizer.json...
[12:26:26] Downloading fsmn-vad_xnnpack.pte...
[12:26:27] Downloading whisper_tiny_en_encoder_xnnpack.pte...
[12:26:30] Downloading whisper_tiny_en_decoder_xnnpack.pte...
[12:26:35] Downloading whisper_tokenizer.json...
[12:26:36] Downloading smolLm2_135M_8da4w.pte...
[12:26:52] Downloading smollm_tokenizer.json...
[12:26:53] Downloading deeplabV3_xnnpack_fp32.pte...
[12:26:59] Downloading xnnpack_crnn_english.pte...
[12:27:00] Downloading xnnpack_craft_quantized.pte...
[12:27:07] Downloading t2i_tokenizer.json...
[12:27:07] Downloading t2i_encoder.pte...
[12:27:41] Downloading t2i_unet.pte...
[12:29:22] Downloading t2i_decoder.pte...
[12:29:38] Pushing models to device...
/Users/msluszniak/react-native-executorch/packages/react-native-e...: 1 file pushed, 0 skipped. 186.4 MB/s (90992000 bytes in 0.465s
/Users/msluszniak/react-native-executorch/packages/react-native-e... 1 file pushed, 0 skipped. 338.9 MB/s (351613824 bytes in 0.989s
adb: error: failed to copy '/Users/msluszniak/react-native-executorch/packages/react-native-executorch/common/rnexecutorch/tests/integration/assets/models/deeplabV3_xnnpack_fp32.pte' to '/data/local/tmp/rnexecutorch_tests/deeplabV3_xnnpack_fp32.pte': remote write failed: No space left on device
/Users/msluszniak/react-native-executorch/packages/react-native-e... 1 file pushed, 0 skipped. 485.2 MB/s (167933320 bytes in 0.330s

packages/react-native-executorch/common/rnexecutorch/tests/run_tests.sh

chmjkb added 20 commits January 22, 2026 08:43

wip

252748f

feat: basic on-device gtest run

3bc9aff

feat: extend cmake to use 3rd party android libs

f05bcac

feat(wip): add a run_tests script to run tests on emulator

0a6a8ae

feat: add some tests along with 3p stubs

147894b

feat: add more tests

a596818

feat: add llm tests

bbe3291

chore: remove logs, add jsi stubs

c885a99

chore: void cast to avoid [[nodiscard]] warnings

34cdb66

feat: add tti tests

64e2d17

chore: avoid redownloading when model file exists

43513ac

chore: add .pte files to gitignore

bee34e1

chore: separate integration and unit tests, add some more tests

3ae1a3d

chore: use typed tests for basemodel tests

88059a7

feat: improve tests

1d416c3

feat: add some more tests

e397e23

chore: remove pointless tests

c565649

feat: add vertical ocr tests

da04969

chore: rename variable

6ea237d

fix: expect throw rnexecutorch error instead of std

53c7106

chmjkb self-assigned this Jan 26, 2026

chmjkb added the test Issue and PR related to tests or testing infrastructure label Jan 26, 2026

chore: update readme and the run tests script

4540c13

chmjkb marked this pull request as ready for review January 26, 2026 12:43

chmjkb added 2 commits January 26, 2026 14:33

chore: add missing assets

916b3b3

fix: use proper image name in vertical ocr test

8863fa2

chmjkb linked an issue Jan 26, 2026 that may be closed by this pull request

Add CI tests #541

Open

msluszniak reviewed Jan 26, 2026

View reviewed changes

msluszniak mentioned this pull request Jan 27, 2026

Refactor code of OCR and VerticalOCR to utilise only two points in bounding boxes #760

Open

chore: review suggestions

17189b6

chmjkb requested a review from msluszniak January 27, 2026 10:50

msluszniak reviewed Jan 27, 2026

View reviewed changes

chmjkb commented Jan 27, 2026

View reviewed changes

packages/react-native-executorch/common/rnexecutorch/tests/run_tests.sh Outdated Show resolved Hide resolved

chmjkb and others added 2 commits January 27, 2026 12:40

Apply suggestion from @chmjkb

8fc251d

chore: review changes

22baead

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add native integration tests setup #754

test: add native integration tests setup #754

chmjkb commented Jan 24, 2026

Uh oh!

chmjkb commented Jan 26, 2026

Uh oh!

msluszniak commented Jan 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msluszniak Jan 27, 2026

Uh oh!

msluszniak Jan 27, 2026

Uh oh!

msluszniak commented Jan 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

test: add native integration tests setup #754

Are you sure you want to change the base?

test: add native integration tests setup #754

Conversation

chmjkb commented Jan 24, 2026

Description

Introduces a breaking change?

Type of change

Tested on

Testing instructions

Screenshots

Related issues

Checklist

Additional notes

Uh oh!

chmjkb commented Jan 26, 2026

Uh oh!

msluszniak commented Jan 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msluszniak Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

msluszniak Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

msluszniak commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

msluszniak commented Jan 27, 2026 •

edited

Loading