Skip to content

fix: Additional tests optimization when running on github actions. #293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
avinash2692 merged 24 commits into from
Jan 16, 2026
+248 −152
Merged

fix: Additional tests optimization when running on github actions. #293

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
65fcb5e
Additional tests cleanup and skipping when running on github actions.
avinash2692 Jan 7, 2026
35c4c90
Merge branch 'main' into fix/286-tests-cleanup
avinash2692 Jan 7, 2026
83d20d8
Merge branch 'main' into fix/286-tests-cleanup
avinash2692 Jan 7, 2026
f22afe6
moving all tests to IBM_GRANITE_4_MICRO_3B
avinash2692 Jan 8, 2026
f6e8f97
Merge branch 'fix/286-tests-cleanup' of https://github.com/generative…
avinash2692 Jan 8, 2026
d998dc6
Merge branch 'main' into fix/286-tests-cleanup
avinash2692 Jan 8, 2026
8b328a0
changing granite4 micro ollama name to latest
avinash2692 Jan 9, 2026
8431f28
Adding step to download granite micro
avinash2692 Jan 9, 2026
2f99525
Minor changes to make tests run
avinash2692 Jan 9, 2026
3823a22
Fixing some wayward tests still in llama
avinash2692 Jan 9, 2026
fee236f
DRYing conftest
avinash2692 Jan 12, 2026
c5d73b0
Merge branch 'main' into fix/286-tests-cleanup
avinash2692 Jan 12, 2026
65f176e
Merge branch 'main' into fix/286-tests-cleanup
avinash2692 Jan 12, 2026
1060da4
Merge remote-tracking branch 'origin/main' into fix/286-tests-cleanup
avinash2692 Jan 12, 2026
5d6c535
Merge remote-tracking branch 'origin/main' into fix/286-tests-cleanup
avinash2692 Jan 15, 2026
4d85f17
Adding metadata to notebooks to help with automated testing.
avinash2692 Jan 15, 2026
2a7c7cc
enabling notebooks to run with pytest docs
avinash2692 Jan 15, 2026
4081477
Merge branch 'main' into fix/286-tests-cleanup
avinash2692 Jan 15, 2026
9a73fa3
reverting cause plugins can only be initiated at top level
avinash2692 Jan 15, 2026
9298672
Merge branch 'main' into fix/286-tests-cleanup
nrfulton Jan 16, 2026
419fcd0
updating dev with nbmake
avinash2692 Jan 16, 2026
8836f7a
removing openai name from granite micro
avinash2692 Jan 16, 2026
e7719c6
Merge branch 'main' into fix/286-tests-cleanup
avinash2692 Jan 16, 2026
956ba93
small change
avinash2692 Jan 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 8 additions & 11 deletions .github/workflows/quality.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@ name: Verify Code Quality
on:
workflow_call:


concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.number || github.ref_name }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.number || github.ref_name }}
cancel-in-progress: true

env:
CICD: 1
Expand All @@ -15,10 +14,10 @@ env:
jobs:
quality:
runs-on: ubuntu-latest
timeout-minutes: 90 # TODO: need to reduce this after we figure out our testing strategy.
timeout-minutes: 90 # TODO: need to reduce this after we figure out our testing strategy.
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12'] # Need to add 3.13 once we resolve outlines issues.
python-version: ["3.10", "3.11", "3.12"] # Need to add 3.13 once we resolve outlines issues.
steps:
- uses: actions/checkout@v4
- name: Free disk space
Expand All @@ -39,18 +38,16 @@ jobs:
- name: Check style and run tests
run: pre-commit run --all-files
- name: Send failure message pre-commit
if: failure() # This step will only run if a previous step failed
if: failure() # This step will only run if a previous step failed
run: echo "The quality verification failed. Please run precommit "
- name: Install Ollama
run: curl -fsSL https://ollama.com/install.sh | sh
- name: Start serving ollama
run: nohup ollama serve &
- name: Pull Llama 3.2:1b model
run: ollama pull llama3.2:1b

- name: Pull model granite4:micro
run: ollama pull granite4:micro
- name: Run Tests
run: uv run -m pytest -v test
- name: Send failure message tests
if: failure() # This step will only run if a previous step failed
if: failure() # This step will only run if a previous step failed
run: echo "Tests failed. Please verify that tests are working locally."

13 changes: 4 additions & 9 deletions docs/examples/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
"""Allows you to use `pytest docs` to run the examples."""
"""Allows you to use `pytest docs` to run the examples.

To run notebooks, use: uv run --with 'mcp' pytest --nbmake docs/examples/notebooks/
Copy link
Contributor

@jakelorocco jakelorocco Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add nbmake to the dev section of the pyproject imports.

"""

import pathlib
import subprocess
Expand Down Expand Up @@ -43,14 +46,6 @@ def pytest_collect_file(parent: pytest.Dir, file_path: pathlib.PosixPath):

return ExampleFile.from_parent(parent, path=file_path)

# TODO: Support running jupyter notebooks:
# - use nbmake or directly use nbclient as documented below
# - install the nbclient package
# - run either using python api or jupyter execute
# - must replace background processes
# if file_path.suffix == ".ipynb":
# return ExampleFile.from_parent(parent, path=file_path)


class ExampleFile(pytest.File):
def collect(self):
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/compositionality_with_generative_slots.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/context_example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/document_mobject.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
15 changes: 10 additions & 5 deletions docs/examples/notebooks/georgia_tech.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6fDEbLHL_hkK"
"id": "6fDEbLHL_hkK",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -134,14 +137,14 @@
" strategy=RejectionSamplingStrategy(loop_budget=5),\n",
" user_variables={\"name\": name, \"notes\": notes},\n",
" return_sampling_results=True,\n",
" )\n",
" ) # type: ignore\n",
" if email_candidate.success:\n",
" return str(email_candidate.result)\n",
" else:\n",
" return email_candidate.sample_generations[0].value\n",
"\n",
"\n",
"m = mellea_org.start_session()\n",
"m = mellea.start_session()\n",
"print(\n",
" write_email(\n",
" m,\n",
Expand Down Expand Up @@ -556,11 +559,13 @@
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"display_name": "mellea-public",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python"
"name": "python",
"version": "3.12.10"
}
},
"nbformat": 4,
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/instruct_validate_repair.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/m_serve_example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/mcp_example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -58,7 +61,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/model_options_example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/sentiment_classifier.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/simple_email.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
10 changes: 8 additions & 2 deletions docs/examples/notebooks/table_mobject.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VDaTfltQY3Fl"
"id": "VDaTfltQY3Fl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -56,7 +59,10 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9EurAUSz_1yl"
"id": "9EurAUSz_1yl",
"tags": [
"skip-execution"
]
},
"outputs": [],
"source": [
Expand Down
2 changes: 1 addition & 1 deletion mellea/backends/model_ids.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ class ModelIdentifier:

IBM_GRANITE_4_MICRO_3B = ModelIdentifier(
hf_model_name="ibm-granite/granite-4.0-micro",
ollama_name="ibm/granite4:micro",
ollama_name="granite4:micro",
watsonx_name="ibm/granite-4-h-small",
)
# todo: watsonx model is different from ollama model - should be same.
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ dev = [
"pytest-asyncio",
"mypy>=1.17.0",
"python-semantic-release~=7.32",
"nbmake>=1.5.5",
]

notebook = [
Expand Down
Loading