Commit 8aa86ce
committed
Add comprehensive integration tests for tool calling
## Test Coverage
### 1. Basic Tool Calling Tests (test_tool_calling_no_infinite_loop)
Tests 4 models to ensure basic tool calling works without infinite loops:
- meta.llama-4-scout-17b-16e-instruct
- meta.llama-3.3-70b-instruct
- cohere.command-a-03-2025
- cohere.command-r-plus-08-2024
Verifies:
- Tool is called when needed
- Model stops after receiving tool results
- No infinite loops occur
### 2. Model-Specific Tests
- test_meta_llama_tool_calling: Validates Meta Llama models specifically
- test_cohere_tool_calling: Validates Cohere models return expected content
### 3. Multi-Step Tool Orchestration Test (test_multi_step_tool_orchestration)
Simulates realistic diagnostic workflows with 6 tools (2 models tested):
- meta.llama-4-scout-17b-16e-instruct
- cohere.command-a-03-2025
Tools simulate monitoring scenarios:
- check_status: Current resource health
- get_events: Recent failure events
- get_metrics: Historical trends
- check_changes: Recent deployments
- create_alert: Incident creation
- take_action: Remediation actions
Verifies:
- Agent makes multiple tool calls (2-8)
- Respects max_sequential_tool_calls limit
- Eventually stops (no infinite loops)
- Handles OCI limitation (1 tool call at a time)
## Test Results
All 8 tests passing across 4 models:
✅ Basic tool calling (4 models × 1 test = 4 tests)
✅ Model-specific tests (2 tests)
✅ Multi-step orchestration (2 models × 1 test = 2 tests)
## Documentation
Added comprehensive test documentation including:
- Prerequisites (OCI auth, environment setup)
- Running instructions
- What each test verifies
- Model compatibility notes1 parent b7d2ebb commit 8aa86ce
File tree
2 files changed
+577
-2
lines changed- libs/oci/tests
- integration_tests/chat_models
- unit_tests/chat_models
2 files changed
+577
-2
lines changed
0 commit comments