Skip to content

Commit 4cebd24

Browse files
committed
Add gemini example using litellm
1 parent c29afd0 commit 4cebd24

File tree

10 files changed

+786
-0
lines changed

10 files changed

+786
-0
lines changed
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
*.so
6+
.Python
7+
build/
8+
develop-eggs/
9+
dist/
10+
downloads/
11+
eggs/
12+
.eggs/
13+
lib/
14+
lib64/
15+
parts/
16+
sdist/
17+
var/
18+
wheels/
19+
*.egg-info/
20+
.installed.cfg
21+
*.egg
22+
23+
# Environments
24+
.env**
25+
.venv
26+
env/
27+
venv/
28+
ENV/
29+
env.bak/
30+
venv.bak/
31+
32+
# IDE
33+
.idea/
34+
.vscode/
35+
*.swp
36+
*.swo
37+
38+
# Git
39+
.git
40+
.gitignore
41+
42+
# Misc
43+
.DS_Store
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# syntax=docker/dockerfile:1.3
2+
FROM python:3.12-slim
3+
COPY --from=ghcr.io/astral-sh/uv:0.6.4 /uv /uvx /bin/
4+
5+
# Install system dependencies
6+
RUN apt-get update && apt-get install -y \
7+
htop \
8+
vim \
9+
curl \
10+
tar \
11+
python3-dev \
12+
postgresql-client \
13+
build-essential \
14+
libpq-dev \
15+
gcc \
16+
cmake \
17+
netcat-openbsd \
18+
nodejs \
19+
npm \
20+
&& apt-get clean \
21+
&& rm -rf /var/lib/apt/lists/**
22+
23+
# Install tctl (Temporal CLI)
24+
RUN curl -L https://github.com/temporalio/tctl/releases/download/v1.18.1/tctl_1.18.1_linux_arm64.tar.gz -o /tmp/tctl.tar.gz && \
25+
tar -xzf /tmp/tctl.tar.gz -C /usr/local/bin && \
26+
chmod +x /usr/local/bin/tctl && \
27+
rm /tmp/tctl.tar.gz
28+
29+
RUN uv pip install --system --upgrade pip setuptools wheel
30+
31+
ENV UV_HTTP_TIMEOUT=1000
32+
33+
# Copy pyproject.toml and README.md to install dependencies
34+
COPY 10_async/10_temporal/100_gemini_litellm/pyproject.toml /app/100_gemini_litellm/pyproject.toml
35+
COPY 10_async/10_temporal/100_gemini_litellm/README.md /app/100_gemini_litellm/README.md
36+
37+
WORKDIR /app/100_gemini_litellm
38+
39+
# Copy the project code
40+
COPY 10_async/10_temporal/100_gemini_litellm/project /app/100_gemini_litellm/project
41+
42+
# Copy the test files
43+
COPY 10_async/10_temporal/100_gemini_litellm/tests /app/100_gemini_litellm/tests
44+
45+
# Copy shared test utilities
46+
COPY test_utils /app/test_utils
47+
48+
# Install the required Python packages with dev dependencies
49+
RUN uv pip install --system .[dev]
50+
51+
WORKDIR /app/100_gemini_litellm
52+
53+
ENV PYTHONPATH=/app
54+
55+
# Set test environment variables
56+
ENV AGENT_NAME=at100-gemini-litellm
57+
58+
# Run the ACP server using uvicorn
59+
CMD ["uvicorn", "project.acp:acp", "--host", "0.0.0.0", "--port", "8000"]
60+
61+
# When we deploy the worker, we will replace the CMD with the following
62+
# CMD ["python", "-m", "run_worker"]
Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# [Temporal] Using Alternative Models with LiteLLM (Gemini)
2+
3+
**Part of the [OpenAI SDK + Temporal integration series](../README.md)**
4+
5+
## What You'll Learn
6+
7+
This tutorial demonstrates how to use Google's Gemini models (or any other LLM provider) with the OpenAI Agents SDK through LiteLLM. The key insight is that LiteLLM provides a unified interface, allowing you to swap models without changing your agent code structure.
8+
9+
**Key insight:** You can use the same OpenAI Agents SDK patterns with any LLM provider supported by LiteLLM - Gemini, Anthropic Claude, Mistral, and many more.
10+
11+
## Prerequisites
12+
- Development environment set up (see [main repo README](https://github.com/scaleapi/scale-agentex))
13+
- Backend services running: `make dev` from repository root (includes Temporal)
14+
- Temporal UI available at http://localhost:8233
15+
- **Google Gemini API key** (see setup below)
16+
- Understanding of OpenAI Agents SDK basics (see [060_open_ai_agents_sdk_hello_world](../060_open_ai_agents_sdk_hello_world/))
17+
18+
## Setup
19+
20+
### 1. Get a Gemini API Key
21+
22+
1. Go to [Google AI Studio](https://aistudio.google.com/apikey)
23+
2. Create a new API key
24+
3. Copy the key for the next step
25+
26+
### 2. Configure the API Key
27+
28+
Add to your environment or `manifest.yaml`:
29+
30+
**Option A: Environment variable**
31+
```bash
32+
export GEMINI_API_KEY="your-gemini-api-key-here"
33+
```
34+
35+
**Option B: In manifest.yaml**
36+
```yaml
37+
agent:
38+
env:
39+
GEMINI_API_KEY: "your-gemini-api-key-here"
40+
```
41+
42+
### 3. Install LiteLLM Dependency
43+
44+
The `pyproject.toml` already includes `litellm>=1.52.0`. When you run the agent, dependencies are installed automatically.
45+
46+
## Quick Start
47+
48+
```bash
49+
cd examples/tutorials/10_async/10_temporal/100_gemini_litellm
50+
uv run agentex agents run --manifest manifest.yaml
51+
```
52+
53+
**Monitor:** Open Temporal UI at http://localhost:8233 to see workflow execution.
54+
55+
## Key Code Changes
56+
57+
The main difference from OpenAI examples is using `LitellmModel`:
58+
59+
```python
60+
from agents.extensions.models.litellm_model import LitellmModel
61+
62+
# Create a LiteLLM model pointing to Gemini
63+
gemini_model = LitellmModel(model="gemini/gemini-2.0-flash")
64+
65+
agent = Agent(
66+
name="Gemini Assistant",
67+
instructions="You are a helpful assistant powered by Gemini.",
68+
model=gemini_model, # Use the LiteLLM model instead of default
69+
)
70+
71+
# Run works exactly the same way
72+
result = await Runner.run(agent, user_messages)
73+
```
74+
75+
## Supported Models
76+
77+
LiteLLM supports many providers. Just change the model string:
78+
79+
| Provider | Model String Example |
80+
|----------|---------------------|
81+
| Google Gemini | `gemini/gemini-2.0-flash`, `gemini/gemini-1.5-pro` |
82+
| Anthropic | `anthropic/claude-3-sonnet-20240229` |
83+
| Mistral | `mistral/mistral-large-latest` |
84+
| Cohere | `cohere/command-r-plus` |
85+
| AWS Bedrock | `bedrock/anthropic.claude-3-sonnet` |
86+
87+
See [LiteLLM Providers](https://docs.litellm.ai/docs/providers) for the full list.
88+
89+
## Why LiteLLM?
90+
91+
**Model Flexibility:** Switch between providers without code changes - just update the model string.
92+
93+
**Unified Interface:** Same OpenAI Agents SDK patterns work with any provider.
94+
95+
**Cost Optimization:** Easily compare costs across providers by switching models.
96+
97+
**Fallback Support:** LiteLLM supports automatic fallbacks if a provider is unavailable.
98+
99+
## Architecture Notes
100+
101+
The Temporal integration remains identical:
102+
- Workflows are durable and survive restarts
103+
- LLM calls are wrapped as activities automatically
104+
- Full observability in Temporal UI
105+
- Automatic retries on failures
106+
107+
The only change is the model provider - everything else works the same.
108+
109+
## When to Use
110+
111+
- Want to use non-OpenAI models with OpenAI Agents SDK
112+
- Need to compare model performance across providers
113+
- Building multi-model systems with fallbacks
114+
- Cost optimization across different providers
115+
- Regulatory requirements for specific model providers
116+
117+
## Troubleshooting
118+
119+
**"GEMINI_API_KEY environment variable is not set"**
120+
- Ensure you've exported the API key or added it to manifest.yaml
121+
122+
**"Model not found" errors**
123+
- Check the model string format matches LiteLLM's expected format
124+
- See [LiteLLM Providers](https://docs.litellm.ai/docs/providers) for correct model names
125+
126+
**Rate limiting errors**
127+
- Gemini has different rate limits than OpenAI
128+
- Consider adding retry logic or using LiteLLM's built-in retry support
129+
130+
**Previous:** [090_claude_agents_sdk_mvp](../090_claude_agents_sdk_mvp/) - Claude SDK integration
Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Agent Manifest Configuration
2+
# ---------------------------
3+
# This file defines how your agent should be built and deployed.
4+
5+
# Build Configuration
6+
# ------------------
7+
# The build config defines what gets packaged into your agent's Docker image.
8+
# This same configuration is used whether building locally or remotely.
9+
#
10+
# When building:
11+
# 1. All files from include_paths are collected into a build context
12+
# 2. The context is filtered by dockerignore rules
13+
# 3. The Dockerfile uses this context to build your agent's image
14+
# 4. The image is pushed to a registry and used to run your agent
15+
build:
16+
context:
17+
# Root directory for the build context
18+
root: ../../../ # Up to tutorials level to include test_utils
19+
20+
# Paths to include in the Docker build context
21+
# Must include:
22+
# - Your agent's directory (your custom agent code)
23+
# These paths are collected and sent to the Docker daemon for building
24+
include_paths:
25+
- 10_async/10_temporal/100_gemini_litellm
26+
- test_utils
27+
28+
# Path to your agent's Dockerfile
29+
# This defines how your agent's image is built from the context
30+
# Relative to the root directory
31+
dockerfile: 10_async/10_temporal/100_gemini_litellm/Dockerfile
32+
33+
# Path to your agent's .dockerignore
34+
# Filters unnecessary files from the build context
35+
# Helps keep build context small and builds fast
36+
dockerignore: 10_async/10_temporal/100_gemini_litellm/.dockerignore
37+
38+
39+
# Local Development Configuration
40+
# -----------------------------
41+
# Only used when running the agent locally
42+
local_development:
43+
agent:
44+
port: 8000 # Port where your local ACP server is running
45+
host_address: host.docker.internal # Host address for Docker networking (host.docker.internal for Docker, localhost for direct)
46+
47+
# File paths for local development (relative to this manifest.yaml)
48+
paths:
49+
# Path to ACP server file
50+
# Examples:
51+
# project/acp.py (standard)
52+
# src/server.py (custom structure)
53+
# ../shared/acp.py (shared across projects)
54+
# /absolute/path/acp.py (absolute path)
55+
acp: project/acp.py
56+
57+
# Path to temporal worker file
58+
# Examples:
59+
# project/run_worker.py (standard)
60+
# workers/temporal.py (custom structure)
61+
# ../shared/worker.py (shared across projects)
62+
worker: project/run_worker.py
63+
64+
65+
# Agent Configuration
66+
# -----------------
67+
agent:
68+
# Type of agent - either sync or async
69+
acp_type: async
70+
71+
# Unique name for your agent
72+
# Used for task routing and monitoring
73+
name: at100-gemini-litellm
74+
75+
# Description of what your agent does
76+
# Helps with documentation and discovery
77+
description: An AgentEx agent using Gemini via LiteLLM
78+
79+
# Temporal workflow configuration
80+
# This enables your agent to run as a Temporal workflow for long-running tasks
81+
temporal:
82+
enabled: true
83+
workflows:
84+
# Name of the workflow class
85+
# Must match the @workflow.defn name in your workflow.py
86+
- name: at100-gemini-litellm
87+
88+
# Queue name for task distribution
89+
# Used by Temporal to route tasks to your agent
90+
# Convention: <agent_name>_task_queue
91+
queue_name: at100_gemini_litellm_queue
92+
93+
# Optional: Credentials mapping
94+
# Maps Kubernetes secrets to environment variables
95+
# Common credentials include:
96+
credentials:
97+
- env_var_name: REDIS_URL
98+
secret_name: redis-url-secret
99+
secret_key: url
100+
# - env_var_name: GEMINI_API_KEY
101+
# secret_name: gemini-api-key
102+
# secret_key: api-key
103+
104+
# Optional: Set Environment variables for running your agent locally as well
105+
# as for deployment later on
106+
env:
107+
# Set your Gemini API key here or in your environment
108+
# GEMINI_API_KEY: "<YOUR_GEMINI_API_KEY_HERE>"
109+
110+
111+
# Deployment Configuration
112+
# -----------------------
113+
# Configuration for deploying your agent to Kubernetes clusters
114+
deployment:
115+
# Container image configuration
116+
image:
117+
repository: "" # Update with your container registry
118+
tag: "latest" # Default tag, should be versioned in production
119+
120+
imagePullSecrets:
121+
- name: my-registry-secret # Update with your image pull secret name
122+
123+
# Global deployment settings that apply to all clusters
124+
# These can be overridden using --override-file with custom configuration files
125+
global:
126+
agent:
127+
name: "at100-gemini-litellm"
128+
description: "An AgentEx agent using Gemini via LiteLLM"
129+
130+
# Default replica count
131+
replicaCount: 1
132+
133+
# Default resource requirements
134+
resources:
135+
requests:
136+
cpu: "500m"
137+
memory: "1Gi"
138+
limits:
139+
cpu: "1000m"
140+
memory: "2Gi"
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Gemini LiteLLM Tutorial

0 commit comments

Comments
 (0)