Skip to content

Commit b9711fd

Browse files
committed
Add gemini example using litellm
1 parent c29afd0 commit b9711fd

File tree

9 files changed

+755
-0
lines changed

9 files changed

+755
-0
lines changed
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
*.so
6+
.Python
7+
build/
8+
develop-eggs/
9+
dist/
10+
downloads/
11+
eggs/
12+
.eggs/
13+
lib/
14+
lib64/
15+
parts/
16+
sdist/
17+
var/
18+
wheels/
19+
*.egg-info/
20+
.installed.cfg
21+
*.egg
22+
23+
# Environments
24+
.env**
25+
.venv
26+
env/
27+
venv/
28+
ENV/
29+
env.bak/
30+
venv.bak/
31+
32+
# IDE
33+
.idea/
34+
.vscode/
35+
*.swp
36+
*.swo
37+
38+
# Git
39+
.git
40+
.gitignore
41+
42+
# Misc
43+
.DS_Store
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# syntax=docker/dockerfile:1.3
2+
FROM python:3.12-slim
3+
COPY --from=ghcr.io/astral-sh/uv:0.6.4 /uv /uvx /bin/
4+
5+
# Install system dependencies
6+
RUN apt-get update && apt-get install -y \
7+
htop \
8+
vim \
9+
curl \
10+
tar \
11+
python3-dev \
12+
postgresql-client \
13+
build-essential \
14+
libpq-dev \
15+
gcc \
16+
cmake \
17+
netcat-openbsd \
18+
nodejs \
19+
npm \
20+
&& apt-get clean \
21+
&& rm -rf /var/lib/apt/lists/**
22+
23+
# Install tctl (Temporal CLI)
24+
RUN curl -L https://github.com/temporalio/tctl/releases/download/v1.18.1/tctl_1.18.1_linux_arm64.tar.gz -o /tmp/tctl.tar.gz && \
25+
tar -xzf /tmp/tctl.tar.gz -C /usr/local/bin && \
26+
chmod +x /usr/local/bin/tctl && \
27+
rm /tmp/tctl.tar.gz
28+
29+
RUN uv pip install --system --upgrade pip setuptools wheel
30+
31+
ENV UV_HTTP_TIMEOUT=1000
32+
33+
# Copy pyproject.toml and README.md to install dependencies
34+
COPY 10_async/10_temporal/100_gemini_litellm/pyproject.toml /app/100_gemini_litellm/pyproject.toml
35+
COPY 10_async/10_temporal/100_gemini_litellm/README.md /app/100_gemini_litellm/README.md
36+
37+
WORKDIR /app/100_gemini_litellm
38+
39+
# Copy the project code
40+
COPY 10_async/10_temporal/100_gemini_litellm/project /app/100_gemini_litellm/project
41+
42+
# Install the required Python packages
43+
RUN uv pip install --system .
44+
45+
WORKDIR /app/100_gemini_litellm
46+
47+
ENV PYTHONPATH=/app
48+
ENV AGENT_NAME=at100-gemini-litellm
49+
50+
# Run the ACP server using uvicorn
51+
CMD ["uvicorn", "project.acp:acp", "--host", "0.0.0.0", "--port", "8000"]
52+
53+
# When we deploy the worker, we will replace the CMD with the following
54+
# CMD ["python", "-m", "run_worker"]
Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# [Temporal] Using Alternative Models with LiteLLM (Gemini)
2+
3+
**Part of the [OpenAI SDK + Temporal integration series](../README.md)**
4+
5+
## What You'll Learn
6+
7+
This tutorial demonstrates how to use Google's Gemini models (or any other LLM provider) with the OpenAI Agents SDK through LiteLLM. The key insight is that LiteLLM provides a unified interface, allowing you to swap models without changing your agent code structure.
8+
9+
**Key insight:** You can use the same OpenAI Agents SDK patterns with any LLM provider supported by LiteLLM - Gemini, Anthropic Claude, Mistral, and many more.
10+
11+
## Prerequisites
12+
- Development environment set up (see [main repo README](https://github.com/scaleapi/scale-agentex))
13+
- Backend services running: `make dev` from repository root (includes Temporal)
14+
- Temporal UI available at http://localhost:8233
15+
- **Google Gemini API key** (see setup below)
16+
- Understanding of OpenAI Agents SDK basics (see [060_open_ai_agents_sdk_hello_world](../060_open_ai_agents_sdk_hello_world/))
17+
18+
## Setup
19+
20+
### 1. Get a Gemini API Key
21+
22+
1. Go to [Google AI Studio](https://aistudio.google.com/apikey)
23+
2. Create a new API key
24+
3. Copy the key for the next step
25+
26+
### 2. Configure the API Key
27+
28+
Add to your environment or `manifest.yaml`:
29+
30+
**Option A: Environment variable**
31+
```bash
32+
export GEMINI_API_KEY="your-gemini-api-key-here"
33+
```
34+
35+
**Option B: In manifest.yaml**
36+
```yaml
37+
agent:
38+
env:
39+
GEMINI_API_KEY: "your-gemini-api-key-here"
40+
```
41+
42+
### 3. Install LiteLLM Dependency
43+
44+
The `pyproject.toml` already includes `litellm>=1.52.0`. When you run the agent, dependencies are installed automatically.
45+
46+
## Quick Start
47+
48+
```bash
49+
cd examples/tutorials/10_async/10_temporal/100_gemini_litellm
50+
uv run agentex agents run --manifest manifest.yaml
51+
```
52+
53+
**Monitor:** Open Temporal UI at http://localhost:8233 to see workflow execution.
54+
55+
## Key Code Changes
56+
57+
The main difference from OpenAI examples is using `LitellmModel`:
58+
59+
```python
60+
from agents.extensions.models.litellm_model import LitellmModel
61+
62+
# Create a LiteLLM model pointing to Gemini
63+
gemini_model = LitellmModel(model="gemini/gemini-2.0-flash")
64+
65+
agent = Agent(
66+
name="Gemini Assistant",
67+
instructions="You are a helpful assistant powered by Gemini.",
68+
model=gemini_model, # Use the LiteLLM model instead of default
69+
)
70+
71+
# Run works exactly the same way
72+
result = await Runner.run(agent, user_messages)
73+
```
74+
75+
## Supported Models
76+
77+
LiteLLM supports many providers. Just change the model string:
78+
79+
| Provider | Model String Example |
80+
|----------|---------------------|
81+
| Google Gemini | `gemini/gemini-2.0-flash`, `gemini/gemini-1.5-pro` |
82+
| Anthropic | `anthropic/claude-3-sonnet-20240229` |
83+
| Mistral | `mistral/mistral-large-latest` |
84+
| Cohere | `cohere/command-r-plus` |
85+
| AWS Bedrock | `bedrock/anthropic.claude-3-sonnet` |
86+
87+
See [LiteLLM Providers](https://docs.litellm.ai/docs/providers) for the full list.
88+
89+
## Why LiteLLM?
90+
91+
**Model Flexibility:** Switch between providers without code changes - just update the model string.
92+
93+
**Unified Interface:** Same OpenAI Agents SDK patterns work with any provider.
94+
95+
**Cost Optimization:** Easily compare costs across providers by switching models.
96+
97+
**Fallback Support:** LiteLLM supports automatic fallbacks if a provider is unavailable.
98+
99+
## Architecture Notes
100+
101+
The Temporal integration remains identical:
102+
- Workflows are durable and survive restarts
103+
- LLM calls are wrapped as activities automatically
104+
- Full observability in Temporal UI
105+
- Automatic retries on failures
106+
107+
The only change is the model provider - everything else works the same.
108+
109+
## When to Use
110+
111+
- Want to use non-OpenAI models with OpenAI Agents SDK
112+
- Need to compare model performance across providers
113+
- Building multi-model systems with fallbacks
114+
- Cost optimization across different providers
115+
- Regulatory requirements for specific model providers
116+
117+
## Troubleshooting
118+
119+
**"GEMINI_API_KEY environment variable is not set"**
120+
- Ensure you've exported the API key or added it to manifest.yaml
121+
122+
**"Model not found" errors**
123+
- Check the model string format matches LiteLLM's expected format
124+
- See [LiteLLM Providers](https://docs.litellm.ai/docs/providers) for correct model names
125+
126+
**Rate limiting errors**
127+
- Gemini has different rate limits than OpenAI
128+
- Consider adding retry logic or using LiteLLM's built-in retry support
129+
130+
**Previous:** [090_claude_agents_sdk_mvp](../090_claude_agents_sdk_mvp/) - Claude SDK integration
Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# Agent Manifest Configuration
2+
# ---------------------------
3+
# This file defines how your agent should be built and deployed.
4+
5+
# Build Configuration
6+
# ------------------
7+
# The build config defines what gets packaged into your agent's Docker image.
8+
# This same configuration is used whether building locally or remotely.
9+
#
10+
# When building:
11+
# 1. All files from include_paths are collected into a build context
12+
# 2. The context is filtered by dockerignore rules
13+
# 3. The Dockerfile uses this context to build your agent's image
14+
# 4. The image is pushed to a registry and used to run your agent
15+
build:
16+
context:
17+
# Root directory for the build context
18+
root: ../../../ # Up to tutorials level to include test_utils
19+
20+
# Paths to include in the Docker build context
21+
# Must include:
22+
# - Your agent's directory (your custom agent code)
23+
# These paths are collected and sent to the Docker daemon for building
24+
include_paths:
25+
- 10_async/10_temporal/100_gemini_litellm
26+
27+
# Path to your agent's Dockerfile
28+
# This defines how your agent's image is built from the context
29+
# Relative to the root directory
30+
dockerfile: 10_async/10_temporal/100_gemini_litellm/Dockerfile
31+
32+
# Path to your agent's .dockerignore
33+
# Filters unnecessary files from the build context
34+
# Helps keep build context small and builds fast
35+
dockerignore: 10_async/10_temporal/100_gemini_litellm/.dockerignore
36+
37+
38+
# Local Development Configuration
39+
# -----------------------------
40+
# Only used when running the agent locally
41+
local_development:
42+
agent:
43+
port: 8000 # Port where your local ACP server is running
44+
host_address: host.docker.internal # Host address for Docker networking (host.docker.internal for Docker, localhost for direct)
45+
46+
# File paths for local development (relative to this manifest.yaml)
47+
paths:
48+
# Path to ACP server file
49+
# Examples:
50+
# project/acp.py (standard)
51+
# src/server.py (custom structure)
52+
# ../shared/acp.py (shared across projects)
53+
# /absolute/path/acp.py (absolute path)
54+
acp: project/acp.py
55+
56+
# Path to temporal worker file
57+
# Examples:
58+
# project/run_worker.py (standard)
59+
# workers/temporal.py (custom structure)
60+
# ../shared/worker.py (shared across projects)
61+
worker: project/run_worker.py
62+
63+
64+
# Agent Configuration
65+
# -----------------
66+
agent:
67+
# Type of agent - either sync or async
68+
acp_type: async
69+
70+
# Unique name for your agent
71+
# Used for task routing and monitoring
72+
name: at100-gemini-litellm
73+
74+
# Description of what your agent does
75+
# Helps with documentation and discovery
76+
description: An AgentEx agent using Gemini via LiteLLM
77+
78+
# Temporal workflow configuration
79+
# This enables your agent to run as a Temporal workflow for long-running tasks
80+
temporal:
81+
enabled: true
82+
workflows:
83+
# Name of the workflow class
84+
# Must match the @workflow.defn name in your workflow.py
85+
- name: at100-gemini-litellm
86+
87+
# Queue name for task distribution
88+
# Used by Temporal to route tasks to your agent
89+
# Convention: <agent_name>_task_queue
90+
queue_name: at100_gemini_litellm_queue
91+
92+
# Optional: Credentials mapping
93+
# Maps Kubernetes secrets to environment variables
94+
# Common credentials include:
95+
credentials:
96+
- env_var_name: REDIS_URL
97+
secret_name: redis-url-secret
98+
secret_key: url
99+
# - env_var_name: GEMINI_API_KEY
100+
# secret_name: gemini-api-key
101+
# secret_key: api-key
102+
103+
# Optional: Set Environment variables for running your agent locally as well
104+
# as for deployment later on
105+
env:
106+
# Set your Gemini API key here or in your environment
107+
# GEMINI_API_KEY: "<YOUR_GEMINI_API_KEY_HERE>"
108+
109+
110+
# Deployment Configuration
111+
# -----------------------
112+
# Configuration for deploying your agent to Kubernetes clusters
113+
deployment:
114+
# Container image configuration
115+
image:
116+
repository: "" # Update with your container registry
117+
tag: "latest" # Default tag, should be versioned in production
118+
119+
imagePullSecrets:
120+
- name: my-registry-secret # Update with your image pull secret name
121+
122+
# Global deployment settings that apply to all clusters
123+
# These can be overridden using --override-file with custom configuration files
124+
global:
125+
agent:
126+
name: "at100-gemini-litellm"
127+
description: "An AgentEx agent using Gemini via LiteLLM"
128+
129+
# Default replica count
130+
replicaCount: 1
131+
132+
# Default resource requirements
133+
resources:
134+
requests:
135+
cpu: "500m"
136+
memory: "1Gi"
137+
limits:
138+
cpu: "1000m"
139+
memory: "2Gi"
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Gemini LiteLLM Tutorial

0 commit comments

Comments
 (0)