[FEATURE] Add cache_strategy="auto" for automatic prompt caching

[kevmyung]

Problem Statement

1. No SDK-level support for conversation caching

   • Prompt caching can significantly reduce costs (up to 90% on cache reads) and latency
   • Users must implement custom hooks to enable caching across tool-use loops

2. Limited scope of existing cache options

   • Current cache_prompt, cache_tools only support system prompt and tool definitions
   • Minimal impact in real-world workflows where conversation history dominates token usage

3. Provider-specific configuration

   • Cache configuration only available in BedrockModel
   • Other providers have cachePoint conversion logic but no way to enable via config
   • Inconsistent developer experience

Proposed Solution

Add cache_strategy parameter to base Model class with hook-based auto-caching.

Key Components

• Model (ABC) : Shared Configuration
  • Add cache_strategy: Optional[str] = None to the base Model class. All provider implementations (BedrockModel, AnthropicModel, LiteLLM, etc.) inherit this configuration automatically.

# Usage (any provider)
agent = Agent(model=BedrockModel(cache_strategy="auto"))
agent = Agent(model=AnthropicModel(cache_strategy="auto"))
agent = Agent(model=LiteLLM(cache_strategy="auto"))

• Agent: Hook Auto-Registration
  • When cache_strategy="auto" is detected, the Agent automatically registers a ConversationCachingHook:

# In Agent.__init__()
if model.get_config().get("cache_strategy") == "auto":
    self.hooks.add_hook(ConversationCachingHook())

• ConversationCachingHook: CachePoint Injection
  • The hook injects a cachePoint block at the last assistant message on each BeforeModelCallEvent

# Before injection
[..., {role: "assistant", content: [...]}, {role: "user", ...}]

# After injection  
[..., {role: "assistant", content: [..., {"cachePoint": {"type": "default"}}]}, {role: "user", ...}]

* This single cache point covers system prompt + tools + conversation history up to that point.

• Provider-Specific Handling (Existing Logic)
  • Each model provider processes the injected cachePoint using existing conversion logic:
    • BedrockModel : Pass-through (native format) | Maintain
    • AnthropicModel : Convert to cache_control | Maintain
    • LiteLLMModel : Convert to cache_control | Fix needed (add message-level handling)

Use Case

Agent workflows with tool usage where multiple model calls occur:

Single-turn scenarios:

• Tool-heavy tasks (search → fetch → analyze → respond)
• Each model call after first assistant message benefits from cache
• Test result: 50-90+% cache hit within single turn (link)

Multi-turn scenarios:

• Conversation history accumulates across turns
• Previous turns fully cached on subsequent turns
• Compounding cost savings over conversation lifetime

Impact:

• 90% cost reduction on cached tokens
• Reduced latency on subsequent model calls

Alternatives Solutions

1. Manual cachePoint injection - Current approach, requires custom hook implementation
2. Agent-level cache_strategy - Rejected because Model owns provider config
3. Automatic system prompt caching only - Insufficient, conversation history dominates

Additional Context

• Backward compatible with existing manual cachePoint insertion
• Extensible for future cache strategies

[Unshure]

Hi @kevmyung, we already have an issue similar to this one here: #756

Do you think this other issue captures this well? We are working on prioritizing that other issue, and want to deduplicate similar issues.

[kevmyung]

Hi @Unshure , thanks for sharing.
While similar to #756, this proposal focuses specifically on the implementation of automated cache point insertion!

[dbschmigelski]

Thank you for the proposal. This is great to see and we definitely want more of it.

There are a couple core questions that we need to answer with strong justifications for each.

1. Does this belong on the Agent or on the Model?
2. Should this be implemented using Hooks? And if so, should the implementor know about that or should it be abstracted as you proposed here?

For 2, we are discussing this currently for a variety of issues and will have strong guidance very soon.

Also,

What we need to design carefully is how we defend against the following #1219 which exposes that there are certain provider+model quarks. I see us having three options

1. Have a single ConversationCachingHook which applies the cachePoint, then apply filtering logic inside the model
2. Some new method on the model interface which only lets us set the cachePoint if the model "approves" of its safety.
3. Have multiple ConversationCachingHook implementations per provider

I am leaning towards 1 which is what you have proposed here as CachePoint is an abstraction that already exists. This also makes us robust in the event we switch mid execution from one model to the next. But, as part of our planning I would not want to release this knowing that we will make issues like #1219 occur frequently. We will need to be fairly robust.

In either approach, and this is lower level than the comments above, I am hesitant to introduce this as a simple string. Strings don't provide us with an extensible path in the future compared to some class or dictionary approach.

Thanks, and I'll continue to think through the design and its implications.

[kevmyung]

Thank you for the feedback!

1. Agent vs Model
I believe Model is the right place. Caching behavior is provider-specific (different APIs, different limitations), and the Model already knows its own capabilities. Agent should remain model-agnostic.

2. Hooks + Abstraction
Agreed. Happy to align with the guidance once available.

3. Defending against #1219
Option 2 seems cleaner. Adding a supports_caching property to the Model interface lets each provider declare its capability explicitly. The hook is only registered when the model supports it.

4. String → Class/Dict
Good point. I'll refactor cache_strategy: str to a CacheConfig dataclass accordingly.

Happy to discuss further!

[Unshure]

Following up on this, to give some more feedback here:

• Lets not implement as a hook. Lets keep the feature scoped to model providers for now
• Lets just implement this as a feature on the Bedrock Model provider class. Not every model provider will have caching, so while we figure out how to include this more broadly, im fine with adding it just to BedrockModel for now

Otherwise, this generally looks good! Happy to review a pr that aligns with this feedback