fix(guard): cache custom serve routes

[MasterPtato]

Fixes RVT-5324

[linear]

RVT-5324 Cache customserve route targets

[MasterPtato]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• fix: remove graceful requirement for metrics aggregator #4060
• fix(guard): cache custom serve routes #4059 👈 (View in Graphite)
• fix(gateway): configure cors for error responses #4058
• chore(guard): clean up guard #4057
• fix: fix active req caculation, add hws metrics #4053
• main

---

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

PR Review: fix(guard): cache custom serve routes

Summary

This PR fixes an issue where custom serve routes were not being cached properly. The key change is modifying the route cache to store RoutingOutput instead of just RouteConfig, allowing both regular routes and custom serve handlers to be cached.

Code Quality - Positive Changes

• Better abstraction: Caching RoutingOutput instead of RouteConfig is the right design - it treats both routing types uniformly
• Improved random selection: Using rand::seq::SliceRandom::choose() instead of manual index calculation is cleaner and handles empty slices correctly
• Code consolidation: Moved caching logic before the match statement, eliminating duplication
• Better imports: Using use rand::seq::SliceRandom instead of glob import follows Rust best practices

The refactoring in proxy_service.rs:100-136 improves the flow:

1. Check cache first
2. Call routing function if cache miss
3. Cache the result immediately after receiving it
4. Then process the cached/fresh result

This is much cleaner than the previous approach where caching happened after processing.

Potential Issues

1. TODO Comment Needs Attention (cache/actor.rs:35)

The path() method per the documentation at request_context.rs:18 includes both path AND query. This means different query parameters will produce different cache keys. This might be intentional for actor routing OR it might cause unnecessary cache misses if query params do not affect routing.

Recommendation: Clarify if query params should be in cache key, then either remove the TODO with an explanatory comment or modify the code to exclude query params.

2. Missing Test Coverage

This PR does not include tests for the new caching behavior. Consider adding:

• Unit test to verify custom serve routes are cached
• Integration test to confirm cache hits work for both RoutingOutput::Route and RoutingOutput::CustomServe
• Test to verify empty target lists are handled gracefully

3. Cache Key Documentation (request_context.rs:12-19)

The new comments are helpful but could be clearer with examples like "example.com:8080" for host field.

Performance Considerations

Positive impact:

• Custom serve routes will now benefit from caching (10-minute TTL), reducing routing function overhead
• Moving the cache insert before the match statement ensures consistent caching behavior
• Using SliceRandom::choose is potentially more efficient for large target lists

No concerns:

• Caching RoutingOutput (which includes Arc) is lightweight since it is reference-counted
• The 10-minute TTL and 10,000 entry limit remain appropriate

Security Concerns

No issues identified. The changes do not introduce security vulnerabilities.

Recommendations Summary

Priority	Item	Action
High	Resolve TODO comment	Clarify if query params should be in cache key
Medium	Add test coverage	Test custom serve route caching
Low	Improve documentation	Add examples to RequestContext field comments

Verdict: Approve with Minor Suggestions

The PR successfully fixes the issue and improves code quality. The core implementation is sound. The main action item is resolving the TODO comment about query parameters in the cache key - this could affect correctness depending on your routing requirements.

Before merging:

• Resolve the TODO at cache/actor.rs:35
• Consider adding tests for the new caching behavior

Great work on the cleaner abstraction!

[pkg-pr-new]

More templates

• actor-actions
• actor-actions-vercel
• example-ai-agent
• example-ai-agent-vercel
• example-ai-and-user-generated-actors-freestyle
• chat-room
• chat-room-vercel
• example-cloudflare-workers
• example-cloudflare-workers-hono
• cross-actor-actions
• cross-actor-actions-vercel
• example-cursors
• example-cursors-raw-websocket
• example-cursors-raw-websocket-vercel
• example-cursors-vercel
• custom-serverless
• custom-serverless-vercel
• example-sqlite
• example-elysia
• example-elysia-vercel
• example-experimental-durable-streams-ai-agent
• example-experimental-durable-streams-ai-agent-vercel
• hello-world
• hello-world-vercel
• example-hono
• hono-react
• hono-react-vercel
• example-hono-vercel
• example-kitchen-sink
• example-kitchen-sink-vercel
• multi-region
• multi-region-vercel
• native-websockets
• native-websockets-vercel
• next-js
• example-raw-fetch-handler
• example-raw-fetch-handler-vercel
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-raw-websocket-handler-proxy-vercel
• example-raw-websocket-handler-vercel
• example-react
• example-react-vercel
• scheduling
• scheduling-vercel
• state
• state-vercel
• example-stream
• example-stream-vercel
• example-trpc
• example-trpc-vercel

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@4059

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@4059

@rivetkit/db

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/db@4059

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@4059

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@4059

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@4059

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@4059

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@4059

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@4059

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@4059

commit: ea52472