Refactor to continuous background preimage generation and L1 finality caching

[yoshidan]

Description

This PR significantly refactors the architecture to switch from on-demand preimage generation to a continuous, always-on background generation model. This improves response latency and ensures preimages are readily available for finalized blocks.

Key Changes

• Continuous Preimage Collection: Introduced PreimageCollector, which runs in the background to continuously fetch L1 finality updates and generate preimages for the range between the agreed L2 block and the finalized L2 block.

• Decoupled Serving: The HTTP server now serves already generated preimages from the repository (FilePreimageRepository), decoupling the generation logic from the request path.
  L1 Finality Caching: Implementing FinalizedL1Repository to persistently store the latest finalized L1 headers.

• API Updates:

  • get_preimage: Updated to return pre-generated data from storage.
  • get_finalized_l1: New endpoint to retrieve finalized L1 headers by hash.
  • get_latest_metadata & list_metadata: New endpoints for better visibility into the available preimages.

• Concurrency & Maintenance:

  • Preimage collection supports parallel execution (max_concurrency).
  • Added PreimagePurger to automatically clean up expired preimages and headers based on TTL.

[Copilot]

Pull request overview

This PR refactors the preimage maker from an on-demand request-response model to a continuous background generation architecture. The system now proactively generates preimages for finalized L2 blocks and caches L1 finality headers, improving response latency by decoupling generation from serving.

Key Changes:

• Introduced PreimageCollector for continuous background preimage generation between agreed and finalized L2 blocks
• Implemented file-based repositories (FilePreimageRepository, FileFinalizedL1Repository) for persistent storage with TTL-based expiration
• Added PreimagePurger for automated cleanup of expired data
• Replaced synchronous derivation endpoint with new endpoints: get_preimage, get_finalized_l1, get_latest_metadata, and list_metadata

Reviewed changes

Copilot reviewed 25 out of 30 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
server/src/web.rs	New HTTP API layer with endpoints for retrieving pre-generated preimages and metadata
server/src/collector.rs	Background worker that continuously fetches L1 finality updates and generates preimages with parallel execution support
server/src/purger.rs	Scheduled task for cleaning up expired preimages and L1 headers based on TTL
server/src/data/preimage_repository.rs	Repository trait and metadata model for preimage storage
server/src/data/file_preimage_repository.rs	File-based implementation of preimage repository with in-memory metadata cache
server/src/data/finalized_l1_repository.rs	Repository trait for L1 finality header storage
server/src/data/file_finalized_l1_repository.rs	File-based implementation for storing L1 finality headers
server/src/derivation/host/single/handler.rs	Refactored to separate DerivationConfig from DerivationRequest, supporting the new architecture
server/src/derivation/host/single/config.rs	Added configuration parameters for collector intervals, TTL, concurrency, and storage directories
server/src/main.rs	Updated to spawn collector and purger tasks alongside the HTTP server with graceful shutdown handling
server/tests/e2e.rs	Refactored E2E tests to validate the new API endpoints and continuous collection model
server/tests/inspect.rs	New inspection test for manual derivation verification
Makefile	Updated to create storage directories, use newer Optimism version (v1.16.3), and configure initial parameters
.github/workflows/ci.yaml	Added Foundry version specification
tool/output.sh, tool/derive.sh	Removed obsolete shell scripts for manual derivation

Comments suppressed due to low confidence (1)

server/src/derivation/host/single/config.rs:56

• The initial_claimed_l2 parameter is required (no default value) but lacks documentation explaining what value should be provided or how to determine it. Consider adding a default value or more detailed documentation explaining the expected input.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[siburu]

@yoshidan I still see the warning: This branch has conflicts that must be resolved.
It is necessary to fix some conflicts, right?

[yoshidan]

@siburu I fixed the conflict now.