feat(cli): implements content-addressable store for the dev CLI build outputs, reducing disk usage by ericallam · Pull Request #2725 · triggerdotdev/trigger.dev

ericallam · 2025-12-02T15:30:58Z

Implements a content-addressable store for dev CLI build outputs, significantly reducing disk usage when making iterative code changes during development.

Problem

When running trigger.dev dev, each code change creates a new build directory in .trigger/tmp/build-XXXXX/. Even minor changes resulted in ~22MB of duplicated files per build because esbuild outputs were being fully copied each time, despite most files (chunks, dependencies) being identical between builds.

Solution

Introduced a content-addressable store at .trigger/tmp/store/ that deduplicates build outputs using hardlinks:

Content-addressable storage: Files are stored by their content hash (using esbuild's built-in outputFile.hash), so identical content is only stored once
Hardlinks: Build directories contain hardlinks to store files instead of copies - same inode means zero additional disk space
Hash caching: File hashes are stored in BuildManifest.outputHashes to avoid recomputation when copying to worker directories

Tested with 12 iterative builds in a reference project:

Metric	Without Dedup	With Dedup	Savings
12 builds	~264 MB	68.7 MB	195 MB (74%)

First build: 22MB (populates the store)
Subsequent builds: ~1.7-4.3MB each (only changed files)
Savings increase with more builds as unchanged files are simply hardlinked

Backwards Compatibility

outputHashes is optional in BuildManifest - falls back to computing hashes if not present
Store is only used in dev mode, no impact on deploy builds
Store is cleared when dev session starts fresh (clearTmpDirs)

… outputs, reducing disk usage

changeset-bot · 2025-12-02T15:31:03Z

🦋 Changeset detected

Latest commit: df5ef35

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 26 packages

Name	Type
trigger.dev	Patch
d3-chat	Patch
references-d3-openai-agents	Patch
references-nextjs-realtime	Patch
references-realtime-hooks-test	Patch
references-realtime-streams	Patch
references-telemetry	Patch
@trigger.dev/build	Patch
@trigger.dev/core	Patch
@trigger.dev/python	Patch
@trigger.dev/react-hooks	Patch
@trigger.dev/redis-worker	Patch
@trigger.dev/rsc	Patch
@trigger.dev/schema-to-json	Patch
@trigger.dev/sdk	Patch
@trigger.dev/database	Patch
@trigger.dev/otlp-importer	Patch
@internal/cache	Patch
@internal/clickhouse	Patch
@internal/redis	Patch
@internal/replication	Patch
@internal/run-engine	Patch
@internal/schedule-engine	Patch
@internal/testcontainers	Patch
@internal/tracing	Patch
@internal/zod-worker	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

coderabbitai · 2025-12-02T15:31:12Z

Walkthrough

This pull request adds a content-addressable store for CLI build outputs and threads an optional storeDir through the build and dev pipelines. Build outputs are hashed and can be written via a store-aware writer that hardlinks from a persistent .trigger/tmp/store location. Manifests gain an optional outputHashes map and copyManifestToDir can use the store to deduplicate files when copying manifests and outputs. Utilities for sanitizing hashes, computing file hashes, and resolving the store directory are included.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Review createFileWithStore in packages/cli-v3/src/utilities/fileSystem.ts — hardlink/copy fallbacks, permissions, and atomicity.
Review copyDirWithStore and computeFileHash in packages/cli-v3/src/build/manifests.ts — traversal, symlink handling, hash lookup order, and failure fallbacks.
Verify propagation of storeDir through getBundleResultFromBuild, createBuildManifestFromBundle, bundleWorker, and dev session flows in packages/cli-v3/src/*.
Confirm BuildManifest schema change (outputHashes) in packages/core/src/v3/schemas/build.ts is consistent with manifest construction and consumers.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 41.18% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title accurately summarizes the main feature: implementing a content-addressable store to reduce disk usage for dev CLI build outputs.
Description check	✅ Passed	The PR description is comprehensive and follows the template structure with sections covering Problem, Solution, Test Results, and Backwards Compatibility, though the Checklist items are not explicitly marked as completed.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch ea-branch-105

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (3)

packages/cli-v3/src/utilities/fileSystem.ts (1)
53-62: Consider atomic existence check or accept TOCTOU as acceptable risk.

The synchronous existsSync check followed by async unlink creates a potential time-of-check-to-time-of-use (TOCTOU) race condition. In practice, this is unlikely to cause issues in a single-process dev CLI, but for robustness consider either:

Catching and ignoring ENOENT errors from unlink

Using fs.rm with force: true which ignores missing files

Apply this diff for a more robust approach:
-  // Remove existing file at destination if it exists (hardlinks fail on existing files)
-  if (fsSync.existsSync(filePath)) {
-    await fsModule.unlink(filePath);
-  }
+  // Remove existing file at destination if it exists (hardlinks fail on existing files)
+  try {
+    await fsModule.unlink(filePath);
+  } catch (e: any) {
+    // Ignore ENOENT - file doesn't exist, which is fine
+    if (e.code !== 'ENOENT') throw e;
+  }
packages/cli-v3/src/build/manifests.ts (2)
54-56: Extract sanitizeHashForFilename to avoid duplication.

This function is identical to sanitizeHashForFilename in fileSystem.ts (lines 23-25). Consider extracting it to a shared location to maintain DRY principles.

Example refactor:
+// In fileSystem.ts, export the function:
-function sanitizeHashForFilename(hash: string): string {
+export function sanitizeHashForFilename(hash: string): string {
   return hash.replace(/\//g, "_").replace(/\+/g, "-");
 }

+// In manifests.ts, import and use it:
+import { sanitizeHashForFilename } from "../utilities/fileSystem.js";
+
-function sanitizeHashForFilename(hash: string): string {
-  return hash.replace(/\//g, "_").replace(/\+/g, "-");
-}
62-65: Consider collision risk with 16-character hash.

Using only the first 16 characters of a SHA-256 hash (64 bits) for content addressing may have collision risks in larger codebases. While unlikely in practice, the full hash or at least 32 characters (128 bits) would provide stronger collision resistance.

Consider using more hash characters:
-  return createHash("sha256").update(contents).digest("hex").slice(0, 16);
+  return createHash("sha256").update(contents).digest("hex"); // Full 64 chars, or .slice(0, 32) for 128 bits

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9f27422 and 64cc995.

⛔ Files ignored due to path filters (1)

references/d3-chat/src/trigger/chat.ts is excluded by !references/**

📒 Files selected for processing (7)

.changeset/polite-eels-divide.md (1 hunks)
packages/cli-v3/src/build/bundle.ts (9 hunks)
packages/cli-v3/src/build/manifests.ts (2 hunks)
packages/cli-v3/src/dev/devSession.ts (5 hunks)
packages/cli-v3/src/utilities/fileSystem.ts (1 hunks)
packages/cli-v3/src/utilities/tempDirectories.ts (1 hunks)
packages/core/src/v3/schemas/build.ts (1 hunks)

🧰 Additional context used

📓 Path-based instructions (4)

**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead

Files:

packages/core/src/v3/schemas/build.ts
packages/cli-v3/src/utilities/tempDirectories.ts
packages/cli-v3/src/utilities/fileSystem.ts
packages/cli-v3/src/build/manifests.ts
packages/cli-v3/src/dev/devSession.ts
packages/cli-v3/src/build/bundle.ts

{packages/core,apps/webapp}/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use zod for validation in packages/core and apps/webapp

Files:

packages/core/src/v3/schemas/build.ts

**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use function declarations instead of default exports

Files:

packages/core/src/v3/schemas/build.ts
packages/cli-v3/src/utilities/tempDirectories.ts
packages/cli-v3/src/utilities/fileSystem.ts
packages/cli-v3/src/build/manifests.ts
packages/cli-v3/src/dev/devSession.ts
packages/cli-v3/src/build/bundle.ts

**/*.{js,ts,jsx,tsx,json,md,css,scss}

📄 CodeRabbit inference engine (AGENTS.md)

Format code using Prettier

Files:

packages/core/src/v3/schemas/build.ts
packages/cli-v3/src/utilities/tempDirectories.ts
packages/cli-v3/src/utilities/fileSystem.ts
packages/cli-v3/src/build/manifests.ts
packages/cli-v3/src/dev/devSession.ts
packages/cli-v3/src/build/bundle.ts

🧠 Learnings (11)