feat (chart): add Langfuse retention management with ClickHouse TTL and hard delete

[a-klos]

This pull request introduces support for automated data retention for Langfuse traces in ClickHouse, targeting setups that do not use Langfuse Enterprise's built-in data retention. It adds Helm chart options, Kubernetes CronJobs for both TTL enforcement and hard deletes, as well as shared environment configuration for ClickHouse access. The documentation is also updated to explain these new features.

Langfuse Trace Retention Automation

• Added a new langfuseRetention configuration section in values.yaml to enable and customize automatic trace retention, including TTL and optional hard deletion, with customizable schedules and ClickHouse table/column mapping.
• Introduced two new Kubernetes CronJobs:
  • One to apply idempotent ALTER TABLE ... MODIFY TTL statements for automatic data expiration.
  • Another (optional) CronJob for deterministic hard deletes using ALTER TABLE ... DELETE WHERE ..., with mutation sync settings.
• Created a shared Helm template (rag.langfuseRetentionClickhouseEnv) to inject consistent ClickHouse connection and retention settings into the CronJobs' environment.

Documentation

• Updated the README.md to document the new trace retention options, configuration examples, and operational notes for non-Enterprise users.

[Copilot]

Pull request overview

This pull request adds automated data retention management for Langfuse traces stored in ClickHouse, targeting deployments without Langfuse Enterprise's built-in retention features. The implementation uses Kubernetes CronJobs to apply TTL policies and optionally perform hard deletes on Langfuse trace tables.

Changes:

• Added langfuseRetention configuration section in values.yaml with configurable retention period, schedules, and ClickHouse table mappings
• Created two CronJob templates: one for applying TTL policies and another for deterministic hard deletes
• Implemented a shared Helm helper template for consistent ClickHouse connection configuration across retention jobs

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
infrastructure/rag/values.yaml	Adds langfuseRetention configuration section with retention settings, CronJob schedules, image configuration, and ClickHouse table mappings
infrastructure/rag/templates/langfuse-retention-cronjob.yaml	Implements CronJob for applying idempotent TTL policies to Langfuse tables using ALTER TABLE MODIFY TTL
infrastructure/rag/templates/langfuse-retention-hard-delete-cronjob.yaml	Implements optional CronJob for hard deletion of expired rows using ALTER TABLE DELETE WHERE with configurable mutation sync
infrastructure/rag/templates/_helpers.tpl	Adds shared template for ClickHouse connection environment variables used by both retention CronJobs
infrastructure/README.md	Documents the new trace retention feature with configuration examples and operational notes for non-Enterprise users