feat: add infra for segments metrics (CM-708) #3728

ulemons · 2026-01-05T14:55:22Z

Introduce Segment Aggregate States for Scalable CDP Metrics

Summary

This PR introduces a new architecture for computing CDP dashboard metrics that is designed to be scalable, predictable, and safe at large data volumes.

The main change is the introduction of a segment-level aggregate states datasource, built daily via a COPY pipe, and three lightweight sinks that derive metrics for:

subprojects
projects
project groups

Heavy aggregation and DISTINCT logic is moved out of the sinks and into a single batch pipeline.

New Components

1. Datasource

cdp_segment_metrics_agg_states_ds

A new datasource that stores one row per subproject segment per daily snapshot, containing:

activity counters as aggregation states
DISTINCT member and organization states (uniqCombined)
hierarchy identifiers (segmentId, parentId, grandparentId)

This datasource uses AggregatingMergeTree and is optimized for hierarchical rollups via *Merge() functions.

2. COPY Pipe (Core of the Change)

cdp_segment_metrics_agg_states_copy.pipe

This is the most complex and important component introduced in this PR.

Responsibilities:

Reads existing aggregate datasources:
- cdp_member_segment_aggregates_ds
- cdp_organization_segment_aggregates_ds
Finalizes per-(segment, member/org) metrics
Computes:
- total activities
- last-30-days activities
- DISTINCT member states
- DISTINCT organization states
Attaches hierarchy metadata from segments
Writes a complete daily snapshot into cdp_segment_metrics_agg_states_ds

This pipe runs once per day and centralizes all heavy computation and DISTINCT logic.

3. Sinks

All sinks are intentionally simple and fast.
They only:

read the latest daily snapshot
merge aggregation states
format the output
export to Kafka

`cdp_dashboard_metrics_subproject_sink.pipe`

Publishes metrics at subproject level by finalizing segment-level states.

`cdp_dashboard_metrics_project_sink.pipe`

Rolls up subproject states to project level using parentId and state merging.

`cdp_dashboard_metrics_project_group_sink.pipe`

Rolls up subproject states to project group

Note

Introduces a scalable, state-based pipeline for CDP segment metrics.

New cdp_segment_metrics_ds (AggregatingMergeTree) storing per-segment daily states: count (total/last-30) and uniqCombined (members/orgs), with hierarchy IDs
COPY pipe cdp_segment_metrics_copy_pipe computes states from existing member/org aggregates and latest activities snapshot; restricts to valid segments, computes once per latest snapshot, reuses empty states; writes full daily snapshot
Three lightweight sinks finalize/roll up latest snapshot and export to Kafka (cdp_dashboard_metrics_per_segment_sink): cdp_segment_metrics_subproject_sink, cdp_segment_metrics_project_sink, cdp_segment_metrics_project_group_sink (rollups via parentId/grandparentId)
Scheduled runs: COPY at 09:00; sinks at 09:30/09:35/09:40

^{Written by Cursor Bugbot for commit 87dac21. This will update automatically on new commits. Configure here.}

github-actions · 2026-01-07T09:06:28Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2026-01-07T09:06:38Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/libs/tinybird/pipes/cdp_dashboard_metrics_subproject_sink.pipe

services/libs/tinybird/pipes/cdp_segment_metrics_agg_states_copy_pipe.pipe

services/libs/tinybird/datasources/cdp_segment_metrics_ds.datasource

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe

services/libs/tinybird/pipes/cdp_segment_metrics_agg_states_copy_pipe.pipe

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe

github-actions · 2026-01-07T14:55:25Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2026-01-07T14:57:50Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe

services/libs/tinybird/pipes/cdp_segment_metrics_subproject_sink.pipe

github-actions · 2026-01-07T15:18:51Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe

services/libs/tinybird/pipes/cdp_segment_metrics_project_sink.pipe

github-actions · 2026-01-07T16:37:42Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe

github-actions · 2026-01-07T16:52:03Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/libs/tinybird/datasources/cdp_segment_metrics_ds.datasource

github-actions · 2026-01-07T17:25:50Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe

github-actions · 2026-01-07T17:39:29Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/libs/tinybird/pipes/cdp_segment_metrics_subproject_sink.pipe

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe

github-actions · 2026-01-08T08:38:33Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

services/libs/tinybird/pipes/cdp_segment_metrics_subproject_sink.pipe

epipav

Looks good 👍 added one comment and a nitpick

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe

epipav · 2026-01-08T16:45:56Z

services/libs/tinybird/pipes/cdp_segment_metrics_project_sink.pipe

+    FROM segments AS s
+    LEFT JOIN
+        cdp_segment_metrics_ds AS sa


NIT: AFAIK, it's better to keep the smaller table on the right-hand side of the join for performance. If we see bad performance, let's try switching places

at the moment we have something like ~10 s, would that be reasonable for now ?

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe

ulemons self-assigned this Jan 5, 2026

ulemons added the Feature Created by Linear-GitHub Sync label Jan 5, 2026

ulemons requested a review from epipav January 5, 2026 15:01

ulemons marked this pull request as ready for review January 7, 2026 09:06

cursor bot reviewed Jan 7, 2026

View reviewed changes

epipav requested changes Jan 7, 2026

View reviewed changes

cursor bot reviewed Jan 7, 2026

View reviewed changes

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe Outdated Show resolved Hide resolved

cursor bot reviewed Jan 7, 2026

View reviewed changes

services/libs/tinybird/datasources/cdp_segment_metrics_ds.datasource Show resolved Hide resolved

cursor bot reviewed Jan 7, 2026

View reviewed changes

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe Outdated Show resolved Hide resolved

ulemons force-pushed the feat/total-segment-metrics branch from 355ef96 to bd87c2e Compare January 7, 2026 17:39

cursor bot reviewed Jan 7, 2026

View reviewed changes

services/libs/tinybird/pipes/cdp_segment_metrics_subproject_sink.pipe Show resolved Hide resolved

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe Outdated Show resolved Hide resolved

ulemons added 7 commits January 8, 2026 09:38

feat: add infra for segments metrics

8159c6b

fix: review comments

b7a92e7

fix: format

19b5e31

fix: format

61dd2e2

fix: project slug anomaly

41ef1ee

fix: use snapshot in the activityRelations

ab05144

fix: cursor suggestions

4e416ec

ulemons force-pushed the feat/total-segment-metrics branch from bd87c2e to 4e416ec Compare January 8, 2026 08:38

ulemons changed the title ~~feat: add infra for segments metrics~~ feat: add infra for segments metrics (CM-708) Jan 8, 2026

feat: optimize copy pipe

8995a85

cursor bot reviewed Jan 8, 2026

View reviewed changes

services/libs/tinybird/pipes/cdp_segment_metrics_subproject_sink.pipe Show resolved Hide resolved

epipav requested changes Jan 8, 2026

View reviewed changes

feat: remove snapshot

f903075

cursor bot reviewed Jan 8, 2026

View reviewed changes

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe Show resolved Hide resolved

services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe Show resolved Hide resolved

fix: grouping fix

87dac21

feat: add infra for segments metrics (CM-708) #3728

Are you sure you want to change the base?

feat: add infra for segments metrics (CM-708) #3728

Conversation

ulemons commented Jan 5, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Introduce Segment Aggregate States for Scalable CDP Metrics

Summary

New Components

1. Datasource

2. COPY Pipe (Core of the Change)

3. Sinks

cdp_dashboard_metrics_subproject_sink.pipe

cdp_dashboard_metrics_project_sink.pipe

cdp_dashboard_metrics_project_group_sink.pipe

Uh oh!

github-actions bot commented Jan 7, 2026

Uh oh!

github-actions bot commented Jan 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 7, 2026

Uh oh!

github-actions bot commented Jan 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 7, 2026

Uh oh!

Uh oh!

github-actions bot commented Jan 7, 2026

Uh oh!

Uh oh!

github-actions bot commented Jan 7, 2026

Uh oh!

Uh oh!

github-actions bot commented Jan 7, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

Uh oh!

epipav left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

epipav Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

ulemons Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ulemons commented Jan 5, 2026 •

edited by cursor bot

Loading

`cdp_dashboard_metrics_subproject_sink.pipe`

`cdp_dashboard_metrics_project_sink.pipe`

`cdp_dashboard_metrics_project_group_sink.pipe`

ulemons Jan 8, 2026 •

edited

Loading