Skip to content

Conversation

@antiguru
Copy link
Member

If we have many updates per source or target, propagate_all will update
an operator's pointstamps as many times as there are updates. Updating
the mutable antichain behind pointstamps can be expensive.

This change extracts the updates from the target and source changes, and
applies them in bulk on the pointstamps in the hope of reducing the cost
of maintaining the antichain.

Signed-off-by: Moritz Hoffmann antiguru@gmail.com

If we have many updates per source or target, `propagate_all` will update
an operator's pointstamps as many times as there are updates. Updating
the mutable antichain behind pointstamps can be expensive.

This change extracts the updates from the target and source changes, and
applies them in bulk on the pointstamps in the hope of reducing the cost
of maintaining the antichain.

Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>
@frankmcsherry
Copy link
Member

frankmcsherry commented Aug 27, 2025

Ah, so I understand what is going on here and it seems like a good idea. In particular, there can be a quadratic behavior here if an antichain for some target contains many times each with maybe a count of one, and then receives a batch of updates that retract each of those times. I bet we could test that theory with a pretty simple rig (but also can just believe it is true).

For code clarity, I was hoping to find something around Peekable that would allow a take_while adapter, perhaps something like https://docs.rs/itertools/latest/itertools/trait.Itertools.html#method.peeking_take_while.

@antiguru
Copy link
Member Author

Closing in favor of #703

@antiguru antiguru closed this Aug 28, 2025
@antiguru antiguru deleted the propagate_all_batch branch August 28, 2025 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants