feat: Support arbitrary boolean expressions with subqueries (OR, NOT) #3791
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
❌ 5 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
Author
|
@coderabbitai review |
These tests verify the RFC "Arbitrary Boolean Expressions with Subqueries": - OR with multiple subqueries (initial snapshot, move-in, move-out) - NOT with subqueries (exclusion, move-in on removal, move-out on addition) - Mixed conditions (subqueries combined with field filters) - Complex boolean expressions (De Morgan's laws: NOT (A AND B), NOT (A OR B)) - Edge cases (NULL values, empty subqueries, tag consistency) - Resume state preservation Tests tagged with @tag :dnf_subqueries require the RFC implementation to pass. These tests currently verify that the expected behavior (no 409/ must_refetch on OR/NOT subquery changes) will work once implemented. The 15 passing tests verify current correct behavior. The 6 excluded tests define the expected behavior for the RFC implementation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 1 of RFC "Arbitrary Boolean Expressions with Subqueries". Implements Electric.Replication.Eval.Decomposer which converts WHERE clause ASTs to Disjunctive Normal Form (DNF). This enables the system to track which conditions caused a row's inclusion in a shape. Key features: - Simplification: double negation elimination, nested AND/OR flattening - Negation Normal Form: De Morgan's laws push NOT to literals - DNF conversion: distribution of AND over OR - Position assignment: deterministic mapping for stable client state - Subquery detection: identifies complex expressions requiring DNF The decomposer handles: - Simple AND/OR combinations - NOT expressions with De Morgan transformation - Nested NOT (double negation elimination) - Mixed subquery and field conditions - Position stability across shape restarts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Extends the Shape struct with: - dnf_decomposition: The result of decomposing the WHERE clause into DNF - position_to_dependency_map: Maps DNF positions to dependency handles This enables tracking which subquery conditions affect which positions in the boolean expression, supporting arbitrary boolean expressions with subqueries (OR, NOT, multiple IN clauses). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implements Phase 3 of the RFC: - compute_active_conditions/4: Evaluates each atomic subexpression and returns a list of booleans indicating which conditions are satisfied - evaluate_dnf/2: Evaluates DNF disjuncts against active conditions - evaluate_conjunction/2: Evaluates a single conjunction - satisfied_disjuncts/2: Returns indices of satisfied disjuncts These functions support arbitrary boolean expressions by allowing per-position evaluation and DNF-based inclusion checking. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update consumer.ex: Don't invalidate shapes when they have a valid DNF decomposition that can handle OR/NOT/multiple subqueries - Update shape.ex: Generate DNF-aware tag structure when we have a valid decomposition - one pattern per disjunct - Update subquery_moves.ex: Handle DNF-aware move-out patterns with correct position indexing per disjunct This enables shapes with complex boolean expressions (OR, NOT, multiple IN clauses) to avoid invalidation and use proper move-in/move-out handling instead. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Ensures that unexpected subexpression formats don't cause errors during tag structure generation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Documents completed work: - DNF Decomposer module with tests - Shape module DNF integration - WhereClause active conditions computation - Consumer invalidation updates - Integration test scaffold Documents remaining work: - NOT inversion in MoveHandling - OR move-out when all disjuncts false - Deduplication for rows in multiple disjuncts - Existing test updates Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add with_dependency_handles/2 to compute position-to-dependency mappings
- Add position_negated?/2 to check if a position is negated (NOT IN)
- Update process_move_ins to return {state, notification} tuple
- For negated positions (NOT IN), move-in triggers move-out and vice versa
- Update Consumer to handle notifications from both move-ins and move-outs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When a value moves OUT of a NOT IN subquery, rows with that value should move INTO the outer shape. This requires removing the NOT from the WHERE clause when querying for new rows. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add exclusion clauses to move-in queries for OR shapes to prevent duplicate inserts when row already in shape via another disjunct - Fix SQL escaping in tag hash computation to handle apostrophes - Fix nil handling in make_tags when stack_id or shape_handle is nil - Update router tests to expect correct behavior now that OR and NOT IN shapes are properly supported: - NOT IN subquery generates move-out instead of 409 - OR with subquery generates move-out/move-in properly - Two subqueries at same level work with move-in - Update subquery_router_test to use collect_messages for move-out tests where control messages may arrive before expected deletes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
All critical functionality for DNF-based subquery handling is now complete: - NOT inversion in MoveHandling - OR deduplication via exclusion clauses - Router tests updated to expect proper behavior instead of 409 - All 1389 tests pass Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This adds the active_conditions field to row messages, which tells clients which atomic conditions in the DNF decomposition are satisfied for each row. This enables proper move-out handling when rows match multiple disjuncts. Changes: - Add active_conditions field to NewRecord, UpdatedRecord, DeletedRecord - Compute active_conditions in fill_move_tags using DNF decomposition - Include active_conditions in LogItems message headers - Generate active_conditions SQL for initial snapshots in querying.ex - Add active_conditions parsing to elixir-client Headers struct - Add test for active_conditions in response headers Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create changeset for arbitrary boolean expressions feature - Update implementation plan with completed work - Apply mix format to all changed files Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When a shape has a subquery combined with other conditions in its WHERE clause (e.g., `parent_id IN (SELECT ...) AND status = 'published'`), changes could be incorrectly skipped if the sublink value was in a pending move-in, even when the new record didn't match other parts of the WHERE clause (like a column equality check). The fix adds a check to verify the new record actually matches the full WHERE clause before deciding to skip a change based on move-in coverage. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
robacourt
force-pushed
the
rob/arbitrary-boolean-expressions-with-subqueries
branch
from
5a3dbbb to
3105a9c
Compare
Contributor
|
Found 8 test failures on Blacksmith runners: Failures
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements RFC "Arbitrary Boolean Expressions with Subqueries" which extends Electric's subquery support from single
IN (SELECT ...)conditions to arbitrary boolean expressions with OR, NOT, and multiple subqueries.What's Changed
WHERE project_id IN (SELECT ...) OR assigned_to IN (SELECT ...)WHERE project_id NOT IN (SELECT ...)WHERE (a IN sq1 AND b='x') OR c NOT IN sq2Key Implementation Details
DNF Decomposer (
lib/electric/replication/eval/decomposer.ex)Active Conditions (
lib/electric/shapes/where_clause.ex)active_conditionsarray indicating which atomic conditions are satisfiedMove-in/Move-out Handling (
lib/electric/shapes/consumer/move_handling.ex)Message Format (
lib/electric/log_items.ex,lib/electric/shapes/querying.ex)active_conditionsarray included in row message headersactive_conditionsactive_conditionsfieldTest plan
test/electric/plug/subquery_router_test.exs)test/electric/replication/eval/decomposer_test.exs)test/electric/shapes/where_clause_test.exs)Test Coverage
Related
docs/rfcs/arbitrary-boolean-expressions-with-subqueries.mdpackages/sync-service/IMPLEMENTATION_PLAN.md🤖 Generated with Claude Code