refactor: dropping unnecessary indexes from db #19451

benesjan · 2026-01-09T03:26:52Z

We were fetching note hash index and event commitment index to verify that a given note or event exists. This was redundant because we were already fetching tx effect and verifying the corresponding commitment is present in the effect.

For this reason in this PR I drop the indexes. (Note that for notes we still need to fetch nullifier index to verify whether the note has already been nullified or not)

This change forces me to rework how events get ordered (we want them to be returned in the order in which they were emitted). Before the events were ordered by this global event commitment index. With this PR they get ordered by block_number-tx_index_in_block-commitment_index_in_tx. We can do this cheaply because we already had all this info returned from node from the getTxEffect endpoint.

Fixes https://linear.app/aztec-labs/issue/F-218/remove-note-and-nullif-indexes-from-note-and-event-storage

benesjan · 2026-01-09T03:27:09Z

refactor: dropping unnecessary indexes from db #19451 👈 (View in Graphite)
refactor: atomic reorg handling #19364
refactor: dropping noteHashLeafIndexMap #19443
next

This stack of pull requests is managed by Graphite. Learn more about stacking.

yarn-project/pxe/src/storage/private_event_store/private_event_store.test.ts

yarn-project/pxe/src/storage/note_store/note_store.ts

yarn-project/pxe/src/notes/note_service.test.ts

yarn-project/pxe/src/contract_function_simulator/oracle/note_packing_utils.ts

yarn-project/pxe/src/storage/note_store/note_store.test.ts

As Martin mentioned [here](#19327 (comment)) makes sense to have the reorg handling be atomic. In this PR I tackle that. Note that this PR doesn't need to be in stack with #19443 but #19451 depends on both so putting this one into stack was the only way to unblock myself working on #19451.

yarn-project/pxe/src/storage/note_store/note_store.test.ts

AztecBot · 2026-01-12T22:50:43Z

Flakey Tests

🤖 says: This CI run detected 3 tests that failed, but were tolerated due to a .test_patterns.yml entry.

\033FLAKED\033 (8;;http://ci.aztec-labs.com/11226004f376fe16�11226004f376fe168;;�):  yarn-project/end-to-end/scripts/run_test.sh simple src/e2e_p2p/multiple_validators_sentinel.parallel.test.ts "collects attestations for validators in proposer node when block is not published" (115s) (code: 1) group:e2e-p2p-epoch-flakes (\033Jan Beneš\033: refactor: dropping unnecessary indexes from db (#19451))
\033FLAKED\033 (8;;http://ci.aztec-labs.com/675aba67f5f557a2�675aba67f5f557a28;;�): yarn-project/end-to-end/scripts/run_test.sh web3signer src/composed/web3signer/e2e_multi_validator_node_key_store.test.ts (39s) (code: 1) (\033Jan Beneš\033: refactor: dropping unnecessary indexes from db (#19451))
\033FLAKED\033 (8;;http://ci.aztec-labs.com/d46cd2faf32d5fea�d46cd2faf32d5fea8;;�):  yarn-project/end-to-end/scripts/run_test.sh simple src/e2e_p2p/gossip_network.test.ts (428s) (code: 1) group:e2e-p2p-epoch-flakes (\033Jan Beneš\033: refactor: dropping unnecessary indexes from db (#19451))

yarn-project/pxe/src/notes/note_service.ts

mverzilli

Nice!

mverzilli · 2026-01-13T09:24:53Z

yarn-project/pxe/src/storage/note_store/note_store.ts

-        await this.#notes.set(noteIndex, dao.toBuffer());
-        await this.#notesToScope.set(noteIndex, scope.toString());
-        await this.#nullifierToNoteId.set(dao.siloedNullifier.toString(), noteIndex);
+        const noteId = dao.siloedNullifier.toString();


This makes so much more sense this way. I would suggest renaming noteId to noteNullifier or even just nullifier: I'm currently editing this same file and I keep coming back to "what was noteId again?" So it seems there's no reason to have that "naming indirection"

In #rewindNullifiersAfterBlock we have noteIdsToReinsert variable which would not work well with the noteNullifier naming so I think it's better to keep it as is.

then I think a little comment above the corresponding async maps clarifying semantics would be helpful so people don't need to read the whole file to know "I see, noteId is just the nullifier"

Added it in b169e49

Anyway if you would feel like changing that later on feel free to do so. If you come up with some reasonable name for noteIdsToReinsert then I think it's a good change.

yarn-project/pxe/src/storage/note_store/note_store.ts

mverzilli · 2026-01-13T15:49:24Z

yarn-project/pxe/src/storage/note_store/note_store.ts

-        await this.#notes.set(noteIndex, dao.toBuffer());
-        await this.#notesToScope.set(noteIndex, scope.toString());
-        await this.#nullifierToNoteId.set(dao.siloedNullifier.toString(), noteIndex);
+        const noteId = dao.siloedNullifier.toString();


then I think a little comment above the corresponding async maps clarifying semantics would be helpful so people don't need to read the whole file to know "I see, noteId is just the nullifier"

We were fetching note hash index and event commitment index to verify that a given note or event exists. This was redundant because we were already fetching tx effect and verifying the corresponding commitment is present in the effect. For this reason in this PR I drop the indexes. (Note that for notes we still need to fetch nullifier index to verify whether the note has already been nullified or not) This change forces me to rework how events get ordered (we want them to be returned in the order in which they were emitted). Before the events were ordered by this global event commitment index. With this PR they get ordered by `block_number-tx_index_in_block-commitment_index_in_tx`. We can do this cheaply because we already had all this info returned from node from the `getTxEffect` endpoint. Fixes https://linear.app/aztec-labs/issue/F-218/remove-note-and-nullif-indexes-from-note-and-event-storage Co-authored-by: Jan Beneš <janbenes1234@gmail.com>

benesjan mentioned this pull request Jan 9, 2026

refactor: atomic reorg handling #19364

Merged

benesjan force-pushed the 01-09-refactor_dropping_unnecessary_indexes_from_db branch from f4ea108 to e7a50d3 Compare January 9, 2026 03:27

benesjan force-pushed the 01-06-refactor_atomic_reorg_handling branch from eba9c04 to e5f9c22 Compare January 9, 2026 03:27

benesjan mentioned this pull request Jan 9, 2026

refactor: dropping noteHashLeafIndexMap #19443

Merged