Run PRAGMA optimize after commiting ingestion to the database
#579
+6
−11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is recommended practice for SQLite databases with short-lived connections such as the connection for ingestion: https://sqlite.org/pragma.html#pragma_optimize Note the callout of "low-quality indexes" on this page: https://sqlite.org/queryplanner-ng.html#howtofix Due to the nature of both topics and contract IDs, we have a huge variance in the K -> V ratio. Things like XLM have tons of event rows given fee events occur on every txn, and "transfer" is obviously a popular topic1 symbol. This explicitly breaks an indexing rule: > A low-quality index is one where there are **more than 10 or 20 > rows in the table that have the same value** for the > left-most column of the index. The solution is running `ANALYZE events;`, but this would take too long and grabs an exclusive lock on the DB, so we must periodically run `ANALYZE` via the `PRAGMA optimize;` directive which we execute at the end of every ingestion commit.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Ensures that we run query planner optimization at the end of every ledger ingest.
Stray optimization also included: since Core manages ledger entries now, the
ledgerEntriesfield of the db cache was unused entirely, so it has been removed.Why
See details in faa3d51 and this Slack thread.
Known limitations
It is unclear what the upper bound on this
PRAGMAis: obviously if it exceeds ledger close time this is hugely problematic. The alternative is to ask DB operators to runANALYZE eventsregularly which seems silly.