HDDS-13994. Open Datanode RocksDB in read-only mode if the disk is full#9790
Open
siddhantsangwan wants to merge 2 commits intoapache:masterfrom
Open
HDDS-13994. Open Datanode RocksDB in read-only mode if the disk is full#9790siddhantsangwan wants to merge 2 commits intoapache:masterfrom
siddhantsangwan wants to merge 2 commits intoapache:masterfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In the Datanode for Schema V3, we have one RocksDB per volume. If the disk this DB is using gets full, RocksDB operations will fail with an exception and Ozone handles this by closing the DB. This means not only writes, but reads will also fail in this situation.
For example, I simulated a disk full in ozone and captured the db behaviour (grep for
IOError(NoSpace)andNo space left on device):With our recent improvements in this area, it's hard for the Datanode data disk (
hdds.datanode.dir, where the block files are) to get full - it should only happen if some other process takes up too much space or the cluster is misconfigured. By default RocksDB is on the same disk.Handling
We have a class
DatanodeStoreCachethat has a map for db path -> db handle. This serves as a cache and most operations (read/write etc.) go through this cache for getting the db handle. In this class, before returning the db handle, we can check if the db is closed. If closed, try to first open it in read-write mode. If that fails with a no space exception, try opening in read-only mode.Nice AI generated flowchart (the
RetryWindowElapsedcode path is not present in this PR, please ignore that. It's just an illustration of what we can try in followup pull requests):Other points
testNoSpaceOpenFallsBackToReadOnly.UNHEALTHYand the pipeline is closed (there's a case by case check).Next pull requests
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13994
How was this patch tested?
Added unit tests.
Also did manual testing using a docker compose cluster and simulating a full volume. Some grepped logs: