Prune skippable directories based on query path#612
Open
zapdos26 wants to merge 4 commits intoenvmodules:mainfrom
Open
Prune skippable directories based on query path#612zapdos26 wants to merge 4 commits intoenvmodules:mainfrom
zapdos26 wants to merge 4 commits intoenvmodules:mainfrom
Conversation
When a user requests a specific deep module path (e.g. tools/cat15/pkg10/3.0), the directory walker no longer enters sibling directories that cannot match. This avoids scanning the entire subtree under a modulepath root. For a 3000-modulefile tree (tools/category/pkg/version layout): - module load tools/cat15/pkg10/3.0: 13,007 → 157 syscalls (98.8% reduction) - module avail tools/cat15/pkg10: 13,005 → 155 syscalls (98.8% reduction) - module avail tools/cat15: 13,005 → 535 syscalls (95.9% reduction) Broad queries (module avail, wildcards, --contains) are unaffected. Signed-off-by: zapdos26 <zapdoskid@gmail.com>
Verify that deep module queries return correct results when directory pruning skips non-matching sibling directories. Tests cover: - 3-level and 2-level specific path queries (pruned) - top-level broad queries (not pruned) - exact version queries (pruned) - load through deep path (pruned) - wildcard queries (not pruned, returns all matches) Signed-off-by: zapdos26 <zapdoskid@gmail.com>
zapdos26
changed the title
When pruning non-matching sibling directories during a deep module query, still read the directory to pick up .modulerc and .version files. This ensures cross-directory aliases and symbol definitions are not lost when their containing directory is pruned. Adds a test for a cross-directory alias defined in cat1/.modulerc that references a module in cat2/. Signed-off-by: zapdos26 <zapdoskid@gmail.com>
When the query path contains virtual names (symbols/aliases like 'fld', 'sfld', 'dadj') that don't correspond to real directories, multi-depth pruning incorrectly skipped sibling directories needed for cross-directory alias resolution defined in .modulerc files. Gate pruning on filesystem existence: verify every path component in the query is a real directory before enabling prunespec. Virtual names fail the isdirectory check, so pruning is safely disabled. Real deep paths still get full multi-depth pruning for lstat savings. Signed-off-by: zapdos26 <zapdoskid@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds directory pruning for deep module queries, improving performance by skipping non-matching sibling directories during module discovery. The pruning logic is applied only to specific queries without wildcards, ensuring broad queries still return all results. Comprehensive tests are included to verify correct behavior.
Directory pruning enhancements
prunespecargument tofindModulesFromDirsAndFilesandfindModulesto support pruning directories based on the query path, and implemented logic to skip directories that cannot match the query. (tcl/modfind.tcl.in)getModulesto enable pruning only for specific deep queries without wildcards, preventing unnecessary directory scans. (tcl/modfind.tcl.in)Testing improvements
testsuite/modulefiles.deep/prunetest/cat1/pkg1/1.0testsuite/modulefiles.deep/prunetest/cat1/pkg1/2.0testsuite/modulefiles.deep/prunetest/cat1/pkg2/1.0testsuite/modulefiles.deep/prunetest/cat1/pkg2/2.0testsuite/modulefiles.deep/prunetest/cat2/pkg1/1.0testsuite/modulefiles.deep/prunetest/cat2/pkg1/2.0testsuite/modulefiles.deep/prunetest/cat2/pkg2/1.0testsuite/modulefiles.deep/prunetest/cat2/pkg2/2.0testsuite/modules.80-deep/026-pruning-deep.exp)