docs: site skeleton & initial placeholder content #62
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Lawrence Lane <llane@nvidia.com>
…language from training paradigms - Remove specific node count limits (16 nodes) lacking code evidence - Change 'Unlimited nodes' to 'Large multi-node clusters' for accuracy - Replace 'webdataset format' with 'Energon data loader' (verified in code) - Remove subjective time estimates (minutes/hours) from setup complexity - Improve precision of scalability descriptions throughout Signed-off-by: Lawrence Lane <llane@nvidia.com>
- Add complete fsdp config examples showing all 4 parallelism dimensions - Replace specific bandwidth numbers with general high-bandwidth requirement - Clarify pipeline bubble efficiency without unverified percentages - Remove unverified 2× memory claim for optimizer state sharding - Add runtime verification examples for checking parallelism config - Add note about automatic DP calculation in automodel - Improve DP calculation example with concrete numbers Signed-off-by: Lawrence Lane <llane@nvidia.com>
… structure, progressive disclosure, and clearer examples Signed-off-by: Lawrence Lane <llane@nvidia.com>
…bleshooting detail - Change content_type from tutorial to how-to (correct classification) - Improve progressive disclosure with clearer step labels - Add verified configuration parameters from source code - Enhance troubleshooting with specific symptoms and actionable solutions - Add checkpoint structure details and contents - Improve configuration override explanation with three-layer precedence - Add missing checkpoint configuration options - Fix list spacing for markdown lint compliance Signed-off-by: Lawrence Lane <llane@nvidia.com>
…l vs megatron); add automodel track (training + inference); add megatron track (data prep + training + inference); update index to route users by use case Signed-off-by: Lawrence Lane <llane@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
* init Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add sigma_min/amx Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add sigma_min/max Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename fientune.py to train.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add from_config Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pass scheduler and model Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * introduce NeMoWanPipeline Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add mode Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update build_model_and_optimizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update NeMoWanPipeline Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move examples Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * lint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * more lint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix 3rdparty & pyproject Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add torch Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * revert 3rdparty Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
Signed-off-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
* add tests Signed-off-by: linnan wang <wangnan318@gmail.com> * update test Signed-off-by: linnan wang <wangnan318@gmail.com> * update Signed-off-by: linnan wang <wangnan318@gmail.com> * update Signed-off-by: linnan wang <wangnan318@gmail.com> --------- Signed-off-by: linnan wang <wangnan318@gmail.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
* adding tests * ruff lint * ruff lint * ruff lint * Explicit mcore path override to use Megatron-Bridge's pinned submodule commit Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Update Megatron-Bridge submodule to latest main with correct Megatron-LM commit (3cbe5c68) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Add Mcore WAN pretrain mock test to CI/CD Signed-off-by: Pablo Garay <pagaray@nvidia.com> * lintfix Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix slow Docker build from Megatron-LM source Signed-off-by: Pablo Garay <pagaray@nvidia.com> * ci: Update gpu runners to use self-hosted-nemo (#48) * ci: Update gpu runners to use self-hosted-nemo Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Use uv run in test_mcore_wan_pretrain Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Ensure uv group megatron-bridge is used for test_mcore_wan_pretrain Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Update TRANSFORMERS_OFFLINE environment variable to 0 and increase timeout in test_mcore_wan_pretrain * Update TRANSFORMERS_OFFLINE environment variable to 0 and increase timeout in test_mcore_wan_pretrain Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Revert GHA changes Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Move uv run group call to L2_Mcore_Mock_Tests_GPU Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Set test back to 5 minute timeout Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Megatron fixes (#49) * Enhance DiT and Wan layer specifications - Updated `get_query_key_value_tensors` method in `dit_attention.py` to include an `output_gate` parameter and set `split_qkv` to default to `True`. - Modified `WanLayerWithAdaLN` class in `wan_layer_spec.py` to add `rotary_pos_cos_sin` parameter for improved positional encoding handling. * Implement ProcessGroupCollection initialization in DiT and Wan models - Added initialization of `pg_collection` in both `DiTCrossAttentionModel` and `WanModel` to ensure proper handling of process groups. - This change checks if `pg_collection` exists and is not None before assigning it, enhancing the robustness of the models. * Update CONTRIBUTING.md to include detailed setup instructions for development environment and Docker container usage. Added sections for building and running the container, as well as setting the PYTHONPATH for DFM. * Refactor import statements in dit_model.py to streamline dependencies. Removed redundant import of ProcessGroupCollection, enhancing code clarity and maintainability. * Refactor code style in DiT and Wan models - Updated string quotes in `dit_model.py` and `wan_model.py` for consistency, changing from single to double quotes. - Reformatted the `get_query_key_value_tensors` method call in `dit_attention.py` for improved readability by breaking it into multiple lines. * Revert M4 changes * Ruff * Ruff * Lint --------- Co-authored-by: Abhinav Garg <abhinavg@stanford.edu> * Revert "Revert GHA changes" This reverts commit d7ad1ab. * tempfortest: timeout setting Signed-off-by: Pablo Garay <pagaray@nvidia.com> * workflow dispatch Signed-off-by: Pablo Garay <pagaray@nvidia.com> * update Signed-off-by: Pablo Garay <pagaray@nvidia.com> * add logging Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Update test configuration for Mcore WAN pretraining - Increased the number of processes per node from 1 to 2 for distributed training. - Set the number of training iterations to 10 to enhance the training process. * More changes * Lint --------- Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Pablo Garay <pagaray@nvidia.com> Co-authored-by: Abhinav Garg <abhinavg@stanford.edu> Co-authored-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Reapply "Revert GHA changes" This reverts commit fdb911f. Signed-off-by: Pablo Garay <pagaray@nvidia.com> * update path per request Signed-off-by: Pablo Garay <pagaray@nvidia.com> * lintfix Signed-off-by: Pablo Garay <pagaray@nvidia.com> * update CONTRIBUTING.md Signed-off-by: Pablo Garay <pagaray@nvidia.com> * lintfix Signed-off-by: Pablo Garay <pagaray@nvidia.com> * adding v run --group megatron-bridge * update test * ruff lint * restore Dockerfile.ci * update .github/workflows/cicd-main.yml --------- Signed-off-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Huy Vu2 <huvu@login-eos02.eos.clusters.nvidia.com> Co-authored-by: Pablo Garay <pagaray@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Abhinav Garg <abhinavg@stanford.edu> Signed-off-by: Lawrence Lane <llane@nvidia.com>
* introduce step_scheduler section Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add step_scheduler section Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * lint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm dead code Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
* replace torch.stack with torch.cat Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
lbliii
force-pushed
the
llane/site-config-and-skeleton
branch
from
0ffe23d to
f2c7c94
Compare
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR provides:
note: the content staged here can either be used as a starting point or just as a reference to be deleted. I tried my best to make the content realistic, but ultimately these articles need SME input/direction. Future sections are yet to be determined.
To preview docs: