fix: add global step number in hf moe chpt

[YashasviChaurasia]

Description of the change

add global step to checkpoint path for moe models finetuned with sharded_state.
Example: hf_converted_checkpoint -> safetensors-1234

Related issue number

How to verify the PR

Was the PR tested

• I have added >=1 unit test(s) for every new method I have added.
• I have ensured all unit tests pass

[github-actions]

Thanks for making a pull request! 😃
One of the maintainers will review and advise on the next steps.

[dushyantbehl]

Can you rename this to safetensors_converted_checkpoint-{state.global_step}

if this would require big changes to llmb we can ignore

[ashokponkumar]

There will be no change required in llmb at all. But a long name might create issues at times with cos and other file systems. How about safetensors-{state.global_step} or something simpler

[dushyantbehl]

safetensors-{state.global_step} good for me too thanks @ashokponkumar

[dushyantbehl]

Can we check if the path exists and the files exist before printing this message?
or print some error in the other case

[YashasviChaurasia]

we do just that in the code below the path.join , if the dir does not exist that means accelerate callback didn't create dir and hence we use base dir , also as we know accelerate callback would always run before trainer controller callback hence this also ensures logic to check if hf checkpoint or not, in what scenario would we need a error btw?:

        if os.path.isdir(hf_converted_path):
            kwargs["hf_path"] = hf_converted_path
        else:
            kwargs["hf_path"] = base_path