no distributed view in tensorboard

[ltm920716]

System Info

torch=2.1.2
torch-tb-profilier==0.4.3
cuda=11.8

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

hihi，
today I test the ‘Multiple GPUs one node’ part with 2 rtx4090 on one node，and I set the ‘--use_profilier’ parameter，but there is not ‘distributed’ view in tensorboard，I cannot find the reason，help please，thanks！

Error logs

no error，but missing distributed view in tensorboard

Expected behavior

show distributed view by torch.profilier in tensorboard

[wukaixingxp]

Hi! Can you share your command for this task? Did you see two workers in the profiler results? Can you share some screenshots? This may be a Pytorch Profiler with Tensorboard issue mentioned here.

[ltm920716]

hello @wukaixingxp,
thanks for your reply!

command:

torchrun --nnodes 1 --nproc_per_node 2 ./recipes/finetuning/finetuning.py --enable_fsdp --model_name ../models/Meta-Llama-3-8B-Instruct --batch_size_training 1 --use_peft --peft_method lora --dataset alpaca_dataset --save_model --low_cpu_fsdp --dist_checkpoint_root_folder model_checkpoints21 --dist_checkpoint_folder fine-tuned --pure_bf16  --output_dir fsdp_run21 --use_profiler --profiler_dir profiler21

and tensorboard screenshots:

there is no distributed view, by the way there is no 'model_checkpoints21' dir in my current workpath by --dist_checkpoint_root_folder

[init27]

Closing since the issue discussion is linked above-please reopen if you have any questions or new comments. Thanks!