How to profiling ENTIRE pytorch code when GPUs are present?

Question

I want to profile my entire training and eval pytorch code. I am using custom dataloaders (e.g. torchmeta library) and novel pytorch libraries (e.g. higher library) and I se