Single Machine Multi-GPU Distributed Training Advantage over Non-Distributed Training

Question

We are not talking about the multiple machines/hosts.

Assuming we have a single host/machine with multiple GPUs, what is the advantage of adopting multi-GPU distribut