Shouldn't multi-GPU mode be faster? with toy example

Question

I am trying 2 GPU model (both are Nvidia 2070 SUPER) on ubuntu, but seems multi-GPU mode is indeed slower than single mode.

I tried many times with different depth.