Since VirtualBox 6.0 support for Hyper-V has been added, but it is still experimental, which explains the bad performance (https://docs.oracle.com/en/virtualization/virtualbox/6.0/admin/hyperv-support.html).
I was able to fix this issue via disabling WSL2 in the Windows Features.
If you go into Control Panel > All Control Panel Items > Programs and Features > Windows Features
you will find a list, where you can disable or enable some Windows features.
Probably Hyper-V is already disabled in the features and you may wonder why VirtualBox still doesn't work properly. That is because WSL2 uses Hyper-V under the hood, even if it is not enabled for your custom use.
To make your VirtualBox usable again you have to uncheck the features Virtual Machine Platform
and Windows Subsystem for Linux
too and restart your PC for the effect to take place.
Clearly this will disable your PC to use WSL2 but at any time you may reactivate these features to work with WSL2 again. If you are not forced to use VirtualBox and WSL2 at the same time this works like a charm.