Each time the device is being rotated, Android will destroy the current activity and re-create it with the new configuration.
An easy way to avoid a
OK, never mind. I think I just missed the correct one:
android:screenOrientation="user"
Together with the other configChanges values, this works perfectly.