I have the issue in two completely different situations, which makes it really weird.
This can happen if the buttons are placed close to the bottom edge of the display. There's a conflict with the system edge gesture to present Control Center (or App Switcher on iPhone X and iPad). It delays touches for your buttons until it's determined the user isn't performing a system gesture.
If you would like, you can tell the system to give precedence to your gestures over the system gestures.
override func preferredScreenEdgesDeferringSystemGestures() -> UIRectEdge {
return .bottom
}
Do note however that this will require the user swipe twice to activate the system gesture.