I have a UIButton underneath a (transparent) UIView. The UIView above has a UISwipeGestureRecognizer added to it, and that is its only purpose - to detect certain swipe gestures
In my case, I fixed it by not using a button, but rather a UITapGestureRecognizer. My pan gesture recognizer was added to the background view, and the tap gesture was added to a view above it.