SwiftUI Move Buttons in List

Question

In my macOS SwiftUI App, I want to rearrange items in a List which are Buttons but I can only drag the items when I am outside of the buttons frame