When a standard WPF button is clicked, it gets highlighted in blue (probably using the blue color from whatever Windows theme is set), and it stays highlighted until you interac
That is simply the focussed state. To turn it off you will have to change the focussed state. It's easiest by using Blend.
I do not recommend setting Focusable to false because it interferes with using the keyboard