I find that I have similar conversations a lot. On the one hand, you have a compact, efficient method of achieving your goal. On the other hand, you have something that the rest of your team might not understand, making it hard to maintain in the future.
My general rule is to ask if the technique being used is something that it is reasonable to expect programmers in general to know. In this case, I think that it is reasonable to expect programmers to know how to use boolean operators, so using xor in an if statement is okay.
As an example of something that wouldn't be okay, take the trick of using xor to swap two variables without using a temporary variable. That is a trick that I wouldn't expect everybody to be familiar with, so it wouldn't pass code review.