As I've explained in question, any mention of groups either in pom.xml or on command line resulted in reduction of executed tests count. Only way I've managed to avoid this is by using mavens profiles like this:
test-slow
maven-surefire-plugin
slow
and then running tests with
mvn -P test-slow test