I have a few native libraries that are fairly big and bloating the application binary size. I looked into APK splitting but maintaining and QAing multiple APKs is not something
First off, if you're worring about binary size, you don't really need arm64-v8a, all those devices can run the armeabi-v7a binaries just fine. Only if you really need to cram the last extra performance out of it, it might be worthwhile.
As for armeabi and ARMv6; Android itself doesn't officially support it any longer, since Android 4.4 (October 2013) - and since Android 4.0 it should be much less common (from that version, AOSP source requires modifications to still build for ARMv6). So in practice, if you aren't supporting versions below 4.4, you can drop that one without any significant loss.
Also, for x86; many of those devices ship with surprisingly decent emulation of arm binaries, so those can manage with the armeabi-v7a version just fine as well.