I had the same trouble, and solved it by renaming my mipmap-anydpi directory to mipmap-anydpi-v26.
Apparently the ic_launcher.xml file confuses older Android versions, and this hides it from all but O. At any rate, my icons now work on all versions (down to SDK 11, at least).