I discovered that all the Swift libraries (libswiftCore.dylib, libswiftCoreGraphics.dylib,...) exist twice in the IPA. After decompressing the IPA, there is one version of these
I've found that playing around with 'Embedded content contains Swift' value yielded some good results, and was actually crucial for my app with an Apple Watch extension to pass validation.
I have written an answer about the details for the Apple Watch fix here.