I discovered that all the Swift libraries (libswiftCore.dylib, libswiftCoreGraphics.dylib,...) exist twice in the IPA. After decompressing the IPA, there is one version of these
I've found that playing around with 'Embedded content contains Swift' value yielded some good results, and was actually crucial for my app with an Apple Watch extension to pass validation.
I have written an answer about the details for the Apple Watch fix here.
I am experiencing the same issue. I have read some answers which suggest that it isn't a big deal and I have to say that it looks like it isn't.
My app contains 3 targets - main app and 2 today extensions. When I create an archive its size is a whooping 126 MB. Most of it is Swift support libraries which are about 45 MB in my case and they are included twice.
However, when I check the binary size through test flight it is 23.6 MB. Now, this is still a lot considering that my application is a simple utility.
However, it is not too much if you take into account that Swift libraries have to be included. So far I would say that even though these libraries are included twice in the IPA they are not included twice in the actual binary.
This obviously doesn't answer the original question and I am also eagerly waiting for someone to come up with the exact solution, but it might be helpful.
I think that if your code already uses Swift, you have no set the Flag "Embedded Content contains Swift code" in Build Options to NO, otherwise Xcode will add the libraries because it thinks that you use some frameworks that have swift code