I\'ve found another question which brings more details regarding the problem and possible solutions. It seems like there is a known bug which is a subject for future improvement
I had exactly the same problems recently. The quick answer is you can't :) Even if you declare "private" modulemap, it can be always imported by your framework users. Please note, that usually, it is not a concern, especially with open source. You just say "this is an internal module, don't use it".
But (there is always but) - you can have behavior, that effectively works the same - allows you to use your Objective-C classes withing same framework target, without making them public. It works in closed source setup, I'm not 100% sure how would it behave with pods.
The case a bit too complex to paste everything here. I'm adding a link to my article about the topic, maybe it will help you. But speaking honestly - it might be a bit of overhead in your setup.
Creating Swift framework with private Objective-C members. The Good, the Bad, and the Ugly
Github example project
I found another question which brings more details regarding the problem and possible solutions (which don't work though). It seems like there is a known bug which is a subject for future improvements.
Objective C classes within an iOS Swift-based dynamic framework