I know Flash can use ExternalInterface to call Javascript functions but I don\'t know if it can call applet also like that. Maybe it can be done by SWF -> JS -> Applet, and back
Yes. Sockets. Your Java applet can open a server socket, probably a HTTP socket, and Flash/Flex can post content to the socket. Your applet will have to be signed by a known CA to be trusted, but once that's done you can pretty much do what you want.
Applets can communicate with JS, and JS can control applets. See these examples of Java/JavaScript interaction.