My company has a large C# code base. Well over half of this code is our core engine for creating, reading, modifying, calculating
Expose your .NET API as ASMX web services and you should be good to go.
EDIT: For more heavy-usage scenarios, it would be worth looking into Windows Communication Foundation (WCF). This has built-in, configurable support for security, streaming, different transport scenarios (HTTP, TCP/IP, local named pipes). You are not restricted to SOAP message encoding, but that would probably be the easiest way to interop with Java.
I'm not too sure about your exact scenario, but if you're dealing with large files and the .NET code and Java code are both running locally, you can just save the file to the user's hard drive using .NET and then fetch it from your Java app.