We make use of a java application that manages a pinpad via the serial port. This works perfectly on windows with the Sun Comm.jar, the supplied dll and the properties file.
It's been a while since I've done this. I remember that you can't just open /dev/ttyS0 as a file and I/O to it.
I remember having the same problem with Sun IO jars also.
The solution I remember settling on was to create a small command-line program in C that echoed stdin to the serial port and serial port in to stdout. Then, I launched this program from my java process. I don't have the code but there are lots of examples on the web for c programs that write to linux serial ports.