Encoding issue on filename with Java 7 on OSX with jnlp/webstart

Question

I have this problem that has been dropped on me, and have been a couple of days of unsuccessful searches and workaround attempts.

I have now an internal java swing prog

Answer 1

I take it it's acceptable to have maximal ASCII representation of the file name, which works in virtually any encoding.

First, you want to use specifically NFKD, so that maximum information is retained in the ASCII form. For example, "2⁵" becomes "25"rather than just "2", "fi" becomes "fi" rather than "" etc once the non-ascii and non-control characters are filtered out.

String str = "XXXYYY_è_ABCD/";
str = Normalizer.normalize(str, Normalizer.Form.NFKD);
str = str.replaceAll( "[^\\x20-\\x7E]", "");
//The file name will be XXXYYY_e_ABCD no matter what system encoding

You would then always pass filenames through this filter to get their filesystem name. You only lose is some uniqueness, I.E file asdé.txt is the same as asde.txt and in this system they cannot be differentiated.

Answer 2

Shot in the dark: File Encoding does not influence the way how the file names are created, just how the content gets written into the file - check this guy here: http://jonisalonen.com/2012/java-and-file-names-with-invalid-characters/

Here is a short entry from Apple: http://developer.apple.com/library/mac/#qa/qa1173/_index.html

Comparing this to http://docs.oracle.com/javase/tutorial/i18n/text/normalizerapi.html I would assume you want to use

normalized_string = Normalizer.normalize(target_chars, Normalizer.Form.NFD);

to normalize the file names before you pass them to the File constructor. Does this help?

Answer 3

It's a bug in the old-skool java File api, maybe just on a mac? Anyway, the new java.nio api works much better. I have several files containing unicode characters and content that failed to load using java.io.File and related classes. After converting all my code to use java.nio.Path EVERYTHING started working. And I replaced org.apache.commons.io.FileUtils (which has the same problem) with java.nio.Files...

...and be sure to read and write the content of file using an appropriate charset, for example:

Files.readAllLines(myPath, StandardCharsets.UTF_8)

Answer 4

I don't think there is a real solution to this problem, right now.

Meantime I came to the conclusion that the "C" environment variables printed from inside the program are from the Java Web Start sandbox, and (by design, apparently) you can't influence those using the jnlp.

The accepted (as accepted by the company) workaround/compromise was of launching the jnlp using javaws from a bash script.

Apparently, launching the jnlp from browser or from finder creates a new sandbox environment with the LANG not setted (so is setted to "C" that is equal to ASCII). Launching the jnlp from command line instead prints the right LANG from the system default, inheriting it from the shell.

This permits to at least preserve the autoupdating feature of the jnlp and dependencies.

Anyway, we sent a bug report to Oracle, but personally I'm not hoping it to be resolved anytime soon, if ever.

Answer 5

EDIT: After experimenting with OS X some more I realized my answer was totally wrong, so I'm redoing it.

If your JVM supports -Dfile.encoding=UTF-8 on the JVM command line, that might fix the issue. I believe that is a standard property but I'm not certain about that.

HFS Plus, like other POSIX-compliant file systems, stores filenames as bytes. But unlike Linux's ext3 filesystem, it forces filenames to be valid decomposed UTF-8. This can be seen here with the Python interpreter on my OS X system, starting in an empty directory.

$ python
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
>>> import os
>>> os.mkdir('\xc3\xa8')
>>> os.mkdir('e\xcc\x80')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 17] File exists: 'e\xcc\x80'
>>> os.mkdir('\x8f')
>>> os.listdir('.')
['%8F', 'e\xcc\x80']
>>> ^D
$ ls
%8F è

This proves that the directory name on your filesystem cannot be Mac-Roman encoded (i.e. with byte value 8F where the è is seen), as long as it's an HFS Plus filesystem. But of course, the JVM is not assured of an HFS Plus filesystem, and SMB and NFS do not have the same encoding guarantees, so the JVM should not assume this scheme.

Therefore, you have to convince the JVM to interpret file and directory names with UTF-8 encoding, in order to read the names as java.lang.String objects correctly.