Java: Detect non-displayable chars for a given Character Encoding

Question

I\'m currently working on an application to validate and parse CSV-files. The CSV files have to be encoded in UTF-8, although sometimes we get files in a false encoding. The

Answer 1

Byte sequences that cannot be decoded correctly will be replaced with the "replacement character", \uFFFD, which is displayed like this: �. However, if the output device doesn't support that character, it is likely to use a question mark (?) instead.

So, after decoding the UTF-8 data into String objects, search for occurrences of \uFFFD.

Alternatively, if you set up an InputStreamReader with an instance of CharsetDecoder that you create yourself, you can get a lot more control. For example, you can specify that if any byte sequences that cannot be decoded, an Exception should be raised. Or you can ignore them. Or you can specify a different character as the replacement character.

Answer 2

If the text is German and the encoding isn't UTF-8, it's probably windows-1252. Or something compatible with windows-1252, like ISO-8859-15. That being the case, Laforge's GuessEncoding should be all you need. I've used it quite a bit and never had a problem, and that's working almost exclusively with English text; German should be even easier to detect.

I see he still hasn't specified a license on his blog or in the source files, but I know those classes are used in Groovy, so that shouldn't be a problem.