Files.readAllBytes vs Files.lines getting MalformedInputException

Question

I would have thought that the following two approaches to read a file should behave equally. But they don\'t. The second approach is throwing a MalformedInputException

Answer 1

This has to do with character encoding. Computers only deal with numbers. To store text, the characters in the text have to be converted to and from numbers, using some scheme. That scheme is called the character encoding. There are many different character encodings; some of the well-known standard character encodings are ASCII, ISO-8859-1 and UTF-8.

In the first example, you read all the bytes (numbers) in the file and then convert them to characters by passing them to the constructor of class String. This will use the default character encoding of your system (whatever it is on your operating system) to convert the bytes to characters.

In the second example, where you use Files.lines(...), the UTF-8 character encoding will be used, according to the documentation. When a sequence of bytes is found in the file that is not a valid UTF-8 sequence, you'll get a MalformedInputException.

The default character encoding of your system may or may not be UTF-8, so that can explain a difference in behaviour.

You'll have to find out what character encoding is used for the file, and then explicitly use that. For example:

String content = new String(Files.readAllBytes(Paths.get("_template.txt")),
        StandardCharsets.ISO_8859_1);

Second example:

Stream<String> lines = Files.lines(Paths.get("_template.txt"),
        StandardCharsets.ISO_8859_1);

Answer 2

Files.lines by default uses the UTF-8 encoding, whereas instantiating a new String from bytes will use the default system encoding. It appears that your file is not in UTF-8, which is why it is failing.

Check what encoding your file is using, and pass it as the second parameter.

Answer 3

To complement Jesper's answer, what happens here (and is undocumented!) is that Files.lines() creates a CharsetDecoder whose policy is to reject invalid byte sequences; that is, its CodingErrorAction is set to REPORT.

This is unlike what happens for nearly all other Reader implementations provided by the JDK, whose standard policy is to REPLACE. This policy will result in all unmappable byte sequences to emit a replacement character (U+FFFD).

Answer 4

2017 use:

 Charset.forName("ISO_8859_1") instead of Charsets.ISO_8859_1