ISO-8859-1 encoding and binary data preservation

Question

I read in a comment to an answer by @Esailija to a question of mine that

ISO-8859-1 is the only encoding to fully retain the original binary data, wi

Answer 1

For an encoding to retain original binary data, it needs to map every unique byte sequence to an unique character sequence.

This rules out all multi-byte encodings (UTF-8/16/32, Shift-Jis, Big5 etc) because not every byte sequence is valid in them and thus would decode to some replacement character (usually ? or �). There is no way to tell from the string what caused the replacement character after it has been decoded.

Another option is to ignore the invalid bytes but this also means that infinite different byte sequences decode to the same string. You could replace invalid bytes with their hex encoding in the string like "0xFF". There is no way to tell if the original bytes legitimately decoded to "0xFF" so that doesn't work either.

This leaves 8-bit encodings, where every sequence is just a single byte. The single byte is valid if there is a mapping for it. But many 8-bit encodings have holes and don't encode 256 different characters.

To retain original binary data, you need 8-bit encoding that encodes 256 different characters. ISO-8859-1 is not unique in this. But what it is unique in, is that the decoded code point's value is also the byte's value it was decoded from.

So you have the decoded string, and encoded bytes, then it is always

(byte)str.charAt(i) == bytes[i] 

for arbitrary binary data where str is new String(bytes, "ISO-8859-1") and bytes is a byte[].

---

It also has nothing to do with Java. I have no idea what his comment means, these are properties of character encodings not programming languages.

Answer 2

"\u00F6" is not a byte array. It's a string containing a single char. Execute the following test instead:

public static void main(String[] args) throws Exception {
    byte[] b = new byte[] {(byte) 0x00, (byte) 0xf6};
    String s = new String(b, "ISO-8859-1"); // decoding
    byte[] b2 = s.getBytes("ISO-8859-1"); // encoding
    System.out.println("Are the bytes equal : " + Arrays.equals(b, b2)); // true
}

To check that this is true for any byte, just improve the code an loop through all the bytes:

public static void main(String[] args) throws Exception {
    byte[] b = new byte[256];
    for (int i = 0; i < b.length; i++) {
        b[i] = (byte) i;
    }
    String s = new String(b, "ISO-8859-1");
    byte[] b2 = s.getBytes("ISO-8859-1");
    System.out.println("Are the bytes equal : " + Arrays.equals(b, b2));
}

ISO-8859-1 is a standard encoding. So the language used (Java, C# or whatever) doesn't matter.

Here's a Wikipedia reference that claims that every byte is covered:

In 1992, the IANA registered the character map ISO_8859-1:1987, more commonly known by its preferred MIME name of ISO-8859-1 (note the extra hyphen over ISO 8859-1), a superset of ISO 8859-1, for use on the Internet. This map assigns the C0 and C1 control characters to the unassigned code values thus provides for 256 characters via every possible 8-bit value.

(emphasis mine)