What is the Unicode U+001A Character? Aka 0x1A

Question

The U+001A character appears frequently in error messages relating to character encoding. What is the U+001A character?

Answer 1

As far as I can tell U+001A is a legacy character in Unicode. Its only reason for existence is that it was already defined in ASCII as the substitute character ("... used in the place of a character that is recognized to be invalid or in error or that cannot be represented on a given device."). It was also sometimes used to end a character stream (which is probably a common source of problems)

In Unicode that function is taken over by the U+FFFD REPLACEMENT CHARACTER.

Answer 2

U+001A is defined in the Unicode Standard as a control character with the name SUBSTITUTE, and it belongs to a group characterized as follows, in chapter 16 of the standard: “There are 65 code points set aside in the Unicode Standard for compatibility with the C0 and C1 control codes defined in the ISO/IEC 2022 framework [...] The Unicode Standard provides for the intact interchange of these code points, neither adding to nor subtracting from their semantics. The semantics of the control codes are generally determined by the application with which they are used. However, in the absence of specific application uses, they may be interpreted according to the control function semantics specified in ISO/IEC 6429:1992.”

ISO 6429 is effectively equivalent to ECMA 48, which mentions this code as having the short name SUB, too, and defines it as follows: “SUB is used in the place of a character that has been found to be invalid or in error. SUB is intended to be introduced by automatic means.” This reflects the definition of this control code in Ascii.

Thus, in general, U+001A may be used to indicate a character-level data error, such as the presence of bytes, in purported character data, that have no interpretation in the character encoding being applied. Loosely speaking, it would thus mean “bad character data”, but more appropriately “malformed data, when trying to interpret data as characters”. However, in Unicode, U+FFFD REPLACEMENT CHARACTER is more appropriate, as it has specific Unicode semantics.

Since the question has been tagged with “xml”, it needs to be noted that in XML 1.0, U+001A is forbidden, by clause 2.2 Characters. Note that the comment “any Unicode character, excluding the surrogate blocks, FFFE, and FFFF” is misleading (but comments are non-normative); U+001A is a Unicode character, though it is not a graphic character and its effect is not defined in the Unicode Standard.

Answer 3

That's the Ctrl+Z control code. It's kinda special in Windows, which inherited it from DOS which inherited it from CP/M. Its legacy use was as an end-of-text marker, similar to how Ctrl+D is used in Unix.

Seeing it in an error message or used as the fall-back character for a failed encoding conversion is however quite unusual. I'd double-check the code and make sure it isn't U+003F or U+FFFD, the more typical encoding fallback characters. Or just a plain quirk of the specific code you are dealing with.