Characters really allowed in XML

Question

Due to some parser error when parsing certain (supposedly) XML data, I had a look at the XML standard to figure out what is really allowed. My current qualms are with regard to

Answer 1

Our friendly wikipedia has a section devoted to this. I think the explanation is in much easier terms. http://en.wikipedia.org/wiki/XML#Escaping

Valid Characters

Unicode code points in the following ranges are valid in XML 1.0 documents:[9]

• U+0009, U+000A, U+000D: these are the only C0 controls accepted in XML 1.0;
• U+0020–U+D7FF, U+E000–U+FFFD: this excludes some (not all) non-characters in the BMP (all surrogates, U+FFFE and U+FFFF are forbidden);
• U+10000–U+10FFFF: this includes all code points in supplementary planes, including non-characters.

XML 1.1[10] extends the set of allowed characters to include all the above, plus the remaining characters in the range U+0001–U+001F. At the same time, however, it restricts the use of C0 and C1 control characters other than U+0009, U+000A, U+000D, and U+0085 by requiring them to be written in escaped form (for example U+0001 must be written as
or its equivalent). In the case of C1 characters, this restriction is a backwards incompatibility; it was introduced to allow common encoding errors to be detected.

The code point U+0000 is the only character that is not permitted in any XML 1.0 or 1.1 document.

Escaping

XML provides escape facilities for including characters which are problematic to include directly. For example:

• The characters "<" and "&" are key syntax markers and may never appear in content outside a CDATA section.[13]
• Some character encodings support only a subset of Unicode. For example, it is legal to encode an XML document in ASCII, but ASCII lacks code points for Unicode characters such as "é".
• It might not be possible to type the character on the author's machine.

• Some characters have glyphs that cannot be visually distinguished from other characters: examples are

  • non-breaking space ( ) " "

  • compare space ( ) " "

  • Cyrillic Capital Letter A (А) "А"

  • compare Latin Capital Letter A (A) "A"

There are five predefined entities:

• < represents "<"
• > represents ">"
• & represents "&"
• ' represents '
• " represents "

All permitted Unicode characters may be represented with a numeric character reference. Consider the Chinese character "中", whose numeric code in Unicode is hexadecimal 4E2D, or decimal 20,013. A user whose keyboard offers no method for entering this character could still insert it in an XML document encoded either as 中 or 中. Similarly, the string "I <3 Jörg" could be encoded for inclusion in an XML document as "I <3 Jörg".

"�" is not permitted, however, because the null character is one of the control characters excluded from XML, even when using a numeric character reference.[14] An alternative encoding mechanism such as Base64 is needed to represent such characters.