Why is the encoding so important and why do XML parsers get confused?
If, for example, an XML document saved as ASCII is sent to an OAI harvester which thinks it is receiving UTF-8, the following will happen: • ASCII values less than 127 will be fine, but • any ASCII values greater than 127 will be a problem. For example, pi (ASCII 227) in binary is 11100011. The XML parser will read this as the first byte of a three byte UTF-8 character. Completely different to what was intended! There’s a good tutorial on XML and character encoding at http://skew.org/xml/tutorial/.
