How windows notepad interpret characters

问题

I was wondering how windows interpret characters, for instance:

I maked a file with an Hexeditor with the 3 bytes E3 81 81. Those bytes are the "ぁ" character encoded as UTF-8.

I open the notepad and it displays "ぁ"

I don't specified the encoding of the file, i just created the bytes. and the notepad interpret it correctly.

Is the notepad guessing what encoding probably is? or is the Hex editor saving those bytes with a specific encoding.

回答1:

If the file only contains these three bytes, then there is no information at all about which encoding to use.

A byte is just a byte, and there is no way to include any encoding information in it. Besides, the hex editor doesn't even know that you intended to decode the data as text.

Notepad normally uses ANSI encoding, so if it reads the file as UTF-8 then it has to guess the encoding based on the data in the file.

If you save a file as UTF-8, Notepad will put the BOM (byte order mark) EF BB BF at the beginning of the file.

回答2:

Notepad makes an educated guess. I don't know the details, but loading the first few kilobytes and trying to convert them from UTF-8 is very simple, so it probably does something similar to that.

回答3:

...and sometimes it gets it wrong... https://ychittaranjan.wordpress.com/2006/06/20/buggy-notepad/

回答4:

There is an easy and efficient way to check whether a file is in UTF-8. See Wikipedia: http://en.wikipedia.org/w/index.php?title=UTF-8&oldid=581360767#Advantages, fourth bullet point. Notepad probably uses this.

Wikipedia claims that Notepad used the IsTextUnicode function, which checks whether a patricular text is written in UTF-16 (it may have stopped using it in Windows Vista, which fixed the "Bush hid the facts" bug): http://en.wikipedia.org/wiki/Bush_hid_the_facts.