Reading UTF-8 text files with ReadList

Question

Is it possible to use ReadList to read UTF-8 (or any other) encoded text files using ReadList[..., Word], or is it ASCII-only? If it\'s ASCII-only

Answer 1

This seems to work

FromCharacterCode[ToCharacterCode[ReadList["raw.php.txt", Word]], "UTF-8"]

The timings I get for the linked test file are

FromCharacterCode[ToCharacterCode[ReadList["test.txt", Word]], "UTF-8"]); // Timing

(* ==> {0.000195, Null} *)

Import["test.txt", "Text"]; // Timing

(* ==> {0.01784, Null} *)

Answer 2

If I leave out Word, this works:

$CharacterEncoding = "UTF-8";

ReadList["UTF8.txt"]

This however is a failure, because the data is not read as strings.

Please try this on a larger file and report its performance:

FromCharacterCode[BinaryReadList["UTF8.txt"], "UTF-8"]