Read/write .txt file with special characters

Question

I open Notepad (Windows) and write

Some lines with special characters
Special: Žđšćč

and go to Save As...

Answer 1

Are you using the character the conversion as part of servlet request/response ? If yes, request.setEncoding("UTF-8")
or
response.setCharacterEncoding("UTF-8")

should solve your purpose.

Answer 2

Notepad does not save special symbols correctly. I had a similar problem and I used Notepad++ instead and selected UTf-8 encoding from there. When I did this, my program no longer crashed when applying String library methods to it unlike when I created the text file in Notepad.

Answer 3

Notepad may not be able to handle non-ascii characters. Try another text editor. If you want to stick to what's available in windows install, try wordpad.

Answer 4

It's the output console which doesn't support those characters. Since you're using Eclipse, you need to ensure that it's configured to use UTF-8 for this. You can do this by Window > Preferences > General > Workspace > Text File Encoding > set to UTF-8.

See also:

• Unicode - How to get the characters right?

---

Update as per the updated question and the comments, apparently the UTF-8 BOM is the culprit. Notepad by default adds the UTF-8 BOM on save. It look like that the JRE on your HTC doesn't swallow that. You may want to consider to use the UnicodeReader example as outlined in this answer instead of InputStreamReader in your code. It autodetects and skips the BOM.

FileInputStream fis = new FileInputStream(new File(fileName));
UnicodeReader ur = new UnicodeReader(fis, "UTF-8");
BufferedReader in = new BufferedReader(ur);

---

Unrelated to the actual problem, it's a good practice to close resources in finally block so that you ensure that they will be closed in case of exceptions.

BufferedReader reader = null;
try {
    reader = new BufferedReader(new UnicodeReader(new FileInputStream(fileName), "UTF-8"));
    // ...
} finally {
    if (reader != null) try { reader.close(); } catch (IOException logOrIgnore) {}
}

Also unrelated, I'd suggest to put Pattern p = Pattern.compile(","); outside the loop, or even make it a static constant, because it's relatively expensive to compile it and it's unnecessary to do this everytime inside a loop.

Answer 5

"Not all sequences of bytes are valid UTF-8."

See

http://en.wikipedia.org/wiki/UTF-8

under "Invalid byte sequences" for specific details.

Answer 6

Your code looks right - but a very common, and easy, error is to misstake what is printed to screen to what's in the String. Check with a debugger if the string isn't already read right.