How is this website fixing the encoding?

Question

I am trying to turn this text:

×וויר. העתיד של רשתות חברתיות והתקשורת ×©×œ× ×•

Into this text:

Answer 1

Since the issue was a MySQL fault with double-encoded UTF8 strings, MySQL is the right way to solve it.

Running the following commands will solve it -

• mysqldump $DB_NAME -u $DB_USER -p -h $DB_HOST.EXAMPLE.NET --add-drop-table --default-character-set=latin1 > export.sql - latin1 is used here to force MySQL not to split the characters, and should not be used otherwise.
• cp export{,.utf8}.sql - making a backup copy.
• sed -i -e 's/latin1/utf8/g' export.utf8.sql - Replacing the latin1 with utf8 in the file, in order to import it as UTF-8 instead of 8859-1.
• mysql $DB_NAME -u $DB_USER -p -h $DB_HOST.EXAMPLE.NET < export.utf8.sql - import everything back to the database.

This will solve the issue in about ten minutes.

Answer 2

You might want to look here - the accepted answer to this question shows a way how to guess the encoding of a byte[]. All you have to ensure then, is getting the proper bytes from the gibberish. Guessing might always fail, of course...

Answer 3

If you look closely at the gibberish, you can tell that each Hebrew character is encoded as 2 characters - it appears that של is encoded as של.

This suggests that you are looking at UTF8 or UTF16 as ASCII. Converting to UTF8 will not help because it is already ASCII and will keep that encoding.

You can read each pair of bytes and reconstruct the original UTF8 from them.

Here is some C# I came up with - this is very simplistic (doesn't fully work - too many assumptions), but I could see some of the characters converted properly:

private string ToProperHebrew(string gibberish)
{
   byte[] orig = Encoding.Unicode.GetBytes(gibberish);
   byte[] heb = new byte[orig.Length / 2];

   for (int i = 0; i < orig.Length / 2; i++)
   {
     heb[i] = orig[i * 2];
   }

   return Encoding.UTF8.GetString(heb);
}

If appears that each byte was re-encoded as two bytes - not sure what encoding was used for this, but discarding one byte seemed to be the right thing for most doubled up characters.

Answer 4

Based on Oded's and Teddy's answers, I came up with this method, which worked for me:

public String getProperHebrew(String gibberish){
    byte[] orig = gibberish.getBytes(Charset.forName("windows-1252"));

    try {
        return new String(orig, "UTF-8");
    } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
        return "";
    }
}

Answer 5

gibberish.encode('windows-1252').decode('utf-8', 'replace')

Answer 6

You can use the meta tag to set the proper encoding for your page. Here is an example how you can do that:

<meta http-equiv="Content-Type" content="text/html; charset=Windows-1255" />

I suppose that this encoding would do the work.