PHP/MySQL encoding problems. � instead of certain characters

Question

I have come across some problems when inputting certain characters into my mysql database using php. What I am doing is submitting user inputted text to a database. I cannot

Answer 1

The likeliest cause of the problem is that the database connection is set to latin1 but you are feeding it text encoded in UTF-8. The simplest way to solve this is to convert your input into what the client expects:

$quote = iconv("UTF-8", "WINDOWS-1252//TRANSLIT", $quote);

(What MySQL calls latin1 is windows-1252 in the rest of the world.) Note that many characters, such as the quotation dash U+2015 that you use there, cannot be represented in this encoding and will be converted into something else. Ideally you should change the column encoding to utf8.

An alternative solution: set the database connection to utf8. It doesn't matter how the columns are encoded: MySQL internally converts text from the connection encoding into the storage encoding, you can keep the columns as latin1 if you want to. (If you do, the quotation dash U+2015 will be turned into a question mark ? because it's not in latin1)

How to set the connection encoding depends on what library you are using: if you use the deprecated MySQL library it's mysql_set_charset, if MySQLi it's mysqli_set_charset, if PDO add encoding=utf8 to the DSN.

If you do this you'll have set the page encoding to UTF-8 with the Content-Type header. Otherwise you would be having the same problem with the browser: feeding it text encoded in UTF-8 when it's expecting something else:

header("Content-Type: text/html; charset=utf-8");

Answer 2

You will need to set your database in utf-8 yes. There is many ways to do it. By changin the config file, via phpmyadmin or by calling php function (sorry memory blank) right before insert and update the mysql.

Unfortunately, i think you will have to re-enter any data you entered before.

One thing you also need to know, from personnal experience, make sure all table with relation have the same collation or you won'T be able to JOIN them.

as reference: http://dev.mysql.com/doc/refman/5.6/en/charset-syntax.html

Also, i can be a apache setting. We've experienced the same issue on 'free-hosting' server as well as on my brother's server. Once switched to another server, all the charater's became neat. Verfiy you apache setting, sorry but i can't bting more light on apache's config.

Answer 3

The solutions provided are helpful if starting from scratch. Putting all possible connections to UTF-8 is indeed the safest. UTF-8 is the most used charset on the net for a variety of reasons.

Some suggestions and a word of warning:

• copy the tables you want to sanitize with a unique prefix (tmp_)
• although your db-connection is forced to utf8, check you General Settings collation, change to utf8_bin if that was not done yet
• you need to run this on the local server
• the funny char error is mostly due to mixing LATIN1 with UTF-8 configurations. This solution is designed for this. It could work with other used char-sets that LATIN1 but I haven't checked this
• check these tmp_tables extensively before copying back to the original

Builds the 2 array needed for the magic:

$chars = get_html_translation_table(HTML_ENTITIES, ENT_QUOTES, "UTF-8");
$LATIN1 = $UTF8 = array();
while (list($key,$val) = each ($chars)) {
    $UTF8[] = $key;
    $LATIN1[] = $val;
}

Now build up the routines you need: (tables->)rows->fields and at each field call

$row[$field] = mysql_real_escape_string(str_replace($LATIN1 , $UTF8 , $row[$field]));
$q[] = "$field = '{$row[$field]}'";

Finally build up and send the query:

mysql_query("UPDATE $table SET " . implode(" , " , $q) . " WHERE id = '{$row['id']}' LIMIT 1");

Answer 4

As mentioned by others, you need to convert to UTF8 from end to end if you want to support "special" characters. This means your web page, PHP, mysql connection and mysql table. The web page is fairly simple, just use the meta tag for UTF8. Ideally your headers would say UTF8 also.

<meta http-equiv="content-type" content="text/html; charset=utf-8" />

Set your PHP to use UTF8. Things would probably work anyway, but it's a good measure to do this:

mb_internal_encoding('UTF-8');
mb_http_output('UTF-8');
mb_http_input('UTF-8');

For mysql, you want to convert your table to UTF8, no need to export/import.

ALTER TABLE table_name CONVERT TO CHARACTER SET utf8

You can, and should, configure mysql to default utf8. But you can also run the query:

 SET NAMES UTF8

as the first query after establishing a connection and that will "convert" your database connection to UTF8.

That should solve all your character display problems.

Answer 5

Get rid of everything you just need to follow these two points, every problem regarding special languages characters will be resolved.

1- You need to define the collation of your table to be utf8_general_ci.

2- define <meta http-equiv="content-type" content="text/html; charset=utf-8"> in the HTML after head tag.

2- You need to define the mysql_set_charset('utf8',$link_identifier); in the file where you made connection with the database and right after the selection of database like 'mysql_select_db' use this 'mysql_set_charset' this will allow you to add and retrieve data properly in what ever the language it is.

Answer 6

change the MySQL collation to utf8_unicode_ci or utf8_general_ci, including the table and the database.