Php/ODBC encoding problem

Question

I use ODBC to connect to SQL Server from PHP. In PHP I read some string (nvarchar column) data from SQL Server and then want to insert it to mysql database. When I try to insert such value to mysql database table I get this mysql error:

Incorrect string value: '\xB3\xB9ow...' for column 'name' at row 1

For string with all ASCII characters everything is fine, the problem occurs when non-ASCII characters (from some European languages) exist.

So, in more general terms: there is a Unicode string in MS SQL Server database, which is retrieved by PHP trough ODBC. Then it is put in sql insert query (as value for utf-8 varchar column) which is executed for mysql database.

Can someone explain to me what is happening in this situation in terms of encoding? At which step what character encoding convertions may take place?

I use: PHP 5.2.5, MySQL5.0.45-community-nt, MS Sql Server 2005.

PHP have to run on Linux platform.

UPDATE: The error doesn't occur when I call utf8_encode($s) on this string and use that value in mysql insert query, but then the inserted string doesn't display correctly in mysql database (so that utf8 encoding only worked for enforcing proper utf8 string, but it loses correct characters).

Answer 1

First you have the encoding of the DB. Then you have the encoding used by the ODBC client.

If the encoding of your ODBC client connection does not match the one of the DB, the ODBC layer will automatically transcode your data, in some cases.

The trick here is to force the encoding of the ODBC client connection.

For an "all UTF-8" setup :

$conn=odbc_connect(DB_DSN,DB_USR,DB_PWD);
odbc_exec($conn, "SET NAMES 'UTF8'");
odbc_exec($conn, "SET client_encoding='UTF-8'");

// processing here

This works perfectly with PostgreSQL + Php 5.x. The exact syntax and options depends on the DB vendor.

You can find very useful and clear additional info for MySql here : http://dev.mysql.com/doc/refman/5.0/fr/charset-connection.html

hope this helps.

Answer 2

Maybe you can use the PDO extension, if it will make any difference?

There is a user contributed comment here that suggests to change the data types in sql server to somethig else, if this is not possible look at the users class that casts fields.

Answer 3

I have no experience with ODBC via PHP, but with the mysql functions PHP seems to default to ASCII and UTF8 connections need to be made explicit if you want to avoid trouble.

Are you sure PHP and the MySQL server are communicating in UTF8? Until PHP 6 the Unicode support tends to be annoyingly inconistent like that.

I remember that the MySQL docs mention a connection string parameter to tweak the Unicode encoding.

From your description it sounds like PHP is treating the connection as ASCII-only.