“Incorrect string value” when trying to insert UTF-8 into MySQL via JDBC?

Question

This is how my connection is set:
Connection conn = DriverManager.getConnection(url + dbName + \"?useUnicode=true&characterEncoding=utf-8\", userName, password

Answer 1

I you only want to apply the change only for one field, you could try serializing the field

class MyModel < ActiveRecord::Base
  serialize :content

  attr_accessible :content, :title
end

Answer 2

MySQL's utf8 permits only the Unicode characters that can be represented with 3 bytes in UTF-8. Here you have a character that needs 4 bytes: \xF0\x90\x8D\x83 (U+10343 GOTHIC LETTER SAUIL).

If you have MySQL 5.5 or later you can change the column encoding from utf8 to utf8mb4. This encoding allows storage of characters that occupy 4 bytes in UTF-8.

You may also have to set the server property character_set_server to utf8mb4 in the MySQL configuration file. It seems that Connector/J defaults to 3-byte Unicode otherwise:

For example, to use 4-byte UTF-8 character sets with Connector/J, configure the MySQL server with character_set_server=utf8mb4, and leave characterEncoding out of the Connector/J connection string. Connector/J will then autodetect the UTF-8 setting.

Answer 3

Got the same problem, to save the data with utf8mb4 needs to make sure:

1. character_set_client, character_set_connection, character_set_results are utf8mb4: character_set_client and character_set_connection indicate the character set in which statements are sent by the client, character_set_results indicates the character set in which the server returns query results to the client.
   See charset-connection.

2. the table and column encoding is utf8mb4

For JDBC, there are two solutions:

Solution 1 (need to restart MySQL):

1. modify my.cnf like the following and restart MySQL:

   [mysql]
   default-character-set=utf8mb4
   
   [mysqld]
   character-set-server=utf8mb4
   collation-server=utf8mb4_unicode_ci
   

this can make sure the database and character_set_client, character_set_connection, character_set_results are utf8mb4 by default.

2. restart MySQL

3. change the table and column encoding to utf8mb4

4. STOP specifying characterEncoding=UTF-8 and characterSetResults=UTF-8 in the jdbc connector,cause this will override character_set_client, character_set_connection, character_set_results to utf8

Solution two (don't need to restart MySQL):

1. change the table and column encoding to utf8mb4

2. specifying characterEncoding=UTF-8 in the jdbc connector,cause the jdbc connector doesn't suport utf8mb4.

3. write your sql statment like this (need to add allowMultiQueries=true to jdbc connector):

   'SET NAMES utf8mb4;INSERT INTO Mytable ...';
   

this will make sure each connection to the server, character_set_client,character_set_connection,character_set_results are utf8mb4.
Also see charset-connection.

Answer 4

my solution is change the column type from varchar(255) to blob

Answer 5

In my case, I tried everything above, nothing worked. I am pretty sure, my database looks like below.

mysql  Ver 14.14 Distrib 5.7.17, for Linux (x86_64) using  EditLine wrapper

Connection id:      12
Current database:   xxx
Current user:       yo@localhost
SSL:            Not in use
Current pager:      stdout
Using outfile:      ''
Using delimiter:    ;
Server version:     5.7.17-0ubuntu0.16.04.1 (Ubuntu)
Protocol version:   10
Connection:     Localhost via UNIX socket
Server characterset:    utf8
Db     characterset:    utf8
Client characterset:    utf8
Conn.  characterset:    utf8
UNIX socket:        /var/run/mysqld/mysqld.sock
Uptime:         42 min 49 sec

Threads: 1  Questions: 372  Slow queries: 0  Opens: 166  Flush tables: 1  Open tables: 30  Queries per second avg: 0.144

so, I look up the column charset in every table

show create table company;

It turns out the column charset is latin. That's why, I can not insert Chinese into database.

 ALTER TABLE company CONVERT TO CHARACTER SET utf8;

That might help you. :)

Answer 6

this is not the recommendation solution.. But worth to share. Since my project are upgrade the DBMS from old Mysql to newest (8). But I cant change the table structure, only the DBMS config (mysql). The solution for mysql server.

test on Windows mysql 8.0.15 on mysql config search for

sql-mode="....."

uncomment it. Or in my case just type/add

sql-mode="NO_ENGINE_SUBSTITUTION"

why not recommended solution. because if you use latin1 (my case).. the data insert successly but not the content (mysql not respond with error!!) . for example you type info like this

bla \x12

it save

bla [] (box)

okay.. for my problem.. I can change the field to UTF8.. But there is small problem.. see above answer about other solution is failed because the word is not inserted because contain more than 2 bytes (cmiiw).. this solution make your insert data become box. The reasonable is to use blob.. and you can skip my answer.

Another testing related to this were.. using utf8_encode on your code before save. I use on latin1 and it was success (I'm not using sql-mode)! same as above answer using base64_encode .

My suggestion to analys your table requirement and tried to change from other format to UTF8