“Incorrect string value” when trying to insert UTF-8 into MySQL via JDBC?

Question

This is how my connection is set:
Connection conn = DriverManager.getConnection(url + dbName + \"?useUnicode=true&characterEncoding=utf-8\", userName, password

Answer 1

I wanted to combine a couple of posts to make a full answer of this since it does appear to be a few steps.

1. Above advice by @madtracey

/etc/mysql/my.cnf or /etc/mysql/mysql.conf.d/mysqld.cnf

[mysql]
default-character-set=utf8mb4

[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0

[mysqld]
##
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4'
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

Again from advice above all jdbc connections had characterEncoding=UTF-8and characterSetResults=UTF-8 removed from them

With this set -Dfile.encoding=UTF-8 appeared to make no difference.

I could still not write international text into db getting same failure as above

Now using this how-to-convert-an-entire-mysql-database-characterset-and-collation-to-utf-8

Update all your db to use utf8mb4

ALTER DATABASE YOURDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Run this query that gives you what needs to be rung

SELECT CONCAT(
'ALTER TABLE ',  table_name, ' CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;  ', 
'ALTER TABLE ',  table_name, ' CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;  ')
FROM information_schema.TABLES AS T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` AS C
WHERE C.collation_name = T.table_collation
AND T.table_schema = 'YOURDB'
AND
(C.CHARACTER_SET_NAME != 'utf8mb4'
    OR
 C.COLLATION_NAME not like 'utf8mb4%')

Copy paste output in editor replace all | with nothing post back into mysql when connected to correct db.

That is all that had to be done and all seems to work for me. Not the -Dfile.encoding=UTF-8 is not enabled and it appears to work as expected

E2A Still having an issue ? I certainly am in production so it turns out you do need to check over what has been done by above, since it sometimes does not work, here is reason and fix in this scenario:

show create table user

  `password` varchar(255) CHARACTER SET latin1 NOT NULL,
  `username` varchar(255) CHARACTER SET latin1 NOT NULL,

You can see some are still latin attempting to manually update the record:

ALTER TABLE user CONVERT TO CHARACTER SET utf8mb4;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

So let's narrow it down:

mysql> ALTER TABLE user change username username varchar(255) CHARACTER SET utf8mb4 not NULL;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
mysql> ALTER TABLE user change username username varchar(100) CHARACTER SET utf8mb4 not NULL;
Query OK, 5 rows affected (0.01 sec)

In short I had to reduce the size of that field in order to get the update to work.

Now when I run:

mysql> ALTER TABLE user CONVERT TO CHARACTER SET utf8mb4;
Query OK, 5 rows affected (0.01 sec)
Records: 5  Duplicates: 0  Warnings: 0

It all works