Error “invalid byte sequence” while restoring PostgreSQL database

Question

Earlier today, I was trying to restore my PostgreSQL (8.1.22) database from production using pgAdmin III. However, after the restoration procedure finished, it started throwing errors like:

WARNING: errors ignored on restore: 4 

Also, upon investigation I found that out of all the tables 3 tables hadn't been restored (contained 0 rows). When I checked the log, I found the follwoing error near the 3 tables:

pg_restore: [archiver (db)] Error from TOC entry 5390; 0 442375 TABLE DATA tablename postgres
pg_restore: [archiver (db)] COPY failed: ERROR:  invalid byte sequence for encoding "UTF8": 0xea0942
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
CONTEXT:  COPY tablename , line 7875

I tried researching my problem on Google but that yielded no results. Kindly help in restoring these three tables without any errors.

Answer 1

Older versions of PostgreSQL were not as strict on UTF-8 compliance than newer versions. Presumably you're trying to restore data containing invalid UTF-8 from such an older version into a newer version.

The invalid strings must be cleaned up. You may follow that procedure for each table that wasn't imported due to these errors:

1. Extract the contents of the table from the dump file into a SQL plain text file:

   pg_restore --table=tablename --data-only dumpfile >plaintext.sql
   

2. Remove the invalid characters in a text editor or automatically with iconv:

   iconv -c -f UTF-8 -t UTF-8 <plaintext.sql >plaintext-cleaned.sql
   

3. Import the sanitized data:

   psql dbname < plaintext-cleaned.sql