pandas to_sql gives unicode decode error

Question

I have a pandas dataframe I loaded via read_csv that I am trying to push to a database via to_sql when I attempt

df.to_sql(\"assessmentinfo_pivot\", util.ENGINE)         


        

Answer 1

I experienced the exact same issue with the combination pymysql and pandas.to_sql

Update, here is what worked for me:

Instead of passing the charset as an argument, try attaching it directly to the connection string:

connect_string = 'mysql+pymysql://{}:{}@{}:{}/{}?charset=utf8'.format(DB_USER, DB_PASS, DB_HOST, DB_PORT, DATABASE)

The problem seems to happen in pymysql and the cause for the error seemingly is that the encoding you define is not properly forwarded and set when the pymsql connection is set.

For the sake of debugging, I harcoded

encoding = 'utf-8

in the pymysql _do_execute_manyfunction and that explained it to me.