Pandas - Writing an excel file containing unicode - IllegalCharacterError

Question

I have the following code:

import pandas as pd

x = [u\'string with some unicode: \\x16\']
df = pd.DataFrame(x)

If I try to write this datafram

Answer 1

Not a Unicode issue as such... \x16 (or in Unicode strings \u0016 refers to the same character) is ASCII control code 22 (SYN). Pandas says it's invalid to have control codes (other than tab and newlines) in an Excel file, and though I don't know much about Excel files it would certainly be impossible to include them in an XML 1.0 file, which is what's inside a xlsx.

So most likely there is no way to include arbitrary character sequences (with control codes) in an Excel. You should filter them out before writing, or if you really need to preserve the original data use some form of ad hoc encoding recognised only by your application.