Microsoft Excel mangles Diacritics in .csv files?

Question

I am programmatically exporting data (using PHP 5.2) into a .csv test file.
Example data: Numéro 1 (note the accented e). The data is utf-8 (

Answer 1

Writing a BOM to the output CSV file actually did work for me in Django:

def handlePersoonListExport(request):
    # Retrieve a query_set
    ...

    template = loader.get_template("export.csv")
    context = Context({
        'data': query_set,
    })

    response = HttpResponse()
    response['Content-Disposition'] = 'attachment; filename=export.csv'
    response['Content-Type'] = 'text/csv; charset=utf-8'
    response.write("\xEF\xBB\xBF")
    response.write(template.render(context))

    return response

For more info http://crashcoursing.blogspot.com/2011/05/exporting-csv-with-special-characters.html Thanks guys!

Answer 2

UTF-8 doesn't work for me in office 2007 without any service pack, with or without BOM (U+ffef or 0xEF,0xBB,0xBF , neither works) installing sp3 makes UTF-8 work when 0xEF,0xBB,0xBF BOM is prepended.

UTF-16 works when encoding in python using "utf-16-le" with a 0xff 0xef BOM prepended, and using tab as seperator. I had to manually write out the BOM, and then use "utf-16-le" rather then "utf-16", otherwise each encode() prepended the BOM to every row written out which appeared as garbage on the first column of the second line and after.

can't tell whether UTF-16 would work without any sp installed, since I can't go back now. sigh

This is on windows, dunno about office for MAC.

for both working cases, the import works when launching a download directly from the browser and the text import wizard doesn't intervence, it works like you would expect.

Answer 3

The CSV format is implemented as ASCII, not unicode, in Excel, thus mangling the diacritics. We experienced the same issue which is how I tracked down that the official CSV standard was defined as being ASCII-based in Excel.

Answer 4

With Ruby 1.8.7 I encode every field to UTF-16 and discard BOM (maybe).

The following code is extracted from active_scaffold_export:

<%                                                                                                                                                                                                                                                                                                                           
      require 'fastercsv'                                                                                                                                                                                                                                                                                                        
      fcsv_options = {                                                                                                                                                                                                                                                                                                           
        :row_sep => "\n",                                                                                                                                                                                                                                                                                                        
        :col_sep => params[:delimiter],                                                                                                                                                                                                                                                                                          
        :force_quotes => @export_config.force_quotes,                                                                                                                                                                                                                                                                            
        :headers => @export_columns.collect { |column| format_export_column_header_name(column) }                                                                                                                                                                                                                                
      }                                                                                                                                                                                                                                                                                                                          

      data = FasterCSV.generate(fcsv_options) do |csv|                                                                                                                                                                                                                                                                           
        csv << fcsv_options[:headers] unless params[:skip_header] == 'true'                                                                                                                                                                                                                                                      
        @records.each do |record|                                                                                                                                                                                                                                                                                                
          csv << @export_columns.collect { |column|                                                                                                                                                                                                                                                                              
            # Convert to UTF-16 discarding the BOM, required for Excel (> 2003 ?)                                                                                                                                                                                                                                     
            Iconv.conv('UTF-16', 'UTF-8', get_export_column_value(record, column))[2..-1]                                                                                                                                                                                                                                        
          }                                                                                                                                                                                                                                                                                                                      
        end                                                                                                                                                                                                                                                                                                                      
      end                                                                                                                                                                                                                                                                                                                        
    -%><%= data -%>

The important line is:

Iconv.conv('UTF-16', 'UTF-8', get_export_column_value(record, column))[2..-1]