How do you change headers in a CSV File with FasterCSV then save the new headers?

Question

I'm having trouble understanding the :header_converters and :converters in FasterCSV. Basically, all I want to do is change column headers to their appropriate column names.

something like:

FasterCSV.foreach(csv_file, {:headers => true, :return_headers => false, :header_converters => :symbol, :converters => :all} ) do |row|
    puts row[:some_column_header] # Would be "Some Column Header" in the csv file.

execpt I don't umderstand :symbol and :all in the converter parameters.

Answer 1

The :all converter means that it tries all of the built-in converters, specifically:

:integer:   Converts any field Integer() accepts.
:float:     Converts any field Float() accepts.
:date:      Converts any field Date::parse() accepts.
:date_time: Converts any field DateTime::parse() accepts.

Essentially, it means that it will attempt to convert any field into those values (if possible) instead of leaving them as a string. So if you do row[i] and it would have returned the String value '9', it will instead return an Integer value 9.

Header converters change the way the headers are used to index a row. For example, if doing something like this:

FastCSV.foreach(some_file, :header_converters => :downcase) do |row|

You would index a column with the header "Some Header" as row['some header'].

If you used :symbol instead, you would index it with row[:some_header]. Symbol downcases the header name, replaces spaces with underscores, and removes characters other than a-z, 0-9, and _. It's useful because comparison of symbols is far faster than comparison of strings.

If you want to index a column with row['Some Header'], then just don't provide any :header_converter option.

---

EDIT:

In response to your comment, headers_convert won't do what you want, I'm afraid. It doesn't change the values of the header row, just how they are used as an index. Instead, you'll have to use the :return_headers option, detect the header row, and make your changes. To change the file and write it out again, you can use something like this:

require 'fastercsv'

input = File.open 'original.csv', 'r'
output = File.open 'modified.csv', 'w'
FasterCSV.filter input, output, :headers => true, :write_headers => true, :return_headers => true do |row|
  change_headers(row) if row.header_row?
end
input.close
output.close

If you need to completely replace the original file, add this line after doing the above:

FileUtils.mv 'modified.csv', 'original.csv', :force => true

Answer 2

I've found a simple approach for solving this problem. The FasterCSV library works just fine. I'm sure that ~7 years between when the post was created to now may have something to do with it, but I thought it was worth noting here.

When reading CSV files, the FasterCSV :header_converters option isn't well documented, in my opinion. But, instead of assigning a symbol (header_converters: :symbol) one can assign a lambda (header_converters: lambda {...}). When the CSV library reads the file it transforms the headers using the lambda. Then, one can save a new CSV file that reflects the transformed headers.

For example:

options = {
  headers: true,
  header_converters: lambda { |h| HEADER_MAP.keys.include?(h.to_sym) ? HEADER_MAP[h.to_sym] : h }
}

table = CSV.read(FILE_TO_PROCESS, options)

File.open(PROCESSED_FILE, "w") do |file|
  file.write(table.to_csv)
end

Answer 3

Rewriting CSV file headers is a common requirement for anyone converting exported CSV files into imports.

I found the following approach gave me what I needed:

lookup_headers = { "old": "new", "cat": "dog" } # The desired header swaps

CSV($>, headers: true, write_headers: true) do |csv_out|
  CSV.foreach( ARGV[0],
               headers: true, 
               # the following lambda replaces the header if it is found, leaving it if not...
               header_converters: lambda{ |h| lookup_headers[h] || h}, 
               return_headers: true) do |master_row|

    if master_row.header_row?
      # The headers are now correctly replaced by calling the updated headers
      csv_out << master_row.headers
    else
      csv_out << master_row
    end
  end
end

Hope this helps!