Rails importing CSV fails due to mal-formation

问题

I get a CSV:MalFormedCSVError when I try to import a file using the following code:

  def import_csv(filename, model)
    CSV.foreach(filename, :headers => true) do |row|
      item = {}
      row.to_hash.each_pair do |k,v|
          item.merge!({k.downcase => v})
      end
        model.create!(item)
    end
  end

The csv files are HUGE, so is there a way I can just log the bad formatted lines and CONTINUE EXECUTION with the remainder of the csv file?

回答1:

You could try handling the file reading yourself and let CSV work on one line at a time. Something like this:

File.foreach(filename) do |line|
  begin
    CSV.parse(line) do |row|
      # Do something with row...
    end
  rescue CSV::MalformedCSVError => e
    # complain about line
  end
end

You'd have to do something with the header line yourself of course. Also, this won't work if you have embedded newlines in your CSV.

回答2:

One problem with using File to manually go through each line in the file is that CSV files can contain fields with \n (newline character) in them. File will take that to indicate a newline and you will end up trying to parse a partial row.

Here is an another approach that might work for you:

@csv = CSV.new('path/to/file.csv')

loop do
  begin
    row = @csv.shift
    break unless row
    # do stuff
  rescue CSV::MalformedCSVError => error
    # handle the error
    next
  end
end

The main downside that I see with this approach is that you don't have access to the CSV row string when handling the error, just the CSV::MalformedCSVError itself.

来源：https://stackoverflow.com/questions/8125719/rails-importing-csv-fails-due-to-mal-formation