Rails: Faster way to perform updates on many records

Question

In our Rails 3.2.13 app (Ruby 2.0.0 + Postgres on Heroku), we are often retreiving a large amount of Order data from an API, and then we need to update or create each order in o

Answer 1

For PostgreSQL, there are several issues that the above approach does not address:

1. You must specify an actual table, not just an alias, in the update target table.
2. You cannot repeat the target table in the FROM phrase. Since you are joining the target table to a VALUES table (hence there is only one table in the FROM phrase, you won't be able to use JOIN, you must instead use "WHERE ".

3. You don't get the same "free" casts in a VALUES table that you do in a simple "UPDATE" command, so you must cast date/timestamp values as such (#val_cast does this).

   class ActiveRecord::Base
   
     def self.update!(record_list)
       raise ArgumentError "record_list not an Array of Hashes" unless record_list.is_a?(Array) && record_list.all? {|rec| rec.is_a? Hash }
       return record_list if record_list.empty?
   
       (1..record_list.count).step(1000).each do |start|
         field_list, value_list = convert_record_list(record_list[start-1..start+999])
         key_field = self.primary_key
         non_key_fields = field_list - [%Q["#{self.primary_key}"], %Q["created_at"]]
         columns_assign = non_key_fields.map {|field| "#{field} = #{val_cast(field)}"}.join(",")
         value_table = value_list.map {|row| "(#{row.join(", ")})" }.join(", ")
         sql = "UPDATE #{table_name} AS this SET #{columns_assign} FROM (VALUES #{value_table}) vals (#{field_list.join(", ")}) WHERE this.#{key_field} = vals.#{key_field}"
         self.connection.update_sql(sql)
       end
   
       return record_list
     end
   
     def self.val_cast(field)
       field = field.gsub('"', '')
       if (column = columns.find{|c| c.name == field }).sql_type =~ /time|date/
         "cast (vals.#{field} as #{column.sql_type})"
       else
         "vals.#{field}"
       end
     end
   
     def self.convert_record_list(record_list)
       # Build the list of fields
       field_list = record_list.map(&:keys).flatten.map(&:to_s).uniq.sort
   
       value_list = record_list.map do |rec|
         list = []
         field_list.each {|field| list <<  ActiveRecord::Base.connection.quote(rec[field] || rec[field.to_sym]) }
         list
       end
   
       # If table has standard timestamps and they're not in the record list then add them to the record list
       time = ActiveRecord::Base.connection.quote(Time.now)
       for field_name in %w(created_at updated_at)
         if self.column_names.include?(field_name) && !(field_list.include?(field_name))
           field_list << field_name
           value_list.each {|rec| rec << time }
         end
       end
   
       field_list.map! {|field| %Q["#{field}"] }
   
       return [field_list, value_list]
     end
   end