Invoking a large set of SQL from a Rails 4 application

Question

I have a Rails 4 application that I use in conjunction with sidekiq to run asynchronous jobs. One of the jobs I normally run outside of my Rails application is

Answer 1

I had the same problem with a set of sql statements that I needed to execute all in one call to the server. What worked for me was to set up an initializer for Mysql2 adapter (as explained in infused answer) but also do some extra work to process multiple results. A direct call to ActiveRecord::Base.connection.executewould only retrieve the first result and issue an Internal Error.

My solution was to get the Mysql2 adapter and work directly with it:

client = ActiveRecord::Base.connection.raw_connection

Then, as explained here, execute the query and loop through the results:

client.query(multiple_stms_query)
while client.next_result
  result = client.store_result
  # do something with it ...
end

Answer 2

Executing one query is - as outlined by other people - quite simply done through

ActiveRecord::Base.connection.execute("SELECT COUNT(*) FROM users")

You are talking about a 20.000 line sql script of multiple queries. Assuming you have the file somewhat under control, you can extract the individual queries from it.

script = Rails.root.join("lib").join("script.sql").read # ah, Pathnames

# this needs to match the delimiter of your queries
STATEMENT_SEPARATOR = ";\n\n"


ActiveRecord::Base.transaction do
  script.split(STATEMENT_SEPARATOR).each do |stmt|
    ActiveRecord::Base.connection.execute(stmt)
  end
end

If you're lucky, then the query delimiter could be ";\n\n", but this depends of course on your script. We had in another example "\x0" as delimiter. The point is that you split the script into queries to send them to the database. I wrapped it in a transaction, to let the database know that there is coming more than one statement. The block commits when no exception is raised while sending the script-queries.

If you do not have the script-file under control, start talking to those who control it to get a reliable delimiter. If it's not under your control and you cannot talk to the one who controls it, you wouldn't execute it, I guess :-).

UPDATE

This is a generic way to solve this. For PostgreSQL, you don't need to split the statements manually. You can just send them all at once via execute. For MySQL, there seem to be solutions to get the adapter into a CLIENT_MULTI_STATEMENTS mode.