Invoking a large set of SQL from a Rails 4 application

Question

I have a Rails 4 application that I use in conjunction with sidekiq to run asynchronous jobs. One of the jobs I normally run outside of my Rails application is

Answer 1

1. use available database tools to handle the complex queries, such as views, stored procedures etc and call them as other people already suggested (ActiveRecord::Base.connection.execute and ModelClass.find_by_sql for example)- it might very well cut down significantly on query preparation time in the DB and make your code easier to handle
   • http://dev.mysql.com/doc/refman/5.0/en/create-view.html
   • http://dev.mysql.com/doc/connector-cpp/en/connector-cpp-tutorials-stored-routines-statements.html
2. abstract your query input parameters into a hash so you can pass it on to sidekiq, don't send SQL strings as this will probably degrade performance (due to query preparation time) and make your life more complicated due to funny SQL driver parsing bugs
3. run your complex queries in a dedicated named queue and set concurrency to such a value that will prevent your database of getting overwhelmed by the queries as they smell like they could be pretty db heavy
   • https://github.com/mperham/sidekiq/wiki/API
   • https://github.com/mperham/sidekiq/wiki/Advanced-Options
4. have a look at Squeel, its a great addition to AR, it might be able to pull some of the things you are doing
   • https://github.com/activerecord-hackery/squeel
   • http://railscasts.com/episodes/354-squeel

I'll assume you use MySQL for now, but your mileage will vary depending on the DB type that you use. For example, Oracle has some good gems for handling stored procedures, views etc, for example https://github.com/rsim/ruby-plsql

Let me know if some of this stuff doesn't fit your use case and I'll expand