“MySQL server has gone away” with Ruby on Rails

Question

After our Ruby on Rails application has run for a while, it starts throwing 500s with \"MySQL server has gone away\". Often this happens overnight. It\'s started doing this

Answer 1

Using reconnect: true in the database.yml will cause the database connection to be re-established AFTER the ActiveRecord::StatementInvalid error is raised (As Dave Cheney mentioned).

Unfortunately adding a retry on the database operation seemed necessary to guard against the connection timeout:

begin
  do_some_active_record_operation
rescue ActiveRecord::StatementInvalid => e
  Rails.logger.debug("Got statement invalid #{e.message} ... trying again")
  # Second attempt, now that db connection is re-established
  do_some_active_record_operation
end

Answer 2

Something else to check is Unicorn config is correct. See before_fork and after_fork handling of ActiveRecord connection here: https://gist.github.com/nebiros/2776085#file-unicorn-rb

Answer 3

I had this problem when sending really large statements to MySQL. MySQL limits the size of statements and will close the connection if you go over the limit.

set global max_allowed_packet = 1048576; # 2^20 bytes (1 MB) was enough in my case

Answer 4

This is probably caused by the persistent connections to MySQL going away (time out is likely if it's happening over night) and Ruby on Rails is failing to restore the connection, which it should be doing by default:

In the file vendor/rails/actionpack/lib/action_controller/dispatcher.rb is the code:

if defined?(ActiveRecord)
  before_dispatch { ActiveRecord::Base.verify_active_connections! }
  to_prepare(:activerecord_instantiate_observers) {ActiveRecord::Base.instantiate_observers }
end

The method verify_active_connections! performs several actions, one of which is to recreate any expired connections.

The most likely cause of this error is that this is because a monkey patch has redefined the dispatcher to not call verify_active_connections!, or verify_active_connections! has been changed, etc.

Answer 5

The connection to the MySQL server is probably timing out.

You should be able to increase the timeout in MySQL, but for a proper fix, have your code check that the database connection is still alive, and re-connect if it's not.