Rails app hangs when there's a server sent event (SSE) that requires database actions

问题

I'm learning & doing SSE for the first time in rails! My controller code:

  def update
    response.headers['Content-Type'] = 'text/event-stream'
    sse = SSE.new(response.stream, event: 'notice')
    begin
      User.listen_to_creation do |user_id|
        sse.write({id: user_id})
      end
    rescue ClientDisconnected
    ensure
      sse.close
    end
  end

Front end:

  var source = new EventSource('/site_update');
  source.addEventListener('notice', function(event) {
    var data = JSON.parse(event.data)
    console.log(data)
  });

Model pub/sub

class User
  after_commit :notify_creation, on: :create

  def notify_creation
      ActiveRecord::Base.connection_pool.with_connection do |connection|
        self.class.execute_query(connection, ["NOTIFY user_created, '?'", id])
      end
  end

  def self.listen_to_creation
    ActiveRecord::Base.connection_pool.with_connection do |connection|
      begin
        execute_query(connection, ["LISTEN user_created"])
        connection.raw_connection.wait_for_notify do |event, pid, id|
          yield id
        end
      ensure
        execute_query(connection, ["UNLISTEN user_created"])
      end
    end
  end

  def self.clean_sql(query)
    sanitize_sql(query)
  end

  private

  def self.execute_query(connection, query)
    sql = self.clean_sql(query)
    connection.execute(sql)
  end
end

I've noticed that if I'm writing to SSE, something trivial like in a tutorial like... sse.write({time_now: Time.now}), everything works great. In command line, CTRL+C successfully shuts down the local server.

However, whenever I need to write something that requires some kind of database action, for example when I'm doing a postgres pub/sub as in this tutorial, then CTRL+C doesn't shut down the local server, it's just stuck and hangs and requires me to manually kill the PID.

On the actual spun up server, sometimes a page refresh will hang forever as well. Other times, it will throw a timeout error:

ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use):

Unfortunately this issue persists in production as well, where i'm using Heroku. I just get lots of timeout errors. But I think I have Heroku properly configured, and also local settings... my understanding is I just need to have a sizable pool (I have 5) to pull connections from and allow multiple threads. Below you'll find some config code.

THERE ARE NO ENV VARIABLES, DEFAULTS USED!

# config/database.yml
default: &default
  adapter: postgresql
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  timeout: 5000

development:
  <<: *default
  database: proper_development


# config/puma.rb
workers Integer(ENV['WEB_CONCURRENCY'] || 1)

threads_count = Integer(ENV['MAX_THREADS'] || 5)
threads threads_count, threads_count

preload_app!

rackup      DefaultRackup
port        ENV['PORT']     || 3000
environment ENV['RACK_ENV'] || 'development'

on_worker_boot do
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
end

If it's helpful here's the output when I run rails s

=> Booting Puma
=> Rails 5.0.2 application starting in development on http://localhost:3000
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 4.3.3 (ruby 2.4.0-p0), codename: Mysterious Traveller
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://127.0.0.1:3000
* Listening on tcp://[::1]:3000
Use Ctrl-C to stop

回答1:

The issue here seems to be the lack of consistency between the puma threads and the database connections. If some connection was initiated by middleware etc through AR, the code you have written can lead to two connections being held in the same request cycle until you receive a notification and the thread finishes its job. AR caches connections per thread, so if a request was made and connection was checked out from the pool it will be held by that. Look at this issue for more details. If you end up using the connection pool to check out one more connection and make that connection wait till you get a notification from Postgres, potentially two connections can be held by the same Puma thread that is waiting.

To see this in action, start a new Rails server instance in development and make a request to your SSE endpoint. If you were getting timeouts before you might see two connections to Postgres while you made just one request to a newly launched server. So, even though your number of threads and connection pool size was same, you might run out of free connections from the pool. An easier way might be to just add this line in development after you checkout a connection to see how many cached connections are being held right now.

def self.listen_to_creation
    ActiveRecord::Base.connection_pool.with_connection do |connection|
      # Print or log this array
      p ActiveRecord::Base.connection_pool.instance_variable_get(:@thread_cached_conns).keys.map(&:object_id)

      begin
        execute_query(connection, ["LISTEN user_created"])
.........
.........

Also, the snippets you have posted seem to indicate you are running up to 16 threads on a connection pool of size 5 in development environment, so that is a different issue.

To fix this, you would want to investigate which thread is holding the connection and if you can reuse it for your notification or just increase the DB pool size.

Now, coming to SSE itself here. Since a SSE connection blocks and reserves a thread in your current setup. If you have multiple requests to this endpoint you might quickly starve out of Puma threads itself making requests wait. This might work in case you are not expecting a lot of requests to this endpoint but if you are, you would need more free threads so you might even want to increase the Puma thread count. Ideally, though a non blocking server would work better here.

Edit: Also, forgot to add that SSE in rails has an issue of keeping alive dead connections if it doesn't know the connection is dead. You might have threads endlessly waiting this way until some data comes and they realize the connection is no longer valid.

来源：https://stackoverflow.com/questions/60637013/rails-app-hangs-when-theres-a-server-sent-event-sse-that-requires-database-ac