问题
I'm learning & doing SSE for the first time in rails! My controller code:
def update
response.headers['Content-Type'] = 'text/event-stream'
sse = SSE.new(response.stream, event: 'notice')
begin
User.listen_to_creation do |user_id|
sse.write({id: user_id})
end
rescue ClientDisconnected
ensure
sse.close
end
end
Front end:
var source = new EventSource('/site_update');
source.addEventListener('notice', function(event) {
var data = JSON.parse(event.data)
console.log(data)
});
Model pub/sub
class User
after_commit :notify_creation, on: :create
def notify_creation
ActiveRecord::Base.connection_pool.with_connection do |connection|
self.class.execute_query(connection, ["NOTIFY user_created, '?'", id])
end
end
def self.listen_to_creation
ActiveRecord::Base.connection_pool.with_connection do |connection|
begin
execute_query(connection, ["LISTEN user_created"])
connection.raw_connection.wait_for_notify do |event, pid, id|
yield id
end
ensure
execute_query(connection, ["UNLISTEN user_created"])
end
end
end
def self.clean_sql(query)
sanitize_sql(query)
end
private
def self.execute_query(connection, query)
sql = self.clean_sql(query)
connection.execute(sql)
end
end
I've noticed that if I'm writing to SSE, something trivial like in a tutorial like... sse.write({time_now: Time.now})
, everything works great. In command line, CTRL+C
successfully shuts down the local server.
However, whenever I need to write something that requires some kind of database action, for example when I'm doing a postgres pub/sub as in this tutorial, then CTRL+C
doesn't shut down the local server, it's just stuck and hangs and requires me to manually kill the PID
.
On the actual spun up server, sometimes a page refresh will hang forever as well. Other times, it will throw a timeout error:
ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use):
Unfortunately this issue persists in production as well, where i'm using Heroku. I just get lots of timeout errors. But I think I have Heroku properly configured, and also local settings... my understanding is I just need to have a sizable pool (I have 5
) to pull connections from and allow multiple threads. Below you'll find some config code.
THERE ARE NO ENV VARIABLES, DEFAULTS USED!
# config/database.yml
default: &default
adapter: postgresql
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
timeout: 5000
development:
<<: *default
database: proper_development
# config/puma.rb
workers Integer(ENV['WEB_CONCURRENCY'] || 1)
threads_count = Integer(ENV['MAX_THREADS'] || 5)
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
ActiveRecord::Base.establish_connection
end
If it's helpful here's the output when I run rails s
=> Booting Puma
=> Rails 5.0.2 application starting in development on http://localhost:3000
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 4.3.3 (ruby 2.4.0-p0), codename: Mysterious Traveller
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://127.0.0.1:3000
* Listening on tcp://[::1]:3000
Use Ctrl-C to stop
回答1:
The issue here seems to be the lack of consistency between the puma threads and the database connections. If some connection was initiated by middleware etc through AR, the code you have written can lead to two connections being held in the same request cycle until you receive a notification and the thread finishes its job. AR caches connections per thread, so if a request was made and connection was checked out from the pool it will be held by that. Look at this issue for more details. If you end up using the connection pool to check out one more connection and make that connection wait till you get a notification from Postgres, potentially two connections can be held by the same Puma thread that is waiting.
To see this in action, start a new Rails server instance in development and make a request to your SSE endpoint. If you were getting timeouts before you might see two connections to Postgres while you made just one request to a newly launched server. So, even though your number of threads and connection pool size was same, you might run out of free connections from the pool. An easier way might be to just add this line in development after you checkout a connection to see how many cached connections are being held right now.
def self.listen_to_creation
ActiveRecord::Base.connection_pool.with_connection do |connection|
# Print or log this array
p ActiveRecord::Base.connection_pool.instance_variable_get(:@thread_cached_conns).keys.map(&:object_id)
begin
execute_query(connection, ["LISTEN user_created"])
.........
.........
Also, the snippets you have posted seem to indicate you are running up to 16 threads on a connection pool of size 5 in development environment, so that is a different issue.
To fix this, you would want to investigate which thread is holding the connection and if you can reuse it for your notification or just increase the DB pool size.
Now, coming to SSE itself here. Since a SSE connection blocks and reserves a thread in your current setup. If you have multiple requests to this endpoint you might quickly starve out of Puma threads itself making requests wait. This might work in case you are not expecting a lot of requests to this endpoint but if you are, you would need more free threads so you might even want to increase the Puma thread count. Ideally, though a non blocking server would work better here.
Edit: Also, forgot to add that SSE in rails has an issue of keeping alive dead connections if it doesn't know the connection is dead. You might have threads endlessly waiting this way until some data comes and they realize the connection is no longer valid.
来源:https://stackoverflow.com/questions/60637013/rails-app-hangs-when-theres-a-server-sent-event-sse-that-requires-database-ac