Ruby Eventmachine & HTTP requests via proxy

送分小仙女□ 提交于 2019-12-11 07:15:58

问题


I'm coding on a high scalable web harvester running on top of Eventmachine. All works fine and fast. Recently I'm trying to fire the requests through a bunch of proxies which also works fine, e.g.:

EventMachine.run do
  connect_opts = { :proxy => { :host => '11.12.13.14', :port => 3128 } }
  request_opts = { :proxy => { :authorization => ['jdoe', 'mysecretpass'] } }
  req = EventMachine::HttpRequest.new('http://www.example.com/', connect_opts).get request_opts
  req.callback { }
end

I'm iterating over hundreds of proxies and firing several hundred requests per second. What I need now is to know which proxy was used for which request to store this metadata in a db. How would you fetch this information from the req object?

I hoped there's is some method like req.proxy_foo (equivalent to req.response, req.response_header etc.) to get this information but didn't find any appropriate way yet.

来源:https://stackoverflow.com/questions/10097824/ruby-eventmachine-http-requests-via-proxy

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!