How get best performance rails requests parallel sidekiq worker

Question

My rails app have one sidekiq worker. The worker will do 2500 requests to a api external, the response is a xml. How get best performance for this worker?

Answer 1

Inside the worker, spawn application level Threads.

For example, Create 10 ruby threads to process the 2500 external api requests(means each ruby thread will process 250 requests).

  # threads will contain the threads
  threads = [] 

  external_requests.each_slice(250) do |group| 
    threads << Thread.new(group) do  |tsrc|
      tsrc.each do |ex_request|
        # Do your external call here 
      end 
    end
  end

  threads.each(&:join) # wait for all threads to finish