How to properly call Akka HTTP client for multiple (10k - 100k) requests?

Question

I\'m trying to write a tool for batch data upload using Akka HTTP 2.0-M2. But I\'m facing akka.stream.OverflowStrategy$Fail$BufferOverflowException: Exceeded configure

Answer 1

Akka absolutely enables backpressure, you're just not taking advantage of it. Instead of dispatching multiple single requests, you can use a single Flow to send all of your requests through. From the documentation:

final Flow<HttpRequest, HttpResponse, Future<OutgoingConnection>> connectionFlow = 
  Http.get(actorSystem).outgoingConnection("127.0.0.1", 8082);

You can then use this Flow to process your HttpRequest objects:

HttpRequest httpRequest = HttpRequest.GET("/test")

//imitates your for-loop example of 100 requests
Source.from(() -> Collections.nCopies(100, httpRequest).iterator()) 
      .via(connectionFlow)
      .runForeach(...)