ECONNRESET when opening a large number of connection in small time period

Question

I have situation where I want to create large number of entities on orion. I am using docker version of Orion and mongo with this docker-compose.

version: "3"
services:
  mongo:
    image: mongo:3.4
    volumes:
      - /data/docker-mongo/db:/data/db
      - /data/docker-mongo/log/mongodb.log:/var/log/mongodb/mongod.log
    command: --nojournal
  orion:
    image: fiware/orion
    volumes:
      - /data/docker-mongo/log/contextBroker.log:/tmp/contextBroker.log
    links:
      - mongo
    ports:
      - "1026:1026"
    command: -dbhost mongo

Now problems happens when I want to upload 2000 entities (opening new connection for each, I know it can be done different but for now this is request), I successfully create no more than 600 (or less never exact number) of them rest fail to create with error:

"error": {
            "errno": "ECONNRESET",
            "code": "ECONNRESET",
            "syscall": "read"
        },

So I assume this issue has something to do with maxConnections, reqPoolSize etc settings in Orion. But in docker I failed to locate Orion config file, I have no way of knowing when I type commands like contextBroker -maxConnections 123456 that setting is being accepted by Orion and docker container.

Also log of Orion is empty, and i cannot determined what is causing this issue when Orion is running on docker.

So main questions:

• Can Orion running on docker be used in same manner as Orion running on VM (are there some fallbacks)
• And how do I check this problem when Orion is running in docker, because I read a lot of docs/issues but no luck (or I missed something).

If you have some advice/soultion it would really help. Thanks

{
"orion" : {
"version" : "1.13.0-next",
"uptime" : "2 d, 15 h, 46 m, 34 s",
"git_hash" : "ae72acf9e8eeaacaf4eb138f7de37bfee4514c6b",
"compile_time" : "Fri May 4 10:12:18 UTC 2018",
"compiled_by" : "root",
"compiled_in" : "1901fd6bb51a",
"release_date" : "Fri May 4 10:12:18 UTC 2018",
"doc" : "https://fiware-orion.readthedocs.org/en/master/"
}
}



{ Error: socket hang up
at createHangUpError (_http_client.js:313:15)
at Socket.socketOnEnd (_http_client.js:416:23)
at Socket.emit (events.js:187:15)
at endReadableNT (_stream_readable.js:1090:12)
at process._tickCallback (internal/process/next_tick.js:63:19) code: 'ECONNRESET' }


error:
{ Error: connect ECONNREFUSED ipofvirtualm:1026
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)
 errno: 'ECONNREFUSED',
 code: 'ECONNREFUSED',
 syscall: 'read',
 address: 'ipofvm',
 port: 1026 },

options:
{ method: 'POST',
 uri: 'http://ip:1026/v2/entities?options=keyValues',
 headers:
  { 'Fiware-Service': 'some service',
    'Fiware-ServicePath': 'some servicepath' },
 body:
  { id: 'F0B935',
    type: 'Transaction',
    refEmitter: 'F0B935',
    refReceiver: '7501JXG',
    refCapturer: 'testtdata',
    date: '12/12/2017 13:25',
    refTransferredResources: 'testtdata',
    transferredLoad: 92 },
 json: true,
 callback: [Function: RP$callback],
 transform: undefined,
 simple: true,
 resolveWithFullResponse: false,
 transform2xxOnly: false },

I am using request promise library for making calls, i try others they had same issue. Now since i cannot send u all 2000 responses i will try to describe. So it when i start to send this it behave. It create like 30 entities then next few or more return response saying ECONNRESET then it start creating again and so on.

What confuse me is that it is not failing totally meaning it works but not as intended. Also it seem that Orion close socket or hang it up some period then he is open again and create as normal and so on. If u need any more info ask, and thanks for quick answer.

Answer 1

instead of opening a new connection per entity why don't you use

POST /v2/op/update 

and create all entities in just one batch? or a couple of batches

See some code at

https://github.com/Fiware/dataModels/blob/master/Weather/WeatherObserved/harvest/spain_weather_observed_harvest.py#L235

Answer 2

With regards to CLI argument passing to CB running inside docker, use the command line in docker compose file, eg:

command: -dbhost mongo -maxConnections 123456

However, I'm not sure that would help to solve the problem, as Orion should deal with your use case without any special customization. Looking to the error message (which seems to be about some problema at TCP layer) I wonder if docker networking layer is acting as bottleneck in some way...

In addition, the suggestion done by Jose Manuel Cantera about using POST /v2/op/update would be a good idea. It would reduce connection stress at network layer and may help to alleviate the problem.

If you cannot change your update strategy, maybe using an inter-request delay (100-200ms) could also help.