What is the most efficient way to scrape multiple single and branch OIDs?

问题

I am always in a war between maximum speed and minimal resource consumption, so my goal is to find the best combination for speed and resource consumption.
For each snmp device I would like to scan the oids under several branches.
Each branch has a dynamic amount of oids under it so i don't know what specific oids i need, I just know that I need all oids under a branch.
I have some devices that only support SNMPv1 so for those devices I write code that is compatible with SNMPv1.
For the rest of the devices I use SNMPv2.

SNMPv1

Let's say I have two OIDs that I would like to walk their branch('1.3.6.1.2.1.4' and '1.3.6.1.2.1.6').
By branch OID I mean all the OIDs under a branch.
I have the following code:

cmdGen = cmdgen.AsynCommandGenerator()
cmdGen.asyncNextCmd(
  cmdgen.CommunityData('public', mpModel=1),
  cmdgen.UdpTransportTarget(('192.168.0.101', 161)),
  (str('1.3.6.1.2.1.4'),str('1.3.6.1.2.1.6'),),
  (__cbFun_Walk, (cmdgen.CommunityData('public', mpModel=1), cmdgen.UdpTransportTarget(('192.168.0.101', 161)))))
cmdGen.snmpEngine.transportDispatcher.runDispatcher()

This works well but the only problem is that I can only stop all walks at once, therefore I can't stop each walk individually, so all walks will finish once the longest walk has finished.
Obviously this is inefficient.
I could also write 2 asyncNextCmd for the branch OIDs:

cmdGen = cmdgen.AsynCommandGenerator()
cmdGen.asyncNextCmd(
        cmdgen.CommunityData('public', mpModel=1),
        cmdgen.UdpTransportTarget(('192.168.0.101', 161)),
        (str('1.3.6.1.2.1.4'),),
        (__cbFun_Walk, (cmdgen.CommunityData('public', mpModel=1),
        cmdgen.UdpTransportTarget(('192.168.0.101', 161))))) 
cmdGen.asyncNextCmd(
        cmdgen.CommunityData('public', mpModel=1),
        cmdgen.UdpTransportTarget(('192.168.0.101', 161)),
        (str('1.3.6.1.2.1.6'),),
        (__cbFun_Walk, (cmdgen.CommunityData('public', mpModel=1),
        cmdgen.UdpTransportTarget(('192.168.0.101', 161)))))
cmdGen.snmpEngine.transportDispatcher.runDispatcher()

I don't know SNMP too well, but I assume that the second code has some drawbacks.
For example on some clients I have hundreds of SNMP devices, so I opened an asyncCmd for each network device simultaneously.
This resulted in a lot of unresponsive devices and too high CPU usage.

SNMPv2

I would also want to try to understand how the bulk works and if I can use it to make my code more efficient.
Lets say I have 2 branches that I would like to walk.
1.3.6.1.2.1.4.20 that has 5 oids, and 1.3.6.1.2.1.4.21 that has 39 oids.
I get all the values in both branches but i also get more values then i want aswell.
The amount of values that I get is always the branch with the highest amount of oids times the amount of branches that I have.
For example the branch with the most oids has 39 oids, and the amount of branches is 2, so 39*2=78, that means the getBulk will return 78 oids.
I want the getBulk to return all the branch oids for each branch and not anything more, so in my case I want 44 oids(39+5 = 44).
This is my code:

cmdGen = cmdgen.CommandGenerator()

errorIndication, errorStatus, errorIndex, varBindTable = cmdGen.bulkCmd(
  cmdgen.CommunityData('public'),
  cmdgen.UdpTransportTarget(('192.168.0.101', 161)),
  0, 1,
  '1.3.6.1.2.1.4.21', '1.3.6.1.2.1.4.20'
)

if errorIndication:
  print errorIndication
elif errorStatus:
  print '%s at %s\n' % (
    errorStatus.prettyPrint(),
    errorIndex and varBindTable[-1][int(errorIndex)-1] or '?'
  )
else:
  for varBindTableRow in varBindTable:
    for name, val in varBindTableRow:
      print str(name.prettyPrint()) + ' = ' + str(val.prettyPrint())

So what is the most efficient way to scrape multiple branch OIDs for both SNMPv1 and SNMPv2?

回答1:

The first thing to realize is that you can't instruct SNMP agent to return you precisely a "branch". Such operation is not supported in SNMP (at protocol level). The manager is in control in regards to what OIDs to request and when to stop. Agent can return the exact single OID (GET) or next OID (GETNEXT/GETBULK) or approximate number of next OIDs at once (GETBULK).

In case when you have dynamic OIDs, you can't reliably anticipate how many of them are there at any given moment. Hence I don't think you could get strictly the branch OIDs not more or less in a single SNMP operation.

Having said that I think you should try to use GETBULK wherever possible. Aim at the maximum anticipated number of OIDs to grab them in one GETBULK operation. But check in your code that you indeed get everything (agent may not return the requested number of OIDs all at once).

As for pysnmp API you are using, to stop its GETBULK/GETNEXT iteration you are supposed to return True from your callback function. You could analyze what OIDs you are getting from the agent there and return True as they get beyond the branch you are interested in. That would terminate one branch retrieval, the rest will keep running.

The design where you send multiple parallel queries to the same SNMP agent for different OIDs is obviously much less efficient.