Uhandled error in pysnmp twisted client with more than 1000 hosts

问题

I have this code:

from twisted.internet import reactor
from twisted.internet import defer, task
from pysnmp.entity import engine, config
from pysnmp.carrier.twisted import dispatch
from pysnmp.carrier.twisted.dgram import udp
from pysnmp.entity.rfc3413.twisted import cmdgen


import __webimport__
import tools.config
from tools.database import makedsn
import psycopg2


def cmp_varBinds(varBind, varName):
    if varName[0] in str(varBind[0]):
        return True


def cbFun(cbCtx, ip, varNames):
    (errorIndication, errorStatus, errorIndex, varBinds) = cbCtx
    if varBinds and any(map(cmp_varBinds, varBinds[0], varNames)):
        print ip, [str(x[1]) for x in varBinds[0]]
        df = defer.Deferred()
        df.addCallback(cbFun, ip=ip, varNames=varNames)
        return df  # This also indicates that we wish to continue walkin


def parallel(iterable, count, callable, *args, **named):
    coop = task.Cooperator()
    work = (callable(elem, *args, **named) for elem in iterable)
    return defer.DeferredList([coop.coiterate(work) for i in xrange(count)])





def fetch(host):
    id, ip, community, hc = host
    snmpEngine = engine.SnmpEngine()
    snmpEngine.registerTransportDispatcher(dispatch.TwistedDispatcher())
    config.addV1System(snmpEngine, 'test-agent', community)
    config.addTargetParams(snmpEngine, 'myParams', 'test-agent', 'noAuthNoPriv', 1)

    config.addTargetAddr(
                     snmpEngine, 'myRouter', config.snmpUDPDomain,
                     (ip, 161), 'myParams', timeout=1
    )

    # Transport
    config.addSocketTransport(
                          snmpEngine,
                          udp.domainName,
                          udp.UdpTwistedTransport().openClientMode()
    )

    getCmdGen = cmdgen.NextCommandGenerator()
    varNames = [('1.3.6.1.2.1.2.2.1.11', None),
            ('1.3.6.1.2.1.2.2.1.12', None),
            ('1.3.6.1.2.1.2.2.1.13', None),
            ('1.3.6.1.2.1.2.2.1.14', None)]
    df = getCmdGen.sendReq(snmpEngine, 'myRouter', varNames)
    df.addCallback(cbFun, ip=ip, varNames=varNames)
    return df


dsn = makedsn(tools.config.main_db)
connection = psycopg2.connect(dsn)
cursor = connection.cursor()
cursor.execute("""SELECT e.id, e.ip, e.snmpcomm, e.hccnt
           FROM snmp_ports sp, equipment e
           WHERE e.snmp = 'Y' and sp.equipment = e.id
           GROUP BY e.id,e.ip,e.snmpcomm,e.hccnt
           ORDER BY e.id""")
hosts = cursor.fetchall()


finished = parallel(hosts, len(hosts), fetch)
finished.addErrback(log.err)
finished.addCallback(lambda ign: reactor.stop())
reactor.run()

I get 4000 hosts from database and ask each of it. if i set limit 1000 in sql query it works fine. But when hosts are more than 1000 i get an error:

Unhandled Error
Traceback (most recent call last):
  File "crawler.py", line 98, in <module>

  File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1192, in run

  File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1201, in mainLoop

--- <exception caught here> ---
  File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/internet/base.py", line 824, in runUntilCurrent

  File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/pysnmp/carrier/base.py", line 52, in _cbFun

  File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/pysnmp/entity/engine.py", line 64, in __receiveMessageCbFun

  File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/pysnmp/proto/rfc3412.py", line 274, in receiveMessage

  File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/pysnmp/smi/builder.py", line 299, in importSymbols

  File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/pysnmp/smi/builder.py", line 270, in loadModules

pysnmp.smi.error.SmiError: MIB file "__SNMPv2-MIB.py[co]" not found in search path

And then script stops. Why is this happenning?

回答1:

What pysnmp you are using? Make sure you are using the latest pysnmp release.

As a side note - your approach is super-inefficient as you seem to re-initialize [heavy] SnmpEngine on each GET operation. Better way would be to keep a single persistent SnmpEngine instance per process/thread.

回答2:

If this only happens when you increase the concurrency level then a likely explanation is that you've run into the limit in the number of open files the platform will let you have at any one time. Each open socket counts towards this limit, as do open "regular" files (from the filesystem).

If you use up all allowed files then Python can't read source code for modules off disk because that platform won't let it open them.

It's not totally obvious that this is what's happening in this case since (if it is) pysnmp is handling the real exception and re-raising a new exception that hides the details.

If this is the problem then you can address it by raising the open files limit. Most of the time you can do this by running:

$ ulimit -Sn 2048

Read more about ulimit (help ulimit in your shell, on the internet) for more details about controlling that limit.

来源：https://stackoverflow.com/questions/19442141/uhandled-error-in-pysnmp-twisted-client-with-more-than-1000-hosts