Managing multiple Twisted client connections

Question

I\'m trying to use Twisted in a sort of spidering program that manages multiple client connections. I\'d like to maintain of a pool of about 5 clients working at one time. The

Answer 1

Since each of your clients needs to update a database, instinctively I think I'd piggyback off the connection pool -- see here for more (the whole doc is recommended for some important design patterns that often emerge when using twisted).

Answer 2

I don't know if you are forced to use Twisted, otherwise you might want to give Gevent a try.

Answer 3

The best option is really just to do the obvious thing here. Don't have a loop, or a repeating timed call; just have handlers that do the right thing.

Keep a central connection-management object around, and make event-handling methods feed it the information it needs to keep going. When it starts, make 5 outgoing connections. Keep track of how many are in progress, maintain a list with them in it. When a connection succeeds (in connectionMade) update the list to remember the connection's new state. When a connection completes (in connectionLost) tell the connection manager; its response should be to remove that connection and make a new connection somewhere else. In the middle, it should be fairly obvious how to fire off a request for the names you need and stuff them into a database (waiting for the database insert to complete before dropping your IRC connection, most likely, by waiting for the Deferred to come back from adbapi).