Increasing throughput in a python script

Question

I\'m processing a list of thousands of domain names from a DNSBL through dig, creating a CSV of URLs and IPs. This is a very time-consuming process that can take several hours.

Answer 1

The vast majority of the time here is spent in the external calls to dig, so to improve that speed, you'll need to multithread. This will allow you to run multiple calls to dig at the same time. See for example: Python Subprocess.Popen from a thread . Or, you can use Twisted ( http://twistedmatrix.com/trac/ ).

EDIT: You're correct, much of that was unnecessary.