Python — Speed Up Imports?

Question

I have 10000s custom (compiled to \'.so\') modules that I\'d like to use in python. The usage of the modules will be consequential (modules are used one after the o

Answer 1

I would first question whether import is really the technique you want to be using to access thousands of code fragments - the full import process is quite expensive, and loading (non-shared) dynamic modules at all isn't particularly cheap, either.

Second, the code as you have written it clearly isn't what you're actually doing. The import statement doesn't accept strings at runtime, you would have to be using importlib.import_module() or calling __import__() directly.

Finally, the first step in optimising this would be to ensure that the first directory on sys.path is the one that contains all these files. You may also want to run Python with the -vv flag to dial up the verbosity on the import attempts. Be warned that this is going to get very noisy if you're doing that many imports.