How to import all submodules?

Question

I have a directory structure as follows:

| main.py
| scripts
|--| __init__.py
   | script1.py
   | script2.py
   | script3.py

From ma

Answer 1

To just load all submodules of a package, you can use this simple function:

def import_submodules(module):
    """Import all submodules of a module, recursively."""
    for loader, module_name, is_pkg in pkgutil.walk_packages(
            module.__path__, module.__name__ + '.'):
        importlib.import_module(module_name)

Use case: load all database models of a Flask app, so that Flask-Migrate could detect changes to the schema. Usage:

import myproject.models
import_submodules(myproject.models)

Answer 2

I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/. Usage is like this:

1. Include the automodinit package into your setup.py dependencies.
2. Add the following to the beginning of the __init__.py file:

__all__ = ["I will get rewritten"]
# Don't modify the line above, or this line!
import automodinit
automodinit.automodinit(__name__, __file__, globals())
del automodinit
# Anything else you want can go after here, it won't get modified.

That's it! From now on importing a module will set __all__ to a list of .py[co] files in the module and will also import each of those files as though you had typed:

for x in __all__: import x

Therefore the effect of from M import * matches exactly import M.

automodinit is happy running from inside ZIP archives and is therefore ZIP safe.

Answer 3

I've played around with Joe Kington's Answer and have built a solution that uses globals and get/setattr and thus doesn't need eval. A slight modification is that instead of directly using the packages __path__ for walk_packages, I use the packages parent directory and then only import modules starting with __name__ + ".". This was done to reliably get all subpackages from walk_packages - in my use case I had a subpackage named test which caused pkgutil to iterate over the test package from python's library; furthermore, using __path__ would not recurse into the packages subdirectories. All these issues were observed using jython and python2.5, the code below is only tested in jython thus far.

Also note that OPs question only talks about importing all modules from a package, this code recursively imports all packages too.

from pkgutil import walk_packages
from os import path

__all__ = []
__pkg_prefix = "%s." % __name__
__pkg_path = path.abspath(__path__[0]).rsplit("/", 1)[0] #parent directory

for loader, modname, _ in walk_packages([__pkg_path]):
    if modname.startswith(__pkg_prefix):
        #load the module / package
        module = loader.find_module(modname).load_module(modname)
        modname = modname[len(__pkg_prefix):] #strip package prefix from name
        #append all toplevel modules and packages to __all__
        if not "." in modname:
            __all__.append(modname)
            globals()[modname] = module
        #set everything else as an attribute of their parent package
        else:
            #get the toplevel package from globals()
            pkg_name, rest = modname.split(".", 1)
            pkg = globals()[pkg_name]
            #recursively get the modules parent package via getattr
            while "." in rest:
                subpkg, rest = rest.split(".", 1)
                pkg = getattr(pkg, subpkg)
            #set the module (or package) as an attribute of its parent package
            setattr(pkg, rest, module)

As a future improvement I'll try to make this dynamic with a __getattr__ hook on the package, so the actual modules are only imported when they are accessed...