How to import all submodules?

Question

I have a directory structure as follows:

| main.py
| scripts
|--| __init__.py
   | script1.py
   | script2.py
   | script3.py

From ma

Answer 1

I was writing a small personal library and adding new modules all the time so I wrote a shell script to look for scripts and create the __init__.py's. The script is executed just outside of the main directory for my package, pylux.

I know it probably isn't the answer you're looking for, but it servered its purpose for me and it might be useful to someone else, too.

#!/bin/bash

echo 'Traversing folder hierarchy...'

CWD=`pwd`


for directory in `find pylux -type d -exec echo {} \;`;
do
    cd $directory
    #echo Entering $directory
    echo -n "" > __init__.py

    for subdirectory in `find . -type d -maxdepth 1 -mindepth 1`;
    do
        subdirectory=`echo $subdirectory | cut -b 3-`
        #echo -n '    ' ...$subdirectory
        #echo -e '\t->\t' import $subdirectory
        echo import $subdirectory >> __init__.py
    done

    for pyfile in *.py ;
    do
        if [ $pyfile = $(echo __init__.py) ]; then
            continue
        fi
        #echo -n '    ' ...$pyfile
        #echo -e '\t->\t' import `echo $pyfile | cut -d . -f 1`
        echo import `echo $pyfile | cut -d . -f 1` >> __init__.py
    done
    cd $CWD

done


for directory in `find pylux -type d -exec echo {} \;`;
do
    echo $directory/__init__.py:
    cat $directory/__init__.py | awk '{ print "\t"$0 }'
done

Answer 2

This works nicely for me in Python 3.3. Note that this works only for submodules which are in files in the same directory as the __init__.py. With some work however it can be enhanced for supporting submodules in directories too.

from glob import iglob
from os.path import basename, relpath, sep, splitext

def import_submodules(__path__to_here):
    """Imports all submodules.
    Import this function in __init__.py and put this line to it:
    __all__ = import_submodules(__path__)"""
    result = []
    for smfile in iglob(relpath(__path__to_here[0]) + "/*.py"):
        submodule = splitext(basename(smfile))[0]
        importstr = ".".join(smfile.split(sep)[:-1])
        if not submodule.startswith("_"):
            __import__(importstr + "." + submodule)
            result.append(submodule)
    return result

Answer 3

Edit: Here's one way to recursively import everything at runtime...

(Contents of __init__.py in top package directory)

import pkgutil

__all__ = []
for loader, module_name, is_pkg in  pkgutil.walk_packages(__path__):
    __all__.append(module_name)
    _module = loader.find_module(module_name).load_module(module_name)
    globals()[module_name] = _module

I'm not using __import__(__path__+'.'+module_name) here, as it's difficult to properly recursively import packages using it. If you don't have nested sub-packages, and wanted to avoid using globals()[module_name], though, it's one way to do it.

There's probably a better way, but this is the best I can do, anyway.

Original Answer (For context, ignore othwerwise. I misunderstood the question initially):

What does your scripts/__init__.py look like? It should be something like:

import script1
import script2
import script3
__all__ = ['script1', 'script2', 'script3']

You could even do without defining __all__, but things (pydoc, if nothing else) will work more cleanly if you define it, even if it's just a list of what you imported.

Answer 4

Simply works, and allows relative import inside packages:

def import_submodules(package_name):
    """ Import all submodules of a module, recursively

    :param package_name: Package name
    :type package_name: str
    :rtype: dict[types.ModuleType]
    """
    package = sys.modules[package_name]
    return {
        name: importlib.import_module(package_name + '.' + name)
        for loader, name, is_pkg in pkgutil.walk_packages(package.__path__)
    }

Usage:

__all__ = import_submodules(__name__).keys()

Answer 5

This is based on the answer that kolypto provided, but his answer does not perform recursive import of packages, whereas this does. Although not required by the main question, I believe recursive import applies and can be very useful in many similar situations. I, for one, found this question when searching on the topic.

This is a nice, clean way of performing the import of the subpackage's modules, and should be portable as well, and it uses the standard lib for python 2.7+ / 3.x.

import importlib
import pkgutil


def import_submodules(package, recursive=True):
    """ Import all submodules of a module, recursively, including subpackages

    :param package: package (name or actual module)
    :type package: str | module
    :rtype: dict[str, types.ModuleType]
    """
    if isinstance(package, str):
        package = importlib.import_module(package)
    results = {}
    for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):
        full_name = package.__name__ + '.' + name
        results[full_name] = importlib.import_module(full_name)
        if recursive and is_pkg:
            results.update(import_submodules(full_name))
    return results

Usage:

# from main.py, as per the OP's project structure
import scripts
import_submodules(scripts)

# Alternatively, from scripts.__init__.py
import_submodules(__name__)

Answer 6

Not nearly as clean as I would like, but none of the cleaner methods worked for me. This achieves the specified behaviour:

Directory structure:

| pkg
|--| __init__.py
   | main.py
   | scripts
   |--| __init__.py
      | script1.py
      | script2.py
      | script3.py

Where pkg/scripts/__init__.py is empty, and pkg/__init__.py contains:

import importlib as _importlib
import pkgutil as _pkgutil
__all__ = [_mod[1].split(".")[-1] for _mod in
           filter(lambda _mod: _mod[1].count(".") == 1 and not 
                               _mod[2] and __name__ in _mod[1],
                  [_mod for _mod in _pkgutil.walk_packages("." + __name__)])]
__sub_mods__ = [".".join(_mod[1].split(".")[1:]) for _mod in
                filter(lambda _mod: _mod[1].count(".") > 1 and not 
                                    _mod[2] and __name__ in _mod[1],
                       [_mod for _mod in 
                        _pkgutil.walk_packages("." + __name__)])]
from . import *
for _module in __sub_mods__:
    _importlib.import_module("." + _module, package=__name__)

Although it's messy, it should be portable. I've used this code for several different packages.