How to find all child modules in Python?

Question

While it is fairly trivial in Python to import a \"child\" module into another module and list its attributes, it becomes slightly more difficult when you want to import all

Answer 1

using dir() and imp module

Answer 2

When I was a kind and just beginning programming in Python I've written this for my modular IRC bot:

    # Load plugins

    _plugins = []

    def ifName(name):
        try:
            return re.match('([^_.].+)\.[^.]+', a).group(1)
        except:
            return None

    def isValidPlugin(obj):
        from common.base import PluginBase
        try:
            if obj.__base__ == PluginBase:
                return True
            else:
                return False
        except:
            return False

    plugin_names = set(ifilter(lambda a: a!=None, [ifName(a) for a in os.listdir(os.path.join(os.getcwd(), 'plugins'))]))
    for plugin_name in plugin_names:
        try:
            plugin = __import__('plugins.'+plugin_name, fromlist=['plugins'])
            valid_plugins = filter(lambda a: isValidPlugin(a), [plugin.__getattribute__(a) for a in dir(plugin)])
            _plugins.extend(valid_plugins)
        except Exception, e:
            logger.exception('Error loading plugin %s', plugin_name)

    # Run plugins

    _plugins = [klass() for klass in _plugins]

It's not secure or "right" way, but maybe it we'll be useful nevertheless. It's very old code so please don't beat me.

Answer 3

The solution above traversing the filesystem for finding submodules is ok as long as you implement every plugin as a filesystem based module.

A more flexible way would be an explicit plugin list in your main module, and have every plugin (whether a module created by file, dynamically, or even instance of a class) adding itself to that list explicitly. Maybe via a registerPlugin function.

Remember: "explicit is better than implicit" is part of the zen of python.

Answer 4

pkgutil.walk_packages() seems to be the right way to do this. Here's how I found a list of the available modules, and then imported one of them:

$ mkdir foo
$ touch foo/__init__.py
$ touch foo/bar.py
$ python
Python 3.8.2 (default, Jul 16 2020, 14:00:26) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import foo
>>> from pkgutil import walk_packages
>>> list(walk_packages(foo.__path__))
[ModuleInfo(module_finder=FileFinder('/home/don/git/zero-play/zero_play/foo'), name='bar', ispkg=False)]
>>> from importlib import import_module
>>> bar = import_module('foo.bar')

Answer 5

You can try globbing the directory:

import os
import glob

modules = glob.glob(os.path.join('/some/path/to/modules', '*.py'))

Then you can try importing them:

checked_modules
for module in modules:
    try:
        __import__(module, globals(), locals()) # returns module object
    except ImportError:
        pass
    else:
        checked_modules.append(module)

Answer 6

I think the best way to do this sort of plugin thing is using entry_points and the API for querying them.