Dynamic class loading in Python 2.6: RuntimeWarning: Parent module 'plugins' not found while handling absolute import

Question

I am working on a plugin system where plugin modules are loaded like this:

def load_plugins():
   plugins=glob.glob(\"plugins/*.py\")
   instances=[]
   for         


        

Answer 1

If the plugins directory does not have an __init__.py, it isn't a package, so when you create plugins.whatever, Python warns you that such a thing shouldn't really exist. (It couldn't be created by "import plugins.whatever" no matter what your path is.)

Also,

• Don't split on /, which is unportable. Use os.path.split.
• Don't use .split(".py") to get the name without the extension, which is buggy. Use os.path.splitext.
• Don't use getattr with a string literal. getattr(plugin, "__init__") is spelled plugin.__init__.
• I am confused why you are calling a module-level __init__ function. This doesn't seem right. Perhaps you want a "set_logger" function or better, to instantiate a class that takes a logger.
• Don't use L = L + some_other_list to extend a list, use the extend method, which has better performance and is more idiomatic.
• Don't squash unknown exceptions by except Exception. If you cannot plan to do something sane in response to an exception, your program cannot go on sanely.

Answer 2

If the directory plugins were a real package (contained __init__.py fine), you could easily use pkgutils to enumerate its plugin files and load them.

import pkgutil
# import our package
import plugins
list(pkgutil.iter_modules(plugins.__path__))

However, it can work without a plugin package anyway, try this:

import pkgutil
list(pkgutil.iter_modules(["plugins"]))

Also it is possible to make a package that only exists at runtime:

import types
import sys
plugins = types.ModuleType("plugins")
plugins.__path__ = ["plugins"]

sys.modules["plugins"] = plugins
import plugins.testplugin

However that hack that was mostly for fun!

Answer 3

The Python imp documentation has been updated since this was answered. It now addresses this issue specifically in the find_module() method.

This function does not handle hierarchical module names (names containing dots). In order to find P.M, that is, submodule M of package P, use find_module() and load_module() to find and load package P, and then use find_module() with the path argument set to P.__path__. When P itself has a dotted name, apply this recipe recursively.

Note that P.__path__ is already a list when supplying it to find_module(). Also note what the find_module() documentation says about finding packages.

If the module is a package, file is None, pathname is the package path and the last item in the description tuple is PKG_DIRECTORY.

So from the OP's question, to import the plugin without the RuntimeError warnings, follow the directions in the updated Python documentation:

# first find and load the package assuming it is in
# the current working directory, '.'

f, file, desc = imp.find_module('plugins', ['.'])
pkg = imp.load_module('plugins', f, file, desc)

# then find the named plugin module using pkg.__path__
# and load the module using the dotted name

f, file, desc = imp.find_module(name, pkg.__path__)
plugin = imp.load_module('plugins.' + name, f, file, desc)

Answer 4

The problem here is with the dot ('.') in the module name:

imp.load_module('plugins.'+name, f, file, desc)

Don't include a '.' after 'plugins', or Python will think it's a module path.

Answer 5

You can try adding below statement at the beginning of import statements.

from __future__ import absolute_import