Python accessing modules from package that is distributed over different directories

Question

I have a question regarding one single module that is distributed over multiple directories.

Let\'s say I have these two file and directories:

~/lib/         


        

Answer 1

This is addressed by Implicit Namespace Packages in Python 3.3. See PEP-420.

Answer 2

You can't without an extreme amount of trickery that pulls one package structure into the other. Python requires that all modules in a package be under a single subdirectory. See the os source to learn how it handles os.path.

Answer 3

Python does indeed remember that there was a xxx package. This is pretty much necessary to achieve acceptable performance, once modules and packages are loaded they are cached. You can see which modules are loaded by looking the the dictionary sys.modules.

sys.modules is a normal dictionary so you can remove a package from it to force it to be reloaded like below:

import sys
print sys.modules
import xml
print sys.modules
del sys.modules['xml']
print sys.modules

Notice that after importing the xml package it is the dictionary, however it is possible to remove it from that dictionary too. This is a point I make for pedagogical purposes only, I would not recommend this approach in a real application. Also if you need to use your misc and util packages together this would not work so great. If at all possible rearrange your source code structure to better fit the normal Python module loading mechanism.

Answer 4

This is an adaptation of an answer to a similar question.

Following up on @Gary's answer, the PEP 420 page says to use the following code on shared __init__.py packages.

__init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

This code should be placed inside the xxx directory's __init__.py. See the *s below

someroot/
├── graphics
│   └── python
│       └── xxx
│           ├── ****__init__.py****
│           └── misc
│               ├── __init__.py
│               ├── module3.py
│               └── module4.py
└── lib
    └── python
        └── xxx
            ├── ****__init__.py****
            └── util
                ├── __init__.py
                ├── module1.py
                └── module2.py

Some setup.sh file to add to the Python Path:

libPath=someroot/lib/python/
graphicsPath=someroot/graphics/python/
export PYTHONPATH=$PYTHONPATH:$libPath:$graphicsPath

Python test code (tested on Python versions 2.7.14 and 3.6.4 using pyenv):

import xxx.util.module1
import xxx.misc.module3 # No errors