Python: sharing common code among a family of scripts

Question

I'm writing a family of Python scripts within a project; each script is within a subdirectory of the project, like so:

projectroot
  |
  |- subproject1
  |    |
  |    |- script1.main.py
  |    `- script1.merger.py
  |
  |- subproject2
  |    |
  |    |- script2.main.py
  |    |- script2.matcher.py
  |    `- script2.merger.py
  |
  `- subproject3
       |
       |- script3.main.py
       |- script3.converter.py
       |- script3.matcher.py
       `- script3.merger.py

Now several of the scripts share some code. The shared code is best considered part of the project itself, and not something I would compile separately and make a library out of, or drop in a sitewide PYTHONPATH. I could place that code in various places, such as in the projectroot directory itself, or in a child directory of projectroot called common (perhaps).

However, most of the ways I have thought of so far involve making packages out of my subprojects with empty __init__.py files and using relative imports (or redundantly messing with sys.path in every subproject. Worse, it seems like building a package structure around this family of scripts runs afoul of the following warning from the rejected PEP-3122:

Attention! This PEP has been rejected. Guido views running scripts within a package as an anti-pattern.

If scripts within a package is anti-patternish, how can I set things up in a way which keeps the common code in the same project? Or is a module and package-based system acceptable here? Which is the cleanest approach? (FWIW I would prefer to have a file such as shared.py or common.py in the project root directory, rather than making a utility directory that is a sibling to the "real" subprojects.)

Answer 1

I suggest putting trivial "launcher" scripts at the top level of your project, and making each of the subproject folders into packages. The modules in the packages can import each other or common code can be factored out into a common package.

Here's what the structure would look like, if we assume the various merger modules can be refactored into a shared version:

projectroot
  |- script1.py # launcher scripts, see below for example code
  |- script2.py
  |- script3.py
  |
  |- common
  |    |- __init__.py
  |    |- merger.py # from other packages, use from ..common import merger to get this
  |
  |- subproject1
  |    |- __init__.py # this can be empty
  |    |- script1_main.py
  |
  |- subproject2
  |    |- __init__.py
  |    |- script2_main.py
  |    |- script2_matcher.py
  |
  |- subproject3
       |- __init__.py
       |- script3_main.py
       |- script3_converter.py
       |- script3_matcher.py

The launcher scripts can be very simple:

from subproject1 import script1_main

if __name__ == "__main__":
    script1_main.main()

That is, all it does is import the appropriate "scriptN_main" module and run a function in it. Using a simple script may also have some small benefits for script startup speed, since the main module can have its compiled bytecode cached to a .pyc file, while scripts are never cached.

Note: I renamed your modules, swapping _ characters in for the . characters. You can't have a . in an identifier (such as a module name), since Python expects it to indicate attribute access. That meant those modules could never be imported. (I'm guessing that this is an artifact of the example files only, not something that you have in your real code.)

Answer 2

Please use setuptools to distribute both scripts and libraries:

e.g.

from setuptools import setup

setup(
    # other arguments here... (e.g. packages / package_dir)
    entry_points = {
        'console_scripts': [
            'script1 = subproject1.script1:main',
            'script2 = subproject2.script2:main',
        ],
    }
)

If you can write all you code as libraries, and don't need separate modules to have your entry points then this is the tool for you. If you have scripts, that's fine too, but you will need a main function you can reference (see example above)

Answer 3

My preference would be a separate "bin" or "scripts" directory, with subprojects as libraries / packages:

projectroot
  |
  |- scripts
  |
  |- lib
  |    |
  |    `- matcher.py
  |    `- merger.py
  |    `- subproject1
  |    `- subproject2
  |    `- subproject3

The idea being your scripts can each reference any subprojects necessary as usual packages. And your subprojects can also reference each other with imports.

You can then also have a main or shared script that sets up the subproject packages for you, if that helps.