Is a Scripts directory an anti-pattern in Python? If so, what's the right way to import?

Question

I\'ve always created scripts directories in every project I\'ve built because they\'re useful for putting infrequently used executable scripts. In Python, I\'ll always put an

Answer 1

The link in the question only says about running scripts that reside in a package directory which is a potential problem because... well... packages are not scripts and scripts are not packages. They serve different purposes and are invoked in different ways, so if you mix them together, there'll be a mess at some point.

Since Python itself has had a Scripts directory for ages and no one complains, it's in no way an anti-pattern.

See How do I separate my executable files from my library files? on how we dealt with executable scripts at my last occupation. It never caused any problems that I'm aware of.

Answer 2

Personally, having a __init__.py in one's scripts dir feels a bit off, but I can also see why it's useful here (and in IDEs).

That said, if they're already being run as a Python module, then maybe they're not true scripts, whatever that even means (related: do you have a shebang on these files?). Hard to say without context, but perhaps they are closer to tools modules, forming part of your overall Python codebase.

In which case, with the addition of an __init__.py at project level (project_dir in your example), you can then use normal importing in your would-be scripts:

from some_modules_dir import foo

meaning that your original python -m scripts.some_script makes absolute (sorry) sense...

Answer 3

In general, if your Python project contains multiple packages that are native to the project, then there should probably be a single top-level package that all other packages are subpackages of (after all they are part of the same project so there should be some unifying justification for their existence). Since a directory counts as a package only when it has an __init__.py, the other way to say this is that if any two sibling directories in your project both have an __init__.py, then their parent should have one as well. So assuming that you have good reasons to have a "scripts" package and a "modules" package directly within your project root, which you very well might have, your project root directory probably should be a package as well.

If your top-level package is not the project root, it is usually considered fine to have some "loose" Python scripts adjacent to the top-level package. Like this:

project_root/
    top_level_package/
        __init__.py
        module.py
        subpackage/
            __init__.py
            anothermodule.py
    adjacent_script.py
    adjacent_script_2.py

The "loose" scripts can import directly from the packages and modules because they are in the same directory as the top-level package. The assumption with this construction is that the top-level package contains all "interesting" code of your project (your "selling point", or the "meat" of your project if you want), while the "loose" adjacent scripts function only as entry points to specific functionality that you may want to access from the top-level package. For example, you may have an adjacent script for starting the test suite, for installing the software, or to start the application if your project is an application.

Concerning your particular pattern, I'm under the impression that you distinguish "scripts" from "modules" because the scripts are meant as an interface between you and the modules. If it is only for you as a developer and you are using the "scripts" infrequently, as you said, it is probably fine to stick with your recipe as long as the root directory has an __init__.py. If, however, you (or someone else) also use the "scripts" as an end user, or if you use them frequently, it is neater to provide a script adjacent to the top-level package that bundles the functionality from your "scripts" directory in a single command (or maybe a few if they offer very dissimilar functionality) with a subcommand for each "script". You might want to use the argparse standard module for that adjacent script. In either case, it would probably be more appropriate to call the "scripts" tools.

If you have only a few tools and they don't contain any code other than glue between the command line and your modules, you may also consider to just move them to the top directory.

---

Some sources:

• argparse
• on distributing packages (note that the setup.py script used in distutils is also an example of an adjacent script)
• official Python tutorial on modules and packages
• PYTHONPATH

Answer 4

Setuptools can create scripts automatically if you give it a list of entry points (i.e. main() functions). If you're already using Setuptools, it's very easy to turn on. Then you can merge your scripts directory with your other packages.