Python CLI program unit testing

Question

I am working on a python Command-Line-Interface program, and I find it boring when doing testings, for example, here is the help information of the program:

Answer 1

I would not test the program as a whole this is not a good test strategy and may not actually catch the actual spot of the error. The CLI interface is just front end to an API. You test the API via your unit tests and then when you make a change to a specific part you have a test case to exercise that change.

So, restructure your application so that you test the API and not the application it self. But, you can have a functional test that actually does run the full application and checks that the output is correct.

In short, yes testing the code is the same as testing any other code, but you must test the individual parts rather than their combination as a whole to ensure that your changes do not break everything.

Answer 2

You can use standard unittest module:

# python -m unittest <test module>

or use nose as a testing framework. Just write classic unittest files in separate directory and run:

# nosetests <test modules directory>

Writing unittests is easy. Just follow online manual for unittesting

Answer 3

Maybe too little too late, but you can always use

import os.system
result =  os.system(<'Insert your command with options here'>
assert(0 == result)

In that way, you can run your program as if it was from command line, and evaluate the exit code.

(Update after I studied pytest) You can also use capsys. (from running pytest --fixtures)

capsys Enable text capturing of writes to sys.stdout and sys.stderr.

The captured output is made available via ``capsys.readouterr()`` method
calls, which return a ``(out, err)`` namedtuple.
``out`` and ``err`` will be ``text`` objects.

Answer 4

Start from the user interface with functional tests and work down towards unit tests. It can feel difficult, especially when you use the argparse module or the click package, which take control of the application entry point.

The cli-test-helpers Python package has examples and helper functions (context managers) for a holistic approach on writing tests for your CLI. It's a simple idea, and one that works perfectly with TDD:

1. Start with functional tests (to ensure your user interface definition) and
2. Work towards unit tests (to ensure your implementation contracts)

Functional tests

NOTE: I assume you develop code that is deployed with a setup.py file or is run as a module (-m).

• Is the entrypoint script installed? (tests the configuration in your setup.py)
• Can this package be run as a Python module? (i.e. without having to be installed)
• Is command XYZ available? etc. Cover your entire CLI usage here!

Those tests are simplistic: They run the shell command you would enter in the terminal, e.g.

def test_entrypoint():
    exit_status = os.system('foobar --help')
    assert exit_status == 0

Note the trick to use a non-destructive operation (e.g. --help or --version) as we can't mock anything with this approach.

Towards unit tests

To test single aspects inside the application you will need to mimic things like command line arguments and maybe environment variables. You will also need to catch the exiting of your script to avoid the tests to fail for SystemExit exceptions.

Example with ArgvContext to mimic command line arguments:

@patch('foobar.command.baz')
def test_cli_command(mock_command):
    """Is the correct code called when invoked via the CLI?"""
    with ArgvContext('foobar', 'baz'), pytest.raises(SystemExit):
        foobar.cli.main()

    assert mock_command.called

Note that we mock the function that we want our CLI framework (click in this example) to call, and that we catch SystemExit that the framework naturally raises. The context managers are provided by cli-test-helpers and pytest.

Unit tests

The rest is business as usual. With the above two strategies we've overcome the control a CLI framework may have taken away from us. The rest is usual unit testing. TDD-style hopefully.

Disclosure: I am the author of the cli-test-helpers Python package.

Answer 5

This isn't for Python specifically, but what I do to test command-line scripts is to run them with various predetermined inputs and options and store the correct output in a file. Then, to test them when I make changes, I simply run the new script and pipe the output into diff correct_output -. If the files are the same, it outputs nothing. If they're different, it shows you where. This will only work if you are on Linux or OS X; on Windows, you will have to get MSYS.

Example:

python mycliprogram --someoption "some input" | diff correct_output -

To make it even easier, you can add all these test runs to your 'make test' Makefile target, which I assume you already have. ;)

If you are running many of these at once, you could make it a little more obvious where each one ends by adding a fail tag:

python mycliprogram --someoption "some input" | diff correct_output - || tput setaf 1 && echo "FAILED"

Answer 6

So my question is, what is the best way to do testing with CLI program, can it be as easy as unit testing with normal python scripts?

The only difference is that when you run Python module as a script, its __name__ attribute is set to '__main__'. So generally, if you intend to run your script from command line it should have following form:

import sys

# function and class definitions, etc.
# ...
def foo(arg):
    pass

def main():
    """Entry point to the script"""

    # Do parsing of command line arguments and other stuff here. And then
    # make calls to whatever functions and classes that are defined in your
    # module. For example:
    foo(sys.argv[1])


if __name__ == '__main__':
    main()

Now there is no difference, how you would use it: as a script or as a module. So inside your unit-testing code you can just import foo function, call it and make any assertions you want.