How do you add additional files to a wheel?

Question

How do control what files are included in a wheel? It appears MANIFEST.in isn't used by python setup.py bdist_wheel.

UPDATE:

I was wrong about the difference between installing from a source tarball vs a wheel. The source distribution includes files specified in MANIFEST.in, but the installed package only has python files. Steps are needed to identify additional files that should be installed, whether the install is via source distribution, egg, or wheel. Namely, package_data is needed for additional package files, and data_files for files outside your package like command line scripts or system config files.

Original Question

I have a project where I've been using python setup.py sdist to build my package, MANIFEST.in to control the files included and excluded, and pyroma and check-manifest to confirm my settings.

I recently converted it to dual Python 2 / 3 code, and added a setup.cfg with

[bdist_wheel]
universal = 1

I can build a wheel with python setup.py bdist_wheel, and it appears to be a universal wheel as desired. However, it doesn't include all of the files specified in MANIFEST.in.

What gets installed?

I dug deeper, and now know more about packaging and wheel. Here's what I learned:

I upload two package files to the multigtfs project on PyPi:

• multigtfs-0.4.2.tar.gz - the source tar ball, which includes all the files in MANIFEST.in.
• multigtfs-0.4.2-py2.py3-none-any.whl - The binary distribution in question.

I created two new virtual environments, both with Python 2.7.5, and installed each package (pip install multigtfs-0.4.2.tar.gz). The two environments are almost identical. They have different .pyc files, which are the "compiled" Python files. There are log files which record the different paths on disk. The install from the source tar ball includes a folder multigtfs-0.4.2-py27.egg-info, detailing the installation, and the wheel install has a multigtfs-0.4.2.dist-info folder, with the details of that process. However, from the point of view of code using the multigtfs project, there is no difference between the two installation methods.

Explicitly, neither has the .zip files used by my test, so the test suite will fail:

$ django-admin startproject demo
$ cd demo
$ pip install psycopg2  # DB driver for PostGIS project
$ createdb demo         # Create PostgreSQL database
$ psql -d demo -c "CREATE EXTENSION postgis" # Make it a PostGIS database 
$ vi demo/settings.py   # Add multigtfs to INSTALLED_APPS,
                        # Update DATABASE to set ENGINE to django.contrib.gis.db.backends.postgis
                        # Update DATABASE to set NAME to test
$ ./manage.py test multigtfs.tests  # Run the tests
...
IOError: [Errno 2] No such file or directory: u'/Users/john/.virtualenvs/test/lib/python2.7/site-packages/multigtfs/tests/fixtures/test3.zip'

Specifying additional files

Using the suggestions from the answers, I added some additional directives to setup.py:

from __future__ import unicode_literals
# setup.py now requires some funky binary strings
...
setup(
    name='multigtfs',
    packages=find_packages(),
    package_data={b'multigtfs': ['test/fixtures/*.zip']},
    include_package_data=True,
    ...
)

This installs the zip files (as well as the README) to the folder, and tests now run correctly. Thanks for the suggestions!

Answer 1

Have you tried using package_data in your setup.py? MANIFEST.in seems targetted for python versions <= 2.6, I'm not sure if higher versions even look at it.

After exploring https://github.com/pypa/sampleproject, their MANIFEST.in says:

# If using Python 2.6 or less, then have to include package data, even though
# it's already declared in setup.py
include sample/*.dat

which seems to imply this method is outdated. Meanwhile, in setup.py they declare:

setup(
    name='sample',
    ...
    # If there are data files included in your packages that need to be
    # installed, specify them here.  If using Python 2.6 or less, then these
    # have to be included in MANIFEST.in as well.
    package_data={
        'sample': ['package_data.dat'],
    },
    ...
)

(I'm not sure why they chose a wildcard in MANIFEST.in and a filename in setup.py. They refer to the same file)

Which, along with being simpler, again seems to imply that the package_data route is superior to the MANIFEST.in method. Well, unless you have to support 2.6 that is, in which case my prayers go out to you.

Answer 2

Before you make any changes in MANIFEST.in or setup.py you must remove old output directories. Setuptools is caching some of the data and this can lead to unexpected results.

rm -rf build *.egg-info

If you don't do this, expect nothing to work correctly.

Now that is out of the way.

1. If you are building a source distribution (sdist) then you can use any method below.

2. If you are building a wheel (bdist_wheel), then include_package_data and MANIFEST.in are ignored and you must use package_data and data_files.

INCLUDE_PACKAGE_DATA

This is a good option, but bdist_wheel does not honor it.

setup(
    ...
    include_package_data=True
)

# MANIFEST.in
include package/data.json

DATA_FILES for non-package data

This is most flexible option because you can add any file from your repo to a sdist or bdist_wheel

setup(
    ....
    data_files=[
        ('output_dir',['conf/data.json']),
    ]
    # For sdist, output_dir is ignored!
    #
    # For bdist_wheel, data.json from conf dir in root of your repo 
    # and stored at `output_dir/` inside of the sdist package.
)

PACKAGE_DATA for non-python files inside of the package

Similar to above, but for a bdist_wheel let's you put your data files inside of the package. It is identical for sdist but has more limitations than data_files because files can only source from your package subdir.

setup(
    ...
    package_data={'package':'data.json'},
    # data.json must be inside of your actual package
)

Answer 3

You can use package_data and data_files in setup.py to specify additional files, but they are ridiculously hard to get right (and buggy).

An alternative is to use MANIFEST.in and add include_package_data=True in setup() of your setup.py as indicated here.

With this directive, the MANIFEST.in will be used to specify the files to include not only in source tarball/zip, but also in wheel and win32 installer. This also works with any python version (i tested on a project from py2.6 to py3.6).

Answer 4

You can specify extra files to install using the data_files directive. Is that what you're looking for? Here's a small example:

from setuptools import setup
from glob import glob

setup(
    name='extra',
    version='0.0.1',
    py_modules=['extra'],
    data_files=[
        ('images', glob('assets/*.png')),
    ],
)

Answer 5

I had config/ directory with JSON files in it, which I needed to add to the wheel package. So, I've added these lines to MANIFEST.in:

recursive-include config/ *.json

The following directive to setup.py:

setup(
 ...
 include_package_data=True,
)

And nothing worked. Until I've created an empty file called __init__.py inside config/ directory.

(Python 3.6.7, wheel 3.6.7, setuptools 39.0.1)

Answer 6

"A wheel is a ZIP-format archive ..." (http://wheel.readthedocs.org/en/latest)

So, treat the .whl file just like a .zip file. Add a file with:

• Python ZipFile object
• any archive/zip file utility, like 7-zip, Winzip, Ark, file-roller, etc.