Packaging and shipping a python library and scripts, the professional way

Question

I have the task of packaging and shipping a commercial application bundle, which will include:

1. a python library (developed by us)
2. some python programs dep

Answer 1

For windows I am doing this:

1. Use cx_freeze like this:

   c:\Python27\Scripts\cxfreeze.bat source.py --target-dir=Bin 
   --base-name=Win32GUI --icon=icon.ico --target-name=Executable.exe
   

   This creates directory with runnable program.
   You can use another packing application such as PyInstaller (Windows, Linux), Py2Exe (Windows), or Py2App (Mac), but I got the best results with Cx_freeze (probably all platforms you mentioned). It has also very good support and is still actively maintained.

2. Pack everything using Inno Setup (Windows).
   You can use any method to create installer you like. I am not familiar with other platforms.

3. Test it in VMware virtual machine on clean OS installation just to be sure.

Actually, it is very easy to do, I am not sure what to add.

---

Edit: You can also use Portable Python for your bundled environment on Windows.

Answer 2

As far as I can see Python is distributed via a msi file. It seems natural to me that you create a msi installer as well. To create msi files you can use e.g. the Windows Installer Extensions toolkit. If you create your own msi and include the other msi into yours then you will get into trouble to install the pyhthon package during your own msi installation routine since Windows does only allow one installation at a time. The easiest way would be to repackage the python msi into your own msi and distribute this one. You can decompile a msi into a xml file which can be used to create a new msi with dark which is part of the Windows Installer Extensions toolkit.

If you did manage it to deploy the your whole suite via a tar file it should be possible to use a zip file on Windows as well if the pyhthon and your other stuff is xcopy deployable. When you must do some actions like setting registry keys, environment variables, creating shortcuts you should use msi since it was designed for this task. But be warned: Msi is a difficult topic. If you need to get something quick you should check how far you can get with a zip file and some scripts. A msi based installation will make servicing and patching much easier but to appreciate these advanced features you will need to invest some weeks to learn it.

Answer 3

You could use Makeself which is just like a tar.gz, but produces a .sh which self-extracts and allows for execution of a custom install script (don't ask me about Windows). This avoids bundling an installed Python which almost certainly won't work - you can include an installer instead.

Your code and the dependencies developed by you should be included as packages created by sdist, which can be installed by PIP and easyinstall into a virtualenv based on your python. In your Manifest.in you can easily include only pyc files and everything else that is necessary and exclude py files so nobody sees your sources. Dependencies will be installed automatically by downloading them, but you can avoid that by including them in your archive, like your dependencies. Just put them into a directory and add "-f file:path_to_your_directory" to you PIP call.

Answer 4

The proper way to do this (w/o shipping own Python) is to create Python Eggs, which then can be installed by Python package manager like easy_install or pip. The setup and install is performed by setup.py, which is just a Python script, so you can include all kinds of non-standard setup procedures if needed.

When using pip, it's easy to install eggs in virtualenv, which would make it isolated from user's Python installation (apart of the interpreter binary itself).

Answer 5

This is not a complete answer but just a bunch of ideas. I wrote an installer for a client that incorporated some ideas that might be useful to you.

It was Linux only so I focussed on just that. We needed to ship specific custom versions of mySQL, lighttpd, python, memcached, a few 3rd party Python modules and some custom scripts. We needed to launch all these services without any problems and let the user control them using regular initscripts. It should work fine on a bunch of popular distros and therefore shouldn't rely on distro specific stuff.

What I did was as follows.

1. Created a 500MB (I'm don't recollect the size) file and formatted it as an ext3fs file system.
2. Mounted it at a point using a loopback device.
3. Ran deb-bootstrap on the mountpoint to create a custom Debian install.
4. Chrooted inside the partition and then ran a bunch of scripts which did an apt-get install on all our dependencies, installed all the eggs and other packages which were necessary for the app, installed the app itself in /opt (inside the chroot), installed supervisord (to do process management) and set things up. Now, this partition was a completely self contained Linux filesystem that contained the application and everything needed to run it. You could dump it anywhere, chroot inside it and launch the app. The only dependency it had with the outside world were the ports it would use for its services and the supervisord control socket. This was the main point. We were able to include exactly what we needed (compiled files, .pycs only etc.) for a few of the applications and didn't have to bother with any limitations in standard installation tools.
5. After this, we packaged a few extra scripts that would go into the external operating system. These were custom made for each distro that we would have to support. This part was distro specific. There were scripts that would go into /etc/init.d and some scripts that would setup the database and stuff at the beginning.
6. We then created an archive of the entire filesystem using makeself. It would checksum stuff and all that and provide a self extracting archive which if run would untar the whole thing into /opt on the host machine, chroot inside the directory and run a setup script that would ask the user a few questions like db username/password etc. and set things up. After that, it would fetch the scripts I mentioned in step 5 and put them on the host OS.

The initscripts would simply chroot into the partition and start supervisord. It would then take care of launching all the services we cared about. Shutting down the application was simply a matter of connecting to running supervisord and running a command. We wrapped this in the initscript so that the user experience was UNIX like.

Now, we'd give clients the self extracting .run file. They'd run it, get asked a few questions and it would create a directory under /opt which contained our app and all it's dependencies. The init scripts would be modified to start our app on bootup and things would work as expected.

I think step 4 gives you the freedom to install whatever you want, however you want so that things would work fine.

Answer 6

I am having the same issue, however most information i could find was only about python packages. I do not have a complete solution yet but, I have a few suggestions.

1. Keep potential dependencies for the python interpreter in mind, eg. on Linux python depends on the glibc. Try either to compile it statically or use a very common glibc version. You might want to have a look at Activestate’s python distribution; the issue is they want a LOT of money for it.

2. Consider using a package format like DEB or RPM instead of tar.gz for linux. This would allow you to resolve dependencies, if there are any and would make an update process easier.

3. Even as you, want to distribute you application as a tar.gz I would recommend building a package for this application. Not necessarily for the user, but for you when you are building the environment, which should run on a customer site. It would allow to test and build and update the environment more easily.