Automatically deleting pyc files when corresponding py is moved (Mercurial)

Question

(I foresaw this problem might happen 3 months ago, and was told to be diligent to avoid it. Yesterday, I was bitten by it, hard, and now that it has cost me real money, I am kee

Answer 1

You need:

1) A real deployment infrastructure, even if it's just a shell script, which does everything. Cloning/checking out an updated copy from source control is not a deployment strategy.

2) Any deployment system should completely clean the directory structure. My usual preference is that each deployment happens to a new directory named with a date+timestamp, and a symlink (with a name like "current") is updated to point to the new directory. This gives you breadcrumbs on each server should something go wrong.

3) To fix whatever is running the Python code. New .py source files should always take precedence over cached .pyc files. If that is not the behavior you are seeing, it is a bug, and you need to figure out why it is happening.

Answer 2

What I have actually done:

1) I am considering Nicholas Knight's suggestion about using a proper deployment strategy. I have been reading about Buildout and Collective.hostout to learn more. I need to decide whether such heavy-weight strategies are worthwhile for my project's relatively simple requirements.

2) I have adopted Ry4an's update hook concept, in the short-term, until I decide.

3) I ignored Ry4an's warning about overkill, and wrote a Python script to only delete stray .pyc files.

#!/usr/bin/env python
""" Searches subdirectories of the current directory looking for .pyc files which
    do not have matching .py files, and deletes them.

    This is useful as a hook for version control when Python files are moved.
    It is dangerous for projects that deliberately include Python 
    binaries without source.
"""
import os
import os.path
for root, dirs, files in os.walk("."):
    pyc_files = filter(lambda filename: filename.endswith(".pyc"), files)
    py_files = set(filter(lambda filename: filename.endswith(".py"), files))
    excess_pyc_files = filter(lambda pyc_filename: pyc_filename[:-1] not in py_files, pyc_files)
    for excess_pyc_file in excess_pyc_files:
        full_path = os.path.join(root, excess_pyc_file)
        print "Removing old PYC file:", full_path
        os.remove(full_path)

My update hooks now call this rather than the "find" commands suggested by others.

Answer 3

I use the .hgignore file to skip versionning of all my .pyc and .py~ (editor's temp files). For example, this is my version :

# use glob syntax.
syntax: glob

.directory
*.pyc
*~
*.o
*.tgz
*.tbz2
*.gz
*.bz2

Also adding a hook on update to remove them is also a interesting trick if you want to not only ignore noise but remove it from your local workspace area.

Answer 4

Here's a unix one-liner that will delete .pyc files each time you run hg update.

Add this to your hgrc file:

[hooks]
preupdate.cleanpyc = hg status --no-status --removed --deleted --include "**.py" --rev .:$HG_PARENT1 --print0 | xargs -0 -I '{}' rm -f '{}c'

This runs just prior to update, and gets all .py files which will be removed or deleted when the update is performed, and then deletes corresponding .pyc files.

Here's a quick breakdown of how it works:

hg status --no-status --removed --deleted --include "**.py" --rev .:$HG_PARENT1

This gets all files removed (e.g. hg forget) or deleted (hg rm, hg mv, etc) between the current revision . and the destination ($HG_PARENT). You could add --subrepos to get all changes in sub-repositories as well if you use that feature.

xargs -0 -I '{}' rm -f '{}c'

This simply adds a 'c' to the end of each file name returned from hg status and tries to delete it. The -f flag for rm ensures that it doesn't error if the .pyc file does not exist.

Note that mercurial automatically deletes empty directories after an update, but orphaned .pyc files often cause directories to be left around. Since this runs before update, it ensure that empty directories are properly deleted.

Answer 5

i use this script to delete the .pyc files in the current folder, this script could be used alone or include it in the exit function to delete the .pyc files when u exit.

import os
files = [f for f in os.listdir('.') if os.path.isfile(f) and '.pyc' in str(f)]
for f in files : os.unlink(os.getcwd()+'/'+f)

Answer 6

1. Do not store .pyc files in the repository.
2. Automatize .pyc delete with: find . -name '*.pyc' -delete
3. While develop use -B argument in Python.