Mapping module imports in Python for easy refactoring

Question

I have a bunch of Python modules I want to clean up, reorganize and refactor (there\'s some duplicate code, some unused code ...), and I\'m wondering if there\'s a tool to make

Answer 1

chuckmove is a tool that lets you recursively rewrite imports in your entire source tree to refer to a new location of a module.

chuckmove --old sound.utils --new media.sound.utils src

...this descends into src, and rewrites statements that import sound.utils to import media.sound.utils instead. It supports the whole range of Python import formats. I.e. from x import y, import x.y.z as w etc.

Answer 2

Writing a script that does this probably wouldn't be very complicated (though there are different syntaxes for import to handle),

It's trivial. There's import and from module import. Two syntax to handle.

Do you know of a more exact term for what I'm trying to do? Code reorganization?

Design. It's called design. Yes, you're refactoring an existing design, but...

Rule One

Don't start a design effort with what you have. If you do, you'll only "nibble around the edges" making small and sometimes inconsequential changes.

Rule Two

Start a design effort with what you should have had if you'd only been smarter. Think broadly and clearly about what you're really supposed to be doing. Ignore what you did.

Rule Three

Design from the ground up (or de novo as some folks say) with the correct package and module architecture.

Create a separate project for this.

Rule Four

Test First. Write unit tests for your new architecture. If you have existing unit tests, copy them into the new project. Modify the imports to reflect the new architecture and rewrite the tests to express your glorious new simplification.

All the tests fail, because you haven't moved any code. That's a good thing.

Rule Five

Move code into the new structure last. Stop moving code when the tests pass.

You don't need to analyze imports to do this, BTW. You're just using grep to find modules and classes. The old imports and the tangled relationships among the old imports doesn't matter, and doesn't need to be analyzed. You're throwing it away. You don't need tools smarter than grep.

If feel an urge to move code, you must be very disciplined. (1) you must have test(s) which fail and then (2) you can move some code to pass the failing test(s).

Answer 3

Python's modulefinder does this. It is quite easy to write a script that will turn this information into an import graph (which you can render with e.g. graphviz): here's a clear explanation. There's also snakefood which does all the work for you (and using ASTs, too!)

You might want to look into pylint or pychecker for more general maintenance tasks.

Answer 4

Modulefinder may not work with Python 3.5*, but pydeps worked very well:

Installation:

sudo apt install python-pygraphviz
pip install pydeps

Then, in the directory where you want to map from,

pydeps --max-bacon=0 .

..to create a map of maximum depth.

*An issue in Python 3.5 but not 3.6 caused the problems with modulefinder, similar to this