Scientific Computing & Ipython Notebook: How to organize code?

Question

I\'m using Ipython Notebook to my research. As my file grows bigger, I constantly extract code out, things like plot method, fitting method etc.

I think I need a way to

Answer 1

You should ideally have a library hierarchy. I would organize it as follows:

Package wsautils

Fundamental, lowest level package [No dependencies]

stringutils.py: Contains the most basic files such string manipulation dateutils.py: Date manipulation methods

Package wsadata

• Parsing data, dataframe manipulations, helper methods for Pandas etc.
• Depends on [wsautils]
  • pandasutils.py
  • parseutils.py
  • jsonutils.py [this could also go in wsautils]
  • etc.

Package wsamath (or wsastats)

Math related utilities, models, PDF, CDFs [Depends on wsautils, wsadata] Contains: - probabilityutils.py - statutils.py etc.

Package wsacharts [or wsaplot]

• GUI, Plotting, Matplotlib, GGplot etc
• Depends on [wsautils, wsamath]
  • histogram.py
  • pichart.py
  • etc. Just an idea, you could also just have a single file here called chartutils or something

You get the idea. Create more libraries as necessary without making too many.

Few other tips:

• Follow the principles of good python package management thoroughly. Read this http://python-packaging-user-guide.readthedocs.org/en/latest/installing/
• Enforce strict dependency management via a script or a tool such that there are no circular dependencies between packages
• Define the name and purpose of each library/module well so that other users also can intuitively tell where a method/utility should go
• Follow good python coding standards (see PEP-8)
• Write test cases for every library/package
• Use a good editor (PyCharm is a good one for Python/iPython)
• Document your APIs, methods

Finally, remember that there are many ways to skin a cat and the above is just one that I happen to like. HTH.