Yes, this file is required if you want directory to be treated as module.
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.
https://docs.python.org/3/tutorial/modules.html#packages
I try to create non-empty __init__.py. You have great possibility to document module, to get rid of the nested imports for users/developers by providing the most useful objects(classes/functions) at the first level... ...actually to be as simple in use as possible in contrast to - let's say - the java imports
Edit after question update
The default importers/finders (examine the sys.meta_path) are:
- BuiltinImporter - searches for/loads built-in modules
- FrozenImporter - searches for/loads frozen modules (e.g. *.pyc)
- PathFinder - the one you are interested in, allow to search for/load modules based on file system
The third is the __init__.py thing (actually FrozenImporter as well).
ThePathFinder searches for module in paths from sys.path (and in __path__ defined in packages). The module could be either standalone python file (if it is in the root of searched path) or directory with __init__.py.
Referring to your example:
foo/
bar/
__init__.py
baz.py
If you create _init__.py in foo/, foo.bar.baz will be available (as you said).
If you add foo/ to sys.path or pass it through PYTHONPATH=foo/, bar.baz will be available (note without parent module foo).
If you write your own finder (and Loader) you can load for example any file you want despite where it is. That gives you great power. For example take a look on stack-overflow-import, exposes code based on SO's search results.
