How can I safely create a nested directory?

Question

What is the most elegant way to check if the directory a file is going to be written to exists, and if not, create the directory using Python? Here is what I tried:

Answer 1

If you consider the following:

os.path.isdir('/tmp/dirname')

means a directory (path) exists AND is a directory. So for me this way does what I need. So I can make sure it is folder (not a file) and exists.

Answer 2

import os
if os.path.isfile(filename):
    print "file exists"
else:
    "Your code here"

Where your code here is use the (touch) command

This will check if the file is there if it is not then it will create it.

Answer 3

Starting from Python 3.5, pathlib.Path.mkdir has an exist_ok flag:

from pathlib import Path
path = Path('/my/directory/filename.txt')
path.parent.mkdir(parents=True, exist_ok=True) 
# path.parent ~ os.path.dirname(path)

This recursively creates the directory and does not raise an exception if the directory already exists.

(just as os.makedirs got an exist_ok flag starting from python 3.2 e.g os.makedirs(path, exist_ok=True))

---

Note: when i posted this answer none of the other answers mentioned exist_ok...

Answer 4

I have put the following down. It's not totally foolproof though.

import os

dirname = 'create/me'

try:
    os.makedirs(dirname)
except OSError:
    if os.path.exists(dirname):
        # We are nearly safe
        pass
    else:
        # There was an error on creation, so make sure we know about it
        raise

Now as I say, this is not really foolproof, because we have the possiblity of failing to create the directory, and another process creating it during that period.

Answer 5

Check if a directory exists and create it if necessary?

The direct answer to this is, assuming a simple situation where you don't expect other users or processes to be messing with your directory:

if not os.path.exists(d):
    os.makedirs(d)

or if making the directory is subject to race conditions (i.e. if after checking the path exists, something else may have already made it) do this:

import errno
try:
    os.makedirs(d)
except OSError as exception:
    if exception.errno != errno.EEXIST:
        raise

But perhaps an even better approach is to sidestep the resource contention issue, by using temporary directories via tempfile:

import tempfile

d = tempfile.mkdtemp()

Here's the essentials from the online doc:

mkdtemp(suffix='', prefix='tmp', dir=None)
    User-callable function to create and return a unique temporary
    directory.  The return value is the pathname of the directory.

    The directory is readable, writable, and searchable only by the
    creating user.

    Caller is responsible for deleting the directory when done with it.

New in Python 3.5: pathlib.Path with exist_ok

There's a new Path object (as of 3.4) with lots of methods one would want to use with paths - one of which is mkdir.

(For context, I'm tracking my weekly rep with a script. Here's the relevant parts of code from the script that allow me to avoid hitting Stack Overflow more than once a day for the same data.)

First the relevant imports:

from pathlib import Path
import tempfile

We don't have to deal with os.path.join now - just join path parts with a /:

directory = Path(tempfile.gettempdir()) / 'sodata'

Then I idempotently ensure the directory exists - the exist_ok argument shows up in Python 3.5:

directory.mkdir(exist_ok=True)

Here's the relevant part of the documentation:

If exist_ok is true, FileExistsError exceptions will be ignored (same behavior as the POSIX mkdir -p command), but only if the last path component is not an existing non-directory file.

Here's a little more of the script - in my case, I'm not subject to a race condition, I only have one process that expects the directory (or contained files) to be there, and I don't have anything trying to remove the directory.

todays_file = directory / str(datetime.datetime.utcnow().date())
if todays_file.exists():
    logger.info("todays_file exists: " + str(todays_file))
    df = pd.read_json(str(todays_file))

Path objects have to be coerced to str before other APIs that expect str paths can use them.

Perhaps Pandas should be updated to accept instances of the abstract base class, os.PathLike.

Answer 6

For a one-liner solution, you can use IPython.utils.path.ensure_dir_exists():

from IPython.utils.path import ensure_dir_exists
ensure_dir_exists(dir)

From the documentation: Ensure that a directory exists. If it doesn’t exist, try to create it and protect against a race condition if another process is doing the same.