Race-condition creating folder in Python

Question

I have a urllib2 caching module, which sporadically crashes because of the following code:

if not os.path.exists(self.cache_location):
    os.mkdir(self.cach         


        

Answer 1

The code I ended up with was:

import os
import errno

folder_location = "/tmp/example_dir"

try:
    os.mkdir(folder_location)
except OSError as e:
    if e.errno == errno.EEXIST and os.path.isdir(folder_location):
        # File exists, and it's a directory,
        # another process beat us to creating this dir, that's OK.
        pass
    else:
        # Our target dir exists as a file, or different error,
        # reraise the error!
        raise

Answer 2

In Python 3.x, you can use os.makedirs(path, exist_ok=True), which will not raise any exception if such directory exists. It will raise FileExistsError: [Errno 17] if a file exists with the same name as the requested directory (path).

Verify it with:

import os

parent = os.path.dirname(__file__)

target = os.path.join(parent, 'target')

os.makedirs(target, exist_ok=True)
os.makedirs(target, exist_ok=True)

os.rmdir(target)

with open(target, 'w'):
    pass

os.makedirs(target, exist_ok=True)

Answer 3

Instead of

if not os.path.exists(self.cache_location):
    os.mkdir(self.cache_location)

you could do

try:
    os.makedirs(self.cache_location)
except OSError:
    pass

As you would end up with the same functionality.

_{DISCLAIMER: I don't know how Pythonic this might be.}

---

Using SQLite3, might be a bit of overkill, but would add a lot of functionality and flexibility to your use case.

If you have to do a lot of "selecting", concurrent inserting and filtering, it's a great idea to use SQLite3, as it wont add too much complexity over simple files (it could be argued that it removes complexity).

---

Rereading your question (and comments) I can better understand your problem.

What is the possibility that a file could create the same race condition?

If it is small enough, then I'd do something like:

if not os.path.isfile(self.cache_location):
    try:
        os.makedirs(self.cache_location)
    except OSError:
        pass

Also, reading your code, I'd change

else:
    # Our target dir is already a file, or different error,
    # relay the error!
    raise OSError(e)

to

else:
    # Our target dir is already a file, or different error,
    # relay the error!
    raise

as it's really what you want, Python to reraise the exact same exception _{(just nitpicking)}.

---

One more thing, may be this could be of use for you (Unix-like only).

Answer 4

Could you catch the exception and then test whether the file exists as a directory or not?

Answer 5

When you have race conditions often EAFP(easier to ask forgiveness than permission) works better that LBYL(look before you leap)

Error checking strategies