IOError: No space left on device - which device?

Question

I\'m uploading a small file (8.5 Mb) to a flask test server.

When the file finishes uploading, the server reports:

    File \"/home/ubuntu/.virtualenvs/e         


        

Answer 1

Werkzeug stores files over a certain size in your temp directory using tempfile.TemporaryFile(), but take into account that those files are unlinked for security. You won't see them listed in the directory. tempfile.gettempdir() is the correct method of determining the directory used for this.

Your /var/tmp directory is probably configured for a smaller partition. Check with df -h to see if that partition still has enough space left. You also need to check for free inodes, with df -i.

It could also be that a process (possibly yours) is hanging on to such unlinked files too long and the space hasn't been returned to the OS yet. You can check for processes holding on to deleted files with:

lsof -nP | grep '/var/tmp' | grep '(deleted)'

or

find /proc/*/fd -ls | grep '/var/tmp' | grep  '(deleted)'

Answer 2

If you can get a shell on the server, try typing df -h and looking for any entries which show Use% of 100%, or Avail of less than your file size.

Answer 3

@Tom Hunt's comment was on the right track.

This unix SE answer explains what happened.

As a protection against low disc space, some daemons automatically "shadows" the current /tmp/ dir with a ram disc if the the root partition runs out of disc space. Sadly there's no automatic reversion of that process once enough disc space is free again.

I unmounted /tmp directory and followed Nitesh's suggestion:

sudo umount /tmp
sudo echo 'MINTMPKB=0' > sudo /etc/default/mountoverflowtmp

and now uploads are working properly.

Answer 4

Try df -i , maybe there are no free inodes.

EDIT:

another option, find werkzeug pid, let it be 777,

• run strace -p 777 &> /tmp/strace_log
• try to upload a file
• stop strace
• find that No space left on device message, it will be like write(1, "blah blah ..."..., 57) = -1 ENOSPC (No space left on device) first argument to write is a file descriptor
• go up and try to locate specific open(... = X syscall , that X is a file descriptor that will fail in "write" syscall step

Pretty cumbersome, I know, but that's debugging.