How do I find out what inotify watches have been registered?

Question

I have my inotify watch limit set to 1024 (I think the default is 128?). Despite that, yeoman, Guard and Dropbox constantly fail, and tell me to up my inotify limit. Before do

Answer 1

1. The default maximum number of inotify watches is 8192; it can be increased by writing to /proc/sys/fs/inotify/max_user_watches.
   You can use sysctl fs.inotify.max_user_watches to check current value.

2. Use tail -f to verify if your OS does exceed the inotify maximum watch limit.
   The internal implementation of tail -f command uses the inotify mechanism to monitor file changes.
   If you've run out of your inotify watches, you'll most likely to get this error:

   tail: inotify cannot be used, reverting to polling: Too many open files

3. To find out what inotify watches have been registered, you may refer to this, and this. I tried, but didn't get the ideal result. :-(

Reference:
https://askubuntu.com/questions/154255/how-can-i-tell-if-i-am-out-of-inotify-watches
https://unix.stackexchange.com/questions/15509/whos-consuming-my-inotify-resources
https://bbs.archlinux.org/viewtopic.php?pid=1340049

Answer 2

inotify filesystem options

sysctl fs.inotify

opened files

lsof | grep inotify | wc -l

Increase the values like this

• sysctl -n -w fs.inotify.max_user_watches=16384
• sysctl -n -w fs.inotify.max_user_instances=512

Answer 3

I already answered this in the same thread on Unix Stackexchange as was mentioned by @cincodenada, but thought I could repost my ready-made answer here, seeing that no one really has something that works:

---

I created a premade script, inotify-consumers, which is pretty fast (< 0.1 sec), that lists the top offenders for you:

$ inotify-consumers  

   INOTIFY
   WATCHER
    COUNT     PID     CMD
----------------------------------------
    6688    27262  /home/dvlpr/apps/WebStorm-2018.3.4/WebStorm-183.5429.34/bin/fsnotifier64
     411    27581  node /home/dvlpr/dev/kiwi-frontend/node_modules/.bin/webpack --config config/webpack.dev.js
      79     1541  /usr/lib/gnome-settings-daemon/gsd-xsettings
      30     1664  /usr/lib/gvfs/gvfsd-trash --spawner :1.22 /org/gtk/gvfs/exec_spaw/0
      14     1630  /usr/bin/gnome-software --gapplication-service

Here you quickly see why the default limit of 8K watchers is too little on a development machine, as just WebStorm instance quickly maxes this when encountering a node_modules folder with thousands of folders. Add a webpack watcher to guarantee problems ...

Just copy the contents of the script (or the file on GitHub) and put it somewhere in your $PATH, like /usr/local/bin. For reference, the main content of the script is simply this

find /proc/*/fd \
    -lname anon_inode:inotify \
    -printf '%hinfo/%f\n' 2>/dev/null \
    \
    | xargs grep -c '^inotify'  \
    | sort -n -t: -k2 -r 

In case you are wondering how to increase the limits, here's how to make it permanent:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Answer 4

The following terminal command worked perfectly for me on my Ubuntu 16.04 Machine:

for foo in /proc/\*/fd/*; do readlink -f $foo; done |grep inotify |cut -d/ -f3 |xargs -I '{}' -- ps --no-headers -o '%p %U %a' -p '{}' |uniq -c |sort -n

My problem was that I had a good majority of my HDD loaded as a folder in Sublime Text. Between /opt/sublime_text/plugin_host 8992 and /opt/sublime_text/sublime_text, Sublime had 18 instances of inotify while the rest of my programs were all between 1-3.

Since I was doing Ionic Mobile App development I reduced the number of instances by 5 by adding the large Node.js folder "node_modules" to the ignore list in the Sublime settings.

"folder_exclude_patterns": [".svn", ".git", ".hg", "CVS", "node_modules"]

Source: https://github.com/SublimeTextIssues/Core/issues/1195

Answer 5

Based on the excellent analysis of cincodenada, I made my own one-liner, which works better for me:

find /proc/*/fd/ -type l -lname "anon_inode:inotify" -printf "%hinfo/%f\n" | xargs grep -cE "^inotify" | column -t -s:

It helps to find all inotify watchers and their watching count. It does not translate process ids to their process names or sort them in any way but that was not the point for me. I simply wanted to find out which process consumes most of the watches. I then was able to search for that process using its process id.

You can omit the last column command if you don't have it installed. It's only there to make the output look nicer.

Okay, as you can see, there is a similar and less fork hungry approach from @oligofren. Better you use his simple script. It's very nice. I was also able to shrink my one-liner because I was not aware of the -lname parameter of find which comes in very handy here.

Answer 6

Since this is high in Google results, I'm copy-pasting part of my answer from a similar question over on the Unix/Linux StackExchange:

I ran into this problem, and none of these answers give you the answer of "how many watches is each process currently using?" The one-liners all give you how many instances are open, which is only part of the story, and the trace stuff is only useful to see new watches being opened.

This will get you a file with a list of open inotify instances and the number of watches they have, along with the pids and binaries that spawned them, sorted in descending order by watch count:

sudo lsof | awk '/anon_inode/ { gsub(/[urw]$/,"",$4); print "/proc/"$2"/fdinfo/"$4; }' | while read fdi; do count=$(sudo grep -c inotify $fdi); exe=$(sudo readlink $(dirname $(dirname $fdi))/exe); echo -e $count"\t"$fdi"\t"$exe; done | sort -nr > watches

If you're interested in what that big ball of mess does and why, I explained in depth over on the original answer.