Using inotify to keep track of all files in a system

Question

Question:

Can inotify be used to reliably record files in a [linux] system?

Details:

I am attempting to use inotifywait to track users movements (currently using bash, but it has been suggested that I migrate to a scripting language). Ultimately I want to add new files to a database upon creation (create, moved_from), update existing rows in a database upon file modification (modify, attrib, move_to), and finally remove a row upon file deletion (delete). I am, however, running into many problems as even an action as seemingly simple as save, generates many inotifywait messages. Observe the following commands and their output (note, the use of /home/user/ is purely for example purposes):

Examples:

Example 1: Listen for file creation:

$ inotifywait -mr /home/user/ -e create  --format %w:%f:%e:%T --timefmt %T

Touch:

$touch test.txt
/home/user/:test.txt:CREATE:21:35:30

Open a new file with vim then issue :w command:

$vim test2.txt
/home/user/:test2.txt:CREATE:21:35:30

Open an existing file with vim then issue :w command:

$vim test2.txt
/home/user/:4913:CREATE:21:35:30
/home/user/:test2.txt:CREATE:21:35:30

Open a new file with gedit then click save:

$gedit test3.txt
/home/user/:test3.txt~:CREATE:21:35:30

Open an existing file with gedit then click save:

$gedit test3.txt
/home/user/:.goutputstream-HN3ZDW:CREATE:21:35:30
/home/user/:test3.txt~:CREATE:21:35:30

Note that not only are two new files displayed as having ben created (4913 and .goutputstream-HN3ZDW), but also that the only file being created is test3.txt~ and not test3.txt, even though the file test3.txt is created when checked with the ls command. For completeness, here is the above example, but with a few more options.

Example 1: Listen for file creation, modification, deltion, and movement:

$ inotifywait -mr /home/user/ -e create -e modify -e delete -e moved_to -e moved_from --format %w:%f:%e:%T --timefmt %T

Touch:

$touch test.txt
/home/user/:test.txt:CREATE:21:35:30

Open a new file with vim then issue :w command:

$vim test2.txt
/home/user/:test2.txt:CREATE:22:12:32

Open an existing file with vim then issue :w command:

$vim test2.txt
/home/user/:4913:CREATE:22:04:35
/home/user/:4913:DELETE:22:04:35
/home/user/:test2.txt:MOVED_FROM:22:04:35
/home/user/:test2.txt~:MOVED_TO:22:04:35
/home/user/:test2.txt:CREATE:22:04:35
/home/user/:test2.txt~:DELETE:22:04:35

Open a new file with gedit then click save:

$gedit test3.txt
/home/user/:test3.txt~:CREATE:21:35:30

Open an existing file with gedit then click save:

$gedit test3.txt
/home/user/:.goutputstream-0WQ2DW:CREATE:22:06:34
/home/user/:test3.txt~:CREATE:22:06:34
/home/user/:.goutputstream-0WQ2DW:MOVED_FROM:22:06:34
/home/user/:test3.txt:MOVED_TO:22:06:34

Basically my question is "is it possible to use inotify to update a file in a database"? For example, if a user edits a file and saves it, I want it to be reflected in the database as an update to that file, and not a brand new file replacing a completely different file. Any help would be greatly appreciated, even if it's a suggestion pointing me in a different direction.

Answer 1

inotify tells you what happens like it happens.

Gedit, like most editors, saves by first writing a temporary file then moving that file into place. This avoids overwriting the file with a half-written version in case the editor or the whole system crashes while the file is being written. Vim takes a different approach (this can be configured, I won't go into details here — see e.g. why inode value changes when we edit in “vi” editor?): it first creates a temporary backup file, then writes the new file.

If you want these to be recorded as a single editing event, you'll have to perform some pattern recognition on the even log. A create-write-move sequence that replaces an existing file and a create-move-create delete sequence like vim's would be the archetypal patterns. Note that the pattern might be interleaved with other events.

I have a suspicion that there's a better way to do what you want to do, but I don't understand what you're trying to do. If you're trying to log user actions, you have already found a way, but there are simpler ways: loggedfs or the audit subsystem. If you want to keep a backup of all file versions, either hook up the editor to a version control system (this lets users control what gets backed up) or use a versioning filesystem such as copyfs. You can even store the files in the database directly, by using a filesystem like mysqlfs or postgresqlfs (admittedly neither project looks maintained).