File scan daemon (OS X) in Python

Question

I'm currently using Python's function,

os.walk('/')

to loop through my whole filesystem on OS X. My aim is to make a personal daemon that keeps track of:

1. Newly made files/dirs
2. Adjusted/touched files/dirs
3. Deleted files (maybe)

The idea

This is more of a precautious function I wanted to add to my Macs to be able to see if weird stuff gets placed in my directories unwanted, so if my Macs ever get infected by some (yet unknown) trojan I can maybe detect it myself already. Also I'm looking into adding features later to maybe shut down my internet connections etc. when something off is detected. This is maybe an irrational function, but as it's just a personal script I think it's not that bad :P.

What I want to achieve

So my main question is. After the first run I will save an array of the whole filesystem and their metadata (creation data, modification date). After that the I want the daemon to run in the background in a "watching" mode, practically mirroring the last known stored array of the filesystem with a newly looped one. The problem now is that when I run the script to test it litteraly starts to burn my CPU, making my MacBook starting to get hiccups after a while. I want to add sleeps between each directory step "os.walk()" makes in my for loop. My question is: What is a right sleep time? My MacBook's Disk Utility says I have 183.867 folders and 1.013.320 files making a total of 1.197.187 entries (as Folders are practically files too). So setting my code to:

time.sleep(0.001)

..would approximately take those 1.2Million entries to be dealt in about 2 minutes. I have no clue if this is good and I prefer to make it more dynamically based on the total files/folders count.

An extra feature Based on my question I noticed that the OS X Disk Utility already knows my total Files and Folders. Can Python get this data too without doing an extreme loop-through? Maybe calling a Terminal function built-in to OS X. This way I could also keep an indicator if using a little GUI for my Daemon's status.

Thanks in advance!

Answer 1

This is not a direct answer but directed to the requirement to track:

Newly made files/dirs
Adjusted/touched files/dirs
Deleted files (maybe)

You can use : http://pyinotify.sourceforge.net/ which ties with inotify and will send events on file changes, deletion and creation. This will avoid the walk through of large directories.

This is wrapper over Inotify . This is available for Linux. I also see that there are libraries and modules for OSX in fink and macports. So this should be more elegant solution.

Answer 2

Not a complete answer, but a few pointers:

For OSX, fseventsd implements a mechanism similar to inotify. fslogger is an example of how it can be used. pymacadmin seems to allow you to interface it from Python.

What you want to implement is similar to Tripwire .