Inotify resource exhaustion

[kazkansouh]

Hello,

Thank you for making such a useful tool :)

So recently I noticed that when running pspy, that it was not very well picking up some short lived cron scripts. After more investigation, I realised the issue was related to running multiple copies of the tool. The first instance works fine, but the second instance misses lots of short lived processes. This is because in its default configuration it attempts to use inotify on almost every directory it has access to, exhausting the users supply of inotify watches. Then the second instance is unable to get notified in a timely fashion when a process start.

Linux limits the number of inotify watches with the value in /proc/sys/fs/inotify/max_user_watches. From the Debian based machines I've looked at this is set to 8192. Thus, on the machine I was testing on (fairly basic Ubuntu 18.04 with Python and PHP) the number of directories are:

$ find /usr /tmp /etc /home /var /opt -type d 2>/dev/null | wc -l
12518

Thus, just running once instance of this tool completely exhausts the limit for inotify watches. Also, this could also be unfair to other CTF players.

Moreover, I tested watching different directories. It appears that probably the only file that needs to be watched to provide a good user experience is /etc/ld.so.cache. I suspect that the vast majority of the time the processes that will be of interest are dynamic.

Probably, the better solution is to replace the attempt at blanket inotify watches to the whole file system and instead use a short list of directories. E.g. something like (with bin directories to catch any static binary executions):

• /etc
• /bin
• /usr/bin
• /usr/local/bin

Possibly, it could print a warning of the ld.so.cache if is not found in one of the directories.

Of course, the user will always be free to configure the tool as they want, but in a default config I dont think it should inadvertently perform a resource exhaustion attack!

I hope this is of some use, it took a while to properly identify. Also, while tracking down the issue, I came across some other minor improvements that I'll submit a PR for you to consider.

[DominicBreuker]

hey, thanks for your feedback!

absolutely true. in almost all cases it will be enough to monitor ld.so.cache or just a bunch of libraries or them system binaries themselves. back when I built this I was a little paranoid that I could miss executions of static binaries hidden in non-standard locations. but the exhaustion is indeed limiting the value. annoying on shared CTF boxes, not particularly stealthy in other situations.

maybe the best way to go would be monitoring of 10 to 15 standard directories, possibly depending on distro. using the existing flags you could add other directories as needed.

[kazkansouh]

Yep, I completely understand about being paranoid and trying to get everything possible, and it generally does a good job.

maybe the best way to go would be monitoring of 10 to 15 standard directories, possibly depending on distro. using the existing flags you could add other directories as needed.

👍 I think this is the way forward, maybe even just /etc by default, it seems reasonable for anything that is FHS complaint?

Just running a quick test on here shows the following:

• left: ./pspy64 -d /etc -r "" -t 40
• right: ./pspy64 -t 40

It looks like just using just /etc is performing a bit better than the full recursive approach. But, its worth noting that the recursive one was better at catching commands typed by hand into the terminal. I,.e., it caught all three ls commands as opposed to just one on the left.

[QuantumShadow87-cloud]

Live an we learn an we code to create better