I woke up this morning, got online through my new UMTS/HSPDA modem, and discovered that one of my servers had a load average of 239+

Not the best way to start the day.

Turns out an rsync backup job between two servers had gone nuts and was spinning them up through the roof. A couple

sudo killall rsync

commands, and everything started settling down.

What's important to note is that while the CPU was pegged, and the load average was over 200, I was still able ssh in, run top, ps, netstat, and navigate around looking at log files, with very little delay. It was just a little slow, but no more than a second wait for anything.

Also, all of my apps which used MySQL as their backend were all dead, with "database refused connection" errors. All of my apps which used PostgreSQL as their backend were a little slow, but were all still up and functioning without errors.

So, Debian and PostgreSQL for the win!

If you have a server with a world facing ssh server, you've probably seen brute force attacks in your logs. Some machine starts hammering your ssh server, trying all sorts of logins (staff, root, a, admin, etc...) over and over and over again.

This is bad on a lot of fronts.

I use two simple iptables rules to block any IP address who has made more than 3 ssh connections or attempted connections within the past 3 minutes. So your would-be brute force attacker, gets three tries, and then is locked out for a minimum of three minutes. However, since 99% of the attacks are run by an automated bot, it will either: give up after the connection is refused multiple times, or it will keep hammering away on the closed door, which keeps the running count of attempted connections in the past 3 minutes over 3, keeping the door closed.

First:

iptables -I INPUT -i eth1 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name DEFAULT --rsource

Then run:

iptables -I INPUT -i eth1 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 180 --hitcount 4 --name DEFAULT --rsource -j DROP

I'd also recommend using the script in my post on blocking IP addresses using iptables to deal with any persistent folks, or people poking too hard on your web site, or other services.