I woke up this morning, got online through my new UMTS/HSPDA modem, and discovered that one of my servers had a load average of 239+
Not the best way to start the day.
Turns out an rsync backup job between two servers had gone nuts and was spinning them up through the roof. A couple
sudo killall rsync
commands, and everything started settling down.
What’s important to note is that while the CPU was pegged, and the load average was over 200, I was still able ssh in, run top, ps, netstat, and navigate around looking at log files, with very little delay. It was just a little slow, but no more than a second wait for anything.
Also, all of my apps which used MySQL as their backend were all dead, with “database refused connection” errors. All of my apps which used PostgreSQL as their backend were a little slow, but were all still up and functioning without errors.
So, Debian and PostgreSQL for the win!
If you have a server with a world facing ssh server, you’ve probably seen brute force attacks in your logs. Some machine starts hammering your ssh server, trying all sorts of logins (staff, root, a, admin, etc…) over and over and over again.
This is bad on a lot of fronts.
I use two simple iptables rules to block any IP address who has made more than 3 ssh connections or attempted connections within the past 3 minutes. So your would-be brute force attacker, gets three tries, and then is locked out for a minimum of three minutes. However, since 99% of the attacks are run by an automated bot, it will either: give up after the connection is refused multiple times, or it will keep hammering away on the closed door, which keeps the running count of attempted connections in the past 3 minutes over 3, keeping the door closed.
iptables -I INPUT -i eth1 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name DEFAULT --rsource
iptables -I INPUT -i eth1 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 180 --hitcount 4 --name DEFAULT --rsource -j DROP
I’d also recommend using the script in my post on blocking IP addresses using iptables to deal with any persistent folks, or people poking too hard on your web site, or other services.
This is mostly for own use, but: If you’ve ever had a server which netstat showed was listening on one or more ports you weren’t expecting, you can use this command to find out which process is listening there:
fuser -vn tcp 4444
Which in this case happens to be owned by JBoss, and not some linux version of a windows worm:)
For more info on fuser, check out the man page, or the simple help below:
Usage: fuser [ -a | -s | -c ] [ -n SPACE ] [ -SIGNAL ] [ -kimuv ] NAME...
[ - ] [ -n SPACE ] [ -SIGNAL ] [ -kimuv ] NAME...
Show which processes use the named files, sockets, or filesystems.
-a display unused files too
-c mounted FS
-f silently ignored (for POSIX compatibility)
-i ask before killing (ignored without -k)
-k kill processes accessing the named file
-l list available signal names
-m show all processes using the named filesystems
-n SPACE search in this name space (file, udp, or tcp)
-s silent operation
-SIGNAL send this signal instead of SIGKILL
-u display user IDs
-v verbose output
-V display version information
-4 search IPv4 sockets only
-6 search IPv6 sockets only
- reset options
udp/tcp names: [local_port][,[rmt_host][,[rmt_port]]]