Thankfully Linux isn’t a platform which has a significant problem with Viruses, however it is always better to be safe than sorry. Luckily ClamAV is an excellent free anti-virus solution for Linux servers. However, at least on RedHat Enterprise 5 (RHEL5) the default install doesn’t offer any automated scanning and alerting. So here is what I’ve done:

The following steps assume you are using RHEL5, but should apply to other Linux distributions as well.

First, you’ll want to install ClamAV:

yum install clamav clamav-db clamd
/etc/init.d/clamd start

On RHEL5 at least this automatically sets up a daily cron job that uses freshclam to update the virus definitions, so that’s good.

Next I recommend removing the test virus files, although you can save this until after you test the rest of the setup:

rm -rf /usr/share/doc/clamav-0.95.3/test/

Now we want to setup our automation. I have a daily cron job that scans the entire server which can take several minutes, and then an hourly cron job that only scans files which were created or modified within the last hour. This should provide rapid notification of any infection without bogging your server down for 5 minutes every hour. The hourly scans run in a couple of seconds.

Each scanning script then checks the scan logs to see if there were any infected files found, and if so immediately sends you a notification e-mail (you could set this address to your mobile phone’s SMS account if you wanted).

The Daily Scan:

emacs /etc/cron.daily/clamscan_daily

Paste in:

#!/bin/bash

# email subject
SUBJECT="VIRUS DETECTED ON `hostname`!!!"
# Email To ?
EMAIL="me@domain.com"
# Log location
LOG=/var/log/clamav/scan.log

check_scan () {

	# Check the last set of results. If there are any "Infected" counts that aren't zero, we have a problem.
	if [ `tail -n 12 ${LOG}  | grep Infected | grep -v 0 | wc -l` != 0 ]
	then
		EMAILMESSAGE=`mktemp /tmp/virus-alert.XXXXX`
		echo "To: ${EMAIL}" >>  ${EMAILMESSAGE}
		echo "From: alert@domain.com" >>  ${EMAILMESSAGE}
		echo "Subject: ${SUBJECT}" >>  ${EMAILMESSAGE}
		echo "Importance: High" >> ${EMAILMESSAGE}
		echo "X-Priority: 1" >> ${EMAILMESSAGE}
		echo "`tail -n 50 ${LOG}`" >> ${EMAILMESSAGE}
		sendmail -t < ${EMAILMESSAGE}
	fi

}

clamscan -r / --exclude-dir=/sys/ --quiet --infected --log=${LOG}

check_scan

chmod +x /etc/cron.daily/clamscan_daily

The Hourly Scan:

emacs /etc/cron.hourly/clamscan_hourly

Paste in:

#!/bin/bash

# email subject
SUBJECT="VIRUS DETECTED ON `hostname`!!!"
# Email To ?
EMAIL="me@domain.com"
# Log location
LOG=/var/log/clamav/scan.log

check_scan () {

	# Check the last set of results. If there are any "Infected" counts that aren't zero, we have a problem.
	if [ `tail -n 12 ${LOG}  | grep Infected | grep -v 0 | wc -l` != 0 ]
	then
		EMAILMESSAGE=`mktemp /tmp/virus-alert.XXXXX`
		echo "To: ${EMAIL}" >>  ${EMAILMESSAGE}
		echo "From: alert@domain.com" >>  ${EMAILMESSAGE}
		echo "Subject: ${SUBJECT}" >>  ${EMAILMESSAGE}
		echo "Importance: High" >> ${EMAILMESSAGE}
		echo "X-Priority: 1" >> ${EMAILMESSAGE}
		echo "`tail -n 50 ${LOG}`" >> ${EMAILMESSAGE}
		sendmail -t < ${EMAILMESSAGE}
	fi

}

find / -not -wholename '/sys/*' -and -not -wholename '/proc/*' -mmin -61 -type f -print0 | xargs -0 -r clamscan --exclude-dir=/proc/ --exclude-dir=/sys/ --quiet --infected --log=${LOG}
check_scan

find / -not -wholename '/sys/*' -and -not -wholename '/proc/*' -cmin -61 -type f -print0 | xargs -0 -r clamscan --exclude-dir=/proc/ --exclude-dir=/sys/ --quiet --infected --log=${LOG}
check_scan

chmod +x /etc/cron.hourly/clamscan_hourly

Protected System

You should now have a well protected system with low impact to system performance and rapid alerting. Anti-Virus is only one piece of protecting a server, but hopefully this makes it easy to implement for everyone.

On the receiving side, you have things like blacklists, spamassassin, bayesian filtering, and lots more. I’ll probably cover this side of things in greater depth in another post.

On the sending side, first and foremost, you have to ensure your server is not acting as an open relay, and allowing spam to be sent through it. After that’s done, you want to be sure that e-mail you send isn’t flagged as spam by people receiving it. And, being a good e-mail citizen, you you want to support the anti-spam standards that are out there.

There are four primary standards for verifying senders and servers.

Sender Policy Framework (SPF) – from their FAQ:

Sender Policy Framework (SPF) is an attempt to control forged e-mail. SPF is not directly about stopping spam – junk email. It is about giving domain owners a way to say which mail sources are legitimate for their domain and which ones aren’t. While not all spam is forged, virtually all forgeries are spam. SPF is not anti-spam in the same way that flour is not food: it is part of the solution.

SenderId – a Microsoft technology which is very similar to SPF:

The Sender ID framework, developed jointly by Microsoft and industry partners, addresses a key part of the spam problem: the difficulty of verifying a sender’s identity.

DomainKeys – from Wikipedia:

DomainKeys is an e-mail authentication system designed to verify the DNS domain of an e-mail sender and the message integrity.

DKIM – an evolved form of DomainKeys, from Wikipedia:

DKIM uses public-key cryptography to allow the sender to electronically sign legitimate emails in a way that can be verified by recipients. Prominent email service providers implementing DKIM (or its slightly different predecessor, DomainKeys) include Yahoo and Gmail. Any mail from these domains should carry a DKIM signature, and if the recipient knows this, they can discard mail that hasn’t been signed, or that has an invalid signature.

SenderId is primarily used by Microsoft mail services like Hotmail/MSN, while DomainKeys and DKIM are primarily used by Yahoo. SPF is used by many mail services.

I’m going to walk you through setting up these anti-spam technologies. I will be setting them up for my domain, digitalsanctuary.com, and using my mail server, which is postfix running on Debian. Your setup and requirements may vary.

Let’s start with SPF. First, you need generate an SPF record for your domain(s), that essentially lists what mail servers are allowed to send mail from that domain. The easiest way to set up the SPF record is with a tool, like this online wizard.

First, I type in my domain name, and press the “Begin” button. The setup for digitalsanctuary.com is actually very simple. I only run one mail server. It runs on the same IP as my DNS A record for digitalsanctuary.com, and it’s the MX record for the domain as well. I don’t send e-mail through other mail servers. So my SPF record looks like this:

“v=spf1 a mx ~all”

Which says that mail from @digitalsanctuary.com can only come from the A or MX record IP addresses for the digitalsanctuary.com domain. Using the wizard you can setup your own domain SPF tuned to your needs. Once you have the record, you need to add it as a TXT entry in your DNS. Instructions are on the lower part of the SPF wizard I pointed you to. I use SoftLayer’s DNS servers, and on their web admin, I had to enter a “@” in the NAME field for the record for it to work correctly. Once that’s done, you can verify that it shows up correctly using a DNS query:

Motoko:~ modoc$ dig digitalsanctuary.com TXT

; <<>> DiG 9.4.2-P2 <<>> digitalsanctuary.com TXT
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36859
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;digitalsanctuary.com. IN TXT

;; ANSWER SECTION:
digitalsanctuary.com. 796 IN TXT "v=spf1 a mx -all"

;; Query time: 8 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sun Feb 22 22:40:48 2009
;; MSG SIZE rcvd: 67

If you see your SPF record showing up in the Answer Section, then it should be working.

You can further test this by sending e-mail to check-auth@verifier.port25.com which will send you a reply with a detailed breakdown of various spam checks.

The reply at this point, should look like this:

Summary of Results
==========================================================
SPF check: pass
DomainKeys check: neutral
DKIM check: neutral
Sender-ID check: pass
SpamAssassin check: ham

There’s the SPF check pass, and wait! There’s also a Sender-ID check pass. How did that happen? SenderId is very similar to SPF. You can read about the differences here (be aware that the openspf folks have a very specific view point on the topic). Essentially they use the same DNS record syntax, and both systems are used to check the validity of the sending server based on who’s sending it, however they match who’s sending the mail at different layers of the SMTP stack. SPF uses the “MAIL FROM” data from the “envelope” layer which is passed as a header value during the SMTP communication. SenderId instead uses the “best” address from the list of address headers in the message itself. If you aren’t a SMTP expert, that sounds pretty confusing. Luckily for most of you, the difference really doesn’t matter. Your MAIL FROM envelope value, and the From address header, will typically be the same. If your outgoing mail system is more complex than this, then you’re likely enough of an SMTP expert to understand the difference, and setup a separate SenderId DNS record. For the rest of us, we’ve just killed two birds with one stone.

At this point you’ll probably want to register your SPF record as your SenderId record with Microsoft. You can do that here: https://support.msn.com/eform.aspx?productKey=senderid&page=support_senderid_options_form_byemail&ct=eformts&scrx=1.

Next, let’s tackle DKIM. We get another two for one deal here, since DKIM is the evolution of DomainKeys. I recommend using this guide to install DKIMProxy. You need to install a number of Perl modules, listed on that page, and I suggest using CPAN to do so. While CPAN isn’t perfect, it does handle a lot of the dependancies for you.

You basically need to install the following Perl modules:

• Crypt::OpenSSL::RSA
• Digest::SHA
• Digest::SHA1
• Error
• Mail::Address
• MIME::Base64
• Net::DNS
• Net::Server
• Mail::DKIM

Then you’ll need to download the DKIMProxy software, I used version 1.1. Follow the build instructions in the guide I linked to above. Once it’s built, you continue to follow the guide and create your public and private keys. Then add the public key into the DNS for your domain. Then, instead of passing everything in as arguments to dkimproxy.out, I go into this directory (assuming you used the install location in the guide): /usr/local/dkimproxy/etc/ and copy dkimproxy_out.conf.example to dkimproxy_out.conf. Then edit that file and setup your keyfile location, domain, and other changes if needed.

Assuming you copied over the example control script into /etc/init.d/, you can test your setup and start dkimproxy, by typing: /etc/init.d/dkimproxy start

Assuming it starts up cleanly, you’re ready to configure Postfix to use dkimproxy to sign outgoing messages. You may currently be sending mail to your mail sever on port 25, but we can’t simply sign all messages delivered to port 25, since that’s where all your incoming mail arrives as well. So we’re going to setup postfix to listen of port 587 as well. When we send outgoing e-mail, it will go to port 587, and then will be run through the dkimproxy filter to be signed, before being sent out.

Use this guide: Setting up the outbound proxy with Postfix to set things up. Once you’ve configured postfix, reload the postfix configuration.

Now you’ll want to ensure that your mail client is pointed to port 587 instead of port 25.

Next, send another test e-mail to check-auth@verifier.port25.com. Your reply should look like this:

Summary of Results
==========================================================
SPF check: pass
DomainKeys check: pass
DKIM check: pass
Sender-ID check: pass
SpamAssassin check: ham

If it looks like that, then you’re all set! You’ve successfully implemented the four major anti-spam mail sending standards. Congratulations!!

Not the best way to start the day.

Turns out an rsync backup job between two servers had gone nuts and was spinning them up through the roof. A couple

sudo killall rsync

commands, and everything started settling down.

What’s important to note is that while the CPU was pegged, and the load average was over 200, I was still able ssh in, run top, ps, netstat, and navigate around looking at log files, with very little delay. It was just a little slow, but no more than a second wait for anything.

Also, all of my apps which used MySQL as their backend were all dead, with “database refused connection” errors. All of my apps which used PostgreSQL as their backend were a little slow, but were all still up and functioning without errors.

So, Debian and PostgreSQL for the win!

This is bad on a lot of fronts.

I use two simple iptables rules to block any IP address who has made more than 3 ssh connections or attempted connections within the past 3 minutes. So your would-be brute force attacker, gets three tries, and then is locked out for a minimum of three minutes. However, since 99% of the attacks are run by an automated bot, it will either: give up after the connection is refused multiple times, or it will keep hammering away on the closed door, which keeps the running count of attempted connections in the past 3 minutes over 3, keeping the door closed.

First:

iptables -I INPUT -i eth1 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name DEFAULT --rsource

Then run:

iptables -I INPUT -i eth1 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 180 --hitcount 4 --name DEFAULT --rsource -j DROP

I’d also recommend using the script in my post on blocking IP addresses using iptables to deal with any persistent folks, or people poking too hard on your web site, or other services.

First, I’ll assume you are already using iptables. If you need help setting that up, use Google, Debian comes with it out of the box.

I have a small script called “block” which looks like this:

#!/bin/bash
sudo iptables -I INPUT -s $1 -j DROP
sudo bash -c "iptables-save > /etc/network/iptables.save"

Whenever I find a “bad” IP in my logs or notifications, I just run:

block bad.ip.add.18

Substituting the bad ip for that nonesense above. This adds it to the list of IP address which iptables will simply drop any incoming packets from, and saves the in memory iptables configuration, so that it is preserved through reboots.

Then in your /etc/network/interfaces file, just add this at the bottom:

post-up iptables-restore /etc/network/iptables.save