PHP and FastCGI

I’ve just converted my server and many sites from mod_php under Apache to FastCGI and php-cgi. In theory this should perform better, scale better, and generally BE better. However I’m new to it, so if you find something that is broken on being weird, let me know!

Want an even better job? :)

While the ATG job opportunity at TOMS is pretty sweet, Spark::red is also hiring!  I might be a bit biased but I think Spark::red is an amazing place to work.

First, the jobs:  We’re looking for sys admins, jboss admins, or technical ATG folks (architects, deployment specialists, etc…) to join our sys ops and support/deployment/load testing/performance tuning/troubleshooting teams in Boston (Chelmsford, MA) and Seattle (Redmond, WA).  You can read more about what skills we’re looking for here: Spark::red Oracle ATG Commerce Hosting Careers.  Don’t worry about the which location for which job, we’re looking to fill the roles in either city.

Why would you want to work with us?  Here’s a list:

  • Amazing team of smart, talented, driven, funny people
  • Working with a combination of Fortune 1000s, Internet Retailer 500s, emerging retailers, hip companies, international trendsetters, and Hollywood award shows!
  • Working with happy clients!  We love our clients and they love us.
  • Great salaries – we want to get and keep the best in the business
  • Great benefits
  • Flexibility, power, huge career growth potential!
  • Large scale infrastructure
  • Interesting problems to solve
  • Halo!
  • You get to work with me:)

 

rsync is MUCH faster than scp

Working at a hosting company that specializes in J2EE application hosting, we move EARs around a lot.  Typically I’ve just been using scp since rsync can easily get you into trouble if you’re pushing deltas you’re not super careful.  And if you’re not pushing deltas, and pushing full ears, it makes more sense to use scp rather than rsync over ssh, which just adds extra overhead.  Right?

Apparently wrong.  After doing some real testing here are some surprising numbers.  All of this was done on private Gig-E networks between beefy RHEL 5 linux servers.

318 MB Exploded EAR file:
scp – 85.8 seconds
rsync – 9 seconds

199 MB compressed tgz EAR file:
scp – 14.7 seconds
rsync – 6.3 seconds

Even doing unprimed transfers, rsync is 2-10 times faster than scp.  The moral of this story is for transfers (scripted or manual) rsync can be MUCH faster.  And of course if you can use primed syncs, rsync is MUCH faster again.

Automated ClamAV Virus Scanning

Automating Linux Anti-Virus Using ClamAV and Cron

Thankfully Linux isn’t a platform which has a significant problem with Viruses, however it is always better to be safe than sorry. Luckily ClamAV is an excellent free anti-virus solution for Linux servers. However, at least on RedHat Enterprise 5 (RHEL5) the default install doesn’t offer any automated scanning and alerting. So here is what I’ve done:

The following steps assume you are using RHEL5, but should apply to other Linux distributions as well.

First, you’ll want to install ClamAV:

yum install clamav clamav-db clamd
/etc/init.d/clamd start

On RHEL5 at least this automatically sets up a daily cron job that uses freshclam to update the virus definitions, so that’s good.

Next I recommend removing the test virus files, although you can save this until after you test the rest of the setup:

rm -rf /usr/share/doc/clamav-0.95.3/test/

Now we want to setup our automation. I have a daily cron job that scans the entire server which can take several minutes, and then an hourly cron job that only scans files which were created or modified within the last hour. This should provide rapid notification of any infection without bogging your server down for 5 minutes every hour. The hourly scans run in a couple of seconds.

Each scanning script then checks the scan logs to see if there were any infected files found, and if so immediately sends you a notification e-mail (you could set this address to your mobile phone’s SMS account if you wanted).

The Daily Scan:

emacs /etc/cron.daily/clamscan_daily

Paste in:

#!/bin/bash

# email subject
SUBJECT="VIRUS DETECTED ON `hostname`!!!"
# Email To ?
EMAIL="me@domain.com"
# Log location
LOG=/var/log/clamav/scan.log

check_scan () {

	# Check the last set of results. If there are any "Infected" counts that aren't zero, we have a problem.
	if [ `tail -n 12 ${LOG}  | grep Infected | grep -v 0 | wc -l` != 0 ]
	then
		EMAILMESSAGE=`mktemp /tmp/virus-alert.XXXXX`
		echo "To: ${EMAIL}" >>  ${EMAILMESSAGE}
		echo "From: alert@domain.com" >>  ${EMAILMESSAGE}
		echo "Subject: ${SUBJECT}" >>  ${EMAILMESSAGE}
		echo "Importance: High" >> ${EMAILMESSAGE}
		echo "X-Priority: 1" >> ${EMAILMESSAGE}
		echo "`tail -n 50 ${LOG}`" >> ${EMAILMESSAGE}
		sendmail -t < ${EMAILMESSAGE}
	fi

}

clamscan -r / --exclude-dir=/sys/ --quiet --infected --log=${LOG}

check_scan
chmod +x /etc/cron.daily/clamscan_daily

The Hourly Scan:

emacs /etc/cron.hourly/clamscan_hourly

Paste in:

#!/bin/bash

# email subject
SUBJECT="VIRUS DETECTED ON `hostname`!!!"
# Email To ?
EMAIL="me@domain.com"
# Log location
LOG=/var/log/clamav/scan.log

check_scan () {

	# Check the last set of results. If there are any "Infected" counts that aren't zero, we have a problem.
	if [ `tail -n 12 ${LOG}  | grep Infected | grep -v 0 | wc -l` != 0 ]
	then
		EMAILMESSAGE=`mktemp /tmp/virus-alert.XXXXX`
		echo "To: ${EMAIL}" >>  ${EMAILMESSAGE}
		echo "From: alert@domain.com" >>  ${EMAILMESSAGE}
		echo "Subject: ${SUBJECT}" >>  ${EMAILMESSAGE}
		echo "Importance: High" >> ${EMAILMESSAGE}
		echo "X-Priority: 1" >> ${EMAILMESSAGE}
		echo "`tail -n 50 ${LOG}`" >> ${EMAILMESSAGE}
		sendmail -t < ${EMAILMESSAGE}
	fi

}

find / -not -wholename '/sys/*' -and -not -wholename '/proc/*' -mmin -61 -type f -print0 | xargs -0 -r clamscan --exclude-dir=/proc/ --exclude-dir=/sys/ --quiet --infected --log=${LOG}
check_scan

find / -not -wholename '/sys/*' -and -not -wholename '/proc/*' -cmin -61 -type f -print0 | xargs -0 -r clamscan --exclude-dir=/proc/ --exclude-dir=/sys/ --quiet --infected --log=${LOG}
check_scan

chmod +x /etc/cron.hourly/clamscan_hourly

Protected System

You should now have a well protected system with low impact to system performance and rapid alerting. Anti-Virus is only one piece of protecting a server, but hopefully this makes it easy to implement for everyone.