A Content Delivery Network, or CDN, is essentially a system of geographically distributed web servers which serve static content, typically images, video, and other bandwidth intensive files. This serves two purposes: it keeps your servers from having to handle those requests and it serves those files to the end user from a low latency server closer to the user (network-wise). Both of these aspects improve the user’s perception of page and site performance. CDNs can also be extremely useful for things like streaming video or other very high bandwidth uses.

How do CDNs work?

CDNs typically work in one of two ways: for some you have to deploy the files to the CDN manually via FTP or some similar mechanism while others work as a transparent proxy automatically loading the files from the source or origin (your servers) into the CDN as users request them. The latter is preferable as you don’t need to take the CDN into consideration when building your application’s page and referencing media, this also makes handling non-production environments more complex. Also it allows the media to be reloaded from the origin based on cache expiration headers, so you don’t need to do anything special during deployments of new media. However those CDN solutions also seem to be more expensive, so it’s a balance you have to weigh yourself.

Roll Your Own Apache Pseudo CDN

You can also roll a pseudo-CDN yourself using Apache. I call it a pseudo-CDN because unlike Akamai and other large providers you don’t get the advantages of hundreds or thousands of geographically distributed servers. You also don’t get lots of fancy math routing user’s requests to the quickest servers based on location, network congestion, and more. What you do get is transparent proxying and off-loading the request handling from your application servers.

This means you don’t have to do anything special or complex when coding your web application and your JSPs to facilitate the CDN, and it means that your application servers are freed up from having to handle the requests for static media, large and small, which means they have more CPU time available for handling the real dynamic processing of your web application.

Apache makes this simple by way of the mod_disk_cache module. I’d recommend avoiding the mod_mem_cache. Even though it sounds like it would be the preferred caching mechanism, I have had significant problems with mem_cache, and have abandoned it. If you’re using Linux (and you should be) the kernel’s ability to aggressively cache recently accessed files means that when you’re using mod_disk_cache, Apache will cache the files you specify on the local hard drive and will use all available RAM to cache those files in memory for rapid serving. If you plan on using mod_gzip and mod_disk_cache together, please read my post on the issues encountered using them together.

There are two parts to this: first, the time it takes the JSP servlet to generate the HTML response, and secondly the time it takes to transmit that HTML response back to the user’s browser.

Caching content sections

The easiest way to reduce the time it takes for the JSP servlet to generate the response is by reducing the amount of dynamic content on the page. Or more precisely by reducing the amount of real-time or unique individual content on the page.

The Cache droplet is THE most under-utilized ATG droplet.

The Cache droplet caches the rendered output of the contents of the oparam based on a content key (such as category, user gender, logged in/logged out state, etc..) for a configured period of time. This can be very useful for things like navigation menus dynamically built based on the catalog. The catalog won’t change too often, so this dynamically generated menu can be safely cached for hours. Or for some or all of a category or product page, when you set the key to the category id or product id.

Look at your pages and evaluate what parts of the page don’t change that frequently. Even if you can only cache the page or block for five minutes, that can be a huge performance win.

Read on for more….

I’ll talk about things like repository caching and other related topics when I dive into the server side of performance tuning your ATG application in a future post.

Reduce CPU utilization in page rendering

You’re also going to want to look at potential CPU draining elements of your pages. Think about what has to happen to render the page. Don’t use RQLQueryForEach if you have the Repository Id you want, just use the RepositoryLookup droplet. Don’t loop through collections if you don’t have to. Think through it. If you have to do something computationally expensive, don’t do it twice. If you need the same data later just store the first result in a param or JSP variable.

Reduce the size of the HTML

Now that we’ve reduced the CPU time need to render the JSP, we need to reduce the amount of time it takes to transmit to the end user. First, we want to shrink the generated HTML. The smaller the output HTML is, the less time it will take to transfer across the wire to the end user. Also, you want to keep the HTML output under 100k, as Google does not index page content over the first 100k (that’s 100k ungzipped). Also, the smaller the HTML is, the quicker the browser can parse it out into the DOM tree.

Don’t use inline JavaScript or CSS styles in your JSP. Extract them out into separate JavaScript and CSS files. This reduces the size of the output HTML, and increases the amount of content that the browser can cache locally (more on this in my future post on secondary media).

Simplify the html structure as much as you can. Don’t use more tags than you need. I’ve seen an amazing amount of useless nested divs on pages.

Move comments from HTML comments to JSP comments (if you can). Most comments shouldn’t be seen by end users anyhow.

Reduce spaces. This is huge. JSPs (especially ones heavy in ATG droplets) tend to generate a huge amount of extraneous spaces. Part of it is that you generally want to format the JSP files using spaces and tabs to make it easy to see the tag hierarchy structure. It makes development and maintenance much easier. Unfortunately these linefeeds, tabs, and spaces get multiplied by iterative or recursive droplets, and other outputs.

For instance the ATG home page source has 431 lines. 103 are blank. The BestBuy televisions page has 2519 lines and 902 are blank. If you take that BestBuy page, and remove the extra blank lines, comments, and inline JavaScript and CSS, it goes from 98k to 80k. That’s a 20% size reduction, or a 20% transfer speed improvement.

If you’re using JBoss the Tomcat engine has an option to strip out extraneous whitespace. It doesn’t remove all of it, but it helps with multiple blank lines very well. In the tomcat55.sar/conf/web.xml file add the following init param to the <servlet-name>jsp</servlet-name> section:

<init-param>
     <param-name>trimSpaces</param-name>
     <param-value>true</param-value>
</init-param>

This, and other helpful performance configurations are listed in the Tuning Site Performance section of the Installation and Configuration Guide for JBoss Application Server ATG documentation. You should really read through the whole thing, and generally follow it.

Compressing the HTML response using gzip

Now we’ve reduced the size of the HTML the JSP is outputting. We can further shrink the data we need to send from our server cluster to the end user’s browser, hence making the HTML transfer faster, by compressing the HTML using gzip. Most modern browsers understand gzipped content and decompress it on the fly. Some don’t, and we’ll work around those in a moment.

Assuming you’re using JBoss for your application server and Apache for your web server, you actually have two options for gzipping the content: You can compress it from within JBoss Tomcat, or you can compress it as it passes through the Apache web server.

If you want to compress it within JBoss you can configure Tomcat to do that by editing the tomcat55.sar/conf/server.xml file. Find the connector you are using (the one with port=”8080″ for HTTP, or port=”8009″ if you’re using AJP) and add the following properties to that section:

compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml"

That should use gzip compression for output that is html or xml, and is greater than 2k in size.

I prefer to perform the gzip compression in Apache instead of JBoss/Tomcat. Firstly the Apache deflate module is more flexible than the Tomcat option, allowing greater control of compression based on path, mimetype, extension, browser, and more. Second in a three-tier infrastructure, the web servers are typically acting as pass-through proxies, plus handling some caching perhaps (more on this later), but are generally light on CPU utilization, but higher on thread and bandwidth usage. App servers on the other hand are using this CPUs a great deal to actually handle the dynamic application. Gzipping the outgoing content does use some CPU power, so I prefer to have that load be on the web servers who can handle it easily, versus having it impact the application performance.

To make Apache gzip compress the output, you need to enable mod_deflate.

You then need to configure the module using settings similar to this:

# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE no-gzip 

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSIE\s7  !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|swf|flv)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary

You can read more about this configuration, plus some issues that arise when using mod_disk_cache and mod_deflate together here. You can get around this by only gzipping html output from your JSPs, which you won’t be caching anyhow, however doing that in a clean fashion appears to be tricky. The AddOutputFilterByType directive doesn’t work with proxied content, and the mod_filter based approach for that is only built into Apache 2.1 and 2.2, so if you’re on 2.0 it’s no good. When I have a good solution for Apache 2.0 + ATG, I’ll post about it.

By gzipping the HTML we can reduce the BestBuy page further, from 80k to 14k, an additional 80% reduction. We’ve shrunk the data that has to transfer to the end user’s computer by a total of 85% at this point. This means the end user’s browser can parse the HTML dom tree, display the page, and start loading the secondary media (images, css, js, etc…) 85% faster. It also means your request handling thread is now free that much faster (after doing the actual processing of the JSP servlet) and is available to handle a new request.

Summary

So at this point we’ve made the JSP as efficient as possible, used the Cache droplet wherever we could, reduced the size of the output HTML, and gzipped the HTML for faster transmission to the end user. We’ve taken a 98k page and ended up only having to move 14k across the wire.

Next we’ll look at all those secondary media files that your page needs: JavaScript, CSS, images, video, Flash, etc…

The Background: Apache is a web server with many different modules you can load in to enhance it. Two common ones are mod_deflate and mod_cache (or mod_disk_cache).

Mod_deflate compresses content that is sent to the webserver using gzip. It can take 100k of html, css, or javascript, and compress it down to ~10k, before transmitting it to the user’s browser. The browser then uncompresses it, and displays the page. Most web servers (depending on how your site/application is structured anyway) are not CPU limited. Therefore, you can spend some extra CPU doing the compression, and get much faster content delivery times to your users, who are often bandwidth limited. Not only does this make pages load faster for your users, but it also allows request handling threads to complete sooner, letting your web server handle more requests.

Some web browsers are not able to handle gzipped content correctly, therefore it’s important to add in some logic to only send gzipped content to browsers who can handle it. Also, there are different types of files which are already compressed and hence trying to gzip them is a waste of time and resources, such as images, video, etc…

A common configuration may look like this:

<Location />
# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE no-gzip 

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSIE\s7  !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|swf|flv)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

This basically says:

“For files under /”
“Compress them”
“Unless it’s Netscape 4.x, then only compress text/html files”
“Or, if it’s Netscape 4.06-4.08, then don’t compress any files”
“But if it’s IE, don’t compress any files” – NOTE: this is different than the common version you see floating around which turns back on compression for IE. If you are loading content from a Flash swf within IE 6, that content can’t be compressed, even though IE 6 handles it fine. Flash doesn’t for some reason. So this setting is safer. If you aren’t using Flash, feel free to change this.
“but if it’s IE7, undo the no compression settings we made before, activating compression”
“but don’t compress already compressed files like images and video”
“Set the response Vary header to User-Agent so that any upstream caching or proxying won’t cache the wrong version and send a compressed version to a browser which can’t handle it, or an uncompressed version to a browser that should have gotten the compressed file”

Confused yet? :)

Mod_disk_cache allows you to specify various files to be cached on the web server and lets you set a cache expiration time, etc… It’s of great value when those files are being served out of a web application, and not coming from the local disk. For instance if Apache is serving files from an ATG instance, mod_disk_cache, lets you have the web server cache images, css, js, videos, etc… from your WAR. There’s also a memory based cache, mod_mem_cache, but it’s more trouble than it’s worth, and you can trust the linux kernel to cache recently accessed files in memory anyhow.

Got it?

So this is where it gets tricky.

If a response has a Vary header set, mod_disk_cache will cache a different version of that file for each value of the Header that Vary references.

So for a file compressed as above, there will be a different version cached for each User-Agent. In theory this will mean that browsers which support gzip compressed content, will get the compressed content, and browsers which don’t, will get the uncompressed version.

In practice, due to the amazing tiny variations of the full User-Agent string, you end up with thousands of copies of the same file in your cache. On a disk cache only a few hours old, there were over 4,400 cached copies of the same javascript file. Each with a slightly different User Agent string, even though there were less than 10 actual browser types represented.

This is a problem for several reasons: Firstly, you end up using far more disk space than you really need. Secondly, you negate the kernel’s in-memory file caching, since those 4,000+ version of the single file are being accessed, it won’t be able to simply keep the two different files (compressed and uncompressed) in memory. Thirdly, you make cleaning out the cache much slower, since you have to delete these thousands of extra files and their containing directories.

The Solution: I’m not sure… Any ideas?

The Google crawl bot does not support cookies, therefore JBoss uses the jsessionid query parameter in order to maintain a session state without cookies. These query parameters can impact your Google rank and indexing efficiency as the same page can be indexed multiple times with different session ids, and dilute your ranking. Also, it leads to ugly links.

If you want to still be able to support non-cookie using users, but would like Google to see cleaner links, you can use Apache’s mod_rewrite to modify the links for the Google bot only, leaving the normal functionality available to the rest of your users.

Assuming you have mod_rewrite enabled in your Apache instance, use this configuration in your apache config:

	# This should strip out jsessionids from google
	RewriteCond %{HTTP_USER_AGENT} (googlebot) [NC]
	ReWriteRule ^(.*);jsessionid=[A-Za-z0-9]+(.*)$ $1$2 [L,R=301]

This rule says for request where the user agent contains “googlebot” (with case insensitive matching), rewrite the URL without the jsessionid. It seems to work nicely.

When using RichFaces with Seam, things work just fine on my local development JBoss instance. But when I deploy the same EAR file up to my production JBoss instance, which is sitting behind an Apache proxy, everything works EXCEPT the rich/ajax stuff.

The issue was that the JavaScript located here: ContextRoot/a4j_3_1_4.GAorg.ajax4jsf.javascript.AjaxScript

would not load.

My Apache proxy was configured like this:

	ProxyPass /10MinuteMail balancer://mycluster/10MinuteMail/
	ProxyPass /10MinuteMail/* balancer://mycluster/10MinuteMail/
	ProxyPassReverse /10MinuteMail http://127.0.0.1:8080/10MinuteMail

With mycluster defined like this:

        
                AddDefaultCharset off
                Order deny,allow
                Allow from all

		BalancerMember http://127.0.0.1:8080
                #Allow from .example.com
        

Again, this configuration worked fine for everything EXCEPT that RichFaces JavaScript.

Since I am only using one node for 10MinuteMail, there is no real need for a load balancer configuration, so I replaced the configuration with this:

	ProxyPass /10MinuteMail http://127.0.0.1:8080/10MinuteMail
	ProxyPass /10MinuteMail/ http://127.0.0.1:8080/10MinuteMail/
        ProxyPassReverse /10MinuteMail/ http://127.0.0.1:8080/10MinuteMail/

Which works, and fixed the RichFaces reference.

So there’s your solution. However I have no idea what the actual root cause is.

I used Apache 2.2 and mod_proxy_http to wrap all of the services in the Apache running on port 80.

Any Apache 2.2 installation should come with mod_proxy and mod_proxy_http. You may need to activate them with something like this:
LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so
LoadModule proxy_http_module /usr/lib/apache2/modules/mod_proxy_http.so

then set a basic configuration for security:
ProxyRequests Off
ProxyPreserveHost On

<Proxy *>
Order deny,allow
Allow from all
</Proxy>

It’s best to ensure all the applications you are trying to wrap are listening at context roots, not the base URL/port. So you want hudson on :9000/hudson/ or whatever your setup has (for instance: java -jar hudson.war –httpPost=9000 –prefix=/hudson ). Same for Jira and everything else.

Then setup your proxy mappings in your apache configs.

ProxyPass /jira http://localhost:8080/jira
ProxyPassReverse /jira http://localhost:8080/jira
<Location /jira>
Order allow,deny
Allow from all
</Location>

Repeat for all of your applications. Then just restart apache to pickup all the changes:

apachectl restart

So now we have Jira at host/jira, Confluence at host/confluence, Hudson at host/hudson, and so on. Pretty, easy to book mark, easy to cross link. It just looks more professional.

	<welcome-file-list>
		<welcome-file>index.html</welcome-file>
	</welcome-file-list>

Everything worked great, when I was hitting the tomcat instance in JBoss. However, as soon as I deployed to my production server, and Apache2 was handling the end user connections and talking to JBoss via ajp13, I kept getting directory listings, instead of the index page. If I typed “index.html” into the browser, it worked fine, but just going to the root of the web-app context root always delivered a directory listing. Not at all what I wanted.

By putting an empty index.html file in my web-app, things worked. I’m assuming that Apache checks to see if the resource exists, and if not, does the whole directory listing thing, ignoring the welcome-file mapping in the web-app. I’m only guessing here, but that seems to be what happened. If so, I’d call this a flaw in the Apache to JBoss request negotiation. Once Apache hands off the request for the index.html, after it’s validated the resource exists, then Seam leaps in and everything works as expected. For whatever reason, it’s only an issue when a request is issued for a directory “/”.

Anyhow, I don’t have all the answers, but I do have a solution for this issue if it bites you. And I wasn’t able to find it on google myself, so hopefully this will be helpful.