• We’ve added several new clients (including a well known member of the Fortune 1000!)
• We’ve added 101 new dedicated servers and over 20 cloud computing instances
• We’ve opened an office in Boston
• We’ve earned our PCI Level 1 MSP Certification
• We’ve hired several employees, a large number of contractors, and even picked up an intern!
• We’ve gone from 3 data centers in the USA to 13 data centers and 16 points-of-presence including several international facilities
• While our competitors have raised prices, we’ve managed to keep prices the same or actually reduce them in many places.  All this while providing newer more powerful hardware across the board
• We’ve introduced a new Oracle ATG Commerce Standard Hosting Package which provides excellent performance, stability, and security, for a very affordable package price
• Served well over 200 TB of content
• Partnered with JBoss/Redhat, Oracle, Akamai, Keynote, Knowledge Path, and many other industry leaders in order to provide the absolute best hosting, support, and related services
• More more!

I just discovered that you can setup OmniFocus to pull from your iCloud synced Reminders. That means if you use OmniFocus, as I do, that you can create new ToDo items via Siri:

Siri, remind me to call Bob about Oracle tomorrow…

Siri, remind me to order shampoo…

and so on….  They show up in your OmniFocus Inbox automatically.  It’s an amazing tool for ubiquitous capture.  As soon as anything pops into my head, I can use Siri to get it out of my head and into OmniFocus.

I’ve also started using Siri to set my morning alarm, timers for cooking, timers for time-boxing tasks, etc…  I’m still learning all the ways to use Siri, but every new thing I find is great!

7) Port binding

Say What?

8) Concurrency

Again, throwing Java and PHP under the bus, they tie back to “factor” number six where everything is stateless.  They also ban PID files, daemon processes, etc…  What they’re proposing is so far outside anything I’ve done or seen anyone do in a real world app I don’t even know how to respond.

I’m not sure how spinning up large numbers of processes on the same hardware works with the idea of having to vend all your own network servers.  How do you handle port binding, port conflict, and port discovery across a cluster?

9) Disposability

This ties back to “factor”s six and eight, stateless standalone processes which can spawn and die quickly without any impact to the overall system.  This forces you to avoid caching, anything complex, and so on.  I’d much rather have big powerful instances with caches, stateful data, a mature container, solid network systems, and all that.

Building a system around stateless job queues and assuming that instances can and will die often enough to warrant building a system around that seems like an odd place to spend your effort.  Big JBoss instances are VERY stable.  They run and they run and they run for months at a time without issue.  There’s no need to build complex queues and job management system, relying heavily on your external systems (like your database, message queues, redis, etc…) to handle all your state because you’re scared to do it yourself.  I’m not sure why relying on MySQL or RabbitMQ for long running state management is any better than just having your app do it.  Let parsing, network overhead, and re-work.

10) Dev/prod parity

This “factor” really covers two separate points: Firstly it pushes frequent production deployments every few hours, made by the code developers (not an ops team).  Secondly it requires that developer instance use identical backing services for everything as production, use the same operating system, etc…

The first point is problematic to me on a few fronts.  The frequency of releases implies no real QA process, beyond perhaps some automated unit tests.  While waterfall-esque QA isn’t a solve-all approach, it DOES generally reduce the number of issues that go into production.  I understand the agile approach makes this trade-off happily against the increased ease of getting fixes deployed out quickly.  However for large players production bugs, production downtime, production impact are VERY costly.  Many large eCommerce sites will lose hundreds of thousands of dollars per hour of downtime (downtime being defined here as anything that prevents users from easily placing orders – not just a site availability outage).  Many sites are used for other critical system or contain large amounts of our privacy information.  Minor code bugs can have catastrophic impacts if they are pushed to production without testing.  Developers are awful testers.  I say this as a developer.  We think about changes based on the intended change and may test that successfully.  But we won’t test the rest of the entire site recession suite, and hence we miss unintended impacts of our code change.  We also tend not to have the meticulous attention to detail that a good QA person does.  A developer unit testing a change is VERY different from a good QA person testing the same change or performing a full regression test suite.

It’s also not PCI compliant to have code developers pushing code changes into production (the idea is that it’s too easy for a malicious developer to push a malicious change out to prod without anyone knowing).

The second point I agree with, to a degree, but not entirely.  I run RHEL 5 in production, but am not going to force any developers to run that as their laptop OS as it’s a pretty terrible desktop system.  OS X is my preferred OS and works pretty well for building things for deploying on Linux systems, including RHEL.  I can agree with wanting things to be as similar as is easily and legally possible.  If you can run the same DB software locally as is used in production, great!  But you probably can’t use full production data locally (either due to size, privacy concerns or other security factors) therefore there will always be differences.  Some external systems you can tie into the same as production, others you’ll need to tie into testing/staging versions which differ from their production counterparts more often than you might expect (PayPal is a great example of this).  Others may not be accessible at all, and will need to be ignored or stubbed out for local work.

11) Logs

They say that all apps should just write to stdout in an unbuffered stream for any and all logging.  Can you imagine if all the apps you install did that?  If redis server just dumped to stdout instead of writing it’s own log file?  Ugh.  What a mess.  There’s also often very reasons that different sets of log output should be handled, directed, and stored differently.  I don’t want to put the onus on everyone who runs the app (developers or otherwise) to properly configure something like syslog-ng and ensure the app’s output is directed there, etc…  Way overly complex.  Also unbuffered writes to stdout carries with it a massive CPU (and occasionally IO) performance penalty.  High volume logs, especially in production, especially under heavy load or when things are going wrong, can generate a TON of lines per second.  Buffering these in memory in the app and writing in batches to file is much more efficient than unbuffered dumping to stdout with the assumption that stdout is being re-routed through another system entirely (syslog-ng, maybe a remote spunk server, maybe other things), parsed, and then written out to a file or db somewhere else.

12) Admin processes

Nothing really to say here.

Summary:

Despite the initial claim that these rules apply to any programming language I think it’s pretty clearly not the case.  These rules are a terrible fit for PHP, Java (J2EE, Seam, ATG), and probably many other languages and frameworks.  Maybe they work great with RoR, I don’t know.

Some of the points here are sound.  Others I think are misguided even if you’re not using a language or framework that pushes you away from the given rule by its very nature.  Some of them fall down on things like PCI compliance, performance, or other real world aspects of running a high performance, high availability, complex production web app.

Not a bad read, but certainly not even close to guidelines I’d recommend anyone take to heart.

The goals are noble: to build scalable, portable, easily deployable web applications, while being language and framework agnostic. Unfortunately I found several of the points to run against my experience and personal opinion, so I wanted to go over the 12 tenets they recommend and offer my thoughts on each.  Please keep in mind that much of my experience is on ATG, J2EE, and large complex applications.

1) Codebase

Overall this is pretty normal, but in some cases I don’t completely agree with the idea that shared code can only be handled through independently built-packaged libraries included via some sort of dependency manager.  I haven’t really used a good J2EE level dependency manager, perhaps Maven makes this easy?

2) Dependancies

I have several issues with this “factor”.  Firstly, Java doesn’t have a packaging system for distributing libraries and common app bundles.  Next, I’m not sure what a “dependency declaration manifest” actually DOES at the app level or what “dependency isolation tools” I should be using.  ATG has module dependencies defined within the MANIFEST.MF files which is great, but isn’t something you can use to handle extra-ear dependencies (JBoss, JDBC drivers, native apps, etc…).

Further they expect the app to provide/package ALL system tools it may need.  The examples they use are ImageMagick and curl.  This is crazy for many reasons: first, many of these tools are different on each platform, packaging/building/installing these tools on each different platform is a massive effort and not something easily bundled into your app, nor should your Java/Ruby/PHP developers have to deal with multi-platform C++ build issues, secondly most platforms have their own package installation and dependency management system (yum, apt-get, etc…) which ensure supported platform specific versions, which not only will work, but also may be required for support for Enterprise Linux distress for instance.

Also: where does it end?  It’s turtles all the way down.  ImageMagick on RHEL 5, installed via  yum, has 148 dependencies.  Curl has 310.  A J2EE application, in EAR form, may depend on: JBoss 5.1 EAP, JDK 1.6.0_27 (the correct version for the platform and 32 or 64 bit), JDK extended crypto policy files, Oracle JDBC driver jar file, various JBoss server config overrides, curl, NFS mounts, ImageMagick, mplayer, Apache front end configured with SSL certs, proxy configs, etc…  It’s insane to say all that needs to be handled within your EAR deployment mechanism.

3) Config

This starts off making sense: keep your configs separate from your code.  Don’t hardcode configs into your source code.  But then they say you should use environment variables to handle all of your configs….  They also say you shouldn’t “group” config values based on the environment (dev, stage, prod, etc…).  Both ATG and JBoss Seam have very sensible configuration setups with config files, environment level groupings for environment specific overrides, etc…  It works great.  Much better than trying to deal with env variables…

4) Backing Services

No argument here.  Nothing that shouldn’t already be common sense either.

5) Build, release, run

No argument here.  Nothing that shouldn’t already be common sense either.

6) Processes

Heading into the deep end again:)  They push for stateless, share-nothing processes that rely on a database (or similar service) for all persistent or shared data/state.  That’s okay for very simple or very low traffic applications, but in my experience more complex and/or higher traffic web apps benefit hugely from having sticky sessions with local RAM based session state.
A user’s session on a serious eCommerce application contains a lot of data with somewhat complex relationships: basic user profile data (username, name, gender), extended user profile data (shopping preferences, shipping address(es), shopping history), current cart data (which could be as simple as a list of SKUs, or could be full copies of SKU data to avoid catalog data changes from impacting cart items, with cloned pricing to avoid pricing changes impacting the cart, coupons, promotions, shipping calculations, shipping methods, tax lookup data, and more), session history data (as simple as breadcrumbs or as complex as browsing history data being used to calculate recommendations, promotions, cross-sell, and up-sell), and checkout flow state and related data.

They seem to recommend that at the end of every page request, this complex relationship of Objects (Java components or beans or whatever you use) should be serialized or mapped back to database backing tables (manually or via an ORM), persisted in a series of inserts and updates (the most expensive type of operations) to a remote database across the network, and those objects should be cleaned up/purged/garbage collected.  Then at the start of every page request, a cookie identifier should be used to identify all of that data again, probably with multiple separate sequential lookups, select it out of the database, parse it back into new Objects you’ve created again, and bound together based on their relationships, before you can service the request.

This is CPU, memory, and network intensive.  You’d be creating massive additional load on your database and dramatically increasing your app server CPU utilization and increasing the request response time.  It’s MUCH easier to build session state data once, keep it around for the life of the session, and just use intelligent sticky session routing on your load balancer/proxy.

It’s late, so I’ll address the last 6 tenets in another post!  Thanks for reading!

If you were on that list, I apologize! If you’re planning a similar migration, be careful!

This guy is VERY smart! Well worth the read. You can also see some follow-up discussion involving Patrick here: http://news.ycombinator.com/item?id=3214397

You will be missed.