As you know if you've been following my blog, 10MinuteMail starting dying with OutOfMemory errors seemly randomly. Sometimes it would happen after 2 days of uptime, sometimes after a week. I ran load tests and profilers but could not reproduce the issue. On the upside I can say that the 10 Minute Mail application can handle a TON of load:)

I made some code changes and optimizations on Friday, and since then, no problems. Obviously that's too early to tell for sure, but I found something last night, which might have explained the previous issues, and it is the root cause, then my latest fixes should have solved it.

I THINK the problem is..... porn. I was watching the inbox size for 10MinuteMail last night, which usually goes between 0k and 2MB in size, and it leapt from 540k to 45MB in a couple of seconds, and then went to 86MB a second later. So first I need to back up a little.

When 10MinuteMail was crashing with OutOfMemory errors, I would wake up and find the server useless due to the memory errors, and inbox would have swollen to a gigabyte or more. I assume this was a cause and effect situation, namely that the server would tank, and then the inbox would pile up afterward.

What I found was that some people used their 10MinuteMail e-mail addresses to sign up for things like the "asianscans" google group, and other lists that periodically would deliver e-mails filled with large (2 MB) images as attachments. 15 10 MB e-mails is 150 MB in your inbox in a few seconds. The old code would parse and load all of the attachments (even though I never completed the attachment download code). Now with 150 MB of attachments being read through a 8k byte array, and into these large in memory objects, that can take a little while. Get a big enough inbox, and it becomes impossible to clean it up as you just keep falling further and further behind. Eventually you run out of memory trying to load up a 800 MB inbox.

So instead of the server running out of memory, tanking, and then letting the inbox build up, I think that what was really happening is that the inbox filled up with huge attachements very quickly, the server fell further and further behind trying to process them, and eventually ran out of memory, tanked, and then of course the inbox just got worse.

The latest code ignores attachments, and took about 4 seconds to process the 80+ MB inbox I saw last night.

I'm knocking on wood that this was in fact the root cause to the earlier problems, and that the future should be smooth sailing:)

As a side note, last month had over 2.1 million page hits to 10MinuteMail.com! Thanks guys! I'm glad people find it useful!

This entry was posted on Monday, November 5th, 2007 at 4:55 pm and is filed under 10MinuteMail. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.