Jan Kubr

Posts Tagged ‘downtime’

Recent Amazon S3 downtime and what did I do about it

In Uncategorized on February 22, 2008 at 15:21

Amazon S3 was inaccessible for a few hours last week. Although this was the first major problem of the service during the last two years and no data was lost, it wasn’t much fun. I did get a complaint from a Flempo user not being able to access their files (Flempo tasks and documents attachments, profile pictures, and custom logos are stored on S3).

Although it is very unlikely such a thing will happen again any time soon, I did think about an emergency strategy for such situations. It might be surprising, but I didn’t have local copies of the files stored on S3. I mean S3 is redundant and  highly available service, so what would be the point to back them up, right? And no one is doing this (I’ve asked people using S3 for quite busy web applications and they said they have their files only on S3). But here you go, the reason to have such a copy is that you have a place to serve your files from when S3 is not available. And there is a very good second reason: Say a customer accidentally deletes an attachment and asks me to recover it from the backup. Oh, but there’s no backup I would have had to say, too bad.

OK so now there is a backup. All the attachments, profile pictures, and custom logos are synchronized with a Flempo server hard disk folder three times a day. I’m not running out of space there any time soon and even then I can always compress the stuff or something. I’ll figure it out when the time is here. If you’re interested in how I did this technically, you only need to read this tutorial.

Flempo has just become more stable and reliable.