Recent Amazon S3 downtime and what did I do about it

Amazon S3 was inaccessible for a few hours last week. Although this was the first major problem of the service during the last two years and no data was lost, it wasn’t much fun. I did get a complaint from a Flempo user not being able to access their files (Flempo tasks and documents attachments, profile pictures, and custom logos are stored on S3).

Although it is very unlikely such a thing will happen again any time soon, I did think about an emergency strategy for such situations. It might be surprising, but I didn’t have local copies of the files stored on S3. I mean S3 is redundant and  highly available service, so what would be the point to back them up, right? And no one is doing this (I’ve asked people using S3 for quite busy web applications and they said they have their files only on S3). But here you go, the reason to have such a copy is that you have a place to serve your files from when S3 is not available. And there is a very good second reason: Say a customer accidentally deletes an attachment and asks me to recover it from the backup. Oh, but there’s no backup I would have had to say, too bad.

OK so now there is a backup. All the attachments, profile pictures, and custom logos are synchronized with a Flempo server hard disk folder three times a day. I’m not running out of space there any time soon and even then I can always compress the stuff or something. I’ll figure it out when the time is here. If you’re interested in how I did this technically, you only need to read this tutorial.

Flempo has just become more stable and reliable.

On Amazon S3 and competitive advantage

As I already reported, Flempo is using Amazon S3 to store task and document attachments. Yesterday I also set up regular database backups using s3sync. That means the database is backed up twice a day using an encrypted channel to a highly reliable storage.

Make the attachments be stored on S3 was very easy. All I needed to do was to change my usage of the file_column plugin to the (newer) attachment_fu one. That took me a couple of hours, but then all you need to do is change the :storage option from :file_system to :s3. So basically if you use attachment_fu already, it would take you 30 minutes to make the switch to S3, 10 of which you would need to sign up for the service..

Attachment_fu does not allow you to easily create attachments from a file you already have opened. But thanks to Craig Ambrose who wrote an article on migrating from file_column to attachment_fu, this wasn’t a big hassle either. Thanks to his code I was able to migrate the old attachments as well as make e-mails with attachments working. (If you send an e-mail with attachments to Flempo, it will not only create/comment a task, but also automatically add any attachments you sent with the e-mail.)

You can find many tutorials on how to use s3sync to do the backups as well. All that is very easy. Too easy actually. Now note that S3 is cheap. Very cheap. This all got me thinking again about competitive advantage and the difficulties to be in the web applications business. Recently I’ve read an article by Joel Spolsky called When there’s muck, there’s brass where he says:

“The trouble is, the market pays for solutions to gnarly problems, not solutions to easy problems.”

Now let’s see what the situation with web applications is: hardware (including highly reliable storage) is cheap, infrastructure software is free and so powerful that you can do very complicated things amazingly quickly. You can outsource so much you need just a few people to serve millions of customers.

That’s great you say. And it is. The “problem” is that it is very easy for anyone. These tools are available to anyone with an Internet connection. I need to ask myself the obvious question: If it’s so easy for anyone to make a webapp, why would people pay for mine? And how long does it take before the competition (meaning one guy anywhere in the world) develops something better than I have?

Somewhere else I read you need to increase the gap between you and your competition. You need to make steps that are hard for your competition, but bring value to the customers.

The thing is the technical stuff is not of one the big steps anymore. It used to be hard to distribute your software. Not anymore. It used to be hard to implement a web application (not to design it though!). Although there is a big difference between some messy PHP code and a clean tested code within a web framework, it is not that hard to do it right anymore. It used to be very difficult (read: expensive) for small Internet businesses to provide their customers with highly reliable architecture. With Amazon’s web services (don’t forget EC2 and the new baby SimpleDB!) this is not the case anymore, either.

I mean there are still only few people who can code and set this all up and make it work. Especially outside the US I’d say. But I do think it is not where the hard part is (or will be soon) anymore. It is in “what is my app about,” “what features shall I put in and what to leave out,” in the user interface design, community management and so on. In short, in the non-geeky stuff.

Flempo is backed by Amazon S3 now

OK all the attachments in Flempo are served from the Amazon S3 web service now. The migration of the old attachments took a bit longer than expected and thus no polishing I was promising in the post about Open Web Awards, but the contest doesn’t bring any traffic to the application anyway, so no big deal I guess; AS3 was far more important. More information about AS3 coming.

How does Amazon S3 make you free

I was telling a friend that Flempo is going to use Amazon S3 to store attachment files. He said, well discs are pretty cheap now and this and this friend can help you set up a RAID cluster from them etc etc.

Sounds great, right? You’d have YOUR OWN hard-discs and have a total control over them. Well you might have a control over the discs, but not so much over the data. And that is what matters. Having my own cluster would be one big pain for me:

1. The setup cost would be much higher than with AS3. Mainly because there is no setup cost with AS3.. But it’d take several weeks to set everything up by myself (with or without whichever friend). But setup is not really the issue.

2. Would the monthly cost be higher? I’m actually not sure about this one. How much maintenance would an own cluster need? Two hours a week in average? Less? More? How much would that cost? You can kind of calculate how much AS3 would cost. Except you have only a very rough idea (at least I have at the moment) how much data your users will transfer in and out and how many requests will you need. But even putting a lot in results in pretty low costs.

Now imagine I had a contractor taking care of my own discs. I’d need him/her to be available 24/7 because should something go wrong, I’d want it to be fixed immediately. That would be super expensive. But what more, it’d be a pain. You’d need to monitor the whole thing much better than you do your connection to AS3, do all the backups etc. I.e. this won’t make me free, really.

Until I have a dedicated system administrator (if ever) and need a very very large cluster, AS3 is the way to go. But probably not even then; many startups requiring large infrastructure use Amazon web services now.

The beauty is someone else does the hard work for you, you pay only for what you use, and can focus on what your core business is (which is not really building a highly reliable distributed storage). You can automate things rather than hiring someone just because you can (which I cannot, anyway..) and then be dependent on his/her knowledge.

The whole thing about Flempo (and I will need to stress it more) is that it allows you to be flexible and organized and on top of things at the same time. And the motivation is really very selfish – I want to be like that at the first place. Using AS3 is one more step to my own freedom without affecting the users’ experience.