Amazon Simple Queue Service

I'm absolutely loving working with Amazon Web Services at the new job.  Almost all of our dev infrastructure is running on EC2, as are all of our production and test deployments.  With the exception of our personal machines, we're pretty much entirely in the clouds.

For the last few weeks, I've been working on our newest product.  My part of the system involved managing a large number of asynchronous backend processing tasks.  Messaging seemed like the right way to farm these tasks out to processing nodes, and so I was faced with my first real decision.  Should I go with a traditional messaging system or should we try out something like Simple Queue Service?

After my years at JBoss, I'm very comfortable with JMS.  I've worked with enough customers on messaging issues to know how to build a robust, scalable messaging infrastructure.  I'm really impressed with what the guys on the HornetQ team are doing, and I was pretty sure I would choose HornetQ if decided to deploy our own messaging system.

However, when I thought about the needs of our system, I realized that I wasn't really looking for a high performance message bus.  And, although I hear HornetQ clustering is pretty straight forward, I wasn't really interesting in deploying and managing a cluster of messaging servers.  What I really wanted is exactly what Amazon SQS provides.

SQS requires no setup or management and is almost infinitely scalable.  It doesn't offer fast throughput, but it's very robust and dependable.  Although I haven't seen any service outages, the kind of system I'm working on doesn't even require high availability.  I just need to know that I can reliably push out messages that will eventually be processed, and I need to know that if the amount of work increases that I can scale up the number of processing nodes to receive those messages.

That's what SQS gives.  In fact, it's even a little better than I originally thought.  I thought that at first I might need to monitor the messaging queue and manually spin up and shut down instances based on our load.  Of course, I would plan to write some sort of management system to perform that task, probably sooner rather than later.  But it turns out that with Amazon it's pretty easy to monitor queue levels and automatically start up and shut down worker nodes, regulating queue levels like a thermostat.  

I'm seriously sold on this.   I have reliable messaging system with almost perfect scalability and adaptability.  I was able to focus on my domain task rather than worrying about my infrastructure, and in the end I produced my entire system in less time than I would have normally allocated just to setting up the infrastructure I needed.  

I should say that just because SQS met my needs, it doesn't mean that you can necessarily throw away your messaging infrastructure and move to Amazon too.  If my requirements for throughput were different or if the nature of my tasks were different, SQS might not have been a good fit.  If I had more complex message selection criteria or needed more control of delivery of messages, SQS would not be a good fit.  The SQS interface simply is not as flexible as other messaging services.  But, if it does fit your task, I highly recommend giving it a try.  I'm very pleased with the service so far.