In the last few days I’ve been experimenting with ActiveMQ, a JMS broker implementation by the Apache Foundation. I’m looking for innovative solutions to make our enterprise infrastructure more robust and flexible and this project looks very promising. For a list of interesting highlights:
- automatic clustering via multi-cast discovery as well as static routing configurations
- support for multiple communication protocols including Openwire (a native binary protocol), Stomp (an open text-based protocol), XMPP, REST and many more
- client implementations available for a wide range of programming languages
- support for queues (one and only one paradigm) and topics (publish-subscriber one-to-many message distribution)
- optional persistence via a built-in message store or standard database integration
and even more obscure features. JMS is a Java technology and API, but ActiveMQ allows you to apply the same thing to different environments, even in a mixed configuration. The point of messaging is that you can decouple applications and components so that they can exchange data items with no need to synchronize or to know each other. It’s a basic principle in Enterprise Integration Patterns.
Some alternative solutions that I evaluated and discarded:
- using a Jabber server (e.g. ejabberd): limited functionality, high overhead
- using an MTA (e.g. postfix): only supports persistent queues, high overhead
- Apache qpid: not mature yet
- RabbitMQ: lack of mature client code
- Spread Toolkit: no persistence, no one-and-only-one
- Proprietary MOM (e.g. TIBCO, IBM Websphere MQ): expensive
- Everything running on a single Athlon64 3200+, 2GB RAM box
- Gentoo Linux amd64, kernel 2.6.23, glibc 2.7, Sun JDK 1.6.0
- two instances of ActiveMQ 5.0.0 auto-networked with multi-cast discovery and talking Openwire between them
- one non-persistent queue
- one producer injecting 6byte long non-persistent messages on the master instance
- two competing consumers fetching messages in auto-acknowledge mode from the master instance
The producer process runs single thread bursting a configurable amount of messages at max speed, each message contains a 6-digit progressive counter. Optionally messages can be padded with dummy data, spread on a configurable amount of different queues or made persistent.
Each consumer runs in a separate single thread process printing a line at most every second with a time stamp and the 6digit code found in the last message received. The time stamp is zeroed every time a message is received with a code less than the last that was seen. This way, it’s easy to find running time for multiple repeated bursts.
Even if the two instances are interconnected, the default configuration only loads the master instance. In default configuration ActiveMQ can enqueue and dequeue approx 195K messages per minute, or 3250 messages per second, which is not bad at all. No message accumulates in queue and consumers fetch messages as the same speed as the producer. My first test was trying to see how this number would change by adding or removing consumers.
In this graph we see that performance is slightly better with only one consumer and gets significantly worse when adding more consumers. For more than 8 parallel consumers throughput gets stabilized around 90K messages a minute.
Next, I checked what happens for bigger payloads. We can see a slight decrease in message rate while moving from 6 to 500 bytes per message, then we enter a sort of bandwidth limit saturating between 400 and 500 Kbytes per second (and message rate decreasing accordingly).
When spreading messages over multiple queues (subscribing all of them in each consumer) we can see a slight decrease in message rate that anyhow keeps above 160K messages per minute. When I tried to increase queues to 1024 ActiveMQ crashed, running out of file descriptors. Apparently a file is created on disk and kept open for each active queue, therefore there is a stringent limit on the number of concurrent queues you can handle.
Now, in all of the above cases traffic flowed from producer to consumer via one single message broker. In this configuration the second broker only acts as a fail-over to be used in case the master instance goes down. This only works if producer and consumer somehow coordinate (e.g. using the same priority order in the list of available brokers) and you don’t need to balance load among a cluster of consumers.
A more advanced configuration could use store and forward so that producers can pump messages on a random broker and messages would always reach subscribers registered for that queue, even if they are connected to a different broker. ActiveMQ easily supports this configuration and in a mesh of brokers in the same LAN your message normally gets to the consumer in at most 2 hops (e.g. traversing two brokers). To test what happens in this topology, I configured our consumers to connect to the other broker. We can see a drop in throughput, almost halved. It’s not bad but I believe they can do better here. After all, adding one hop should only affect latency, not throughput. Actually the inter-broker connection looked partially unstable, sometime losing sync and automatically reconnecting.
Now, all of this was for non-persistent queues. This means every time you restart the broker all messages in queue get wiped out. None the less, messages are kept in RAM (or on disk, when queue length gets above a configurable threshold) while there’s no consumer to serve a queue. You have several ways to control message lifetime, including expiration limit. This is OK if you mostly care for speed and can afford losing messages, but in case you have to deliver 100% of your messages you have to use a persistent queue. Messages can be marked persistent by adding the relevant attribute in the producer code.
When using persistence we see a dramatic drop in performance. The AMQ built-in message storage (default choice) is slower by a factor of 3 while MySQL is slower by a factor of 10. Adding the built-in high performance journal on top of mysql gets even worse (even if it’s supposed to help, according to documentation). We also see a very different behavior between producer and consumer interfaces. With persistence consumers become much slower than producers, by as much as a factor of 10. Persistence is a nice feature but adds a huge penalty. None the less with AMQ you can still run 900 messages a second.
ActiveMQ would support a master slave configuration where the slave would replicate every persistent message, to be used for disaster recovery in case the master storage suddenly becomes unusable. I tried this configuration by it kept on crashing (out of sync after each temporary disconnection between master and slave), so I guess it’s not production grade yet.
Note: in all of the above tests each instance of ActiveMQ took around 150M of resident RAM (almost 900M in virtual) and ActiveMQ was the most CPU intensive process in top output.
Conclusion: ActiveMQ 5.0.0 appears to be very good for non persistent messaging and queueing. Also provides good performance for persistent messaging using the built-in AMQ storage. Don’t try using an external SQL database for storage because the performance penalty is huge. Benchmark data can be found here ActiveMQ benchmark data