In the last few days I’ve been experimenting with ActiveMQ, a JMS broker implementation by the Apache Foundation. I’m looking for innovative solutions to make our enterprise infrastructure more robust and flexible and this project looks very promising. For a list of interesting highlights:
and even more obscure features. JMS is a Java technology and API, but ActiveMQ allows you to apply the same thing to different environments, even in a mixed configuration. The point of messaging is that you can decouple applications and components so that they can exchange data items with no need to synchronize or to know each other. It’s a basic principle in Enterprise Integration Patterns.
Some alternative solutions that I evaluated and discarded:
The producer process runs single thread bursting a configurable amount of messages at max speed, each message contains a 6-digit progressive counter. Optionally messages can be padded with dummy data, spread on a configurable amount of different queues or made persistent.
Each consumer runs in a separate single thread process printing a line at most every second with a time stamp and the 6digit code found in the last message received. The time stamp is zeroed every time a message is received with a code less than the last that was seen. This way, it’s easy to find running time for multiple repeated bursts.
Even if the two instances are interconnected, the default configuration only loads the master instance. In default configuration ActiveMQ can enqueue and dequeue approx 195K messages per minute, or 3250 messages per second, which is not bad at all. No message accumulates in queue and consumers fetch messages as the same speed as the producer. My first test was trying to see how this number would change by adding or removing consumers.
In this graph we see that performance is slightly better with only one consumer and gets significantly worse when adding more consumers. For more than 8 parallel consumers throughput gets stabilized around 90K messages a minute.
Next, I checked what happens for bigger payloads. We can see a slight decrease in message rate while moving from 6 to 500 bytes per message, then we enter a sort of bandwidth limit saturating between 400 and 500 Kbytes per second (and message rate decreasing accordingly).
When spreading messages over multiple queues (subscribing all of them in each consumer) we can see a slight decrease in message rate that anyhow keeps above 160K messages per minute. When I tried to increase queues to 1024 ActiveMQ crashed, running out of file descriptors. Apparently a file is created on disk and kept open for each active queue, therefore there is a stringent limit on the number of concurrent queues you can handle.
Now, in all of the above cases traffic flowed from producer to consumer via one single message broker. In this configuration the second broker only acts as a fail-over to be used in case the master instance goes down. This only works if producer and consumer somehow coordinate (e.g. using the same priority order in the list of available brokers) and you don’t need to balance load among a cluster of consumers.
A more advanced configuration could use store and forward so that producers can pump messages on a random broker and messages would always reach subscribers registered for that queue, even if they are connected to a different broker. ActiveMQ easily supports this configuration and in a mesh of brokers in the same LAN your message normally gets to the consumer in at most 2 hops (e.g. traversing two brokers). To test what happens in this topology, I configured our consumers to connect to the other broker. We can see a drop in throughput, almost halved. It’s not bad but I believe they can do better here. After all, adding one hop should only affect latency, not throughput. Actually the inter-broker connection looked partially unstable, sometime losing sync and automatically reconnecting.
Now, all of this was for non-persistent queues. This means every time you restart the broker all messages in queue get wiped out. None the less, messages are kept in RAM (or on disk, when queue length gets above a configurable threshold) while there’s no consumer to serve a queue. You have several ways to control message lifetime, including expiration limit. This is OK if you mostly care for speed and can afford losing messages, but in case you have to deliver 100% of your messages you have to use a persistent queue. Messages can be marked persistent by adding the relevant attribute in the producer code.
When using persistence we see a dramatic drop in performance. The AMQ built-in message storage (default choice) is slower by a factor of 3 while MySQL is slower by a factor of 10. Adding the built-in high performance journal on top of mysql gets even worse (even if it’s supposed to help, according to documentation). We also see a very different behavior between producer and consumer interfaces. With persistence consumers become much slower than producers, by as much as a factor of 10. Persistence is a nice feature but adds a huge penalty. None the less with AMQ you can still run 900 messages a second.
ActiveMQ would support a master slave configuration where the slave would replicate every persistent message, to be used for disaster recovery in case the master storage suddenly becomes unusable. I tried this configuration by it kept on crashing (out of sync after each temporary disconnection between master and slave), so I guess it’s not production grade yet.
Note: in all of the above tests each instance of ActiveMQ took around 150M of resident RAM (almost 900M in virtual) and ActiveMQ was the most CPU intensive process in top output.
Conclusion: ActiveMQ 5.0.0 appears to be very good for non persistent messaging and queueing. Also provides good performance for persistent messaging using the built-in AMQ storage. Don’t try using an external SQL database for storage because the performance penalty is huge. Benchmark data can be found here ActiveMQ benchmark data
Well, I’m not sure this qualifies as “useless knowledge”…
You know, I have to honor that “mostly” in the big title
Thanks for sharing your data on ActiveMQ.
We are trying to enqueue messages inside tomcat but the call queueSender#send blocks indefinetly. Surprisingly the same code works on a different machine.
What can be the cause of this problem ?
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(“”, “”, “tcp://192.168.98.141:61616″);
javax.jms.QueueConnection connection = connectionFactory.createQueueConnection();
queueSession = connection.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);
javax.jms.Queue queue = queueSession.createQueue(queueName);
QueueSender sender = queueSession.createSender(queue);
TextMessage message2 = queueSession.createTextMessage();
// set some props message2.setStringProperty(ICPWSConsts.ICP_CUSTOMERID_PROP, customerIdentifier);
Unfortunately, in my experience, ActiveMQ has a great feature set but suffers from reliability issues. There is one specific app that I wrote a year ago and it’s interaction with ActiveMQ has been one of the biggest support nightmares ever since. The main issue has been around how it handles message delivery failure, i.e. the receiving end of the message queue does database work and the database goes down for an hour and so message delivery is failing – database comes back up but messages don’t empty from the queue – plus other instances of it just more or less randomly stopping delivery. Maybe it’s my app or other network-related issues – but my personal experience with ActiveMQ is that it is not really solid production-quality code. But, in all fairness, it could just be the way I’m using it.
[…] This post was mentioned on Twitter by brucesnyder, Domingo Suarez. Domingo Suarez said: RT @bubbl: ActiveMQ : Performance http://ff.im/-aHX1V (via @brucesnyder) […]
Social comments and analytics for this post…
This post was mentioned on Twitter by bubbl: ActiveMQ : Performance http://ff.im/-aHX1V…
[…] one is posted by @bubble. The test covers throughput vs message size and throughput vs persistence. The […]
[…] The numbers are fair. Others on the net have observed similar results: http://www.mostly-useless.com/blog/2007/12/27/playing-with-activemq/ […]