Part of my job is to select technical people to be hired in the company.  Few months ago I happened to interview a guy for a developer position.  This guy has a degree in chemistry but his career then oscillated between system administration and programming in the java world.

After a while talking about his experience he said that, being in his 30s, he was looking for some stability.  In particular, he said, he was frightened by how quickly our knowledge can become obsolete in the programming world.  For example in 5 years Java could become obsolete because everybody could move to the next big thing (.net, he said), making Java skills useless and forcing you to learn everything from scratch.  He added that system administration and programming skills in the open source world will last longer.

On one hand I agree: open source skills will last longer.  In the open source world we never have to push the new thing just to force everybody to buy the new version.  We’re not at the mercy of some company’s agenda.  We grow software incrementally, evolving technology on top of the existing good old stuff, with continuous innovation and experimentation patch after patch and using Darwinism to select stuff that works and re-use it forever.

On the other hand I think this guy doesn’t understand what programming is really about, on a deeper level. In our world technologies evolve very quickly.  Every day new emerging technologies provide better ways to do the same thing and even in the evolutionary open source world, we have to keep up with the fast-pace advancement.

That’s why good schools won’t teach you a specific programming language or a specific application or operating system.  Good schools won’t teach you the technology of the day.  They will teach you how to learn new stuff quickly, how to be your own self-teacher.  They will prepare you for a whole life of research and study.

As Jeff Atwood once wrote “how lasts about five years, but why is forever”

Last weekend I went to PyCon Due, the 2nd Italian conference on the Python programming language.  It was in Firenze, a few minutes walk from where I live, so it’s been very handy.  This year the conference was much bigger than last year’s.

For starters, on Friday Richard Stallman spoke at Palazzo Vecchio.  I couldn’t go (I had to work) but luckily enough I listened to RMS in a speech he gave at Hackmeeting 2002 in Bologna, so I’m blessed already.  I missed the opportunity to visit the outstanding Salone del 500, tough.

The next two days were at Auditorium Duomo, where the guys from python.it organized three parallel tracks running from 9am-7pm, with little breaks for some rest and socialization opportunity.  There were many well-known speakers, including Raymond Hettinger, Brian Fitzpatrick and of course Alex Martelli.  I attended talks about Django, Google App Engine, how to listen to our users, Plone, nginx’s mod_wsgi, Zope3, Twisted, Python 3.0 and Python inner workings.

A long list of companies sponsored the event: some big 900-pound gorillas like Google and Skype, as well as local fast-growing Italian companies.  Of course they took the opportunity to try and hire high profile developers, presenting their business and collecting contacts.  Too bad the company I work for didn’t participate.

All in all, with over 300 attendees I can say the conference was a big success, even if some talks were a little under par in terms of quality or effectiveness.  I didn’t like the “teach me” formula but most talks were  definitely outstanding.  And did I mention the hostesses were beautiful?  Go python.it go!  You totally rock!

from The Book of Mozilla, 7:15

And so at last the beast fell and the unbelievers rejoiced.
But all was not lost, for from the ash rose a great bird.
The bird gazed down upon the unbelievers and cast fire
and thunder upon them. For the beast had been
reborn with its strength renewed, and the
followers of Mammon cowered in horror.

Looking at this dithered GIF makes me so nostalgic…
Today AOL announced Netscape Navigator has been discontinued. Time for a new chapter in Mozilla’s book.

Exactly ten years ago, on Feb 9th 1998, Bruce Perens and Eric Raymond began the Open Source movement. It was just a different way to explain what had been already happening for quite a few years, and make it understandable for the business world. And it worked very well.

Building on the ground-breaking work of great leaders like Richard Stallman and Linus Torvalds, we laid out the software technology that leads many markets of today’s world, entering the mainstream. When I say we it’s because I have been an active contributor of this community and an advocate of the open source concept since the beginning.

Nowadays I use open source for everything, my parents use Linux on their desktop, my friends are eager to buy an Asus EEE and major governments have state funded programs to adopt or develop open source. Several open source companies are making billions and almost everybody is saving huge money just by using it.

The idea spread even out of the software world, generating parallel incarnations in the culture generation and distribution area. Try to imagine a world without open source: no Firefox, no Open Office, no MySQL, no OLTP project, no Asus EEE, no Creative Commons, no Wikipedia, no peer-to-peer. Lately, all the biggest innovations in the software world came from the open source community are were possible because of it. There are still are some areas where Open Source has to work in order to go truly mainstream, for example Desktop applications, but we can clearly see it’s going to happen sooner or later.

Thanks to Linus, Richard, Bruce, Eric, Larry, Guido, to friends at Ferrara Linux Users Group, and to everybody who helped and believed in this amazing challenge.  Where will we stand in 2018?  I can’t wait to see that.

In the last few days I’ve been experimenting with ActiveMQ, a JMS broker implementation by the Apache Foundation. I’m looking for innovative solutions to make our enterprise infrastructure more robust and flexible and this project looks very promising. For a list of interesting highlights:

• automatic clustering via multi-cast discovery as well as static routing configurations
• support for multiple communication protocols including Openwire (a native binary protocol), Stomp (an open text-based protocol), XMPP, REST and many more
• client implementations available for a wide range of programming languages
• support for queues (one and only one paradigm) and topics (publish-subscriber one-to-many message distribution)
• optional persistence via a built-in message store or standard database integration

and even more obscure features. JMS is a Java technology and API, but ActiveMQ allows you to apply the same thing to different environments, even in a mixed configuration. The point of messaging is that you can decouple applications and components so that they can exchange data items with no need to synchronize or to know each other. It’s a basic principle in Enterprise Integration Patterns.

Some alternative solutions that I evaluated and discarded:

• using a Jabber server (e.g. ejabberd): limited functionality, high overhead
• using an MTA (e.g. postfix): only supports persistent queues, high overhead
• Apache qpid: not mature yet
• RabbitMQ: lack of mature client code
• Spread Toolkit: no persistence, no one-and-only-one
• Proprietary MOM (e.g. TIBCO, IBM Websphere MQ): expensive

Anyhow, for my experiments I connected ActiveMQ using Perl clients, using acme‘s Net::Stomp module. The basic setup was like this:

• Everything running on a single Athlon64 3200+, 2GB RAM box
• Gentoo Linux amd64, kernel 2.6.23, glibc 2.7, Sun JDK 1.6.0
• two instances of ActiveMQ 5.0.0 auto-networked with multi-cast discovery and talking Openwire between them
• one non-persistent queue
• one producer injecting 6byte long non-persistent messages on the master instance
• two competing consumers fetching messages in auto-acknowledge mode from the master instance

The producer process runs single thread bursting a configurable amount of messages at max speed, each message contains a 6-digit progressive counter. Optionally messages can be padded with dummy data, spread on a configurable amount of different queues or made persistent.

Each consumer runs in a separate single thread process printing a line at most every second with a time stamp and the 6digit code found in the last message received. The time stamp is zeroed every time a message is received with a code less than the last that was seen. This way, it’s easy to find running time for multiple repeated bursts.

Even if the two instances are interconnected, the default configuration only loads the master instance. In default configuration ActiveMQ can enqueue and dequeue approx 195K messages per minute, or 3250 messages per second, which is not bad at all. No message accumulates in queue and consumers fetch messages as the same speed as the producer. My first test was trying to see how this number would change by adding or removing consumers.

In this graph we see that performance is slightly better with only one consumer and gets significantly worse when adding more consumers. For more than 8 parallel consumers throughput gets stabilized around 90K messages a minute.

Next, I checked what happens for bigger payloads. We can see a slight decrease in message rate while moving from 6 to 500 bytes per message, then we enter a sort of bandwidth limit saturating between 400 and 500 Kbytes per second (and message rate decreasing accordingly).

When spreading messages over multiple queues (subscribing all of them in each consumer) we can see a slight decrease in message rate that anyhow keeps above 160K messages per minute. When I tried to increase queues to 1024 ActiveMQ crashed, running out of file descriptors. Apparently a file is created on disk and kept open for each active queue, therefore there is a stringent limit on the number of concurrent queues you can handle.

Now, in all of the above cases traffic flowed from producer to consumer via one single message broker. In this configuration the second broker only acts as a fail-over to be used in case the master instance goes down. This only works if producer and consumer somehow coordinate (e.g. using the same priority order in the list of available brokers) and you don’t need to balance load among a cluster of consumers.

A more advanced configuration could use store and forward so that producers can pump messages on a random broker and messages would always reach subscribers registered for that queue, even if they are connected to a different broker. ActiveMQ easily supports this configuration and in a mesh of brokers in the same LAN your message normally gets to the consumer in at most 2 hops (e.g. traversing two brokers). To test what happens in this topology, I configured our consumers to connect to the other broker. We can see a drop in throughput, almost halved. It’s not bad but I believe they can do better here. After all, adding one hop should only affect latency, not throughput. Actually the inter-broker connection looked partially unstable, sometime losing sync and automatically reconnecting.

Now, all of this was for non-persistent queues. This means every time you restart the broker all messages in queue get wiped out. None the less, messages are kept in RAM (or on disk, when queue length gets above a configurable threshold) while there’s no consumer to serve a queue. You have several ways to control message lifetime, including expiration limit. This is OK if you mostly care for speed and can afford losing messages, but in case you have to deliver 100% of your messages you have to use a persistent queue. Messages can be marked persistent by adding the relevant attribute in the producer code.

When using persistence we see a dramatic drop in performance. The AMQ built-in message storage (default choice) is slower by a factor of 3 while MySQL is slower by a factor of 10. Adding the built-in high performance journal on top of mysql gets even worse (even if it’s supposed to help, according to documentation). We also see a very different behavior between producer and consumer interfaces. With persistence consumers become much slower than producers, by as much as a factor of 10. Persistence is a nice feature but adds a huge penalty. None the less with AMQ you can still run 900 messages a second.

ActiveMQ would support a master slave configuration where the slave would replicate every persistent message, to be used for disaster recovery in case the master storage suddenly becomes unusable. I tried this configuration by it kept on crashing (out of sync after each temporary disconnection between master and slave), so I guess it’s not production grade yet.

Note: in all of the above tests each instance of ActiveMQ took around 150M of resident RAM (almost 900M in virtual) and ActiveMQ was the most CPU intensive process in top output.

Conclusion: ActiveMQ 5.0.0 appears to be very good for non persistent messaging and queueing. Also provides good performance for persistent messaging using the built-in AMQ storage. Don’t try using an external SQL database for storage because the performance penalty is huge. Benchmark data can be found here ActiveMQ benchmark data

I know how it feels…

Google led the creation of Open Handset Alliance, a consortium involving a number of telco and manufacturers . As its first act the Alliance released Android, an open source operating system for mobiles complete of SDK and API. The SDK includes a working emulator and half a dozen example applications. The idea to establish an open platform for mobile developers is very good but not particularly new: project Openmoko has been working to a similar concept for several months and went as far as to release a developer version of the handset.

Currently there are several competing platforms with a significant presence on the market: Symbian OS, Microsoft Windows Mobile, Apple iPhone, Motorola MotoMAGX, Qualcomm BREW and RIM Blackberry to name a few. Trying to build a business on top of mobile applications is very difficult because you have to make several different versions of your application and the market is very fluid. Now, with the huge money and pressure that Google can inject into the field there’s a chance they can impose a common platform.

They chose to build on top of several open source technologies: Linux, OpenGL, SQLite and Webkit. The latter is particularly interesting because it’s a rendering engine derived from the KDE project (and going to be merged back into konqueror) and already powers the iPhone, s60 and Safari. This could become the rendering engine widely used both on the web and on mobile handsets.

To bootstrap the project and quickly build a critical mass of third party developers, Google issued a contest with a 10 million dollar prize for the best applications that will be developed in the next few months. There are a few countries not allowed to participate to the contest. As usual there are countries forbidden by US laws (Cuba, Iran, Syria, North Korea, Sudan, and Myanmar) and on top of that there are countries excluded because of local restrictions: Quebec and… Italy!

So Telecom Italia is one of the members of Open Handset Alliance, but Italians can’t participate to the challenge. Great. How is this possible? Simple: in Italy contests are regulated by a very strict law requiring that issuers create accurate documentation, put the prize in a protected guarantee fund and seek permission of the Ministry of Economy. Google probably found that going through all of this was not worthwhile. Italian bureaucracy and over-legislation at work. Do you remember my last post on this topic?

This weekend I’ve been to PyCon Uno, the first italian conference dedicated to the Python programming language. It’s been very interesting, in particular Alex Martelli‘s keynote on managing the tech development and a presentation on sqlalchemy, but this post is not about the conference.

Marco Pesenti Gritti’s final keynote was about One Laptop Per Child, a project to bring a low cost computer to children in those third world countries where the benefits of worldwide information and education technology would be otherwise impossible. With the help of UN and MIT, they are trying to build a 100$ book-size laptop with a 1200×900 colour display and consuming just about 2W. It’s going to be developed using open source software only, and to use Python and GTK+ to develop a revolutionary environment that discards the desktop/office metaphor and puts emphasis on children activities. Several states already joined the project and did preliminary testing on some of their schools.

I think they did a very good job and hope this is going to be a great success.

Is there anybody out there who doesn’t know who Steve Jobs and Bill Gates are? Everybody and his brother know that they are the leaders of Apple and Microsoft and have been dominating the personal computer industry for the last 30 years. In all of this time, they’ve been very careful to appear together very rarely, but last Wednesday they shared the stage at the D: All things Digital conference. They chatted about how the industry evolved during their kingdom and talked about how they think our digital future is going to be. A warm and friendly chat, like a get together for long-standing friends who finally relax and nostalgically recall the common past. But wait, look at the picture, is that a real smile?

They’ve been enemies so far, trying to extend their business by attacking the market occupied by the other company, but now they have common enemies to fight. Google is moving users one level up in the stack, feeding them with application that run inside a normal Internet browser, storing data on the server, and basically making the personal computer a commodity that nobody cares about. And they use AJAX, a technology paradigm where neither Apple nor Microsoft have experience. Hand-held manufactures are on their way to provide a new computing concept, where you carry your important data and applications in your pocket and always on. Meanwhile, the Open Source movement is making operating systems and standard applications a commodity.

They know their traditional business is in danger, so their companies are both busy at differentiating, trying to find an escape path. Microsoft spent the last years investing in game consoles (the Xbox), internet search engines, portals and advertising (MSN) and now they’re trying to merge with Yahoo and become a media company. Apple spent the last years investing in portable players (iPod), music distribution (iTunes), cell phones (iPhone) and basically dismissed the computer hardware market when they stopped having their own technological platform, standardizing on the same Intel processors that run Windows.

This get-together appears as one more signal the personal computer industry reached a dead end and a new industry is taking the lead. That is not a real smile at all. Their eyes are nervously trying to convey a message: “let’s stop our fight and work together to save the business”. I won’t be surprised if they announce a joint venture or a strategic partnership in a few months.

TICKER="QQQQ"
PS1="\[\033[01;32m\]\`GET 'http://download.finance.yahoo.com/d/quotes.csv?s=$TICKER&f=sl1d1'|cut -f2 -d,\` \u@\h\[\033[01;34m\] \w \$\[\033[00m\] "
export PS1

Happy monitoring!