Distributed stress tester with Erlang Part 1

One day, not too long ago, I was asked by a client, “What would happen if we send the server five times the volume of traffic currently being sent?” They were asking this question because they had just acquired another company and was planning on integrating that companies services with theirs. The answer, quite frankly, was no. Our servers’ CPUs were hovering near 50% utilization at all times, and we didn’t want to risk maxing out.

However, as luck might have it, we were getting new hardware in two weeks. That gave me two weeks to figure out just how much load our new hardware could handle.

Examining the Problem

We needed to figure out how much load the new servers could handle, so we needed a way of flooding the network with requests. We had a regression tool, but it just wasn’t cut out for this type of work. It sent one message at a time, and did pre and post processing. No, we needed something better, something faster.

We could use perl, make it concurrently send messages to the server via curl. That would work. But it wasn’t distributed, so unless we coordinated our testing, it would be limited by that one user’s bandwidth. Not enough to reach current production traffic. And a hassle to coordinate, so that won’t work.

Then I remembered the conference I went to a while back, CodepaLOUsa, there was a session I attended by Bryan Hunter about Erlang.

Erlang: Enter Stage Right

Erlang has this neat ability to make distributed, cross-network computing extremely simple. It was designed for this type of job. To connect to a cluster (or “cloud”) of computers is only one line of code:

net_adm:ping('remoteNode@hostname').

Not only does that connect you to the remote node, it also connects you to every node connected with the remote node. And when your pool of nodes is a wandering army of co-worker laptops, it’s a good thing Erlang gossips. To manage our constantly changing network of nodes, we setup one, immobile, always on node at a development server. Whenever a laptop was booted, an Erlang node was launched, which connected to the coordinating node at the development server. Because everyone knew where that was, everyone knew where everyone else was too.

Let’s Work Together

Now that we have a cluster of nodes, let’s do something with them! To send a command to all the nodes is also incredibly simple in Erlang. It’s also just one line of code:

rpc:multicall(ModuleName, FunctionName, [Array of Parameters]).

It’s as simple as that. That line of code will call

ModuleName:FunctionName(Array of Parameters).

on all nodes connected to you, including yourself.

But Sir, How Will We Know What To Do?

Now that we have all the nodes connected, and know how to tell them to do something, how do we deploy our module that actually does the stress testing across all the nodes? In a normal situation, you’d have to email everyone a copy of your binary, or have them update it using some sort of tool. Or if you’re savvy, you could have a cron job check for new versions and update every so often. But that’s all too complicated. All the hassle to send that out and get everyone to update, or all the red tape for management to approve a new deploy. Too much work, let’s solve it with Erlang. In Erlang, deploying your module to all the nodes connected to you is, you guessed it, one line:

c:nl(ModuleName).

Once we compile our stress tester, we can deploy it using that single line of code, and then we can make all the nodes execute it using rpc:multicall.

What Next?

Now we have our network of laptops connected, and everyone has the code and can run it, what is actually in our code for stress testing? Join me next time for that!

To Be Continued…

Be Sociable, Share!

No related posts.

No Comments.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>