Twisted.web vs Tornado, part deux

Published on September 17th, 2009 by Matt Heitzenroder in apparatus Comments Tools:

The first time I conducted a performance test between Twisted.web and Tornado, the results were not very meaningful in retrospect. I have three good reasons for redoing the performance tests:

Apache Benchmark errors – the twisted.web test would crash ab once the concurrency reached a certain level, thus not revealing the real performance of twisted.web.
Unrealistic server environment – I did test of my MacBookPro, not a dedicated server.
Controversy – the ensuing conversation of twisted.web vs tornado warrants a more accurate test with better tools and hardware.

This time, I performed the test with the aid of my friend Travis (thank you!) using The Grinder, a legitimate load testing tool and several virtual machines from the Rackspace Cloud. The servers were Ubuntu Jaunty 9.04 with 2GB or RAM and quad-core 2GHz processors.

The Grinder script that I used to conduct the test just did a simple request. The Grinder test and preferences are available for download. I changed the IP address in the Grinder test to switch between twisted.web and tornado. The following command was used to start the tests:

1	$ for i in 100 200 300 400 500 750 1000; do java -Dgrinder.threads=$i -cp grinder-3.2/lib/grinder.jar net.grinder.Grinder grinder.pref; done;

I used the same, simple ”hello, world” response apps for twisted.web and tornado as I did for the previous test. Each app was run as a single threaded application. I also used Python 2.6.2.

The raw data that resulted from The Grinder and html output from the GrinderAnalysis is available for download.

The following results were compiled into a spreadsheet from the GrinderAnalysis tool. The X-axis is an individual test that represents number of concurrent requests. The Y-axis is the mean response time in milliseconds. Lower mean response time is better.

Python HTTP Server Performance

Below are examples of the response times graphs (blue) that I used to compile the above spreadsheet. The entire output from GrinderAnalytics is available for download, as mentioned above. Transactions per second (TPS) is the graph in the green and is maximum number of iterations of the grinder test that successfully ran in a one second interval.

100 Concurrent Requests

500 Concurrent Requests

1000 Concurrent Requests

This entry was posted on Thursday, September 17th, 2009 at 4:18 pm and is filed under apparatus. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Christian Toivola

Nice benchmarks - I noticed you are part of our Django meetup group but never attended.. we're going on 1 year this month, you should give it a try - I know that other members of our group would be interested to learn more about Twisted in case you have working experience with it. On another note, it would be interesting to see how Erlang-based alternatives would fare against these two, but I guess this comparison was strictly python due in part to the outrage in the community.

roder

@Christian Toivola

Thanks. Guilty as charged, I have never showed up to the django meetup. I'll have a guest in town on the next two meetup dates, but perhaps I'll be able to make the September with my guest (it's actually Dan, the other blogger)

fancypantser

I think the data smell fishy. You are returning "hello world" on a Ubuntu Jaunty server w/2GB of RAM and quad-core 2GHz processors ... and the best response time you can manage on either of the two servers is 500msec? I don't buy it. On a server I have handy, a dual-core 3GHz Ubuntu machine w/2GB RAM, the same Tornado "hello world" averages 50-80msec per response with tests using 100, 250, and 500 concurrent connections.

What else is the machine doing? Is this a long-distance trip or are you on the same switch?

roder

@fancypantser Actually, that's a good point. I used the public IP address and not the internal IP address when I was using the Rackspace cloud VMs. So I suspect this is was a long distance trip out to the bare metal's switch and back to VMs that I conducted the test on. It may not be the perfect, low-latency test, but it does paint a picture of a realistic "interwebz" test.

sean

Right, it's a realistic indication of the heavy 'cloud tax' that Rackspace Cloud/Mosso imposes. Latency has been a noted issue for at least 2 years with their offerings but the company line is 'hey, it scales well -- the numbers start high but don't go much higher under heavy load!'

Name

i agree. this data is total bullshit. he's running grinder from his home computer, which is at a 500 ms lag to rackspace.

roder

@Name That's hilarious... i love that you are insulting the work I did and said what I did wrong to make the data bullshit. If you had clearly read the post, you would have noticed that all tests were performed on the Rackspace Cloud as I said in the post.

Name

i don't doubt that twisted is faster but you're measuring almost entirely network lag. the benchmark needs to be redone. the data is shit.

roder

You know, if you actually gave me productive feedback I'd definately listen to your input and redo the test. Instead, you're just insulting...why?

Janitha Karunaratne

Good work, but may I suggest you please fix the scales on the bottom 6 graphs... the scales are different between left and right and gives a false sense of what's going on visually (between left and right).

roder

I understand exactly what you're saying, but that's the output of Grinder Analyzer and also The Grinder. It does give you a sense of is how long it takes the test to do 100 tests at the variable concurrent amount.