Twisted.web vs Tornado Performance Test

Published on September 11th, 2009 by Matt Heitzenroder in apparatus Comments Tags: http, performance, tornado, twisted, twisted.web Tools:

If you look about this blog, there have been a number posts about the twisted framework. Dan, Igor, and I have been playing with txAMQP for sometime. We’ve never run any twisted code in a high-scale production environment, we’re just enthusiasts. So when I read the comments from the Tornado announcement and in the HackerNews thread, I asked #twisted.web on freenode:

“Who runs a large-scale production environment with twisted.web?”

The channel member responded that I speak to abstractbill since he’s written twisted code for justin.tv. He had already commented on hackernews and therefore, I already knew where he stood. Users were asking and I just needed to see for myself:

“What’s the performance difference between Twisted.web and Tornado?”

So I performed the following tests using Apache Benchmark the same test that finiteloop used. However, I varied concurrent connections in addition to the benchmarked performance of requests per second.

The performance test was performed on my MacBookPro Intel Core Duo 2.4GHz, 4GB of RAM. Each python webserver was run in one, single thread. The code for the tests is available for Twisted.web and Tornado. Both webservers benchmarked by increasing the concurrent connections until the servers closed the connection and prevented ab from completing the test.

Tornado outperforms Twisted.web, but in the scale of those numbers the difference is negligible as finiteloop suggests. What is noticeable is that after 50 concurrent connections, twisted.web’s webserver began to close the connection and not complete the test.

The results do not really stop me from wanting to continue using twisted.web, however there is some room for improvement in the twisted.web framework. I applaud Facebook for releasing Tornado as open source and FriendFeed for create a really nice async webserver in python. I look forward to it’s continued development.

This entry was posted on Friday, September 11th, 2009 at 3:43 pm and is filed under apparatus. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Name

If you're running Tornado on Mac OS X and not on Linux, you're using the select() backend and not the epoll backend. Thus your performance numbers are heavily skewed against Tornado's strengths in terms of number of simultaneous connections it can support.

Yes, it'd be nice if it used libevent so that it could take advantage of kqueue, but it doesn't.

anonymous

Apache benchmark is totally unrealistic. As well you should test a web framework when it is loaded up with apps.

KevBurnsJr

I've noticed some strange behavior with ab. Suggest testing with httperf as well.

Philip Jenvey

What was the CPU utilization like?

Glyph

If I remember correctly, the closed-connections bug is a bug in some versions of 'ab'. Have you tried httperf instead?

roder

I'll try that again with httperf and let you know. Good response btw.

Tofu

I think its funny that they did all that work and got little improvement. Why not improve twisted.web?

I ran some tests. Included webmachine for kicks.

http://thetofu.com/webtest/

Arek

How did your test for 200 concurrent users come out: Twisted ~720 , Tornado ~430 . Then both hover around 700-800 requests per second for 100 and 50 concurrent users.

That major difference during 200 simultaneous conns seems off . I did not see that happen on my machine. Is there anything you noticed during this test ?

Tofu

I didnt notice anything differnt and I thought that was odd too. I meant to run it again a few more times, but I figured I got the information that I wanted. That performance was not an issue between twisted and tornado.

BTW, Dustin has started work on tornado with twisted support. http://github.com/dustin/tornado/tree/master

etrepum

I'm pretty sure that problem you're seeing is a bug in Mac OS X, I've seen it in a lot more than just Twisted and I've never seen it on any other platform. Not sure why Tornado doesn't have it, maybe poll instead of select?

Kenneth Reitz

I can't wait to start using tornado. I love what I see. LOVE.

Thanks for giving the comparison.

exarkun

> What is noticeable is that after 50 concurrent connections, twisted.web’s webserver began to close the connection and not complete the test.

This is strange. I've run this benchmark against Twisted Web many times in the past, and written lots of apps based on Twisted web. I've never seen this problem before. Indeed, I downloaded your twisted.web.py and ran ab against it at various levels of concurrency. The tests all completed, up to the highest setting I tried, 1000:

$ ab -c 1000 -n 10000 http://localhost:8080/
...
Concurrency Level: 1000
Time taken for tests: 6.25909 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0

I wonder if you can investigate the cause of the closed connections you saw? Or suggest what other factors might be involved which cause the behavior in your environment which are missing from mine.

dkoepke

There's a fast path for locally-directed packets. Matt's test, like finiteloop's, performed ab on a remote host. This is probably the difference you're seeing.

exarkun

Also, I thought this might only show up after many more connections (I only ran 10,000 - but you ran 100,000). However, I get the same results when running 100,000 total requests.

roder

We're going to re-run the tests again today on much better hardware today. I'll post the results again and let you know as soon as I do. I suspect it's the fact it's my Mac.

exarkun

I've investigated this a bit further and what I found suggests that this behavior may result from the platform's use of syn cookies. When I was initially unable to reproduce this, I was using Ubuntu Hardy, which disables syn cookies by default. After some experimenting on Ubuntu Jaunty, after a tip from Thomas Herve, I was able to reproduce the problem (although only at a much higher concurrency level - around 500). Jaunty enables syn cookies by default. Then, returning to Hardy and enabling syn cookies, I was able to produce the behavior there. I was also able to provoke this behavior from ab running against the Tornado server - which is perfectly sensible if this explanation is correct, since it's primarily do to something the platform TCP stack is doing, not something either Twisted or Tornado is doing.

I haven't dug into this so deeply that I'm entirely confident that this completely explains the behavior, but I think it's a pretty good start. A direction for further investigation would be to write a client like ab but which reports more identifying information about each connection so that the failing connections can be traced through the entire network.

One thing I'm not sure about is whether OS X uses syn cookies. Some web pages I've found with Google suggest that it does, but it would be nice to get the information from a more authoritative/reliable source.

Jacques Mattheij

That sounds like either a platform issue or a very subtle bug in twister, it would be very good to be able to tell which. Given the results of other users for now my money is on platform issues (maybe a library, a language version or something like that).

You should list as many bits of info as you can to see if somebody can reproduce this.

Twisted.web vs Tornado Performance Test

Recent Posts

Tags

Archives