The Abstract Truth

Twitter versus The Stock Market

Posted by rbpasker on May 6, 2008

A friend sent me a link to a page with the Twitter Stats.

Wow, 3.5 million tweets per week, not counting locked and direct tweets. That’s about 350 tweets PER MINUTE (rules of thumb: ~10K minutes in a week; ~PI*10^7 seconds in a year). Even if the number of private tweets and direct messages drive the volume 5x to 1750 tweets per minute, its still not much. A 1750-person company delivers an average of 1 email message/minute/person. Even Exchange can handle that.

The data feed produced by the Options Price Reporting Authority contains every transaction on every stock and option exchange around the world. The current projections for OPRA are here, but let me republish the table for you:

Yes, that’s 701,000 messages PER SECOND, or about 423 BILLION messages per week.

Where’s the bottleneck again?

8 Responses to “Twitter versus The Stock Market”

  1. Matt said

    I don’t even use Twitter and know nothing about the bottleneck except for what I read, but is the problem in message sending or message display? Is it 3rd-party (including RSS feeds) access? Is it HTML rendering? Do we know enough about the architecture of Twitter to know that any one potential area of slowdown couldn’t cause the bottleneck on the whole site?

    Or are those questions kinda besides the point?

    Certainly it seems that any traditional messaging infrastructure could handle that kinda load with traditional scaling strategies.

  2. rbpasker said

    > any traditional messaging infrastructure could handle that kinda load

    exactly my point.

    nobody knows and nobody’s talking about what the real problem is.

  3. Someone told me it had something to do with Rails? Twitter just fired its CTO too. The question I’m interested in is whether Twitter will be a friendster and lose their market because they couldn’t meet demand.

  4. Matt said

    I take this whole Rails fault thing seriously ’cause I’m a Rails evangelist and technologist. This seems to me a pretty clear case of an architectural problem and not a platform problem.

    Twitter is a messaging system with a nifty web interface (should be pretty close to static, right?), a read-only xml interface (rss), and an xml api (3rd party integration) … at least as far as I understand it. Rails can certainly be used to handle the web and rss interfaces, which should basically be serving up static files or dynamic files that contain 90% cached data. Rails could also be used as a thin layer exposing the api and calling through to the messaging system underneath. Now, if one of those layers is the whole bottleneck, Rails can be blamed.

    Sensibly, Twitter should be a messaging backend (XMPP, JMS … something that’s been solved before) with services that handle writing static files or cached data for the user-exposed layers. I could see there being issues with badly performing 3rd-party clients (which could be handled through a good API) and I could see there being issues of bursty-traffic (which could be handled through clever use of EC2 and thorough de-coupling of the services), but I can’t really see how the interface layers could be optimally designed and still be the bottleneck.

  5. Matt said

    Sorry … I didn’t read that well enough before posting. In regards to my last comment, if static files are being served, Rails makes sense as a technology to generate them, but then it’s not really a Rails issue anymore. Once the files are static it’s a webserver issue.

  6. Bob is right, the finance industry has already solved the problem a long long time ago!
    Why re-inventing the wheel?


  7. I was wondering also whether the old IRC system/infrastructure could be used for that?

  8. ARI ZILKA said

    those volumes are _much_ smaller than I was told. But make a lot more sense. Thanks Bob for tracking this down.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: