Comments on: Twitter versus The Stock Market

By: ARI ZILKA

ARI ZILKA — Thu, 08 May 2008 06:39:43 +0000

those volumes are _much_ smaller than I was told. But make a lot more sense. Thanks Bob for tracking this down.

By: David Berrebi

David Berrebi — Wed, 07 May 2008 10:40:20 +0000

I was wondering also whether the old IRC system/infrastructure could be used for that?

By: David Berrebi

David Berrebi — Wed, 07 May 2008 10:39:18 +0000

Bob is right, the finance industry has already solved the problem a long long time ago!
Why re-inventing the wheel?

By: Matt

Matt — Tue, 06 May 2008 23:33:35 +0000

Sorry … I didn’t read that well enough before posting. In regards to my last comment, if static files are being served, Rails makes sense as a technology to generate them, but then it’s not really a Rails issue anymore. Once the files are static it’s a webserver issue.

By: Matt

Matt — Tue, 06 May 2008 23:31:27 +0000

I take this whole Rails fault thing seriously ’cause I’m a Rails evangelist and technologist. This seems to me a pretty clear case of an architectural problem and not a platform problem.

Twitter is a messaging system with a nifty web interface (should be pretty close to static, right?), a read-only xml interface (rss), and an xml api (3rd party integration) … at least as far as I understand it. Rails can certainly be used to handle the web and rss interfaces, which should basically be serving up static files or dynamic files that contain 90% cached data. Rails could also be used as a thin layer exposing the api and calling through to the messaging system underneath. Now, if one of those layers is the whole bottleneck, Rails can be blamed.

Sensibly, Twitter should be a messaging backend (XMPP, JMS … something that’s been solved before) with services that handle writing static files or cached data for the user-exposed layers. I could see there being issues with badly performing 3rd-party clients (which could be handled through a good API) and I could see there being issues of bursty-traffic (which could be handled through clever use of EC2 and thorough de-coupling of the services), but I can’t really see how the interface layers could be optimally designed and still be the bottleneck.

By: Chris Neumann

Chris Neumann — Tue, 06 May 2008 22:43:06 +0000

Someone told me it had something to do with Rails? Twitter just fired its CTO too. The question I’m interested in is whether Twitter will be a friendster and lose their market because they couldn’t meet demand.

By: rbpasker

rbpasker — Tue, 06 May 2008 22:41:34 +0000

> any traditional messaging infrastructure could handle that kinda load

exactly my point.

nobody knows and nobody’s talking about what the real problem is.

By: Matt

Matt — Tue, 06 May 2008 16:42:59 +0000

I don’t even use Twitter and know nothing about the bottleneck except for what I read, but is the problem in message sending or message display? Is it 3rd-party (including RSS feeds) access? Is it HTML rendering? Do we know enough about the architecture of Twitter to know that any one potential area of slowdown couldn’t cause the bottleneck on the whole site?

Or are those questions kinda besides the point?

Certainly it seems that any traditional messaging infrastructure could handle that kinda load with traditional scaling strategies.