The Abstract Truth

Archive for May, 2008

Twitter versus The Stock Market

Posted by rbpasker on May 6, 2008

A friend sent me a link to a page with the Twitter Stats.

Wow, 3.5 million tweets per week, not counting locked and direct tweets. That’s about 350 tweets PER MINUTE (rules of thumb: ~10K minutes in a week; ~PI*10^7 seconds in a year). Even if the number of private tweets and direct messages drive the volume 5x to 1750 tweets per minute, its still not much. A 1750-person company delivers an average of 1 email message/minute/person. Even Exchange can handle that.

The data feed produced by the Options Price Reporting Authority contains every transaction on every stock and option exchange around the world. The current projections for OPRA are here, but let me republish the table for you:

Yes, that’s 701,000 messages PER SECOND, or about 423 BILLION messages per week.

Where’s the bottleneck again?

Posted in tech | 8 Comments »

Solving The Twitter Problem(s)

Posted by rbpasker on May 6, 2008

The net is atwitter with solutions to the “Twitter Problem,” which is that Twitter keeps going down, and it is annoying the bloggers themselves, who use it as a broadcast platform.

But there are really a bunch of different problems, many of which will be solved independently:

  • Decentralization, a la DNS, of the network of servers/services providing Twitter-like functionality, for redundancy.

  • Scalability using a server-side implementation that can handle the volume of messages on the busiest nodes efficiently. I’ve proposed Tervela‘s product as the hub

  • Inverting the protocol from “pull” to “push,” to reduce the amount of unproductive polling that takes place.

  • A universal addressing scheme, based on existing standards, like RFC 822‘s mailbox addressing. (This is what Jabber uses). A URL-based system would work, too.

  • Proper language mappings (such as Actors) that are designed to handle large numbers of messages with minimal overhead.

  • A common API/wire format, hopefully something much simpler than XMPP, to make it simple to write applications that can participate in this network.

  • Gateways to other messaging systems, such as email and SMS

  • Bridges (PDF) to synchronous protocols, like HTTP

    The good thing is, lots of people are finally starting to think about this problem. The bad thing is, lots of people are finally starting to think about this problem (which means it will take a LONG time to solve).

    • Posted in tech | 2 Comments »

     
    %d bloggers like this: