The Abstract Truth

Now that sonic.net has added self-service subdomains, you can now read this blog at: http://blog.pasker.net . i would have added theabstracttruth.pasker.net, as well, but WordPress wants $10 for each separate domain, and it didn’t seem worth it.

Note that this is in addition to http://theabstracttruth.wordpress.com . both will work, but in case I decide to change blogging systems, its probably better to use http://blog.pasker.net for web and RSS .

I recently gave a presentation to the technical management and architects at a Very Large Web Property on messaging systems, AKA Message-Oriented Middleware, AKA Enterprise Services Bus, etc.

This was not meant to be an introduction, because they already have messaging systems in use. Rather, I wrote this presentation (in about 15 minutes) as a springboard to remind everyone what messaging system are capable of, so we could talk further about how to capitalize on the products they were already using.

A friend sent me a link to a page with the Twitter Stats.

Wow, 3.5 million tweets per week, not counting locked and direct tweets. That’s about 350 tweets PER MINUTE (rules of thumb: ~10K minutes in a week; ~PI*10^7 seconds in a year). Even if the number of private tweets and direct messages drive the volume 5x to 1750 tweets per minute, its still not much. A 1750-person company delivers an average of 1 email message/minute/person. Even Exchange can handle that.

The data feed produced by the Options Price Reporting Authority contains every transaction on every stock and option exchange around the world. The current projections for OPRA are here, but let me republish the table for you:

Yes, that’s 701,000 messages PER SECOND, or about 423 BILLION messages per week.

Where’s the bottleneck again?

The net is atwitter with solutions to the “Twitter Problem,” which is that Twitter keeps going down, and it is annoying the bloggers themselves, who use it as a broadcast platform.

But there are really a bunch of different problems, many of which will be solved independently:

In his latest polemic, Herr Hildebrand interprets my robustness problems as a dichotomy between the Thread and the Process. (Just joking, Hal!) Like earlier authors, I will also refrain from providing precise definitions, except to bastardize Justice Potter Stewart’s old saw, “I know a [process|thread] when I see one.”

Well, Hal is exactly right that the process model has huge robustness benefits over threads due to the memory and scheduler isolation enforced between processes by the OS and MMU. He is also correct that much robustness is lost when running many applications in a multi-threaded environment (like the JVM), creating operational behavioral dependency among applications that didn’t exist at design time.

But rather than bowing to the false architectural dichotomy between the Process Model and the Thread Model, I propose an architecture that uses each of them to their best advantage, using processes to separate applications, and threads to separate transactions within applications, and also proposing a number of improvements to the JVM that would make this hybrid model as cheap to run as the monolithic JVM model we currently had.

Before continuing, let’s imagine for a minute a JVM that has some special properties:

1. it has no transactional state, which most people would read as “share nothing”
   
2. the transactional state is cached locally, and the ACI (but not D) cache is shared across all processes via shared memory
   
3. a fully operational JVM can be launched from scratch, ready to process transactions, in, say, under a second
   

With such a JVM, it is not hard to imagine the proposed solution: the runtime architecture consist of one jar file (i.e., one application) per JVM, and that each JVM handles multiple simultaneous transactions in separate threads. An errant application could then be recycled by recycling the entire JVM around it. For greater robustness within a single application, multiple JVMs could run the same jar.

What you now have is a mini-cluster that provides the robustness of the Process Model with the multiprocessing and “forking” speed of the Thread Model.

In order to achieve this, the JVM would:

1. have to be “pickle-able,” by which I mean that you can run a VM, get it to a steady state ready to process transactions, and then pickle it to disk, just like a VMWare image.
   
2. support sharable read-only data, for all code and constant data
   
3. the ability to pass TCP endpoints around among processes, so that a dispatcher process can funnel transactions to the right JVM without a copy.
   

I know some of these things have been worked on in the past, but I wonder if some of the specialized JVM vendors who have a stake in enterprise software (Oracle/WebLogic/JRockit, Azul Systems, Sun, IBM) shouldn’t start looking at this problem again.

Yesterday I wrote about the pitfalls of trying to do Distributed Lock Management. Well, it turns out my friends at Terracotta have been hard at work on an awesome management tool to let users detect the problem in a console. Nice job! I wonder, however, how easy it will be for a developer to spot the problem in the case where there are more than two locks involved in the deadlock.

What I would like to see is an automatic solution to the problem, one that detects such a deadly embrace and chooses a victim to kill. Detecting the deadlock would have to be heuristic, in the sense of watching the locks to see how long they usually take, and considering only those locks which exceed the normal holding time.

In a distributed locking case, such as with Terracotta, the deadlock detector could System.exit(), let the other VM continue along, and the management system would automagically restart the victim VM. It wouldn’t prevent the problem from happening again in 10 seconds, but it might at least ring lots of bells so someone can come look at the problem, rather than having the whole cluster deadlocked. In the single VM case, we’d have to wait for a proper solution to Thread.stop(), which I also talked about yesterday.

The other question I have about deadlock detection is whether some of it can be done via static analysis, but this is not my area of expertise. An alternative would be to use AOP to instrument the locks. I’m sure someone has already done this.

Well, after getting a much deserved woodshed-ing from Hal Hildebrand for misunderstanding his post on what he’s been struggling with, I have been thinking and chatting with people about what a a “system programming” or “infrastructure” edition for Java might contain.

One of the characteristics that makes application infrastructure so unique is that it is a container for other people’s random code. Java generally does a pretty good job of dealing with loadable code by providing such features as class loaders, a code security model, a component security model, and a threading architecture.

There are, however, still a number of robustness issues with Java as it exists today:

• hot code relaoding – this is an age-old robustness problem, especially for operational issues like rolling upgrades. and I think the Zero Turnaround guys may have a splendid general purpose solution in their Java Rebel product.
• runaway thread healing – These are threads that go into an endless loop or permanent I/O block. We used to have the ability to set an ExecuteThread timeout in WLS, whereby a watchdog timer would kill any thread that didn’t complete within a configurable time period. But then Sun deprecated Thread.stop(), and suggested instead cooperative thread death using a state variable. This abdication of responsibility for robustness from the VM to user code is similar to the cooperative transaction manager timeout, about which my colleague Pete Holditch says as
  

  There is no easy answer - there isn’t really a facility in J2SE or J2EE as they stand today to allow a thread to be safely and asynchronously terminated.

  I’d like to see a permanent solution to this problem, even if it means implementing transactional memory in the JVM.
  

• memory quotas – Another great way to test an application server’s robustness is to leak memory. Providing a quota system that limits (hopefully, heuristically) the ability for a component to allocate memory would prevent bad code from killing the whole server with an OOME.
• deadlock management – Before you go hitting the “comment” button, note that I went through Distributed Lock Manager hell in VMS 25 years ago, so I know the pitfalls here. Nevertheless, Azul has done some great stuff (pdf) in this area, and I think its ripe for attention.

So rather than just complaining, here are some real-life problems that Java Platform, Infrastructure Edition could solve.

A colleague recently detailed the trials and tribulations of a piece of infrastructure he is building in Java, and it seems that after beating his head against the wall, he won, but not without a few scars.

The thing that I have always loved about Java is its productiveness and expressiveness. I could write scads of code that worked well and correctly very quickly.

But Hal has indirectly pointed out a failure of Java that I think has been true since since its earliest days when I started using it in mid-1995:

Java is a lousy language for building infrastructure.

All of the techniques that Java exposes to operate on software itself are complex, slow, or bloaty, including reflection, proxies, class loading, weak references, bytecode manipulation, and compiler APIs. Since many of the pieces of WebLogic software I wrote had to deal with other user’s users’ code (Servlets, plug-ins, etc.), I spent a fair bit of time arm wrestling with Java, either trying to find the right technique, implementing a workaround, doing code generation, or chasing performance problems. When I transitioned to the chief architect role, the WebLogic team members experienced many of the same problems while writing EJB and RMI in Java (Sun’s RMI was in C). I also think much of the excitement over AOP was a result of people being fed up with using Java for system programming; AOP cut across these problems by bypassing or augmenting the language itself. Hal’s post tells me that the problem is still not solved.

This is not to say that I don’t think Java is a good language for server-side programming – after all, WebLogic pioneered Java on the server – but rather that the focus on “enterprise edition” has been on server-side application code, rather than infrastructure code.

IBM had a variant of PL/I called PL/S that they used internally for writing writing system software. We have editions of Java for the client, for enterprise apps, for mobile, and for real-time. Perhaps we need an edition of Java for infrastructure software.

Java Platform, Infrastructure Edition anyone?

Update: Here’s Hal’s response.

Please read: The Twitter Problem(s), too.

I’ve been arguing – again – that we need a SaaS universal messaging hub that can send and receive messages among a variety of different synchronous and asynchronous message services, such email, instant messaging, SMS, RSS, and Comet.

I say “again,” because the company I co-founded in 1999, Kenamea, was originally conceived as “big message switch in the cloud,” which, among other things, solved a problem that is finally being recognized: the enormous and unproductive polling that takes place in HTTP and Ajax.

At the time, nay-sayers claimed that HTTP servers would have no problem scaling with all the zillions of Ajax requests, and of course, they didn’t, not without tweaking. No lesser a publication than WebLogic Developer’s Journal [WEG], featured a “how-to” article, My friend, colleague, and Scala/Lift maven extraordinaire David Pollak even showed us how to do it “right,” using actors. So, yes, maybe the existing httpd’s scale, but not without serious architectural work. I suspect even more work will be forthcoming, as amount of Ajax traffic continues to increase.

The place where much of the unproductive polling is taking place right now is on social networking status feeds, and the web is aflame dealing with issues.

The answer to these kinds of problems is to reverse the protocol: instead of web services, applications, and RIAs polling web sites and web services for changed information, there needs to be a big message hub in the cloud that takes in all the updates from around the web and republishes it to all the other services that are interested in those updates.

Kenamea certain had capability to do this. Using the Kenamea message switch, server applications, such as Twitter, could send messages to client applications (e.g., IM, email, web browser) without the client having to continuously ask Twitter if there were any new Tweets (which is how it works today).

If I were going to do this right now, I would base the technology on the Tervela Tervela TMX Message Switch, which, out of the box, has the ability to switch 1.5 million messages per second, with the predictability and reliability necessary to run the trading system for the largest banking institutions out there. [Note: I am an Advisor to Tervela].

In conjunction with complex-event processing systems, message switching will have a revolutionary impact on the efficiency of distribution and delivery of real-time data around the web. You heard it hear first. Again.