In his latest polemic, Herr Hildebrand interprets my robustness problems as a dichotomy between the Thread and the Process. (Just joking, Hal!) Like earlier authors, I will also refrain from providing precise definitions, except to bastardize Justice Potter Stewart’s old saw, “I know a [process|thread] when I see one.”

Well, Hal is exactly right that the process model has huge robustness benefits over threads due to the memory and scheduler isolation enforced between processes by the OS and MMU. He is also correct that much robustness is lost when running many applications in a multi-threaded environment (like the JVM), creating operational behavioral dependency among applications that didn’t exist at design time.

But rather than bowing to the false architectural dichotomy between the Process Model and the Thread Model, I propose an architecture that uses each of them to their best advantage, using processes to separate applications, and threads to separate transactions within applications, and also proposing a number of improvements to the JVM that would make this hybrid model as cheap to run as the monolithic JVM model we currently had.

Before continuing, let’s imagine for a minute a JVM that has some special properties:

1. it has no transactional state, which most people would read as “share nothing”
   
2. the transactional state is cached locally, and the ACI (but not D) cache is shared across all processes via shared memory
   
3. a fully operational JVM can be launched from scratch, ready to process transactions, in, say, under a second
   

With such a JVM, it is not hard to imagine the proposed solution: the runtime architecture consist of one jar file (i.e., one application) per JVM, and that each JVM handles multiple simultaneous transactions in separate threads. An errant application could then be recycled by recycling the entire JVM around it. For greater robustness within a single application, multiple JVMs could run the same jar.

What you now have is a mini-cluster that provides the robustness of the Process Model with the multiprocessing and “forking” speed of the Thread Model.

In order to achieve this, the JVM would:

1. have to be “pickle-able,” by which I mean that you can run a VM, get it to a steady state ready to process transactions, and then pickle it to disk, just like a VMWare image.
   
2. support sharable read-only data, for all code and constant data
   
3. the ability to pass TCP endpoints around among processes, so that a dispatcher process can funnel transactions to the right JVM without a copy.
   

I know some of these things have been worked on in the past, but I wonder if some of the specialized JVM vendors who have a stake in enterprise software (Oracle/WebLogic/JRockit, Azul Systems, Sun, IBM) shouldn’t start looking at this problem again.

Yesterday I wrote about the pitfalls of trying to do Distributed Lock Management. Well, it turns out my friends at Terracotta have been hard at work on an awesome management tool to let users detect the problem in a console. Nice job! I wonder, however, how easy it will be for a developer to spot the problem in the case where there are more than two locks involved in the deadlock.

What I would like to see is an automatic solution to the problem, one that detects such a deadly embrace and chooses a victim to kill. Detecting the deadlock would have to be heuristic, in the sense of watching the locks to see how long they usually take, and considering only those locks which exceed the normal holding time.

In a distributed locking case, such as with Terracotta, the deadlock detector could System.exit(), let the other VM continue along, and the management system would automagically restart the victim VM. It wouldn’t prevent the problem from happening again in 10 seconds, but it might at least ring lots of bells so someone can come look at the problem, rather than having the whole cluster deadlocked. In the single VM case, we’d have to wait for a proper solution to Thread.stop(), which I also talked about yesterday.

The other question I have about deadlock detection is whether some of it can be done via static analysis, but this is not my area of expertise. An alternative would be to use AOP to instrument the locks. I’m sure someone has already done this.

Well, after getting a much deserved woodshed-ing from Hal Hildebrand for misunderstanding his post on what he’s been struggling with, I have been thinking and chatting with people about what a a “system programming” or “infrastructure” edition for Java might contain.

One of the characteristics that makes application infrastructure so unique is that it is a container for other people’s random code. Java generally does a pretty good job of dealing with loadable code by providing such features as class loaders, a code security model, a component security model, and a threading architecture.

There are, however, still a number of robustness issues with Java as it exists today:

• hot code relaoding – this is an age-old robustness problem, especially for operational issues like rolling upgrades. and I think the Zero Turnaround guys may have a splendid general purpose solution in their Java Rebel product.
• runaway thread healing – These are threads that go into an endless loop or permanent I/O block. We used to have the ability to set an ExecuteThread timeout in WLS, whereby a watchdog timer would kill any thread that didn’t complete within a configurable time period. But then Sun deprecated Thread.stop(), and suggested instead cooperative thread death using a state variable. This abdication of responsibility for robustness from the VM to user code is similar to the cooperative transaction manager timeout, about which my colleague Pete Holditch says as
  

  There is no easy answer – there isn’t really a facility in J2SE or J2EE as they stand today to allow a thread to be safely and asynchronously terminated.

  I’d like to see a permanent solution to this problem, even if it means implementing transactional memory in the JVM.
  

• memory quotas – Another great way to test an application server’s robustness is to leak memory. Providing a quota system that limits (hopefully, heuristically) the ability for a component to allocate memory would prevent bad code from killing the whole server with an OOME.
• deadlock management – Before you go hitting the “comment” button, note that I went through Distributed Lock Manager hell in VMS 25 years ago, so I know the pitfalls here. Nevertheless, Azul has done some great stuff (pdf) in this area, and I think its ripe for attention.

So rather than just complaining, here are some real-life problems that Java Platform, Infrastructure Edition could solve.

A colleague recently detailed the trials and tribulations of a piece of infrastructure he is building in Java, and it seems that after beating his head against the wall, he won, but not without a few scars.

The thing that I have always loved about Java is its productiveness and expressiveness. I could write scads of code that worked well and correctly very quickly.

But Hal has indirectly pointed out a failure of Java that I think has been true since since its earliest days when I started using it in mid-1995:

Java is a lousy language for building infrastructure.

All of the techniques that Java exposes to operate on software itself are complex, slow, or bloaty, including reflection, proxies, class loading, weak references, bytecode manipulation, and compiler APIs. Since many of the pieces of WebLogic software I wrote had to deal with other user’s users’ code (Servlets, plug-ins, etc.), I spent a fair bit of time arm wrestling with Java, either trying to find the right technique, implementing a workaround, doing code generation, or chasing performance problems. When I transitioned to the chief architect role, the WebLogic team members experienced many of the same problems while writing EJB and RMI in Java (Sun’s RMI was in C). I also think much of the excitement over AOP was a result of people being fed up with using Java for system programming; AOP cut across these problems by bypassing or augmenting the language itself. Hal’s post tells me that the problem is still not solved.

This is not to say that I don’t think Java is a good language for server-side programming – after all, WebLogic pioneered Java on the server – but rather that the focus on “enterprise edition” has been on server-side application code, rather than infrastructure code.

IBM had a variant of PL/I called PL/S that they used internally for writing writing system software. We have editions of Java for the client, for enterprise apps, for mobile, and for real-time. Perhaps we need an edition of Java for infrastructure software.

Java Platform, Infrastructure Edition anyone?

Update: Here’s Hal’s response.