Java, open source, standards, and conformance

[Revised 5/30/04]
My colleague Simon Phipps has just posted a lengthy piece entitled On Java and Openness. I agree with him pretty much 100%.
A lot depends on whether you approach compatibility and interoperability from a “standards” position or not. The “standards” viewpoint is that there are going to be multiple implementations, that this is a good thing, and that you need to make sure that these implementations can interoperate. This was always the IETF approach: a proposed standard with only one implementation was viewed as problematic, because you couldn’t distinguish between the intention of the standard and the accidents of implementation.
I started at Sun in 1985, and my first job was to do an implementation of NFS for MS-DOS from spec. All of the previous implementations (for various types of Unix as well as VMS) had been based on the source code, so I guess that I made Rusty Sandberg and the other spec-writers pretty nervous. But it worked. Later I worked with folks from FTP, Microsoft, and JSB to develop the Windows Sockets specification, and we recognized from the start that the conformance verification model was going to be key to acceptance.
I’ve always taken an object-oriented approach to standards: I believe that a standard is an object with one method, bool Conform(implementation i). If an implementation, i, conforms to the standard, I can use it. Of course this presupposes that the standard is adequately specified and the Conform() method is trustworthy, but that’s why doing standards is hard work. It also explains why the Reference Implementation (RI) is such an important concept – that if (or rather when) the language of the standard proves inadequate or ambiguous, there is an authoritative answer: the standard is what the RI does. Critically, the RI is not supposed to be the only implementation; it should be optimized for clarity and conformance rather than performance, size, efficiency, etc.
Everything that I’ve seen about the open source movement suggests that it is designed to encourage group participation on a single code base, rather than the creation of multiple independent implementations. Where’s the 100% compatible clone of Perl, or Apache or Sendmail? In that respect (paradoxically?), open source leads to a monoculture, just as much as Windows does. When you actually have multiple implementations, you have to face the question of whether compatibility is important or not. For platform technologies – systems that other people rely on to make their software work – the answer seems to be yes. And despite the absolutists that Simon cites, there is no evidence whatsoever that a free market approach can sustain compatibility, except by preferring one choice and allowing the others to wither. To me, monoculture – even a free monoculture – seems dangerous. Diversity is good.