Information wants to be free (as in speech)

I know when this blog started life it was meant to be about writing software. But I write software so much that writing about writing software doesn't always have as much appeal as you might think. So I write about other stuff.

Often stuff that annoys me. Is it me, or is there no shortage of that?

From the particularly clueless department comes the US Congress (why is it parliaments keep generating so much nonsense?) which is considering enforcing a standardized DRM (digital rights management) scheme on the music industry. Why is this suddenly so pressingly important? Well apparently, because of "inconvenience for consumers". Frankly, that looks to me to be a load of cobblers, it seems consumers have no inconvenience buying an iPod and using iTunes :-). Rather, all of Apple's competitors who have missed the boat would stand to benefit. How convenient.

Meanwhile, proprietary document formats, not least of all Microsoft's Office formats show, show where real inconvenience (and far worse) lie not simply for consumers, but for entire industries.

So where is the US Congress interest in mandating that document formats used for government business, or required for public purposes (such as the common legal requirement that records be maintained for a period of time) must be in an open, published format? The US state of Massachusetts appears to be heading in this direction. Surely if moves are going to be made to enforce interoperability (which, of course, I am entirely for) then rather than addressing a minor aspect of an emerging market, it would make much more sense to address a genuinely important aspect of interoperability, namely office formats.

Why is this important? Two reasons really. One impacts on the creator of documents. In essence, when using a proprietary format, use of that content even by its creator is ultimately determined by the developer of the piece of software used to create it. Sure, the license terms of that software, or its fundamental operation are unlikely to change from day to day. But try opening a 10 year old word document in the latest version of word. Does it open an utterly faithful version of the document as created? Chances are that it won't. Is this really important? I'll let you decide for your own purposes.
And in another 5 years will it open at all?
No problem you might reply, I'll just run the old version of the app I used to create that document. While that might be feasible for the next 5 or 10 years, is it feasible for the next two or three decades?

Open published standardized formats provide a guarantee which no software developer (none, not even the very biggest) can. That your work will be accessible (in essence continue to exist) in perpetuity.
As documentation becomes increasingly electronic, this is not simply a luxury, it is fundamentally important. Our culture is really what we create. If we lose the ability to read, to listen to, to experience our cultural creations, we lose much of that culture.

There is a second reason why open document formats are profoundly beneficial to the entire online world, and increasingly the entire developed world. They provide a platform for innovation. By standardizing document formats, users are able to exercise genuine choice over the software they use, rather than being locked into whatever choice they or someone else made at whatever point they made it. We are all then much less dependent on the choices software developers make which might have more to do with their strategic advantage than it does about our needs as users.

OK, so how in theory do open formats benefit this? And in practice is there any evidence that this is not so much smithsonian (invisible) hand waving?
Right now if someone sends you a word document, how do you view it? You use Word. OK, some applications will translate Word formats and make the accessible, even editable (Apple's TextEdit and Pages are an obvious example). So we have interoperability right? Only on the most trivial level. Complex aspects of any document format, particularly when it is both unpublished, and liable to change are extremely difficult to make work properly. Even Microsoft has had reasonably well known difficulties when upgrading file formats, and this involved the team who actually developed the file formats in the first place, with full access to the existing file format documentation.
Stable, published document formats allow much greater and more reliable interoperability of applications like OpenOffice with your Office documents, as well as enabling smaller developers who might wish to specialize in providing particular functionality, say a useful grammar checker, that would work with whatever Word processing application a user chose to work with to develoop such software. This is not to say that this is impossible now, simply very difficult, and risky for a software developer on two fronts. One, because there is no guarantee that the underlying document format won't change at any time, and two, because the US DCMA (Digital Copyright Millennium Act) might actually make reverse engineering those file formats illegal (regardless of your opinion of that, I'll tell you the reality of small software developers. We steadfastly avoid any situation which has even the slightest whiff of potential legal issues like this. There are several features we have no developed for Style Master precisely because of this kind of issue).

But is this not all simply hand waving and theory? Well, let's compare two significant areas of software use over the last decade.
A decade ago, the web was a niche game. While emerging in popular consciousness, few people really used it. There were few published standards, and a large number of very small developers developing browsers, editors, servers, and other web software. Microsoft launched IE with Windows 95.
Over the next decade, while we might focus on the growing and until recently seemingly complete dominance of IE as a browser, looking at the web as an entire ecosystem, we find significant flux in almost every aspect of that ecosystem. There is no one dominant server app for the web (with numerous open source and commercial applications in this space). No web editing software dominates - several applications from large companies like Macromedia, Adobe and Microsoft remain viable, while hundreds if not thousands of small developers (like us) develop viable, in many cases innovative software which interoperate with one another and with the big players, because of the stable published underlying formats like HTML and CSS.
Even among browsers, several browsers remain more than viable, despite Microsoft's seeming dominance for most of this decade. And thank goodness for that, as IE has essentially been unchanged since the release of version 6 nearly half a decade ago.

Let's compare this with the space of Office productivity software.

In 1995, while Microsoft Office was certainly a commercial success, it had far from the almost complete dominance of Office in 2005. Today, Word Perfect Office and Lotus Smart Suite and OpenOffice between them haggle over at most 10% of the Office market (accurate figures are difficult to come across). The entire decade has seen a concentration of Office's market dominance.

Why the difference? There are many, from abysmal business decisions by competitors to users familiarity with their existing application, to Microsoft's enormous marketing budget (literally billions of dollars a year).
But above all else, once you start using Office (this is largely true of other such apps too) you are locked into the product because of all the content you create in that application's proprietary format. It is a huge risk to transition to another platform, regardless of any other benefits, for the fear of losing access to your existing documents.

So, in one field, despite the enormous marketing and R&D budgets, despite their ability to attract and keep the best and brightest people, Microsoft is one player among many (dominant in browsers, but in no other aspect of the web) while in another, the utterly dominate.
The difference? In both cases all the variables are almost identical And you can't accuse MS of not trying to dominate the web the way it has Operating Systems and Office apps. The difference is that on the web there are open, standardized document formats (HTML/XHTML, CSS, JPEG and so on) and other open standards (protocols like HTTP) while in the Office space there are no such levelers of the playing field.

There is a third and possibly most important reason than either of these, why open document formats are profoundly important.
The network effect.

Google alone indexes in excess of 8 billion web pages. Increasingly Google indexes not just HTML based pages, but many other kinds of format, among them Office formats. But this indexing suffers from, in the case of closed formats like office, similar problems as faced by developers of applications seeking interoperability with Office documents. Formats can change at any time, and Google must reverse engineer them to gain access to their content (my guess is they most likely index the plain text available in Office formats, with reference to little if any structured markup, like headings). This diminishes the value of content in such formats, and will increasingly so as search engines get smarter at using document semantics to help boost result relevance.

If, like me, you believe "information wants to be free", then this is about much more than the ability of markets to function as they should to provide us with solutions. Using proprietary formats constrains the freedom and value of information that is at odds with the world of the web.

So what can you do if you care?

Simple really. Use open published document formats as much as is feasible. Don't lock others into your file format choices by sending them proprietary formatted documents (this goes for any document format, not simply Microsoft's). Where feasible, send back any proprietary formatted documents to whomever sent them to you, asking they simply send them as plain text, or the appropriate open format.
Investigate alternatives to your current software which will read and write open standard formats.
Make this an issue when making decisions in future about system choices.

Many of us however have significant investments in proprietary formats, so making the change will require an effort, there is no doubt. But the benefits to us individually, and more importantly to the world of information will be profound.

Plus it is going to happen. Depend on it. You may as well get on the train early and get a good seat.