This year is the 20th anniversary of when I started writing XCB, and I’ve been thinking about how best to implement network protocols ever since. I have a lot to say on that topic but in this post I just want to tell the story of how XCB’s protocol specification language came to be.

This story begins over sushi lunch on the last Thursday of August, 2001. I’m not sure when Keith Packard and Professor Bart Massey began discussing how a C-language client library for the X protocol ought to work, but Bart had been passing around a paper abstract on the topic for a couple months by then. Most of XCB’s eventual design principles were already described in that abstract, even though no code existed yet. In particular, Bart wrote:

It is expected that [XCB] will be implemented as directly as possible from the X Protocol specification. It would be nice to at least partially automate the generation of the API implementation directly from the X Protocol document: this concept is currently being explored.

The person exploring that concept was me, and that Thursday I was telling Bart and Keith that I didn’t think I could get it to work. The existing human-readable documentation on the X protocol followed some consistent typographical conventions that seemed parsable, but after writing a small pile of Perl with limited success, I concluded that was a dead end.

Over lunch, Bart and Keith suggested that I could instead try writing new protocol specifications with code generation in mind. Specifically, Bart suggested that I use the m4 text preprocessor to express each part of the protocol as a macro invocation.

Of the people who have heard of m4, the most likely response to any mention of it is some combination of disgust or horror. I think most people have only experienced it in conjunction with either Sendmail configuration or autoconf, and in both cases m4 was used to paper over complexity by burying it in layers of complex macros. I don’t think it’s fair to blame m4 for that, though.

Anyway, I didn’t know any of that. So while Keith told Bart over the weekend that he couldn’t believe Bart would inflict m4 on me, and while Bart was starting to feel guilty about it, I was naively forging ahead and making it work.

XCB’s code generation continued to be based on m4 for the next three years, until Josh Triplett took Bart’s “Open Source UNIX Software Development” class and replaced all my beautiful m4 macros with a combination of XML and XSLT. Despite that, we became good friends. In turn, Josh’s XSLT was later replaced with Python, which is the combination that XCB still uses now.

Today, XCB consists of about 2,900 lines of hand-written C, and about 4,000 lines of Python that’s used only at build time. The protocol specifications total almost 19,000 lines of XML. By contrast, libX11 (which XCB was intended to replace) is about 93,000 lines of hand-written C, not counting the many libraries that implement various protocol extensions.

There has only been one CVE assigned to XCB, ever: CVE-2013-2064. By contrast, libX11 and its extension libraries have had over 50, in some cases because similar bugs were replicated across many of the libraries. This highlights the value of the Language-theoretic Security (LangSec) philosophy, which I only just learned is a specific thing people study. (See The Seven Turrets of Babel: A Taxonomy of LangSec Errors and How to Expunge Them for a comprehensive overview of security issues in protocol implementations.)

Besides reducing the amount of code to maintain and making the implementation more resistant to security bugs, making the protocol descriptions into pure data instead of code enabled new software to be built that was difficult to write before.

My favorite example is that in 2009, Wireshark’s protocol dissector for X was replaced with code auto-generated from XCB’s XML protocol descriptions. Before that, it could only dissect the core protocol, but this change allowed it to support all the X protocol extensions as well. As extensions have been added and changed over time, Wireshark has only needed to be rebuilt against newer xcbproto, and it immediately gained support for the new extensions.

Bart’s original stated goal, that we should automate the generation of the API implementation, really paid off. This was the first project where I saw the value of machine-readable protocol specifications. I’ve kept trying to replicate that result ever since, but I’ve never been completely satisfied with the available options for specification languages or methodologies.

I intend for this to be the first post in a series about protocol implementations I’ve worked on, where each one will highlight different aspects of desirable features for protocol implementers. Then I’ll wrap up the series with discussion of the connection between network protocols and an aspect of formal language theory which I recently became aware of, called “subsequential finite state transducers”.

Until then, I hope you enjoyed this peek behind the curtains of XCB’s development!