» ironclad's history

Monday, 29 December A.D. 2014 @ 7:32 PM

Like a lot of Common Lisp packages that I've written, Ironclad was inspired by somebody else. The following is my recollection of the history of Ironclad.

In one of the perennial debates on comp.lang.lisp about whether Common Lisp was a decent large for writing high-performance code, Pierre R. Mai posted an implementation of MD5 written for CMUCL that was at least competitive with implementations written in C. (This was over a decade ago and MD5 was still on the cusp of being secure.) Hash algorithms were one place where people thought Common Lisp couldn't compete with C—the requirements for efficient, unboxed arithmetic were thought to be beyond the reach of Common Lisp implementations.

Given the close history of SBCL and CMUCL, Pierre's implementation was quickly made available as an addon in SBCL's contrib/ directory. Intrepid SBCL hackers Christophe Rhodes and Alexey Dejneka also figured out how to compile portable Common Lisp implementations of modulo-2^32 arithmetic into efficient machine code. Obviously, the portable implementation wasn't going to be equally efficient on all implementations, but it was a good starting point.

Being in college, and needing something to avoid doing work for my classes, I started writing an implementation of SHA-1 that could be added as a contrib to SBCL. It was rejected, for reasons that I can't recall, but seemed reasonable at the time. However, it also set me to thinking: it would be silly to have separate MD5, SHA-1, SHA-2, SHA-512, etc. packages; it would be difficult to swap out implementations if you needed to, and each package was likely to have slightly different naming conventions, calling conventions, etc. etc. What you really wanted was a common interface for all of them.

And thus was the first release of Ironclad conceived.

I will not detail here the exact path by which bits were added to Ironclad. Hash algorithms came first, and then the big names in encryption algorithms; the AES competition was taking place around this time, so I added a few of the promising candidates from that competition. Basic, functional, insecure versions of RSA were added. There wasn't any grand plan to what algorithms were chosen: anytime I felt that my studies were too tedious is when something tended to be added to Ironclad.

Various refactorings took place along the way. The encryption algorithms and encryption modes had been implemented with macros masquerading as C++ templates and as such, took quite a long time to compile when changes were made. Changing them to better utilize generic function dispatch improved compilation and load time while maintaining performance. I distinctly remember getting frustrated when several bugs were reported in hash algorithms and having to change several different copies of code multiple times. I had cut-and-pasted code because I wasn't convinced that proper refactorings could provide the same performance, but the maintenance burden convinced me to do some benchmarking, and it turned out I had been wrong.

The best refactoring took place when I was writing non-Ironclad code and realized that I would really like to read and write integers of different endiannesses. Ironclad had this capability, of course, but it seemed silly to pull all of Ironclad in for this one bit of functionality. Thus was nibbles split out as a separate library, and slowly it gained its own set of improvements (inline assembly for SBCL, for instance), which in turn improved Ironclad as well.

I work significantly less on Ironclad than I used to. I still try to answer questions and investigate bug reports filed on Github, and an occasional week of nights spent hacking will produce something useful. But personal factors (e.g. no longer being in college) have lessened my motivation to write Ironclad particularly and Common Lisp generally.

There have also been numerous changes in the cryptographic landscape over Ironclad's lifetime. Increasing emphasis has been placed on not merely having secure algorithms, but implementing them in a secure way. Ironclad blatantly ignores much of the received wisdom about implementing algorithms securely, and a thorough audit would turn up many rookie mistakes. (See, for instance, the list of best practices at cryptocoding.net.) Not to mention that some of the techniques for implementing algorithms securely would be well-nigh impossible to implement in portable Common Lisp (bit-masking of raw pointer values comes to mind). I don't have any good solutions to this problem in general; I suppose you could write appropriately unportable code for those implementations that expose the necessary bits, and provide wrappers to known-good crypto implementations in C. But half the fun of Ironclad was doing all of this in Common Lisp, rather than writing FFI code.

The public-key algorithm implementations are particularly bad; improving those would be a lot of work. All the cryptographic tricks alluded to above are likely required, and then there's also things like writing your own bignum library, since implementations's bignums were likely not written with cryptographic applications in mind. I simply do not have the inclination to track down all the relevant standards (for padding messages and the like), academic papers (for telling you how not to do things), and reading other crypto implementations (for showing you how things might be done properly, if you can puzzle out the why from the lack of comments) to do the requisite work.

However, it is clear that Ironclad has been useful despite its many flaws in this area; the patches and the well-wishes I have received over the years are abundant evidence of this. If people wanted to improve Ironclad, the following suggestions may prove useful: