Introduction rewritten on: 2024-02-05
Clojure Experience Report
These days, it is entirely possible to go through an entire career as a software engineer without ever encountering a fundamentally different way to write software. We have settled on a single basic methodology: the programmer constructs a program by writing text into a "source file", which will be translated by another program into a format suitable for execution by the computer. This methodology has two key components: first, the format in which the program is constructed by a human is textual; and second, there is a separation between the "source code" produced by the programmer and the "executable" that the computer actually runs.
We use text to represent programs for good reason: no-one has ever been able to come up with an alternative that can effectively express the highly abstract concepts involved in programming. Graphical programming systems do exist but they tend to be quite restricted. Scratch is an excellent educational system that allows the programmer to construct programs by graphically manipulating blocks. However, its limitations quickly become apparent if you try to use it to build complex software; many schools in the UK start teaching programming using Scratch but the national curriculum now requires them to introduce textual programming languages too. Besides computing education, graphical programming systems are sometimes used to allow "non-programmers" to do some limited programming. This is particularly popular in electronics — where systems such as Flowcode may be used to program microcontrollers — and especially in industrial computing, such as assembly lines in factories, where Programmable Logic Controllers are routinely programmed using ladder logic. In all such cases, though the computers may be part of a large and complex electromechanical system, the software running on each microcontroller is both relatively simple and has a predictable structure. Where the problem space is sufficiently constrained, solutions may be specified in a convenient graphical language. However, all successful general-purpose programming languages are textual.
In contrast, the reasons behind the separation of "source code" from the "executable" are less obvious. Of course, the requirements for the efficient operation of a computer's processor and the requirements for comprehension by humans are quite different — there are certainly good reasons why we do not program directly in machine code — but those requirements constrain the programming interface rather than the programming methodology. There is no reason why a programmer could not directly edit a running program: if the machine code and the source code are two different representations of the same program, an edit made to the latter could be immediately reflected in the former. Indeed, this is essentially how two programming systems developed at Xerox PARC in the 1970s worked: both Smalltalk and Interlisp had the concept of an "image", a complete environment consisting of both the running code and the means to edit that code.
The Interlisp-D manual describes Interlisp as a "resident" programming environment. Rather than editing source files on disk that you then compile to create a separate executable — or even, as with modern dialects of Lisp, separate source files on disk that you then load into a running Lisp program — an Interlisp programmer would edit the running code in memory using the SEdit structure editor, which was included in the Interlisp environment. The source code would be saved to disk, in what Interlisp called "symbolic files", but this would be managed automatically by the Interlisp system. Likewise, a Smalltalk image contained both a source code database and a powerful editor — more like a modern IDE than a simple text editor — along with the running application.
This sort of system encourages an interactive style of software development that is unfamiliar to most programmers today. Since even experienced Smalltalk programmers struggle to explain what it is like to those who haven't tried it, I won't even try. However, most importantly, both of these programming systems, which are held up today by those who knew them as paragons of interactive programming, provided a single source of truth for the contents of a program. In both Interlisp and Smalltalk, the programmer never directly edited source files on disk: the files on disk were merely the saved state of the program that was interactively modified by the programmer within the running environment. In effect, source code and machine code were always just two different views of the same program.
This is quite different from systems that separate the source code from the executable program, in which there is always a danger that the two may diverge. This is particularly likely if we do not compile the entire program from scratch every time — perhaps because it is too slow to do so — and instead split the program into components which are compiled separately and then linked together to form the complete executable. In that case, it is all too easy for a programmer to edit the source files for two components but only recompile one of them; or to recompile all the changed source files but forget to relink the results. It becomes even more confusing when some parts of the program depend on other parts: perhaps the programmer has only edited one source file, but other parts of the program depend on the code defined in that source file. In that case, some source files need to be recompiled even though they themselves have not been changed. There are many ways in which the final executable might not accurately reflect the content of all of the source files. This can be extremely frustrating for the programmer: they may spend hours trying to debug a failing test, when in fact the bug has already been fixed but the relevant source file has not been recompiled. Perhaps worse, all the tests may pass when in fact the source contains a bug.
In 1976, Stuart Feldman solved this problem by writing a program called Make. This program keeps track of dependencies: if the programmer changes one source file, Make knows not only to recompile that file but also which other files depend on that file and so must be recompiled in turn. After building a program with Make, we can be confident that the resulting executable is entirely consistent with all of its source code.
Stuart Feldman described the event that prompted him to create Make:
Make originated with a visit from Steve Johnson (author of yacc, etc.),
storming into my office, cursing the Fates that had caused him to waste a
morning debugging a correct program (bug had been fixed, file hadn't been
compiled, cc *.o was therefore unaffected). As I had spent a part
of the previous evening coping with the same disaster on a project I was
working on, the idea of a tool to solve it came up. It began with an elaborate
idea of a dependency analyzer, boiled down to something much simpler, and
turned into Make that weekend.
Make was released with V7 UNIX in 1979 and since then we have been largely free of such disasters. New build tools have been written which improve on Make in various ways, but most programs today are built with some sort of dependency tracking that ensures that the executables we run stay in sync with the source code that we edit.
However, there has recently been a renewal of interest in interactive programming. Perhaps the most popular of the new interactive programming environments are Jupyter Notebooks. These are commonly used by data scientists for exploratory work: their interactive and graphical nature make them particularly well-suited for that. They consist of a continuously running Python process along with a number of cells containing Python code. When a cell is evaluated, the code within it is executed by the running process. Then the results, which can be textual or graphical, are displayed. By interleaving explanatory text with the Python cells and their results, data scientists can create a sort of interactive document: it's a data science experiment that contains both the code and the results and can be re-run with the click of a button. It's quite a cool idea... but it has some issues. In particular, data scientists can evaluate any cell at any time and in any order. They can also edit the Python code in a cell and not evaluate it. Since all cells are executed within the same continuously running Python process, so each cell has access to the results of any cell that has already been evaluated, and they can all implicitly depend on each other.
Joel Grus gave a great talk at JupyterCon in 2018, entitled "I don't like notebooks". During the talk, he says that his "number one fundamental complaint" about Jupyter Notebooks is that "notebooks have tons and tons of hidden state that's easy to screw up and difficult to reason about". He provides multiple examples of notebooks in which the result of evaluating cells is not consistent with the source code visible in those cells. Sometimes this is because the source code has been edited but not yet re-run. Sometimes it is because a different cell, whose code depends on the result of a cell that has been edited and re-run, has not itself been re-run. In effect, Jupyter Notebooks have recreated the "disaster" that prompted Stuart Feldman to write Make more than 40 years ago.
I say all this because I found programming in Clojure to be a lot like working with Jupyter Notebooks. Clojure is a sort of Lisp and the recommended programming methodology is to use interactive programming: that is, you keep a Clojure process continuously running, then use a plugin for your favourite text editor or IDE to send Clojure forms to that process to be loaded. In fact, this method of programming is not just recommended: it is pretty much required. While you can at least work around the issues with Jupyter Notebooks by periodically restarting the Python process and evaluating all cells in order from the beginning, this isn't really possible with Clojure. Python starts in tens of milliseconds at most but Clojure takes seconds. That's for an empty program: once your Clojure program has progressed beyond "Hello World", it is not unusual to have to wait tens of seconds for it to start. A thirty second pause every time you make a change to the source code and want to see the result does not make for a productive developer experience, thus interactive programming is required.
In the absence of dependency tracking, I found myself frequently getting very frustrated with Clojure. For example, I would write some tests, they would all pass, then I would refactor my code, the tests would still pass, so I would carry on with my work. Later, perhaps when I pushed my changes to CI, all the tests would fail. It would turn out that they only passed because the previous version of my code — the code before the refactor — was still resident in the running Clojure process. My tests weren't actually testing the code in the source files at all. I was bitten by some variety of this problem nearly every day, which did not endear Clojure to me.
It seems that I am not the only person to have suffered in this way. Referring to the continuously running Clojure process as the REPL, an experienced Clojure programmer wrote in A Clojure REPL Workflow for Beginners:
Once you start getting used to using a REPL, you also need to learn how to keep some of the REPL state in your head. Keep in mind what you've evaluated and what you haven't. Not knowing is how you quickly have bugs where you think the code has been evaluated and your function definition updated but [it] has not.
Similarly, the documentation for Leiningen, a popular build tool for Clojure, says:
Keep in mind that while keeping a running process around is convenient, it's easy for that process to get into a state that doesn't reflect the files on disk: functions that are loaded and then deleted from the file will remain in memory, making it easy to miss problems arising from missing functions (often referred to as "getting slimed").
Two things strike me about this: first, it's kind of amazing that a new programming language, invented in the 21st century, could suffer from this basic problem that was solved in the 1970s. Even more amazing is that so many people seem to just sort of accept it, as if requiring the programmer to keep track of program state in their head is a totally normal and reasonable thing to do.
Second, it's interesting that the Leiningen documentation refers to this problem as "getting slimed". It is a reference to SLIME, the Superior Lisp Interaction Mode for Emacs. This is an extension for the Emacs text editor that integrates it with a Common Lisp REPL: it allows the programmer to load Lisp forms into a running Lisp process. I assume this comes from the early years of Clojure, when Emacs users would try to use SLIME to work with a Clojure REPL. (These days, people writing Clojure code in Emacs have their own extension, CIDER, that does a similar job to SLIME but targets Clojure rather than Common Lisp.) It's interesting because I haven't been able to find any references to "getting slimed" that pre-date Clojure: do Common Lisp programmers not have this problem? What do they do differently? There's a whole different tradition of Lisp programming that came out of MIT around the same time as Interlisp was being developed at Xerox Parc — with Lisp Machines and Symbolics and the Texas Instruments Explorer — which eventually resulted in Common Lisp. I don't think these systems were "resident programming environments" in the same way that Interlisp and Smalltalk were, so how did they avoid "getting slimed"? Maybe they didn't. Maybe they just didn't have a name for the problem.
Unsurprisingly, Clojure programmers haven't just resigned themselves to
constant sliming. Perhaps most famously, Stuart Sierra's blog article,
My
Clojure Workflow, Reloaded, describes how he ensures that he can restart
his application cleanly without having to restart his REPL. The key innovation
was clojure.tools.namespace,
which is a library for dependency tracking of Clojure code. To help structure
applications such that they could be cleanly restarted without actually
restarting the Clojure process itself, Stuart Sierra developed a library for
lifecycle management called
Component.
Later, other people created alternatives to Component such as
Mount and
Integrant.
So if I were to program in Clojure again now, I should be able to avoid the pain of "getting slimed". I wish I had understood all this sooner though: my initial experiences with Clojure really soured my opinion of the language. It also didn't help that when I went looking for advice about this, the most common response I found in Clojure forums was, "Yeah, you have to get used to that." I struggled to understand why anyone would put up with such a frustrating system. The answer seems to be that people think it is a price worth paying in exchange for the "interactive programming" experience, which Clojure programmers tend to call "REPL-driven development". Practicalli says:
REPL driven development is the foundation of working with Clojure effectively. The REPL is an instant feedback workflow that continually runs your code without the need to manually run an explicit compile-build-run cycle.
But I never really found it that great. It seems you have two options: either
you use something like clojure.tools.namespace/refresh to reload
your code after you make a change, in which case the programming experience
isn't really any different from the old edit-compile-test loop; or you use
your editor's REPL integration to reload individual Clojure forms into a
running process, in which case you are vulnerable to "getting slimed". So what
benefit does the latter bring to make the risk of sliming worth it?
When developing an application with little state — such as a web backend, in which each HTTP request is handled individually — there doesn't seem to be any great advantage to REPL-based development. Using the Clojure REPL, you could change a handler function and reload it. But why is that better than recompiling the webserver and restarting it? Sure, the Clojure compiler is slow and Clojure programs start excruciatingly slowly, so a full recompilation and restart would be painful with Clojure. But with another language, one that has a fast compiler and starts quickly, I can achieve the same thing without any danger of getting slimed. In fact, by using a file watcher such as entr to automatically recompile and restart the server whenever I change the source code, I can have my new handler function live and under test even faster than I can with Clojure and a REPL integration.
Perhaps live code reloading offers more of an advantage when developing an application with a lot of state. If you have an application in which state is gradually built up over time — especially one in which the state is modified by user interaction — then maybe you don't want to tear down all that state and start again from scratch every time you experiment with a change to the program. For example, Phil Hagelberg and Bryce Covert both claim that REPL-based development is a great help when developing games. I don't have any experience of game development but I can imagine that the REPL makes it fast and fun to experiment with different game mechanics. Then again, I would have thought that you'd have test cases and saved games so restarting the program and restoring an interesting state would also be fast. Smaller changes, such as tweaking the value of some parameter, can be tested interactively with a debugger and so don't require a REPL. In fact, a good debugger was something that I really missed when working with Clojure: it is not trivial to inspect the state of a running Clojure program, including the call stack and local variables, using a REPL. It certainly doesn't help with post-mortem debugging: I never found an equivalent to loading a core dump into a debugger.
It's possible, probable even, that I just missed the point. I spent over a year really trying to get into Clojure, including pair programming with a few experienced Clojure programmers, and attending a Clojure conference, so I think I gave it a good go, but it just didn't click for me.
Maybe that won't be the case for you. Maybe you'll love the REPL-based workflow. Let's set aside the programming methodology for now and just assume you start programming in Clojure, what should you expect?
Ease of interoperation with Java is a deliberate design goal of Clojure: by
making the vast Java ecosystem easily available to Clojure programmers,
Clojure has avoided the niche ecosystem problem that often plagues new
languages. But it comes at quite a cost: Clojure is a very leaky abstraction
over Java. In order to program effectively in Clojure, you not only need to
understand Java and the JVM, but you must also deal with Java runtime
behaviour in your Clojure programs. You might want to use Clojure's facility
for functional programming to write a neat functional data-processing
pipeline... but some underlying Java method will throw an exception. It will
be hard to predict where or when this might happen. You must be prepared to
deal with the Java standard library and with Java's object-oriented idioms in
addition to Clojure's functional programming style. Clojure provides some
macros, such as gen-class and doto, to make this
easier, but it is still a thing you must understand and do.
In a similar way, if you are programming in ClojureScript then you must understand the underlying JavaScript platform. This also means that functions from the Clojure standard library sometimes behave differently in ClojureScript: you are expected to understand Java and JavaScript well enough to predict this.
Clojure error messages appear as Java stacktraces emitted by the Clojure runtime. They are often very long (many pages long!) and have little to do with the actual error. Eric Normand wrote:
I've recently shifted my thinking about Clojure error messages. It is more useful to think of them as non-existent than to think of them as bad... it's not so much that Clojure's errors are bad. It's more that they're accidental. Clojure's core functions are, for the most part, implemented without checks on the arguments. They only code the "happy path". They assume that the arguments are of the correct type and shape, and proceed without caution... What happens if they're not correct? That's up to chance.
As a result, it is often very difficult to tell from an error log what it was that actually went wrong.
Finally, what is the language itself like? There are a lot of different ways to do almost the same thing. For example, suppose you have one or more sequences (arrays, lists, vectors, whatever) and you would like to iterate over them, performing some operation on each item in turn. In Clojure, there are many different ways to do this, including:
mapmapvforloopintodoseq
These are all slightly different. Most obviously, into is for
creating a new data structure using the elements of your sequence,
whereas doseq is only for side effects, e.g. performing I/O using
each element (it always returns nil). But if the data structure
you want is another sequence, you could use either
map or into... but map returns a lazy
sequence whereas mapv eagerly evaluates its arguments into a
vector... or you could use for to get a lazy sequence but it
behaves differently if you have more than one input sequence.
loop is used with recur because Clojure, due to
limitations of the JVM, does not have true tail recursion: you have to use the
special loop/recur construct instead of just calling a function
recursively.
The argument in favour of all these different ways to do almost the same thing is that your choice signals intent. This may be a reasonable argument if you are using one of the functions for iterating over a sequence that I didn't include in the above list, such as:
filterkeepremovereduce
The choice of map, filter or reduce
tells me something about the sort of transformation that you want to perform
on the data. But then why do we also have into?
Is keep preferable to composing map
and filter? I haven't even mentioned transducers yet: what should
I infer from your decision to use
reduce rather than transduce? Perhaps most
importantly, I will never correctly infer your intent if you use a function
that I haven't seen before... and there are a lot of them. I can always read a
simple loop but for anything unfamiliar I will have to trawl
through the documentation, or experiment in the REPL, to try to work out what
it was that you were trying to achieve (and I still won't be sure why you did
it this way rather than one of the dozens of other possible ways).
The Clojure standard library is full of this sort of thing: many functions, sometimes subtly different, all with overlapping functionality. For any given algorithm, there are umpteen different ways to implement it.
This contrasts with the idea of "orthogonality" that guided the design of UNIX. Rob Pike described this idea in the context of programming language design:
You have to pick the right set of features. Not just features for features' sake. The way I like to think of this is to think of the entire world of programming as a vector space of very high dimension. And what you want to do is to find the basis set that covers that vector space. So that you can write the programs you want by combining the appropriate, orthogonal (because that's what a basis set is) set of features. And what happens when you add features for expressiveness or for fun, is that it tends to add more non-basis vectors into that space, so there become many paths to get to a particular solution.
There are a lot of non-orthogonal features in Clojure. Some might object to my calling them features — in typical Lisp style, they're mostly library functions and macros rather than special language features — but from the point of view of a developer using the things, the effect is the same. There is a fundamental difference between a standard library providing many different functions for, say, serializing data to different formats, and a standard library providing 37 different ways to iterate over a list.
I found it interesting to read an interview with Rich Hickey, the creator of Clojure, about his motivations:
I was a C++ developer for a long time. I taught it at NYU for a little while. I worked in scheduling systems and broadcast automation all in C++. Then I moved from that to Java and C#... I discovered Lisp after ten years of C++ and said to myself, "What have I been doing with my life?" I realized it was just much more productive for me. I fell in love with it right away... I spent several years subsequent to that continuing to do my commercial work in Java and C#, but all the while hoping at some point to be able to get Lisp into the mix. That's what drove me to doing Clojure, because I saw that either .NET or the JVM were requirements for all the customers I was encountering. Lisp really couldn't reach those platforms, so I decided to make a Lisp that could.
That makes a lot of sense. If the JVM is a hard requirement, then Clojure is a reasonable choice. If you are already experienced in Java and heavily invested in the JVM, then Clojure's entanglement with the host platform is a benefit rather than a cost. Further, if you really love Lisp, then Clojure may be the obvious choice for a JVM language rather than, say, Kotlin.
However, if the JVM is not a requirement, then the argument in favour of Clojure is weaker. If you love Lisp and are happy with a large, featureful version, then why not use Common Lisp? If you prefer something more minimal, why not Scheme or Racket? If it's functional programming that you want, how about OCaml? Without the JVM constraint, the benefit of Clojure is not so obvious.
Personally, I became interested in Clojure after watching Rich Hickey's StrangeLoop 2011 talk, Simple Made Easy. It is a great talk. Regardless of your opinions on Clojure, I highly recommend it. At the time, I thought I saw similarities between Rich Hickey's idea of simplicity and the UNIX idea of orthogonality. However, although there are some similarities, in the context of programming language design they have played out very differently.