Vendoring with Quicklisp, Make, Git, and Nix
Max Rottenkolber <firstname.lastname@example.org>
I have a pet project called Athens (source), it is a server application built with CCL. What it does is entirely irrelevant for the topic at hand. What follows are my thoughts on dependency management for Lisp applications, and my experience with some tools—old and new—that solved my woes.
For being a seemingly simple application, Athens has an awful lot of dependencies of different kinds. A dozen primary dependencies include both third-party systems from Quicklisp, as well as homegrown systems not in Quicklisp. These primary dependencies pull in countless secondary dependencies from Quicklisp, which in turn depend on C libraries provided by the OS such as OpenSSL and LibXML2. Ouch.
I want building Athens to be trivial and predictable. No chasing down dependencies. I also want to be in control of when dependencies are upgraded, and be able to reproduce build and test environments faithfully. I am fine with only supporting the Unix-like operating systems supported by CCL, so platform support is restricted to those.
Making things easy
Athens has a human-readable Makefile that describes how the application is built. You can
cd into its source directory and type
make to produce an executable (which includes the CCL kernel and the Athens Lisp image) which will act like a reasonable Unix citizen.
Right now, its build dependencies (minus ubiquitous core utilities and GNU Make) are CCL, OpenSSL and LibXML2, the latter two of which are also runtime dependencies that are dynamically loaded as shared objects.
The Make rule to build Athens invokes
ccl to run the actual build script which is written in Lisp. It also loads a project-local Quicklisp installation! Athens ships with Quicklisp, and includes all its Lisp dependencies checked into the repository as part of the Quicklisp installation. The release manager (me) makes sure all is in order on the
Athens vendors a bunch of homegrown and pinned dependencies (currently
xmls-1.7, because its API changed and I have not found the time to restore compatibility in my downstream library, yet) in
lib/. The remaining third-party dependencies are managed via the fabulous Quicklisp. The above rule installs Quicklisp into the source tree and links all non-Quicklisp systems to its
local-projects/. In order to upgrade to a new Quicklisp dist I would do
…and check in the resulting
quicklisp/ tree into the repository (after testing that the resulting Athens binary built with the new dist works as expected.) To avoid excessive repository bloat, Athens has some Quicklisp specific patterns in its
I manage the vendored dependencies in
lib/ (including quicklisp-bootstrap) with git-subtree when possible.
Nixing the build
In addition to its
Makefile, Athens comes with a default.nix file that enables it to be built by Nix. To build Athens using Nix you can do
…which performs the build in tidy environment with all build dependencies provided, and produces a so-called Nix derivation of Athens. Likewise, Athens (and its runtime dependencies) can be installed in a Nix environment via
We can also interactively test Athens and its build in a controlled environment:
By default, Nix will use the dependencies from the active system or user environment, but it does not restrict you to your current system. On of its cool features is that you can reproduce the build environment for any previous or future version of nixpkgs (a Git repository of Nix expressions for every piece of software distributed by Nix.) To test Athens with the most recent nixpkgs you could do
default.nix contains a few Lisp specific bits that warrant an explanation. It starts out by declaring its build dependencies. Since Athens is a CCL application this includes
ccl and its dependencies
libxml2, as well as a utility
makeWrapper provided by Nix (to be explained later.)
Runtime dependencies are declared by inheriting
libxml2 (along with the version string for this build of Athens) into the derivation.
preBuild hook specifies some steps to ensure a clean build. CCL will try to load shared objects for OpenSSL and LibXML2 provided by the operating system (on NixOS these won’t be present in the usual locations at all.) To ensure that it will only use the shared objects specified for the build we set
LD_LIBRARY_PATH accordingly. The remaining bits tell ASDF where to cache compiled Lisp code during the build by setting
XDG_CACHE_HOME to a suitable location, and to cease its attempt to load system and user configurations for the source registry (which could mess with the build) via
Nix will implicitly build software using
make, and that works just fine for Athens so we can get away without declaring a
Makefile does not have an
install target though, a custom
installPhase is needed. This is also where
makeWrapper comes into play. When the CCL kernel loads the Athens Lisp image, it will dynamically open the shared libraries used by Athens (take a look at CCL’s REVIVE-SHARED-LIBRARIES function for the gritty details.) To make sure it finds the share objects specified in the derivation, the Athens executable is replaced by a shell script that again sets
LD_LIBRARY_PATH and then
execs the actual executable.
Finally, we need to inhibit Nix’s default behavior of stripping ELF symbols from the resulting executables. This just happens to lobotomize the executable produced by CCL.
So where does this leave us? If you are a user of Nix then you can let Nix figure out the dependencies and produce a clean build of Athens easily. Or put alternatively, Athens is left with only a single dependency: Nix. Of course, it is still possible to build and run Athens without Nix by manually resolving its its dependencies, and using the Makefile directly.