Analysis of existing ASDF files

Summary of ASDF data

Direct access to the automatically generated data.

Access to the discussion in comp.lang.lisp. Direct your comments there or by private email.

Background

ASDF is a system for declaring dependencies between the components of a Lisp program or library. Developed quite long ago by Dan Barlow, it was intended to be an improvement over MK-DEFSYSTEM, including an "object-oriented" architecture that allows for extensibility: from defining new kinds of components (C sources, Java sources, etc) to new operations.

However, the word "architecture" is perhaps not really descriptive of ASDF, for it basically allows you to plug anything in:

  • Define whatever operations you want, without regard for definitions found in other libraries.
  • Define :around, :after and :before methods without actual control of other people's decisions.
  • Populating the *.asd file with all kind of forms that have side effects: defpackages, loading files, creating files...
  • Expecting that the *.asd files have side effects that will be visible in your loaded or compiled files.

The list is endless, precisely because ASDF is so little specified: the system definition files do not even have a format and can contain arbitrary lisp code.

This lack of actual design may seem appealing and quite in the Lisp spirit of do-it-yourself, but it is a nightmare when one wants different pieces of software to coexist or actually take the libraries that you so carefully developed and use them outside of the ASDF world.

For instance, assume that I am a package maintainer for Debian and I want to distribute precompiled versions of all useful lisp libraries. How can I do it? I actually can not. I can precompile them, but I must do it in a hackish way, also installing all the sources for your system, registering them with ASDF and also placing a copy of your side-effect-full *.asd files somewhere to be loaded.

From a more fundamental point of view, the fact that *.asd files contain forms with side effects also prevents anybody from writing a more intellingent and more efficient ASDF. Note that ASDF has to load every *.asd file before knowing dependencies and it has to load them even if it only wants to study those dependencies. That is stupid, because this will cause all the side effects in all those *.asd files to take place, sometimes with the consequence of Lisp files being compiled and loaded, files being created or even errors signaled because of missing dependencies. All that just to build the dependency tree!

But what bothers me right now is that this structure of *.asd files also prevents other delivery and build systems. For instance those which are not based on a load-load-load-and-dump-image model. Take for instance ECL. This is the Common Lisp implementation that I develop. This implementation is extremely portable while having a decent performance, it is based on a C core, compiles using C and loads files using the dynamic linker of your operating system.

ECL can not dump memory images, because this is not portable across operating systems and even among versions of the same system. Instead we used shared libraries: each lisp file is translated to a C file which is then compiled and loaded. Multiple lisp files can be combined into a single binary, which may be shipped as a FASL, shared library, statically linked library or even an executable program. Everything is explained in the manual. All this can be done to some extent with ASDF.

The reality

I have made a simple study of existing software that is based on ASDF. The results are summarized in a companying set of tables. A preliminary conclusion of this study is the following one: side effects in an ASDF file belong mostly to a few families

  • Those that tweak ASDF
    • New components
    • New load-op / perform operations for those components
    • Loading systems that are needed for those components
  • Redundant ones that can be moved into separate files
    • Package definitions
    • Operations that are done at the end, such as PROVIDE
  • Questionable coding practices
    • Reading version numbes from files using #. forms
    • Loading manually systems instead of listing them as dependencies
    • Defining options for the files to be compiled

Ideas for the future

Edit: much of the following includes possible ways to improve ASDF, none of which are part of ASDF 2 as we know it.

Better ASDF file writing

Many of these things can be very easily fixed through better practices. For instance, it does not make sense to provide a "PERFORM :AFTER" method just to use (PROVIDE :MY-SYSTEM) when you can insert that statement in one of the files that is loaded, or simply rely on ASDF to be properly hooked to REQUIRE/PROVIDE.

Similarly, the package definitions should live in a source file that is separate and is listed in the system itself, so that it is available at run time even if the *.asd file was not loaded.

ASDF dependencies for systems

A bit more tricky is the situation with components. The first question we should answer is whether it makes sense to allow arbitrary components in system definitions.

In addition to this we must think carefuly of a better way to declare components such that ASDF gets to know which files have to be delievered with an applications and which ones are only needed when building the library or program itself.

The other question is where to place these definitions. I would advocate for a separate Lisp file that can be listed in the system definition itself, with something like

(defsystem :my-system
   :system-dependencies
 ((:file "my-components") ; for my components
          :cffi ; for other components provided by CFFI
         :cffi-grovel)

This way we could explicitely tell ASDF that we need to load other files or even packages in order to properly process the system definition.

Package-independent system definitions

It is important that system definitions remain readable even without those packages. For that we have to find a different mechanism for registering components so that instead of using

(defsystem :my-system
   :system-dependencies (:ffi)
   :components ((ffi-c:preprocessed-ffi "ffi") ...

we can use symbols without package prefixes

(defsystem :my-system
   :system-dependencies (:ffi)
   :components ((:preprocessed-ffi "ffi") ...

Listing delivered files

The system definition should include files that are not built but which are part of the system nevertheless. For instance, resource files: images, databases, documentation, etc. This information can be used to install and move the libraries in a simpler way than it is done now -- which amounts to moving everything, including sources.

Passing options to systems

One of the uses that one finds in the *.asd files is to set up options for building and compiling files. This could be replaced with different models

  • Making use of *FEATURES*
  • Just defining variables in the CL-USER package before building the program
  • Creating an ASDF API for declaring options and assigning them values
  • Defining variables in the CL-USER package and declaring default values as DEFSYSTEM options.

I personally think the last one is a compromise between polluting a namespace (the case of *features*) and tightly coupling our compiled code to ASDF. If we rely on CL-USER variables we can, inside the compiled code, use DEFVAR to supply default values when the user did not provide any.

Separating ASDF's task

ASDF now has two different sets of uses

  • People who develop with ASDF
  • People who use the libraries from ASDF

The one does not exclude the other, but the needs are different and may be even more different in the future. For developers ASDF covers all these tasks

  • Loading files so that other files can be compiled
  • Compiling files
  • Loading the resulting files
  • Testing

If we focus on the users then we only need one thing

  • Loading possibly compiled files

The compiling phase can be abstracted out and even eliminated if we distribute precompiled files, but in any case the user does not want and does not need to care about whether files are compiled or not and how.

So in a sense we are merging in ASDF multiple tasks: from being a makefile that drives the compilers and linkers, to being the dynamic linker that looks for dependencies of libraries and loads them. These two tasks can be better separated and even better defined.

In particular the loading phase is so simple that one may even think of a bare-bones ASDF system that is capable of loading only precompiled files and which can be shipped anywhere. Or even better, delivering programs with their own "loaders", just a set of LOAD statements automatically generated by ASDF so that the resulting executable can be moved around and installed anywhere without ASDF itself.

Locating our files

This is another use I have found in lisp software. In ASDF 2 there exist two API functions for locating the pathname of a system. I strongly dislike them because of my personal dislike towards anything that binds my code to a certain build system. I instead advocate the use of logical pathname translations as a mean to pass around locations for files. For instance one may define automatically a logical pathname for each system COMMON-LISP:SYSTEMS;MY-SYSTEM or one may explicitely supply it as an option to DEFSYSTEM

(defsystem :my-system
   :logical-pathname "ES.SOFTWARE:MY-SYSTEM" ...

Using logical pathnames you may simply find the actual location via (translate-logical-pathname ...) or make use of logical pathnames in your software (if there are no weird file names around). The advantage is that these pathnames can be defined by any other loader system, or even by some file that uses your library without ASDF itself.