Compiling Clojure projects in Emacs

Another post in the not-so-series about Emacs configuration. Today I will describe my configuration for managing the compilation-error-regexp-alist variable in a way that is meaningful for the current project I’m working on. Some time ago I faced a problem that the compilation-error-regexp-alist variable contains far too many entries for different languages by default. Since then, I’ve figured out how to solve this problem for myself, so I decided to explain it here, with the hope that it will be useful for others. Actually, most of this isn’t Clojure-specific at all and can be applied to any other language as easily. Yet, there will be some parts that are useful specifically for Clojure projects but may also benefit other languages that share the same problem with file paths.

For those unfamiliar with the compile command in Emacs, it is essentially a way of running external programs, such as make, lein, or anything else, really. There’s a similar command in Emacs, called async-shell-command, which also can be used for that, but the key difference between these two is that the buffer, created by compile automatically enters compile-mode. This mode is responsible for highlighting and counting errors, and warnings, and provides ways of jumping to problems directly from that buffer. And to configure how the errors are recognized we need to use the compilation-error-regexp-alist and compilation-error-regexp-alist-alist variables. The first one contains symbols, that we want to check against, and the second one stores pairs with such symbols and the regular expression used to match the particular error. For example, to compile a Guile project, we can set this variable to:

(setq compilation-error-regexp-alist '(guile-file guile-line))

The compilation-error-regexp-alist-alist contains, among others, these two entries:

'(;; ...
  (guile-file "^In \\(.+\\..+\\):\n" 1 nil nil 0)
  (guile-line "^ *\\([0-9]+\\): *\\([0-9]+\\)" nil 1 2)
  ;; ...
  )

But now, that the compilation-error-regexp-alist variable is set to contain only Guile-related entries, the compilation buffer will only pick up problems that match these regular expressions from compilation-error-regexp-alist-alist.

The system may look simple, however, there’s a problem. The compilation-error-regexp-alist variable is not buffer-local, and while you can make it, you still need a way to know what kind of project you’re compiling and some hook to set these variables right before the compilation starts. And even if we knew, we would need a way of setting this variable on per project basis to some values that are meaningful for the project. So how can we do that?

The compile function

To answer our questions, let’s look at the compile function signature:

(defun compile (command &optional comint)
  ;; ...
  )

If we check the documentation for this function, there’s a vague line about the comint argument:

If optional second arg COMINT is t the buffer will be in Comint mode with ‘compilation-shell-minor-mode’.

However, this documentation isn’t exactly on point. If we look at the body of this function, we’ll see that the only time the comint argument is used is at the end of the function when we call the compilation-start function:

(defun compile (command &optional comint)
  ;; ...
  (compilation-start command comint))

The signature of compilation-start tells us that its second argument is actually called mode and it is described as:

MODE is the major mode to set in the compilation buffer. Mode may also be t meaning use ‘compilation-shell-minor-mode’ under ‘comint-mode’.

So there’s a way of setting a different mode for the compilation buffer. And that’s exactly what we need to solve our problem.

Defining a new major mode for compilation of a specific language

So this step can be done for any language, not just Clojure. If you frequently compile, say, Lua, you may want to analyze Lua stack traces, and prevent other regular expressions from the compilation-error-regexp-alist to interfere. So to workaround this, you can simply create a mode, which sets its own local variables to the values, needed by compile. Here’s a mode I’ve defined for Clojure:

(defvar clojure-compilation-error-regexp-alist nil
  "Alist that specifies how to match errors in Clojure compiler output.
See `compilation-error-regexp-alist' for more information.")

(defvar clojure-compilation-error-regexp-alist-alist nil
  "Alist of values for `clojure-compilation-error-regexp-alist'.")

(defvar-local clojure-compilation-project nil
  "Current root of the project being compiled.")

(defvar-local clojure-compilation-project-files nil
  "Current list of files belonging to the project being compiled.")

(define-derived-mode clojure-compilation-mode compilation-mode
  "Clojure(Script) Compilation"
  "Compilation mode for Clojure output."
  (setq-local compilation-error-regexp-alist
              clojure-compilation-error-regexp-alist)
  (setq-local compilation-error-regexp-alist-alist
              clojure-compilation-error-regexp-alist-alist)
  (setq-local clojure-compilation-project (project-current t))
  (setq-local clojure-compilation-project-files
              (project-files clojure-compilation-project)))

I’ll explain why the other variables are needed later, our main focus here is clojure-compilation-error-regexp-alist and clojure-compilation-error-regexp-alist-alist ones.

Now we can define our rules for highlighting. For example, if you use clj-kondo, an excellent linter for Clojure, you might want to capture the filename from its messages and distinguish warnings and errors. You can do it like this:

(setq clojure-compilation-error-regexp-alist
      '(clj-kondo-warning clj-kondo-error))
(setq clojure-compilation-error-regexp-alist-alist
      '((clj-kondo-error "^\\(/[^:]+\\):\\([[:digit:]]+\\):\\([[:digit:]]+\\): error"
                         1 2 3 2)
        (clj-kondo-warning "^\\(/[^:]+\\):\\([[:digit:]]+\\):\\([[:digit:]]+\\): warning"
                           1 2 3 1)))

Because our clojure-compilation-mode mode is derived from the compilation-mode, we can use everything that is defined in compilation-mode, so upon activation, we locally set the regexp-alist variables to our predefined Clojure-related rules. The last thing left is to teach Emacs how to automatically enable this mode, and this can be done via project.el.

Tweaking project.el to help compile understand what mode to use

As the title of this section suggests, we’re going to teach project.el new tricks. But why project.el all of a sudden? Didn’t I say that we’re going to use compile?

Well, I mainly use project-compile because it does an appropriate thing from the project root. I don’t use projectile, which is another project management package for Emacs, so if you use it, you’re on your own here.

The first thing we need to do is create a new variable in the project group:

(defcustom project-compilation-mode nil
  "Mode to run the `compile' command with."
  :type 'symbol
  :group 'project
  :safe #'symbolp
  :local t)

Next, we’re going to advise the compilation-start function, which is used by compile and recompile functions to use this mode if the mode wasn’t passed explicitly:

(define-advice compilation-start (:filter-args (args) use-project-compilation-mode)
  (let ((cmd (car args))
        (mode (cadr args))
        (rest (cddr args)))
    (if (and (null mode) project-compilation-mode)
        (append (list cmd project-compilation-mode) rest)
      args)))

Basically, we reconstruct the argument list with the mode we’re interested in. Unfortunately, the project-compile function doesn’t support specifying compilation mode, and calls compile without any arguments, so we kinda have to do it this way. As a bonus, it will also work for recompile, which gets called when you press g in the compilation buffer. It would be nicer, if project-compile accepted the same arguments as compile, and there was a project-recompile function, so we could do things without using define-advice, but we have to work with what we have¹.

This advice only does anything to the arguments if the mode wasn’t set to a non-nil value, and when project-compilation-mode is set. Which, we can do via .dir-locals.el or .dir-locals-2.el if the first one is under version control:

;;; Directory Local Variables            -*- no-byte-compile: t -*-
;;; For more information see (info "(emacs) Directory Variables")

((nil . ((project-compilation-mode . clojure-compilation-mode))))

Now, if we compile this project, we’ll trigger our advice, which will update args, and compilation will enter clojure-compilation-mode. The errors will now use the rules we’ve defined for the current mode, and the process is automatic enough. Alternatively, project-compilation-mode can be set via a mode hook:

(add-hook 'clojure-mode-hook
          (lambda ()
            (setq-local project-compilation-mode 'clojure-compilation-mode)))

The downside of this method is that when called from the non-Clojure buffer, the value of this variable will be unset.

Dynamically extracting filenames from compiler output

Example configuration for clj-kondo from above is nice, but there’s a problem. Clj-kondo works with files, but when actually compiling Clojure code with the compiler, there is no actual project information left. Instead, we have namespace information, which is not exactly mapped to files in the project. E.g. we can get reflection warnings from something outside of our project, like from a dependency or a build system plugin, which would look like this:

$ lein kaocha
Reflection warning, /tmp/form-init5271780372383707418.clj:1:1014 - call to static method invokeStaticMethod on clojure.lang.Reflector can't be resolved (argument types: unknown, java.lang.String, unknown).
Reflection warning, kaocha/runner.clj:156:73 - call to java.io.File ctor can't be resolved.
Reflection warning, test_project/core.clj:8:3 - reference to field getMessage can't be resolved.

Here, I’m using kaocha, a test runner for Clojure which has some problems with reflection. Nothing serious, but when *warn-on-reflection* is set to true in a lein profile, this gets in the log. However, these messages are useful for us, as they demonstrate that even though reflection warnings have filename and line/column information, they can be outside of our project root.

Another thing to notice is that the files that do belong to our project don’t use the full path from the project root. In other words full path from the project root to the core.clj should be src/test_project/core.clj. This is important, because there may be other directories, which contain Clojure files, for example, test/test_project/bar.clj, which would be represented as just test_project/bar.clj in the compilation log. Names of such directories are arbitrary and can be configured for each project separately.

So, in order to jump to such a file from this kind of log entry, we need to figure out if this file belongs to our project. What’s great about compilation-error-regexp-alist-alist is that instead of specifying a group that represents the path part in the regular expression, we can specify a function to call in the compilation buffer. This function is called from the end of the matched part of our regular expression, so we can do a simple search for the filename from there. Then we can look if this path can be found in a set of all project files, and if it is, we return the path and the directory relative to the project root. Here’s the function:

(defun clojure-compilation-filename-fn (rule-name)
  "Create a function that gets the filename from the error message.

RULE-NAME is a symbol in `clojure-compilation-error-regexp-alist-alist'.
It is used to obtain the regular expression, which is used for a
backward search in order to extract the filename from the first
group."
  (lambda ()
    "Get a filename from the error message and compute the relative directory."
    (let* ((regexp (car (alist-get rule-name clojure-compilation-error-regexp-alist-alist)))
           (filename (save-match-data
                       (re-search-backward regexp)
                       (substring-no-properties (match-string 1)))))
      (if-let ((file (seq-find
                      (lambda (s)
                        (string-suffix-p filename s))
                      clojure-compilation-project-files)))
          (let* ((path-in-project
                  (substring file (length (project-root clojure-compilation-project))))
                 (dir (substring path-in-project 0 (- (length filename)))))
            (cons filename dir))
        filename))))

Now you can see why clojure-compilation-mode sets the clojure-compilation-project and clojure-compilation-project-files variables. We reuse them for every line that matches our regular expression, so it’s better to cache them beforehand since project sources rarely change during compilation. Here’s how we use it:

(let ((rule-name 'clojure-reflection-warning))
  (add-to-list 'clojure-compilation-error-regexp-alist rule-name)
  (add-to-list 'clojure-compilation-error-regexp-alist-alist
               '(rule-name
                 "^Reflection warning,[[:space:]]*\\([^:]+\\):\\([0-9]+\\):\\([0-9]+\\).*$"
                 (clojure-compilation-filename-fn rule-name)
                 2 3 2)))

This can be further automated into a separate function which sets both alists.

This function creates a closure over this regular expression and uses it to find the path to the file. If the path is not a part of the project, we just return it as is. Emacs will ask us for the base directory to search this file in, which isn’t the best solution, but we can’t do much here, given that it’s impossible to know from which jar the namespace came.

My complete configuration for Clojure compilation can be found here.