Compiling Clojure projects in Emacs - Jumping into dependencies
In the previous post on the subject I’ve described how one can create a custom compilation mode for any language.
In the Dynamically extracting filenames from compiler output section I’m talking about various issues with how Clojure reports problem locations.
The main problem is when the problem is inside of a dependency, and be it your own library, or a third-party one it’s equally tedious to go and look into it because the dependency usually is a
jar file somewhere in the
And the actual problem is that the error message doesn’t really tell you what dependency to look into, it just uses a path to the file as if it was a file in a project.
We can always ignore such files, but it’s not great, as we’re effectively closing our eyes on existing problems in our codebase.
Since the previous post I’ve tweaked the function that gets the filename for the given compilation output line - now it can find files in the project’s dependencies as well.
It’s still not entirely reliable, as theoretically there might be some name clashes, but I have yet to see any actual problems with this approach.
This approach involves dealing with the
I had to update the macro I’m using to generate these helper modes.
The main changes are that the supplementary
compile-add-error-syntax function is now adding a
HIGHLIGHT parameter to the
compilation-error-regexp-alist-alist, and automatically specifies that it should only be applied to the
The face is chosen automatically based on the
(cl-defun compile-add-error-syntax (mode name regexp &key file line col (level 'error) hyperlink highlight) "Register new compilation error syntax. Add NAME symbol to `compilation-error-regexp-alist', and then add REGEXP FILE LINE and optional COL LEVEL info to `compilation-error-regexp-alist-alist'." (or file (error "Missing value for :file keyword")) (or line (error "Missing value for :line keyword")) (let ((faces '(compilation-info-face compilation-warning-face compilation-error-face)) (level (cond ((eq level 'info) 0) ((eq level 'warn) 1) ((eq level 'error) 2) (t (error "Unsupported level type: %S" level)))) (mode (symbol-name (or mode 'compilation)))) (add-to-list (intern (concat mode "-error-regexp-alist")) name) (add-to-list (intern (concat mode "-error-regexp-alist-alist")) (list name regexp file line col level hyperlink (list highlight (nth level faces))))))
Changes to the macro itself are quite small:
(defmacro define-project-compilation-mode (base-name &rest body) (declare (indent 1)) (let* ((name (symbol-name base-name)) (doc-name (capitalize (replace-regexp-in-string "-compilation$" "" name))) (current-project-root (intern (concat name "-current-project"))) (current-project-files (intern (concat name "-current-project-files"))) (compilation-mode-name (intern (concat name "-mode")))) `(progn (defvar ,(intern (concat name "-error-regexp-alist")) nil ,(concat "Alist that specifies how to match errors in " doc-name " compiler output. See `compilation-error-regexp-alist' for more information.")) (defvar ,(intern (concat name "-error-regexp-alist-alist")) nil ,(concat "Alist of values for `" (downcase doc-name) "-compilation-error-regexp-alist'. See `compilation-error-regexp-alist-alist' for more information.")) (defvar-local ,current-project-root nil ,(concat "Current root of the project being compiled. Set automatically by the `" (symbol-name compilation-mode-name) "'.")) (defvar-local ,current-project-files nil ,(concat "Current list of files belonging to the project being compiled. Set automatically by the `" (symbol-name compilation-mode-name) "'.")) (define-compilation-mode ,compilation-mode-name ,(concat doc-name " Compilation") ,(concat "Compilation mode for " doc-name " output.") (setq-local ,current-project-root (project-current t)) (setq-local ,current-project-files (project-files ,current-project-root)) ,@body) (provide ',compilation-mode-name))))
,@body inside the call to
define-compilation-mode as it really was a mistake in the previous version of the macro.
This way we can extend the initialization step of the mode with additional expressions in the
Working with project’s
First things first, we need to query our project for
It can be done with
lein classpath or
clojure -Spath if you’re using
For this post, I’m going to continue with
lein, as it is what I use at work, and I’m unfamiliar with most
deps commands, i.e. for example, I don’t know what the analog to
The code below defines some functions that in the end will return a list of strings.
These strings are paths to
.jar archives being used by our project:
(defun clojure-compilation--split-classpath (classpath) "Split the CLASSPATH string." (split-string classpath ":" t "[[:space:]\n]+")) (defun clojure-compilation--get-project-dependencies* (command _deps-file _mod-time) "Call COMMAND to obtain the classpath string. DEPS-FILE and MOD-TIME are used for memoization." (thread-last command shell-command-to-string clojure-compilation--split-classpath (seq-filter (lambda (s) (string-suffix-p ".jar" s))))) (fset 'clojure-compilation--get-project-dependencies-memo (memoize #'clojure-compilation--get-project-dependencies*)) (defun clojure-compilation--get-lein-project-dependencies (root) "Obtain classpath from lein for ROOT." (let* ((project-file (expand-file-name "project.clj" root)) (mod-time (file-attribute-modification-time (file-attributes project-file)))) (clojure-compilation--get-project-dependencies-memo "lein classpath" project-file mod-time))) (defun clojure-compilation--get-deps-project-dependencies (root) "Obtain classpath from deps for ROOT." (let* ((project-file (expand-file-name "deps.edn" root)) (mod-time (file-attribute-modification-time (file-attributes project-file)))) (clojure-compilation--get-project-dependencies-memo "clojure -Spath" project-file mod-time))) (defun clojure-compilation-get-project-dependencies (project) "Get dependencies of the given PROJECT. Returns a list of all jar archives." (when (bound-and-true-p tramp-gvfs-enabled) (let ((root (project-root project))) (cond ((file-exists-p (expand-file-name "deps.edn" root)) (clojure-compilation--get-deps-project-dependencies root)) ((file-exists-p (expand-file-name "project.clj" root)) (clojure-compilation--get-lein-project-dependencies root))))))
define-project-compilation-mode macro that I’ve defined above we can create a
(defvar-local clojure-compilation-project-deps nil "List of project's dependencies") (defvar-local clojure-compilation-project-deps-mod-time nil "Accumulated modification time of all project's libraries") (define-project-compilation-mode clojure-compilation (require 'tramp-gvfs) (setq-local clojure-compilation-project-deps (clojure-compilation-get-project-dependencies clojure-compilation-current-project)) (setq-local clojure-compilation-project-deps-mod-time (seq-reduce #'+ (mapcar (lambda (f) (time-to-seconds (file-attribute-modification-time (file-attributes f)))) clojure-compilation-project-deps) 0)))
Upon initializing, it will query
deps) for the project’s classpath, and store it in the
In addition to that we store the accumulated modification time of all of our dependencies.
lein classpath every time project is re-compiled is quite slow, and we only really need to do it when
project.clj was recently changed, so I’m using a simple memoization function:
(defun memoize (fn) "Create a storage for FN's args. Checks if FN was called with set args before. If so, return the value from the storage and don't call FN. Otherwise calls FN, and saves its result in the storage. FN must be referentially transparent." (let ((memo (make-hash-table :test 'equal))) (lambda (&rest args) (let ((value (gethash args memo))) (or value (puthash args (apply fn args) memo))))))
We memoize the call to
clojure-compilation--get-project-dependencies* by its
filename, and the file’s modification timestamp.
So, if the filename or timestamp changes, we re-compute the dependencies.
Now we look at the function that is used in the
But before that, let’s define some rules:
(compile-add-error-syntax 'clojure-compilation 'some-warning "^\\([^:[:space:]]+\\):\\([0-9]+\\) " :file #'clojure-compilation-filename :line 2 :level 'warn :hyperlink 1 :highlight 1) (compile-add-error-syntax 'clojure-compilation 'clj-kondo-warning "^\\(/[^:]+\\):\\([[:digit:]]+\\):\\([[:digit:]]+\\): warning" :file 1 :line 2 :col 3 :level 'warn :hyperlink 1 :highlight 1) (compile-add-error-syntax 'clojure-compilation 'clj-kondo-error "^\\(/[^:]+\\):\\([[:digit:]]+\\):\\([[:digit:]]+\\): error" :file 1 :line 2 :col 3 :hyperlink 1 :highlight 1) (compile-add-error-syntax 'clojure-compilation 'kaocha-tap "^not ok.*(\\([^:]*\\):\\([0-9]*\\))" :file #'clojure-compilation-filename :line 2 :hyperlink 1 :highlight 1) (compile-add-error-syntax 'clojure-compilation 'clojure-fail "^.*\\(?:FAIL\\|ERROR\\) in.*(\\([^:]*\\):\\([0-9]*\\))" :file #'clojure-compilation-filename :line 2 :hyperlink 1 :highlight 1) (compile-add-error-syntax 'clojure-compilation 'clojure-reflection-warning "^Reflection warning,[[:space:]]*\\([^:]+\\):\\([0-9]+\\):\\([0-9]+\\)" :file #'clojure-compilation-filename :line 2 :col 3 :level 'warn :hyperlink 1 :highlight 1) (compile-add-error-syntax 'clojure-compilation 'clojure-performance-warning "^Performance warning,[[:space:]]*\\([^:]+\\):\\([0-9]+\\):\\([0-9]+\\)" :file #'clojure-compilation-filename :line 2 :col 3 :level 'warn :hyperlink 1 :highlight 1) (compile-add-error-syntax 'clojure-compilation 'clojure-syntax-error "^Syntax error .* at (\\([^:]+\\):\\([0-9]+\\):\\([0-9]+\\))" :file #'clojure-compilation-filename :line 2 :col 3) (compile-add-error-syntax 'clojure-compilation 'kaocha-unit-error "^ERROR in unit (\\([^:]+\\):\\([0-9]+\\))" :file #'clojure-compilation-filename :line 2 :hyperlink 1 :highlight 1) (compile-add-error-syntax 'clojure-compilation 'eastwood-warning "^\\([^:[:space:]]+\\):\\([0-9]+\\):\\([0-9]+\\):" :file #'clojure-compilation-filename :line 2 :col 3 :level 'warn :hyperlink 1 :highlight 1)
These are the rules I’m using at work, it’s quite handy to be able to jump from the compilation buffer and see if the problem is fixable.
clojure-compilation-filename is defined as follows:
(defun clojure-compilation-filename () "Function that gets filename from the error message. If the filename comes from a dependency, try to guess the dependency artifact based on the project's dependencies." (when-let ((filename (substring-no-properties (match-string 1)))) (or (clojure-compilation--find-file-in-project filename) (when-let ((dep (clojure-compilation--find-dep filename))) (concat (expand-file-name dep) "/" filename)))))
It splits the task into two parts. First, it checks if the file is part of the project:
(defun clojure-compilation--find-file-in-project (file) "Check if FILE is part of the currently compiled project." (seq-find (lambda (s) (string-suffix-p file s)) clojure-compilation-current-project-files))
It’s a rather simple filter of the
clojure-compilation-current-project-files var we create when the
The second part is similar, but it tries to find a matching dependency. Unfortunately, knowing the file name doesn’t mean that we’ll be able to find the dependency itself, as the artifact name may not have the same name. But this is somewhat rare, so we can do it like this:
(defun clojure-compilation--file-exists-jar-p (jar file) "Check if FILE is present in the JAR archive." (with-temp-buffer (when (zerop (call-process "jar" nil (current-buffer) nil "-tf" jar)) (goto-char (point-min)) (save-match-data (re-search-forward (format "^%s$" (regexp-quote file)) nil t))))) (defun clojure-compilation--find-dep* (file _project _deps-mod-time) "Find FILE in current project dependency list. PROJECT and DEPS-MOD-TIME are used for memoizing the call." (when (not (string-empty-p file)) (seq-find (lambda (d) (clojure-compilation--file-exists-jar-p d file)) clojure-compilation-project-deps))) (fset 'clojure-compilation--find-dep-memo (memoize #'clojure-compilation--find-dep*)) (defun clojure-compilation--find-dep (file) "Find FILE in current project dependency list." (clojure-compilation--find-dep-memo file clojure-compilation-current-project clojure-compilation-project-deps-mod-time))
There’s a lot going on, but the idea is basically the same as for the project’s dependencies.
clojure-compilation--find-dep* by the
file, current project, and accumulated modification time of all dependencies.
If any of the dependencies changes we’ll re-compute the whole thing.
Otherwise, if the
file is present multiple times in the compilation output we will avoid searching for it multiple times thanks to the memoization.
For example, here’s a log from
I have omitted any lines that are related to the project itself, so we’re only looking at warnings inside the dependencies:
-*- mode: clojure-compilation; default-directory: "~/some/project/" -*- Clojure Compilation started at Mon Oct 2 15:29:30 lein do clean, check Reflection warning, me/raynes/fs.clj:517:42 - reference to field getName can't be resolved. Reflection warning, clojure/data/xml.clj:337:17 - call to method createXMLStreamReader can't be resolved (target class is unknown). Reflection warning, instaparse/util.clj:5:3 - call to java.lang.RuntimeException ctor can't be resolved. Reflection warning, instaparse/util.clj:11:3 - call to java.lang.IllegalArgumentException ctor can't be resolved. Reflection warning, ring/util/servlet.clj:88:24 - call to method write on javax.servlet.ServletOutputStream can't be resolved (argument types: unknown).
So, given a
me/raynes/fs.clj and a dependency list like:
(... "~/.m2/repository/org/clojure/data.xml/0.0.8/data.xml-0.0.8.jar" "~/.m2/repository/instaparse/instaparse/1.4.8/instaparse-1.4.8.jar" "~/.m2/repository/me/raynes/fs/1.4.6/fs-1.4.6.jar" "~/.m2/repository/ch/qos/logback/logback-core/1.2.11/logback-core-1.2.11.jar" "~/.m2/repository/ch/qos/logback/logback-classic/1.2.11/logback-classic-1.2.11.jar" "~/.m2/repository/ring-logger/ring-logger/1.1.1/ring-logger-1.1.1.jar" "~/.m2/repository/ring/ring-servlet/1.10.0/ring-servlet-1.10.0.jar" "~/.m2/repository/ring/ring-core/1.10.0/ring-core-1.10.0.jar" ...)
me/raynes/fs.clj is not part of the project, we will have to go through every
.jar archive, get its file listing, and check if any have this file.
I’m not sure if it is possible for two archives in the classpath list to have the same file, and that never happened to me yet, so I assume it is a reliable enough way of doing this.
Thus, the result of
clojure-compilation-filename will be
But what should we do with it?
Emacs actually can open this kind of path with TRAMP if the
gvfs is present on the system, hence the check in the
gvfs is not available there’s no point in analyzing dependencies, because, without it, Emacs can’t go inside such an archive.
And I’m not sure why - Emacs can open archives much like DIRED opens directories, and then we can open files from these archives, it just has to be done in two steps and there’s no special path syntax to open an archive, and jump to a file immediately.
gvfs however, Emacs can mount the archive, and jump to the
me/raynes/fs.clj file inside of it in one go, thus highlighting the problem in the dependency:
;; ---8<--- (defn find-files "Find files matching given pattern." [path pattern] (find-files* path #(re-matches pattern (.getName %)))) ;; ---8<---
The only downside of this approach is that, for some reason, Emacs takes a very long time to unmount these archives when I close it.
This has nothing to do with the method, and probably an issue with the
gvfs support in general.
gvfs package isn’t available it’s probably possible to use the jarchive package, however, the filename format returned by the
clojure-compilation-filename function has to be changed to include the
jar:file:// scheme and use
! as a separator:
(defun clojure-compilation-filename () ;; ---8<--- (when-let ((dep (clojure-compilation--find-dep filename))) (concat "jar:file://" (expand-file-name dep) "!" filename)) ;; ---8<--- )
Though, unfortunately, I couldn’t make it work with
jarchive because for some reason Emacs transforms
jar:file:/foo/bar right before a file is opened, and
jarchive specifically looks for
jar:file:/// as a prefix.
Maybe there’s some kind of a setting for that.
If you know about such, let me know too!
With all in place, Clojure warnings are fully intractable from the compilation buffer.
Even though Clojure is a language where we rarely use edit-compile-check cycle, I find it tremendously useful to be able to call
lein check, and other tools like clj-kondo or eastwood.
Same with the kaocha test runner.
Emacs is very configurable.
I would say that the
compilation-error-regexp-alist isn’t the most straightforward interface for configuring how errors are parsed, but it gets the job done and is very versatile.
It’s a general pattern in Emacs, many such configurations accept functions in arbitrary places, allowing users to extend the interface even more than it is possible with just regular parameters.
Because of that Emacs is also an infinite time sink, and seeing how my configuration grows by the day even after using Emacs for many years is both inspiring and scary.
Inspiring because Emacs shows that such a configurable system is possible.
Scary because I don’t know if the process will ever stop.
I hope this post was useful, and gave you the idea of how you can create custom handlers for the compilation buffer.
The whole configuration for Clojure can be found here.
Macros for defining language compilation modes are here.
The required advice for the
compilation-start function is available here.
Let me know if you had any problems with this code, or have ideas on any possible improvements!