Dynamic font-lock for Fennel

I’ve been working on adding a dynamic font-locking for fennel-mode for a while now. The first attempt started years ago, in July 2022, and were quite crude. It consisted of sending a special code to the REPL that would return me a list of macro names, and then I created a regular expression that matched all these names for syntax highlighting. This had several problems.

First, this meant that any macro known to the REPL would be highlighted. This is not great, as macros in Fennel can be loaded from any file that you require, but it doesn’t mean that they’re available to your file in particular.

For example, imagine you have this code:

As you can see, this code requires the module :foo, defines a few functions, and finally calls them. Ignore the fact that the module foo is unused for now - turn your attention to the colors instead.

As you can see, fennel-mode uses colors to tell the user a few things. Special forms and macros are highlighted in a blueish color. Inbuilt functions are highlighted in purple. Names of declared functions are also in purple, but not their calls.

However, you can see that for some reason, bar is highlighted blue, as if it was a special or a macro. This is because I’m using my old implementation of dynamic highlighting. If we look into the foo module, we’ll see that it requires a macro module, that defines bar as a macro. It’s not a problem - macros in Fennel are only available in the module they’re required, so there’s no name conflict here, as you would expect. However, highlighting is not very smart.

When syntax highlighting is applied, a request is sent to the REPL to get all available macros. Then no distinction is made for where these macros are coming from, hence the wrong highlighting.

Another tricky case, that I don’t see implemented around is scoping:

In Fennel, the macro special, which is used to define macros, obeys the scoping rules. Thus when we create a macro inside of the do’s scope, this macro should not be highlighted outside of it. But again, because the initial implementation of dynamic highlighting didn’t care about anything but macro names, the macro was incorrectly highlighted in this case as well.

Because of these problems, Phil and I decided that this won’t be a good feature to bring into fennel-mode in this state. The branch rested for almost three years, up until recently, when I started working on fennel-mode again.

Recently, I added support for Flymake via the fennel-ls language server. That was another old branch of mine, where I added linting via Fennel’s own linter plugin, but Phil mentioned that the plugin was never supposed to be used like this. It was also suggested that since fennel-ls already existed back then, and it was working with eglot there was no real need for using Flymake with Fennel’s linter plugin.

I gave this branch a rest, and after a while, when I started using fennel-ls myself, I decided to contribute a small change so that it could be used from Flymake. Personally, I don’t need a full-blown language server for Fennel, because everything I need is provided by the Fennel Proto REPL, but I do want some linting capabilities. So after merging the Flymake branch into the main fennel-mode branch, I noticed this old dynamic-font-lock branch, and decided that “it’s time to do this”.

Thus, I’m happy to announce, that all the problems listed above were fixed! Take a look at this:

Of course, when we just load the file, nothing is highlighted yet, as the syntax highlighting is dynamic. However, when we load the module into the REPL, the highlighting is applied correctly:

Once the module is loaded, highlighting is cached, and no further requests to the REPL are sent. This doesn’t, however, mean that when we change the code, the highlighting won’t react:

I wanted to make this feature for quite some time. I’m a big fan of CIDER’s dynamic font-locking and have been using it for some years. It never occurred to me, however, that CIDER doesn’t do scoping:

But Clojure macros are not scoped, like in Fennel, so that’s probably fine. What’s not as fine is that it will highlight the macro before it was declared:

Which would be an error if we were to try to reload this file. But, for what it’s worth, I think CIDER’s solution is fine, because again - no special scoping rules, and unless the file loads without errors, no dynamic syntax highlighting would be applied. For Fennel, however, I wanted a more robust solution.

For now, this is still available in a feature branch, but I’m positive that I’ll merge this soon, after I give it a bit more testing, and gather some feedback from the users. If you’re a fennel-mode user, and you use my fennel-proto-repl module, I encourage you to try out dynamic font-locking and see if it works for you!