New Fennel Proto REPL and call for testing

In the previous post I’ve described how to define a simple protocol, upgrade the stock Fennel REPL with it, and create a simple client that works with this setup. And at the end, I mentioned that I was working on a proper client implementation and a more robust protocol as part of the fennel-mode package. So today I’m presenting the new fennel-proto-repl module!

It’s an experimental implementation of the new Fennel REPL client, that is based on the message-callback system, and it provides more robust ways of communicating with the REPL process. We’re not going to remove the old REPL integration support, and it going to remain the default one since I expect that some rough edges still need to be refined, and we still need to wait until Fennel will release the next stable version anyway. But if you’re using the development version of Fennel or don’t mind trying things, you can update fennel-mode and try the new REPL implementation by adding the following hook to your Emacs configuration:

(add-hook 'fennel-mode-hook 'fennel-proto-repl-minor-mode)

This new minor mode enables the integration with the new REPL implementation, meaning that all usual commands, such as Ctrl+Alt+x (eval-defun) will use the new REPL implementation.

At the end of the previous post I mentioned that the protocol-based REPL may not work in all cases, but this limitation should now be lifted in the latest Fennel (that’s why the Fennel update is required). I’ve sent a patch that makes inbuilt Fennel REPL more dynamic, and now the currently running REPL can be changed at runtime. Thus, if your application uses a custom REPL, like the one in the min-love2d-fennel repo, you should be able to use new Proto REPL, you’ll just need to update the Fennel dependency in your application.

So this post is a call for testing! If you experience hiccups or problems with the stock fennel-repl mode in Emacs, give fennel-proto-repl a try!

What’s new

Here I’ll briefly list enhancements that are now possible thanks to the protocol. There aren’t a lot of new features, the new client mostly provides stability-related improvements, yet, there are some things I’d like to note.

Programmatic API

First of all, now it is possible to send both synchronous and asynchronous messages to the REPL process and use a system of callbacks to get back the results. Previously it was possible but it was quite finicky because of how comint¹ works, and thus all requests sent outside of the REPL were synchronous (like completions or documentation fetching). Again, I’ve described it in detail in the previous post. The REPL itself is still synchronous - all messages are processed sequentially, but the client is now decoupled and doesn’t have to proactively wait for each message to be processed.

The new API includes these two functions: fennel-proto-repl-send-message, and fennel-proto-repl-send-message-sync. The first one will never block Emacs and should be used whenever you want to send some code to the REPL but you don’t require the result immediately. Here’s an example:

(fennel-proto-repl-send-message
 :eval "(+ 1 2 3)"
 (lambda (vals) (message "%S" vals)))

In this case, the result will be shown when the REPL is ready to process the message. You can additionally provide error callback and printing callback to capture error messages or data that the REPL requested to print.

(fennel-proto-repl-send-message-sync :eval "(+ 1 2 3)") ;; => ("6")

This will block Emacs until the REPL is ready to process the message, so it should be used with care. The message, however, can timeout after a configurable amount of time (5 seconds by default), so the block will not be indefinite.

Most of the features in the new client are implemented via these two functions, so it should be enough for other packages to integrate with the new client if necessary. I’m already thinking about updating my ob-fennel package to use this new REPL implementation, as it will fix a lot of bugs and problems in general.

Multiple REPLs

Support for multiple REPL sessions received some improvements. Previously, with the comint-based REPL, in order to have more than one REPL active at a time, it was required to rename the existing REPL buffer, and then start a new one. Otherwise, the function that starts the REPL just jumped to the already running one. This worked, however, the in-buffer commands, such as eval-defun or eval-last-sexp started working with the new REPL buffer unless it also was renamed. If there wasn’t a REPL buffer with the default name these commands didn’t work.

Fennel Proto REPL introduces a way to link a certain buffer to a certain REPL with the fennel-proto-repl-link-buffer function. This function provides a list of currently running REPLs to choose from, and after choosing one all in-buffer commands start working with that REPL. Additionally to that, it’s no longer necessary to manually rename the REPL buffer to start a new one - the fennel-proto-repl function always starts a new REPL and automatically links the current buffer to it.

Improved error reporting

The protocol changes how the errors are reported to the client, so the client could better analyze the stack traces and make them clickable. Now, when the error occurs, a special buffer will pop up, displaying the detailed message and a clickable stack trace:

The REPL itself shows a shortened message to avoid too much visual noise.

Improved Xref, Eldoc, and completion-at-point support

Fennel REPL has a command that can provide you with the target file and line of the definition you’re interested in, but it’s a bit tedious to copy it from the REPL every time. So the new Xref integration uses a special protocol OP that will automate most of the things.

Eldoc support was already part of the fennel-mode, but it was implemented poorly. The new implementation is asynchronous, and should never block Emacs if the REPL is busy, like what’s happened with an older implementation. We will remove eldoc support from basic fennel-repl in the future as it causes lots of problems. Unfortunately, asynchronous Eldoc is a feature of Emacs 28+. I’ve decided not to implement support for older versions of Emacs because in the case of Fennel, there’s no way of querying for documentation concurrently with the currently running task, like can be done in other languages, and blocking Emacs is not fun.

Completion at point also received some enhancements and should be much faster now, because everything is now processed in batches. The previous implementation requested a list of completions with one command and then sent a request for each item to get its kind. The new implementation does an initial request, and then a second request that gets kinds for all results of previous requests and caches them until the next completion happens.

Overall, all these features should be more robust, because we no longer require parsing the output of the REPL, as we had to with comint.

Rough edges

The implementation is not perfect, so there are some things to be concerned about.

The client code is still can be improved performance-wise. Nothing crazy, but if the process starts spitting out lots of output, it can start piling up and Emacs will chug on it. This happens when printing huge tables, like _G, or when a tight loop produces a lot of output, like (for [i 1 1000000] (print i)). Disabling process logging can help, and this is why the logging is disabled by default. Though it should not be a major problem because large amounts of output are rare, and data is usually small.

The buffer linking mechanism is not as user-friendly as something more automatic, like linking REPLs based on project structure, I agree. But I hope it is not as bad, and we can always improve this part later as we gather more feedback from the users.

Lastly, the overall complexity of the client may seem daunting, and there’s a lot of code but the architecture is quite simple, it’s just that there’s a lot of code for various features.

I hope that it will be good enough to be the default REPL one day. However, as I’ve mentioned, there may be problems with embedded REPLs or just the fact that you can’t update the old Fennel version due to some circumstances. For these cases, there’s always the old implementation of the REPL.

The Protocol library

The protocol behind the new client is in fact editor-agnostic and can be used by other editors. Because of that, I’ve released the protocol as a library: fennel-proto-repl-protocol, and I’m looking forward to others trying it and implement support in their editors.

The protocol setup is quite simple - all you have to do is either require the protocol library from the REPL and call the provided function with an editor-specific format function. Or you can send the whole protocol code to a running REPL without using require as the fennel-proto-repl module does.

Here’s an example for JSON based messaging:

>> (fn format-json [env data]
     (: "{%s}" :format
        (table.concat
         (icollect [_ [k v] (ipairs data)]
           (: "%s: %s" :format (env.fennel.view k)
              (: (case v
                   {:list data} (.. "[" (table.concat data " ") "]")
                   {:string data} (env.fennel.view data)
                   {:sym data} (tostring data)
                   _ (env.protocol.internal-error "Wrong data kind" (env.fennel.view v)))
                 :gsub "\n" "\\\\n"))) ", ")))
#<function: 0x563fec938af0>
>> ((require :protocol) format-json)
{"id": 0, "op": "init", "status": "done", "protocol": "0.1.0", "fennel": "1.3.1-dev", "lua": "PUC Lua 5.4"}

For more info, see the project’s readme.

All in all, I hope this will help make better tooling for Fennel in the future, and will bring the programming experience closer to what’s available for other Lisps.