Extending Emacs with Fennel

After watching this year’s EmacsConf and seeing Guile Emacs being resurrected I thought to myself - why limit ourselves to Guile? Sure, Guile isn’t just a Scheme implementation, thanks to its compiler-tower-based design. Other languages exist for Guile VM, such as Emacs Lisp, and Guile manual lists the following languages with various stages of completeness:

ECMAScript
Brainfuck
Lua
Ruby
Python

Sure, it would be nice, if Emacs could natively run all of these, but we have to understand, that Guile Lua is not PUC Lua. Even LuaJIT has difficulties maintaining full compatibility with modern releases of Lua, having us stuck in the realm somewhere between Lua 5.1 and 5.2, while PUC Lua is already 5.4. Future releases would bring more differences, making PUC Lua and Luajit diverge even more.

And Lua is a simple language, unlike something like JS, Ruby, or Python. So I wouldn’t bet on the fact that Guile Ruby would be exactly the same as Ruby. Maybe it will, but it’ll probably take years.

So I thought, why not just use Lua as is? The only thing we can’t do is to run Fennel functions from Emacs Lisp. There were attempts at bringing Lua VM into Emacs, running in the same process, however, as I know, it isn’t stable enough.

However, Emacs can connect to a Fennel REPL, like it does for many other languages, such as Common Lisp or Clojure. Sure, the language is running in a separate process, and this comes with a lot of nuances, but it’s still a possible route.

So I made a small package: require-fennel.el. It’s capable of loading Fennel (or Lua) modules, and defining a set of functions on the Emacs side for every Fennel function in the module.

Running Fennel from Emacs Lisp

For example, we can load Fennel itself into Emacs like this:

(require-fennel fennel)

Now we have all sorts of functions available, for example, here’s fennel.eval:

(fennel.eval "(fcollect [i 1 10] (* i i))")
;; => [1 4 9 16 25 36 49 64 81 100]

Or, we can create a file greet.fnl:

(fn greet [name]
  "Greets NAME with a message."
  (print (.. "Hello, " name " from Fennel!")))

and load it:

(require-fennel greet)

(greet "Andrey")
(require-fennel foo)
(require-fennel vec-add)

Code Snippet 1: "Hello, Andrey from Fennel!" will be displayed in the echo area of Emacs

Let’s ask Emacs to describe the greet function:

greet is a interpreted-function.

(greet NAME)

Greets NAME with a message.

As can be seen, Emacs can show us the function signature and its documentation string obtained from Fennel.

Unlike Emacs Lisp, Fennel functions support destructuring with special syntax based on how data literals are written in Fennel. For example, let’s create a function that accepts a map, and destructures its keys:

(fn foo [{:x x :y y}]
  "Accepts a map with the X and Y keys, and adds them together."
  (+ x y))

After loading it in Emacs we’ll see the following documentation:

foo is a interpreted-function.

(foo ((:y . y) (:x . x)))

Accepts a map with the X and Y keys, and adds them together.

I took some creative liberties and made it so Fennel hash tables are represented as association lists in Emacs Lisp. So the ((:x . 1) (:y . 2)) in Elisp is equal to {:x 1 :y 2} in Fennel. The argument list on the Emacs Lisp side is a bit unconventional, but I think that’s OK.

The same goes for vector destructuring:

(fn vec-add [[x0 y0] [x1 y1]]
  "Adds two 2D vectors."
  [(+ x0 x1) (+ y0 y1)])

Emacs’ documentation uses lists:

vec-add is a interpreted-function.

(vec-add (X0 y0) (X1 y1))

Adds two 2D vectors.

I couldn’t make it work with vectors for some reason, and as can be seen, Emacs tries to upcase parameters, but it doesn’t recognize that y0 is also a parameter. Probably fine, as such documentation would never be generated for Emacs Lisp functions.

We can call such a function with both vectors and lists:

(vec-add [1 2] [1 2])   ;=> [2 4]
(vec-add '(1 2) '(1 2)) ;=> [2 4]

That’s because Fennel doesn’t have a list datatype, it only has associative and sequential tables. Thus I’m using Elisp vectors as the result type.

Data conversion

Here’s how conversion table when passing data from Elisp to Fennel:

Elisp	Fennel
`[1 2 3]`	`[1 2 3]`
`(1 2 3)`	`[1 2 3]`
`((:foo . 1) (:bar . 2))`	`{:foo 1 :bar 2}`
`#s(hash-table test equal data ("foo" 1 "bar" 2))`	`{:foo 1 :bar 2}`
`:foo`	`"foo"`
`'false`	`false`
`"foo"`	`"foo"`

And here’s the conversion table when receiving data from Fennel in Emacs:

Fennel	Elisp
`[1 2 3]`	`[1 2 3]`
`{:foo 1 :bar 2}`	`((:foo . 1) (:bar . 2))`
`(values 1 2 3)`	`(1 2 3)`
`(fn [])`	`(lambda (&rest args) ...)`
`userdata`	`#<udata: 0x55b66bd6ac28>`
`true`	`t`
`nil`, `false`	`nil`

A small note about hash tables and multiple value returns. As can be seen, both are returned as lists. I specifically chose the cons notation for association lists, i.e. (a . b) because this way we can tell apart hash tables and mustivalues. There are no cons cells in Fennel, so no other data structure will be represented like that.

Because multiple values are returned as lists, we have to call apply, if we wish to compose two Fennel functions. In other words, in Fennel we can do this:

(fn rest [_ ...] ...)

(fn first [x _] x)

(first (rest 1 2 3)) ;=> 2

{: first : rest}

Code Snippet 2: fr.fnl

In Elisp, however, we would have to do:

(require-fennel fr)

(apply #'fr.first (fr.rest 1 2 3)) ;=> 2

If we were to call it as (fr.first (fr.rest 1 2 3)) the result would be [2 3], because fr.rest returned a list of return values, and lists are converted to vectors or hash maps depending on their structure.

That’s the basic gist of this package.

More examples

After implementing it I was like a kid in a toy store. With a childish grin, I started loading all my crazy Fennel libraries that I made over the years.

For example, here’s cljlib:

(require-fennel cljlib)

(cljlib.conj [1] 2) ;=> [1 2]

Or, here’s a JSON parser I made:

(require fennel json)

(json.decode "{\"foo\": [1, 2, 3]}")
;; (("foo" . [1 2 3]))

(json.encode '(("foo" . [1 2 3])))
;; "{\"foo\": [1, 2, 3]}"

Here’s a crazy one - I’m loading my Fennel port of a Clojure library clj-http into Emacs, and calling it from Elisp:

(require-fennel io.gitlab.andreyorst.fnl-http)
(require-fennel io.gitlab.andreyorst.fnl-http.client :as client) ; nested module

(client.get "http://httpbin.org/get" '((:as . :json)))

;; => (("headers" . (("Content-Length" . "198")
;;                   ("Access-Control-Allow-Credentials" . t)
;;                   ("Access-Control-Allow-Origin" . "*")
;;                   ("Content-Type" . "application/json")
;;                   ("Connection" . "keep-alive")
;;                   ("Server" . "gunicorn/19.9.0")
;;                   ("Date" . "Mon, 16 Dec 2024 22:56:35 GMT")))
;;     ("reason-phrase" . "OK")
;;     ("length" . 198)
;;     ("protocol-version" . (("name" . "HTTP")
;;                            ("minor" . 1)
;;                            ("major" . 1)))
;;     ("request-time" . 214)
;;     ("trace-redirects" . [])
;;     ("status" . 200)
;;     ("body" . (("args" . [])
;;                ("headers" . (("Host" . "httpbin.org")
;;                              ("X-Amzn-Trace-Id" . "Root=1-6760b023-39ca8e413573df5d14d09486")))
;;                ("url" . "http://httpbin.org/get"))))

Code Snippet 3: I've formatted nested alists to use dot notation for better reading clarity. In other words changed (a (b . c) (d . e)) into (a . ((b . c) (d . e))) which are the same thing.

The client made an HTTP request from the Fennel side and passed the response back to Emacs. The response body was in JSON, but specifying ((:as . :json)) as options map forced parsing the response body into Fennel tables. Which, upon arriving in Emacs Lisp were converted into association lists.

So now we can call Fennel from Elisp. But why stop there? Let’s call Emacs Lisp from Fennel!

Running ELisp from Fennel?..

Since this package uses the Fennel Proto REPL, which is a custom protocol that can be extended at runtime, we can create a set of functions that will allow us to call back to Elisp. Here’s how we extend the protocol:

(let [format-elisp _G.___repl___.pp]
  (protocol.env-set!
   :emacs {:call (->> {:__index
                       (fn [_ k]
                         (fn [...]
                           (protocol.message
                            [[:id {:sym protocol.id}]
                             [:op {:string :require-fennel/call}]
                             [:fun {:sym k}]
                             [:arguments {:list (fcollect [n 1 (select "#" ...)]
                                                  (format-elisp (pick-values 1 (select n ...))))}]])
                           (. (protocol.receive) :data)))}
                      (setmetatable {}))
           :var (->> {:__index
                      (fn [_ k]
                        (protocol.message [[:id {:sym protocol.id}]
                                           [:op {:string :require-fennel/var}]
                                           [:var {:sym k}]])
                        (. (protocol.receive) :data))}
                     (setmetatable {}))
           :eval (fn [expr]
                   (protocol.message [[:id {:sym protocol.id}]
                                      [:op {:string :require-fennel/eval}]
                                      [:expr {:sym expr}]])
                   (. (protocol.receive) :data))})
  {:id 1000 :nop ""})

I had to update the protocol core a bit, adding the protocol.env-set! and protocol.receive functions. The env-set! allows us to create user-visible definitions. So we add the emacs table that holds three fields.

The call field is a table with some metatable trickery. Since it’s an empty table, any key would trigger the __index metamethod. This metamethod can be another table or a function - in our case, it is the latter. This function returns a handler that calls into protocol.message that sends a custom OP require-fennel/call that we can handle in our package.

The same goes for the var field, except it doesn’t return a function, it just returns a value.

The last field, eval, is a combination of both, kinda. It is a function, not a table, and it accepts one argument - an expression to evaluate in Emacs.

However, we can’t use these just yet. The second part of the trick comes from the new ability to extend the main protocol handler with custom handlers:

(cl-defgeneric fennel-proto-repl-handle-custom-op (_op _message _callbacks)
  "Handler for a custom protocol OP.
The first argument OP is used for dispatching.  Accepts the whole
MESSAGE, and its CALLBACKS.  The default implementation of unknown OP is
a nop."
  nil)

With this generic function, we can define new handlers in separate packages. These are the methods I’ve added in require-fennel.el:

(cl-defmethod fennel-proto-repl-handle-custom-op ((op (eql :require-fennel/call)) message callbacks)
  "Custom handler for the require-fennel/call OP.
Accepts a MESSAGE and its CALLBACKS.
The MESSAGE contains a function to evaluate and its arguments."
  (fennel-proto-repl-send-message
   nil
   (format "{:id %s :data %s}"
           (plist-get message :id)
           (apply (plist-get message :fun) (plist-get message :arguments)))
   nil))

(cl-defmethod fennel-proto-repl-handle-custom-op ((op (eql :require-fennel/var)) message callbacks)
  "Custom handler for the require-fennel/var OP.
Accepts a MESSAGE and its CALLBACKS.
The MESSAGE contains a var which value is then returned to Fennel."
  (fennel-proto-repl-send-message
   nil
   (format "{:id %s :data %s}"
           (plist-get message :id)
           (require-fennel--elisp-to-fennel (eval (plist-get message :var))))
   nil))

(cl-defmethod fennel-proto-repl-handle-custom-op ((op (eql :require-fennel/eval)) message callbacks)
  "Custom handler for the require-fennel/var OP.
Accepts a MESSAGE and its CALLBACKS.
The MESSAGE contains an expression which is evaluated and its result is returned to Fennel."
  (fennel-proto-repl-send-message
   nil
   (format "{:id %s :data %s}"
           (plist-get message :id)
           (require-fennel--elisp-to-fennel (eval (plist-get message :expr))))
   nil))

These three fields and methods allow us to do anything with Emacs, and it happens in the same process, so we can access the full Emacs state. For example:

;; Welcome to Fennel Proto REPL 0.6.0
;; Fennel version: 1.5.1-dev
;; Lua version: PUC Lua 5.4
>> (emacs.call.emacs-uptime)
"19 minutes, 17 seconds"
>> emacs.var.emacs-version
"31.0.50"
>> (emacs.eval "(with-current-buffer \" *fennel-elisp*\" (buffer-substring-no-properties (point-min) (point-max)))")
";; Welcome to Fennel Proto REPL 0.6.0
;; Fennel version: 1.5.1-dev
;; Lua version: PUC Lua 5.4
>> (emacs.call.emacs-uptime)
\"19 minutes, 17 seconds\"
>> emacs.var.emacs-version
\"31.0.50\"
>> (emacs.eval \"(with-current-buffer \\\" *fennel-elisp*\\\" (buffer-substring-no-properties (point-min) (point-max)))\")
>> "
>>

Crazy!

Fennelmacs

So now we can write Fennel scripts that can obtain data from Emacs, process it, and pass back the results. Thus, our goal is achieved - we’ve added a second language to Emacs!

Of course, it’s not the same as what Guile Emacs does. We’re still running code in a separate process, adding communication between the two via some protocol. It’s a neat trick, nothing more. Guile Emacs, if it ever becomes the thing, will be more powerful, as we would be able to leverage all these different languages in the same process, possibly sharing the memory, calling functions interchangeably, etc. Some may even write Emacs packages in Brainfuck, who knows.

Still, this is really cool, in my opinion, and shows the power of Emacs pretty well. If you’re into Fennel, like me, I hope you’ll enjoy using this package as I enjoyed making it. See you at Fennel Conf!

Comment via email