Fennel libraries as single files

I’m pleased to announce that most of my libraries for fennel are now shipped as single files! It was a long-standing issue with Fennel for me, as there was no clear way to ship both macros and functions in the same file, but after some experimentation, I figured out a way. After a bit of testing with my new library for asynchronous programming I can say that it works well enough to be used in most other libraries. Although the implementation is a bit rough and manual, that’s what I’ll try to describe here today.

But first, let’s talk modules. This will be a long post, so buckle up.

Modules in various programming languages

I find it fascinating that a lot of languages have some kind of a module system, yet most of them hide it from the user behind some special keywords and don’t allow us to operate on modules as we can do with other objects. Lua is not one of these languages - its module system is ingenious and yet very simple. First, let’s look at some other languages so you can understand what I’m getting at.

For example, here’s Elixir, a language I sometimes tinker with. I’m using an interactive elixir shell, but it’ll be the same even if I were to put this module into a file:

iex(1)> defmodule Lib do
...(1)>   def foo(), do: "foo"
...(1)>   def bar(), do: "bar"
...(1)> end
{:module, Lib,
 <<70, 79, 82, 49, 0, 0, 5, 180, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 169,
   0, 0, 0, 18, 10, 69, 108, 105, 120, 105, 114, 46, 70, 111, 111, 8, 95, 95,
   105, 110, 102, 111, 95, 95, 10, 97, 116, ...>>, {:bar, 0}}
iex(2)> Lib
Lib
iex(3)> Lib.foo()
"foo"

You can see that after the module was defined, we can see a tuple with some keywords, the module symbol, and other stuff. If we try to examine Lib in the shell it just prints Lib. Yet, we can call the function foo from this module via the dot syntax: Lib.foo()

If we try to inspect what Lib actually is, we can see that it has a bunch of information but, at least to me, this tells almost nothing of substance:

iex(4)> i Lib
Term
  Lib
Data type
  Atom
Module bytecode
  []
Source
  iex
Version
  [249975233796634777878553557963106734802]
Compile options
  [:no_spawn_compiler_process, :from_core, :no_core_prepare, :no_auto_import]
Description
  Call Lib.module_info() to access metadata.
Raw representation
  :"Elixir.Lib"
Reference modules
  Module, Atom
Implemented protocols
  IEx.Info, Inspect, List.Chars, String.Chars

Let’s look at another language, that I don’t often tinker with, but a lot of other people do - Python. This time I’ll put this into a file, as I don’t know the shell capabilities of Python, and if it is possible to do stuff like in Elixir. Here’s the same module:

# lib.py
def foo ():
    return "foo"

def bar ():
    return "bar"

Now, if we import it, we can at least see some indication of what lib is when we try to inspect it:

>>> import lib
>>> lib
<module 'lib' from '/tmp/lib.py'>
>>> lib.foo()
'foo'

What is this <module 'lib' from '/tmp/lib.py'> thing actually? Python defines different keywords for working with modules, like from, import, as, and maybe others, as well as a special set of functions to query the module, like dir. But the module itself is probably some kind of an object:

>>> type(lib)
<class 'module'>

Here’s another language, that I know better than Python or Elixir - Clojure. In Clojure, similar to Elixir, you can define modules in the REPL, so let’s do this:

user> (ns lib)
nil
lib> (defn foo [] "foo")
#'lib/foo
lib> (defn bar [] "bar")
#'lib/bar
lib> (in-ns 'user)
#namespace[user]
user> (require 'lib)
nil
user> (lib/foo)
"foo"

Once again, we can see that the namespace is some kind of #namespace[user] thingy. If we try to reference the namespace as is in the REPL, we get an error:

user> lib
Syntax error compiling at (*cider-repl blog/posts:localhost:44133(clj)*:0:0).
Unable to resolve symbol: lib in this context

However, Clojure provides us with a set of ns-* functions to work with namespaces. For instance, we can get all interned symbols of the namespace as a map:

user> (ns-interns 'lib)
{bar #'lib/bar, foo #'lib/foo}

We can even go back to the lib module, define a private function, return to the user module, and inspect the lib module again:

user> (in-ns 'lib)
#namespace[lib]
lib> (defn ^:private baz [] "baz")
#'lib/baz
lib> (in-ns 'user)
#namespace[user]
user> (ns-interns 'lib)
{baz #'lib/baz, bar #'lib/bar, foo #'lib/foo}
user> (ns-publics 'lib)
{bar #'lib/bar, foo #'lib/foo}

Yet, modules are something hidden from us, we usually don’t operate on them directly.

So that brings us to Fennel, and therefore Lua. I will demonstrate things in Fennel just for better clarity, as it provides better inspection capabilities.

What separates Lua runtime from the rest is that its modules are just tables. When we’re writing a library, or just a module in our application, the module is just a file on the file system, and at the end of that file the last statement returns a table (or a function):

;; lib.fnl
(fn foo [] "foo")
(fn bar [] "bar")
(fn baz [] "baz")

{:foo foo
 :bar bar}

In this case, the table holds references to local functions, and thus the baz function is completely private. When we require this module with the require function it somehow finds the file of this module on disc, loads it, evaluates the top level, and returns the table, which we just then store in a variable, same as any other value:

>> (local lib (require "lib"))
>> (lib.foo)
"foo"

If we try to reference lib here, we’ll see that it is just a table, nothing special:

>> lib
{:bar #<function: 0x55f4cfecd350>
 :foo #<function: 0x55f4cff20610>}
>> (getmetatable lib)
nil

We can check if it has a metatable that hides some module-related trickery, but it’s not. It’s a plain ol’ Lua table.

Lua module system

But Lua doesn’t stop there. I mentioned that require evaluates the top level, so if we put something like (print "hi!") into a module, and require it, we’ll see the text being printed:

;; qux.fnl
(fn qux [] "qux")
(print "hi!")
{:qux qux}

Here’s how it looks in the Fennel REPL:

>> (require :qux)
hi!
{:qux #<function: 0x55f4cfe7a1e0>}	"./qux.fnl"

But if we do a second require the print won’t happen:

>> (require :qux)
{:qux #<function: 0x55f4cfe7a1e0>}

This is because while the modules are plain tables, the require function is actually smarter than just read+eval. Module loading process in Lua ensures that modules are only loaded once, and are cached in a special package table.

package.loaded - caching module definitions

>> package.loaded.qux
{:qux #<function: 0x55f4cfe7a1e0>}

So each time we call (require :something) it first looks if package.loaded.something exists, and if it is, it just returns the value from there. This way, we can unload any module, just by setting a key in the package.loaded to nil, and then require will re-evaluate our module. Fennel actually provides a REPL command for that, called reload.

However, it’s still not that simple. Lua’s require is actually extensible. If the module was not found in the package.loaded the next place require checks is package.preload.

package.preload - preloading modules via anonymous functions

This table stores modules in a similar way, except instead of storing tables, it stores functions that when called will load a module.

We can utilize this in the REPL:

>> (fn package.preload.my-lib []
     (print "loading my-lib")
     (fn hello [name]
       (.. "hi " name "!"))
     {:hello hello})
#<function: 0x55f4d4a23d50>
>> package.preload.my-lib
#<function: 0x55f4d4a23d50>

I’ve created a function in that table, that when called will signal that the module is loaded, define a local function, and return a table, referencing that function. So it’s essentially the same as the file approach, except it is stored in a function, which execution is delayed until someone tries to require the module. Lua was meant to be able to work in filesystem-less environments, and this is one of the solutions for storing libraries.

When we require the my-lib module, we see the loading my-lib message, but instead of providing a second value with the file from where the module was read, the value is :preload::

>> (require :my-lib)
loading my-lib
{:hello #<function: 0x55f4d4a4ecf0>}	":preload:"

This indicates that the library was indeed loaded from the package.preload table. We can then use it as normal:

>> (local my-lib (require :my-lib))
nil
>> (my-lib.hello "Bob")
"hi Bob!"

This already gives us some power, and Fennel utilizes package.preload to provide fully self-contained AOT-compiled Lua files that can be shipped without any external dependencies. Yet, there’s a final step in this system that we can discuss.

package.searchers - extending the module system

If the module was found neither in the package.loaded nor in the package.preload, the require function starts using special package.searchers:

>> package.searchers
[#<function: 0x55f4cfc8b930>
 #<function: 0x55f4cfc8b970>
 #<function: 0x55f4cfc8b9b0>
 #<function: 0x55f4cfc8b9f0>
 #<function: 0x55f4cff32450>]

It’s another table in the package table, that stores a list of functions, that will try to find and load the given module by their own means. Let’s write our own:

>> (table.insert
    package.searchers
    (fn [mod]
      (io.write "searching " mod " with a custom searcher\n")
      (fn loader []
        (fn bye [name]
          (: "bye %s" :format name))
        {:bye bye})
      (values loader mod)))

This searcher function in our case will work for any module regardless of its existence, so by doing this we kinda broke the loading mechanism. Still, this might be useful in some sandboxing scenarios. Alternatively, this mechanism can be used if the environment you run Lua in doesn’t have a real file system, so finding a module requires some trickery, and thus you can extend the general require mechanism. What our function does is not as complicated - it’s similar to what we did in the preload example, except what makes it different is that it basically allows us to do whatever we want before this loader function is given back to the runtime.

We can see it works:

>> (require :asdf)
searching asdf with a custom searcher
{:bye #<function: 0x55ec3e592ae0>}	"asdf"
>> (require :qwer)
searching qwer with a custom searcher
{:bye #<function: 0x55ec3e529460>}	"qwer"
>> (require :asdf)
{:bye #<function: 0x55ec3e592ae0>}
>>

And that it obeys the same caching rules.

This package.searchers table is the sole reason why Fennel works as seamlessly as it is. Setting up Fennel’s package searcher means that when we’re trying to require some module if there is a module.fnl file on the FENNEL_PATH, Fennel will compile this module, and then load it as a normal Lua module. If you want to implement your own Lua-based language, this is definitely the place to embed your compiler.

Now that you have an idea of how modules work in Lua, let’s discuss the main topic of this post - single-file Fennel libraries, and complications related to it.

Fennel libraries with macros

A library in Fennel obeys the exact same rules as a Lua library, except the Fennel compiler adds some steps to the whole process of library loading. As I explained in the searchers topic, the compiler is invoked on the module file before it is given to the Lua runtime. This means, that when we’re loading a Fennel file, it first gets compiled to Lua, and only then it is loaded. This works well up until we introduce compile-time features into our modules - mainly macros.

If we look back at Clojure and Elixir examples, I didn’t use macros in the modules - let’s fix this:

lib> (defmacro unless [test & body] `(when-not ~test ~@body))
#'lib/my-do
lib> unless
Syntax error compiling at (*cider-repl blog/posts:localhost:44133(clj)*:0:0).
Can't take value of a macro: #'lib/unless
lib> (ns-interns 'lib)
{baz #'lib/baz, bar #'lib/bar, unless #'lib/unless, foo #'lib/foo}

If we try to take a value of the macro, we get an error, as macros are a compile-time construct. Yet, it is referenced in the ns-intern as a var. And we can go back to the user module, and use our macro from there:

lib> (in-ns 'user)
#namespace[user]
user> (lib/unless true (println "this should never be printed"))
nil

In Elixir, it is similar, as you also can provide a macro in your module, and upon requiring the module you can use a macro from it. Here’s an example macro from the docs:

iex(2)> defmodule Unless do
...(2)>   defmacro unless(clause, do: expression) do
...(2)>     quote do
...(2)>       if(!unquote(clause), do: unquote(expression))
...(2)>     end
...(2)>   end
...(2)> end
{:module, Unless,
 <<70, 79, 82, 49, 0, 0, 6, 72, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 191,
   0, 0, 0, 20, 13, 69, 108, 105, 120, 105, 114, 46, 85, 110, 108, 101, 115,
   115, 8, 95, 95, 105, 110, 102, 111, 95, 95, ...>>, {:unless, 2}}
iex(3)> Unless.unless true, do: IO.puts "this should never be printed"
this should never be printed
** (UndefinedFunctionError) function Unless.macro_unless/2 is undefined or private.
      However, there is a macro with the same name and arity.
      Be sure to require Unless if you intend to invoke this macro
    Unless.unless(true, [do: :ok])
    iex:3: (file)

Except, we can’t use this macro right away - unlike in Clojure, we need to require the module:

iex(3)> require Unless
Unless
iex(4)> Unless.unless true, do: IO.puts "this should never be printed"
nil

Because Elixir’s macros are entirely Elixir’s concept, BEAM knows nothing about them, and thus you can use them only in Elixir, and it explains why the shell requires some additional steps. Perhaps these can be automated, as Clojure can do it without any additional require calls, but I don’t know for sure how it is done in Clojure or in Elixir.

I’m bringing Elixir and Clojure here because both languages are hosted on their respective runtimes - Erlang’s BEAM and JVM. Fennel is similarly hosted on Lua, but on the other hand, is different from these two when it comes to macros.

Let’s add the unless macro to the lib.fnl module:

;; lib.fnl
(fn foo [] "foo")
(fn bar [] "bar")
(macro unless [test ...]
  `(when (not ,test) ,...))
{:foo foo :bar bar :unless unless}

Trying to load this module, however, will result in a compile error:

(local lib (require :lib))
./lib.fnl:5:27 Compile error: tried to reference a macro without calling it

{:foo foo :bar bar :unless unless}
* Try renaming the macro so as not to conflict with locals.

We can’t simply export our macro alongside our functions. What should we do then?

The fennel language reference suggests that macros can’t be referenced and exported alongside functions And that macro is supposed to be used only as file-local macros, and thus these can’t be exported. Additionally to that, when writing macros, we can’t access any surrounding code. The reference section then says that in order to export macros we can put them into a separate module and use import-macros to import them to the arbitrary code.

The whole macro system in Fennel is not as advanced as in other lisps or as in Elixir, but it gets the job done, it just requires some squats to be done.

Macro modules

A macro module is exactly the same as an ordinary Fennel module - it shares the same file extension, same syntax, but is meant to be used in conjunction with import-macros or require-macros specials. To write a macro module we can simply put our unless macro into a separate file, replacing macro with fn:

;; unless.fnl
(fn unless [test ...]
  `(when (not ,test) ,...))

{:unless unless}

That’s right, in Fennel, macros are ordinary functions running in a compiler environment, and now we actually can export them as an ordinary table. The compiler environment defines additional functions that we can use in macros, such as quote and unquote which we’re using here in the forms of ` and ,. Now we can require this module with import-macros:

>> (import-macros Unless :unless)
nil
>> (Unless.unless true (print "this should never be printed"))
nil
>> (Unless.unless false (print "this should be printed"))
this should be printed
nil

There’s also another special that can import macros from a module: require-macros. Unlike import-macros it doesn’t put macros in a binding, instead, it pollutes the current scope with all of the macros from the module:

>> (require-macros :unless)
nil
>> (unless true (print "this should be printed"))
nil

So (import-macros name module) is much like (local name (require module)) but for macro modules, and it uses require-macros underneath.

import-macros and require

Speaking of the relationship between import-macros and require, they indeed work similarly. Fennel’s developers understood that Lua’s module system is very flexible and decided to make a similar interface for macros.

You couldn’t see it, but in the previous example, when I called require-macros on the unless module again, it didn’t re-read the module from disk. Instead, it used the same technique of caching, much like package.loaded but instead, it uses fennel.macro-loaded to store macros. We can see that by requiring Fennel in the REPL and inspecting the table:

>> (local fennel (require :fennel))
nil
>> fennel.macro-loaded
{:unless {:unless #<function: 0x56224adc40a0>}}

We can even do the same thing as with the package.loaded, and set the unless module to nil with (set fennel.macro-loaded.unless nil) If we then add some side-effecting code, like (print "macros!") to the module, and require it again, we’ll see that Fennel will re-read our module:

>> (set fennel.macro-loaded.unless nil)
nil
>> (import-macros Unless :unless)
macros!
nil
>> (import-macros Unless :unless)
nil

But only once, much like with ordinary require.

Similarly to Lua, Fennel provides its own table with searcher functions for macros:

>> fennel.macro-searchers
[#<function: 0x56224ac5a740> #<function: 0x56224ac5a7f0>]

So the system is extremely similar, the only thing it lacks is something akin to package.preload but for macros. With that in mind let me introduce you to the library problem.

The library problem

Let’s imagine, we want to write a library that provides you with a lot of functions, and with some convenience macros that remove some boilerplate. In reality, almost anything can be done without macros, but macros are great for writing code that you don’t want to write yourself. So let’s say we want to add polymorphic dispatch to Fennel by adding multimethods.

Let’s write the functions first:

(fn multi [dispatch-fn]
  (let [methods {}]
    (setmetatable
     {:add-method (fn [dispatch-val fn1]
                    (tset methods dispatch-val fn1))
      :get-method (fn [dispatch-val]
                    (. methods dispatch-val))
      :remove-method (fn [dispatch-val]
                       (tset methods dispatch-val nil))}
     {:__call (fn [self ...]
                (let [method (self.get-method (dispatch-fn ...))]
                  (method ...)))})))

(fn add-method [multi dispatch-val method]
  (multi.add-method dispatch-val method))

(fn get-method [multi dispatch-val]
  (multi.get-method dispatch-val))

(fn remove-method [multi dispatch-val]
  (multi.remove-method dispatch-val))

{:multi multi
 :add-method add-method
 :get-method get-method
 :remove-method remove-method}

The idea is simple. We create an object with the function multi and give it a dispatch-fn - the function it will invoke with the arguments to determine what method it needs to call. If there is a method, it will be called with the same arguments as the dispatch-fn functions. I also provided convenience functions that provide some abstraction over the inner implementation of the multimethod object. Let’s try it:

>> (local mult (require :multimethods))
nil
>> (local interact (mult.multi (fn [a b] (.. a.type " " b.type))))
nil
>> (mult.add-method interact "bunny lion" (fn [bunny lion] "bunny runs away"))
nil
>> (mult.add-method interact "lion bunny" (fn [lion bunny] "lion eats bunny"))
nil
>> (mult.add-method interact "lion lion" (fn [lion1 lion2] "lions fight"))
nil
>> (mult.add-method interact "bunny bunny" (fn [bunny1 bunny2] "bunnies mate"))
nil
>> (interact {:type "bunny"} {:type "lion"})
"bunny runs away"
>> (interact {:type "lion"} {:type "bunny"})
"lion eats bunny"

I opted for a simple example, where I just get the type field of each object and concatenate them into a string. Then we dispatch on that string.

The library is now complete, but it’s a bit tedious to type all of these anonymous functions every time, let’s add an interface similar to how Clojure does this. We’ll add two macros defmulti, which given a name and a dispatch fn will create a local variable by itself, and defmethod which will act the same as add-method except it will construct the anonymous function for us. We start by creating the macro module:

(fn defmulti [name dispatch-fn]
  `(local ,name (mult.multi ,dispatch-fn)))

(fn defmethod [name dispatch-val ...]
  `(mult.add-method ,name ,dispatch-val (fn ,...)))

{:defmulti defmulti
 :defmethod defmethod}

We already have the multimethod itself, so let’s try our defmethod macro:

>> (require-macros :multimethod-macros)
nil
>> (defmethod interact "ship rock" [ship _rock] (.. ship.name " sinks"))
nil
>> (interact {:type "ship" :name "Wasa"} {:type "rock"})
"Wasa sinks"

It works, but did you notice the problem? Let’s try this again, but instead, let’s call our library something else in the REPL:

>> (local mmethods (require :multimethods))
nil
>> (require-macros :multimethod-macros)
nil
>> (defmulti interact (fn [a b] (.. a.type " " b.type)))
unknown:2:? Compile error: unknown identifier: mult

Looking into the macroexpansion of defumilti we see that it does exactly what we told it:

>> (macrodebug (defmulti interact (fn [a b] (.. a.type " " b.type))))
(local interact (mult.multi (fn [a b] (.. a.type " " b.type))))

Except, the mult is nowhere to be found.

So, here’s the problem - we need to reuse functions from our function module in the macro module. Unfortunately, there’s literally no way to do it without making some assumptions about the module name. Instead of using available binding in the macro, we can require the library because it’s cached:

(fn defmulti [name dispatch-fn]
  `(local ,name (let [mmethods# (require :multimethods)]
                  (mmethods#.multi ,dispatch-fn))))

(fn defmethod [name dispatch-val ...]
  `(let [mmethods# (require :multimethods)]
     (mmethods#.add-method ,name ,dispatch-val (fn ,...))))

{:defmulti defmulti
 :defmethod defmethod}

With this change, everything starts working:

>> (local mmethods (require :multimethods))
nil
>> (require-macros :multimethod-macros)
nil
>> (defmethod interact "ship rock" [ship _rock] (.. ship.name " sinks"))
nil
>> (interact {:type "ship" :name "Wasa"} {:type "rock"})
"Wasa sinks"

But there’s another problem - what if the user of our library doesn’t like that the module file name is multimethods.fnl and wants to rename it? Suppose they already have their own multimethods module and want to try out ours alongside it, so they rename it to aorst-multimethods.fnl. Required macros will suddenly stop working because in our macros we hardcoded the multimethods module name.

init.fnl and init-macros.fnl

One solution to this problem is to use Lua’s other hidden module mechanic, related to how the searchers work.

In Lua, we can load a file by giving require a module name that matches that file, like require "foo" will probably load the foo.lua file. However, we can also load a directory in the same way - instead of creating foo.lua we can create a directory foo and write the contents of foo.lua into foo/init.lua. Then, calling require "foo" will seamlessly load code from init.lua, but it will look like we’re loaded a directory.

Fennel supports the same thing for ordinary modules with init.fnl, but in addition to that, it also supports this for macro modules with init-macros.fnl. We can re-implement our multimethod library this way, but we’ll change how we require the library in the macros:

;; multimethods-lib/init-macros.fnl
(local lib-module ...) ;; for relative require

(fn defmulti [name dispatch-fn]
  `(local ,name (let [mmethods# (require ,lib-module)]
                  (mmethods#.multi ,dispatch-fn))))

(fn defmethod [name dispatch-val ...]
  `(let [mmethods# (require ,lib-module)]
     (mmethods#.add-method ,name ,dispatch-val (fn ,...))))

{:defmulti defmulti
 :defmethod defmethod}

This trick is called relative-require. It’s quite common in Lua, but I decided to use it when I was trying to ship huge libraries with lots of macros and inner modules.

When the module is loaded, Lua passes the module name to the module as an argument. This (local lib-module ...) captures it in the lib-module variable, and then we can refer to it in our macros. If we now look at what our macros expand to, we’ll see that they use the module name correctly:

>> (local mmethods (require :multimethods-lib))
nil
>> (require-macros :multimethods-lib)
nil
>> (macrodebug (defmulti interact (fn [a b] (.. a.type " " b.type))))
(local interact
  (let [mmethods_2_auto (require "multimethods-lib")]
    (mmethods_2_auto.multi (fn [a b] (.. a.type " " b.type)))))
nil

Now our macros are unbound from the library name itself. However, shipping library in this way is not that great. For example, here’s how one of my libraries that uses this technique looks if you peek into the library root:

.
├── CHANGELOG.md
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── doc/
├── fennel-test/
├── impl/
├── init.fnl
├── init-macros.fnl
├── LICENSE
├── Makefile
├── README.md
└── tests/

Basically, it’s the project’s root, and if you want to use this library, the easiest way is to just git-clone the project and require it. But it means that there are lots of unrelated files, and there’s no way to know what files are related unless the project has some kind of manifest, which is not the case for Fennel projects. It would be much better if you could just grab only the init.fnl file, name it however you want, and require both macros and functions from it.

Require-as-include

Fennel has a feature that can do almost that. For multi-file libraries, we can AOT compile all of the Fennel code into Lua, and the compiler will inject all of the dependencies in an interesting way. It does so by tracking what modules we require and splicing them into the code by using package.preload trick I’ve described way earlier in this post:

(fn foo []
  "some-lib's foo")

{: foo}

(local some-lib (require :some-lib))

(fn my-foo []
  (some-lib.foo))

{:my-foo my-foo}

If we call fennel --require-as-include -c my-lib.fnl, we’ll get the following code:

local some_lib
package.preload["some-lib"] = package.preload["some-lib"] or function(...)
  local function foo()
    return "some-lib's foo"
  end
  return {foo = foo}
end
some_lib = require("some-lib")
local function my_foo()
  return some_lib.foo()
end
return {["my-foo"] = my_foo}

As can be seen, the whole module definition of some-lib was included in the file as an entry in the package.preload. In the later code, it is required and used as normal.

However, this doesn’t work with macros, as there is no fennel.macro-preloaded or something similar. And for a long enough time, I wasn’t sure if it was possible to mimic this behavior at compile time. But then I remembered that we indeed have the module name at compile time, as clearly shown in the relative-require example.

Mimicking preload behavior for macros

Here’s what I came up with:

(eval-compiler
  (local lib-module ...)

  (fn defmulti [name dispatch-fn]
    `(local ,name
       (let [mmethods# (require ,lib-module)]
         (mmethods#.multi ,dispatch-fn))))

  (fn defmethod [name dispatch-val ...]
    `(let [mmethods# (require ,lib-module)]
       (mmethods#.add-method ,name ,dispatch-val (fn ,...))))

  (tset macro-loaded lib-module
        {:defmulti defmulti
         :defmethod defmethod}))

(fn multi [dispatch-fn]
  (let [methods {}]
    (setmetatable
     {:add-method (fn [dispatch-val fn1]
                    (tset methods dispatch-val fn1))
      :get-method (fn [dispatch-val]
                    (. methods dispatch-val))
      :remove-method (fn [dispatch-val]
                       (tset methods dispatch-val nil))}
     {:__call (fn [self ...]
                (let [method (self.get-method (dispatch-fn ...))]
                  (method ...)))})))

(fn add-method [multi dispatch-val method]
  (multi.add-method dispatch-val method))

(fn get-method [multi dispatch-val]
  (multi.get-method dispatch-val))

(fn remove-method [multi dispatch-val]
  (multi.remove-method dispatch-val))

{:multi multi
 :add-method add-method
 :get-method get-method
 :remove-method remove-method}

Finally, we can require our library and macros from the same module, and it doesn’t have to be a directory! Here’s how it looks:

>> (local mm (require :combined-multimethods))
nil
>> (require-macros :combined-multimethods)
nil
>> (defmulti interact (fn [a b] (.. a.type " " b.type)))
nil
>> (defmethod interact "ship rock" [ship _rock] (.. ship.name " sinks"))
nil
>> (interact {:type "ship" :name "Wasa"} {:type "rock"})
"Wasa sinks"

So, it’s a combination of techniques, but what exactly is eval-compiler doing here?

If you look closely at evel-compiler body, you’ll see that instead of exporting the table at the end of the scope we simply set it to the macro-loaded table. We use the module name which was set to combined-multimethods when the module was required, and thus the macro-loaded.combined-multimethods now stores the table with macro definitions. Then, when require-macros looks up the macro module combined-multimethods, it sees that it was already loaded, and simply returns it without trying to find a file with these macros. Alternatively, we could define a macro-searcher that, given a specific module-name would do the same thing, loading macros while avoiding trying to find a file, but it’s more work for almost no benefit over what we have here already.

Preprocessor-as-include

There’s still a problem though. What if we want to write a library, that uses another library, that uses macros in this way?

One such example is my cljlib library I keep re-implementing every few months or so. In that library I also wanted to ship macros and functions in the same module, but here’s a problem - this library depends on other libraries, namely lazy-seq and itable. While itable doesn’t provide any macros, lazy-seq does, and cljlib re-uses these macros.

Before some recent changes I made to both lazy-seq and cljlib you had to fiddle with FENNEL_PATH and FENNEL_MACRO_PATH in order to use cljlib, and you had to check out the full repository with submodules. I feel like this is way too much work for most people, and I don’t want anyone to suffer that much, so I changed it to use the trick above and embedded other libraries into it by manually writing the package.preload trick instead of relying on --require-as-include flag. Here’s how it looks in the src/cljlib.fnl file:

;;; itable
(set package.preload.itable
  (or package.preload.itable
      (fn [...]
;;;###include itable/src/itable.fnl
        )))

;;; lazy-seq
(set package.preload.lazy-seq
  (or package.preload.lazy-seq
      (fn [...]
;;;###include lazy-seq/lazy-seq.fnl
        )))

;;; cljlib
(eval-compiler
  (local lib-name (or ... :cljlib))
  ;; macros ...
  )
;; functions ...

These ;;;###include path comments are nothing special from the perspective of Fennel, but I made a small build.fnl script for the library that takes care of them:

(fn spit-lib [path to]
  (with-open [lib (io.open path)]
    (each [line (lib:lines)]
      ;; patching compile-time variable used to store macro module
      ;; name because when loafing the combined file it will always
      ;; equal the main module and will break macros in vendored
      ;; libraries.
      (case (line:match "%(local lib%-name %(or %.%.%. (.*)")
        name (to:write (.. "(local lib-name (or " name "\n"))
        _ (to:write line "\n")))))

(with-open [cljlib (io.open "./cljlib.fnl" :w)]
  (let [main (io.open "src/cljlib.fnl")]
    (each [line (main:lines)]
      (case (line:match ";;;###include (.*)")
        (path) (spit-lib path cljlib)
        _ (cljlib:write line "\n")))))

It is meant to be used only for development, the library is pre-built and ready for anyone to grab. What’s interesting about this script is that it also patches the library it includes. This was a pain to figure out.

You see, in the lazy-seq library I’m using the same eval-compiler trick, and it in turn uses the (local lib-name (or ... :lazy-seq)) trick. However, when this library is spliced into our main file as is, when we load this file as (require :cljlib), the ... becomes "cljlib", and the lazy-seq library, included in this file will write its macros into macro-loaded.cljlib instead of supposed macro-loaded.lazy-seq. This can’t be worked around, as this happens way before package.preload.lazy-seq runs, so the ... provided by the function in that table is never used by eval-compiler. The only way to delay the eval-compiler is to put it into the file or use the searcher that will evaluate this block of code by itself which is way too tricky.

So I made a hack, that while splicing the library code scans for any occurrences of the (local lib-name (or ... pattern, and replaces it with the name used in the or expression, without checking .... This way, the (local lib-name (or ... :lazy-seq)) becomes (local lib-name (or :lazy-seq)) in the final version of cljlib.fnl. Of course, it is extremely finicky, as even an introduction of a newline between lib-name and (or ...) expression will break this. Changing lib-name to something else will also break this down.

But even with this kind of monkey-patching it works well enough, as I control what code goes in both libraries, and the user doesn’t need to think about it at all. All that is needed to do is just to grab the cljlib.fnl from the repo root and put it into the project. I’ve updated Fenneldoc to use the newer version of cljlib and was pleased to see that it still works.

Fixing macros in Fennel

Even with the ability to ship macros in the same file as functions, I find it weird that Fennel’s macro system has all sorts of other limitations.

Firstly, the inability to use functions defined alongside macros is really bad. This is one of the reasons we have to do all these squats with relative-require stuff and require the library itself from the macro. Which in turn makes us more error-prone, if some weird environment will not set the module args (...) properly.

Another thing is that macros can’t use other macros unless they’re defined in a separate module, or in the eval-compiler trick I was featuring in this post. If you define a macro with the macro special, it must be self-contained. Otherwise, you have to move it somewhere. I mean, it’s weird that Fennel has a proper implementation of macros sitting in the compiler, but for some reason macro and macros specials don’t use it.

What I think would fix all this mess is this:

First, we need to ditch macro modules completely. They’re inferior to ordinary fennel modules and only make things complicated.

Next, we need to make it so the macro special sets the macro in the macro-loaded entry of the current module, much like I manually do in the eval-compiler block.

And lastly, when compiling the module to Lua, just load the module at compile time as a whole! This way not only macros will be able to use other macros, but they’ll also be able to use other ordinary functions at compile time!

The only problem left is how to make it so the generated code can use definitions from the library. And I think I know the way, and it’s similar to how Clojure handles things.

So basically, same as in Clojure, macros can only use public definitions. What this means is that these definitions are in the packagr.loaded by the time we’ve required the library. Therefore, the backquote can safely fully qualify any unqualified symbol in the calling position!

In other words, the macro bar from this file:

(fn foo [val] val)

(macro bar [expr]
  `(foo ,expr))

{:foo foo}

After macroexpanding will become:

>> (require-macros (doto :lib require))
nil
>> (macrodebug (bar (+ 1 2 3)))
(package.loaded.lib.foo (+ 1 2 3))

And will work just fine, as it can reach foo, as it is already in the package.loaded by that time. Which is similar to what Clojure does:

user> (defn foo [val] val)
#'user/foo
user> (defmacro bar [expr] `(foo ~expr))
#'user/bar
user> (macroexpand '(bar (+ 1 2 3)))
(user/foo (+ 1 2 3))

And as the final change, we need to make that the fennel.searcher properly sets up macro-loaded in order to ditch the awkward (doto :lib require) thing.

Anyhow, that’s all I got for today. I know that I’m writing a lot of devlog posts about my small game dev marathon, so I try to write some posts in-between so that my blog doesn’t become too focused on one topic. Hope that this was an interesting read!