Andrey Listopadov

Clojure on Fennel part four: Parsing (again)

Mon, 29 Jun 2026 00:29:00 +0300

Other than that, the parsing is complete, and we can look at the compiler part of the ClojureFnl project. But that’s gonna be in the next post.

Sike!

Wasn’t my intention to fake out like this again, but while working on the compiler, I had an important realization that led me to redesign the parser completely. And it’s a bit of a shame, because I already had a decent part of the compiler working, being able to run the REPL and do various cool things with it. But, better sooner than later, I guess.

I had a good amount of the post about the compiler already written too, but now I’ll have to discard all of that. So instead, I decided to talk about the new parser by explaining why the old one failed me. And to do so, I’ll give you a brief look on how the compiler currently works with parsed code.

Old parser and compiler

So, in the previous post, we’ve reached a point where the parser can read code into a tagged tree. I spent a fair amount of time discussing how I wanted this specifically, and now it feels like I’m backpedalling, but tagged tree is not the way forward. Here’s the idea.

Currently, we’re reading this expression (+ 1 2) into:

[:code
 [:list
  [:symbol "+"]
  [:whitespace " "]
  [:number "1"]
  [:whitespace " "]
  [:number "2"]]]

The compiler then walks this tree, taking the first value of each node to decide what to do. First, it sees :code - that’s just an entry point, containing multiple top-level expressions. Next, it sees :list and dispatches to a function dedicated to compiling lists.

This function checks the first element of the list, sees that it is a :symbol, and checks whether this symbol is somehow special. While + doesn’t look that special, it is to the compiler, because Clojure’s + and Fennel’s + are not the same thing. In Clojure, + is a function, in Fennel, it is a special.

So the compiler replaces + with clojure_core_ns.add, and then proceeds over the rest of the forms in the list, recursively calling itself over each. In the end, we get this, written into a string builder:

(clojure_core_ns.add 1 2)

This is a somewhat simplified explanation, because I’m omitting scope resolution, symbol shadowing, and other such things that the compiler currently tracks. For actual specials, like Clojure’s let*, the story is a bit different too. The compiler has a dedicated compile-special function for all Clojure specials provided by cljlib as Fennel macros. And that’s where things started to go haywire.

Look at this code:

(def ^:private foo 42)

It’s read like this:

[:list
 [:symbol "def"]
 [:whitespace " "]
 [:metadata
  [:metadata-entry [:keyword ":private"]]
  [:whitespace " "]
  [:symbol "foo"]]
 [:whitespace " "]
 [:number "42"]]

Here’s a problem - the parser reads metadata ^:private, and metadata in Clojure is attached to symbols, so the grammar I use reads the next value after the metadata and wraps it into a single node:

[:metadata
 [:metadata-entry [:keyword ":private"]]
 [:whitespace " "]
 [:symbol "foo"]]

The compiler, while compiling this list, sees def as the first symbol and enters compile-special.def. But def expects a symbol to bind the value to, while here we have a different node type, called :metadata. So my compiler had to account for all cases where metadata can appear - and it wasn’t easy to do.

I tried to side-step this by always expecting metadata, because it can appear almost anywhere, and discarding it, because Fennel’s concept of metadata is a bit different. In which I mostly succeeded. But this wasn’t the only problematic thing to support.

Here’s another example, now from Edamame:

(defn- read-token
  "Read in a single logical token from the reader"
  ^String [#?(:clj rdr :cljs ^not-native rdr :cljr rdr) _kind initch]
  (loop [sb #?(:clj (StringBuilder.)
               :cljs (StringBuffer.)
               :cljr (StringBuilder.))
         ch initch]
    (if (or (whitespace? ch)
            (macro-terminating? ch)
            (nil? ch))
      (do (when ch
            (r/unread rdr ch))
          (str sb))
      (recur #?(:clj (.append sb ch) :cljs (.append sb ch) :cljr (.Append sb (str ch))) (r/read-char rdr)))))

It reads into this tagged tree:

[:list
 [:symbol "defn-"]
 [:whitespace " "]
 [:symbol "read-token"]
 [:whitespace "\n  "]
 [:string "\"Read in a single logical token from the reader\""]
 [:whitespace "\n  "]
 [:metadata
  [:metadata-entry [:symbol "String"]]
  [:whitespace " "]
  [:vector
   [:conditional
    [:list
     [:keyword ":clj"]
     [:whitespace " "]
     [:symbol "rdr"]
     [:whitespace " "]
     [:keyword ":cljs"]
     [:whitespace " "]
     [:metadata
      [:metadata-entry [:symbol "not-native"]]
      [:whitespace " "]
      [:symbol "rdr"]]
     [:whitespace " "]
     [:keyword ":cljr"]
     [:whitespace " "]
     [:symbol "rdr"]]]
   [:whitespace " "]
   [:symbol "_kind"]
   [:whitespace " "]
   [:symbol "initch"]]]
 [:whitespace "\n  "]
 [:list
  [:symbol "loop"]
  [:whitespace " "]
  [:vector
   [:symbol "sb"]
   [:whitespace " "]
   [:conditional
    [:list
     [:keyword ":clj"]
     [:whitespace " "]
     [:list [:symbol "StringBuilder."]]
     [:whitespace "\n               "]
     [:keyword ":cljs"]
     [:whitespace " "]
     [:list [:symbol "StringBuffer."]]
     [:whitespace "\n               "]
     [:keyword ":cljr"]
     [:whitespace " "]
     [:list [:symbol "StringBuilder."]]]]
   [:whitespace "\n         "]
   [:symbol "ch"]
   [:whitespace " "]
   [:symbol "initch"]]
  [:whitespace "\n    "]
  [:list
   [:symbol "if"]
   [:whitespace " "]
   [:list
    [:symbol "or"]
    [:whitespace " "]
    [:list [:symbol "whitespace?"] [:whitespace " "] [:symbol "ch"]]
    [:whitespace "\n            "]
    [:list
     [:symbol "macro-terminating?"]
     [:whitespace " "]
     [:symbol "ch"]]
    [:whitespace "\n            "]
    [:list [:symbol "nil?"] [:whitespace " "] [:symbol "ch"]]]
   [:whitespace "\n      "]
   [:list
    [:symbol "do"]
    [:whitespace " "]
    [:list
     [:symbol "when"]
     [:whitespace " "]
     [:symbol "ch"]
     [:whitespace "\n            "]
     [:list
      [:symbol "r/unread"]
      [:whitespace " "]
      [:symbol "rdr"]
      [:whitespace " "]
      [:symbol "ch"]]]
    [:whitespace "\n          "]
    [:list [:symbol "str"] [:whitespace " "] [:symbol "sb"]]]
   [:whitespace "\n      "]
   [:list
    [:symbol "recur"]
    [:whitespace " "]
    [:conditional
     [:list
      [:keyword ":clj"]
      [:whitespace " "]
      [:list
       [:symbol ".append"]
       [:whitespace " "]
       [:symbol "sb"]
       [:whitespace " "]
       [:symbol "ch"]]
      [:whitespace " "]
      [:keyword ":cljs"]
      [:whitespace " "]
      [:list
       [:symbol ".append"]
       [:whitespace " "]
       [:symbol "sb"]
       [:whitespace " "]
       [:symbol "ch"]]
      [:whitespace " "]
      [:keyword ":cljr"]
      [:whitespace " "]
      [:list
       [:symbol ".Append"]
       [:whitespace " "]
       [:symbol "sb"]
       [:whitespace " "]
       [:list [:symbol "str"] [:whitespace " "] [:symbol "ch"]]]]]
    [:whitespace " "]
    [:list [:symbol "r/read-char"] [:whitespace " "] [:symbol "rdr"]]]]]]

Yes, it’s abysmal, but bear with me. I had to work with this, after all, thinking that this is a blessing.

The compiler sees defn- and enters the compile-special.defn- function. Defn expects a function name, then a vector for its arguments. Here, instead of the vector, we have metadata node again. This is fine, since I just mentioned above that I managed to sidestep this problem.

Since this is an arglist, we need to compile it in a special way, adding its symbols to the function scope, etc. However, instead of arguments, we see the conditional node:

[:vector
 [:conditional
  [:list
   [:keyword ":clj"] [:whitespace " "] [:symbol "rdr"] [:whitespace " "]
   [:keyword ":cljs"] [:whitespace " "] [:metadata [:metadata-entry [:symbol "not-native"]] [:whitespace " "] [:symbol "rdr"]] [:whitespace " "]
   [:keyword ":cljr"] [:whitespace " "] [:symbol "rdr"]]]
 [:whitespace " "]
 [:symbol "_kind"]
 [:whitespace " "]
 [:symbol "initch"]]

I didn’t think it was allowed in Clojure to use conditional reading inside forms like this. But apparently it’s OK, and my compiler failed to deal with it.

So now, any node can not only be a metadata node, but also a conditional node. This complicates things, but at this point I’m still thinking that I can persevere.

So I did. I added support for almost all specials provided by cljlib. The main thing that was left to do were macros. And then it hit me:

I can’t do macros like that!

Why? Why, of course, because macros don’t emit tagged trees that my compiler understands. They emit code!

Here’s a simple macro:

(defmacro unless [test & body]
  `(when (not ~test)
     ~@body))

We can parse it:

[:list
 [:symbol "defmacro"] [:whitespace " "] [:symbol "unless"] [:whitespace " "]
 [:vector
  [:symbol "test"] [:whitespace " "] [:symbol "&"] [:whitespace " "] [:symbol "body"]]
 [:whitespace " "]
 [:backtick
  [:list
   [:symbol "when"] [:whitespace " "]
   [:list [:symbol "not"] [:whitespace " "] [:unquote [:symbol "test"]]]
   [:whitespace " "] [:unquote-splicing [:symbol "body"]]]]]

We can probably even compile it to something that we could then call during the compiler step. However, what will (unless true (println 42)) thing return?

(when (not true) (println 42)), of course. It’s a Lisp macro, what else did you expect?

But what does the compiler expect?

[:list
 [:symbol "when"]
 [:whitespace " "]
 [:list [:symbol "not"] [:whitespace " "] [:symbol "true"]]
 [:whitespace " "]
 [:list [:symbol "println"] [:whitespace " "] [:number "42"]]]

Oh, it wants this.

I have to convert macro’s output into a string, parse it, and feed it into the compiler if I want macros to work at all. And that’s BAD. If only I realized this sooner!

This was the final nail in the coffin of the tagged tree approach for my project. I knew I had to rewrite the parser to emit actual data structures that I’ll be able to emit from macros as well, and compile them.

New parser reader

So I decided that I need a proper lisp reader that will produce data structures:

;; Welcome to Fennel Proto REPL 0.6.4-dev
;; Fennel version: 1.7.0-dev
;; Lua version: PUC Lua 5.5
;; Work directory: ~/Projects/fennel/ClojureFnl/
>> (local reader (require :impl.reader))
nil
>> (reader.read-string "(def ^:private foo 42)")
(def foo 42)
>> (local {: meta : second} (require :clojure.core))
nil
>> (meta (second (reader.read-string "(def ^:private foo 42)")))
{:private true}

As can be seen, the new reader module provides the read-string function that produces data structures. Notably, there are no longer any metadata nodes - metadata is assigned to the symbol foo in this case.

Same goes for the reader conditionals:

>> (reader.read-string "[0 #?(:clj 1 :cljfnl 2) #?@(:clj [2 3] :cljfnl [3 4]) 5]")
[0 2 3 4 5]

These are now correctly spliced at read time.

This also solved macro problems for the most part:

>> (reader.read-string "`(+ ~x ~@y ~z)")
(clojure.core/seq
 (clojure.core/concat
  (clojure.core/list (quote clojure.core/+))
  (clojure.core/list x)
  y
  (clojure.core/list z)))

And it matches what Clojure itself does:

Clojure 1.12.5
user=> (read-string "`(+ ~x ~@y ~z)")
(clojure.core/seq
 (clojure.core/concat
  (clojure.core/list (quote clojure.core/+))
  (clojure.core/list x)
  y
  (clojure.core/list z)))

Due to this change, the compiler doesn’t have to know about syntax quote (`) at all, and thus macros will correctly return these kinds of lists.

So, now we can read Clojure code into data structures. Previously, the parser returned [:code A B ...] kind of result, where A, B, and the rest are all top-level forms. New reader also does this, but returns them as a list:

>> (reader.read-string-all "1 \"str\" :keyword ::namespaced #:map{:key :val}")
(1 "str" :keyword :user/namespaced {:map/key :val})

However, there’s a problem.

Consider the ::namespaced keyword. Both my reader and Clojure read it as :user/namespaced, but this is because the default namespace in the REPL is user. For my parser, it’s mostly an arbitrary choice because it doesn’t know anything about runtime.

Clojure 1.12.5
user=> (read-string "::x")
:user/x
user=> (ns foo)
nil
foo=> (read-string "::x")
:foo/x

And when compiling a file, the file might have an ns declaration, or even multiple of them:

(ns foo)

(println ::x)

(ns bar)

(println ::x)

If we were to run this code, we would see :foo/x then :bar/x.

My parser currently reads all input, meaning it transforms all top-level forms from text to data. However, this also means that it doesn’t understand anything about namespaces.

Previously, this was fine because the compiler handled this - the tagged tree didn’t resolve anything. Currently, however, we need to construct namespaced keys, and we need to know the namespace. But it’s not possible unless we parse the input expression by expression, instead of all at once.

Hence, I had to update the grammar so it could parse expression by expression. This way, I could read source code form by form, and compile each form separately. And if the compiler encounters an ns declaration, it could update some state, so the reader would know how to read namespaced keywords, and other things that may include namespaces.

But we still need to pass this state to the reader. And my reader is based on a PEG grammar.

Luckily for me, the lpeg library I use has a Carg function that can pass additional arguments into the PEG parser. This way I can write my transformation function T:

(fn T [tag patt]
  (/ (* (Ct (* (Cc tag) patt)) (Carg 1))
     (fn [node state]
       (case node
         ;; ----8<----
         [:macro-keyword bo data]
         (read-macro-keyword data bo state)
         [:conditional data]
         (read-conditional data state)
         [:conditional-splicing data]
         (conditional-splicing data state)
         ;; ----8<----
         _ _))))

This state can be passed anew after reading each expression, or mutated in place - either way, the reader now knows how to access state. The reader itself is still stateless.

The reader now supports all Clojure syntax and produces data structures:

1                                                   ;; 1
0.5                                                 ;; 0.5
1e10                                                ;; 10000000000.0
1/2                                                 ;; 1/2
1N                                                  ;; 1N
1.33333333333333333333333333333333M                 ;; 1.33333333333333333333333333333333M
0xFF                                                ;; 255
017                                                 ;; 15
16rFF                                               ;; 255
\c                                                  ;; "c"
"string"                                            ;; "string"
\u0041                                              ;; "A"
\o101                                               ;; "A"
:keyword                                            ;; :keyword
:namespaced/keyword                                 ;; :namespaced/keyword
::auto-keyword                                      ;; :user/auto-keyword
symbol                                              ;; symbol
namespaced/symbol                                   ;; namespaced/symbol
#'var                                               ;; (var var)
'quoted-symbol                                      ;; (quote quoted-symbol)
(list)                                              ;; (list)
[vector]                                            ;; [vector]
{hash map}                                          ;; {hash map}
#:namespaced{:map 42}                               ;; {:namespaced/map 42}
#::{:auto :namespaced}                              ;; {:user/auto :namespaced}
#{hash set}                                         ;; #{set hash}
#"regexp"                                           ;; "regexp"
@dereferencing                                      ;; (clojure.core/deref dereferencing)
#(+ %1 %2)                                          ;; (fn* [p__0__ p__1__] (+ p__0__ p__1__))
^:meta data                                         ;; data
#_discard                                           ;;
[#?(:cljfnl "reader") #?@(:cljfnl [:conditionals])] ;; ["reader" :conditionals]
nil                                                 ;; nil
true                                                ;; true
false                                               ;; false
##NaN                                               ;; .nan
##Inf                                               ;; .inf
#inst "2022"                                        ;; #<inst: 2022-01-01T00:00:00.000-00:00>
#uuid "c6e8050a-789b-4305-b518-8f0f7c31da7a"        ;; "c6e8050a-789b-4305-b518-8f0f7c31da7a"

Note about data structures: even though we’re working in Fennel, the produced maps, vectors and lists are custom data structures that are implemented in the cljlib library. These are implementations of persistent data structures I’ve talked about in part 2.

Further work

With that, I can work on the compiler, now for real. It’s a shame that I basically have to do all the work on the compiler again from scratch, because the underlying data has changed so much, but this simplifies a lot of things, so I’m OK with that.

One thing that the reader still doesn’t support yet is read-time evaluation with #=expr. This requires a working compiler, and I’ll have to first implement it, then integrate it back into the reader.

Another thing I was thinking about was to drop the LPEG parser altogether. Maybe, when I have the compiler working and have support for all Clojure runtime semantics in place, I’ll make a fork of the Edamame parser and replace the current reader with it, adding support for ClojureFnl via reader conditionals. This will remove a C library dependency, which is a good thing for distribution. I still have another C library for arbitrary-precision numbers, but it can be worked around.

But, lesson learned - when implementing a lisp, even if it is just a transpiler, don’t try to cut corners, and make a proper reader.

Next post, for sure, will be about the compiler!

Comment via email

Clojure on Fennel part three: parsing

Mon, 27 Apr 2026 19:48:00 +0300

The two previous posts were not related to the compiler itself, but were kicked off by the start of the compiler development. I’d say this project was the reason that I made proper immutable data structures for Fennel and Lua. But now we’re finally going to talk about the compiler itself! And we’ll start with the parsing stage - transforming Clojure code into something that can be operated on by the compiler.

Manual single-pass parser and compiler

When I started this project, as I usually do, I underestimated the complexity at hand. My first attempt was a single-pass compiler that accepted a stream of characters of Clojure code and re-compiled it on the fly.

My idea was to write a simple recursive-descent parser that, when accepting a character would decide how to translate it to Fennel, assuming my fennel-cljlib library handled most of the semantics. So, for example, when encountering [ it would translate it to (cljlib.vector and thus [1 2 3] would descend to compile-vector and expand to (cljlib.vector 1 2 3).

The idea is simple, and at first I thought that it would suffice, but then I remembered that there are a lot of other things in Clojure that I have to consider. For example #_ - the ignore form syntax, and ^foo - the metadata syntax. This complicated things a lot, so I added initial support for them and moved on - because I decided that this will be a bootstrap compiler for a proper Clojure parser.

Clojure parsers

After some time I had a compiler that could compile most of the arbitrary code I threw at it. Of course, it was not exactly to the Clojure spec, but it worked and I was going to replace it anyway. So I started looking at available Clojure parsers written in Clojure.

Edamame

I started with Edamame. The choice was based on the fact that the Squint project uses it and my project is similar to Squint, or so I thought. This parser is also used in SCI - a Clojure interpreter, so it seemed like a good fit.

The first thing I did was make sure that Edamame at least compiles with my bootstrap compiler. The fact that it compiles doesn’t mean that it works - it just translated everything into Fennel, with bits of cljlib here and there. So the next step was to make it work.

But then I had a realization that I don’t need to do that at all - I could just parse Edamame with Edamame, and write a compiler against its output! Meaning, I could use Edamame’s parser directly until my compiler is complete enough to compile Edamame. And since my idea was to replace bootstrap compiler in the end anyway, I decided that it doesn’t make much sense to make it work enough to fully compile Edamame into a runnable Fennel code. So I used Edamame to parse Edamame, and got back…

… the same Edamame source.

Imagine my face when I realized that Edamame doesn’t produce an abstract syntax tree with all information, but instead it just produces Clojure data structures, such as lists, vectors, and other stuff. And all of the information about the source code sits inside metadata of these data structures, not in the output. Yes, it does transform some Clojure forms into lists, like @foo is transformed into (deref foo), but that’s still nothing to me, because Fennel has no list data type and its support for metadata is nowhere near compatibility with Clojure.

I get it, I should have read the readme with more attention, and don’t make assumptions based on wild guesses, so it’s fine. Still, I wasn’t dropping the idea to use an existing Clojure parser, and the bootstrap compiler combo. I just need a different parser.

Rewrite-clj

Next one I looked at was rewrite-clj.

This library does return a tree of objects. It’s still a list, but at least it contains nodes that can be translated into Fennel objects. However, it still had a few problems.

First, its source code is much bigger than that of Edamame. Its API is more complicated, and it expects the use of Zippers, which I don’t yet have in cljlib. I didn’t want such a big parser for my compiler, given that it’s job is quite simple - produce an AST that I will always walk in full.

Second, it’s still producing some Clojure data structures that I can’t cleanly map to Fennel, so it would mean that the compiler itself would depend on cljlib. And I didn’t want that.

Luckily for me, their user guide mentioned another parser.

Parcera

Finally, parcera.

Looking at the readme, I finally saw what I wanted to see:

(ns example.core
  (:require [parcera.core :as parcera]))

;;parse clojure code from a string
(parcera/ast (str '(ns parcera.core
                     (:require [clojure.data :as data]
                               [clojure.string :as str]))))

;; => returns a data structure with the result from the parser
(:code
 (:list
  (:symbol "ns")
  (:whitespace " ")
  (:symbol "parcera.core")
  (:whitespace " ")
  (:list
   (:keyword ":require")
   (:whitespace " ")
   (:vector (:symbol "clojure.data")
            (:whitespace " ")
            (:keyword ":as")
            (:whitespace " ")
            (:symbol "data"))
   (:whitespace " ")
   (:vector (:symbol "clojure.string")
            (:whitespace " ")
            (:keyword ":as")
            (:whitespace " ")
            (:symbol "str")))))

Tagged tree. Yes, it’s still a list, but at least this can be easily mapped to Fennel sequential tables. So I opened the code and…

…there’s nothing there! Turns out, this project uses an ANTLR grammar for parsing Clojure.

At this point I was quite frustrated with the fact that there’s no single Clojure parser that could provide something similar to what parcera provides, but written in pure Clojure. Or at least in Java, which I could hand-translate as a last resort. Yes, there’s a Clojure parser inside the Clojure project itself, written in Java, but my goal was to recompile an existing parser written in Clojure.

Grammar-based parser

After a bit of consideration, I decided that a grammar-based approach is not that bad of an idea after all. I couldn’t use the ANTLR grammar directly, but at least I had it, and it seemed complete enough.

In Lua there’s a great library called LPeg that can generate parsers at runtime from a PEG grammar, and I wanted to try it out for quite some time. But to do that I needed a PEG grammar for Clojure. ANTLR and PEG are quite different, but it’s not impossible to translate one grammar into another.

So that’s what I did:

# Clojure PEG Grammar
#
# Adapted from https://github.com/carocad/parcera (ANTLR4 grammar v0.11.6)
#
# Key differences:
#
# - Unlike ANTLR, PEG choices are ordered (/).
#   More specific alternatives need to come first.
# - No separate lexer phase.
#   All rules operate on characters directly.
# - ANTLR fragment rules are prefixed with '_'.
# - ANTLR's SENTINEL (catch-all for invalid tokens) has no direct PEG equivalent.
#   The parser will silently fail on invalid input.

# Parser rules

code        <- input* !.

input       <- ignore / form

ignore      <- whitespace / comment / discard

form        <- literal / collection / reader_macro

# sets and namespaced maps are under dispatch (they start with #)
collection  <- list / vector / map

list        <- '(' input* ')'

vector      <- '[' input* ']'

map         <- '{' input* '}'

# macro_keyword before keyword ('::' is longer than ':')
literal     <- macro_keyword / keyword / string / number / character / symbol

keyword       <- KEYWORD
macro_keyword <- MACRO_KEYWORD
string        <- STRING

# ordering: most-specific prefix first; LONG last (least specific)
number        <- HEXADECIMAL / OCTAL / RADIX / DOUBLE / RATIO / LONG

# NAMED_CHAR first (longest); UNICODE before UNICODE_CHAR (more specific)
character     <- NAMED_CHAR / OCTAL_CHAR / UNICODE / UNICODE_CHAR

symbol        <- SYMBOL

# unquote_splicing before unquote ('~@' vs '~')
reader_macro <- unquote_splicing
              / unquote
              / metadata
              / backtick
              / quote
              / dispatch
              / deref

metadata <- ((metadata_entry / deprecated_metadata_entry) ignore*)+
            ( symbol
            / collection
            / set
            / namespaced_map
            / tag
            / fn
            / unquote_splicing
            / unquote
            / conditional_splicing
            / conditional
            / deref
            / quote
            / backtick
            / var_quote
            )

metadata_entry            <- '^' ignore* (map / symbol / string / keyword / macro_keyword / conditional)

# #^ is deprecated syntax for metadata
deprecated_metadata_entry <- '#^' ignore* (map / symbol / string / keyword / macro_keyword / conditional)

backtick          <- '`' ignore* form

quote             <- "'" ignore* form

# negative lookahead prevents '~' from matching when '~@' follows
unquote           <- '~' !'@' ignore* form

unquote_splicing  <- '~@' ignore* form

deref             <- '@' ignore* form

# conditional_splicing before conditional ('#?@' vs '#?')
# tag last (most general: '#' + symbol)
dispatch <- conditional_splicing
          / conditional
          / set
          / namespaced_map
          / fn
          / regex
          / var_quote
          / symbolic
          / eval
          / tag

# no whitespace allowed between '#' and the delimiter
fn                    <- '#' list

regex                 <- '#' STRING

set                   <- '#{' input* '}'

namespaced_map        <- '#' (macro_keyword / keyword / auto_resolve) ignore* map

auto_resolve          <- '::'

var_quote             <- "#'" ignore* form

discard               <- '#_' ignore* form

tag                   <- '#' symbol ignore* form

conditional           <- '#?' whitespace* list

conditional_splicing  <- '#?@' whitespace* list

# ##Inf, ##-Inf, ##NaN  (allows arbitrary symbolic values)
symbolic              <- '##' ignore* SYMBOL

eval                  <- '#=' ignore* (symbol / list / conditional)

whitespace <- WHITESPACE

comment    <- COMMENT


# Lexical rules (terminals)

HEXADECIMAL <- _SIGN? '0' [xX] [0-9A-Fa-f]+ _BIG_INT?

OCTAL       <- _SIGN? '0' [0-7]+ _BIG_INT?

# base 2-36, then r/R, then digits in that base
RADIX       <- _SIGN? ([2-9] / [12] [0-9] / '3' [0-6]) [rR] [0-9a-zA-Z]+

RATIO       <- _SIGN? _DIGIT+ '/' _DIGIT+

LONG        <- _SIGN? _DECIMAL _BIG_INT?

# FRACTION EXPONENT before FRACTION alone (longer match first)
DOUBLE      <- _SIGN? _DECIMAL+ ((_FRACTION _EXPONENT / _FRACTION / _EXPONENT) 'M'? / 'M')

_BIG_INT  <- 'N'
_FRACTION <- '.' _DIGIT*
_EXPONENT <- [eE] _SIGN? _DIGIT+
_DECIMAL  <- '0' / [1-9] _DIGIT*


STRING <- '"' ([^"\\] / '\\' .)* '"'


# \p{White_Space} approximation
WHITESPACE <- [ \t\n\r\f,]+

COMMENT <- (';' / '#!') [^\r\n]*


# Named characters must not be followed by name-chars (word boundary)
NAMED_CHAR   <- '\\' ('newline' / 'return' / 'space' / 'tab' / 'formfeed' / 'backspace')
                !(_ALLOWED_NAME_CHARACTER / _DIGIT)

UNICODE      <- '\\' 'u' [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]
                !(_ALLOWED_NAME_CHARACTER / _DIGIT)

# octal char: 0-377; longest alternative first
OCTAL_CHAR   <- '\\' 'o' ([0-3] [0-7] [0-7] / [0-7] [0-7] / [0-7])
                !(_ALLOWED_NAME_CHARACTER / _DIGIT)

# any single non-combining-mark character after backslash
UNICODE_CHAR <- '\\' ![\u0300-\u036F\u1DC0-\u1DFF\u20D0-\u20FF] .
                !(_ALLOWED_NAME_CHARACTER / _DIGIT)


# ::/ is NOT a valid macro keyword
MACRO_KEYWORD <- '::' (_SIMPLE_KEYWORD '/')? _SIMPLE_KEYWORD

KEYWORD       <- ':' (_SIMPLE_KEYWORD '/')? (_SIMPLE_KEYWORD / '/')

_SIMPLE_KEYWORD <- _KEYWORD_HEAD _KEYWORD_BODY+
                 / _KEYWORD_HEAD

_KEYWORD_BODY <- _KEYWORD_HEAD / ':'

_KEYWORD_HEAD <- _ALLOWED_NAME_CHARACTER / _DIGIT / [#'] / _SIGN


SYMBOL <- (_SIMPLE_SYMBOL '/')? (_SIMPLE_SYMBOL / '/')

# longer alternatives first for correct PEG matching
_SIMPLE_SYMBOL <- _SIGN _SYMBOL_HEAD _SYMBOL_BODY*
                / _ALLOWED_NAME_CHARACTER (_SYMBOL_HEAD / _DIGIT / ':') _SYMBOL_BODY*
                / _ALLOWED_NAME_CHARACTER
                / _SIGN

_SYMBOL_BODY <- _SYMBOL_HEAD / _DIGIT / ':'

_SYMBOL_HEAD <- _ALLOWED_NAME_CHARACTER / [#'] / _SIGN


# Characters allowed in names: everything EXCEPT reserved characters.
# Equivalent to ANTLR ~[\p{White_Space},()[\]{}"@~^;`\\:#'/0-9+-]
# Note: \p{White_Space} approximated with common whitespace chars here.
_ALLOWED_NAME_CHARACTER <- ![ \t\n\r\f,()\[\]{}"@~^;`\\:#'/0-9+\-] .

_SIGN  <- [+\-]

_DIGIT <- [0-9]

But of course, you can’t just load a PEG grammar into LPeg. Instead, you need to write the grammar programmatically, using LPeg’s API. I’ll spare you the details, but here’s a full source code of the parser.

After conversion work, I finally had a parser that could produce an AST:

>> (local {: parse} (require :impl.parser))
nil
>> (parse "(ns parcera.core
             (:require [clojure.data :as data]
                       [clojure.string :as str]))")
[:code
 [:list
  [:symbol "ns"]
  [:whitespace " "]
  [:symbol "parcera.core"]
  [:whitespace "\n             "]
  [:list
   [:keyword ":require"]
   [:whitespace " "]
   [:vector
    [:symbol "clojure.data"]
    [:whitespace " "]
    [:keyword ":as"]
    [:whitespace " "]
    [:symbol "data"]]
   [:whitespace "\n                       "]
   [:vector
    [:symbol "clojure.string"]
    [:whitespace " "]
    [:keyword ":as"]
    [:whitespace " "]
    [:symbol "str"]]]]]

Or not:

>> (parse "(ns parcera.core
             (:require [clojure.data :as data]
                       [clojure.string :as str])") ;; <- missing ')'
nil

Turns out, LPeg doesn’t have any way to report errors if the input string doesn’t match. It simply returns nil.

Luckily for me, there’s an additional library, called lpeglabel that can help with that. It implements an extension to the LPeg library that allows defining tagged failures:

>> (parse "(ns parcera.core
             (:require [clojure.data :as data]
                       [clojure.string :as str])")
unknown:3:49: Parse error: EOF while reading: Expected an ')'

Now I have a working Clojure parser with a usable output.

The parser

I mentioned before that Clojure metadata and special characters did introduce problems to my original parser. Well, with this grammar, this is no longer an issue:

>> (each [_ s (ipairs ["@foo" "^Foo bar" "^{:baz true} qux" "#_1 2"])]
     (pp (parse s)))
[:code [:deref [:symbol "foo"]]]
[:code [:metadata
        [:metadata-entry [:symbol "Foo"]]
        [:whitespace " "]
        [:symbol "bar"]]]
[:code [:metadata
        [:metadata-entry
         [:map [:keyword ":baz"] [:whitespace " "] [:symbol "true"]]]
        [:whitespace " "]
        [:symbol "qux"]]]
[:code [:discard [:number "1"]] [:whitespace " "] [:number "2"]]

Yes, the parser doesn’t transform @foo into (deref foo), but it’s OK, as I can handle it in the compiler easily now. Metadata is fully supported by the parser and is stripped in the compiler. And discard (#_) actually works in all of its supported ways, i.e. #_#_ 1 2 works fine.

Each parser result starts with :code, an entry point for the compiler. And the compiler itself is just a simple recursive descent over all of the nodes in a loop. So, that’s good, right?

Well, yes and no. There are some cons to this approach.

First, and foremost - this parser doesn’t support streaming, meaning I can’t stream chunks of code and compile them. Well, it’s not impossible, but currently it works on strings only. It is possible to write a second parser that would split text into top-level expressions, but for now it’s an optimization waiting to happen. Right now, this means that files will be parsed in full - having a greater footprint in memory.

Second, and arguably more important - this is a C library dependency. Both LPeg and lpeglabel are C libraries, meaning that the compiler now works only on Lua runtimes that can load object files. And the choice of systems where this compiler can run is now limited to those that can compile these libraries. Not that big of a deal, if you ask me, since C runs almost everywhere, but it’ll make the compiler difficult to ship as a no-dependency binary. It’s possible to compile a Fennel script into a binary and include all of its dependencies, but it would mean that I’ll have to produce binaries for many different operating systems and architectures. Or, alternatively, require people to install LPeg and lpeglabel through luarocks.

For now, I’m willing to accept this. There’s a port of LPeg in pure Lua, but it’s quite slow and doesn’t have support for lpeglabel features. There’s an open request for that, but it’s been sitting open for quite some time.

Other than that, the parsing is complete and we can look at the compiler part of the ClojureFnl project. But that’s gonna be in the next post.

Comment via email

Clojure on Fennel part two: immutable.fnl optimizations

Wed, 15 Apr 2026 05:11:00 +0300

So the next post will hopefully be about the compiler itself.

Unless I get distracted again.

Sike!

While I did some work on the compiler, I’m not feeling ready to talk about it yet. I want the post about it to be, well, more in-depth, so for now, let’s go back to immutable data structures. I got some comments on their performance, which can be summarized to “bad”.

And I agree! The performance is not great, especially when compared to plain Lua tables. However, my benchmarking process was a bit flawed, and I’ve improved it since.

This post won’t be long, but I wanted to make it anyway, to illustrate what I’m dealing with. Again, this is not the final version, I’m still working on possible optimizations as I write this, but some things have improved already, though not by much.

There were some low-hanging fruits that I could tackle:

Vector had a local fragment function with pure arithmetic. I already have the same fragment function in my bitops module that uses native rshift+band. The modulo operation is expensive.
A similar case of using bit operations instead of modulo was in HashMap’s index function.
Increased the popcount table to 256 entries instead of 16. Halves the loop iterations for typical bitmaps.
Probably premature, but replaced most instances of faccumulate, fcollect, etc. with plain for loops. These loops are quite hot, and Fennel compiles them to have additional assignment at each iteration. Should be free, but actually had a small impact.
More pre-allocations when creating Lua tables. Using [nil nil nil nil] tells the Lua compiler to pre-allocate four slots when creating the table, meaning it won’t resize once we insert elements. Tables in Lua resize by a factor of 2 and copy everything on resize, which is not great for performance.
Vector iterator is now stateful. And also fast.
Internal fields for data structures used long, elaborate keys, which meant string comparison in certain cases of table lookup would be unnecessarily long. Again, changing them to shorter keys should not matter that much, but it did have some impact.
HashMap branching factor was increased to 32. After some tests, it proved to be slightly better, despite what I said in my previous post. More on that later.
Hashes are now cached in a weak table. Helps avoid re-hashing the same values at the expense of memory. Not sure how well it will scale.
Optimized array copying for sizes less than 4 by doing direct table construction. Avoids loop overhead.

With these changes in place (and some more I omitted), here are some of the benchmarks:

These images show per-commit changes to the performance. Really helped me track what was worth keeping, and what wasn’t.

Here’s the LuaJIT version of the benchmark:

Here’s another graph:

This is me testing how hash map branching factor affects operations. As can be seen, switching from 16 to 32 makes inserts slower on PUC Lua 5.5 but makes indexing faster. On LuaJIT both operations are faster.

Looking at the charts, the 7989c32 commit with that change (the second orange bar) — insertions are slower than the previous commit, and indexing is faster than the previous commit. Unfortunately, LuaJIT charts are noisy and I can’t confirm the speedup there. Something to do with instabilities in the LuaJIT tracer, I guess.

Note I know, the results for some operations are worse than before all optimizations that took place. This is still a work in progress. Just shows how important relative measuring is.

The results are a mixed bag. Most operations that improved did so by only 1 or 2 milliseconds overall, meaning the per-operation cost is still quite high, compared to Lua tables. Yes, some became faster, so I’m keeping these changes, but I can also see that for some operations, they actually got slower than they were before any optimization work. The reason for that may be that I fixed a few bugs along the way. Anyway, I tried to prioritize transient collection variants, as they benefit most from optimizations given how they’re used, and LuaJIT performance where possible.

Some bars are not meaningful for certain operations. E.g., I had changes that had nothing to do with Persistent Vector, but affected it anyway, so this benchmark is still a bit flawed. At least those differences are within the deviation range.

Anyhow, just wanted to show that optimization is hard, and not all changes yield the results you expect. In the case of this library, I don’t think there are more low-hanging fruits to optimize, or at least I don’t see any. For example, there are a lot of table allocations that are hard to get rid of in the current implementation. A deeper analysis is needed, probably by people with more Lua knowledge.

I’m still satisfied with some improvements I got, but we’ll see if more are on the way.

Update

After a lot more benchmarks, flamegraphs, and throwing stuff at the wall, I think I can’t do much more. I decided to update this post instead of writing another one (and really should have waited before publishing the original) because at that point, I thought there was nothing more to do. Now I think there really isn’t. So the results are in:

Figure 1: PUC Lua 5.5 (green - before, orange - after)

Figure 2: LuaJIT (green - before, orange - after)

As can be seen, the changes are more dramatic on Lua than on LuaJIT. Mostly because it’s hard to optimize for LuaJIT without sacrificing performance on Lua, so I re-prioritized Lua, since LuaJIT is already much faster.

The final benchmark tables are below.

PUC Lua 5.5

PersistentVector

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
build 50000	0.18 ms	18.88 ms	102x	0.378 us	119x	0.440 us
lookup random	0.48 ms	11.44 ms	24x	0.229 us	28x	0.266 us
assoc random	0.31 ms	42.72 ms	139x	0.854 us	222x	1.4 us
pop all	0.14 ms	14.56 ms	106x	0.291 us	150x	0.415 us
iterate	0.60 ms	2.74 ms	5x	0.055 us	17x	0.196 us

TransientVector

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
build 50000	0.19 ms	6.55 ms	35x	0.131 us	41x	0.154 us
assoc random	0.31 ms	17.64 ms	57x	0.353 us	67x	0.408 us
pop all	0.14 ms	5.74 ms	42x	0.115 us	51x	0.142 us

PersistentHashMap

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
insert 50000	2.01 ms	81.44 ms	41x	1.6 us	82x	3.2 us
lookup random	0.92 ms	34.03 ms	37x	0.681 us	81x	1.7 us
contains random	1.82 ms	33.08 ms	18x	0.662 us	49x	1.7 us
dissoc all	0.90 ms	76.97 ms	86x	1.5 us	187x	3.3 us
dissoc 10%	0.18 ms	9.81 ms	56x	2.0 us	112x	3.7 us
iterate	1.91 ms	7.22 ms	4x	0.144 us	4x	0.146 us
lookup structural keys	0.14 ms	2.86 ms	21x	0.572 us	146x	4.1 us
insert collision-heavy	0.13 ms	6.50 ms	51x	1.3 us	134x	3.5 us
mixed 80/20	1.92 ms	53.67 ms	28x	1.1 us	62x	2.3 us

TransientHashMap

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
insert 50000	2.02 ms	38.91 ms	19x	0.778 us	44x	1.7 us
lookup random	1.14 ms	35.56 ms	31x	0.711 us	86x	1.7 us
dissoc all	0.89 ms	44.52 ms	50x	0.890 us	112x	2.0 us
mixed 80/20	4.39 ms	44.87 ms	10x	0.897 us	22x	1.9 us

LuaJIT

PersistentVector

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
build 50000	0.11 ms	8.12 ms	71x	0.162 us	84x	0.177 us
lookup random	0.06 ms	0.67 ms	11x	0.013 us	11x	0.013 us
assoc random	0.04 ms	22.91 ms	559x	0.458 us	731x	0.585 us
pop all	0.02 ms	10.77 ms	718x	0.215 us	657x	0.210 us
iterate	0.02 ms	0.23 ms	12x	0.005 us	26x	0.011 us

TransientVector

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
build 50000	0.05 ms	0.79 ms	16x	0.016 us	12x	0.012 us
assoc random	0.04 ms	2.22 ms	55x	0.044 us	28x	0.022 us
pop all	0.02 ms	0.35 ms	23x	0.007 us	50x	0.016 us

PersistentHashMap

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
insert 50000	0.79 ms	32.32 ms	41x	0.646 us	50x	0.989 us
lookup random	0.34 ms	5.78 ms	17x	0.116 us	43x	0.248 us
contains random	0.30 ms	5.91 ms	20x	0.118 us	40x	0.257 us
dissoc all	0.23 ms	27.02 ms	119x	0.540 us	191x	0.894 us
dissoc 10%	0.05 ms	5.89 ms	118x	1.2 us	121x	1.4 us
iterate	0.07 ms	1.07 ms	16x	0.021 us	30x	0.040 us
lookup structural keys	0.02 ms	2.41 ms	121x	0.483 us	119x	0.498 us
insert collision-heavy	0.28 ms	5.20 ms	19x	1.0 us	36x	1.2 us
mixed 80/20	0.50 ms	46.59 ms	94x	0.932 us	40x	0.513 us

TransientHashMap

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
insert 50000	0.91 ms	30.92 ms	34x	0.618 us	11x	0.242 us
lookup random	0.32 ms	34.90 ms	110x	0.698 us	44x	0.246 us
dissoc all	0.23 ms	36.35 ms	162x	0.727 us	57x	0.271 us
mixed 80/20	1.29 ms	40.71 ms	32x	0.814 us	11x	0.309 us

Conclusion

Yes, the benchmark shows that some operations slowed down, some sped up. Ideally, I’d like to have no slowdown, but that’s that. Still, I wouldn’t trust my LuaJIT benchmark, though - graphs show that for some weird reason unrelated changes affect performance in the modules not touched by those changes at all. So these changes are now merged into the main branch.

Main bottleneck now is table allocation and array copying - things I already spent optimizing a lot of time with mixed results. So I think I can’t do better than that. At least without major algorithmic restructuring.

With that, the next post should be about the compiler.

Comment via email

Clojure on Fennel part one: Persistent Data Structures

Tue, 07 Apr 2026 02:47:00 +0300

Somewhere in 2019 I started a project that aimed to bring some of Clojure features to Lua runtime - fennel-cljlib. It was a library for Fennel that implemented a basic subset of clojure.core namespace functions and macros. My goal was simple - I enjoy working with Clojure, but I don’t use it for hobby projects, so I wanted Fennel to feel more Clojure-like, besides what it already provides for that.

This library grew over the years, I implemented lazy sequences, added immutability, made a testing library, inspired by clojure.test and kaocha, and even made a port of clojure.core.async. It was a passion project, I almost never used it to write actual software. One notable exception is fenneldoc - a tool for documentation generation for Fennel libraries. And I haven’t seen anyone else use it for a serious project.

The reason for that is simple - it was an experiment. Corners were cut, and Fennel, being a Clojure-inspired lisp is not associated with functional programming the same way Clojure is. As a matter of fact, I wouldn’t recommend using this library for anything serious… yet.

Recently, however, I started a new project: ClojureFnl. This is a Clojure-to-Fennel compiler that uses fennel-cljlib as a foundation. It’s still in early days of development, but I’ve been working on it for a few months in private until I found a suitable way to make things work in March. As of this moment, it is capable of compiling most of .cljc files I threw at it, but running the compiled code is a different matter. I mean, it works to some degree, but the support for standard library is far from done.

;; Welcome to ClojureFnl REPL
;; ClojureFnl v0.0.1
;; Fennel 1.6.1 on PUC Lua 5.5
user=> (defn prime? [n]
         (not (some zero? (map #(rem n %) (range 2 n)))))
#<function: 0x89ba7c550>
user=> (for [x (range 3 33 2)
             :when (prime? x)]
         x)
(3 5 7 11 13 17 19 23 29 31)
user=>

However, there was a problem.

My initial implementation of immutable data structures in the itable library had a serious flaw. The whole library was a simple hack based on the copy-on-write approach and a bunch of Lua metatables to enforce immutability. As a result, all operations were extremely slow. It was fine as an experiment, but if I wanted to go further with ClojureFnl, I had to replace it. The same problem plagued immutableredblacktree.lua, an implementation of a copy-on-write red-black tree I made for sorted maps. It did a full copy of the tree each time it was modified.

For associative tables it wasn’t that big of a deal - usually maps contain a small amount of keys, and itable only copied levels that needed to be changed. So, if you had a map with, say, ten keys, and each of those keys contained another map with ten keys, adding, removing or updating a key in the outer map meant only copying these ten keys - not the whole nested map. I could do that reliably, because inner maps were immutable too.

But for arrays the story is usually quite different. Arrays often store a lot of indices, and rarely are nested (or at least not as often as maps). And copying arrays on every change quickly becomes expensive. I’ve mitigated some of the performance problems by implementing my version of transients, however the beauty of Clojure’s data structures is that they’re quite fast even without this optimization.

Proper persistent data structures

Clojure uses Persistent HAMT as a base for its hash maps and sets, and a bit-partitioned trie for vectors. For sorted maps and sets, Clojure uses an immutable red-black tree implementation, but as far as I know it’s not doing a full copy of the tree, and it also has structural sharing properties.

I started looking into existing implementations of HAMT for Lua:

hamt.lua (based on mattbierner/hamt)
- seemed incomplete
ltrie
- no transients
- no hashset
- no ordered map (expectable, different algorithm)
- no compound vector/hash
Michael-Keith Bernard’s gist
- no custom hashing

I could use one of those, notably ltrie seemed the most appropriate one, but given that I’m working on a fennel library that I want later to embed into my Clojure compiler I needed a library implemented in Fennel.

So I made my own library: immutable.fnl. This library features HAMT-hash maps, hash-sets, and vectors, as well as a better implementation of a persistent red-black tree, and lazy linked lists.

Persistent Hash Map

I started the implementation with a Persistent HAMT with native Lua hashing. The data structure itself is a Hash Array Mapped Trie (HAMT) with 16-factor branching. Thus all operations are O(Log16 N), which is effectively O(1) for a practical amount of keys.

As far as I know, Clojure uses branching factor of 32, but for a Lua runtime this would mean that the popcount would be more expensive, and despite a shallower tree, each mutation would need to copy a larger sparse array. With branching factor of 16 a map with 50K entries is ~4 levels deep, which would be ~3 with 32 branching factor. So my logic was that it’ll be a compromise, especially since Lua is not JVM when it comes to performance.

Of course, it’s not as fast as a pure Lua table, which is to be expected. Lua tables are implemented in C, use efficient hashing, and dynamically re-allocated based on key count. So for my implementation most operations are a lot slower, but the total time for an operation is still usable.

Here are some benchmarks:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Regular operations are notably slower, when compared to Lua:

Operation	Lua table	Persistent HashMap	Ratio	per op
insert 50000 random keys	2.05 ms	164.80 ms	80.3x slower	3.3 us
lookup 50000 random keys	0.83 ms	92.51 ms	110.8x slower	1.9 us
delete all	0.78 ms	170.78 ms	219.8x slower	3.4 us
delete 10%	0.14 ms	19.50 ms	136.4x slower	3.9 us
iterate 50000 entries	1.74 ms	6.64 ms	3.8x slower	0.133 us

For transients the situation is a bit better, but not by much:

Operation	Lua table	Transient HashMap	Ratio	per op
insert 50000 random keys	2.05 ms	89.17 ms	43.5x slower	1.8 us
delete all	0.76 ms	104.31 ms	138.0x slower	2.1 us
delete 10%	0.16 ms	12.71 ms	82.0x slower	2.5 us

On LuaJIT numbers may seem worse, but per-operation cost is much lower, it’s just that native table operations are so much faster:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	Persistent HashMap	Ratio	per op
insert 50000 random keys	0.86 ms	49.05 ms	56.8x slower	0.981 us
lookup 50000 random keys	0.27 ms	14.21 ms	53.4x slower	0.284 us
delete all	0.13 ms	48.63 ms	374.1x slower	0.973 us
delete 10%	0.05 ms	6.49 ms	138.1x slower	1.3 us
iterate 50000 entries	0.07 ms	1.80 ms	27.7x slower	0.036 us

Operation	Lua table	Transient HashMap	Ratio	per op
insert 50000 random keys	0.76 ms	22.43 ms	29.6x slower	0.449 us
delete all	0.15 ms	34.16 ms	232.4x slower	0.683 us
delete 10%	0.04 ms	5.02 ms	132.1x slower	1.0 us

With a branching factor of 32 the situation gets worse on PUC Lua, but is slightly better on LuaJIT. So there’s still space for fine-tuning.

For hashing strings and objects I decided to use djb2 algorithm. I am almost as old as this hash function, so seemed like a good fit. JK. The main reason to use it was that it can be implemented even if we don’t have any bit-wise operators, and Lua doesn’t have them in all of the versions. It only uses +, *, and % arithmetic operators, so can be done on any Lua version. It’s prone to collisions, and I try to mitigate that by randomizing it when the library is loaded.

Still, collisions do happen, but HAMT core ensures that they will still resolve correctly by implementing a deep equality function for most objects.

However, when first working on this, I noticed this:

>> (local hash-map (require :io.gitlab.andreyorst.immutable.PersistentHashMap))
nil
>> (local {: hash} (require :io.gitlab.andreyorst.immutable.impl.hash))
nil
>> (hash (hash-map :foo 1 :bar 2))
161272824
>> (hash {:foo 1 :bar 2})
161272824
>> (hash-map (hash-map :foo 1 :bar 2) 1 {:foo 1 :bar 2} 2)
{{:foo 1 :bar 2} 2}

This is an interesting loophole. What object ended up in our hash map as a key - our persistent map or plain Lua table? Well, that depends on insertion order:

>> (each [_ k (pairs (hash-map (hash-map :foo 1 :bar 2) 1 {:foo 1 :bar 2} 2))]
     (print (getmetatable k)))
IPersistentHashMap: 0x824d9b570
nil
>> (each [_ k (pairs (hash-map {:foo 1 :bar 2} 2 (hash-map :foo 1 :bar 2) 1))]
     (print (getmetatable k)))
nil
nil

To reiterate, I’m creating a hash map, with a key set to another persistent hash map, and then insert a plain Lua table with the same content. The Lua table hashes to exactly the same hash, and goes into the same bucket, but there’s no collision, because objects are equal by value. But equality of mutable collections is very loosely defined - it may be equal right now, but the next time you look at it, it’s different. So a different hashing was needed for persistent collections, to avoid these kinds of collision. I ended up salting persistent collections with their prototype address in memory.

Other than that, the HAMT implementation is by the book, and the rest is the interface for interacting with maps.

Main operations:

new - construct a new map of key value pairs
assoc - associate a key with a value
dissoc - remove key from the map
conj - universal method for association, much like in Clojure
contains - check if key is in the map
count - map size, constant time
get - get a key value from a map
keys - get a lazy list of keys
vals - get a lazy list of values
transient - convert a map to a transient

Coercion/conversion:

from - create a map from another object
to-table - convert a map to a Lua table
iterator - get an iterator to use in Lua loops

Transient operations:

assoc! - mutable assoc
dissoc! - mutable dissoc
persistent - convert back to persistent variant, and mark transient as completed

This covers most of the needs in my fennel-cljlib library, as anything besides it I can implement myself, or just adapt existing implementations.

A Persistent Hash Set is also available as a thin wrapper around PersistentHashMap with a few method changes.

A note on PersistentArrayMap.

In Clojure there is a second kind of maps that are ordered, not sorted, called a Persistent Array Map. They are used by default when defining a map with eight keys or less, like {:foo 1 :bar 2}. The idea is simple - for such a small map, a linear search through all keys is faster than with a HAMT-based map.

However, in my testing on the Lua runtime, there’s no benefit in this kind of a data structure, apart from it being an ordered variant. Lookup is slower, because of a custom equality function, which does deep comparison.

Persistent Vector

Persistent Vectors came next, and while the trie structure is similar to hash maps, vectors use direct index-based navigation instead of hashing, with a branching factor of 32. Unlike maps, vector arrays in the HAMT are more densely packed, and therefore a higher branching factor is better for performance. So lookup, update, and pop are O(log32 N), append can be considered O(1) amortized.

Still, compared to plain Lua sequential tables the performance is not as good:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Operation	Lua table	Persistent Vector	Ratio	per op
insert 50000 elements	0.19 ms	21.07 ms	109.7x slower	0.421 us
lookup 50000 random indices	0.47 ms	14.05 ms	29.7x slower	0.281 us
update 50000 random indices	0.32 ms	70.04 ms	221.6x slower	1.4 us
pop all 50000 elements	0.25 ms	24.34 ms	96.2x slower	0.487 us
iterate 50000 elements	0.63 ms	10.16 ms	16.2x slower	0.203 us

Operation	Lua table	Transient Vector	Ratio	per op
insert 50000 elements	0.19 ms	7.81 ms	40.3x slower	0.156 us
update 50000 random indices	0.33 ms	20.76 ms	62.4x slower	0.415 us
pop all 50000 elements	0.25 ms	11.14 ms	44.4x slower	0.223 us

On LuaJIT:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	Persistent Vector	Ratio	per op
insert 50000 elements	0.10 ms	7.62 ms	74.0x slower	0.152 us
lookup 50000 random indices	0.06 ms	0.67 ms	11.8x slower	0.013 us
update 50000 random indices	0.04 ms	29.13 ms	710.4x slower	0.583 us
pop all 50000 elements	0.02 ms	8.62 ms	410.4x slower	0.172 us
iterate 50000 elements	0.02 ms	0.57 ms	28.7x slower	0.011 us

Operation	Lua table	Transient Vector	Ratio	per op
insert 50000 elements	0.05 ms	0.59 ms	11.6x slower	0.012 us
update 50000 random indices	0.04 ms	2.06 ms	51.6x slower	0.041 us
pop all 50000 elements	0.02 ms	0.84 ms	46.7x slower	0.017 us

I think this is an OK performance still. Vectors don’t use hashing, instead it is a direct index traversal via bit-shifting, so there’s no hashing, just index math.

Operations on vectors include:

new - constructor
conj - append to the tail
assoc - change a value at given index
count - element count (constant time)
get - get value at given index
pop - remove last
transient - convert to a transient
subvec - create a slice of the vector in constant time

Transient operations:

assoc! - mutable assoc
conj! - mutable conj
pop! - mutable pop
persistent - convert back to persistent and finalize

Interop:

from - creates a vector from any other collection
iterator - returns an iterator for use in Lua loops
to-table - converts to a sequential Lua table

One notable difference in both vector and hash-map is that it allows nil to be used as a value (and as a key, in case of the hash-map). Vectors don’t have the same problem that Lua sequential tables have, where length is not well-defined if the table has holes in it.

It’s a debate for another time, whether allowing nil as a value (and especially as a key) is a good decision to make, but Clojure already made it for me. So for this project I decided to support it.

Persistent Red-Black Tree

For sorted maps and sorted sets I chose Okasaki’s insertion and Germane & Might’s deletion algorithms. Most of the knowledge I got from this amazing blog post by Matt Might.

I believe the operations are O(Log N), as for any binary tree, but given that the deletion algorithm is tricky, I’m not exactly sure:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Operation	Lua table	PersistentTreeMap	Ratio	per op
insert 50000 random keys	2.10 ms	209.23 ms	99.8x slower	4.2 us
lookup 50000 random keys	0.88 ms	82.97 ms	94.2x slower	1.7 us
delete all	0.74 ms	173.76 ms	234.8x slower	3.5 us

On LuaJIT:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	PersistentTreeMap	Ratio	per op
insert 50000 random keys	0.72 ms	101.08 ms	140.4x slower	2.0 us
lookup 50000 random keys	0.25 ms	12.67 ms	49.9x slower	0.253 us
delete all	0.14 ms	56.14 ms	403.9x slower	1.1 us

The API for sorted maps and sets is the same as to their hash counterparts with a small difference - no transients. Clojure doesn’t do them, and I’m not doing them too.

That’s all for benchmarks. I know that there are many problems with this kind of benchmarking, so take it with a grain of salt. Still, the results are far, far better than what I had with itable.

But there are two more data structures to talk about.

Persistent List

As I mentioned, I made a lazy persistent list implementation a while ago but it had its problems and I couldn’t integrate that library with the current one well enough.

The main problem was that this library uses a single shared metatable per data structure, and the old implementation of lazy lists didn’t. This difference makes it hard to check whether the object is a table, hash-map, list, vector, set, etc. So I reimplemented them.

The reason for old implementation to use different metatables was because I decided to try the approach described in Reversing the technical interview post by Kyle Kingsbury (Aphyr). I know this post is more of a fun joke, but it actually makes sense to define linked lists like that in Lua.

See, tables are mutable, and you can’t do much about it. Closures, on the other hand are much harder to mutate - you can still do it via the debug module, but it’s hard, and it’s not always present. So storing head and tail in function closures was a deliberate choice.

However, it meant that I needed to somehow attach metadata to the function, to make it act like a data structure, and you can’t just use setmetatable on a function. Again, you can do debug.setmetatable but all function objects share the same metadata table. So, while you can do fancy things like this:

>> (fn comp [f g] (fn [...] (f (g ...))))
#<function: 0x7bdb320a0>
>> (debug.setmetatable (fn []) {:__add comp})
#<function: 0x7bd17f040>
>> ((+ string.reverse string.upper) "foo")
"OOF"

You can also notice, that our + overload applied to functions in the string module.

So instead, we use a table, and wrap it with a metatable that has a __call metamethod, essentially making our table act like a function. This, in turn means, that we have to create two tables per list node - one to give to the user, the other to set our __call and use it as a meta-table.

Convoluted, I know. It’s all in the past now - current implementation is a simple {:head 42 :tail {...}} table. Not sure what is worse.

But that meant that I had to rework how lazy lists worked, because previously it was just a metatable swap. Now list stores a “thunk”, that when called replaces itself in the node with the :head and :tail keys. Unless it’s an empty list, of course - in that case we swap the metatable to an empty list one.

So Lists have three metatables now:

IPersistentList
IPersistentList$Empty
IPersistentList$Lazy

Instead of god knows how many in the old implementation.

The list interface is also better now. Previously it was hardcoded how to construct a list from a data structure. Current implementation also hardcodes it, but also allows to build a list in a lazy way from an iterator.

This is better, because now a custom data structure that has weird iteration schema (like maps and sets in this library), we still can convert it to a list. A general case is just:

(PersistentList.from-iterator #(pairs data) (fn [_ v] v))

Meaning that we pass a function that will produce the iterator, and a function to capture values from that iterator. Reminds me of clojure transducers in some way.

Persistent Queue

And the final data structure - a persistent queue. Fast append at the end, and also fast remove from the front.

It’s done by holding two collections - a linked list at the front, and a persistent vector for the rear. So removing from the list is O(1), and appending to the vector is also pretty much O(1).

Interesting things start to happen when we exhaust the list part - we need to move vector’s contents into the list. It is done by calling PersistentList.from on the rear. And building a list out of a persistent vector is an O(1) operation as well! Well, because nothing happens, we simply create an iterator, and build the list in a lazy way. But since indexing the vector is essentially ~O(1), we can say that we still retain this property.

Or at least that’s how I reasoned about this - I’m not that good with time-complexity stuff.

ClojureFnl

That concludes part one about ClojureFnl.

I know that this post was not about ClojureFnl at all, but I had to fix my underlying implementation first. Now, that I have better data structures to build from, I can get back working on the compiler itself. So the next post will hopefully be about the compiler itself.

Unless I get distracted again.

Comment via email