Andrey Listopadov

Clojure on Fennel part four: Parsing (again)

Mon, 29 Jun 2026 00:29:00 +0300

Other than that, the parsing is complete, and we can look at the compiler part of the ClojureFnl project. But that’s gonna be in the next post.

Sike!

Wasn’t my intention to fake out like this again, but while working on the compiler, I had an important realization that led me to redesign the parser completely. And it’s a bit of a shame, because I already had a decent part of the compiler working, being able to run the REPL and do various cool things with it. But, better sooner than later, I guess.

I had a good amount of the post about the compiler already written too, but now I’ll have to discard all of that. So instead, I decided to talk about the new parser by explaining why the old one failed me. And to do so, I’ll give you a brief look on how the compiler currently works with parsed code.

Old parser and compiler

So, in the previous post, we’ve reached a point where the parser can read code into a tagged tree. I spent a fair amount of time discussing how I wanted this specifically, and now it feels like I’m backpedalling, but tagged tree is not the way forward. Here’s the idea.

Currently, we’re reading this expression (+ 1 2) into:

[:code
 [:list
  [:symbol "+"]
  [:whitespace " "]
  [:number "1"]
  [:whitespace " "]
  [:number "2"]]]

The compiler then walks this tree, taking the first value of each node to decide what to do. First, it sees :code - that’s just an entry point, containing multiple top-level expressions. Next, it sees :list and dispatches to a function dedicated to compiling lists.

This function checks the first element of the list, sees that it is a :symbol, and checks whether this symbol is somehow special. While + doesn’t look that special, it is to the compiler, because Clojure’s + and Fennel’s + are not the same thing. In Clojure, + is a function, in Fennel, it is a special.

So the compiler replaces + with clojure_core_ns.add, and then proceeds over the rest of the forms in the list, recursively calling itself over each. In the end, we get this, written into a string builder:

(clojure_core_ns.add 1 2)

This is a somewhat simplified explanation, because I’m omitting scope resolution, symbol shadowing, and other such things that the compiler currently tracks. For actual specials, like Clojure’s let*, the story is a bit different too. The compiler has a dedicated compile-special function for all Clojure specials provided by cljlib as Fennel macros. And that’s where things started to go haywire.

Look at this code:

(def ^:private foo 42)

It’s read like this:

[:list
 [:symbol "def"]
 [:whitespace " "]
 [:metadata
  [:metadata-entry [:keyword ":private"]]
  [:whitespace " "]
  [:symbol "foo"]]
 [:whitespace " "]
 [:number "42"]]

Here’s a problem - the parser reads metadata ^:private, and metadata in Clojure is attached to symbols, so the grammar I use reads the next value after the metadata and wraps it into a single node:

[:metadata
 [:metadata-entry [:keyword ":private"]]
 [:whitespace " "]
 [:symbol "foo"]]

The compiler, while compiling this list, sees def as the first symbol and enters compile-special.def. But def expects a symbol to bind the value to, while here we have a different node type, called :metadata. So my compiler had to account for all cases where metadata can appear - and it wasn’t easy to do.

I tried to side-step this by always expecting metadata, because it can appear almost anywhere, and discarding it, because Fennel’s concept of metadata is a bit different. In which I mostly succeeded. But this wasn’t the only problematic thing to support.

Here’s another example, now from Edamame:

(defn- read-token
  "Read in a single logical token from the reader"
  ^String [#?(:clj rdr :cljs ^not-native rdr :cljr rdr) _kind initch]
  (loop [sb #?(:clj (StringBuilder.)
               :cljs (StringBuffer.)
               :cljr (StringBuilder.))
         ch initch]
    (if (or (whitespace? ch)
            (macro-terminating? ch)
            (nil? ch))
      (do (when ch
            (r/unread rdr ch))
          (str sb))
      (recur #?(:clj (.append sb ch) :cljs (.append sb ch) :cljr (.Append sb (str ch))) (r/read-char rdr)))))

It reads into this tagged tree:

[:list
 [:symbol "defn-"]
 [:whitespace " "]
 [:symbol "read-token"]
 [:whitespace "\n  "]
 [:string "\"Read in a single logical token from the reader\""]
 [:whitespace "\n  "]
 [:metadata
  [:metadata-entry [:symbol "String"]]
  [:whitespace " "]
  [:vector
   [:conditional
    [:list
     [:keyword ":clj"]
     [:whitespace " "]
     [:symbol "rdr"]
     [:whitespace " "]
     [:keyword ":cljs"]
     [:whitespace " "]
     [:metadata
      [:metadata-entry [:symbol "not-native"]]
      [:whitespace " "]
      [:symbol "rdr"]]
     [:whitespace " "]
     [:keyword ":cljr"]
     [:whitespace " "]
     [:symbol "rdr"]]]
   [:whitespace " "]
   [:symbol "_kind"]
   [:whitespace " "]
   [:symbol "initch"]]]
 [:whitespace "\n  "]
 [:list
  [:symbol "loop"]
  [:whitespace " "]
  [:vector
   [:symbol "sb"]
   [:whitespace " "]
   [:conditional
    [:list
     [:keyword ":clj"]
     [:whitespace " "]
     [:list [:symbol "StringBuilder."]]
     [:whitespace "\n               "]
     [:keyword ":cljs"]
     [:whitespace " "]
     [:list [:symbol "StringBuffer."]]
     [:whitespace "\n               "]
     [:keyword ":cljr"]
     [:whitespace " "]
     [:list [:symbol "StringBuilder."]]]]
   [:whitespace "\n         "]
   [:symbol "ch"]
   [:whitespace " "]
   [:symbol "initch"]]
  [:whitespace "\n    "]
  [:list
   [:symbol "if"]
   [:whitespace " "]
   [:list
    [:symbol "or"]
    [:whitespace " "]
    [:list [:symbol "whitespace?"] [:whitespace " "] [:symbol "ch"]]
    [:whitespace "\n            "]
    [:list
     [:symbol "macro-terminating?"]
     [:whitespace " "]
     [:symbol "ch"]]
    [:whitespace "\n            "]
    [:list [:symbol "nil?"] [:whitespace " "] [:symbol "ch"]]]
   [:whitespace "\n      "]
   [:list
    [:symbol "do"]
    [:whitespace " "]
    [:list
     [:symbol "when"]
     [:whitespace " "]
     [:symbol "ch"]
     [:whitespace "\n            "]
     [:list
      [:symbol "r/unread"]
      [:whitespace " "]
      [:symbol "rdr"]
      [:whitespace " "]
      [:symbol "ch"]]]
    [:whitespace "\n          "]
    [:list [:symbol "str"] [:whitespace " "] [:symbol "sb"]]]
   [:whitespace "\n      "]
   [:list
    [:symbol "recur"]
    [:whitespace " "]
    [:conditional
     [:list
      [:keyword ":clj"]
      [:whitespace " "]
      [:list
       [:symbol ".append"]
       [:whitespace " "]
       [:symbol "sb"]
       [:whitespace " "]
       [:symbol "ch"]]
      [:whitespace " "]
      [:keyword ":cljs"]
      [:whitespace " "]
      [:list
       [:symbol ".append"]
       [:whitespace " "]
       [:symbol "sb"]
       [:whitespace " "]
       [:symbol "ch"]]
      [:whitespace " "]
      [:keyword ":cljr"]
      [:whitespace " "]
      [:list
       [:symbol ".Append"]
       [:whitespace " "]
       [:symbol "sb"]
       [:whitespace " "]
       [:list [:symbol "str"] [:whitespace " "] [:symbol "ch"]]]]]
    [:whitespace " "]
    [:list [:symbol "r/read-char"] [:whitespace " "] [:symbol "rdr"]]]]]]

Yes, it’s abysmal, but bear with me. I had to work with this, after all, thinking that this is a blessing.

The compiler sees defn- and enters the compile-special.defn- function. Defn expects a function name, then a vector for its arguments. Here, instead of the vector, we have metadata node again. This is fine, since I just mentioned above that I managed to sidestep this problem.

Since this is an arglist, we need to compile it in a special way, adding its symbols to the function scope, etc. However, instead of arguments, we see the conditional node:

[:vector
 [:conditional
  [:list
   [:keyword ":clj"] [:whitespace " "] [:symbol "rdr"] [:whitespace " "]
   [:keyword ":cljs"] [:whitespace " "] [:metadata [:metadata-entry [:symbol "not-native"]] [:whitespace " "] [:symbol "rdr"]] [:whitespace " "]
   [:keyword ":cljr"] [:whitespace " "] [:symbol "rdr"]]]
 [:whitespace " "]
 [:symbol "_kind"]
 [:whitespace " "]
 [:symbol "initch"]]

I didn’t think it was allowed in Clojure to use conditional reading inside forms like this. But apparently it’s OK, and my compiler failed to deal with it.

So now, any node can not only be a metadata node, but also a conditional node. This complicates things, but at this point I’m still thinking that I can persevere.

So I did. I added support for almost all specials provided by cljlib. The main thing that was left to do were macros. And then it hit me:

I can’t do macros like that!

Why? Why, of course, because macros don’t emit tagged trees that my compiler understands. They emit code!

Here’s a simple macro:

(defmacro unless [test & body]
  `(when (not ~test)
     ~@body))

We can parse it:

[:list
 [:symbol "defmacro"] [:whitespace " "] [:symbol "unless"] [:whitespace " "]
 [:vector
  [:symbol "test"] [:whitespace " "] [:symbol "&"] [:whitespace " "] [:symbol "body"]]
 [:whitespace " "]
 [:backtick
  [:list
   [:symbol "when"] [:whitespace " "]
   [:list [:symbol "not"] [:whitespace " "] [:unquote [:symbol "test"]]]
   [:whitespace " "] [:unquote-splicing [:symbol "body"]]]]]

We can probably even compile it to something that we could then call during the compiler step. However, what will (unless true (println 42)) thing return?

(when (not true) (println 42)), of course. It’s a Lisp macro, what else did you expect?

But what does the compiler expect?

[:list
 [:symbol "when"]
 [:whitespace " "]
 [:list [:symbol "not"] [:whitespace " "] [:symbol "true"]]
 [:whitespace " "]
 [:list [:symbol "println"] [:whitespace " "] [:number "42"]]]

Oh, it wants this.

I have to convert macro’s output into a string, parse it, and feed it into the compiler if I want macros to work at all. And that’s BAD. If only I realized this sooner!

This was the final nail in the coffin of the tagged tree approach for my project. I knew I had to rewrite the parser to emit actual data structures that I’ll be able to emit from macros as well, and compile them.

New parser reader

So I decided that I need a proper lisp reader that will produce data structures:

;; Welcome to Fennel Proto REPL 0.6.4-dev
;; Fennel version: 1.7.0-dev
;; Lua version: PUC Lua 5.5
;; Work directory: ~/Projects/fennel/ClojureFnl/
>> (local reader (require :impl.reader))
nil
>> (reader.read-string "(def ^:private foo 42)")
(def foo 42)
>> (local {: meta : second} (require :clojure.core))
nil
>> (meta (second (reader.read-string "(def ^:private foo 42)")))
{:private true}

As can be seen, the new reader module provides the read-string function that produces data structures. Notably, there are no longer any metadata nodes - metadata is assigned to the symbol foo in this case.

Same goes for the reader conditionals:

>> (reader.read-string "[0 #?(:clj 1 :cljfnl 2) #?@(:clj [2 3] :cljfnl [3 4]) 5]")
[0 2 3 4 5]

These are now correctly spliced at read time.

This also solved macro problems for the most part:

>> (reader.read-string "`(+ ~x ~@y ~z)")
(clojure.core/seq
 (clojure.core/concat
  (clojure.core/list (quote clojure.core/+))
  (clojure.core/list x)
  y
  (clojure.core/list z)))

And it matches what Clojure itself does:

Clojure 1.12.5
user=> (read-string "`(+ ~x ~@y ~z)")
(clojure.core/seq
 (clojure.core/concat
  (clojure.core/list (quote clojure.core/+))
  (clojure.core/list x)
  y
  (clojure.core/list z)))

Due to this change, the compiler doesn’t have to know about syntax quote (`) at all, and thus macros will correctly return these kinds of lists.

So, now we can read Clojure code into data structures. Previously, the parser returned [:code A B ...] kind of result, where A, B, and the rest are all top-level forms. New reader also does this, but returns them as a list:

>> (reader.read-string-all "1 \"str\" :keyword ::namespaced #:map{:key :val}")
(1 "str" :keyword :user/namespaced {:map/key :val})

However, there’s a problem.

Consider the ::namespaced keyword. Both my reader and Clojure read it as :user/namespaced, but this is because the default namespace in the REPL is user. For my parser, it’s mostly an arbitrary choice because it doesn’t know anything about runtime.

Clojure 1.12.5
user=> (read-string "::x")
:user/x
user=> (ns foo)
nil
foo=> (read-string "::x")
:foo/x

And when compiling a file, the file might have an ns declaration, or even multiple of them:

(ns foo)

(println ::x)

(ns bar)

(println ::x)

If we were to run this code, we would see :foo/x then :bar/x.

My parser currently reads all input, meaning it transforms all top-level forms from text to data. However, this also means that it doesn’t understand anything about namespaces.

Previously, this was fine because the compiler handled this - the tagged tree didn’t resolve anything. Currently, however, we need to construct namespaced keys, and we need to know the namespace. But it’s not possible unless we parse the input expression by expression, instead of all at once.

Hence, I had to update the grammar so it could parse expression by expression. This way, I could read source code form by form, and compile each form separately. And if the compiler encounters an ns declaration, it could update some state, so the reader would know how to read namespaced keywords, and other things that may include namespaces.

But we still need to pass this state to the reader. And my reader is based on a PEG grammar.

Luckily for me, the lpeg library I use has a Carg function that can pass additional arguments into the PEG parser. This way I can write my transformation function T:

(fn T [tag patt]
  (/ (* (Ct (* (Cc tag) patt)) (Carg 1))
     (fn [node state]
       (case node
         ;; ----8<----
         [:macro-keyword bo data]
         (read-macro-keyword data bo state)
         [:conditional data]
         (read-conditional data state)
         [:conditional-splicing data]
         (conditional-splicing data state)
         ;; ----8<----
         _ _))))

This state can be passed anew after reading each expression, or mutated in place - either way, the reader now knows how to access state. The reader itself is still stateless.

The reader now supports all Clojure syntax and produces data structures:

;; Clojure                                          ;; Fennel
1                                                   ;; 1
0.5                                                 ;; 0.5
1e10                                                ;; 10000000000.0
1/2                                                 ;; 1/2
1N                                                  ;; 1N
1.33333333333333333333333333333333M                 ;; 1.33333333333333333333333333333333M
0xFF                                                ;; 255
017                                                 ;; 15
16rFF                                               ;; 255
\c                                                  ;; "c"
"string"                                            ;; "string"
\u0041                                              ;; "A"
\o101                                               ;; "A"
:keyword                                            ;; :keyword
:namespaced/keyword                                 ;; :namespaced/keyword
::auto-keyword                                      ;; :user/auto-keyword
symbol                                              ;; symbol
namespaced/symbol                                   ;; namespaced/symbol
#'var                                               ;; (var var)
'quoted-symbol                                      ;; (quote quoted-symbol)
(list)                                              ;; (list)
[vector]                                            ;; [vector]
{hash map}                                          ;; {hash map}
#:namespaced{:map 42}                               ;; {:namespaced/map 42}
#::{:auto :namespaced}                              ;; {:user/auto :namespaced}
#{hash set}                                         ;; #{set hash}
#"regexp"                                           ;; "regexp"
@dereferencing                                      ;; (clojure.core/deref dereferencing)
#(+ %1 %2)                                          ;; (fn* [p__0__ p__1__] (+ p__0__ p__1__))
^:meta data                                         ;; data
#_discard                                           ;;
[#?(:cljfnl "reader") #?@(:cljfnl [:conditionals])] ;; ["reader" :conditionals]
nil                                                 ;; nil
true                                                ;; true
false                                               ;; false
##NaN                                               ;; .nan
##Inf                                               ;; .inf
#inst "2022"                                        ;; #inst "2022-01-01T00:00:00.000-00:00"
#uuid "c6e8050a-789b-4305-b518-8f0f7c31da7a"        ;; #uuid "c6e8050a-789b-4305-b518-8f0f7c31da7a"

Note about data structures: even though we’re working in Fennel, the produced maps, vectors and lists are custom data structures that are implemented in the cljlib library. These are implementations of persistent data structures I’ve talked about in part 2.

Further work

With that, I can work on the compiler, now for real. It’s a shame that I basically have to do all the work on the compiler again from scratch, because the underlying data has changed so much, but this simplifies a lot of things, so I’m OK with that.

One thing that the reader still doesn’t support yet is read-time evaluation with #=expr. This requires a working compiler, and I’ll have to first implement it, then integrate it back into the reader.

Another thing I was thinking about was to drop the LPEG parser altogether. Maybe, when I have the compiler working and have support for all Clojure runtime semantics in place, I’ll make a fork of the Edamame parser and replace the current reader with it, adding support for ClojureFnl via reader conditionals. This will remove a C library dependency, which is a good thing for distribution. I still have another C library for arbitrary-precision numbers, but it can be worked around.

But, lesson learned - when implementing a lisp, even if it is just a transpiler, don’t try to cut corners, and make a proper reader.

Next post, for sure, will be about the compiler!

Comment via email

Better slopes in AABB collision systems

Wed, 17 Jun 2026 19:14:00 +0300

Earlier this year I published a guide on how to implement slopes in AABB collision resolution using bump.lua. The resulting system worked, but was a bit hard to use in-game and had a few issues, so I wouldn’t consider it a viable solution.

The main issue with the old approach was the use of the cross collision response. It allowed the object to move through the slope as if it wasn’t there at all, and then the update loop corrected the y position after the fact. This alone makes writing code the game a lot more complicated than it needs to be:

(fn player-collision-filter [_ obj]
  (if (not= nil obj.slope)
      :cross
      :slide))

(fn grounded? [obj]
  (let [(_ y collisions l)
        (world:check obj obj.x (+ obj.y 1) player-collision-filter)]
    (and (or (faccumulate [res nil i 1 l :until res]
               (let [col (. collisions i)]
                 (when col.other.slope ;; <- eeeeeh...
                   (= obj.y (col.other.slope obj)))))
             (= obj.y y))
         collisions)))

(fn update [player main]
  ;; ---8<---
  (let [(x y collisions l)
        (world:move
         player
         (+ player.x player.x-vel)
         (+ player.y player.y-vel)
         player-collision-filter)]
    (doto player
      (tset :x x)
      (tset :y y))
    (for [i 1 l]
      (let [collision (. collisions i)
            other collision.other]
        (when other.slope
          (case (other.slope player)
            y* (when (>= player.y y*)
                 (world:update player player.x y* player.w player.h) ;; <- yikes!
                 (doto player
                   (tset :y y*)))))))
    ;; ---8<---
    ))

I had to resort to calling the world:update method, which, as described in the bump’s docs, is used to update object’s position in the world without checking collisions. This made slopes really awkward to use and work with. And while I suspected that this was a wrong approach, I decided to go with it anyway, because at the time I didn’t knew better.

Then, while working on the game for the Spring Lisp Game Jam I again needed a custom collision handling that is not supported by bump.lua directly, but a bit simpler this time around. It’s a common thing in games, where you can pass through the block from one side, but not from the other. Like clouds usually behave in platformers - you can cross them from below, but still land on them:

Your browser does not support the video tag.

A cloud platform in the game Rayman (1995)

However, bump.lua by default only supports the following type of collisions: slide, cross, touch, and bounce:

Figure 1: Response types from bump.lua documentation

Solving this using the cross handler seemed wrong, and after a bit of searching I learned that in bump.lua you can define your own collision response functions. So I defined my own collision response called slide-cross (because the main intention was to be able to go through from the sides with the slide response but cross over it from below):

(fn slide-cross [world col x y w h goal-x goal-y filter]
  (let [goal-x (or goal-x x)]
    (var goal-y (or goal-y y))
    (if (<= (+ y h) col.otherRect.y)
        (let [tch col.touch
              move col.move]
          (when (or (not= move.x 0) (not= move.y 0))
            (set goal-y tch.y))
          (set col.slide {:x goal-x :y goal-y})
          (let [(cols len) (world:project col.item tch.x tch.y w h goal-x goal-y filter)]
            (values goal-x goal-y cols len)))
        (let [(cols len) (world:project col.item x y w h goal-x goal-y filter)]
          (values goal-x goal-y cols len)))))

(world:addResponse :slide-cross slide-cross)

With such colliders in place, I could replicate cloud platform behavior:

Your browser does not support the video tag.

I didn’t use clouds themselves in the actual game - instead, this collider was assigned to specific blocks I wanted to be able to run past, acting more as background structures you can still jump on, i.e. pedestals, but the idea of making a custom collider got me thinking. If I can make a custom collision response function, why not try to do it for slopes too?

So that’s what I did. But before that, I had to redo slopes a bit.

Slopes in `bump.lua`

First, let’s start with how slopes are defined:

(fn make-slope [x y x0 y0 x1 y1 slope-type]
  (let [(nx ny) (slope-normal x0 y0 x1 y1 slope-type)]
    {:slope slope-type
     :surface #(+ y0 (/ (* (- y1 y0) (- (clamp x0 $ x1) x0)) (- x1 x0)))
     : x : y : nx : ny
     :h (m/max y0 y1) :w (m/max x0 x1)}))

For example, to define a slope at coordinates x,y that is 16px long and 8px tall (), we can call make-slope like this:

(make-slope x y 0 8 16 0 :floor)

A 45° 8x8 slope () can be defined as follows:

(make-slope x y 0 8 8 0 :floor)

These slopes get taller from left to right, if we want a slope that goes lower from left to right (), we swap the coordinates:

(make-slope x y 0 0 16 8 :floor)

This function returns an object representing the slope. Of course, we’re not limited to defining 16x8 and 8x8 slopes - these are just the most logical choices for a platformer in TIC-80. We can define steeper slopes like 8x16, or use an arbitrary size altogether. As long as it can be clearly represented by the underlying spritework given resolution constraints you work in.

The make-slope constructor computes the slope’s normal vector:

(fn slope-normal [x0 y0 x1 y1 slope-type]
  (let [dx (- x1 x0)
        dy (- y1 y0)
        len (m/sqrt (+ (* dx dx) (* dy dy)))
        nx (/ (- dy) len)
        ny (/ dx len)]
    (if (or (and (= slope-type :floor) (> ny 0))
            (and (= slope-type :ceiling) (< ny 0)))
        (values (- nx) (- ny))
        (values nx ny))))

It will be used in our response function to implement bouncing off the slope surface correctly - one of the things I fell short while making the previous version.

Slope response function for `bump.lua`

Now we can define the response function itself:

(fn slope-response [world col x y w h goal-x goal-y filter]
  (let [goal-x (or goal-x x)
        {: surface :slope slope-type : nx : ny} col.other
        {:x ox :y oy} col.otherRect
        {:y touch-y} col.touch]
    (var goal-y (or goal-y y))
    (case slope-type
      :floor (let [surf-y (m/min (+ oy (surface (- goal-x ox)))
                                 (+ oy (surface (- (+ goal-x w) ox))))]
               (if (> (+ goal-y h) surf-y)
                   (if (> col.normal.y 0)
                       (set goal-y touch-y)
                       (set [goal-y col.normal.x col.normal.y] [(- surf-y h) nx ny]))
                   (set [col.normal.x col.normal.y] [0 0])))
      :ceiling (let [surf-y (m/max (+ oy (surface (- goal-x ox)))
                                   (+ oy (surface (- (+ goal-x w) ox))))]
                 (if (< goal-y surf-y)
                     (if (< col.normal.y 0)
                         (set goal-y touch-y)
                         (set [goal-y col.normal.x col.normal.y] [surf-y nx ny]))
                     (set [col.normal.x col.normal.y] [0 0])))
      _ (error (.. "unknown slope type: " (tostring _))))
    (set col.slide {:x goal-x :y goal-y})
    (local (cols len) (world:project col.item x y w h goal-x goal-y filter))
    (values goal-x goal-y cols len)))

It has two case branches - one that I’ve omitted in the above explanation of make-slope.

The slope-type argument, set to floor in those examples, determines if a slope is a ground one or a one on the ceiling. Another thing that wasn’t supported in the old method.

This response function works as follows:

It gets goal-x and goal-y, as well as other usual parameters, and the collision object.
The slope is always in col.other, because slope itself never collides with anything (they’re never passed to world:move calls).
We get the slope’s rectangle to adjust our object’s position across the slope’s width.
Then we check slope-type
- For floor slopes we get the min of the y coordinate of the slope y.
- Then we check if the goal-y received by the collision response function is below the slope’s surface.
- If so, and the normal vector points up, meaning the collision comes from above the slope, we set the goal-y and normal vector.
For ceiling slopes, the logic is the same, only some signs and checks are inverted.
If approached from the other side of the slope, i.e. from below - the collision acts like a regular AABB, so the slope forms a right triangle.

Handling goal-y and normal vector inside the response function provides much cleaner slopes:

Your browser does not support the video tag.

Note: it may seem that balls float over the slopes without touching, but that's only a visual quirk because ball's collider is still a 8x8 square - it touches the slope wiht a corner, but the sprite makes it look that it's floating. A more precise collider for the ball is needed, but it's out of the scope of this post.

The old implementation couldn’t achieve such smooth gliding over the slope’s surface. Balls get stuck on adjacent slopes, and incorrect normals prevent balls from rolling off smoothly:

Your browser does not support the video tag.

Note: in this clip the ball seems to touch the slope properly, but again, it's just a visual thing - in reality the slope is a little below the sprite's visual representation of it.

Now, when slopes are defined better, we can integrate them into a mock-up platformer to test them in a game environment:

Your browser does not support the video tag.

I don’t use normal vectors here, since in a platformer the jump usually goes up, regardless of the terrain, but depending on the player’s state one could use normals as well, given that we have them in the collision info.

The old implementation of slopes had problems, that can be seen here:

Your browser does not support the video tag.

Notice that when transitioning from slope to solid ground, the player transitions to the falling state for one frame. This was because we had to handle slope collisions outside of the collider itself, in the update loop - we couldn’t cross slopes unless they’re a tiny bit higher than the ground next to it. This problem is now eliminated by handling slopes during bump’s collision resolution step.

But we don’t have to stop there - what we have now is basically an extension to normal AABB with custom height functions that are bound by bounding boxes. So we can create any kind of slopes, like ramps - we just need a proper height function. However, we did a small optimisation that prevents us from doing it right now - we pre-computed slope normals. If we use a height function whose normal vector changes along the x coordinate, we need to handle that in the slope-response function:

 (fn slope-response [world col x y w h goal-x goal-y filter]
   (let [goal-x (or goal-x x)
-        {: surface :slope slope-type : nx : ny} col.other
+        {: surface :slope slope-type} col.other
         {:x ox :y oy} col.otherRect
-        {:y touch-y} col.touch]
+        cx (- (+ goal-x (/ w 2)) ox)
+        (nx ny) (slope-normal (- cx 1) (surface (- cx 1))
+                              (+ cx 1) (surface (+ cx 1)) slope-type)]
     (var goal-y (or goal-y y))
     (case slope-type
       :floor (let [surf-y (m/min (+ oy (surface (- goal-x ox)))
                                  (+ oy (surface (- (+ goal-x w) ox))))]
                (if (> (+ goal-y h) surf-y)
-                   (if (> col.normal.y 0)
-                       (set goal-y touch-y)
-                       (set [goal-y col.normal.x col.normal.y] [(- surf-y h) nx ny]))
+                   (set [goal-y col.normal.x col.normal.y] [(- surf-y h) nx ny])
                    (set [col.normal.x col.normal.y] [0 0])))
       :ceiling (let [surf-y (m/max (+ oy (surface (- goal-x ox)))
                                    (+ oy (surface (- (+ goal-x w) ox))))]
                  (if (< goal-y surf-y)
-                     (if (< col.normal.y 0)
-                         (set goal-y touch-y)
-                         (set [goal-y col.normal.x col.normal.y] [surf-y nx ny]))
+                     (set [goal-y col.normal.x col.normal.y] [surf-y nx ny])
                      (set [col.normal.x col.normal.y] [0 0])))
       _ (error (.. "unknown slope type: " (tostring _))))
     (set col.slide {:x goal-x :y goal-y})

With that in place, we can even make dynamic slopes, like a moving sine wave, and objects will correctly respond:

Your browser does not support the video tag.

For my purposes, I don’t need this kind of flexibility yet, but it’s nice to be able to do it still. I could see this being used in a game like Sonic, where you have loops, which are circles. Defining four slopes for each circle quadrant (two bottom ones as floor slopes and upper ones as ceiling slopes), and handling how to enter and exit them by marking some of them inactive via the cross collision response. Handling normals in this case would prove handy, as Sonic jumps perpendicular to the normal at the current position on the slope.

Your browser does not support the video tag.

However, we lost one additional feature our original slope-response had: these slopes don’t have solid opposite side. Triangular floor slopes prevented objects from entering the slope from beneath, only from the sides and from above the slope. Sine-wave slope will teleport objects to the slope side if an object entered inside - it’s a correctness measure, because of a more complicated geometry.

That’s not a bad thing on its own, since slopes are usually surrounded by other types of ground anyway, but still a thing to consider.

With that, I have almost perfect slopes for bump.lua, and I can use it in systems where I can’t access more sophisticated collision detection systems like Box2D. There are still things to be aware of, though. First, because bump’s collision resolution does a single pass, we can’t handle cases well where a ball touches the slope, another wall, and another ball. Second, the order of collisions can give different outcomes - if a ball lands exactly in the middle of two slopes, the outcome is determined by which slope bump.lua will check first. So it’s good for simple cases, but not a proper slope collision simulation, like one found in non-AABB systems.

On that note, I think I’m satisfied with what I have, and I’ll try to use this in my next game projects! And, as always, the you can try this online here, and a downloadable version of a cart is provided below:

Comment via email

Spring Lisp Game Jam 2026

Sun, 24 May 2026 08:08:00 +0300

This year I decided to participate in the Spring Lisp Game Jam. It’s an annual event, where you have to make a game in any kind of lisp in a limited time, usually a week.

I’ve been putting this away for several years, because every time the jam started I wasn’t ready to spend time on it, because of work or other duties I had at the time. This year, however, I decided to take a vacation, and give it my all. Life, of course, had other plans, but I managed to persevere.

This game jam doesn’t have a particular theme, which is good, as I always wanted to make a particular kind of a game - a movement-oriented platformer. I really enjoyed games like Metroid Fusion and Metroid Dread, which, to my taste, have great platforming, especially in Dread, so I wanted to make something akin to it. On paper, this idea seemed doable. In practice, well, you’ll see.

I decided to make “speed” the main thing of my game, thinking that the core idea will be to maintain speed, and complete room-based obstacle courses. My engine of choice was again TIC-80 as I had some experience with it already, and it’s the most frictionless environment I know for making small games. That’s debatable, of course, given TIC’s restrictions of resolution of 240x136 pixels and a 16-color-only palette. But I like restrictions, they breed creativity, as we all know.

This is more of a postmortem post, as I’m writing this after finishing the game - I simply had no capacity of writing this during the development. So information below is more of how I remember this, and of notes I made along the way.

Day 1

First things first, I chose TIC-80, and it supports programming in lisp out of the box. The language in question is Fennel, which is a nice lisp that compiles to Lua. I needed a few things though.

TIC doesn’t have any built-in functionality besides the essential things for drawing on the screen and reading user’s input. My game needs to deal with collisions, and while other game engines, like LOVE, include a library for that, TIC is certainly not.

A popular library for this is bump.lua, however, TIC doesn’t allow filesystem interactions, so the library must be bundled inside the cart. However, this is a Lua library, and I’m making a game in Fennel.

Fennel has a way of inlining dependencies via the --require-as-include option that could be used, however, it only works when compiling Fennel code to Lua. And I’m not sure if this will pass for the game jam. Plus, Fennel doesn’t compile comments, so I couldn’t just write the game cart in Fennel and then compile it to Lua, I’d need to translate all of the cart metadata by hand.

Luckily, a few years ago I made a port of bump.lua to Fennel for cases like this. I used it in the other game I was making at the time. You can read my old dev-log here. Anyhow, I needed to make an actual game, not just mess with libraries, so I picked a palette and began working.

First day was all about player controller. I needed a solid foundation to build the game around, and none of my previous player controllers were adequate for this. They were too stiff.

At the end of the day I ended up with this:

Your browser does not support the video tag.

This is a bit more than just a player controller - I have a camera system, level editor, placeholder graphics, and powerups already coded.

This is my camera system:

(local camera
  {:x 96
   :y 40
   :update
   (fn [self {: x : y}]
     (doto self
       (tset :x (* (// x 240) 240))
       (tset :y (* (// y 136) 136))))
   :draw
   (fn [{: x : y} remap]
     (map (// x 8) (// y 8) (+ (// x 8) 31) (+ (// y 8) 18) 0 0 0 1 remap))
   :translate
   (fn [{:x cam-x :y cam-y} x y]
     (values (- x cam-x) (- y cam-y)))})

It’s a simple object that has three main methods.

The update method is responsible for camera positioning. It is called on each frame, and it simply clamps the x, y position into the one full screen range. Thus, this is not a free camera, it is locked to the current screen.

I had dynamic cameras in the past, and while I like them, for this game I wanted something different. Essentially, my vision was that each screen will be an obstacle course, although even at this point I already knew this would be a bit problematic.

The draw method simply draws the current visible range of tiles via the map command. I have a remap callback ready in case I’ll need to change some tiles dynamically.

The final method translate is an interesting one. When I draw anything, like the player, or interactable objects, I need to do so in the camera space. So I call (camera:translate obj.x obj.y) to get screen-space coordinates. In other words, object’s coordinates may be 1337,420, but in camera space they as well may be 42,27.

The camera draws the level from the map editor built into TIC, but we need to load it into the game to interact with it properly. So we need a system to do that.

The level editor is another thing I reused from the last game I made with TIC.

(local objects
  {1 :player
   17 :speed-up
   33 :jump-up})

(macro id [x y]
  `(mget (// ,x 8) (// ,y 8)))

(fn obj? [x y]
  (. objects (id x y)))

(fn solid? [x y]
  (fget (id x y) 0))

(fn load-map-to-world []
  (for [x 0 (* 240 8) 8]
    (for [y 0 (* 136 8) 8]
      (case (obj? x y)
        :player (do (mset (// x 8) (// y 8) 0)
                    (doto player
                      (tset :x x)
                      (tset :y y)
                      (tset :spawn-x x)
                      (tset :spawn-y y))
                    (doto camera
                      (tset :x (* x 8))
                      (tset :y (* y 8)))
                    (world:add player player.x player.y player.w player.h))
        :speed-up
        (let [obj {: x : y :h 8 :w 8 :type :cross
                   :handle (fn [self target]
                             (world:remove self)
                             (mset (// self.x 8) (// self.y 8) 0)
                             (set target.max-xvel (+ target.max-xvel 1)))}]
          (world:add obj obj.x obj.y obj.w obj.h))
        :jump-up
        (let [obj {: x : y :h 8 :w 8 :type :cross
                   :handle
                   (fn [self target]
                     (world:remove self)
                     (mset (// self.x 8) (// self.y 8) 0)
                     (set target.jump-vel (- target.jump-vel 2)))}]
          (world:add obj obj.x obj.y obj.w obj.h))
        _ (if (solid? x y)
              (let [obj {: x : y :h 8 :w 8}]
                (world:add obj obj.x obj.y obj.w obj.h)))))))

It’s a simple loop that goes over each map tile, and decides if it is a thing that we need to register in bump’s world. To decide that I use a combination of tile ids and flags. It’s called at the start of the game, and loads the map once.

Finally, the most convoluted part was the player controller:

(fn player.update [self _transition-target]
  (let [grounded? (grounded? self collision-filter)
        wall-bound? (wall-bound? self collision-filter)]
    (echoes:update self)
    (when (bonk? self collision-filter)
      (set self.yvel 0))
    (if grounded?
        (set self.yvel 0)
        (apply-gravity self wall-bound?))
    (if (and (btn right) (or grounded? (not wall-bound?)))
        (doto self
          (tset :flip 1)
          (tset :xvel (lerp self.xvel (if wall-bound? (/ self.max-xvel 5) self.max-xvel) (if grounded? acc 0.01))))
        (and (btn left) (or grounded? (not wall-bound?)))
        (doto self
          (tset :flip -1)
          (tset :xvel (lerp self.xvel (if wall-bound? (/ self.max-xvel -5) (- self.max-xvel)) (if grounded? acc 0.01))))
        grounded?
        (if wall-bound?
            (set self.xvel 0)
            (let []
              (set self.xvel (snap-to-zero (round (lerp self.xvel 0 0.1)))))))
    (when (or (btnp A) (btnp B) (btnp X) (btnp Y))
      (if grounded?
          (set self.yvel self.jump-vel)
          wall-bound?
          (doto self
            (tset :flip (- self.flip))
            (tset :xvel (if (= self.xvel 0)
                            (* 3 (- wall-bound?))
                            (- self.xvel)))
            (tset :yvel (* self.jump-vel 0.75)))))

    (let [(x y cols len) (world:move self (+ self.x self.xvel) (+ self.y self.yvel) collision-filter)]
      (for [i 1 len]
        (let [{: other} (. cols i)]
          (if other.handle
              (other:handle self))))
      (set self.x x)
      (set self.y y))))

This is again a method in the player object, which holds some state, like coordinates and other attributes of the player, like max speed. It’s fairly simple right now, but it’ll get worse in the future days.

One of the things you may have noticed was a trail effect, when the player reaches a certain speed. I wanted it in the game, inspired by Super Metroid’s speed booster effect. However, my small brain couldn’t figure out how to do it properly, so I simply used linear interpolation so each echo lags behind the player object just enough, and used time offsets to delay their animations. Later, I realized that I can do it simply by storing last player position every few frames and drawing the echo this way, but I had no time to fix it.

The animation system is another thing I needed to make myself, as TIC doesn’t have anything like that. There’s a popular library for doing that, called anim8, by the same author of bump.lua. I didn’t use it, as I had to learn it, and wasn’t sure if it’s compatible with TIC at all.

So my animation system is as follows:

(fn draw-anim [self x y flip? speed]
  (when (= (% t speed) 0)
    (set self.frame (% (+ self.frame 1) self.frames)))
  (spr (+ self.start (* self.frame 2)) (- x 4) y 0 1 flip? 0 2 2))

(local run-anim
  {:start 258
   :end 268
   :frames 6
   :frame 0
   :draw draw-anim})

(local jump-anim
  {:start 288
   :end 302
   :frames 8
   :frame 0
   :draw draw-anim})

;; etc

In the player.draw method I call this depending on the player’s state:

(fn player.draw [self]
  (let [(x y) (camera:translate self.x self.y)
        flip? (if (< self.flip 0) 1 0)]
    (echoes:draw self)
    (if (<= 0 (m/abs self.xvel) 0.01)
        (if (not= 0 self.yvel)
            (slow-jump-anim:draw x y flip? player)
            (idle-anim:draw x y flip? 60))
        (not (grounded? self))
        (case (wall-bound? player)
          1 (spr 256 x y 0 1 1 0 1 2)
          -1 (spr 256 x y 0 1 0 0 1 2)
          _ (if (> (m/abs self.xvel) 3)
                (jump-anim:draw x y flip? (m/floor (lerp 10 1 (/ (m/abs self.xvel) (m/abs self.max-xvel)))))
                (slow-jump-anim:draw x y flip? player)))
        (run-anim:draw x y flip? (m/floor (lerp 10 3 (/ (m/abs self.xvel) (m/abs self.max-xvel))))))))

As you can see, I don’t yet have a state machine or anything like it in my player object, which I fixed in later days. However, this was enough to make the animations you’ve seen in the video.

Animation speed is determined by the speed parameter, which is named incorrectly, as it is more of an update interval in frames. I use linear interpolation to play animations faster when player’s speed increases, interpolating from 10 tics to 1 tic. Each frame is 60 tics.

Day 2

Second day I focused on improving animations, as currently they look very janky.

I have Aseprite installed, so I picked up my drawing tablet and sat down working on the run animation. Or I would have done that if Wacom wasn’t a piece of crap.

I had to install drivers, but not just any drivers - the newest drivers are incompatible with my rather old tablet. They are quite big size-wise, and with slow internet it was a pain, since I blindly downloaded the latest driver, and was met with a message, that I should upgrade my device. Luckily, the official web page still has a working driver for my OS and tablet combination, but I was afraid for a second that I would have to resort to using my phone. Not that it is a problem, but I wanted to try Aseprite for this project.

The result is as follows:

Your browser does not support the video tag.

I also added sliding animation, and spikes, as no platformer is complete without spikes, am I right or am I right?

Jumping also has two states now - a slow jump, and a fast jump, each with their own animation. For slow jumps the player sprite is mostly static, changing only when the vertical velocity goes from negative to positive. When a certain amount of speed is reached, however, the jump changes into a Salto Mortale mode, where the player spins frantically, like Samus Aran. Did I mention that Metroid was a huge inspiration for me this time around?

In addition to sliding, I added wall slide, which is not only a visual aid, but also slows you down when falling a bit.

Day 3

I felt satisfied with my player controller so far, so I started working on the level itself. This includes decorations, and level design in general.

Here’s what I got by the end of day three:

Your browser does not support the video tag.

I designed a few tiles for the ground and walls, as well as variants for different biomes:

It’s hard to show the whole map in TIC, so here’s a screenshot from the built-in overview:

Aesthetics for this game were inspired by the palette I’ve chosen, and by the game Animal Well, which I’ve played recently. Or at least I wanted to think that way, as I liked the artstyle, and limited resolution of that game.

Days 4-7

The rest of the days I’ve been working on the level design. I even started to plan out the level on a piece of paper, like so:

Here’s map progress by day:

Figure 1: Day 4

Figure 2: Day 4, again

Figure 3: Day 5

Figure 4: Day 6

At some point I completed the paper plan:

Figure 5: Day 7

Figure 6: Day 8

I was ready to start working on the last bit of the map, when I realized that I made a mistake. My plan contained an extra row on top, when in reality I had just two more.

This meant that I had to trim down the last section, as I couldn’t really throw anything from the middle section, and it wouldn’t make sense to make the top section penetrate down.

Figure 7: Day 9

Still, I had the full map complete by day nine, out of ten days available for the jam. A bit of a close call, but I could release this game any time, as it is essentially complete now.

During these days I also made a main menu screen, and the end screen. The end screen feels a bit barebones, but I had no time to do anything more with it, as I still had to do the most difficult part - sounds and music.

SFX

Sound effects weren’t as hard, when it comes to the player itself. I’m using simple effects for steps, jumps, and slides based on white noise that TIC provides. Timing them to animation frames is easy enough, as I can call sfx from the animation pipeline itself.

Because of that, things like steps are actually in sync with the animation, which I like. However, in the browser the sound may lag, I don’t really know why, but it did for all my previous games. Maybe it also depends on the browser.

Day 10 - Music

As I usually do, I left the hardest part to the last day. If I fail at it, I won’t include it in the game, and release it as is - it’s still a complete game. But, if I succeed, the game will be a bit better.

Music composition is not of my top skills, I’d say I’m pretty bad at it. There are few reasons for that, first and foremost being it’s so clunky to do that on a computer. If I had a MIDI keyboard, or an iPad with GarageBand, I think I would spend more time doing that.

So I reached out to a friend with whom we did a collaboration before, and asked them to help me out. They were kind enough to provide me with an original piece, which I had to port to TIC with the best of my ability.

TIC’s music tracker is a bit weird, and very limiting for someone who’s not versatile enough with trackers in general.

Adapting the song was quite a challenge for me. First of all, the song was written in non-tracker software, as MIDI. Having a piano-roll view of it helped with adapting, but some of the effects that were used were hard to recreate, so I did my best.

For example, delay. I had to create a wave that modulated its volume to act like its two notes repeating after a short delay.

Then there were drums. I need to mention that I’m limited by three channels out of four available. One channel is reserved for in-game sound effects, like steps, jumps, powerups, etc. This leaves me with three channels for music.

I used the first channel for the lead melody, second one for the bass, and the third one for the drum beat. Drums proved that it’s not really feasible to do them in one channel. You can’t overlay two samples, like in a modern MIDI editor. So you have to choose between kick and snare. Or snare and cymbal. Or kick and cymbal. You get the idea.

The game

Anyway, this concludes this dev log, and the game is now available on itch.io. If you like it, consider voting for it on the jam’s page. And if you wish to download it, here’s the cart:

I can complete this game in about 10 minutes, but I think it’ll take a fair bit more time for players unfamiliar with it. After all, I spent a lot of time playing this game during development, and I know it pretty well.

I know that some levels in this game are probably very janky. Many jumps are probably not as obvious as they should have been, or unnecessarily hard. I’ve had no time to get this game play-tested by other people, so that’s that. Still, I fixed the most annoying places that drove even me nuts. And overall, I’m satisfied with what I made.

I hope you like this game, but don’t forget to check other submissions too! If you’re a fellow participant, I wish you luck, and let’s see who’s gonna be the winner! And if not, I encourage you to try it next year. It’s fun :)

Comment via email

Clojure on Fennel part three: parsing

Mon, 27 Apr 2026 19:48:00 +0300

The two previous posts were not related to the compiler itself, but were kicked off by the start of the compiler development. I’d say this project was the reason that I made proper immutable data structures for Fennel and Lua. But now we’re finally going to talk about the compiler itself! And we’ll start with the parsing stage - transforming Clojure code into something that can be operated on by the compiler.

Manual single-pass parser and compiler

When I started this project, as I usually do, I underestimated the complexity at hand. My first attempt was a single-pass compiler that accepted a stream of characters of Clojure code and re-compiled it on the fly.

My idea was to write a simple recursive-descent parser that, when accepting a character would decide how to translate it to Fennel, assuming my fennel-cljlib library handled most of the semantics. So, for example, when encountering [ it would translate it to (cljlib.vector and thus [1 2 3] would descend to compile-vector and expand to (cljlib.vector 1 2 3).

The idea is simple, and at first I thought that it would suffice, but then I remembered that there are a lot of other things in Clojure that I have to consider. For example #_ - the ignore form syntax, and ^foo - the metadata syntax. This complicated things a lot, so I added initial support for them and moved on - because I decided that this will be a bootstrap compiler for a proper Clojure parser.

Clojure parsers

After some time I had a compiler that could compile most of the arbitrary code I threw at it. Of course, it was not exactly to the Clojure spec, but it worked and I was going to replace it anyway. So I started looking at available Clojure parsers written in Clojure.

Edamame

I started with Edamame. The choice was based on the fact that the Squint project uses it and my project is similar to Squint, or so I thought. This parser is also used in SCI - a Clojure interpreter, so it seemed like a good fit.

The first thing I did was make sure that Edamame at least compiles with my bootstrap compiler. The fact that it compiles doesn’t mean that it works - it just translated everything into Fennel, with bits of cljlib here and there. So the next step was to make it work.

But then I had a realization that I don’t need to do that at all - I could just parse Edamame with Edamame, and write a compiler against its output! Meaning, I could use Edamame’s parser directly until my compiler is complete enough to compile Edamame. And since my idea was to replace bootstrap compiler in the end anyway, I decided that it doesn’t make much sense to make it work enough to fully compile Edamame into a runnable Fennel code. So I used Edamame to parse Edamame, and got back…

… the same Edamame source.

Imagine my face when I realized that Edamame doesn’t produce an abstract syntax tree with all information, but instead it just produces Clojure data structures, such as lists, vectors, and other stuff. And all of the information about the source code sits inside metadata of these data structures, not in the output. Yes, it does transform some Clojure forms into lists, like @foo is transformed into (deref foo), but that’s still nothing to me, because Fennel has no list data type and its support for metadata is nowhere near compatibility with Clojure.

I get it, I should have read the readme with more attention, and don’t make assumptions based on wild guesses, so it’s fine. Still, I wasn’t dropping the idea to use an existing Clojure parser, and the bootstrap compiler combo. I just need a different parser.

Rewrite-clj

Next one I looked at was rewrite-clj.

This library does return a tree of objects. It’s still a list, but at least it contains nodes that can be translated into Fennel objects. However, it still had a few problems.

First, its source code is much bigger than that of Edamame. Its API is more complicated, and it expects the use of Zippers, which I don’t yet have in cljlib. I didn’t want such a big parser for my compiler, given that it’s job is quite simple - produce an AST that I will always walk in full.

Second, it’s still producing some Clojure data structures that I can’t cleanly map to Fennel, so it would mean that the compiler itself would depend on cljlib. And I didn’t want that.

Luckily for me, their user guide mentioned another parser.

Parcera

Finally, parcera.

Looking at the readme, I finally saw what I wanted to see:

(ns example.core
  (:require [parcera.core :as parcera]))

;;parse clojure code from a string
(parcera/ast (str '(ns parcera.core
                     (:require [clojure.data :as data]
                               [clojure.string :as str]))))

;; => returns a data structure with the result from the parser
(:code
 (:list
  (:symbol "ns")
  (:whitespace " ")
  (:symbol "parcera.core")
  (:whitespace " ")
  (:list
   (:keyword ":require")
   (:whitespace " ")
   (:vector (:symbol "clojure.data")
            (:whitespace " ")
            (:keyword ":as")
            (:whitespace " ")
            (:symbol "data"))
   (:whitespace " ")
   (:vector (:symbol "clojure.string")
            (:whitespace " ")
            (:keyword ":as")
            (:whitespace " ")
            (:symbol "str")))))

Tagged tree. Yes, it’s still a list, but at least this can be easily mapped to Fennel sequential tables. So I opened the code and…

…there’s nothing there! Turns out, this project uses an ANTLR grammar for parsing Clojure.

At this point I was quite frustrated with the fact that there’s no single Clojure parser that could provide something similar to what parcera provides, but written in pure Clojure. Or at least in Java, which I could hand-translate as a last resort. Yes, there’s a Clojure parser inside the Clojure project itself, written in Java, but my goal was to recompile an existing parser written in Clojure.

Grammar-based parser

After a bit of consideration, I decided that a grammar-based approach is not that bad of an idea after all. I couldn’t use the ANTLR grammar directly, but at least I had it, and it seemed complete enough.

In Lua there’s a great library called LPeg that can generate parsers at runtime from a PEG grammar, and I wanted to try it out for quite some time. But to do that I needed a PEG grammar for Clojure. ANTLR and PEG are quite different, but it’s not impossible to translate one grammar into another.

So that’s what I did:

# Clojure PEG Grammar
#
# Adapted from https://github.com/carocad/parcera (ANTLR4 grammar v0.11.6)
#
# Key differences:
#
# - Unlike ANTLR, PEG choices are ordered (/).
#   More specific alternatives need to come first.
# - No separate lexer phase.
#   All rules operate on characters directly.
# - ANTLR fragment rules are prefixed with '_'.
# - ANTLR's SENTINEL (catch-all for invalid tokens) has no direct PEG equivalent.
#   The parser will silently fail on invalid input.

# Parser rules

code        <- input* !.

input       <- ignore / form

ignore      <- whitespace / comment / discard

form        <- literal / collection / reader_macro

# sets and namespaced maps are under dispatch (they start with #)
collection  <- list / vector / map

list        <- '(' input* ')'

vector      <- '[' input* ']'

map         <- '{' input* '}'

# macro_keyword before keyword ('::' is longer than ':')
literal     <- macro_keyword / keyword / string / number / character / symbol

keyword       <- KEYWORD
macro_keyword <- MACRO_KEYWORD
string        <- STRING

# ordering: most-specific prefix first; LONG last (least specific)
number        <- HEXADECIMAL / OCTAL / RADIX / DOUBLE / RATIO / LONG

# NAMED_CHAR first (longest); UNICODE before UNICODE_CHAR (more specific)
character     <- NAMED_CHAR / OCTAL_CHAR / UNICODE / UNICODE_CHAR

symbol        <- SYMBOL

# unquote_splicing before unquote ('~@' vs '~')
reader_macro <- unquote_splicing
              / unquote
              / metadata
              / backtick
              / quote
              / dispatch
              / deref

metadata <- ((metadata_entry / deprecated_metadata_entry) ignore*)+
            ( symbol
            / collection
            / set
            / namespaced_map
            / tag
            / fn
            / unquote_splicing
            / unquote
            / conditional_splicing
            / conditional
            / deref
            / quote
            / backtick
            / var_quote
            )

metadata_entry            <- '^' ignore* (map / symbol / string / keyword / macro_keyword / conditional)

# #^ is deprecated syntax for metadata
deprecated_metadata_entry <- '#^' ignore* (map / symbol / string / keyword / macro_keyword / conditional)

backtick          <- '`' ignore* form

quote             <- "'" ignore* form

# negative lookahead prevents '~' from matching when '~@' follows
unquote           <- '~' !'@' ignore* form

unquote_splicing  <- '~@' ignore* form

deref             <- '@' ignore* form

# conditional_splicing before conditional ('#?@' vs '#?')
# tag last (most general: '#' + symbol)
dispatch <- conditional_splicing
          / conditional
          / set
          / namespaced_map
          / fn
          / regex
          / var_quote
          / symbolic
          / eval
          / tag

# no whitespace allowed between '#' and the delimiter
fn                    <- '#' list

regex                 <- '#' STRING

set                   <- '#{' input* '}'

namespaced_map        <- '#' (macro_keyword / keyword / auto_resolve) ignore* map

auto_resolve          <- '::'

var_quote             <- "#'" ignore* form

discard               <- '#_' ignore* form

tag                   <- '#' symbol ignore* form

conditional           <- '#?' whitespace* list

conditional_splicing  <- '#?@' whitespace* list

# ##Inf, ##-Inf, ##NaN  (allows arbitrary symbolic values)
symbolic              <- '##' ignore* SYMBOL

eval                  <- '#=' ignore* (symbol / list / conditional)

whitespace <- WHITESPACE

comment    <- COMMENT


# Lexical rules (terminals)

HEXADECIMAL <- _SIGN? '0' [xX] [0-9A-Fa-f]+ _BIG_INT?

OCTAL       <- _SIGN? '0' [0-7]+ _BIG_INT?

# base 2-36, then r/R, then digits in that base
RADIX       <- _SIGN? ([2-9] / [12] [0-9] / '3' [0-6]) [rR] [0-9a-zA-Z]+

RATIO       <- _SIGN? _DIGIT+ '/' _DIGIT+

LONG        <- _SIGN? _DECIMAL _BIG_INT?

# FRACTION EXPONENT before FRACTION alone (longer match first)
DOUBLE      <- _SIGN? _DECIMAL+ ((_FRACTION _EXPONENT / _FRACTION / _EXPONENT) 'M'? / 'M')

_BIG_INT  <- 'N'
_FRACTION <- '.' _DIGIT*
_EXPONENT <- [eE] _SIGN? _DIGIT+
_DECIMAL  <- '0' / [1-9] _DIGIT*


STRING <- '"' ([^"\\] / '\\' .)* '"'


# \p{White_Space} approximation
WHITESPACE <- [ \t\n\r\f,]+

COMMENT <- (';' / '#!') [^\r\n]*


# Named characters must not be followed by name-chars (word boundary)
NAMED_CHAR   <- '\\' ('newline' / 'return' / 'space' / 'tab' / 'formfeed' / 'backspace')
                !(_ALLOWED_NAME_CHARACTER / _DIGIT)

UNICODE      <- '\\' 'u' [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]
                !(_ALLOWED_NAME_CHARACTER / _DIGIT)

# octal char: 0-377; longest alternative first
OCTAL_CHAR   <- '\\' 'o' ([0-3] [0-7] [0-7] / [0-7] [0-7] / [0-7])
                !(_ALLOWED_NAME_CHARACTER / _DIGIT)

# any single non-combining-mark character after backslash
UNICODE_CHAR <- '\\' ![\u0300-\u036F\u1DC0-\u1DFF\u20D0-\u20FF] .
                !(_ALLOWED_NAME_CHARACTER / _DIGIT)


# ::/ is NOT a valid macro keyword
MACRO_KEYWORD <- '::' (_SIMPLE_KEYWORD '/')? _SIMPLE_KEYWORD

KEYWORD       <- ':' (_SIMPLE_KEYWORD '/')? (_SIMPLE_KEYWORD / '/')

_SIMPLE_KEYWORD <- _KEYWORD_HEAD _KEYWORD_BODY+
                 / _KEYWORD_HEAD

_KEYWORD_BODY <- _KEYWORD_HEAD / ':'

_KEYWORD_HEAD <- _ALLOWED_NAME_CHARACTER / _DIGIT / [#'] / _SIGN


SYMBOL <- (_SIMPLE_SYMBOL '/')? (_SIMPLE_SYMBOL / '/')

# longer alternatives first for correct PEG matching
_SIMPLE_SYMBOL <- _SIGN _SYMBOL_HEAD _SYMBOL_BODY*
                / _ALLOWED_NAME_CHARACTER (_SYMBOL_HEAD / _DIGIT / ':') _SYMBOL_BODY*
                / _ALLOWED_NAME_CHARACTER
                / _SIGN

_SYMBOL_BODY <- _SYMBOL_HEAD / _DIGIT / ':'

_SYMBOL_HEAD <- _ALLOWED_NAME_CHARACTER / [#'] / _SIGN


# Characters allowed in names: everything EXCEPT reserved characters.
# Equivalent to ANTLR ~[\p{White_Space},()[\]{}"@~^;`\\:#'/0-9+-]
# Note: \p{White_Space} approximated with common whitespace chars here.
_ALLOWED_NAME_CHARACTER <- ![ \t\n\r\f,()\[\]{}"@~^;`\\:#'/0-9+\-] .

_SIGN  <- [+\-]

_DIGIT <- [0-9]

But of course, you can’t just load a PEG grammar into LPeg. Instead, you need to write the grammar programmatically, using LPeg’s API. I’ll spare you the details, but here’s a full source code of the parser.

After conversion work, I finally had a parser that could produce an AST:

>> (local {: parse} (require :impl.parser))
nil
>> (parse "(ns parcera.core
             (:require [clojure.data :as data]
                       [clojure.string :as str]))")
[:code
 [:list
  [:symbol "ns"]
  [:whitespace " "]
  [:symbol "parcera.core"]
  [:whitespace "\n             "]
  [:list
   [:keyword ":require"]
   [:whitespace " "]
   [:vector
    [:symbol "clojure.data"]
    [:whitespace " "]
    [:keyword ":as"]
    [:whitespace " "]
    [:symbol "data"]]
   [:whitespace "\n                       "]
   [:vector
    [:symbol "clojure.string"]
    [:whitespace " "]
    [:keyword ":as"]
    [:whitespace " "]
    [:symbol "str"]]]]]

Or not:

>> (parse "(ns parcera.core
             (:require [clojure.data :as data]
                       [clojure.string :as str])") ;; <- missing ')'
nil

Turns out, LPeg doesn’t have any way to report errors if the input string doesn’t match. It simply returns nil.

Luckily for me, there’s an additional library, called lpeglabel that can help with that. It implements an extension to the LPeg library that allows defining tagged failures:

>> (parse "(ns parcera.core
             (:require [clojure.data :as data]
                       [clojure.string :as str])")
unknown:3:49: Parse error: EOF while reading: Expected an ')'

Now I have a working Clojure parser with a usable output.

The parser

I mentioned before that Clojure metadata and special characters did introduce problems to my original parser. Well, with this grammar, this is no longer an issue:

>> (each [_ s (ipairs ["@foo" "^Foo bar" "^{:baz true} qux" "#_1 2"])]
     (pp (parse s)))
[:code [:deref [:symbol "foo"]]]
[:code [:metadata
        [:metadata-entry [:symbol "Foo"]]
        [:whitespace " "]
        [:symbol "bar"]]]
[:code [:metadata
        [:metadata-entry
         [:map [:keyword ":baz"] [:whitespace " "] [:symbol "true"]]]
        [:whitespace " "]
        [:symbol "qux"]]]
[:code [:discard [:number "1"]] [:whitespace " "] [:number "2"]]

Yes, the parser doesn’t transform @foo into (deref foo), but it’s OK, as I can handle it in the compiler easily now. Metadata is fully supported by the parser and is stripped in the compiler. And discard (#_) actually works in all of its supported ways, i.e. #_#_ 1 2 works fine.

Each parser result starts with :code, an entry point for the compiler. And the compiler itself is just a simple recursive descent over all of the nodes in a loop. So, that’s good, right?

Well, yes and no. There are some cons to this approach.

First, and foremost - this parser doesn’t support streaming, meaning I can’t stream chunks of code and compile them. Well, it’s not impossible, but currently it works on strings only. It is possible to write a second parser that would split text into top-level expressions, but for now it’s an optimization waiting to happen. Right now, this means that files will be parsed in full - having a greater footprint in memory.

Second, and arguably more important - this is a C library dependency. Both LPeg and lpeglabel are C libraries, meaning that the compiler now works only on Lua runtimes that can load object files. And the choice of systems where this compiler can run is now limited to those that can compile these libraries. Not that big of a deal, if you ask me, since C runs almost everywhere, but it’ll make the compiler difficult to ship as a no-dependency binary. It’s possible to compile a Fennel script into a binary and include all of its dependencies, but it would mean that I’ll have to produce binaries for many different operating systems and architectures. Or, alternatively, require people to install LPeg and lpeglabel through luarocks.

For now, I’m willing to accept this. There’s a port of LPeg in pure Lua, but it’s quite slow and doesn’t have support for lpeglabel features. There’s an open request for that, but it’s been sitting open for quite some time.

Other than that, the parsing is complete and we can look at the compiler part of the ClojureFnl project. But that’s gonna be in the next post.

Comment via email

Clojure on Fennel part two: immutable.fnl optimizations

Wed, 15 Apr 2026 05:11:00 +0300

So the next post will hopefully be about the compiler itself.

Unless I get distracted again.

Sike!

While I did some work on the compiler, I’m not feeling ready to talk about it yet. I want the post about it to be, well, more in-depth, so for now, let’s go back to immutable data structures. I got some comments on their performance, which can be summarized to “bad”.

And I agree! The performance is not great, especially when compared to plain Lua tables. However, my benchmarking process was a bit flawed, and I’ve improved it since.

This post won’t be long, but I wanted to make it anyway, to illustrate what I’m dealing with. Again, this is not the final version, I’m still working on possible optimizations as I write this, but some things have improved already, though not by much.

There were some low-hanging fruits that I could tackle:

Vector had a local fragment function with pure arithmetic. I already have the same fragment function in my bitops module that uses native rshift+band. The modulo operation is expensive.
A similar case of using bit operations instead of modulo was in HashMap’s index function.
Increased the popcount table to 256 entries instead of 16. Halves the loop iterations for typical bitmaps.
Probably premature, but replaced most instances of faccumulate, fcollect, etc. with plain for loops. These loops are quite hot, and Fennel compiles them to have additional assignment at each iteration. Should be free, but actually had a small impact.
More pre-allocations when creating Lua tables. Using [nil nil nil nil] tells the Lua compiler to pre-allocate four slots when creating the table, meaning it won’t resize once we insert elements. Tables in Lua resize by a factor of 2 and copy everything on resize, which is not great for performance.
Vector iterator is now stateful. And also fast.
Internal fields for data structures used long, elaborate keys, which meant string comparison in certain cases of table lookup would be unnecessarily long. Again, changing them to shorter keys should not matter that much, but it did have some impact.
HashMap branching factor was increased to 32. After some tests, it proved to be slightly better, despite what I said in my previous post. More on that later.
Hashes are now cached in a weak table. Helps avoid re-hashing the same values at the expense of memory. Not sure how well it will scale.
Optimized array copying for sizes less than 4 by doing direct table construction. Avoids loop overhead.

With these changes in place (and some more I omitted), here are some of the benchmarks:

These images show per-commit changes to the performance. Really helped me track what was worth keeping, and what wasn’t.

Here’s the LuaJIT version of the benchmark:

Here’s another graph:

This is me testing how hash map branching factor affects operations. As can be seen, switching from 16 to 32 makes inserts slower on PUC Lua 5.5 but makes indexing faster. On LuaJIT both operations are faster.

Looking at the charts, the 7989c32 commit with that change (the second orange bar) — insertions are slower than the previous commit, and indexing is faster than the previous commit. Unfortunately, LuaJIT charts are noisy and I can’t confirm the speedup there. Something to do with instabilities in the LuaJIT tracer, I guess.

Note I know, the results for some operations are worse than before all optimizations that took place. This is still a work in progress. Just shows how important relative measuring is.

The results are a mixed bag. Most operations that improved did so by only 1 or 2 milliseconds overall, meaning the per-operation cost is still quite high, compared to Lua tables. Yes, some became faster, so I’m keeping these changes, but I can also see that for some operations, they actually got slower than they were before any optimization work. The reason for that may be that I fixed a few bugs along the way. Anyway, I tried to prioritize transient collection variants, as they benefit most from optimizations given how they’re used, and LuaJIT performance where possible.

Some bars are not meaningful for certain operations. E.g., I had changes that had nothing to do with Persistent Vector, but affected it anyway, so this benchmark is still a bit flawed. At least those differences are within the deviation range.

Anyhow, just wanted to show that optimization is hard, and not all changes yield the results you expect. In the case of this library, I don’t think there are more low-hanging fruits to optimize, or at least I don’t see any. For example, there are a lot of table allocations that are hard to get rid of in the current implementation. A deeper analysis is needed, probably by people with more Lua knowledge.

I’m still satisfied with some improvements I got, but we’ll see if more are on the way.

Update

After a lot more benchmarks, flamegraphs, and throwing stuff at the wall, I think I can’t do much more. I decided to update this post instead of writing another one (and really should have waited before publishing the original) because at that point, I thought there was nothing more to do. Now I think there really isn’t. So the results are in:

Figure 1: PUC Lua 5.5 (green - before, orange - after)

Figure 2: LuaJIT (green - before, orange - after)

As can be seen, the changes are more dramatic on Lua than on LuaJIT. Mostly because it’s hard to optimize for LuaJIT without sacrificing performance on Lua, so I re-prioritized Lua, since LuaJIT is already much faster.

The final benchmark tables are below.

PUC Lua 5.5

PersistentVector

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
build 50000	0.18 ms	18.88 ms	102x	0.378 us	119x	0.440 us
lookup random	0.48 ms	11.44 ms	24x	0.229 us	28x	0.266 us
assoc random	0.31 ms	42.72 ms	139x	0.854 us	222x	1.4 us
pop all	0.14 ms	14.56 ms	106x	0.291 us	150x	0.415 us
iterate	0.60 ms	2.74 ms	5x	0.055 us	17x	0.196 us

TransientVector

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
build 50000	0.19 ms	6.55 ms	35x	0.131 us	41x	0.154 us
assoc random	0.31 ms	17.64 ms	57x	0.353 us	67x	0.408 us
pop all	0.14 ms	5.74 ms	42x	0.115 us	51x	0.142 us

PersistentHashMap

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
insert 50000	2.01 ms	81.44 ms	41x	1.6 us	82x	3.2 us
lookup random	0.92 ms	34.03 ms	37x	0.681 us	81x	1.7 us
contains random	1.82 ms	33.08 ms	18x	0.662 us	49x	1.7 us
dissoc all	0.90 ms	76.97 ms	86x	1.5 us	187x	3.3 us
dissoc 10%	0.18 ms	9.81 ms	56x	2.0 us	112x	3.7 us
iterate	1.91 ms	7.22 ms	4x	0.144 us	4x	0.146 us
lookup structural keys	0.14 ms	2.86 ms	21x	0.572 us	146x	4.1 us
insert collision-heavy	0.13 ms	6.50 ms	51x	1.3 us	134x	3.5 us
mixed 80/20	1.92 ms	53.67 ms	28x	1.1 us	62x	2.3 us

TransientHashMap

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
insert 50000	2.02 ms	38.91 ms	19x	0.778 us	44x	1.7 us
lookup random	1.14 ms	35.56 ms	31x	0.711 us	86x	1.7 us
dissoc all	0.89 ms	44.52 ms	50x	0.890 us	112x	2.0 us
mixed 80/20	4.39 ms	44.87 ms	10x	0.897 us	22x	1.9 us

LuaJIT

PersistentVector

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
build 50000	0.11 ms	8.12 ms	71x	0.162 us	84x	0.177 us
lookup random	0.06 ms	0.67 ms	11x	0.013 us	11x	0.013 us
assoc random	0.04 ms	22.91 ms	559x	0.458 us	731x	0.585 us
pop all	0.02 ms	10.77 ms	718x	0.215 us	657x	0.210 us
iterate	0.02 ms	0.23 ms	12x	0.005 us	26x	0.011 us

TransientVector

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
build 50000	0.05 ms	0.79 ms	16x	0.016 us	12x	0.012 us
assoc random	0.04 ms	2.22 ms	55x	0.044 us	28x	0.022 us
pop all	0.02 ms	0.35 ms	23x	0.007 us	50x	0.016 us

PersistentHashMap

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
insert 50000	0.79 ms	32.32 ms	41x	0.646 us	50x	0.989 us
lookup random	0.34 ms	5.78 ms	17x	0.116 us	43x	0.248 us
contains random	0.30 ms	5.91 ms	20x	0.118 us	40x	0.257 us
dissoc all	0.23 ms	27.02 ms	119x	0.540 us	191x	0.894 us
dissoc 10%	0.05 ms	5.89 ms	118x	1.2 us	121x	1.4 us
iterate	0.07 ms	1.07 ms	16x	0.021 us	30x	0.040 us
lookup structural keys	0.02 ms	2.41 ms	121x	0.483 us	119x	0.498 us
insert collision-heavy	0.28 ms	5.20 ms	19x	1.0 us	36x	1.2 us
mixed 80/20	0.50 ms	46.59 ms	94x	0.932 us	40x	0.513 us

TransientHashMap

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
insert 50000	0.91 ms	30.92 ms	34x	0.618 us	11x	0.242 us
lookup random	0.32 ms	34.90 ms	110x	0.698 us	44x	0.246 us
dissoc all	0.23 ms	36.35 ms	162x	0.727 us	57x	0.271 us
mixed 80/20	1.29 ms	40.71 ms	32x	0.814 us	11x	0.309 us

Conclusion

Yes, the benchmark shows that some operations slowed down, some sped up. Ideally, I’d like to have no slowdown, but that’s that. Still, I wouldn’t trust my LuaJIT benchmark, though - graphs show that for some weird reason unrelated changes affect performance in the modules not touched by those changes at all. So these changes are now merged into the main branch.

Main bottleneck now is table allocation and array copying - things I already spent optimizing a lot of time with mixed results. So I think I can’t do better than that. At least without major algorithmic restructuring.

With that, the next post should be about the compiler.

Comment via email

Clojure on Fennel part one: Persistent Data Structures

Tue, 07 Apr 2026 02:47:00 +0300

Somewhere in 2019 I started a project that aimed to bring some of Clojure features to Lua runtime - fennel-cljlib. It was a library for Fennel that implemented a basic subset of clojure.core namespace functions and macros. My goal was simple - I enjoy working with Clojure, but I don’t use it for hobby projects, so I wanted Fennel to feel more Clojure-like, besides what it already provides for that.

This library grew over the years, I implemented lazy sequences, added immutability, made a testing library, inspired by clojure.test and kaocha, and even made a port of clojure.core.async. It was a passion project, I almost never used it to write actual software. One notable exception is fenneldoc - a tool for documentation generation for Fennel libraries. And I haven’t seen anyone else use it for a serious project.

The reason for that is simple - it was an experiment. Corners were cut, and Fennel, being a Clojure-inspired lisp is not associated with functional programming the same way Clojure is. As a matter of fact, I wouldn’t recommend using this library for anything serious… yet.

Recently, however, I started a new project: ClojureFnl. This is a Clojure-to-Fennel compiler that uses fennel-cljlib as a foundation. It’s still in early days of development, but I’ve been working on it for a few months in private until I found a suitable way to make things work in March. As of this moment, it is capable of compiling most of .cljc files I threw at it, but running the compiled code is a different matter. I mean, it works to some degree, but the support for standard library is far from done.

;; Welcome to ClojureFnl REPL
;; ClojureFnl v0.0.1
;; Fennel 1.6.1 on PUC Lua 5.5
user=> (defn prime? [n]
         (not (some zero? (map #(rem n %) (range 2 n)))))
#<function: 0x89ba7c550>
user=> (for [x (range 3 33 2)
             :when (prime? x)]
         x)
(3 5 7 11 13 17 19 23 29 31)
user=>

However, there was a problem.

My initial implementation of immutable data structures in the itable library had a serious flaw. The whole library was a simple hack based on the copy-on-write approach and a bunch of Lua metatables to enforce immutability. As a result, all operations were extremely slow. It was fine as an experiment, but if I wanted to go further with ClojureFnl, I had to replace it. The same problem plagued immutableredblacktree.lua, an implementation of a copy-on-write red-black tree I made for sorted maps. It did a full copy of the tree each time it was modified.

For associative tables it wasn’t that big of a deal - usually maps contain a small amount of keys, and itable only copied levels that needed to be changed. So, if you had a map with, say, ten keys, and each of those keys contained another map with ten keys, adding, removing or updating a key in the outer map meant only copying these ten keys - not the whole nested map. I could do that reliably, because inner maps were immutable too.

But for arrays the story is usually quite different. Arrays often store a lot of indices, and rarely are nested (or at least not as often as maps). And copying arrays on every change quickly becomes expensive. I’ve mitigated some of the performance problems by implementing my version of transients, however the beauty of Clojure’s data structures is that they’re quite fast even without this optimization.

Proper persistent data structures

Clojure uses Persistent HAMT as a base for its hash maps and sets, and a bit-partitioned trie for vectors. For sorted maps and sets, Clojure uses an immutable red-black tree implementation, but as far as I know it’s not doing a full copy of the tree, and it also has structural sharing properties.

I started looking into existing implementations of HAMT for Lua:

hamt.lua (based on mattbierner/hamt)
- seemed incomplete
ltrie
- no transients
- no hashset
- no ordered map (expectable, different algorithm)
- no compound vector/hash
Michael-Keith Bernard’s gist
- no custom hashing

I could use one of those, notably ltrie seemed the most appropriate one, but given that I’m working on a fennel library that I want later to embed into my Clojure compiler I needed a library implemented in Fennel.

So I made my own library: immutable.fnl. This library features HAMT-hash maps, hash-sets, and vectors, as well as a better implementation of a persistent red-black tree, and lazy linked lists.

Persistent Hash Map

I started the implementation with a Persistent HAMT with native Lua hashing. The data structure itself is a Hash Array Mapped Trie (HAMT) with 16-factor branching. Thus all operations are O(Log16 N), which is effectively O(1) for a practical amount of keys.

As far as I know, Clojure uses branching factor of 32, but for a Lua runtime this would mean that the popcount would be more expensive, and despite a shallower tree, each mutation would need to copy a larger sparse array. With branching factor of 16 a map with 50K entries is ~4 levels deep, which would be ~3 with 32 branching factor. So my logic was that it’ll be a compromise, especially since Lua is not JVM when it comes to performance.

Of course, it’s not as fast as a pure Lua table, which is to be expected. Lua tables are implemented in C, use efficient hashing, and dynamically re-allocated based on key count. So for my implementation most operations are a lot slower, but the total time for an operation is still usable.

Here are some benchmarks:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Regular operations are notably slower, when compared to Lua:

Operation	Lua table	Persistent HashMap	Ratio	per op
insert 50000 random keys	2.05 ms	164.80 ms	80.3x slower	3.3 us
lookup 50000 random keys	0.83 ms	92.51 ms	110.8x slower	1.9 us
delete all	0.78 ms	170.78 ms	219.8x slower	3.4 us
delete 10%	0.14 ms	19.50 ms	136.4x slower	3.9 us
iterate 50000 entries	1.74 ms	6.64 ms	3.8x slower	0.133 us

For transients the situation is a bit better, but not by much:

Operation	Lua table	Transient HashMap	Ratio	per op
insert 50000 random keys	2.05 ms	89.17 ms	43.5x slower	1.8 us
delete all	0.76 ms	104.31 ms	138.0x slower	2.1 us
delete 10%	0.16 ms	12.71 ms	82.0x slower	2.5 us

On LuaJIT numbers may seem worse, but per-operation cost is much lower, it’s just that native table operations are so much faster:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	Persistent HashMap	Ratio	per op
insert 50000 random keys	0.86 ms	49.05 ms	56.8x slower	0.981 us
lookup 50000 random keys	0.27 ms	14.21 ms	53.4x slower	0.284 us
delete all	0.13 ms	48.63 ms	374.1x slower	0.973 us
delete 10%	0.05 ms	6.49 ms	138.1x slower	1.3 us
iterate 50000 entries	0.07 ms	1.80 ms	27.7x slower	0.036 us

Operation	Lua table	Transient HashMap	Ratio	per op
insert 50000 random keys	0.76 ms	22.43 ms	29.6x slower	0.449 us
delete all	0.15 ms	34.16 ms	232.4x slower	0.683 us
delete 10%	0.04 ms	5.02 ms	132.1x slower	1.0 us

With a branching factor of 32 the situation gets worse on PUC Lua, but is slightly better on LuaJIT. So there’s still space for fine-tuning.

For hashing strings and objects I decided to use djb2 algorithm. I am almost as old as this hash function, so seemed like a good fit. JK. The main reason to use it was that it can be implemented even if we don’t have any bit-wise operators, and Lua doesn’t have them in all of the versions. It only uses +, *, and % arithmetic operators, so can be done on any Lua version. It’s prone to collisions, and I try to mitigate that by randomizing it when the library is loaded.

Still, collisions do happen, but HAMT core ensures that they will still resolve correctly by implementing a deep equality function for most objects.

However, when first working on this, I noticed this:

>> (local hash-map (require :io.gitlab.andreyorst.immutable.PersistentHashMap))
nil
>> (local {: hash} (require :io.gitlab.andreyorst.immutable.impl.hash))
nil
>> (hash (hash-map :foo 1 :bar 2))
161272824
>> (hash {:foo 1 :bar 2})
161272824
>> (hash-map (hash-map :foo 1 :bar 2) 1 {:foo 1 :bar 2} 2)
{{:foo 1 :bar 2} 2}

This is an interesting loophole. What object ended up in our hash map as a key - our persistent map or plain Lua table? Well, that depends on insertion order:

>> (each [_ k (pairs (hash-map (hash-map :foo 1 :bar 2) 1 {:foo 1 :bar 2} 2))]
     (print (getmetatable k)))
IPersistentHashMap: 0x824d9b570
nil
>> (each [_ k (pairs (hash-map {:foo 1 :bar 2} 2 (hash-map :foo 1 :bar 2) 1))]
     (print (getmetatable k)))
nil
nil

To reiterate, I’m creating a hash map, with a key set to another persistent hash map, and then insert a plain Lua table with the same content. The Lua table hashes to exactly the same hash, and goes into the same bucket, but there’s no collision, because objects are equal by value. But equality of mutable collections is very loosely defined - it may be equal right now, but the next time you look at it, it’s different. So a different hashing was needed for persistent collections, to avoid these kinds of collision. I ended up salting persistent collections with their prototype address in memory.

Other than that, the HAMT implementation is by the book, and the rest is the interface for interacting with maps.

Main operations:

new - construct a new map of key value pairs
assoc - associate a key with a value
dissoc - remove key from the map
conj - universal method for association, much like in Clojure
contains - check if key is in the map
count - map size, constant time
get - get a key value from a map
keys - get a lazy list of keys
vals - get a lazy list of values
transient - convert a map to a transient

Coercion/conversion:

from - create a map from another object
to-table - convert a map to a Lua table
iterator - get an iterator to use in Lua loops

Transient operations:

assoc! - mutable assoc
dissoc! - mutable dissoc
persistent - convert back to persistent variant, and mark transient as completed

This covers most of the needs in my fennel-cljlib library, as anything besides it I can implement myself, or just adapt existing implementations.

A Persistent Hash Set is also available as a thin wrapper around PersistentHashMap with a few method changes.

A note on PersistentArrayMap.

In Clojure there is a second kind of maps that are ordered, not sorted, called a Persistent Array Map. They are used by default when defining a map with eight keys or less, like {:foo 1 :bar 2}. The idea is simple - for such a small map, a linear search through all keys is faster than with a HAMT-based map.

However, in my testing on the Lua runtime, there’s no benefit in this kind of a data structure, apart from it being an ordered variant. Lookup is slower, because of a custom equality function, which does deep comparison.

Persistent Vector

Persistent Vectors came next, and while the trie structure is similar to hash maps, vectors use direct index-based navigation instead of hashing, with a branching factor of 32. Unlike maps, vector arrays in the HAMT are more densely packed, and therefore a higher branching factor is better for performance. So lookup, update, and pop are O(log32 N), append can be considered O(1) amortized.

Still, compared to plain Lua sequential tables the performance is not as good:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Operation	Lua table	Persistent Vector	Ratio	per op
insert 50000 elements	0.19 ms	21.07 ms	109.7x slower	0.421 us
lookup 50000 random indices	0.47 ms	14.05 ms	29.7x slower	0.281 us
update 50000 random indices	0.32 ms	70.04 ms	221.6x slower	1.4 us
pop all 50000 elements	0.25 ms	24.34 ms	96.2x slower	0.487 us
iterate 50000 elements	0.63 ms	10.16 ms	16.2x slower	0.203 us

Operation	Lua table	Transient Vector	Ratio	per op
insert 50000 elements	0.19 ms	7.81 ms	40.3x slower	0.156 us
update 50000 random indices	0.33 ms	20.76 ms	62.4x slower	0.415 us
pop all 50000 elements	0.25 ms	11.14 ms	44.4x slower	0.223 us

On LuaJIT:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	Persistent Vector	Ratio	per op
insert 50000 elements	0.10 ms	7.62 ms	74.0x slower	0.152 us
lookup 50000 random indices	0.06 ms	0.67 ms	11.8x slower	0.013 us
update 50000 random indices	0.04 ms	29.13 ms	710.4x slower	0.583 us
pop all 50000 elements	0.02 ms	8.62 ms	410.4x slower	0.172 us
iterate 50000 elements	0.02 ms	0.57 ms	28.7x slower	0.011 us

Operation	Lua table	Transient Vector	Ratio	per op
insert 50000 elements	0.05 ms	0.59 ms	11.6x slower	0.012 us
update 50000 random indices	0.04 ms	2.06 ms	51.6x slower	0.041 us
pop all 50000 elements	0.02 ms	0.84 ms	46.7x slower	0.017 us

I think this is an OK performance still. Vectors don’t use hashing, instead it is a direct index traversal via bit-shifting, so there’s no hashing, just index math.

Operations on vectors include:

new - constructor
conj - append to the tail
assoc - change a value at given index
count - element count (constant time)
get - get value at given index
pop - remove last
transient - convert to a transient
subvec - create a slice of the vector in constant time

Transient operations:

assoc! - mutable assoc
conj! - mutable conj
pop! - mutable pop
persistent - convert back to persistent and finalize

Interop:

from - creates a vector from any other collection
iterator - returns an iterator for use in Lua loops
to-table - converts to a sequential Lua table

One notable difference in both vector and hash-map is that it allows nil to be used as a value (and as a key, in case of the hash-map). Vectors don’t have the same problem that Lua sequential tables have, where length is not well-defined if the table has holes in it.

It’s a debate for another time, whether allowing nil as a value (and especially as a key) is a good decision to make, but Clojure already made it for me. So for this project I decided to support it.

Persistent Red-Black Tree

For sorted maps and sorted sets I chose Okasaki’s insertion and Germane & Might’s deletion algorithms. Most of the knowledge I got from this amazing blog post by Matt Might.

I believe the operations are O(Log N), as for any binary tree, but given that the deletion algorithm is tricky, I’m not exactly sure:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Operation	Lua table	PersistentTreeMap	Ratio	per op
insert 50000 random keys	2.10 ms	209.23 ms	99.8x slower	4.2 us
lookup 50000 random keys	0.88 ms	82.97 ms	94.2x slower	1.7 us
delete all	0.74 ms	173.76 ms	234.8x slower	3.5 us

On LuaJIT:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	PersistentTreeMap	Ratio	per op
insert 50000 random keys	0.72 ms	101.08 ms	140.4x slower	2.0 us
lookup 50000 random keys	0.25 ms	12.67 ms	49.9x slower	0.253 us
delete all	0.14 ms	56.14 ms	403.9x slower	1.1 us

The API for sorted maps and sets is the same as to their hash counterparts with a small difference - no transients. Clojure doesn’t do them, and I’m not doing them too.

That’s all for benchmarks. I know that there are many problems with this kind of benchmarking, so take it with a grain of salt. Still, the results are far, far better than what I had with itable.

But there are two more data structures to talk about.

Persistent List

As I mentioned, I made a lazy persistent list implementation a while ago but it had its problems and I couldn’t integrate that library with the current one well enough.

The main problem was that this library uses a single shared metatable per data structure, and the old implementation of lazy lists didn’t. This difference makes it hard to check whether the object is a table, hash-map, list, vector, set, etc. So I reimplemented them.

The reason for old implementation to use different metatables was because I decided to try the approach described in Reversing the technical interview post by Kyle Kingsbury (Aphyr). I know this post is more of a fun joke, but it actually makes sense to define linked lists like that in Lua.

See, tables are mutable, and you can’t do much about it. Closures, on the other hand are much harder to mutate - you can still do it via the debug module, but it’s hard, and it’s not always present. So storing head and tail in function closures was a deliberate choice.

However, it meant that I needed to somehow attach metadata to the function, to make it act like a data structure, and you can’t just use setmetatable on a function. Again, you can do debug.setmetatable but all function objects share the same metadata table. So, while you can do fancy things like this:

>> (fn comp [f g] (fn [...] (f (g ...))))
#<function: 0x7bdb320a0>
>> (debug.setmetatable (fn []) {:__add comp})
#<function: 0x7bd17f040>
>> ((+ string.reverse string.upper) "foo")
"OOF"

You can also notice, that our + overload applied to functions in the string module.

So instead, we use a table, and wrap it with a metatable that has a __call metamethod, essentially making our table act like a function. This, in turn means, that we have to create two tables per list node - one to give to the user, the other to set our __call and use it as a meta-table.

Convoluted, I know. It’s all in the past now - current implementation is a simple {:head 42 :tail {...}} table. Not sure what is worse.

But that meant that I had to rework how lazy lists worked, because previously it was just a metatable swap. Now list stores a “thunk”, that when called replaces itself in the node with the :head and :tail keys. Unless it’s an empty list, of course - in that case we swap the metatable to an empty list one.

So Lists have three metatables now:

IPersistentList
IPersistentList$Empty
IPersistentList$Lazy

Instead of god knows how many in the old implementation.

The list interface is also better now. Previously it was hardcoded how to construct a list from a data structure. Current implementation also hardcodes it, but also allows to build a list in a lazy way from an iterator.

This is better, because now a custom data structure that has weird iteration schema (like maps and sets in this library), we still can convert it to a list. A general case is just:

(PersistentList.from-iterator #(pairs data) (fn [_ v] v))

Meaning that we pass a function that will produce the iterator, and a function to capture values from that iterator. Reminds me of clojure transducers in some way.

Persistent Queue

And the final data structure - a persistent queue. Fast append at the end, and also fast remove from the front.

It’s done by holding two collections - a linked list at the front, and a persistent vector for the rear. So removing from the list is O(1), and appending to the vector is also pretty much O(1).

Interesting things start to happen when we exhaust the list part - we need to move vector’s contents into the list. It is done by calling PersistentList.from on the rear. And building a list out of a persistent vector is an O(1) operation as well! Well, because nothing happens, we simply create an iterator, and build the list in a lazy way. But since indexing the vector is essentially ~O(1), we can say that we still retain this property.

Or at least that’s how I reasoned about this - I’m not that good with time-complexity stuff.

ClojureFnl

That concludes part one about ClojureFnl.

I know that this post was not about ClojureFnl at all, but I had to fix my underlying implementation first. Now, that I have better data structures to build from, I can get back working on the compiler itself. So the next post will hopefully be about the compiler itself.

Unless I get distracted again.

Comment via email

Implementing SSE with fnl-http and async.fnl

Wed, 30 Jul 2025 01:05:00 +0300

Some time ago, I was working on an HTTP library for Fennel. As a proof of concept, I added a module that implements a simple web server and wanted to experiment with it. The server can serve files from a directory using this simple handler:

(fn handler [{: method : path &as request}]
  (case method
    :GET (let [file (io.open (path:gsub "^/" "") :r)]
           (or file {:status 404 :body "not found"}))
    _ {:status 405
       :body "Method not allowed"}))

This handler is simple - when the client sends a GET request with a given path, for example GET http://localhost:8080/some/file.txt, the handler simply removes the leading / and tries to open the given path as a file handle. If the file is present, we return it as is, and the server will handle the closure of this file once the client disconnects. Otherwise, we return a table with the 404 status code, indicating that the file wasn’t found. Other methods are not allowed, and we respond with a 405 error by matching on method with case.

Now we can require the server module and start a small file-serving server:

(local server
  (require :io.gitlab.andreyorst.fnl-http.server))

(local s
  (server.start handler {:port 8080}))

(s:wait)

The (s:wait) call simply locks the current thread because, without it, the process will just exit.

This is, of course, the simplest web server there is; it just accesses files in the current directory, returning them as application/octet-stream. The handler function can be more sophisticated, handling different methods, parsing arguments, etc. For our purposes today, we won’t need most of these features, but I thought it would be better to familiarize you with how the server is set up.

The topic of today’s post is Server-Sent Events, or SSE for short. It is a protocol for communicating between a server and multiple clients that uses a persistent connection and pushes messages from the server to the client. The client can then process these messages at their own pace and react accordingly.

We’ll need an asynchronous library to handle SSE for multiple clients, so let’s add it to the mix:

(import-macros {: go : go-loop}
  :io.gitlab.andreyorst.async)

(local {: <! : >! : timeout : alts! : pub : sub : chan : close! &as async}
  (require :io.gitlab.andreyorst.async))

(local json
  (require :io.gitlab.andreyorst.json))

I’ve also included a JSON library; both are already used by fnl-http. Now, let’s begin!

SSE stream

We’ll send events to our clients via a channel:

(local event-ch (chan))

This event channel will be our main way to send events to the clients, but we won’t read from it directly. Instead, we can create a “publication” or pub that will allow clients to subscribe to the messages:

(local event-pub
  (pub event-ch #:sse #(async.sliding-buffer 100)))

For the purposes of this post, there’s only one type of message in this event-pub thing, but in a more complex scenario, you can separate messages and subscribe to different topics altogether. Speaking of subscribing, here’s a helper function:

(fn subscribe-to-events [client-ch]
  (sub event-pub :sse client-ch))

Finally, we need a function to register an event:

(fn register-event [event]
  (>! event-ch event))

Now the fun part. Right now, our event system is ready to be accessed by clients, but the events themselves don’t make any sense, as they’re just some objects. We’ll define events as tables with the following structure:

{:event "optional event type"
 :id    "optional event id"
 :data  "event data"}

Since our register-event function simply writes to the event-ch, we need a way to intercept that. We can do so by writing middleware that accepts a channel with events as tables and returns a new channel that will have all of the events formatted:

(fn wrap-format-sse-event [ch]
  (let [out (chan)]
    (go-loop [e (<! ch)]
      (case e
        {: event : id : data}
        (doto out
          (>! (.. "event: " (tostring event) "\n"))
          (>! (.. "id: " (tostring id) "\n"))
          (>! (.. "data: " (tostring data) "\n")))
        {: event : data}
        (doto out
          (>! (.. "event: " (tostring event) "\n"))
          (>! (.. "data: " (tostring data) "\n")))
        {: data}
        (doto out
          (>! (.. "data: " (tostring data) "\n"))))
      (if e
          (recur (<! ch))
          (close! out)))
    out))

Now, the general mechanism is as follows:

When a client connects, we create a channel and subscribe to events.
We wrap this channel with our formatter and some other middleware if needed.
Finally, we return this channel as the response body.

Before we write an HTTP handler that does this, we need to consider one other thing: the connection. If the connection is not used for a long time, it can be closed automatically. Because of that, it is common to send meaningless heartbeat events every now and then. Let’s write another wrapper that will track if the client’s event stream doesn’t have any events for some time and return a heartbeat:

(fn wrap-heartbeat [ch timeout-ms]
  (let [out (chan)]
    (go-loop []
      (let [tout (timeout timeout-ms)]
        (match (alts! [tout ch])
          [_ tout]
          (do (>! out {:event "heartbeat" :data 0})
              (recur))
          [event ch]
          (do (>! out event)
              (recur))
          [nil ch] (close! out))))
    out))

Similarly to our previous wrapper, we accept a channel with events and return a new channel. The asynchronous go-loop, which I didn’t mention before, will run in a separate virtual thread, trying to read from both the given channel and a timeout channel. Whichever returns first will determine what event will be sent to the client. If a normal event comes through, we send it to the out channel as is and restart the timer. If no events come and our timer has been exhausted, we send a heartbeat and restart the timer again. If our events channel returns nil, it means that it was closed, so we also close the out channel.

Finally, we can implement our handler:

(fn handler [{: path : method : headers : content}]
  (case path
    (where "/events" (= :GET method))
    (let [events (chan)]
      (subscribe-to-events events)
      {:status 200
       :headers {:content-type "text/event-stream"}
       :body (-> events
                 (wrap-heartbeat 10_000)
                 wrap-format-sse-event)})
    (where "/notify" (= :POST method))
    (case (tonumber headers.Content-Length)
      len (let [event (json.decode (content:read len))]
            (register-event event)
            {:status 200
             :body "OK"})
      _ (error "malformed Content-Lenght"))
    _
    {:status 405
     :headers {:content-type "application/json"}
     :body (json.encode {:error "Method not allowed"})}))

This handler has two main entry points: /events and /notify. The /events entry is for subscribing to the events stream with a GET request. The /notify entry is for sending events from the client to the server (because right now we don’t want to make the server too complicated to generate events on its own).

Same as before, we can start a server:

(local s
  (server.start handler {:port 12345}))

(s:wait)

Reading the event stream and triggering events

With the server up and running, we can connect a pair of clients.

The first client will subscribe to the event stream by executing a GET request:

(local http
  (require :io.gitlab.andreyorst.fnl-http.client))

(local resp
  (http.get "http://localhost:12345/events" {:as :stream}))

If we look at the response, we’ll see the headers containing text/event-stream as the content type, as per the response from our server, and the body as a stream, as requested by the client:

{:body #<Reader: 0x564ae85ca7d0>
 :headers {:Connection "keep-alive"
           :Content-Type "text/event-stream"
           :Transfer-Encoding "chunked"}
 :http-client #<tcp-client: 0x564ae812fa10>
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 38
 :status 200
 :trace-redirects {}}

We can consume events with a helper function:

(fn consume-events [stream]
  (fn read-line [stream buf]
    (let [c (stream:read 1)]
      (when (not= "\n" c)
        (table.insert buf c)
        (read-line stream buf))))
  (while true
    (let [buf []]
      (read-line stream buf)
      (io.write "[" (os.date) "]\t" (table.concat buf) "\n"))))

(consume-events resp.body)

In the process output, we’ll be able to see heartbeat events appearing every ten seconds:

[Wed Jul 30 00:34:35 2025]	event: heartbeat
[Wed Jul 30 00:34:45 2025]	data: 0
[Wed Jul 30 00:34:45 2025]	event: heartbeat
[Wed Jul 30 00:34:55 2025]	data: 0

Next, we can send a POST request from another process, sending an event:

(local http
  (require :io.gitlab.andreyorst.fnl-http.client))
(local json
  (require :io.gitlab.andreyorst.json))

(http.post "http://localhost:12345/notify"
           {:body (json.encode {:event "message" :id "client-2" :data "Hi from client-2"})})

We immediately see the event in the first client’s output, followed by heartbeat events:

[Wed Jul 30 00:35:03 2025]	event: message
[Wed Jul 30 00:35:03 2025]	id: client-2
[Wed Jul 30 00:35:03 2025]	data: Hi from client-2
[Wed Jul 30 00:35:03 2025]	event: heartbeat
[Wed Jul 30 00:35:13 2025]	data: 0

As you can see, the time when the heartbeat event arrived is exactly ten seconds after the last non-heartbeat event. Of course, we can connect multiple clients subscribed to the event stream, and they will all receive events.

Conclusion¹

This should give you an idea of how to implement an SSE stream in your server using the fnl-http library. Of course, this implementation is only loosely based on the specification, so I wouldn’t suggest it as a general implementation guide, but I hope it provides a clear enough explanation of how things work.

The server module is still very much in a proof-of-concept stage, but as this article shows, it is working quite well. All in all, I believe the ability to return an asynchronous channel as a body from the server is a nice way to handle this kind of event system. The code can be written as usual, and no specific handling for returning asynchronous data via HTTP is required.

Thanks for reading, and I hope you’ll find fnl-http interesting enough to try!

As much as I don’t like “Conclusion” as the last heading in an article, I can’t think of a different one here. ↩︎

Comment via email

Implementing dynamic scope for Fennel and Lua

Mon, 09 Jun 2025 22:31:00 +0300

I’m continuing my work on fennel-cljlib, my port of clojure.core and some other core libraries, focusing on porting missing functions and features to it. One such feature, which I sometimes miss in Lua and Fennel, is dynamic binding.

The Lua VM doesn’t provide dynamic scoping as a language feature, and Fennel itself doesn’t introduce any concepts like Clojure’s Var. However, we can still implement dynamic scoping that works similarly to Clojure and other Lisps using the debug library. Most of the ideas are based on information from the guides at leafo.net. There’s even a “Dynamic scoping in Lua” guide that implements a slightly different version of this feature, requiring variables to be referenced by name via dynamic("var_name") call. While this approach is feasible, I wanted something more in line with how other Lisps work, so let’s explore advancing it further. Luckily for us, Leafo already has all the necessary guides!

But first things first. I wanted to delay working on dynamic scoping as much as possible because it is a feature that’s hard to get right. I already have some experience with implementing dynamic scoping for one of my older libraries that implemented a condition system from Common Lisp in Fennel. This library, however, required special syntax to access all of the dynamically bound symbols and thus did not actually require anything fancy for it to work.

So what does dynamic binding/scoping mean in a language? If you know about lexical and dynamic scoping and wish to skip this tangent, feel free to do so.

In short, lexically scoped variables exist only where their lexical scope allows them to. For example, a variable defined in a block of code will only exist in that block because it is its lexical scope.

When working with languages that have higher-order functions, I often find myself in a situation where I want to refactor some code that uses anonymous functions by moving them out and giving them a name. Sometimes it’s possible; sometimes it’s not.

For example, imagine I wanted to move out this function from map:

(fn some-func [messages]
  (let [extra-data (other-func)]
    (map
     (fn [message]
       (do-stuff message extra-data))
     messages)))

If I were to do so, we would have a problem:

(fn process-message [message]
  ;; oops, extra-data is now an unknown variable
  (do-stuff message extra-data))

(fn some-func [messages]
  (let [extra-data (other-func)]
    (map process-message messages)))

So extra-data is bound lexically, and thus if we look at the lexical scope of the process-messages function, we’ll see that it tries to use extra-data while it’s not defined there. If extra-data were a global variable, it wouldn’t be problematic, but it is a local variable with a lexical scope. We could move the entire let that binds extra-data to the result of calling other-func, but let’s say we don’t want to call it on each iteration of a map because it’s slow and will do the same work repeatedly. So, what are our options here?

Well, we can make it a closure!

(fn make-message-processor [extra-data]
  (fn process-message [message]
    (do-stuff message extra-data)))

(fn some-func [messages]
  (let [extra-data (other-func)]
    (map (make-message-processor extra-data) messages)))

Now, we pass extra-data only once to the make-message-processor function, and it returns a function that has this variable stored in a closure. However, this is still a lexical scope because, as you can see, extra-data is present there.

In a language that uses dynamic scoping, this could be a whole different story. Let’s look at Clojure; although I wouldn’t recommend doing it this way, it is possible to do it this way¹:

(defn process-message [message]
  (do-stuff message extra-data))

(defn some-func [messages]
  (binding [extra-data (other-func)]
    (mapv process-message messages)))

Here, I assume that extra-data is a dynamic variable that obeys the rules of dynamic scoping. The binding call introduces a dynamic scope within which extra-data is set to the value of (other-func). It acts more like a scoped global variable, or at least you can think of it that way.

To introduce a dynamic scope, Clojure uses binding. Let’s look at it briefly:

(defmacro binding
  "binding => var-symbol init-expr

  Creates new bindings for the (already-existing) vars, with the
  supplied initial values, executes the exprs in an implicit do, then
  re-establishes the bindings that existed before.  The new bindings
  are made in parallel (unlike let); all init-exprs are evaluated
  before the vars are bound to their new values."
  {:added "1.0"}
  [bindings & body]
  (assert-args
    (vector? bindings) "a vector for its binding"
    (even? (count bindings)) "an even number of forms in binding vector")
  (let [var-ize (fn [var-vals]
                  (loop [ret [] vvs (seq var-vals)]
                    (if vvs
                      (recur  (conj (conj ret `(var ~(first vvs))) (second vvs))
                             (next (next vvs)))
                      (seq ret))))]
    `(let []
       (push-thread-bindings (hash-map ~@(var-ize bindings)))
       (try
         ~@body
         (finally
           (pop-thread-bindings))))))

It’s a simple idea: a try block without catch statements, only with a finally clause. Before we enter the try block, we set all mentioned variables to their values, and after we’re done with the body, we restore those values.

In Clojure, dynamic bindings work really well, but this is due to a combination of factors. First, the try support in the JVM is excellent, ensuring that finally will perform its intended function. Additionally, the JVM supports thread-local bindings, so even in a multithreaded context, binding still works. Finally, heh, Clojure has Vars, which makes it all possible.

Now, let’s try to implement the same concept in Fennel!

Implementing dynamic scope in Fennel

Before I descend into madness, I would say that we could do the same thing as in Clojure: set some variables, run code in a protected call, and reset the variables afterward. While this approach would indeed work, I wanted to tidy up my understanding of function environments in Lua. It’s a neat concept that Lua and a few other languages have, but Lua is one of the few languages that actually allows users to manipulate function environments. So, let’s explore this idea.

First, we need a way to forcefully set a function’s environment. This could be done in Lua 5.1 via setfenv; however, it was removed starting from Lua 5.2 onward. It can be implemented like this:

(local setfenv
  (or _G.setfenv
      (fn setfenv [f env i]
        (let [i (or i 1)]
          (case (debug.getupvalue f i)
            :_ENV (doto f (debug.upvaluejoin i (fn [] env) 1))
            nil f
            _ (setfenv f env (+ i 1)))))))

Now, we can set an environment for any function.

But what exactly is this function environment? I’ve realized that I never explained that, so here we go.

In Lua, the environment is a table² that stores the names of the variables. Starting from Lua 5.2, the environment is represented by a variable called _ENV, which is what we’re testing for in setfenv above. By default, _ENV has the same value as _G, a table that contains all global variables. However, we can change the function’s environment by modifying the value of _ENV.

For instance, in Lua, we can set _ENV to a table, and all global definitions would end up in that table:

a = 0

local function f (t)
    _ENV = t
    a = 42
    b = 322
    return t
end

local t = {}

f(t)

print(a, b) -- 0, nil
print(t.a, t.b) -- 42, 322

This is a cool feature, and we can actually use environments for sandboxing code, but that’s a story for another time. Let’s return to dynamic scoping.

Looking at this, you might get the idea that if we change the function’s environment and set our dynamic variables in it specifically, once we leave the lexical scope of that _ENV, all changes revert to normal because they never happened in a global environment!

Unfortunately, it’s not that simple. Yes, we can change the function’s environment, but it will only affect that specific function. Moreover, this change is permanent, meaning that we’ll have to reset the function back to its original environment. So, it’s not as straightforward as just changing _ENV around the code we want to run. Of course, we could write getfenv, then wrap the entire thing in a pcall, and safely restore the environment once the work is done.

However, we can’t set the environment of just the function we’re calling. _ENV is stored in a closure, so we’ll need to change all of the functions called by the function we wish to invoke with a custom environment. This makes undoing changes trickier to implement.

Luckily for us, we can bypass the need to roll back the changes to the function’s environment completely! Instead, we can simply clone the function and set its environment as we wish! Here’s an implementation:

(fn clone-function-with-env [f env]
  "Recursively clones the function `f`, and any subsequent functions that it
might call via upvalues.  Sets `env` as environment for the cloned function."
  (let [dumped (string.dump f)
        cloned (load dumped)]
    (var (done? i) (values false 1))
    (while (not done?)
      (case (debug.getupvalue f i)
        (where (name val) (= :function (type val)))
        (let [subf (clone-function-with-env val env)]
          (debug.setupvalue cloned i subf))
        name
        (debug.upvaluejoin cloned i f i)
        nil (set done? true))
      (set i (+ i 1)))
    (setfenv cloned env)))

Finally, we write the function that will call a given function f in a context where the given bindings are dynamically bound:

(fn dynamic-call [bindings f ...]
  "Calls `f` with `bindings` as its root environment."
  (let [new-env (setmetatable bindings {:__index _ENV})
        f* (clone-function-with-env f new-env)]
    (f* ...)))

As well, as a convenience macro for using it like a let but with dynamic binding:

(macro binding [bindings ...]
  (assert-compile (sequence? bindings) "expected a sequence of bindings" bindings)
  (assert-compile
   (= 0 (% (length bindings) 2))
   "expected an even number of forms in binding sequence"
   bindings)
  `(dynamic-call
    ,(faccumulate [res {} i 1 (length bindings) 2]
       (doto res
         (tset (tostring (. bindings i)) (. bindings (+ i 1)))))
    (fn [] ,...)))

Now we can use dynamic binding in Fennel!

Usage example

To illustrate, let’s create some variables that we wish to treat dynamically:

(global foo 21)
(global bar 73)

(print foo bar) ;; 21	73

With globals in place, we can try our binding macro:

(binding [foo 42]
  (print foo bar))
;; 42 73

As can be seen, foo no longer refers to 21, but now it is 42. However, if we try to print foo outside of binding’s scope, we will again get 21:

(binding [foo 42]
  (print foo))
;; 42
(print foo)
;; 21

A keen reader would mention that this example is not so different from using a plain let:

(let [foo 42]
  (print foo))
;; 42
(print foo)
;; 21

And you’d be right! However, where it’s going to be different is when we put functions into the mix:

(fn f []
  (print "f:" foo bar))

How, if we were to call it inside of let, the bindings introduced by it won’t affect the function, because both foo and bar are not lexically present here:

(let [foo 42
      bar 1337]
  (f))
;; still prints:
;; f: 21 73

This is where binding jumps in. Instead of following lexical binding rules, that are natural for most languages, we now introduce dynamic binding of foo and bar:

(binding [foo 42
          bar 1337]
  (f))
;; prints:
;; f: 42 1337
(f)
;; prints:
;; f: 21 73

And, as can be seen above, inside binding’s scope, f sees foo as 42, and bar as 1337, while outside of it, the values are still 21 and 73. So, we never actually changed the values of foo and bar, they’re still 21 and 73, respectfully. Instead, in the scope of binding we changed how f accesses these variables.

This also works with functions that call other functions:

(fn f []
  (print "f:" foo))

(fn g []
  (f)
  (print "g:" bar))

(fn h []
  ((fn [] (print "h:" foo bar))))

(binding [foo 42
          bar 322]
  (f)
  ;; prints:
  ;; f: 42
  (g)
  ;; prints:
  ;; f: 42
  ;; g: 322
  (h)
  ;; prints:
  ;; h: 42 322
  )

That’s pretty much it! This approach has a lot of flaws though.

First, it will only work on ordinary functions, so no tricks with __call metamethod, or native functions are supported.

Second, it won’t work with coroutines either. You can’t use string.dump on something like coroutine.resume directly, so we won’t be able to do (dynamic-call {:foo 42} coroutine.resume some-coroutine). It won’t even work if we wrap coroutine.resume into an anonymous function like (dynamic-call {:foo 42} (fn [] (coroutine.resume coro))), because coro here, while being an upvalue, is not a function, so we can’t clone it.

Finally, it relies on the debug library, and recursive function dumping, which itself is already pretty crazy.

There are probably more things that can go wrong with this.

So why not just set the globals temporarily?

First of all, yes, we could just set the globals temporarily, and restore their values later:

(fn set-globals [globals]
  (collect [name new-val (pairs globals)]
    (let [old-val (. _G name)]
      (set (. _G name) new-val)
      (values name old-val))))

(fn close-handler [old-vals ok? ...]
  (each [name val (pairs old-vals)]
    (set (. _G name) val))
  (if ok?
      ...
      (error ... 0)))

(fn call-with-temp-globals [globals f ...]
  (-> globals
      set-globals
      (close-handler (pcall f ...))))

(call-with-temp-globals {:foo 123 :bar 456} g)
;; prints:
;; f: 123
;; g: 456
(g)
;; prints:
;; f: 21
;; g: 73

While this works, I don’t like the idea that we’re actually changing the values instead of shadowing them in the environment, though this is more akin to the original Clojure implementation. Since Lua is single-threaded it should not be problematic, however, I think it can still mess things up if we were to introduce some kind of an asynchronous scheduler, like in my async.fnl library. This also messes up stacktraces in cases where an error occurs as a result of calling f because you can’t re-throw errors in Lua like in other languages. There’s also a potential to use <close> marker that came in Lua 5.4 to avoid pcall altogether:

local function set_globals(globals)
    local old_values = {}
    for name, new_val in pairs(globals) do
        old_values[name] = _G[name]
        _G[name] = new_val
    end
    return old_values
end

local function close_handler(old_values)
    for name, val in pairs(old_values) do
        _G[name] = val
    end
end

local function dynamic_call_close(globals, f, ...)
    local old_values <close> =
        setmetatable(set_globals(globals), {__close = close_handler})
    return f(...)
end

This would keep stack traces intact, and values would be restored right when we exit dynamic_call_close but will only work in Lua 5.4.

Thus, while doing it with pcall is more generic, I wanted to explore the environment approach first since Lua already provides a mechanism for working with function environments. But for now, I think I’ll leave dynamic scoping out of cljlib as I’m not really sold on any of the ways of doing it that I’ve come up with so far.

I really don’t like this way of refactoring this code, but sometimes it is exactly what you need. Not this time, though. The approach with closures is what I’d use in this case in Clojure too. ↩︎
Everything in Lua is a table, because of course it is. ↩︎

Comment via email

Dynamic font-lock for Fennel

Mon, 10 Mar 2025 01:26:00 +0300

I’ve been working on adding a dynamic font-locking for fennel-mode for a while now. The first attempt started years ago, in July 2022, and were quite crude. It consisted of sending a special code to the REPL that would return me a list of macro names, and then I created a regular expression that matched all these names for syntax highlighting. This had several problems.

First, this meant that any macro known to the REPL would be highlighted. This is not great, as macros in Fennel can be loaded from any file that you require, but it doesn’t mean that they’re available to your file in particular.

For example, imagine you have this code:

Figure 1: All code in this post will be provided as images to better show code highlighting

As you can see, this code requires the module :foo, defines a few functions, and finally calls them. Ignore the fact that the module foo is unused for now - turn your attention to the colors instead.

As you can see, fennel-mode uses colors to tell the user a few things. Special forms and macros are highlighted in a blueish color. Inbuilt functions are highlighted in purple. Names of declared functions are also in purple, but not their calls.

However, you can see that for some reason, bar is highlighted blue, as if it was a special or a macro. This is because I’m using my old implementation of dynamic highlighting. If we look into the foo module, we’ll see that it requires a macro module, that defines bar as a macro. It’s not a problem - macros in Fennel are only available in the module they’re required, so there’s no name conflict here, as you would expect. However, highlighting is not very smart.

When syntax highlighting is applied, a request is sent to the REPL to get all available macros. Then no distinction is made for where these macros are coming from, hence the wrong highlighting.

Another tricky case, that I don’t see implemented around is scoping:

Figure 2: Like, imagine, otherwise I’d have to do this highlighting in CSS manually

In Fennel, the macro special, which is used to define macros, obeys the scoping rules. Thus when we create a macro inside of the do’s scope, this macro should not be highlighted outside of it. But again, because the initial implementation of dynamic highlighting didn’t care about anything but macro names, the macro was incorrectly highlighted in this case as well.

Because of these problems, Phil and I decided that this won’t be a good feature to bring into fennel-mode in this state. The branch rested for almost three years, up until recently, when I started working on fennel-mode again.

Recently, I added support for Flymake via the fennel-ls language server. That was another old branch of mine, where I added linting via Fennel’s own linter plugin, but Phil mentioned that the plugin was never supposed to be used like this. It was also suggested that since fennel-ls already existed back then, and it was working with eglot there was no real need for using Flymake with Fennel’s linter plugin.

I gave this branch a rest, and after a while, when I started using fennel-ls myself, I decided to contribute a small change so that it could be used from Flymake. Personally, I don’t need a full-blown language server for Fennel, because everything I need is provided by the Fennel Proto REPL, but I do want some linting capabilities. So after merging the Flymake branch into the main fennel-mode branch, I noticed this old dynamic-font-lock branch, and decided that “it’s time to do this”.

Thus, I’m happy to announce, that all the problems listed above were fixed! Take a look at this:

Of course, when we just load the file, nothing is highlighted yet, as the syntax highlighting is dynamic. However, when we load the module into the REPL, the highlighting is applied correctly:

Once the module is loaded, highlighting is cached, and no further requests to the REPL are sent. This doesn’t, however, mean that when we change the code, the highlighting won’t react:

Your browser does not support the video tag.

I wanted to make this feature for quite some time. I’m a big fan of CIDER’s dynamic font-locking and have been using it for some years. It never occurred to me, however, that CIDER doesn’t do scoping:

But Clojure macros are not scoped, like in Fennel, so that’s probably fine. What’s not as fine is that it will highlight the macro before it was declared:

Which would be an error if we were to try to reload this file. But, for what it’s worth, I think CIDER’s solution is fine, because again - no special scoping rules, and unless the file loads without errors, no dynamic syntax highlighting would be applied. For Fennel, however, I wanted a more robust solution.

For now, this is still available in a feature branch, but I’m positive that I’ll merge this soon, after I give it a bit more testing, and gather some feedback from the users. If you’re a fennel-mode user, and you use my fennel-proto-repl module, I encourage you to try out dynamic font-locking and see if it works for you!

Comment via email

deps.fnl 0.2.2 released

Sun, 16 Feb 2025 06:30:00 +0300

I just released a new version of the deps.fnl project!

After getting some feedback on the 0.1.0 version and addressing most of it, I’m ready to present a new version with improved dependency and conflict handling.

Major changes

The `deps.fnl` file format

The :deps field was changed back to use maps, as it was during the prototyping stage:

{:deps {"https://gitlab.com/andreyorst/async.fnl"
        {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}
        "https://gitlab.com/andreyorst/json.fnl"
        {:type :git :sha "eebcb40750d6f41ed03fed04373f944dcf297383"}
        "https://gitlab.com/andreyorst/reader.fnl"
        {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}
        "luasocket"
        {:type :rock :version "3.1.0-1"}}
 :paths
 {:fennel ["src/?.fnl" "src/?/init.fnl"]}}

Before this change the deps.fnl file would have looked like this:

{:deps [["https://gitlab.com/andreyorst/async.fnl"
         {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}]
        ["https://gitlab.com/andreyorst/json.fnl"
         {:type :git :sha "eebcb40750d6f41ed03fed04373f944dcf297383"}]
        ["https://gitlab.com/andreyorst/reader.fnl"
         {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}]
        ["luasocket"
         {:type :rock :version "3.1.0-1"}]]
 :paths
 {:fennel ["src/?.fnl" "src/?/init.fnl"]}}

The main reason for a change to the use of vectors was the fact that all of the Luarocks dependencies were using the same tree. Because conflict resolution was made during dependency installation, and maps are unordered, and if any of the libraries had a Luarocks library as its transient dependency, Luarocks would silently override the dependency. This made conflict checking nondeterministic, and thus error-prone.

This is no longer an issue, and thus the format can be changed back to maps.

Conflict resolution

Previously, deps would use a deterministic order of dependencies and the --allow-conflicts flag to resolve conflicts. This meant, that if your project had two dependencies, that had the same transient dependency but of a different version, --allow-conflicts would honor dependencies that appeared earlier in the order. For example, if you had dependencies A, and B, both of which depended on C of versions v1 and v2 like this:

;; project deps.fnl
{:deps [["A" {:version "1"}]
        ["B" {:version "1"}]]}

;; A deps.fnl

{:deps [["C" {:version "1"}]]}

;; B deps.fnl

{:deps [["C" {:version "2"}]]}

The deps script would throw an error:

[A:1]->[C:1] conflicts with [B:1]->[C:2]

Previously, this could be resolved by using the --allow-conflicts flag, meaning that deps would see the conflict, but ignore it, and use the C library from the first one encountered in the deps.fnl. So for the following deps.fnl:

{:deps [["A" {:version "1"}]
        ["B" {:version "1"}]]}

The libraries used would be A version 1, transient C version 1 from A, and B version 1.

However, for the following deps.fnl (A and B appear in the reverse order):

{:deps [["B" {:version "1"}]
        ["A" {:version "1"}]]}

The libraries used would be A version 1, transient C version 2 from B, and B version 1.

This wasn’t clear enough, and could introduce unexpected results when editing deps.fnl.

For example, adding another dependency before A that (unknowingly) also depends on yet another version of C would override the version for the project.

At first, I was thinking about using :exlusions key to specify that we want to exclude certain transient dependencies, but it also introduced the need to do this at arbitrary depth:

{:deps [["A" {:version "1"}]
        ["B" {:version "1"
              :exclusions ["C"]}]]}

And we would still not know what version of C we’re using.

So instead, conflict resolution works by specifying the specific version of dependency C at the same level as A and B:

{:deps {"A" {:version "1"}
        "B" {:version "1"}
        "C" {:version "1"}}}

This way the project author is in full control of what versions will be used.

The --allow-conflicts flag is no more. By default, conflicts are checked, and the only way to resolve conflicts is either to update the library in question, or override it at the root level.

Conflict resolution is now module-based

Another new feature I’m pretty excited about is module-based conflict resolution. In the 0.1.0 version of deps it was not possible to detect conflicts when something is a fork.

What I mean by that is, say, I have a library: async.fnl and someone decides to fork it to change something or fix a bug. They add the following to their project deps.fnl:

{:deps {"https://gitlab.com/some-fork/async.fnl"
        {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}
        "https://gitlab.com/andreyorst/async.fnl"
        {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}}}

The main question thus was - what async.fnl implementation would actually be loaded? We could answer this question if :deps was a list instead of a table because the ordering would rule out and (require :io.gitlab.andreyorst.async) would load the first library. However, with deps stored as a map, we would have gotten different implementations each time because key hashing is randomized.

The problem is - previously deps didn’t see this as a conflict. These two libraries are the same library but different projects.

Now, instead of comparing just library names (URLs in this case) and versions (SHAs for Git dependencies), deps also compares exported modules. So we can detect the conflict because both libraries export io.gitlab.andreyorst.async module. If the person who forked the library chooses to change the module name to io.gitlab.some-fork.async instead, there won’t be a conflict, as this is now a properly different library.

The example above is unlikely to happen as is, but if some library uses a fork as a transient dependency this might happen. This was pointed out to me by Ambrose Bonnaire-Sergeant off the mailing list, and we’ve come up with this solution.

Luarocks (transient) dependencies

Another problem with transient dependencies is the fact that Luaorcks dependency manifests are very lax about versions of the transient dependencies. Usually, you would see just library names, and sometimes version ranges. For example, here’s a fragment from the lapis library manifest describing dependencies:

dependencies = {
  "lua",

  "ansicolors",
  "argparse",
  "date",
  "etlua",
  "loadkit",
  "lpeg",
  "lua-cjson",
  "luaossl",
  "luasocket",
  "pgmoon",
}

The pgmoon dependency itself depends on lpeg. So Luarocks default strategy is to pull the latest available version of the library.

This makes conflict resolution harder, because deps.fnl requires specifying a concrete version of any dependency. Paired with the fact that Luarocks silently overrides already installed dependencies when you give it a specific version to install, and the fact that deps.fnl is now unordered, the only way to get deterministic versions from Luarocks was to ask users to specify all of the transient dependencies:

{:deps {"lapis"
        {:type :rock :version "1.16.0-1"
         :dependencies
         {"loadkit" {:type :rock :version "1.1.0-1"}
          "etlua" {:type :rock :version "1.3.0-1"}
          "luasocket" {:type :rock :version "3.1.0-1"}
          "lua-cjson" {:type :rock :version "2.1.0.10-1"}
          "argparse" {:type :rock :version "0.7.1-1"}
          "lpeg" {:type :rock :version "1.1.0-2"}
          "ansicolors" {:type :rock :version "1.0.2-3"}
          "date" {:type :rock :version "2.2.1-1"}
          "pgmoon" {:type :rock :version "1.16.0-1"
                    :dependencies {"lpeg" {:type :rock :version "1.1.0-2"}}}
          "luaossl" {:type :rock :version "20220711-0"}}}}}

The :dependencies key is akin to the dependencies map in the rock manifest.

The conflict resolution works exactly the same.

Runtime library manipulation with `deps.add-lib`, and `sync-deps`

Something that escaped my previous announcement, because it was developed a bit later is that it is now possible to add libraries while working on the application without restarting the REPL. When the REPL is started via the deps --repl command, a new deps module is available:

>> (require :deps)
{:_VERSION "0.2.0"
 :add-lib #<function: 0x560a243c2990>
 :add-libs #<function: 0x560a245b5820>
 :sync-deps #<function: 0x560a246d8d50>}

Thus, if you’re working on some project and in need of a new dependency, or want to compare a few and choose which one to use, you can quickly do so in the REPL:

>> (deps.add-libs {"bump" {:type :rock :version "3.1.7-1"}
                   "anim8" {:type :rock :version "v2.3.1-1"}})
["bump" "anim8"]
>> (deps.add-lib
    "penlight"
    {:type :rock :version "1.14.0-2"
     :dependencies {"luafilesystem" {:type :rock :version "1.8.0-1"}}})
["pl.input" "pl.path" "pl.array2d" "pl.compat" "pl.luabalanced" "pl.pretty"
 "pl.import_into" "pl.dir" "pl.func" "pl.Map" "pl.comprehension" "pl.app" "pl.seq"
 "pl.sip" "pl.file" "pl.stringx" "pl.class" "pl.Date" "pl.strict" "pl.template"
 "pl.types" "pl.permute" "pl.MultiMap" "pl.text" "pl.utils" "pl.OrderedMap" "pl.xml"
 "pl.url" "pl.tablex" "pl.test" "pl.lapp" "pl.List" "pl.Set" "pl.operator" "pl.data"
 "pl.lexer" "pl.config" "pl.init" "pl.stringio" "lfs"]

The returned value is a table of module names that are now available to require.

Alternatively, the deps.fnl file can be edited, and changes can be synced into the running REPL via the sync-deps call.

New flags

Several new flags were added.

`--tree`

Prints the dependency tree, outlining what dependencies are on your path, and what transient dependencies are brought by each dependency:

[["lapis" "1.16.0-1"
  [["argparse" "0.7.1-1"]
   ["luaossl" "20220711-0"]
   ["pgmoon" "1.16.0-1"
    [["lpeg" "1.1.0-2"]]]
   ["date" "2.2.1-1"]
   ["luasocket" "3.1.0-1"]
   ["loadkit" "1.1.0-1"]
   ["ansicolors" "1.0.2-3"]
   ["lpeg" "1.1.0-2"]
   ["etlua" "1.3.0-1"]
   ["lua-cjson" "2.1.0.10-1"]]]
 ["andreyorst/fennel-test"
  "9aae4dd52be1f5dd1ee97e61c639e6c5a2f9afee"]]

`--fennel-ls`

The deps script is now able to create or update an existing configuration for the fennel-ls project.

When calling deps --fennel-ls a flsproject.fnl file will be created if it doesn’t exist already. This file will contain all of the PATH-related data:

{:fennel-path "test/?.fnl;.deps/git/andreyorst/fennel-test/9aae4dd52be1f5dd1ee97e61c639e6c5a2f9afee/src/?/init.fnl"
 :lua-version "lua54"
 :macro-path ".deps/git/andreyorst/fennel-test/9aae4dd52be1f5dd1ee97e61c639e6c5a2f9afee/src/?/init.fnlm"}

`--profiles`

Another new feature - profiles! Before this flag existed profiles were kind of possible by using --merge and additional deps.fnl files:

;; deps.fnl
{:deps {"https://gitlab.com/andreyorst/reader.fnl"
        {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}}
 :paths
 {:fennel ["src/?.fnl"]}}

;; dev-deps.fnl
{:deps {"https://gitlab.com/andreyorst/fennel-test"
        {:type :git :sha "72a74394f89fdaf15abf11f33752a07e9e0bee91"}}
 :paths {:fennel ["tests/?.fnl"]}}

Having these two files, a final deps map can be obtained with deps --merge dev-deps.fnl:

;; output of `deps --merge dev-deps.fnl --show`
{:deps {"https://gitlab.com/andreyorst/fennel-test"
        {:sha "72a74394f89fdaf15abf11f33752a07e9e0bee91" :type "git"}
        "https://gitlab.com/andreyorst/reader.fnl"
        {:sha "252ea2474cb7399020e6922f700a5190373e6f98" :type "git"}}
 :paths {:clua {} :fennel ["src/?.fnl" "tests/?.fnl"] :lua {} :macro {}}}

This works, but semantically it is a bit unclear. To know that the development “profile” exists you need to actively look for it in the project files. Multiple such profiles also pile up in the project, which is not great.

Now, this can be done a bit more consistently with:

;; deps.fnl
{:deps
 {"https://gitlab.com/andreyorst/reader.fnl"
  {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}}

 :paths
 {:fennel ["src/?.fnl"]}

 :profiles
 {:dev
  {:deps {"https://gitlab.com/andreyorst/fennel-test"
          {:type :git :sha "72a74394f89fdaf15abf11f33752a07e9e0bee91"}}
   :paths {:fennel ["tests/?.fnl"]}}}}

And the same final deps map can be obtained with deps --profiles dev:

;; output of `deps --profiles dev --show`
{:deps {"https://gitlab.com/andreyorst/fennel-test"
        {:sha "72a74394f89fdaf15abf11f33752a07e9e0bee91" :type "git"}
        "https://gitlab.com/andreyorst/reader.fnl"
        {:sha "252ea2474cb7399020e6922f700a5190373e6f98" :type "git"}}
 :paths {:clua {} :fennel ["src/?.fnl" "tests/?.fnl"] :lua {} :macro {}}}

The --merge flag is here to stay for compatibility. Maybe there are other use cases for it too, so it won’t be removed unless deemed unnecessary. Moreover, --profiles actually supports multiple comma-separated profile names, so it is much more compact compared to --merge:

$ deps --profiles dev,test,doc ...
$ deps --merge dev-deps.fnl --merge test-deps.fnl --merge doc-deps.fnl ...

Upcoming work

I’ll probably take a short break from working on deps for a while, or at least I won’t work on it as actively. Both to let it settle a bit, give it time to gather some feedback, and let me rest a bit too. I’m feeling good about what I did so far, and deps in my opinion is out of the alpha stage for sure. So I think it can be safely used in other projects, and I’m still open for feedback!

As for plans, I have something I want to experiment with.

I’ve started outlining an interface for custom backend implementations. Right now there are only two backends: Git and Luarocks, but if the community desires, I’ll extend deps in such a way that custom backends could be added via some sort of a plugin system. Thus other ways to download and add dependencies would be possible. For example, recently I learned that some Lua libraries exist only in SVN. There may be others that are only available through GNU Bazaar, the new hot Jujutsu VCS, or maybe some use Mercurial, I don’t know!

Additionally, I think it is safe to say, that using version control systems for libraries is not a great idea, so I might start working on a dependency packaging system for Fennel libraries that we could later upload to some package archive. Thinking of a name - Fennel Bulbs sounds fun!

Dependency building needs to be changed to be a bit more robust. Right now I think it should work fine, but probably only on *nix-based systems. Windows support for deps.fnl is there, but completely untested. So if any Fennel users happen to do their stuff on Windows - let me know if it works, please!

That’s all for now. Huge thanks to Phil Hagelberg, Ambrose Bonnaire-Sergeant, and Emma B. for the feedback and code reviews!

Comment via email

deps.fnl - a new dependency manager for Fennel projects

Fri, 10 Jan 2025 00:47:00 +0300

I’d like to present a new project, aimed at one of the areas where the Fennel ecosystem can be improved - project dependency management.

Other programming languages have tools to pull in, build, and load external dependencies. Java has Maven, Clojure has Leiningen and other projects like this, Python has Pip and Poetry, and Lua has Luarocks. All these tools provide ways for pulling in dependencies concerning how the language is organized.

Since Fennel runs on Lua, we could use Luarocks, but, well, it’s not really good. It has a lot of design problems and doesn’t fit for Fennel for a number of reasons, like:

It is tailored for Lua dependencies only, so shipping Fennel libraries would require AOT compiling them to Lua;
Luarocks only knows how to build LUA_PATH and LUA_CPATH, but Fennel requires FENNEL_PATH and more importantly FENNEL_MACRO_PATH;
Shipping macros is hard or impossible.

A few notes on why I decided to make a dependency manager

I’ve made a lot of libraries for Fennel and encountered various problems with shipping these libraries to users.

Most of the Fennel libraries I write feature user-exposed macros, which means that the library can’t be shipped as a single file most of the time. I did find a workaround for that eventually, but it still poses some interesting challenges for library writers.

Another problem I encountered are libraries that need other libraries to work. Some of my libraries depend on other libraries I made, and thus I either need to include them in the source code directly for shipping or do some other trickery. And libraries depending on other libraries (transient dependencies) could potentially conflict with other dependencies in the project.

Finally, multifile libraries that can’t be compiled into a single file, since they have multiple modules that serve as entry points are hard to ship because users will have to properly set up PATH in their project.

So I made deps.fnl to solve these problems.

`deps.fnl`

This project consists of a single script, called deps, that works as a drop-in replacement for the fennel executable. This means that you can call deps with any arguments that you specify to Fennel, and it will work transparently. In other words:

deps --require-as-include --compile file.fnl > file.lua

But the main thing that deps was designed to do is to set up the environment for your project. When the deps.fnl file is present in the project, the deps script analyzes this file and pulls in all of the specified libraries, sets all required PATH variables, like LUA_PATH, LUA_CPATH, FENNEL_PATH, and FENNEL_MACRO_PATH, and launches fennel.

For example, here’s a deps.fnl for my fnl-http project:

{:deps
 [["https://gitlab.com/andreyorst/async.fnl"
   {:type "git" :sha "a83b13b397fdfab3ebf7f17a16d21a7ec2674ceb"}]
  ["https://gitlab.com/andreyorst/json.fnl"
   {:type "git" :sha "bfb8d7d03c26619768eefaa47782a3e95dbe5156"}]
  ["https://gitlab.com/andreyorst/reader.fnl"
   {:type "git" :sha "3ff2bc790c8b7922267af5712a64a14572a172cf"}]
  ["luasocket"
   {:type "rock" :version "3.0rc1-2"}]]
 :paths
 {:fennel ["src/?.fnl" "src/?/init.fnl"]}}

As can be seen, this project depends on four libraries: async.fnl, json.fnl, reader.fnl, and luasocket. Previously I had to vendor these dependencies in the repository, and instruct people how to install fnl-http and use it as a library.

Moreover, fnl-http is a multi-file library, so manually setting up PATH for it and all of its dependencies is not a pleasant task. But with deps you can use this library in your project by placing this deps.fnl file at your project root:

{:deps [["https://gitlab.com/andreyorst/fnl-http"
         {:type "git" :sha "ea885a8c767a627206126cc36f1befb1a2d71850"}]]
 :paths {:fennel ["?.fnl"]}}

Then just call deps --repl, and you’re good to go:

$ deps --repl
processing git repo: https://gitlab.com/andreyorst/fnl-http
processing git repo: https://gitlab.com/andreyorst/async.fnl
processing git repo: https://gitlab.com/andreyorst/json.fnl
processing git repo: https://gitlab.com/andreyorst/reader.fnl
processing rock: luasocket 3.0rc1-2
Welcome to Fennel 1.5.1-dev on PUC Lua 5.4!
Use ,help to see available commands.
>> (require :io.gitlab.andreyorst.fnl-http.client)
{:connect #<function: 0x5605682104e0>
 :delete #<function: 0x560568bc64c0>
 :get #<function: 0x5605685315c0>
 :head #<function: 0x5605689c9ca0>
 :options #<function: 0x5605682b0a20>
 :patch #<function: 0x5605687d8f40>
 :post #<function: 0x560568c40270>
 :put #<function: 0x5605681d9d80>
 :request #<function: 0x560568b37450>
 :trace #<function: 0x560568580320>}

In the output above after analyzing the project’s deps.fnl the deps script proceeds to download dependencies. The project’s only dependency is fnl-http, but after downloading it, the deps script check if there’s a deps.fnl in the root of the repository. Since there is, it proceeds to download all transient dependencies. Calling deps --repl the second time would not download anything, unless new dependencies were added to the deps.fnl file or hashes/version were changed.

After processing all of the dependencies the REPL is started as requested, and the library is ready to be used.

Dependency types

Dependencies can be of two types - git and rock.

Git dependencies are just a link to the repository and a commit hash to checkout. Until we have some kind of a registry and a way to package libraries for downloading this is the best we can do. But as seen above, this already makes using libraries a lot simpler.

Rock dependencies are processed with Luarocks. Yes, deps wraps not only the Fennel executable but Luarocks as well. Luarocks is fine enough to be used like this and provides enough command-line arguments for deps needs.

All of the dependencies are stored in the project’s root unless requested elsewhere. I decided not to go the .m2 route. One reason for that is the fact, that Luarocks can’t install two versions of the same library into a single “tree”, as they call it. Instead, it overrides any library already present in the tree. This is probably related to how the LUA_PATH is constructed by Luarocks.

Speaking of PATH=⁣s. The =deps.fnl file contains a mandatory field :paths, that contains a table. This table specifies the project’s path configuration. Each library may require a different path configuration, and when deps script processes a said deps.fnl file, it reads the path table and constructs the final PATH for your project.

You can analyze the path generated for the project with deps --path. It prints the path to standard out in a format similar to luarocks path.

Finally, some Git repositories may not use deps.fnl, but it is still possible to manage them with deps.fnl. For example, my library fennel-test doesn’t yet feature a deps.fnl file, but the deps script itself uses it for testing. To make this work, you can specify :paths field for each dependency:

{:deps [["https://gitlab.com/andreyorst/fennel-test"
         {:type :git
          :sha "01ea080dc8176512c5890462252c5bf214baf137"
          :paths {:fennel ["?.fnl"] :macro ["?.fnl"]}}]]
 :paths {}}

Thus, we can even pull in pure Lua dependencies, and specify the :paths table with something like {:lua ["?.lua"]}, and deps will set everything up.

You can read more information on the deps.fnl file format, and deps script usage in the project’s readme.

I need feedback

I have high hopes for this project, and I’m looking for feedback. If you’re working on some application that requires external dependencies, give deps.fnl a shot. If you’re a library maker like me, consider using deps.fnl so your users have an easier way of using your work. You can submit feedback either via email, or GitLab issues.

If the project gets enough traction and is to community’s satisfaction, Phil and I are discussing the possibility of including this into Fennel itself, so no installation would be required. Since deps is already a drop-in replacement for fennel, once deps is a part of the fennel executable, all commands from the above would work with just fennel itself.

Thanks for reading, and I hope this project will be useful to you!

Comment via email

Extending Emacs with Fennel

Fri, 20 Dec 2024 03:23:00 +0300

After watching this year’s EmacsConf and seeing Guile Emacs being resurrected I thought to myself - why limit ourselves to Guile? Sure, Guile isn’t just a Scheme implementation, thanks to its compiler-tower-based design. Other languages exist for Guile VM, such as Emacs Lisp, and Guile manual lists the following languages with various stages of completeness:

ECMAScript
Brainfuck
Lua
Ruby
Python

Sure, it would be nice, if Emacs could natively run all of these, but we have to understand, that Guile Lua is not PUC Lua. Even LuaJIT has difficulties maintaining full compatibility with modern releases of Lua, having us stuck in the realm somewhere between Lua 5.1 and 5.2, while PUC Lua is already 5.4. Future releases would bring more differences, making PUC Lua and Luajit diverge even more.

And Lua is a simple language, unlike something like JS, Ruby, or Python. So I wouldn’t bet on the fact that Guile Ruby would be exactly the same as Ruby. Maybe it will, but it’ll probably take years.

So I thought, why not just use Lua as is? The only thing we can’t do is to run Fennel functions from Emacs Lisp. There were attempts at bringing Lua VM into Emacs, running in the same process, however, as I know, it isn’t stable enough.

However, Emacs can connect to a Fennel REPL, like it does for many other languages, such as Common Lisp or Clojure. Sure, the language is running in a separate process, and this comes with a lot of nuances, but it’s still a possible route.

So I made a small package: require-fennel.el. It’s capable of loading Fennel (or Lua) modules, and defining a set of functions on the Emacs side for every Fennel function in the module.

Running Fennel from Emacs Lisp

For example, we can load Fennel itself into Emacs like this:

(require-fennel fennel)

Now we have all sorts of functions available, for example, here’s fennel.eval:

(fennel.eval "(fcollect [i 1 10] (* i i))")
;; => [1 4 9 16 25 36 49 64 81 100]

Or, we can create a file greet.fnl:

(fn greet [name]
  "Greets NAME with a message."
  (print (.. "Hello, " name " from Fennel!")))

and load it:

(require-fennel greet)

(greet "Andrey")
(require-fennel foo)
(require-fennel vec-add)

Code Snippet 1: "Hello, Andrey from Fennel!" will be displayed in the echo area of Emacs

Let’s ask Emacs to describe the greet function:

greet is a interpreted-function.

(greet NAME)

Greets NAME with a message.

As can be seen, Emacs can show us the function signature and its documentation string obtained from Fennel.

Unlike Emacs Lisp, Fennel functions support destructuring with special syntax based on how data literals are written in Fennel. For example, let’s create a function that accepts a map, and destructures its keys:

(fn foo [{:x x :y y}]
  "Accepts a map with the X and Y keys, and adds them together."
  (+ x y))

After loading it in Emacs we’ll see the following documentation:

foo is a interpreted-function.

(foo ((:y . y) (:x . x)))

Accepts a map with the X and Y keys, and adds them together.

I took some creative liberties and made it so Fennel hash tables are represented as association lists in Emacs Lisp. So the ((:x . 1) (:y . 2)) in Elisp is equal to {:x 1 :y 2} in Fennel. The argument list on the Emacs Lisp side is a bit unconventional, but I think that’s OK.

The same goes for vector destructuring:

(fn vec-add [[x0 y0] [x1 y1]]
  "Adds two 2D vectors."
  [(+ x0 x1) (+ y0 y1)])

Emacs’ documentation uses lists:

vec-add is a interpreted-function.

(vec-add (X0 y0) (X1 y1))

Adds two 2D vectors.

I couldn’t make it work with vectors for some reason, and as can be seen, Emacs tries to upcase parameters, but it doesn’t recognize that y0 is also a parameter. Probably fine, as such documentation would never be generated for Emacs Lisp functions.

We can call such a function with both vectors and lists:

(vec-add [1 2] [1 2])   ;=> [2 4]
(vec-add '(1 2) '(1 2)) ;=> [2 4]

That’s because Fennel doesn’t have a list datatype, it only has associative and sequential tables. Thus I’m using Elisp vectors as the result type.

Data conversion

Here’s how conversion table when passing data from Elisp to Fennel:

Elisp	Fennel
`[1 2 3]`	`[1 2 3]`
`(1 2 3)`	`[1 2 3]`
`((:foo . 1) (:bar . 2))`	`{:foo 1 :bar 2}`
`#s(hash-table test equal data ("foo" 1 "bar" 2))`	`{:foo 1 :bar 2}`
`:foo`	`"foo"`
`'false`	`false`
`"foo"`	`"foo"`

And here’s the conversion table when receiving data from Fennel in Emacs:

Fennel	Elisp
`[1 2 3]`	`[1 2 3]`
`{:foo 1 :bar 2}`	`((:foo . 1) (:bar . 2))`
`(values 1 2 3)`	`(1 2 3)`
`(fn [])`	`(lambda (&rest args) ...)`
`userdata`	`#<udata: 0x55b66bd6ac28>`
`true`	`t`
`nil`, `false`	`nil`

A small note about hash tables and multiple value returns. As can be seen, both are returned as lists. I specifically chose the cons notation for association lists, i.e. (a . b) because this way we can tell apart hash tables and mustivalues. There are no cons cells in Fennel, so no other data structure will be represented like that.

Because multiple values are returned as lists, we have to call apply, if we wish to compose two Fennel functions. In other words, in Fennel we can do this:

(fn rest [_ ...] ...)

(fn first [x _] x)

(first (rest 1 2 3)) ;=> 2

{: first : rest}

Code Snippet 2: fr.fnl

In Elisp, however, we would have to do:

(require-fennel fr)

(apply #'fr.first (fr.rest 1 2 3)) ;=> 2

If we were to call it as (fr.first (fr.rest 1 2 3)) the result would be [2 3], because fr.rest returned a list of return values, and lists are converted to vectors or hash maps depending on their structure.

That’s the basic gist of this package.

More examples

After implementing it I was like a kid in a toy store. With a childish grin, I started loading all my crazy Fennel libraries that I made over the years.

For example, here’s cljlib:

(require-fennel cljlib)

(cljlib.conj [1] 2) ;=> [1 2]

Or, here’s a JSON parser I made:

(require fennel json)

(json.decode "{\"foo\": [1, 2, 3]}")
;; (("foo" . [1 2 3]))

(json.encode '(("foo" . [1 2 3])))
;; "{\"foo\": [1, 2, 3]}"

Here’s a crazy one - I’m loading my Fennel port of a Clojure library clj-http into Emacs, and calling it from Elisp:

(require-fennel io.gitlab.andreyorst.fnl-http)
(require-fennel io.gitlab.andreyorst.fnl-http.client :as client) ; nested module

(client.get "http://httpbin.org/get" '((:as . :json)))

;; => (("headers" . (("Content-Length" . "198")
;;                   ("Access-Control-Allow-Credentials" . t)
;;                   ("Access-Control-Allow-Origin" . "*")
;;                   ("Content-Type" . "application/json")
;;                   ("Connection" . "keep-alive")
;;                   ("Server" . "gunicorn/19.9.0")
;;                   ("Date" . "Mon, 16 Dec 2024 22:56:35 GMT")))
;;     ("reason-phrase" . "OK")
;;     ("length" . 198)
;;     ("protocol-version" . (("name" . "HTTP")
;;                            ("minor" . 1)
;;                            ("major" . 1)))
;;     ("request-time" . 214)
;;     ("trace-redirects" . [])
;;     ("status" . 200)
;;     ("body" . (("args" . [])
;;                ("headers" . (("Host" . "httpbin.org")
;;                              ("X-Amzn-Trace-Id" . "Root=1-6760b023-39ca8e413573df5d14d09486")))
;;                ("url" . "http://httpbin.org/get"))))

Code Snippet 3: I've formatted nested alists to use dot notation for better reading clarity. In other words changed (a (b . c) (d . e)) into (a . ((b . c) (d . e))) which are the same thing.

The client made an HTTP request from the Fennel side and passed the response back to Emacs. The response body was in JSON, but specifying ((:as . :json)) as options map forced parsing the response body into Fennel tables. Which, upon arriving in Emacs Lisp were converted into association lists.

So now we can call Fennel from Elisp. But why stop there? Let’s call Emacs Lisp from Fennel!

Running ELisp from Fennel?..

Since this package uses the Fennel Proto REPL, which is a custom protocol that can be extended at runtime, we can create a set of functions that will allow us to call back to Elisp. Here’s how we extend the protocol:

(let [format-elisp _G.___repl___.pp]
  (protocol.env-set!
   :emacs {:call (->> {:__index
                       (fn [_ k]
                         (fn [...]
                           (protocol.message
                            [[:id {:sym protocol.id}]
                             [:op {:string :require-fennel/call}]
                             [:fun {:sym k}]
                             [:arguments {:list (fcollect [n 1 (select "#" ...)]
                                                  (format-elisp (pick-values 1 (select n ...))))}]])
                           (. (protocol.receive) :data)))}
                      (setmetatable {}))
           :var (->> {:__index
                      (fn [_ k]
                        (protocol.message [[:id {:sym protocol.id}]
                                           [:op {:string :require-fennel/var}]
                                           [:var {:sym k}]])
                        (. (protocol.receive) :data))}
                     (setmetatable {}))
           :eval (fn [expr]
                   (protocol.message [[:id {:sym protocol.id}]
                                      [:op {:string :require-fennel/eval}]
                                      [:expr {:sym expr}]])
                   (. (protocol.receive) :data))})
  {:id 1000 :nop ""})

I had to update the protocol core a bit, adding the protocol.env-set! and protocol.receive functions. The env-set! allows us to create user-visible definitions. So we add the emacs table that holds three fields.

The call field is a table with some metatable trickery. Since it’s an empty table, any key would trigger the __index metamethod. This metamethod can be another table or a function - in our case, it is the latter. This function returns a handler that calls into protocol.message that sends a custom OP require-fennel/call that we can handle in our package.

The same goes for the var field, except it doesn’t return a function, it just returns a value.

The last field, eval, is a combination of both, kinda. It is a function, not a table, and it accepts one argument - an expression to evaluate in Emacs.

However, we can’t use these just yet. The second part of the trick comes from the new ability to extend the main protocol handler with custom handlers:

(cl-defgeneric fennel-proto-repl-handle-custom-op (_op _message _callbacks)
  "Handler for a custom protocol OP.
The first argument OP is used for dispatching.  Accepts the whole
MESSAGE, and its CALLBACKS.  The default implementation of unknown OP is
a nop."
  nil)

With this generic function, we can define new handlers in separate packages. These are the methods I’ve added in require-fennel.el:

(cl-defmethod fennel-proto-repl-handle-custom-op ((op (eql :require-fennel/call)) message callbacks)
  "Custom handler for the require-fennel/call OP.
Accepts a MESSAGE and its CALLBACKS.
The MESSAGE contains a function to evaluate and its arguments."
  (fennel-proto-repl-send-message
   nil
   (format "{:id %s :data %s}"
           (plist-get message :id)
           (apply (plist-get message :fun) (plist-get message :arguments)))
   nil))

(cl-defmethod fennel-proto-repl-handle-custom-op ((op (eql :require-fennel/var)) message callbacks)
  "Custom handler for the require-fennel/var OP.
Accepts a MESSAGE and its CALLBACKS.
The MESSAGE contains a var which value is then returned to Fennel."
  (fennel-proto-repl-send-message
   nil
   (format "{:id %s :data %s}"
           (plist-get message :id)
           (require-fennel--elisp-to-fennel (eval (plist-get message :var))))
   nil))

(cl-defmethod fennel-proto-repl-handle-custom-op ((op (eql :require-fennel/eval)) message callbacks)
  "Custom handler for the require-fennel/var OP.
Accepts a MESSAGE and its CALLBACKS.
The MESSAGE contains an expression which is evaluated and its result is returned to Fennel."
  (fennel-proto-repl-send-message
   nil
   (format "{:id %s :data %s}"
           (plist-get message :id)
           (require-fennel--elisp-to-fennel (eval (plist-get message :expr))))
   nil))

These three fields and methods allow us to do anything with Emacs, and it happens in the same process, so we can access the full Emacs state. For example:

;; Welcome to Fennel Proto REPL 0.6.0
;; Fennel version: 1.5.1-dev
;; Lua version: PUC Lua 5.4
>> (emacs.call.emacs-uptime)
"19 minutes, 17 seconds"
>> emacs.var.emacs-version
"31.0.50"
>> (emacs.eval "(with-current-buffer \" *fennel-elisp*\" (buffer-substring-no-properties (point-min) (point-max)))")
";; Welcome to Fennel Proto REPL 0.6.0
;; Fennel version: 1.5.1-dev
;; Lua version: PUC Lua 5.4
>> (emacs.call.emacs-uptime)
\"19 minutes, 17 seconds\"
>> emacs.var.emacs-version
\"31.0.50\"
>> (emacs.eval \"(with-current-buffer \\\" *fennel-elisp*\\\" (buffer-substring-no-properties (point-min) (point-max)))\")
>> "
>>

Crazy!

Fennelmacs

So now we can write Fennel scripts that can obtain data from Emacs, process it, and pass back the results. Thus, our goal is achieved - we’ve added a second language to Emacs!

Of course, it’s not the same as what Guile Emacs does. We’re still running code in a separate process, adding communication between the two via some protocol. It’s a neat trick, nothing more. Guile Emacs, if it ever becomes the thing, will be more powerful, as we would be able to leverage all these different languages in the same process, possibly sharing the memory, calling functions interchangeably, etc. Some may even write Emacs packages in Brainfuck, who knows.

Still, this is really cool, in my opinion, and shows the power of Emacs pretty well. If you’re into Fennel, like me, I hope you’ll enjoy using this package as I enjoyed making it. See you at Fennel Conf!

Comment via email

Extending JSON encoder with custom types

Mon, 04 Nov 2024 06:39:00 +0300

One thing I like about Lua is that it has a limited amount of built-in types and data structures. There are no objects, just tables. And if you need a custom behavior, a metatable can be attached to any table to extend it with custom methods that integrate with the rest of Lua runtime.

However, this is both a blessing and a curse. And unfortunately, Lua developers made a huge mistake by breaking this API¹ once already, in a minor release even.

As you may noticed, fnl-http has a lot of stuff. It has asynchronous Channels, and various Readers, which act like objects, but are still ordinary tables underneath. And this is true for a lot of Lua projects, tables are flexible enough to implement many cool things, so people do.

However, this poses an interesting problem - because objects are not a thing in Lua, and a set of default metamethods is quite limited, there’s no default way of extending objects with custom behavior. In Java, you would specify what interfaces are required for an object to have, and it often means that if something doesn’t implement them, a wrapper object is needed. In Clojure, we can get away with protocols and multimethods, but Lua has neither of those. In Lua, and subsequently Fennel, we have to be creative, because even though we can add stuff through metatables, not every value can have one.

While working on fnl-http, I added a basic JSON parser and encoder. The parser part is quite simple, as JSON has a specification and a fixed amount of data types to worry about. It’s not like EDN or XML, for example, which are both extensible data notations that can represent pretty much anything. In JSON you get only these data types:

Number
String
Array
Object
Boolean
Null

The only difference from Lua here is that null is nil and Lua doesn’t have arrays as a separate type. So implementing a parser was easy, and everything pretty much maps to Lua data structures one-to-one.

Implementing JSON encoder, however, was not as easy. For basic data types, it is easy enough, but once we touch custom objects created with metatables things get messy.

For example, let’s say we need a set data structure. A set is similar to a table, except the values are the same as the keys. But, we may want to distinguish them from ordinary tables, if, say, we’re making a library that has a set of functions for working with mathematical sets. Like intersection, difference, etc. Here’s a set constructor:

(fn Set [...]
  (let [data (faccumulate [res {}
                           i 1 (select :# ...)]
               (let [k (select i ...)]
                 (doto res (tset k k))))]
    (setmetatable
     {}
     {:__index data
      :__newindex (fn [_ k _] (tset data k k))})))

As you can see, we create an empty table and set its metatable to a bunch of functions that close over the local data. This technique is quite common in Lua and is called proxy tables. By using the empty table as a base, we ensure that any table access triggers the __index or __newindex metamethod, depending on the action.

However, there’s a problem. If we try to print such an object in the REPL, we don’t get any information about it, even though it has values:

>> (local s (Set :a :b :c))
nil
>> (. s :a)
"a"
>> (set (. s :d) 42)
nil
>> (. s :d)
"d"
>> s
{}

Additionally to that, we have no means of iteration over this data structure, as the default next function will just return nil:

>> (next s)
nil

We can fix this by adding a __pairs metamethod:

(fn Set [...]
  (let [data (faccumulate [res {}
                           i 1 (select :# ...)]
               (let [k (select i ...)]
                 (doto res (tset k k))))]
    (setmetatable
     {}
     {:__index data
      :__newindex (fn [_ k _] (tset data k k))
      :__pairs (fn [_] #(pick-values 1 (next data $2)))})))

Now, printing works, albeit in a bit strange way:

>> (Set :a :b :c)
{:a nil :b nil :c nil}

A few years ago, I remade the pretty-printer for Fennel, and support for custom objects was a huge part of its design. In short, we can fix our object textual representation by using the __fennelview metamethod.

Now, __fennelview is fine and all, but it is not exactly meant for extending existing objects. Of course, given a Lua library, that implements these proxy tables you can add support for Fennel, but it’s more of an ad-hoc solution. So it’s meant for library authors to make the Fennel REPL experience more user-friendly, and make sure that the printed representation can be read back by Fennel.

And it’s not always possible to add __fennelview to existing objects. For example, one common complaint in Lua is that arrays start from 1 and not 0. We can “fix” that by implementing our own Array type:

(fn Array [...]
  (let [vals [...]]
    (setmetatable
     []
     {:__index (fn [_ i]
                 (. vals (+ i 1)))
      :__newindex (fn [i val]
                    (tset vals (- i 1) val))
      :__len #(length vals)
      :__pairs (fn [_] #(next vals $2))})))

Right now, the object above prints like this:

>> (Array 1 2 3)
[1 2 3]

Let’s assume this came from the library, and we wish to print this differently, e.g. as (Array 1 2 3) instead of [1 2 3] to make sure that such arrays are read back as 0-based arrays, and not as ordinary Lua tables. We can do that, but it is tricky because we don’t have any means of accessing the data closure:

(fn wrap [arr]
  (tset (getmetatable arr)
      :__fennelview
    (fn pp-doc-example [t view options indent]
      (let [lines (icollect [i v (pairs t)]
                    (let [v (view v options (+ 7 indent))]
                      (if (= i 1) v
                          (.. "       " v))))]
        (doto lines
          (tset 1 (.. "(Array " (or (. lines 1) "")))
          (tset (length lines) (.. (. lines (length lines)) ")"))))))
  arr)

However, as you can see, we have to use wrap on every Array, because each instance comes with its own metatable, and setting it once wouldn’t work. And if we try to read back the printed representation we’ll get the old representation:

>> (wrap (Array 1 2 3))
(Array 1 2 3)
>> (Array 1 2 3)
[1 2 3]

So we can make another wrapper:

(local Array* Array)

(fn Array [...]
  (wrap (Array* ...)))

>> (Array 1 2 3)
(Array 1 2 3)

So you can see, how this is a bit problematic.

Custom JSON encoders

Now, this was a bit of a tangent, so let’s get back to the main topic - JSON encoding. Because I can’t ask everyone to do this kind of wrapping, I decided to implement a different way of extending the json.encode function. Thus, the library now features two more functions: register-encoder and unregister-encoder. The second one is for fixing mistakes, and won’t probably be used too much.

So, let’s see how we can add support for custom types. First, we need a function that will tell if the given object is an instance of something we want to encode:

(fn Array? [x]
  (and (= :table (type x))
       (not= nil (. x 0))
       Array))

A library may provide a custom predicate to check if the given object is of the same type, like chan? for asynchronous channels. However, it can’t be used as is, because to register an encoder, such a function needs to return a singleton, that can be compared with =. So we return Array from our example function.

Next, we can implement the encoder:

(fn encode-array [arr encode]
  (.. "["
      (-> (fcollect [i 0 (- (length arr) 1)]
            (encode (. arr i)))
          (table.concat ", "))
      "]"))

Finally, we can register it:

(json.register-encoder object object? encode-object)

The first argument is the object itself, and the second one is a function to obtain the object’s type. Internally, register-encoder calls the object? function on the first argument to provide a bit more flexibility, in case object? returns different values for different objects or based on the object’s properties. The third argument is a function that provides the encoding. It accepts a callback to encode nested values as a second argument.

With that, we can register the encoder and encode our array as JSON by calling (json (Array 1 2 3)):

>> (json.register-encoder (Array) Array? encode-array)
nil
>> (json (Array 1 2 3))
"[1, 2, 3]"

This should provide enough flexibility to support any proxy-based objects.

Custom decoders

I decided not to add custom decoder support to this library, yet. JSON doesn’t have any custom types, so it doesn’t make sense to parse its values to anything but default Lua types.

One could argue, that it would be nice to have a way to parse arrays to zero-based ones, given that we can convert zero-based arrays to JSON. While true, I think this only adds unnecessary complexity to the parser.

One reason, registering an encoder requires a singleton as type designator is to ensure that objects can’t be mixed up. If it was allowed to register encoders via string types, it would be possible to register two "Array" types, making ambiguous what encoder would be used or would require to change the name in case of conflict.

JSON has a fixed amount of types, so registering a decoder for, say, Array, would mean that all arrays would be parsed to this custom type in all of the project parts. Meaning, that if you use a different library, that also uses this JSON parser, and internally expects one-based arrays it will break.

I’m open to changing my mind on this, but for now, I feel that custom encoders are sufficient enough to make this library more usable. As a part of this change, I’m moving this library out from the fnl-http repo into its own library, making fnl-http depend on it instead, making it possible for other projects to use this parser more easily. You can find the code here.

Thanks for reading, and I hope this library will be useful!

Foontones

The deprecation of __ipairs metamethod in Lua 5.3 resulted in an inability to implement custom iteration over custom sequential data structures. For example, the zero-indexed array example would benefit from __ipairs because we could specify that the iteration starts from 0. With the default ipairs we would always miss the first value. It’s not hard to fix manually, by overriding ipairs with a custom function that checks for __ipairs metamethod:
```
       (local ipairs* ipairs)
       (fn _G.ipairs [tbl ...]
         (case (getmetatable tbl)
           {:__ipairs ip} (ip tbl ...)
           _ (ipairs* tbl ...)))
```
However, it’s not practical to intrude into Lua’s standard library in such a way. ↩︎

Comment via email

Streaming and receiving chunked multipart data with fnl-http

Sat, 02 Nov 2024 04:46:00 +0300

I’ve been absent for a while - you may have noticed that compared to the previous year, I posted a lot less this time. There are two closely related reasons for that. First, I felt burned out from programming. Second, I finally picked up a guitar after five or more years and started recording again. Thus, now I only do hobby programming when I feel passionate or challenged in an entertaining way. This works for me, but I have less material to write about.

One such passionate project for me this year is fnl-http. It’s an HTTP/1.1 client and server implementation for Fennel made using luasocket and async.fnl - another library of mine.

Last time I posted, I was mostly talking about testing. And I’m glad that I wrote all these tests, because the change I’m going to talk about this time broke a lot of them, and I was able to diagnose problems much quicker. But the main changes are related to the post I made before that: fnl-http Improvements. In that post, the most exciting things for me were chunked transfer encoding and support for multipart requests. I’ve implemented support for reading chunked responses (the client already could send chunked requests) and moved onto implementing multipart requests.

HTTP Server

For some time, the implementation was sufficient enough, so I started working on an POC asynchronous HTTP server. The server is pretty much bare-bones right now, though it features everything you need to implement a proper one. Here’s how it can be used:

(local server (require :io.gitlab.andreyorst.fnl-http.server))
(local json (require :io.gitlab.andreyorst.fnl-http.json))

(fn handler [{: headers &as request}]
  {:status 200
   :headers {:connection (or headers.Connection :keep-alive)
             :content-type :application/json}
   :body (json request)})

(with-open [s (server.start handler {:port 3000})]
  (s:wait))

It’s a rather simple echo server, that sends back the parsed request as JSON. Let’s try it:

>> (local http (require :io.gitlab.andreyorst.fnl-http.client))
nil
>> (http.get "localhost:3000" {:as :json})
{:body {:headers {:Host "localhost:3000"}
        :method "GET"
        :path "/"
        :protocol-version {:major 1 :minor 1 :name "HTTP"}}
 :headers {:Connection "keep-alive"
           :Content-Length "131"
           :Content-Type "application/json"}
 :http-client #<tcp-client: 0x561839afe390>
 :length 131
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 13
 :status 200
 :trace-redirects {}}

Now, of course, this won’t work as is, because not all Lua values can be encoded as JSON. Such examples are functions, and custom objects, like Readers. We can cheat, though, and encode it as a Fennel table and send it back as a string:

(local {: view} (require :fennel))

(fn handler [{: headers &as request}]
  {:status 200
   :headers {:connection (or headers.Connection :keep-alive)
             :content-type :application/json}
   :body (view request)})

Now we can try a POST request:

>> (http.post "localhost:3000" {:body "Hello HTTP"})
{:body "{:content #<Reader: 0x55eeb3016fc0>
 :headers {:Content-Length \"10\" :Host \"localhost:3000\"}
 :length 10
 :method \"POST\"
 :path \"/\"
 :protocol-version {:major 1 :minor 1 :name \"HTTP\"}}"
 :headers {:Connection "keep-alive"
           :Content-Length "183"
           :Content-Type "application/json"}
 :http-client #<tcp-client: 0x55eeb3059470>
 :length 183
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 20
 :status 200
 :trace-redirects {}}

This should give you an idea of how to implement a proper handler.

The reason the content is a Reader is simple - we don’t want to process any more data than we need when parsing the request. Once we obtain the HTTP-related data, such as headers, method, path, and version, we wrap the rest of the data in a Reader object, so the handler function can work on it later or even discard it. In this particular case, it’s as simple as doing (request.content:read (tonumber request.headers.Content-Length)) but it may vary depending on what was sent, and what path the client used. So I’m leaving this open. Using a Reader enables asynchronous and lazy processing of the request’s body.

Multipart requests

What’s interesting, however, is that there may be multiple bodies, if the request is a multipart one. We can send a multipart request like this:

(http.post "localhost:3000"
           {:multipart [{:name "foo" :content "some string"}
                        {:name "file" :filename "some-data.json"
                         :content (io.open "some-data.json")}]})

Ideally, the server should be able to first process the foo part, and then a file part. Additionally, a file can be really big, so we might want to avoid reading it fully into memory. For example, the JSON module of fnl-http can parse files without fully reading them into memory, as it is based around readers.

The question, however, is how do we give this to a handler without reading the request in full first? We don’t know how many parts there will be, and we might not know the sizes of each part. Let’s look at how the handler encodes this request:

>> (http.post "localhost:3000"
           {:multipart [{:name "foo" :content "some string"}
                        {:name "file" :filename "some-data.json"
                         :content (io.open "some-data.json")}]})
{:body "{:headers {:Content-Type \"multipart/form-data; boundary=------------aaec41d0-1c75-4037-a039-91cc2b31584c\"
           :Host \"localhost:3000\"}
 :method \"POST\"
 :parts #<function: 0x55eeb28e79c0>
 :path \"/\"
 :protocol-version {:major 1 :minor 1 :name \"HTTP\"}}"
 :headers {:Connection "keep-alive"
           :Content-Length "256"
           :Content-Type "application/json"}
 :http-client #<tcp-client: 0x55eeb31df9e0>
 :length 256
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 80
 :status 200
 :trace-redirects {}}

You can see, that there’s no content field anymore. Instead, there’s a parts field, and it is a function.

This function is a standard Lua iterator, that you can call until it returns nil or plug it in any of the iteration forms. Let’s change our handler to support both content and parts:

(fn handler [request]
  (case request
    {: content}
    (set request.content
      (if request.headers.Content-Length
          (content:read (tonumber request.headers.Content-Length))
          (= request.headers.Transfer-Encoding "chunked")
          (content:read :*a)))
    {: parts}
    (set request.parts
      (icollect [part parts]
        (doto part (tset :content (part.content:read :*a))))))
  {:status 200
   :body (json request)})

Let’s look at this a bit closer. The case has two branches - one for content and one for parts. The content branch is pretty straightforward - we replace request’s content field value with the data obtained from Reader. If there was a Content-Length header, we used it to determine how much to read. If not, we (rather crudely) check if the chunked Transfer-Encoding was used, and read all of the data from the Reader. We can do so, because chunked readers know when to stop, based on the final chunk header.

The parts branch is similar, except we have to go one level deeper, and it’s a bit simpler to work with the part’s contents. First, the main function of icollect is to produce a sequential table from the iterator values. We do so by replacing the content field of each part. The content field is again a Reader, but it’s a sized reader, so we can almost always read it in full. Even if content length was not specified, parts are separated by boundaries, so we always know when to stop.

Each part is a table of the following structure:

{:content #<Reader>
 :headers {...}
 :length N ;; optional field, if headers had a Content-Length specified
 :name "part name"
 :filename "name of the uploaded file"
 ;; alternatively :filename* can be present instead
 :type "part kind"}

As much information is given to make working with parts easier. And because parts are provided by an iterator, we don’t have to know in advance how many parts are coming.

However, there’s a caveat - we can’t obtain information about each part first, and process the contents later. Moving to the next part consumes the request body until the next boundary is given, so we can’t reverse the Reader back to its content. Thus, doing this won’t work:

(let [all-parts (icollect [part parts] part)]
  (each [_ part (ipairs all-parts)]
    (part.content:read :*a)))

You’ll have to either copy reader contents into another in-memory Reader, or process parts as they appear.

Now, we can try our request again:

>> (http.post "localhost:3000"
              {:multipart [{:name "foo" :content "some string"}
                           {:name "file" :filename "some-data.json"
                            :content (io.open "some-data.json")}]
               :as :json})
{:body
 {:headers
  {:Content-Type
   "multipart/form-data; boundary=------------a03996f2-0d1c-4aa4-97f5-ac90b06cb6ea"
   :Host "localhost:3000"}
  :method "POST"
  :parts [{:content "some string"
           :headers {:Content-Disposition "form-data; name=\"foo\""
                     :Content-Length "11"
                     :Content-Transfer-Encoding "8bit"
                     :Content-Type "text/plain; charset=UTF-8"}
           :length 11
           :name "foo"
           :type "form-data"}
          {:content "{\"value\": \"JSON data\"}\n"
           :filename "some-data.json"
           :headers {:Content-Disposition "form-data; name=\"file\"; filename=\"some-data.json\""
                     :Content-Transfer-Encoding "binary"
                     :Content-Type "application/octet-stream"
                     :Transfer-Encoding "chunked"}
           :name "file"
           :type "form-data"}]
  :path "/"
  :protocol-version {:major 1 :minor 1 :name "HTTP"}}
 :headers {:Connection "keep-alive" :Content-Length "808"}
 :http-client #<tcp-client: 0x55cf2d942450>
 :length 808
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 13
 :status 200
 :trace-redirects {}}

As you can see, the foo part had length of 12, and the file part had a chunked transfer encoding.

Before this post, there was no way of sending multipart with chunked encoding, so this is a pretty good change in my opinion. Definitively expands the possibilities. Still need to test this with some real HTTP servers, as I don’t see this feature used too often though.

Performance

Now, let’s talk performance. I used the wrk tool to measure the server’s RPS with a simple handler that does 50 millisecond delay on each request:

(local async (require :io.gitlab.andreyorst.async))

(fn handler [{: headers &as request}]
  (async.<! (async.timeout 50))
  {:status 200
   :headers {:connection (or headers.Connection "keep-alive")
             :content-length 11
             :content-type "text/plain"}
   :body "hello world"})

The handler function is always running in an asynchronous context, so we can use parking operations such as <! inside of it.

Running it with 12 threads and 400 connections for 30 seconds yields about 2k requests per second:

Running 30s test @ http://localhost:3000
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   146.42ms   94.13ms   2.00s    98.54%
    Req/Sec   205.07     90.83   343.00     62.36%
  69835 requests in 30.09s, 8.00MB read
  Socket errors: connect 0, read 0, write 0, timeout 151
Requests/sec:   2321.21
Transfer/sec:    272.37KB

Not bad for a single-threaded Lua runtime, I would say!

Now, of course, this kind of benchmark is more synthetic than real, so don’t expect it to work as well as it says here in all cases. With a more complex handler, that deals with routing, and data processing this number will go down. Still, I’m pretty satisfied with these results.

And CPU usage is quite low too. When idling a single CPU core is sitting at about 5%, and during the benchmark, it rises to 70%. Removing the asynchronous sleep, raises CPU usage to 100% (single core), peaking at 67MB of RAM and serving about 4k requests per second. So with an asynchronous sleep, our server process is doing some actual sleeping, leaving some resources to the OS!

And given how async.fnl is implemented, we can actually replace (async.<! (async.timeout 50)) with a blocking loop, like (for [i 1 1_000_000] nil) and still get around 400 RPS. That’s a busy loop with one million iterations per handler invocation on each request. So you shouldn’t worry too much when blocking the handler.

Is it a good result? You be the judge.

Comment via email

Interactive game development with LÖVE

Sun, 08 Sep 2024 21:00:00 +0300

This is a continuation of the previous post on game development with the LÖVE game engine. I’m slowly appreciating the freedom it gives, compared to the TIC-80 experience. One of such freedoms is the fact, that LÖVE is a well-behaving console application.

When you launch TIC, it boots up into a console:

If you do so from a terminal emulator, you can see that the same console is there too:


 TIC-80 tiny computer
 version 1.1.2837 Pro (be42d6f)
 https://tic80.com (C) 2017-2024

 hello! type help for help

>ls

[tic80.com]
mtd.lua

>

However, writing in this prompt doesn’t do anything - it’s not sent back into TIC. And unfortunately, there’s no way of reading from standard IN, because there’s no io namespace provided. One could, theoretically, fiddle with sockets, if it is even possible to load them into TIC, which I’m not so sure about.

In LÖVE, however, you get full Lua experience and more! As you may know, Lua’s io.read() is blocking, and there’s no real way of making it non-blocking or checking the input buffer. However, that’s not a problem for LÖVE, as it has threads! So we can set up a separate thread that reads from the stdin.

Why would we want that? Well, you see, when it comes to Lua, I usually do my programming in Fennel, and Fennel has a decent REPL for doing interactive development. I love the idea of programming with a REPL, I do it a lot for my hobby projects, and at work with Clojure. Interactive development makes iterations much faster, and, in my experience, minimizes the amount of bugs in the resulting program. And we can leverage that during game development too.

Adding the REPL

The way we’re going to set this up is as follows:

We create a new, asynchronous handler for the REPL, and spawn it on the main thread. Then, we’re going to spawn another thread, that will listen on stdin, and send everything it reads to the main thread. Finally, once the main thread finishes the evaluation, it sends the data back to the IO thread for printing. After that, I’m going to spice things up by using the Fennel Proto REPL package I made for Emacs for a better REPL experience.

Let’s look at the REPL code:

(require :love.event)

(local fennel
  (require :lib.fennel))

(fn read [cont?]
  (io.write (if cont? ".." ">> "))
  (io.flush)
  (.. (tostring (io.read)) "\n"))

(case ...
  (event-type channel)
  (while true
    (match (channel:demand)
      :write (-> (channel:demand)
                 (table.concat "\t")
                 (io.stdout:write "\n"))
      :error (-> (channel:demand)
                 (io.stderr:write "\n"))
      :read (->> (channel:demand)
                 read
                 (love.event.push event-type)))))

(let [chan (love.thread.newChannel)
      repl (coroutine.wrap #(fennel.repl $...))]
  (repl
   {:readChunk (fn [{: stack-size}]
                 (chan:push :read)
                 (chan:push (< 0 stack-size))
                 (coroutine.yield))
    :onValues (fn [vals]
                (chan:push :write)
                (chan:push vals))
    :onError (fn [errtype err]
               (chan:push :err)
               (chan:push err))
    :moduleName "lib.fennel"})
  (-> (love.filesystem.read "lib/repl.fnl")
      (#(fennel.compile-string $))
      (love.filesystem.newFileData "repl")
      love.thread.newThread
      (: :start :eval chan))
  (set love.handlers.eval repl))

Simply requiring this file would start a REPL. You may be confused, about how’s it going to work, the while true may look especially concerning.

When we require this file, the ... is bound to the module name, so the case doesn’t match, as it expects two values an event-type and the channel. Then we fall into the let block, which defines the channel chan, the REPL handler repl, starts the REPL coroutine, which pushes the :read message to the channel. Finally, we load this file again, using love.filesystem.read, compile it to Lua, and start a new thread with :eval for the event-type, and chan for the channel. This makes case match and starts the while loop, whose sole purpose is to take values off the channel and act accordingly.

Finally, we set the love.handlers.eval handler, matching our :eval event type to call the repl as a callback when the event occurs. The event occurs only when we get the :read message from the channel, then we use read to get data from the stdio, and we push it back to the eval handler so the repl callback can be called.

That’s pretty much it! Now, let’s put it to the test!

Changing code at runtime

Here, I have a simple player controller and a bunch of values that correspond to how the player behaves:

(local fall-vel 300)
(local jump-vel -300)
(local run-vel 200)

(local player
  {:name "player"
   :color [0.3 0.7 0.3 1]
   :xvel 0
   :yvel 0
   :x 50
   :y 50
   :flip? false
   :width 16
   :height 24
   :state :idle
   :extra-jump 10
   :draw
   (fn [p]
     ;; omitted
     )
   :update
   (fn [{: xvel : yvel : extra-jump : flip? : x : y &as p} dt]
     ;; omitted
     )})

We can try and change the run-vel and see if it is picked up by the game:

Your browser does not support the tag

Seems it’s not. Perhaps, we need to recompile the game module?

Your browser does not support the tag

Aw, snap!

By recompiling the entire module, we undefined some variables, and the game crashed! By the way, the reason changing the run speed value didn’t apply is that we were actually defining a completely new speed value, that is not part of the module we’re editing. That’s a quirk of how Lua modules work and how Fennel REPL works. So let’s fix this!

Modules and state

While Lua has some lisp characteristics, it is understandably not an actual lisp. So, unfortunately, we can’t expect the same level of interactivity from it.

In Lua, a module is an ordinary table. When you write Lua, you return the value at the bottom of the file, usually a table, or sometimes a function. Any local variables, defined in that module are locals, stored as closures for the functions that reference them in the exported value.

Because of that, there’s no such thing as entering a module like in many other lisps. The REPL has its own kind of module space, where all locals, defined in the REPL are stored. The same goes for any code that we send to the REPL. So, while we do have a REPL, we can’t enter a specific module and change it partially - the only way to do that is to recompile the module and put the updated version in the package.loaded table. Which we can do.

However, there’s another problem lurking around - state. The video above shows that the error message was:

./game.fnl:68: attempt to index upvalue 'world' (a nil value)

When we’ve recompiled the module, world became nil, why? Well, because world is defined as a nil, and later set in the love.load function:

(var world nil)

;; ---- 8< ----

(fn love.load []
  (set world (bump.newWorld 64))
  (each [_ wall (pairs level)]
    (world:add wall wall.x wall.y wall.width wall.height))
  (world:add player player.x player.y player.width player.height))

However, the love.load runs only when the game starts, and hence after recompiling the module, world becomes nil again, and our game doesn’t really expect that. The same would happen to any other state we’re setting up when the game loads.

Unfortunately, Fennel doesn’t have a defonce or anything like it. The defonce macro is available in many lisps, and its purpose is to ensure that when we’re redefining something in the REPL or re-evaluating the whole module/namespace/package (whatever you guys call it) the value doesn’t get redefined. Fennel lacks this because there’s no way of doing this in Lua, because of how locals are stored (as closed variables).

But there’s a way around this, we can move the definitions to another module, and use require to load them. In Lua require caches everything it loads, so when we run require again, it doesn’t really do anything except get the value from the package.loaded table.

Here’s how we can define the world module:

(local bump (require :lib.bump))

(bump.newWorld 64)

Code Snippet 1: state/world.fnl

That’s it. Now, we can bring it to the main game module using require:

(local world (require :state.world))

This way, when we recompile the module, we get the same world as before, unless we were to recompile the world module.

I moved most of the game state in such modules, and now we can use our REPL.

Using the REPL

Here’s an example of tweaking the player’s motion speed again:

Your browser does not support the tag

I’m sending the code to the REPL, instead of reloading the module, just because it’s a bit more convenient. This way tweaking the game is much nicer and frictionless, although it doesn’t reach the same heights as that one video from Bret Victor.

Here’s me using a REPL to make an ad-hoc level editor:

Your browser does not support the tag

Unfortunately, there’s no way of using the REPL on Android, since there’s no way of running LÖVE from the same environment as Emacs, without installing X11 and running everything through Termux. Probably doable, but I don’t want to go this route. Android development is more backup route anyway, since I have access to my PC again.

Comment via email

Boredom and gamedev

Sun, 08 Sep 2024 00:34:00 +0300

I spent the previous ten days on vacation. Usually, I try to go off once or twice a year to somewhere where I can just passively relax - usually, it is some sea resort. This year I decided to go to the Republic of Türkiye and spend my time at the beach without any major attractions. The problem is - I get bored pretty quickly and my brain wants to do something.

I took with me my trusty Nintendo 2DS in case I have an urge to spend some time looking at the screen and to get myself occupied on the plane, and a phone with some books to read while I’m lying on the beach. However, my phone is also my computer, and I do a lot of coding on it in my spare time, usually when I don’t have access to a proper PC.

The last time I went on such a vacation I got so bored, that I decided that I wanted to do some game dev, and started a personal game dev marathon. Originally, I planned to start once I got back from vacation, but I was so eager that I started early, while I was still at the beach. The marathon was a crunch, and I had mixed feelings about it in the end, which ultimately led to burnout, and I put the idea of making games on the shelf for a while. And while the last time I started doing this I only drew some sprites, the only reason for that was that I couldn’t get any game engine on my phone. This time around, I got so bored that I started programming a game while gradually getting sunburns and taking breaks to go and actually swim in the sea.

Making games on Android

Basically, if the only thing you have is a phone you have two main options - TIC-80 and löve2d. tic has everything you need - an integrated code editor, a sprite editor, a map editor, and even a sound editor if you’re that versatile. however, the experience of using TIC for doing everything is not great, especially on the phone. I mean, that’s what you have to deal with:

Not a lot of screen space to view the code, the keyboard is abysmal, and the editor experience itself has a lot to be desired. Yes, it is authentic, I get it, it’s just not what I’m personally after.

On PC you can actually get a PRO version of TIC-80, which allows you to write a game in an external editor, and load it into the engine. Technically, you could do the same on Android, but there’s a problem - I use Android 14, which has storage restrictions for apps. And TIC-80 can’t access anything except files in its directory stored at /sdcard/Android/data/com.nesbox.tic80/. Which can’t be accessed directly on the phone either - you need a PC to access it. So even if you have a separate text editor, you can’t do your game development with TIC on Android. Moreover, you can’t even get your game out of the engine unless you have a PC at hand.

The other option, as I said, is LÖVE2D. It has the same problem as TIC, as you can’t launch the game from anywhere but /sdcard/Android/data/org.love2d.android/, however, newer versions have a separate dedicated launcher, that can load .love files from anywhere on the filesystem, which is good enough. Alternatively, an older version, like 11.3 can be installed that bypasses the sandboxing restrictions. So that was the route I took.

Making games on Android with LÖVE

I have little experience with LÖVE2D. Previously, I did a bunch of tests with it and made a few demos like the one described in my raymarching post, or the F.A.B.R.I.K. algorithm I implemented back in the day. So nothing major. The raymarching demo barely runs on a PC, and there’s no way to control it on the phone without connecting a gamepad. Unfortunately, I had no gamepad with me this time, so if I wanted to make a game on the phone I had to implement one myself. Fortunately, I have a lot of experience playing games with the on-screen gamepad, so I’m at least well accustomed to the idea.

I had no idea how to interact with the touch screen though, so I found a drop-in gamepad implementation for LÖVE2D in the game-tools repository:

Figure 1: lovepad.lua

Still, I wanted this gamepad to be a bit more advanced, with analog stick imitation, and more buttons, so I ported it to Fennel first and then extended it with more features:

tag"/>

It has all of the buttons of a modern controller, so I could share the control scheme between the on-screen gamepad and a real gamepad. I do understand that the on-screen gamepad should, ideally, be tailored around a game, but this will suffice for now. It was fun to implement this too, and I found a small bug in the lovepad.lua file, while I was porting it.

Next, I added the bump.lua library for basic physics (because I don’t want to deal with box2d yet), and made a simple player controller:

As you can see, I’m using Emacs on the phone, with the Google Keyboard set to PC QWERTY layout - a neat feature. This is my phone setup for programming, and I’m actually using the same configuration as on my regular PC.

After adding a camera library from the author of bump.lua, here’s what the player controller looks like at the moment:

Your browser does not support the tag

I want to tweak this player-controller a bit further, making it more fleshed out, but as you can see it already handles a few things:

Direction facing is tracked based on horizontal velocity
Jumping
- Holding the jump button longer increases the jump height
- Bumping against a ceiling resets the vertical velocity
Crouching and crawling
- When crouched, the player’s height is reduced, and when crawling the width is increased
- When crawling into a narrow space, the player changes to the crouched state if no direction input is given.
The camera follows the player
- Camera pans in the direction of movement, showing more of the upcoming level
- Vertical position only changes when the player is grounded, or when the vertical position is lower than the original camera position

I still need to tweak various speed values and add more movement options like dashing, sliding, etc., but it already feels gameish.

Writing a player controller was the most challenging thing before when I made the first demo game in the marathon. I made this one similar to the previous one, as can be seen in further GAME1 demos, but it’s a complete rewrite, as I wanted to make it differently this time. We’ll see how it evolves over time.

Drawing

While I didn’t draw anything for this game, I have a few updates on how I approach drawing on the phone. In short, I ditched the PixelStudio I’ve talked about in the first post about GAME1 mainly because its UX doesn’t work for me personally. It’s still a good app for pixel art, I just don’t like how it works, and it lacks some features I’d like to have.

Fortunately, there exists a better option - Krita which has a port for Android that doesn’t compromise any features, so you get a full Krita experience. It’s probably better suited for Android tablets, but works well enough for me on the phone. I’m slowly learning pixel art, and here’s a timelapse of me drawing a tree on my phone:

Your browser does not support the tag

I made a few mistakes when using various tools, as I was not well accustomed to Krita’s toolset back then.

Here’s the final result after a bit of refinement:

This isn’t a drawing for the game I’m making, just a sketch, following the principles outlined in this amazing video by Brandon James Greer. But one thing that makes a difference when making games for LÖVE2D and not for TIC-80 is the unrestricted palette. It’s still better to restrict the palette when making pixel art, but using only 16 colors for the entirety of game graphics is too restrictive for me. Especially right now.

One thing I haven’t tried yet is animating. PixelStudio had a decent animation pipeline, and it was possible to export the animation as a sprite-sheet. Still need to experience this in Krita - I know it has animation capabilities, but I’m not sure if they’ll be comfortable to use on the phone or not.

I’ll be making more pixel art for this game I’m making in the future, so I’ll post actual images once I have anything more to share!

Comment via email

fnl-http - testing with fennel-test

Thu, 15 Aug 2024 23:01:00 +0300

fnl-http is my current passion project - I spend a lot of free time tinkering with it, and the last week was spent on testing and fixing bugs.

As you may know, I made a testing framework, called fennel-test, which has a dedicated test runner, and a set of macros for writing tests. I use it in all of my projects, and oftentimes, some project requires more features that are currently available in the framework, so I extend it as needed. I think it’s a great way of designing something - you use it, and enhance it as you encounter new needs.

I want programming to be exciting, so I spent a lot of time making my test report nice to look at. Here’s how the report looks by default:

Test run at Tue Aug  6 23:20:41 2024, seed: 1722975641242
(....)(......)(.)(.....)(.)(........)(...)(...)

The default reporter is called dots. Each test namespace is encoded with parentheses (), each successful test is encoded as a dot ., and failed tests are encoded as F. The runner randomizes the order of tests each time, so if we run it again, the result would be different:

Test run at Tue Aug  6 23:24:10 2024, seed: 1722975850235
(......)(........)(.)(...)(...)(....)(.....)(.)

However, this is all you get with a reporter like this. If a test fails, information about the error is printed after all tests have finished, so you know what test did fail.

There’s another inbuilt reporter, called namespaces:

Test run at Tue Aug  6 23:27:33 2024, seed: 1722976053226
test.http-parser-test: PASS
test.client-test: PASS
test.json-test: PASS
test.httpbin-test: PASS
test.body-test: PASS
test.headers-test: PASS
test.readers-test: PASS
test.url-test: PASS

As you can see, it prints each namespace that is currently being tested, followed by the namespace-wide status. If a test fails, you again see its error message, but only after all tests have finished, which isn’t ideal because tests in the fnl-http project take about two minutes to complete.

Fortunately enough, you can create reporters in the .fennel-test configuration file!

Reporters

A reporter is just a table with a set of functions. Each one is called when the respecting stage of testing is being run. The methods are:

ns-start - called when entering a new test namespace.

Used to print the information about the namespace, such as its name.
ns-report - called when exiting a test namespace.

Used to print the overall status of the namespace.
test-start - called when running a single test from a namespace.

Used to print the information about the test, such as the test’s name.
test-report - called after the test is finished.

Used to print the status of the test.
stats-report - called after running all tests in all namespaces.

Used to print overall statistics of tests - how many tests were executed, how many errors were encountered, were there any warnings, etc.

This was enough for me to implement other kind of reporters, like the one in the project’s readme. However, I wanted a bit more for this project - first, I wanted the report to be fun to look at, and second, I wanted some statistics. And third, actually, I noticed the need to skip tests in case something is not available on the machine.

After a bit of work, here’s what I’ve got:

With this reporter, if a specific test fails, I can immediately see it, without waiting for all tests to pass. In the end, you can see some statistics on the longest tests, and average test time of the namespace. For example, the test.httpbin-test namespace took 90 seconds to complete, but on average each test only takes 11 seconds. I’m not entirely sure why this metric would be necessary, but I’ve seen it in another test runner, so I implemented it in my reporter.

Now, here’s a problem - one of the tests takes a whopping 90 seconds. I don’t want to run this test all the time, especially, when I’m fixing a bug found by some other test. So I introduced a skip-test function as a first step.

Originally, this function had two modes - terminating and non-terminating, but I later removed the second mode. The reason for that is the with-open macro.

Here’s an example of one of the fixtures I use in the fnl-http project:

(use-fixtures
 :once
 (fn [t]
   (with-open [proc (io.popen (.. "podman run  -p " port ":80 kennethreitz/httpbin >/dev/null 2>&1 & echo $!"))]
     (let [pid (proc:read :*l)
           attempts 10]
       (if (wait-for-server attempts)
           (do (t)
               (kill pid))
           (do (kill pid)
               (skip-test (.. "coudln't connect to httpbin server after " attempts " attempts"))))))))

The with-open macro ensures that the resource is closed no matter what. Closing the process, however, doesn’t kill the program running in the background, so I had to get its PID with sketchy echo $! part and kill it later. Killing the process doesn’t release the file descriptor created by popen, so with-open is only responsible for releasing the file handle here.

A sudden bug

What with-open also does is catching, and re-throwing errors. It’s something akin to trying with resources in Java, and other languages that support this kind of abstraction. You can think about it as try/finally without the catch part.

Lua, however, doesn’t really have well-defined exceptions. Instead, functions usually throw string messages about the error and that’s it. Sometimes, however, it’s useful to throw a table instead. Tables are like objects, so you can embed some additional information in those, making it easier to match a particular error. It’s better to convert this table to a string if it ever reaches the user uncaught though.

What makes things harder is that in Lua it’s not possible to re-throw an error without losing its previous stack trace. While it is a common practice to use pcall to catch any errors and to re-throw them if you can’t do anything about the error, this is what causes losing the precious stack trace. More precisely, if the error message comes with the stack trace, and you would just re-throw it with error, you wouldn’t add your stack trace to the trace. Sounds convoluted, yeah?

So Lua gives you another option: xpcall. The Lua manual doesn’t have a thorough explanation of it though:

xpcall (f, msgh [, arg1, ···])

This function is similar to pcall, except that it sets a new message handler msgh.

It works the same as pcall, except, it accepts an additional argument - a function that would append a stack trace to the error. Usually it’s the debug.traceback function:

debug.traceback ([thread,] [message [, level]])

If message is present but is neither a string nor nil, this function returns message without further processing. Otherwise, it returns a string with a traceback of the call stack. The optional message string is appended at the beginning of the traceback. An optional level number tells at which level to start the traceback (default is 1, the function calling traceback).

Knowing all that, let’s look at how I’ve implemented the skip-test function:

(local Skip
  (setmetatable
   {}
   {:__tostring #:Skip
    :__fennelview #:Skip}))

(fn skip-test [reason]
  "Calling this function inside a test or a fixture will stop the test
early and mark it as skipped. The optional `reason` argument is a
message to display in the log if the reporter is configured to do so."
  (error [Skip reason]))

Right, skip-test throws an array with a special table object, defined as a singleton, so the test runner could distinguish between ordinary test errors and skipped tests. In theory, everything should have worked, but for some reason, some of the tests were skipped correctly, but others didn’t.

The culprit? It’s with-open, obviously, otherwise, why would I take such a tangent about it?

Internally, with-open uses the xpcall function to do its job. And theoretically, it should work just fine, however, it doesn’t use debug.traceback if Fennel’s own compiler.traceback is available.

Originally, I thought that’s just how it is, and implemented two modes in skip-test. After a bit of digging and consulting the manual, I found out that compiler.traceback mistakenly converted every message to a string. Fixing that made the skip-test function work in all cases.

Another sudden bug!

After the patch landed, and I updated Fennel to the newest commit, something else broke. Now, the JSON parsing test no longer succeeded, complaining that some number has no integer representation. The number in question is 23456789012000000000000000000000000000000000000000000000000000000000000000000 or, as it is written in the JSON test file that I use, 2.3456789012e+76.

Before the update, everything worked fine, so I started digging again. Turns out, recently, the way numbers are compiled by Fennel has changed:

@@ -534,8 +534,9 @@ (fn compile-sym
 ;; We do gsub transformation because some locales use , for
 ;; decimal separators, which will not be accepted by Lua.
 (fn serialize-number [n]
-  (pick-values 1 (-> (tostring n)
-                     (string.gsub "," "."))))
+  (if (= (math.floor n) n)
+      (string.format "%d" n)
+      (string.gsub (tostring n) "," ".")))

And now it is failing, because indeed, you can’t convert 2.3456789012e+76 to a string using %d. Except, you can if you’re using LuaJIT - it happily converts this number to "-9223372036854775808", which is even worse.

After a bit of searching, I found that the safest way to format integers as stings is by using "%.f format specifier. Strangely enough, the Lua manual doesn’t have any information on most number formatting patterns. A tiny bit of information was found on the lua-users.org page:

To make sure integer numbers are properly converted into string (without scientific notation), rather than using tostring(x), use format.string("%.0f",x)

I don’t really get why %.f would format a number as an integer given that %f formats floats. And, even though it works the same way in C, I have never seen this in my life. But OK, we could just replace %d with %.f, and be done with that, however, the number above is then would appear as "23456789012000000697746671432670411048021854502345623879537911769302235611136". I get it, it’s a big number, and Lua has limited precision for integers, but the main problem is that this number is much harder to read than its e-notation brother 2.3456789012e+76.

So I settled on finding a way to fix this bug so that both PUC Lua and LuaJIT would format this number in the same way. The result is a patch, that changes the compiler and pretty-printer parts of the language. The function that does that, looks horrifying:

(fn serialize-number [n]
  (let [val (if (= (math.floor n) n)
                (let [s1 (string.format "%.f" n)]
                  (if (= s1 (tostring n)) s1
                      (or (faccumulate [s nil i 0 99 :until s]
                            (let [s (string.format (.. "%." i "e") n)
                                  n* (tonumber s)]
                              (when (= n n*)
                                (let [exp (s:match "e%+?(%d+)$")]
                                  ;; Lua stops transforming numbers from
                                  ;; e-notation to integers at e+14
                                  (if (and exp (> (tonumber exp) 14))
                                      s
                                      s1)))
                              s1))))
                  (tostring n))]
        (pick-values 1 (string.gsub val "," "."))))

We start by checking if (tostring n) and (string.format "%.f" n) give the same string. If they are, we just return the string. If not, we try the scientific notation for each power from 0 to 99. I probably should’ve tested for more powers than that, but it’s OK for now.

We’re converting the number to a string using the e-notation and parse it back, comparing it with the original number. What this does, is that if the power is less than the original, we would lose precision, and comparison would fail. If we can’t find the exponent up to 99, we bail out and return the %.f representation. Otherwise, if the numbers match, we check if the exponent is greater than 14 as it is the exponent where PUC Lua starts using the e-notation. Otherwise, we again return the %.f representation.

Phew!

But I’m not done yet!

Let’s try to compile a really huge number:

>> 1e+99999
nil

What?

Let’s do it in an expression then, maybe that’s just the pretty printer freaking out:

>> (+ 1e+309 1)
attempt to perform arithmetic on a nil value (global 'inf')

~~What?~~ I, of course, jest, but the reason this happens is pretty peculiar.

You see - when we do (tostring "%.f" n) on a huge number like 1e+309 it returns the string "inf". The compiler then happily includes it in the output Lua code, which is then run by the Lua VM. In Lua, however, inf is not a reserved identifier or a number, so it treats inf like a variable name. The inf variable is unbound, hence the nil in the first example, and an error in the second.

So we need to account for that:

(fn serialize-number [n]
  (let [val (if (= (math.floor n) n)
                (let [s1 (string.format "%.f" n)]
                  (if (= s1 "inf") "(1/0)" ; portable inf
                      (= s1 "-inf") "(-1/0)"
                      (= s1 (tostring n)) s1 ; no precision loss
                      (or (faccumulate [s nil
                                        i 0 308 ; beyond 308 every number turns to inf
                                        :until s]
                            (let [s (string.format (.. "%." i "e") n)]
                              (when (= n (tonumber s))
                                (let [exp (s:match "e%+?(%d+)$")]
                                  ;; Lua keeps numbers in standard notation up to e+14
                                  (if (and exp (> (tonumber exp) 14))
                                      s
                                      s1)))))
                          s1)))
                (tostring n))]
    (pick-values 1 (string.gsub val "," "."))))

I also brought the exponent limit up to the last number representable in this notation without becoming inf.

Now we can go back to testing!

Counting skipped tests and assertions

The other thing I added to fennel-test was more counters. Counting skipped tests required some tinkering, like counting tests in the whole namespace if the tests were skipped from the :once fixture.

Counting assertions was even harder, because tests in fennel-test are implemented through macros, and assertions are macros too. I had to inject some runner state, conditionally, and modify it at runtime from each assertion. A bit shady but it works.

What is much more shady is the coverage reports for this project.

Coverage report

I have a firm belief that 100% is the only acceptable coverage. However, and this is a big however, it’s important that such coverage comes from public API tests only. Achieving 100% coverage this way means that your program doesn’t have any dead code and that your public API test at least touches all of our code at least once. It doesn’t speak about test quality, of course, and it doesn’t guarantee that the program is bug-free, but those other points alone make a huge difference.

Lua has one and only luacov that you can use to test your code. I’ve used it in some other Fennel projects in the past, but it comes with some nuances you have to deal with.

First, luacov itself is imprecise. In the post about 100% coverage I talk about it in more detail, but basically, I often see situations where the side effects happened, but the coverage report showed code responsible for the side effects as untouched. Other times, the coverage percentage is bigger than its actual value due to code formatting.

With the fnl-http project I hit a different problem. Two, actually.

The first one is related to the --correlate option of the Fennel compiler. I had an idea, that if I used --correlate I could substitute Lua sources used to produce the coverage with Fennel sources. And since the line numbers are matched by specifying the --correlate option, the coverage report would be so much easier to read.

Didn’t work out.

Unfortunately, --coverage is imprecise. Many parts of Fennel AST don’t contain any line information. Here’s an example:

(local {: a
        : b}
  (require :c))

{: a
 : b}

And here’s the corresponding Lua code:

local _local_1_ = require("c") local a = _local_1_["a"]
local b = _local_1_["b"]
-- empty line
-- empty line
return {a = a, b = b}
-- empty line

As you can see, the overall amount of lines is the same, but many lines that are not empty in the Fennel example are empty in Lua. It’s not a problem, as long as you’re measuring the coverage and viewing the report with Lua, but if we replace the Lua code with its Fennel counterpart, the report becomes invalid. I guess, when the report is constructed, luacov checks if the line is empty or not, and if it isn’t, and there were no hits registered, it counts it as a miss. I’m using luacov-console to view the report, maybe it’s a bug from there, I don’t know.

The only way to fix this is to write the code accounting for these nuances of --correlate:

(local {: a                ; local _local_1_ = require("c") local a = _local_1_["a"]
        : b} (require :c)) ; local b = _local_1_["b"]
                           ; <empty line>
{: a : b}                  ; return {a = a, b = b}

There are more examples like that in the project’s code.

The other problem is more peculiar. Luacov uses debug.sethook to build its report. And I use the async.fnl library in this project, which also uses debug.sethook to schedule tasks. So every time anything happens asynchronously, and trust me, there’s a lot of that in fnl-http, the Luacov report is no longer properly calculated.

So, Luacov has a lot of problems, but I understand why it is like that. You can’t really instrument the Lua code in some other way.

Fennel, on the other hand, isn’t like that. Being a language where everything is an expression, we can probably make some compiler plugin that injects some side-effecting code that writes the report for every expression in the code. It would be difficult to do with things like tables and destructuring, but it should be doable. Maybe I’ll try to tackle this problem in the future, who knows?

Anyway, that’s all from me for now! As usual, you can check out the project, which contains all of the stuff mentioned above regarding fennel-test and luacov. Thanks for reading!

Comment via email

fnl-http Improvements

Thu, 01 Aug 2024 22:57:00 +0300

In the last post two weeks ago I described the process of making an asynchronous HTTP/1.1 client from scratch (minus the socket part). At the end, I mentioned that there’s a lot more to implement:

Now, of course, that’s not all that needs to be implemented. In its current state, this client doesn’t support responses that do not specify the Content-Length or produce a response body in chunked encoding. HTTPS is out of the question for now too. Most of the code just takes a “happy path” without much thought given to errors or more complicated scenarios. But it’s fine, I can always make it better down the line.

HTTPS is still out of the question - to be honest, I don’t want to deal with binary protocols that much, and get myself involved with implementing SSL in Lua. I could, of course use a library, but I feel that Luasocket alone is already big enough dependency.

So here are improvements so far.

Chunked transfer-encoding

As previously mentioned, the client didn’t understand chunked responses, so I had to fix it with a chunked body reader:

(fn read-chunk-size [src]
  {:private true}
  ;; TODO: needs to process chunk extensions
  (case (src:read :*l)
    "" (read-chunk-size src)
    line
    (case (line:match "%s*([0-9a-fA-F]+)")
      size (tonumber (.. "0x" size))
      _ (error (format "line missing chunk size: %q" line)))))

(fn chunked-body-reader [src initial-chunk]
  "Reads body in chunks, buffering each fully, and requesting the next
chunk, once the buffer is empty."
  {:private true}
  (var chunk-size initial-chunk)
  (var buffer (or (src:read chunk-size) ""))
  (var more? true)
  (fn read-more []
    ;; TODO: needs to process entity headers after the last chunk.
    (when more?
      (set chunk-size (read-chunk-size src))
      (if (> chunk-size 0)
          (set buffer (.. buffer (or (src:read chunk-size) "")))
          (set more? false)))
    (values (> chunk-size 0) (string-reader buffer)))
  (make-reader
   src
   {:read-bytes (fn [_ pattern]
                  (let [rdr (string-reader buffer)]
                    (case pattern
                      (where n (= :number (type n)))
                      (let [buffer-content (rdr:read pattern)
                            len (if buffer-content (length buffer-content) 0)
                            read-more? (< len n)]
                        (set buffer (buffer:sub (+ len 1)))
                        (if read-more?
                            (let [(_ rdr) (read-more)]
                              (if buffer-content
                                  (.. buffer-content (or (rdr:read (- n len)) ""))
                                  (rdr:read (- n len))))
                            buffer-content))
                      (where (or :*l :l))
                      (let [buffer-content (rdr:read :*l)
                            (_ read-more?) (not (buffer:find "\n"))]
                        (when buffer-content
                          (set buffer (buffer:sub (+ (length buffer-content) 2))))
                        (if read-more?
                            (let [rdr (read-more)]
                              (if buffer-content
                                  (.. buffer-content (or (rdr:read :*l) ""))
                                  (rdr:read :*l)))
                            buffer-content))
                      (where (or :*a :a))
                      (let [buffer-content (rdr:read :*a)]
                        (set buffer "")
                        (while (read-more) nil)
                        (let [rdr (string-reader buffer)]
                          (set buffer "")
                          (case (rdr:read :*a)
                            nil (when buffer-content
                                  buffer-content)
                            data (.. (or buffer-content "") data))))
                      _ (error (tostring pattern)))))
    :read-line (fn [src]
                 (let [rdr (string-reader buffer)
                       buffer-content (rdr:read :*l)
                       read-more? (not (buffer:find "\n"))]
                   (when buffer-content
                     (set buffer (buffer:sub (+ (length buffer-content) 2))))
                   (if read-more?
                       (if buffer-content
                           (.. buffer-content (or (src:read :*l) ""))
                           (src:read :*l))
                       buffer-content)))
    :close (fn [src] (src:close))
    :peek (fn [_ bytes]
            (assert (= :number (type bytes)) "expected number of bytes to peek")
            (let [rdr (string-reader buffer)
                  content (or (rdr:read bytes) "")
                  len (length content)]
              (if (= bytes len)
                  content
                  (let [(_ rdr) (read-more)]
                    (let [data (rdr:read (- bytes len))]
                      (set buffer (.. buffer (or data "")))
                      buffer)))))}))

It’s a mess, but it works. Please, refer to the previous post for the explanation of what a reader is.

The main idea here is that, when the chunked transfer encoding is detected, we create a reader with chunked-body-reader instead of a regular body-reader, and set up an internal string buffer. We then process all reading operations in terms of this buffer, until it depletes, or when we request more data than left in the buffer. After that, we download the next chunk.

We still get a stream-like behavior out of this, although it has a somewhat big issue - if the server responds with a chunk of enormous size, we can fill up all available memory. I’ve thought of reworking this function to fill the buffer with smaller chunks, but it complicates the code even further, so I decided not to. For now, at least.

And this now works both ways! We can send a chunked body to the server if we’ve didn’t specify the Content-Length header, and the body is not a string. This happens automatically for files, other readers, and channels. Yes, with this library, you can stream data to a server from a channel:

(let [ch (chan)]
  (go #(with-open [body (io.open "some-file" :r)]
         (each [line (body:lines)]
           (<! (timeout 300))
           (>! ch (.. line "\n")))
         (close! ch)))
  (http.post "http://some-server.org" {:body ch}))

This will stream the file line by line, sending each line every 300 milliseconds. The server will receive each line as a separate chunk.

I won’t go into much detail on how this is implemented, as I would need to go through a lot of code.

Multipart requests

After I finished with chunked requests, I turned my eyes on multipart/form-data. It’s an interesting part of HTTP, and after a closer look at it, I realized that email is just a multipart request with weird headers. I’ve been editing emails in Emacs for quite some time but never noticed that the head of the mail is exactly like the head of an HTTP request. What a revelation!

That aside, implementing multipart came with a challenge. I don’t know if it’s just me, but I couldn’t find anywhere if the total Content-Length header is required for a multipart request or not. Logically speaking, since every part is separated by a boundary, a server should be able to read a request’s body until the final boundary. So theoretically, it doesn’t have to know the total content length of the request.

However, I’ve been testing my client with the httpbin.org service and, for some reason, it doesn’t parse the multipart request unless the Content-Length header is present. And it’s not just the sum of part lengths, it’s also their headers, empty lines that separate the header part from content, and so forth.

So I had to implement a set of functions to format multipart parts, and count their lengths:

(fn format-multipart-part [{: name : filename : filename*
                            : content :length content-length
                            : headers
                            : mime-type} boundary]
  "Format a single multipart entry.
The part starts with the `boundary`, followed by headers, created from
`name`, optional `filename` or `filename*` for files, `mime-type`, and
`content-length` which is either calculated from `content` or provided
explicitly.

Default headers include `content-disposition`, `content-length`,
`content-type`, and `content-transfer-encoding`. Provide `headers` for
additional or to change the default ones."
  {:private true}
  (let [content (wrap-body content)]
    (format
     "--%s\r\n%s\r\n"
     boundary
     (headers->string
      (collect [k v (pairs (or headers {}))
                :into {:content-disposition (format "form-data; name=%q%s%s" name
                                                    (if filename
                                                        (format "; filename=%q" filename)
                                                        "")
                                                    (if filename*
                                                        (format "; filename*=%s" (urlencode-string filename*))
                                                        ""))
                       :content-length (if (= :string (type content))
                                           (length content)
                                           (or content-length (content:length)))
                       :content-type (or mime-type (guess-content-type content))
                       :content-transfer-encoding (guess-transfer-encoding content)}]
        k v)))))

This function just formats a single multipart header, i.e. it doesn’t include the content. The content is streamed later, so we don’t need it here.

But with this function, we can now calculate the total length of our request:

(fn multipart-content-length [multipart boundary]
  "Calculate the total length of `multipart` body.
Needs to know the `boundary`."
  (+ (accumulate [total 0
                  _ {:length content-length
                     : name
                     : content
                     &as part}
                  (ipairs multipart)]
       (let [content (wrap-body content)]
         (+ total
            (length (format-multipart-part part boundary))
            (if (= :string (type content)) (+ (length content) 2)
                (reader? content)
                (+ 2 (or content-length
                         (content:length)
                         (error (format "can't determine length for multipart content %q" name) 2)))
                (not= nil content-length)
                (+ content-length 2)
                (error (format "missing length field on non-string multipart content %q" name) 2)))))
     (length (format "--%s--\r\n" boundary))))

This function doesn’t construct the whole body either - it only calculates the length based on headers, and the length of content fields that it can reasonably calculate. There’s a small bug in the format-multipart-part function though. We can override any header, including content-length, meaning we can break the request, if we send a string, and provide a different content-length for it either by mistake or deliberately. This can be fixed, of course, by either not allowing to change content length for strings, or by wrapping string bodies in a string reader. I’ll have to think about what method is better here.

That aside, we can now stream the multipart request:

(fn stream-multipart [dst multipart boundary]
  "Write `multipart` entries to `dst` separated with the `boundary`."
  (each [_ {: name : filename
            : content :length content-length
            : mime-type
            &as part}
         (ipairs multipart)]
    (assert (not= nil content) "Multipart content cannot be nil")
    (assert name "Multipart body must contain at least content and name")
    (let [content (wrap-body content)]
      (->> (if (= :string (type content)) content "")
           (.. (format-multipart-part part boundary))
           (dst:write))
      (when (not= :string (type content))
        (stream-body dst content {:content-length (or content-length (content:length))})))
    (dst:write "\r\n"))
  (dst:write (format "--%s--\r\n" boundary)))

Again, I’m not going to dive too deep here. If you’re interested in the code, you can find and read it here.

Here’s an example of a multipart request body:

--a25537b6-51d6-491f-b18b-a33d197ae4c6\r
Content-Disposition: form-data; name="json-test"; filename="test.json"\r
Content-Length: 1438\r
Content-Transfer-Encoding: binary\r
Content-Type: application/octet-stream\r
\r
[
    "JSON Test Pattern pass1",
    {"object with 1 member":["array with 1 element"]},
    {},
    [],
    -42,
    true,
    false,
    null,
    {
        "integer": 1234567890,
        "real": -9876.543210,
        "e": 0.123456789e-12,
        "E": 1.234567890E+34,
        "":  23456789012E66,
        "zero": 0,
        "one": 1,
        "space": " ",
        "quote": "\"",
        "backslash": "\\",
        "controls": "\b\f\n\r\t",
        "slash": "/ & /",
        "alpha": "abcdefghijklmnopqrstuvwyz",
        "ALPHA": "ABCDEFGHIJKLMNOPQRSTUVWYZ",
        "digit": "0123456789",
        "0123456789": "digit",
        "special": "`1~!@#$%^&*()_+-={':[,]}|;.</>?",
        "hex": "\u0123\u4567\u89AB\uCDEF\uabcd\uef4A",
        "true": true,
        "false": false,
        "null": null,
        "array":[  ],
        "object":{  },
        "address": "50 St. James Street",
        "url": "http://www.JSON.org/",
        "comment": "// /* <!-- --",
        "# -- --> */": " ",
        " s p a c e d " :[1,2 , 3

,

4 , 5	   ,  		        6           ,7        ],"compact":[1,2,3,4,5,6,7],
        "jsontext": "{\"object with 1 member\":[\"array with 1 element\"]}",
        "quotes": "&#34; \u0022 %22 0x22 034 &#x22;",
        "/\\\"\uCAFE\uBABE\uAB98\uFCDE\ubcda\uef4A\b\f\n\r\t`1~!@#$%^&*()_+-=[]{}|;:',./<>?"
: "A key can be any string"
    },
    0.5 ,98.6
,
99.44
,

1066,
1e1,
0.1e1,
1e-1,
1e00,2e+00,2e-00
,"rosebud"]
\r
--a25537b6-51d6-491f-b18b-a33d197ae4c6\r
Content-Disposition: form-data; name="description"\r
Content-Length: 27\r
Content-Transfer-Encoding: 8bit\r
Content-Type: text/plain; charset=UTF-8\r
\r
a JSON file to test parsers\r
--a25537b6-51d6-491f-b18b-a33d197ae4c6--\r

And here’s this request issued with the client library:

>> (http.post "http://httpbin.org/post"
              {:multipart [{:name "json-test"
                            :content (io.open "test/data/valid.json")
                            :filename "test.json"}
                           {:name "description" :content "a JSON file to test parsers"}]
               :as :json})
{:body {:args {}
        :data ""
        :files {:json-test "..."} ;; contents were excluded to save some vertical space
        :form {:description "a JSON file to test parsers"}
        :headers {:Content-Length "1946"
                  :Content-Type "multipart/form-data; boundary=------------5af9b616-6e9d-4c19-8aa5-cc0c5f52ca11"
                  :Host "httpbin.org"
                  :X-Amzn-Trace-Id "Root=1-66aad04d-5155e53a4023b5726bdd134e"}
        :url "http://httpbin.org/post"}
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "2118"
           :Content-Type "application/json"
           :Date "Thu, 01 Aug 2024 00:01:18 GMT"
           :Server "gunicorn/19.9.0"}
 :http-client #<SocketChannel: 0x559596d0e200>
 :length 2118
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 388
 :status 200}

As you can see, the JSON got into the files field, and the description into the form field.

Redirects and query-params

Honestly, implementing redirection was a pain in the butt. But I’m glad that I did!

First, I now finally understand how redirection works. Second, I had to do a major code refactoring for it, which made some parts more approachable, and robust. That was a lot of changes, though.

Alongside redirects, I improved the URL parser and implemented a query builder, that can merge multiple queries into one. Here’s an example, again with httpbin.org:

>> (http.get "http://httpbin.org/redirect-to?status_code=302"
             {:query-params {:url "http://httpbin.org/get"}
              :as :json})
{:body {:args {}
        :headers {:Host "httpbin.org"
                  :X-Amzn-Trace-Id "Root=1-66aad21a-4c52d9316b88f5d862a4b535"}
        :url "http://httpbin.org/get"}
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "199"
           :Content-Type "application/json"
           :Date "Thu, 01 Aug 2024 00:08:58 GMT"
           :Server "gunicorn/19.9.0"}
 :http-client #<SocketChannel: 0x559596734da0>
 :length 199
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 394
 :status 200}

As can be seen, the query params in the URL string and the query-params field in the options table were merged. During the merge, all values are URL-encoded, so you can pass in URLs that are not entirely correct, and it should work.

Of course, we can change how we deal with redirects, limiting the total amounts of redirects:

>> (http.get "http://httpbin.org/absolute-redirect/4"
             {:max-redirects 3
              :as :json})
too many redirecs

Or, we can choose not to follow redirects altogether:

>> (http.get "http://httpbin.org/absolute-redirect/4"
             {:follow-redirects? false})
{:body "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: <a href=\"http://httpbin.org/absolute-redirect/3\">http://httpbin.org/absolute-redirect/3</a>.  If not click the link."
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "283"
           :Content-Type "text/html; charset=utf-8"
           :Date "Thu, 01 Aug 2024 00:24:48 GMT"
           :Location "http://httpbin.org/absolute-redirect/3"
           :Server "gunicorn/19.9.0"}
 :http-client #<SocketChannel: 0x5589fcb69530>
 :length 283
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "FOUND"
 :request-time 228
 :status 302}

Changes in the asynchronous API

I’ve decided to make it easier for users of this library to work with the client’s asynchronous API, and you should no longer be required to use async.fnl directly. Instead, every request function now accepts callbacks for processing responses, and exceptions:

(http.get "https://lua-users.org"
          {:async? true}
          (fn on-response [resp] (print resp.body))
          (fn on-raise [err] (print err.status)))

This means, that the only need for async.fnl is if you need to wait for certain requests to complete. It also means, that the http.METHOD functions will no longer return a promise channel, as I promised in the previous post.

In addition to that, it is now possible to read the response body, returned as a stream on the main thread, even if the request was asynchronous. This is possible because I’ve cheated:

(fn <!? [port]
  "Takes a value from `port`.  Will return `nil` if closed.  Will block,
if nothing is available and used on the main thread.  Will park if
used in the `(go ...)` block."
  (if (main-thread?)
      (<!! port)
      (<! port)))

(fn >!? [port val]
  "Puts a `val` into `port`.  `nil` values are not allowed.  Will block,
if no buffer space or consumer are available and used on the main
thread.  Will park if used inside a `(go ...)` block.  Returns `true`
unless `port` is already closed."
  (if (main-thread?)
      (>!! port val)
      (>! port val)))

Now try to do something like that in Clojure! You can read why this works in my post on async.fnl, or watch my talk at Fennelconf 2023 where I explain this with some more examples.

Performance check

Since the client got bigger and more complicated, let’s check the performance! In the previous post, the client couldn’t accept a chunked response, but now it can, meaning we can compare it with Luasocket’s HTTP client:

>> (local lhttp (require :socket.http))
nil
>> (= (lhttp.request "http://lua-users.org/wiki/FiltersSourcesAndSinks")
      (. (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks") :body))
true
>>

Let’s measure the time again:

>> (time (do (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks") nil))
Elapsed 501.31988525391 ms
nil
>> (time (do (lhttp.request "http://lua-users.org/wiki/FiltersSourcesAndSinks") nil))
Elapsed 630.47289848328 ms
nil

It’s about the same. However, we can be slightly faster, if the user doesn’t need to process the body immediately:

>> (time (do (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks" {:as :stream}) nil))
Elapsed 317.51012802124 ms
nil

Of course, the time is saved by not reading the body of the response, which shows that reading the body alone is about 200ms. But Luasocket’s HTTP client can’t do that at all - it always processes the body, even if the generic form is used and a sink is supplied. That’s an advantage, I would say!

And, thanks to the asynchronous processing of requests, we can actually issue multiple requests at the same time:

>> (let [n-requests 10
         ch (chan)]
     (print ";;; luasocket")
     (time
      (let [start (gettime)]
        (for [i 1 n-requests]
          ;; issuing a synchronous request
          (lhttp.request "http://lua-users.org/wiki/FiltersSourcesAndSinks")
          (io.write i " response processed in " (- (gettime) start)
                    " ms since the first request\n"))))
     (print ";;; fnl-http")
     (time
      (let [start (gettime)]
        (for [i 1 n-requests]
          ;; issuing multiple requests. The on-success handler
          ;; will put the request number into the channel `ch`
          ;; on completion.
          (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks"
                    {:async? true} #(>! ch [i (- (gettime) start)]) #nil))
        (for [i 1 n-requests]
          ;; avaiting each response
          (let [[i ms] (<!! ch)]
            (io.write i " response processed in " ms
                      " ms since the first request\n"))))))
;;; luasocket
1 response processed in 0.55836296081543 ms since the first request
2 response processed in 1.0487260818481 ms since the first request
3 response processed in 1.4974920749664 ms since the first request
4 response processed in 1.9714229106903 ms since the first request
5 response processed in 2.4879760742188 ms since the first request
6 response processed in 3.010272026062 ms since the first request
7 response processed in 3.6146268844604 ms since the first request
8 response processed in 4.1402359008789 ms since the first request
9 response processed in 4.6472299098969 ms since the first request
10 response processed in 5.1433019638062 ms since the first request
Elapsed 5143.6059474945 ms
;;; fnl-http
5 response processed in 1.3388519287109 ms since the first request
7 response processed in 1.4401869773865 ms since the first request
9 response processed in 1.5485229492188 ms since the first request
4 response processed in 1.6689200401306 ms since the first request
8 response processed in 1.7789130210876 ms since the first request
6 response processed in 1.9086010456085 ms since the first request
1 response processed in 2.0185270309448 ms since the first request
10 response processed in 2.1092948913574 ms since the first request
2 response processed in 2.2185280323029 ms since the first request
3 response processed in 2.3184778690338 ms since the first request
Elapsed 2318.8180923462 ms

Still ‘bout twice as fast!

And, as you can see, the order of request completion in the asynchronous case is different, since multiple requests were processed in parallel. Well, it’s not really parallel execution, since the Lua runtime only has one thread, and all response reading is synchronous, but the moment we’ve written the request into the socket, the operating system, and networking hardware could act independently. Thus, the response was sent, and the server on the other side received a series of requests, and probably processed them in parallel, resulting in a faster overall response. While we wait for the data to come back to our socket, other tasks can be executed, such as making another request or processing a part of the response. Of course, when we process the response, we do it asynchronously as well, creating opportunities for other asynchronous threads to collaborate.

Neat stuff!

Even further development, I guess?

When I started this project, I just wanted to use async.fnl for something more real and see if it works. After working on it for about a month, and reading the source code of the clj-http client, this project gradually became something like a re-implementation of clj-http in Fennel. So I changed the project name to fnl-http. It’s not a one-to-one port like async.fnl where I studied the source code - here all code is written from scratch without looking at other clients, but still, I tried to make it faithful.

There are still a lot of parts of HTTP/1.1 that I haven’t processed, and I still have some refactoring ideas and TODOs, so stay tuned!

Comment via email

Asynchronous HTTP client for Fennel

Mon, 15 Jul 2024 20:08:00 +0300

A while ago, I made a library for asynchronous programming in Fennel. It’s based on Clojure’s core.async vision of asynchronous programming using only channels. As an experiment, I’ve added a TCP support layer in that library, allowing one to create a TCP channel, and use it in the same way as a regular channel. My original plan was to implement a small asynchronous HTTP client using that, and I’ve tried it a bunch of times in the last year.

A bit later, I felt unusually burned out from programming as a hobby. In April, I made a post about hobbies, saying that I’d like to switch my hobbies from computers for some time. Truth be told, I’ve been in this stance since February, and only now do I once again feel some interest in programming as a hobby.

So today, I’ll walk you through why I decided to write an HTTP client almost from scratch, and why Luasocket’s HTTP client didn’t work out.

Luasocket’s HTTP client

So, there’s an HTTP client in the luasocket library. And while it does support streaming the body through the LTN12 module, the responses are always synchronous. Which means, that if you have a big enough body, that you’d like to process on demand, you only get the convenience of a stream that you’ve provided as a sink. I’d like the body to be returned as some kind of stream that you can process later, reading data from the client, not from some kind of buffer.

The first thing I tried was patching http.open function at runtime. Then we can open the connection and change the close method on the socket, to work with our intentions. Here’s a piece of code that does this:

(let [{: open : request &as http} (require :socket.http)]
  (fn http.open [...] ;; replacing `open` with our version
    (let [h (open ...) ;; using the original `open` inside
          close (. (getmetatable h) :__index :close)]
      (set close-handler #(close h))
      (fn h.close [self]
        (if (not got-body)
            (close-handler self)
            1))
      h))
  (let [(ok? code resp) (request opts)] ;; "thankfully" `http.request`
                                        ;; resolves `http.open`
                                        ;; dynamically
    (set http.open open) ;; restoring the original `http.open`
    (if ok?
        {:status code
         :headers resp
         :body out} ;; the body is now a channel
        (error code))))

There are a lot of missing details, for example, the out is an ordinary channel, however, there’s a sink that passed via the opts table that pumps everything to it on demand. Once the sink is drained the connection is closed.

All in all, this is a very stateful and shaky approach, so I didn’t want to rely on that. Furthermore, the sink is being pumped in chunks that can’t be controlled very well, making it harder to work with the data in a meaningful way. So I knew I should do something else.

I decided to write my own HTTP client based on Luasocket sockets, and streams. Also, I wanted a bit more familiar calling interface, because Luasocket’s HTTP client is not pleasant to use in my opinion. So let’s start with the HTTP parser!

Reader

Actually, before parsing HTTP, we need to be able to read data from a stream. Sockets are kinda like streams already, but we’re not there yet, so let’s create a string reader:

>> (string-reader "foo\nbar\nbaz")
#<Reader: 0x55f6ca3475b0>

The string-reader function accepts a string, and returns a Reader object that provides the following methods:

read - returns a specified amount of bytes, determined either by the number of bytes or by a supported read pattern.
lines - returns a function that when called will return the next logical line.
peek - returns a specified amount of bytes, without moving the caret.
close - closes the Reader.

Here’s the implementation of string-reader:

(fn string-reader [string]
  (var (i closed) (values 1 false))
  (let [len (length string)
        try-read-line (fn [s pattern]
                        (case (s:find pattern i)
                          (start end s)
                          (do (set i (+ end 1)) s)))
        read-line (fn [s]
                    (when (<= i len)
                      (or (try-read-line s "(.-)\r?\n")
                          (try-read-line s "(.-)\r?$"))))]
    (make-reader
     string
     {:close (fn [_]
               (when (not closed)
                 (set i (+ len 1))
                 (set closed true)
                 closed))
      :read-bytes (fn [s pattern]
                    (when (<= i len)
                      (case pattern
                        (where (or :*l :l))
                        (read-line s)
                        (where (or :*a :a))
                        (s:sub i)
                        (where bytes (= :number (type bytes)))
                        (let [res (s:sub i (+ i bytes -1))]
                          (set i (+ i bytes))
                          res))))
      :read-line read-line
      :peek (fn [s pattern]
              (when (<= i len)
                (case pattern
                  (where bytes (= :number (type bytes)))
                  (let [res (s:sub i (+ i bytes -1))]
                    res)
                  _ (error "expected number of bytes to peek"))))})))

It’s a bit big, but there’s a reason for that - strings in Lua are immutable, and we have to maintain the state when reading. What makes this more complicated is the peek method, but we’ll get to that later.

The make-reader function is a generic Reader constructor, that accepts a table of optional methods:

(fn ok? [ok? ...] (when ok? ...))

(fn make-reader [source {: read-bytes : read-line : close : peek}]
  (let [close (if close
                  (fn [_ ...]
                    (ok? (pcall close source ...)))
                  #nil)]
    (-> {: close
         :read (if read-bytes
                   (fn [_ pattern ...]
                     (ok? (pcall read-bytes source pattern ...))) #nil)
         :lines (if read-line
                    (fn []
                      (fn [_ ...]
                        (ok? (pcall read-line source ...))))
                    (fn [] #nil))
         :peek (if peek
                   (fn [_ pattern ...]
                     (ok? (pcall peek source pattern ...)))
                   #nil)}
        (setmetatable
         {:__close close
          :__name "Reader"
          :__fennelview #(.. "#<" (: (tostring $) :gsub "table:" "Reader:") ">")}))))

As you can see, it just wraps everything with pcall and passes the specified source to the methods. Perhaps, there’s a more clear way of doing this, but it’s fine for now.

Here’s a file-reader, for example:

(fn file-reader [file]
  (let [file (case (type file)
               :string (io.open file :r)
               _ file)]
      (make-reader file
                   {:close #(: $ :close)
                    :read-bytes (fn [f pattern] (f:read pattern))
                    :read-line (file:lines)
                    :peek (fn [f pattern]
                            (assert (= :number (type pattern)) "expected number of bytes to peek")
                            (let [res (f:read pattern)]
                              (f:seek :cur (- pattern))
                              res))})))

It accepts either a file handle or a string that is treated as a path to open the file. It’s much shorter than the string-reader since files can be seek’d and the file object is already stateful.

In the end, we can do this:

>> (local rdr (reader.string-reader "foo\nbar\nbaz\nqux"))
nil
>> (rdr:read 2)
"fo"
>> (rdr:read :*l)
"o"
>> (rdr:read :*l)
"bar"
>> (rdr:read :*a)
"baz
qux"

We’ll be implementing a socket Reader soon, but before that, let’s start implementing a set of functions to generate HTTP messages.

HTTP generation

We start with the build-http-request function:

(local HTTP-VERSION "HTTP/1.1")

(fn build-http-request [method request-target ?headers ?content]
  "Formaths the HTTP request string as per the HTTP/1.1 spec."
  (string.format
   "%s %s %s\r\n%s\r\n%s"
   (string.upper method)
   request-target
   HTTP-VERSION
   (or (headers->string ?headers) "")
   (or ?content "")))

As can be seen, it’s just a string.format call, which we fill in with specific parts. The method and request-target are mandatory arguments, as nothing will work without them. ?headers and ?content on the other hand are optional. Let’s implement the headers->string function now:

(fn header->string [header value]
  "Converts `header` and `value` arguments into a valid HTTP header
string."
  (.. (utils.capitalize-header header) ": " (tostring value) "\r\n"))

(fn headers->string [headers]
  "Converts a `headers` table into a multiline string of HTTP headers."
  (when (and headers (next headers))
    (-> (icollect [header value (pairs headers)]
          (header->string header value))
        table.concat)))

We simply check whether headers is not nil, and if it has any keys. If so, we iterate over the headers table, and convert each pair to a string, then concatenating the resulting table:

>> (headers->string {:connection "close" :content-type "application/json"})
"Content-Type: application/json\r
Connection: close\r
"

The utils.capitalize-header function accepts a string in any case, and transforms it to an HTTP case, meaning each word is capitalized and separated with a dash:

>> (utils.capitalize-header "foo_bar")
"Foo-Bar"
>> (utils.capitalize-header "FooBar")
"Foo-Bar"
>> (utils.capitalize-header "FOO BAR")
"Foo-Bar"

I’ll let you think about implementation, as I’m not particularly fond of my own :)

With that, we can generate a request:

>> (build-http-request
    :get "/"
    {:host "localhost:1234"
     :connection "close"
     :content-type "text/plain"}
    "foobar")
"GET / HTTP/1.1\r
Content-Type: text/plain\r
Host: localhost:1234\r
Connection: close\r
\r
foobar"

While we’re at it, let’s make the build-http-response function too:

(fn build-http-response [status reason ?headers ?content]
  "Formats the HTTP response string as per the HTTP/1.1 spec."
  (string.format
   "%s %s %s\r\n%s\r\n%s"
   HTTP-VERSION
   (tostring status)
   reason
   (or (headers->string ?headers) "")
   (or ?content "")))

Same as before, string.format, filling missing parts, no new functions used:

>> (build-http-response
    200 "OK"
    {:connection "close"
     :content-type "application/json"}
    "{\"foo\": \"bar\"}")
"HTTP/1.1 200 OK\r
Content-Type: application/json\r
Connection: close\r
\r
{\"foo\": \"bar\"}"

Now, we can parse these using our Reader objects.

HTTP Parsing

Let’s start with parse-http-response, as we’re more likely to parse responses, rather than requests, as this library is mainly a client. We’ll write parse-http-request as well, but just for fun.

(fn parse-http-response [src read-fn]
  (let [status (read-response-status-line src read-fn)
        headers (read-headers src read-fn)]
    (doto status
      (tset :headers headers)
      (tset :body src))))

Obviously, this isn’t all we really need, but it’s a good start. First, let’s parse the status line, i.e. the first line of the response: HTTP/1.1 200 OK

(fn parse-response-status-line [status]
  ((fn loop [reader fields res]
     (case fields
       [field & fields]
       (let [part (reader)]
         (loop reader fields
               (case field
                 :protocol-version
                 (let [(name major minor) (part:match "([^/]+)/(%d).(%d)")]
                   (doto res
                     (tset field {: name :major (tonumber major) :minor (tonumber minor)})))
                 _ (doto res
                     (tset field (utils.as-data part))))
               ))
       _
       (let [reason (-> "%s/%s.%s +%s +"
                        (string.format
                         res.protocol-version.name
                         res.protocol-version.major
                         res.protocol-version.minor
                         res.status)
                        (status:gsub ""))]
         (doto res
           (tset :reason-phrase reason)))))
   (status:gmatch "([^ ]+)")
   [:protocol-version :status]
   {}))

(fn read-response-status-line [src read-fn]
  (parse-response-status-line (read-fn src :*l)))

You may be wondering, why these functions accept read-fn. Well, this is something we need to do because we’ll later pass in a socket-channel as a stream.

With these functions, we can read and parse the status line:

(read-response-status-line
 (reader.string-reader "HTTP/1.1 200 OK\r\n")
 (fn [src pattern] (src:read pattern)))
{:protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

You may also wonder, why the loop doesn’t just continue to read the reason-phrase, instead relying on status:gsub. Well, reason can include spaces:

>> (read-response-status-line
    (reader.string-reader "HTTP/1.1 404 NOT FOUND\r\n")
    (fn [src pattern] (src:read pattern)))
{:protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "NOT FOUND"
 :status 404}

Onto headers:

(fn parse-header [line]
  (case (line:match " *([^:]+) *: *(.*)")
    (header value) (values header value)))

(fn read-headers [src read-fn ?headers]
  (let [headers (or ?headers {})
        line (read-fn src :*l)]
    (case line
      (where (or "\r" ""))
      headers
      _ (read-headers
         src
         read-fn
         (case (parse-header (or line ""))
           (header value)
           (doto headers (tset header value)))))))

This one is tail-recursive, accumulating parsed headers in the ?headers table:

>> (read-headers
    (reader.string-reader "Content-Type: application/json\r\nConnection: close\r\n\n\r")
    (fn [src pattern] (src:read pattern)))
{:Connection "close" :Content-Type "application/json"}

Finally, for the body, we just return the reader as it is, as we’ve already read from it all the necessary data, and the rest are the contents of the request. Together, we get a fully parsed response:

>> (parse-http-response
    (reader.string-reader "HTTP/1.1 200 OK\r
Content-Type: application/json\r
Connection: close\r
\r
{\"foo\": \"bar\"}")
    (fn [src pattern] (src:read pattern))
    )
{:body #<Reader: 0x55a54832b810>
 :headers {:Connection "close" :Content-Type "application/json"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

We can ensure that the parsed response can be encoded back:

>> (let [req (build-http-response 200 "OK" {:connection :close} "foobar")
         rdr (reader.string-reader req)
         {: body : headers : reason-phrase : status}
         (parse-http-response rdr (fn [src pattern] (src:read pattern)) {:as :raw})]
     (= req (build-http-response status reason-phrase headers body)))
true

Now, let’s parse the request for completeness:

(fn parse-request-status-line [status]
  ((fn loop [reader fields res]
     (case fields
       [field & fields]
       (let [part (reader)]
         (loop reader fields
               (doto res
                 (tset field (utils.as-data part)))))
       _ res))
   (status:gmatch "([^ ]+)")
   [:method :path :http-version]
   {}))

(fn read-request-status-line [src read-fn]
  (parse-request-status-line (read-fn src :*l)))

(fn parse-http-request [src read-fn]
  (let [status (read-request-status-line src read-fn)
        headers (read-headers src read-fn)]
    (doto status
      (tset :headers headers)
      (tset :content (read-fn src :*a)))))

It’s similar, but we need to parse the status line a bit differently. Same as before, let’s ensure that we can do both:

>> (let [req (build-http-request :get "/" {:connection :close} "foobar")
       rdr (reader.string-reader req)
       {: headers : method : path : content}
       (parse-http-request rdr (fn [src pattern] (src:read pattern)) rdr)]
   (= req (build-http-request method path headers content)))
true

Now, we can start implementing the client itself!

Async.fnl

For communication, we’ll use already existing TCP channels, that enclose socket objects. Although, we’ll need a small modification to the async.fnl library, as currently it always reads data in chunks of fixed size.

Here’s the function that returns a socket channel:

(fn set-chunk-size [self pattern-or-size]
  (set self.chunk-size pattern-or-size))

(fn socket-channel [client xform err-handler]
  (let [recv (chan 1024 xform err-handler)
        resp (chan 1024 xform err-handler)
        ready (chan)
        close (fn [self] (recv:close!) (resp:close!) (set self.closed true))
        c (-> {:puts recv.puts
               :takes resp.takes
               :put! (fn [_  val handler enqueue?]
                       (recv:put! val handler enqueue?))
               :take! (fn [_ handler enqueue?]
                        (offer! ready :ready)
                        (resp:take! handler enqueue?))
               :close! close
               :close close
               :chunk-size 1024
               :set-chunk-size set-chunk-size}
              (setmetatable
               {:__index Channel
                :__name "SocketChannel"
                :__fennelview
                #(.. "#<" (: (tostring $) :gsub "table:" "SocketChannel:") ">")}))]
    (go-loop [data (<! recv) i 0]
      (when (not= nil data)
        (case (socket.select nil [client] 0)
          (_ [s])
          (case (s:send data i)
            (nil :timeout j)
            (do (<! (timeout 10)) (recur data j))
            (nil :closed)
            (do (s:close) (close! c))
            _ (recur (<! recv) 0))
          _ (do (<! (timeout 10))
                (recur data i)))))
    (go-loop [wait? true]
      (when wait?
        (<! ready))
      (case (client:receive c.chunk-size)
        data
        (do (>! resp data) (recur true))
        (nil :closed "")
        (do (client:close) (close! c))
        (nil :closed data)
        (do (client:close) (>! resp data) (close! c))
        (nil :timeout "")
        (do (<! (timeout 10)) (recur false))
        (nil :timeout data)
        (do (>! resp data) (<! (timeout 10)) (recur true))))
    c))

It’s a lot of code, but in reality, it isn’t that complicated. The socket-channel function is responsible for three things:

Creating a channel, that has separate queues for puts and takes, such that when we put something to a channel, we won’t be able to read it back immediately.
Creating an asynchronous loop that waits for data from the socket, and puts it into the receive queue
Creating another asynchronous loop, that waits data on the channel, and puts it into the socket.

These loops account for timeouts, as we’re going to set the socket timeout to 0, to ensure non-blocking behavior. Both asynchronous loops employ their own timeout handling.

The resulting object is a channel, but with some custom methods to account for its duplex nature. And, there are also other properties, such as chunk-size and set-chunk-size which we’re going to use with our streams.

Notice, how the second go-loop has a wait? argument. Because the channel is buffered, it will happily read everything until the buffer is full, which is not ideal, since we may want to change the chunk-size later. So we need to delay the actual read, which we do in the take! method using offer!.

Finally, the tcp.chan function acts as a constructor for TCP channels:

(fn tcp.chan [{: host : port} xform err-handler]
  "Creates a channel that connects to a socket via `host` and `port`.
Optionally accepts a transducer `xform`, and an error handler.
`err-handler` must be a fn of one argument - if an exception occurs
during transformation it will be called with the thrown value as an
argument, and any non-nil return value will be placed in the channel.
The read pattern f a socket can be controlled with the
`set-chunk-size` method."
  (assert socket "tcp module requires luasocket")
  (let [host (or host :localhost)]
    (match-try (socket.connect host port)
      client (client:settimeout 0)
      _ (socket-channel client xform err-handler)
      (catch (nil err) (error err)))))

This is all we need on the async.fnl side of things. No more patching Luasocket’s HTTP client.

Now we can start implementing our own client.

HTTP client

We already have most of the things we need, but we need to change the parse-http-response a bit. Remember, our function returns the body as the Reader, but our source is actually a channel. We can account for that by passing in an appropriate read-fn function but our users shouldn’t deal with that. Instead, the body should provide proper Reader methods, so let’s create a body-reader:

(fn body-reader [src read-fn]
  (var buffer "")
  (make-reader
   src
   {:read-bytes (fn [src pattern]
                  (let [rdr (reader.string-reader buffer)
                        buffer-content (rdr:read pattern)]
                    (case pattern
                      (where n (= :number (type n)))
                      (let [len (if buffer-content (length buffer-content) 0)
                            read-more? (< len n)]
                        (set buffer (string.sub buffer (+ len 1)))
                        (if read-more?
                            (if buffer-content
                                (.. buffer-content (or (read-fn src (- n len)) ""))
                                (read-fn src (- n len)))
                            buffer-content))
                      (where (or :*l :l))
                      (let [read-more? (not (buffer:find "\n"))]
                        (when buffer-content
                          (set buffer (string.sub buffer (+ (length buffer-content) 2))))
                        (if read-more?
                            (if buffer-content
                                (.. buffer-content (or (read-fn src pattern) ""))
                                (read-fn src pattern))
                            buffer-content))
                      (where (or :*a :a))
                      (do (set buffer "")
                          (case (read-fn src pattern)
                            nil (when buffer-content
                                  buffer-content)
                            data (.. (or buffer-content "") data)))
                      _ (error (tostring pattern)))))
    :read-line (fn [src]
                 (let [rdr (reader.string-reader buffer)
                       buffer-content (rdr:read :*l)
                       read-more? (not (buffer:find "\n"))]
                   (when buffer-content
                     (set buffer (string.sub buffer (+ (length buffer-content) 2))))
                   (if read-more?
                       (if buffer-content
                           (.. buffer-content (or (read-fn src :*l) ""))
                           (read-fn src :*l))
                       buffer-content)))
    :close (fn [src] (src:close))
    :peek (fn [src bytes]
            (assert (= :number (type bytes)) "expected number of bytes to peek")
            (let [rdr (reader.string-reader buffer)
                  content (or (rdr:read bytes) "")
                  len (length content)]
              (if (= bytes len)
                  content
                  (let [data (read-fn src (- bytes len))]
                    (set buffer (.. buffer (or data "")))
                    buffer))))}))

Now, that’s a lot of code again. The reason for that is the peek method - same as for strings, we need to implement a way to peek at data without modifying the reader, or at least hide the modification from the user. This is essentially what this function does - it creates a string buffer that we can act upon, accounting for possible data in it before actually reading from the channel itself.

With that function, we can update parse-http-response:

(fn parse-http-response [src read-fn {: as}]
  (let [status (read-response-status-line src read-fn)
        headers (read-headers src read-fn)
        stream (body-reader src read-fn)]
    (doto status
      (tset :headers headers)
      (tset :body
            (case as
              :raw (stream:read (or headers.Content-Length :*a))
              :stream stream
              _ (error (string.format "unsupported coersion method '%s'" as)))))))

As you can see, I’ve added another parameter with options. Right now, there’s only one option as that will choose how to return the body. Specifying {:as :raw} to this function will return the contents of the reader. And {:as :stream} will return the stream itself.

Now, we can write the request function:

(local http {})

(fn http.request [method url ?opts]
  "Makes a `method` request to the `url`, returns the parsed response,
containing a stream data of the response. The `method` is a string,
describing the HTTP method per the HTTP/1.1 spec. The `opts` is a
table containing the following keys:

- `:async?` - a boolean, whether the request should be asynchronous.
  The result is an instance of a `promise-chan`, and the body must
  be read inside of a `go` block.
- `:headers` - a table with the HTTP headers for the request
- `:body` - an optional string body.
- `:as` - how to coerce the output.

Several options available for the `as` key:

- `:stream` - the body will be a stream object with a `read` method.
- `:raw` - the body will be a string.
  This is the default value for `as`.

When supplying a non-string body, headers should contain a
\"content-length\" key. For a string body, if the \"content-length\"
header is missing it is automatically determined by calling the
`length` function, ohterwise no attempts at detecting content-length
are made."
  (let [{: host : port &as parsed} (http-parser.parse-url url)
        opts (collect [k v (pairs (or ?opts {}))
                       :into {:as :raw
                              :async? false}]
               k v)
        headers (collect [k v (pairs (or opts.headers {}))
                          :into {:host
                                 (.. host (if port (.. ":" port) ""))
                                 :content-length
                                 (case opts.body
                                   (where body (= :string (type body)))
                                   (length body))}]
                  k v)
        path (utils.format-path parsed)
        req (build-http-request method path headers opts.body)
        chan (tcp.chan parsed)]
    (if opts.async?
        (let [res (promise-chan)]
          (go (>! chan req)
              (>! res (http-parser.parse-http-response
                       chan
                       (make-read-fn <!)
                       opts)))
          res)
        (do (>!! chan req)
            (http-parser.parse-http-response
             chan
             (make-read-fn <!!)
             opts)))))

We’re still missing some important parts, but we’ll get to those in a bit. This function is long, but it’s not that complicated.

First, we parse the given url into its components, mainly the host and port. Then, we prepare the default opts table to include as key and async? key if none were passed.

Next, we prepare headers. The HTTP spec requires the Host header, so we put it in based on the parsed URL. The path part of the request is then formatted from the rest data from the parsed URL.

Finally, we build-http-request, using the method, path, headers and body if supplied. And create the tcp.chan using the parsed object that contains host and port needed for the constructor.

Then, if opts.async? is true, we create a promise-chan, which is like Clojure’s promise, which is only computed once, and then returns the same result over and over. We start the go thread, in which we asynchronously put the request into the socket channel, and then we asynchronously parse the response supplying our read-fn with the <! function to take from the channel asynchronously. And we return res - the promise channel I mentioned earlier.

If the opts.async? wasn’t true, we do all the same thing, but in a blocking way, using >!! and <!! to work with the channel.

Let’s look at parse-url:

(fn parse-authority [authority]
  (let [userinfo (authority:match "([^@]+)@")
        port (authority:match ":(%d+)")
        host (if userinfo
                 (authority:match (.. "@([^:]+)" (if port ":" "")))
                 (authority:match (.. "([^:]+)" (if port ":" ""))))]
    {: userinfo : port : host}))

(fn parse-url [url]
  (let [scheme (url:match "^([^:]+)://")
        {: host : port : userinfo}
        (parse-authority
         (if scheme
             (url:match "//([^/]+)/")
             (url:match "^([^/]+)/")))
        scheme (or scheme "http")
        port (or port (case scheme :https 443 :http 80))
        path (url:match "//[^/]+/([^?#]+)")
        query (url:match "%?([^#]+)#?")
        fragment (url:match "#([^?]+)%??")]
    {: scheme : host : port : userinfo : path : query : fragment}))

Nothing special, just some patterns to match components of the URL:

>> (parse-url "https://lua-users.org/?some-query-param=some-value")
{:host "lua-users.org"
 :port 443
 :query "some-query-param=some-value"
 :scheme "https"}

And the format-path function constructs the path back:

(fn format-path [{: path : query : fragment}]
  (.. "/" (or path "") (if query (.. "?" query) "") (if fragment (.. "?" fragment) "")))

Here’s what it does on the previously parsed URL:

>> (format-path {:host "lua-users.org"
 :port 443
 :query "some-query-param=some-value"
 :scheme "https"})
"/?some-query-param=some-value"

Finally, the mysterious make-read-fn function creates a function that will work with our socket-channel:

(fn make-read-fn [receive]
  "Returns a function that receives data from a socket by a given
`pattern`.  The `pattern` can be either `\"*l\"`, `\"*a\"`, or a
number of bytes to read."
  (fn [src pattern]
    (src:set-chunk-size pattern)
    (receive src)))

Since our channel does reading on demand, we can’t pass the chunk-size/pattern in, so we use the set-chunk-size method instead. Paired with delayed read, I’ve explained earlier, this works really well.

With all of that, we’re ready to do some requests!

>> (http.request :get "http://lua-users.org/" {:headers {:connection "close"} :as :raw})
{:body "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">

<html>

<head>
<title>lua-users.org</title>
<meta http-equiv=\"content-type\" content=\"text/html; charset=ISO-8859-1\">
<LINK TYPE=\"text/css\" REL=\"stylesheet\" HREF=\"/styles/main.css\">
</head>

<body>

<img src=\"/images/lua-users-org.png\" alt=\"lua-users.org logo\">

<br clear=all>
<p>
<b>lua-users.org</b> is an internet site for and by users of the programming language Lua.
</p>
<p>
To find out what Lua is and for documentation, downloads, mailing list info, etc. please visit the <a href=\"http://www.lua.org/\">official Lua web site</a>.  The <b>lua-users.org</b> web is not affiliated with Tecgraf.
</p>
<p>
Currently available resources:
</p>
<ul>
    <li> <a href=\"/wiki/\">lua-users wiki</a> - collaborative web site
    <li> <a href=\"/lists/lua-l/\">lua-l archive</a> - searchable web archive of
        the official Lua mailing list
    <li> <a href=\"MiniCharter.html\">lua-users.org mini charter</a>
    <li> <a href=\"Acknowledgements.html\">acknowledgements</a>
</ul>

</body>

</html>
"
 :headers {:Accept-Ranges "bytes"
           :Connection "close"
           :Content-Length "1055"
           :Content-Type "text/html"
           :Date "Mon, 15 Jul 2024 12:03:00 GMT"
           :ETag "\"61acad05-41f\""
           :Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
           :Server "nginx/1.20.1"
           :Strict-Transport-Security "max-age=0;"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

Or, we can use :as :stream, and process the stream later:

(local resp (http.request :get "http://lua-users.org/" {:headers {:connection "close"} :as :stream}))
nil
>> resp
{:body #<Reader: 0x55d7090707c0>
 :headers {:Accept-Ranges "bytes"
           :Connection "close"
           :Content-Length "1055"
           :Content-Type "text/html"
           :Date "Mon, 15 Jul 2024 12:03:47 GMT"
           :ETag "\"61acad05-41f\""
           :Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
           :Server "nginx/1.20.1"
           :Strict-Transport-Security "max-age=0;"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}
>> (resp.body:read :*l)
"<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">"

However, writing (http.request :get ...) all the time, is a bit tedious. Let’s instead define functions for each HTTP method. And, to do so, we’ll use macros, because I’m too lazy, and I don’t want to write documentation for each function:

(macro define-http-method [method]
  "Defines an HTTP method for the given `method`."
  `(fn ,(sym (.. :http. (tostring method)))
     [url# opts#]
     {:fnl/arglist [url opts]
      :fnl/docstring ,(.. "Makes a `" (string.upper (tostring method))
                          "` request to the `url`, returns the parsed response,
containing a stream data of the response. The `method` is a string,
describing the HTTP method per the HTTP/1.1 spec. The `opts` is a
table containing the following keys:

- `:async?` - a boolean, whether the request should be asynchronous.
  The result is an instance of a `promise-chan`, and the body must
  be read inside of a `go` block.
- `:headers` - a table with the HTTP headers for the request
- `:body` - an optional string body.
- `:as` - how to coerce the output.

Several options available for the `as` key:

- `:stream` - the body will be a stream object with a `read` method.
- `:raw` - the body will be a string.
  This is the default value for `as`.

When supplying a non-string body, headers should contain a
\"content-length\" key. For a string body, if the \"content-length\"
header is missing it is automatically determined by calling the
`length` function, ohterwise no attempts at detecting content-length
are made.")}
     (http.request ,(tostring method) url# opts#)))

(define-http-method get)
(define-http-method post)
(define-http-method put)
(define-http-method patch)
(define-http-method options)
(define-http-method trace)
(define-http-method head)
(define-http-method delete)
(define-http-method connect)

Now, we can call any HTTP method by just writing (http.method-name ...):

>> (http.get "http://lua-users.org/" {:headers {:connection "close"} :as :stream})
{:body #<Reader: 0x55d7095dbb20>
 :headers {:Accept-Ranges "bytes"
           :Connection "close"
           :Content-Length "1055"
           :Content-Type "text/html"
           :Date "Mon, 15 Jul 2024 12:08:28 GMT"
           :ETag "\"61acad05-41f\""
           :Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
           :Server "nginx/1.20.1"
           :Strict-Transport-Security "max-age=0;"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

And we can even see the documentation for each method in our editor:

Now, that body-reader, with a custom buffer, seemed a bit out of the way, so let’s use it for good.

JSON parsing

Let’s extend our parse-http-response function with another as method:

;; ---- 8< ----
      (tset :body
            (case as
              :raw (stream:read (or parsed-headers.Content-Length :*a))
              :json (json.parse stream)
              :stream stream
              _ (error (string.format "unsupported coersion method '%s'" as))))
;; ---- 8< ----

Now, we can implement a simple JSON parser:

(fn parse [rdr]
  "Accepts a reader object `rdr`, that supports `peek`, and `read`
methods.  Parses the contents to a Lua table."
  ((fn loop []
     (case (rdr:peek 1)
       "{" (parse-obj rdr loop)
       "[" (parse-arr rdr loop)
       "\"" (parse-string rdr)
       (where "t" (= "true" (rdr:peek 4))) (do (rdr:read 4) true)
       (where "f" (= "false" (rdr:peek 5))) (do (rdr:read 5) false)
       (where "n" (= "null" (rdr:peek 4))) (do (rdr:read 4) nil)
       (where c (c:match "[ \t\n]")) (loop (skip-space rdr))
       (where n (n:match "[-0-9]")) (parse-num rdr)
       nil (error "JSON parse error: end of stream")
       _ (error (string.format "JSON parse error: unexpected token ('%s' (code %d))" _ (_:byte)))))))

First, let’s look at parse-string:

(fn parse-string [rdr]
  (rdr:read 1)
  ((fn loop [chars escaped?]
     (case (rdr:read 1)
       "\\" (loop chars (not escaped?))
       "\"" (if escaped?
                (loop (.. chars "\"") false)
                chars)
       _ (loop (.. chars _) false)
       ))
   "" false))

This function handles strings, and escaped patterns withing set strings, like nested strings:

>> (parse (reader.string-reader "\"foo \\\"bar\\\"\""))
"foo \"bar\""

Now, let’s look at the parse-arr

(fn parse-arr [rdr parse]
  (rdr:read 1)
  ((fn loop [arr]
     (skip-space rdr)
     (case (rdr:peek 1)
       "]" (do (rdr:read 1) arr)
       _ (let [val (parse)]
           (tset arr (+ 1 (length arr)) val)
           (skip-space rdr)
           (case (rdr:peek 1)
             "," (do (rdr:read 1) (loop arr))
             "]" (do (rdr:read 1) arr)
             _ (error (.. "JSON parse error: expected ',' or ']' after the value: " (json val)))))))
   []))

This function accepts the parse callback to parse nested values, so if we try to parse an array of strings, we’ll call parse on each value, and then expect either a colon or the closing bracket to mark the end of the array.

>> (parse (reader.string-reader "[\"a\", \"b\"]"))
["a" "b"]

The parse-obj works for objects in a similar way:

(fn parse-obj [rdr parse]
  (rdr:read 1)
  ((fn loop [obj]
     (skip-space rdr)
     (case (rdr:peek 1)
       "}" (do (rdr:read 1) obj)
       _ (let [key (parse)]
           (skip-space rdr)
           (case (rdr:peek 1)
             ":" (let [_ (rdr:read 1)
                       value (parse)]
                   (tset obj key value)
                   (skip-space rdr)
                   (case (rdr:peek 1)
                     "," (do (rdr:read 1) (loop obj))
                     "}" (do (rdr:read 1) obj)
                     _ (error (.. "JSON parse error: expected ',' or '}' after the value: " (json value)))))
             _ (error (.. "JSON parse error: expected colon after the key: " (json key)))))))
   {}))

Except, it waits for the : after each key, and then the value:

>> (parse (reader.string-reader "{\"arr\": [\"a\", \"b\"]}"))
{:arr ["a" "b"]}

Finally, there’s the parse-num and code to parse Booleans, and null:

(fn parse-num [rdr]
  ((fn loop [numbers]
     (case (rdr:peek 1)
       (where n (n:match "[-0-9.eE+]"))
       (do (rdr:read 1) (loop (.. numbers n)))
       _ (tonumber numbers)))
   (rdr:read 1)))

With that, we can call http.get passing it {:as :json} as opts:

>> (http.get "http://httpbin.org/get" {:as :json})
{:body {:args {}
        :headers {:Host "httpbin.org"
                  :X-Amzn-Trace-Id "Root=1-6695183d-27c59aff535ae9f11df8c98d"}
        :origin "185.169.167.87"
        :url "http://httpbin.org/get"}
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "199"
           :Content-Type "application/json"
           :Date "Mon, 15 Jul 2024 12:38:21 GMT"
           :Server "gunicorn/19.9.0"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

And our body was successfully parsed.

Asynchronous HTTP calls

Now, this client is asynchronous, meaning we can download lots of resources simultaneously. I’m not saying in parallel, because there are no threads in Lua, but still, we can process several requests at once using our asynchronous library.

Let’s define a simple time macro first:

(macro time [expr]
  `(let [clock# socket.gettime
         start# (clock#)
         res# ,expr
         end# (clock#)]
     (print (.. "Elapsed " (* (- end# start#) 1000) " ms"))
     res#))

With it, we can measure the execution time of a body. Let’s make three asynchronous requests, that upon completion will count the lengths of the body for each, and return their sum:

>> (time
    (let [headers {:connection "close"}
          a (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
          b (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
          c (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
          lengths (chan 3)]
      (go (let [resp (<! a)]
            (>! lengths (length resp.body))))
      (go (let [resp (<! b)]
            (>! lengths (length resp.body))))
      (go (let [resp (<! c)]
            (>! lengths (length resp.body))))
      (+ (<!! lengths) (<!! lengths) (<!! lengths))))
Elapsed 530.24077415466 ms
597

In comparison, three synchronous requests, take almost twice as much time:

>> (time
    (let [headers {:connection "close"}
          a (http.get "http://httpbin.org/get" {:headers headers :as :raw})
          b (http.get "http://httpbin.org/get" {:headers headers :as :raw})
          c (http.get "http://httpbin.org/get" {:headers headers :as :raw})]
      (+ (length a.body) (length b.body) (length c.body))))
Elapsed 1047.5831031799 ms
597

For the reference, here’s Luasocket’s HTTP client:

>> (local lhttp (require :socket.http))
nil
>> (time
    (let [a (lhttp.request "http://httpbin.org/get")
          b (lhttp.request "http://httpbin.org/get")
          c (lhttp.request "http://httpbin.org/get")]
      (+ (length a) (length b) (length c))))
Elapsed 1919.6438789368 ms
717

(Note, the resulting length is different, because Luasocket specifies its user-agent, and httpbin includes it in the answer.)

Further development

Now, of course, that’s not all that needs to be implemented. In its current state, this client doesn’t support responses that do not specify the Content-Length or produce a response body in chunked encoding. HTTPS is out of the question for now too. Most of the code just takes a “happy path” without much thought given to errors or more complicated scenarios. But it’s fine, I can always make it better down the line.

But all in all, it was a fun experiment, and I’m feeling better, as I finally have some use for async.fnl. You can find the source code for this project on my GitLab profile. And by the way, even though it is written in Fennel and uses some Fennel libraries, like async.fnl, this library works with ordinary Lua:

Lua 5.4.4  Copyright (C) 1994-2022 Lua.org, PUC-Rio
> http = require "http"
> http.get("http://httpbin.org/get", {headers = {connection = "close"}}).body
{
  "args": {},
  "headers": {
    "Host": "httpbin.org",
    "X-Amzn-Trace-Id": "Root=1-6695424f-01e0510f2ab49b9f46bd51dc"
  },
  "origin": "185.165.163.78",
  "url": "http://httpbin.org/get"
}
> http.get("http://httpbin.org/get", {headers = {connection = "close"}, as = "stream"}).body
Reader: 0x55a6c0eff710

So if you like what you see, but you don’t use Fennel, you can still try out this library. It just needs to be compiled first.

Anyway, that’s all from me, so… thanks for reading!

Comment via email

Game4 and Autumn Lisp Game Jam

Thu, 19 Oct 2023 18:42:00 +0300

You might be wondering why there were no posts on the game development marathon I’ve been doing. Maybe you’d thought that I gave up after the admittedly underwhelming game3 having no actual game just some basic movement. And yeah, I felt burned up a lot, and considered skipping a month maybe - but then this happened.

So I thought - well, I just make the game during the game jam instead, and rest up until the jam begins. Yes, it means I will technically have less time to do this, but I did spend even less time on game dev last month, so it’s fine. I also got an unplanned vacation, so I guess I’ll actually have free time too.

Though, I lied a bit. The other reason I didn’t do anything is also because the next genre I wanted to work in is a puzzle platformer. And honestly, I don’t know what kind of puzzles to make. None seem appealing.

So after a bit of consideration, I’ve decided that can try to make something like a Lode Runner type of game. I already have the platformer engine from the very first game in this series, and originally I’ve planned that I would make a really simple 2D platformer, then in the subsequent game I will refine and improve it. Similarly, with the top-down games - first make a simple dungeon crawler, then take the blueprint and make an adventure with it. Unfortunately, the second game was a flop, and thus if I am to make the fifth game I’ll have to do everything from scratch.

So I’ve been thinking how I should handle the game. I guess I’m going to ditch the camera here, so the level fits the screen. It will make it a bit easier to reason about, and I guess I can always add camera back later if I really need to.

AI would be an issue though - I’ll need to implement some kind of path-finding for the 2D grid with gravity in mind. In other words, enemies should understand that they can jump down from cliffs if it will bring them on the same elevation with the player. Additionally, enemies shouldn’t just follow the shortest path to the player, as it will make them very predictable.

And if I have time, I can try putting on a twist to the formula. Not sure what kind of twist yet, but we’ll see I guess.

The jam starts in ~12 hours, which unfortunately means that it will begin while my work week isn’t yet ended. So I will have one day less already. Additionally, right now I’m in another city, and will lose another day on train to get back home. So this leaves me with 8 days instead of 10.

Hopefully it will not be an issue. Wish me luck!

Comment via email

Game3 - Results

Sun, 01 Oct 2023 17:12:00 +0300

I gave myself 1 extra day on this game because in total I was only able to work on this game for 5 days, two of which were on the train. So this game is now done, although I couldn’t make any sounds for it. Designing explosion and blaster sounds that sound good is hard. The game isn’t really complete, and you get practically invincible once you get 3 power-ups. As wasn’t able to do much this time, so a lot of sprites are left unused. Anyhow, here’s the game:

Play Game3 - Shoot 'Em Up on itch.io

Since the last post, I changed a lot. First, bump is back and all collisions are handled through it again.

I made some more weapons, the default one has a rapid-fire rate, but low damage:

Your browser does not support the video tag.

There are three more weapons - laser gun, magic gun, and rocket gun. Each of these has different damage values. Originally I planned to implement the overheat mechanic, but ran out of time and passion to do that.

It was fun implementing the flame that appears when the ship is taken down. It is done as a particle effect, as the ship simply constantly emits flame particles when burning, and they get removed once their animation ends. Similar to how the bullets interact with enemies.

If you want to download this game, here’s the cartridge:

The next game on my list is a puzzle platformer. For now, I’m not sure if I will have time or passion to do it, and I honestly don’t know what types of puzzles should it have. We’ll see. I already have a pretty decent engine for a platforming game, so it’s a good time to refine it.

Comment via email

Game3 w2/3

Sun, 24 Sep 2023 20:55:00 +0300

Another week passed, but I didn’t manage to squeeze out much spare time to make any significant progress with the game. Not like I didn’t do anything, but the game is far from being playable. This week I, unfortunately, could spend only a single day working on a game, because every day after work I had some personal stuff to take care of, which took all of my after-work hours. But it doesn’t matter, the challenge is a challenge! Hopefully, the next week will be a bit less busy. Although the stuff happening right now is very exciting and very important, so if the next week is busy again - so be it.

Anyway, here’s the progress so far:

Your browser does not support the video tag.

It’s a prototype. I’m experimenting with enemy patterns, like a simple sine wave, or tracking enemies. No collision has been implemented yet. This is my plan for the next and final week. As well as doing more animation, as the ship is just a cardboard right now, making weapons work, and not just fly with you, and implementing a scoring system. Probably that’s the most I’ll be able to do.

Another thing I want to look into is rotating sprites, as I want to spawn meteorites semi-randomly after some time passes in the game, so the game becomes harder. These meteorites will shatter, and smaller parts will also damage the player, making it harder to maneuver. But that’s probably the least prioritized thing as of this moment.

Comment via email

Game3 W?/4

Sat, 16 Sep 2023 01:59:00 +0300

I’m working on the next game, although I gave myself an extended period of rest. After failing to complete the last one I needed to distract myself a bit from this activity, so I worked a bit on Fennel stuff, played some games, and I think I’m back in business by now. I have an interesting thing going on with my productivity - it comes in waves. What I mean is that I can go onto programming frenzy for several months, and then once I feel burned out, I can just play games for another full month straight. Though I get tired of games too, and if by that time I feel that I’m not ready for programming yet, I can read some technical books, or even just some manga if I still feel too tired to be productive.

So this five month marathon is a bit hard for me, as I basically have to give up on long periods of rest, and juggle my free time to actually have some time to work on these projects. While these games are nothing special, I feel like I’m still learning new stuff, and thus this is beneficial for me still. Though, at some I actually thought of dropping the whole idea, and starting a more complex project that I was thinking about for a lot of time already. But I decided to leave it for the next year, as I still need a lot to learn. But, back on the topic.

This post actually marks the end of only the second week of the current game. How come, you might ask? To put it simply, I miscalculated a bit, and in reality I had another week on the previous game by the time I dropped it. So, technically, I could probably finish that game, but the burnout got me, so that’s that.

As for this game, I have a little to show. As it is a side-scrolling shooter, I designed a basic ship with some shooting modules, and some powerup icons. I considered to not making any graphics for this one, and doing everything with just primitives, like triangles, circles and so on, but after trying to draw a ship like this I dropped the idea and returned to Pixel Studio. Here are the results so far:

The ship turned out to be a bit too big, but I decided that I can make a game with larger assets for a change. So the enemy also uses 4 sprites, instead of just one, as in my previous games.

And here are the enemies animated:

Movement animations

Explosion animations

Obviously, I need to add more enemy variety, with different patterns, but designing enemies is hard. BTW, if you’re curious how I draw things on the phone, here’s a time-lapse:

Your browser does not support the video tag.

Though, I usually don’t do it in the landscape mode, portrait mode works better for me, especially with a stylus. This process was recorded in landscape just for viewing convenience.

On the technical side of things, I already implemented a basic movement with linear interpolation for the flying modules, and implemented some enemy patterns, but there’s no actual interactions yet. I’m kinda torn on the idea of including bump.lua again, as I feel like that this game can be done without any collision library. At the same time, having a collision library may make things much simpler, so I’m still thinking how to approach this game.

For the record, I basically dropped the idea of making any kind of music for these games because I just can’t get myself to work on it. I do plan to take some courses on music composition for future projects, though. Sound effects are challenging enough already by themselves.

That’s all for this week! Hopefully, I’ll manage to finish the game this time, though I already lost one week. Let’s say, this is week 1/3 instead of 2/4 shall we? Feels less demotivating this way.

Comment via email

Fennel libraries as single files

Sun, 27 Aug 2023 20:45:00 +0300

I’m pleased to announce that most of my libraries for fennel are now shipped as single files! It was a long-standing issue with Fennel for me, as there was no clear way to ship both macros and functions in the same file, but after some experimentation, I figured out a way. After a bit of testing with my new library for asynchronous programming I can say that it works well enough to be used in most other libraries. Although the implementation is a bit rough and manual, that’s what I’ll try to describe here today.

But first, let’s talk modules. This will be a long post, so buckle up.

Modules in various programming languages

I find it fascinating that a lot of languages have some kind of a module system, yet most of them hide it from the user behind some special keywords and don’t allow us to operate on modules as we can do with other objects. Lua is not one of these languages - its module system is ingenious and yet very simple. First, let’s look at some other languages so you can understand what I’m getting at.

For example, here’s Elixir, a language I sometimes tinker with. I’m using an interactive elixir shell, but it’ll be the same even if I were to put this module into a file:

iex(1)> defmodule Lib do
...(1)>   def foo(), do: "foo"
...(1)>   def bar(), do: "bar"
...(1)> end
{:module, Lib,
 <<70, 79, 82, 49, 0, 0, 5, 180, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 169,
   0, 0, 0, 18, 10, 69, 108, 105, 120, 105, 114, 46, 70, 111, 111, 8, 95, 95,
   105, 110, 102, 111, 95, 95, 10, 97, 116, ...>>, {:bar, 0}}
iex(2)> Lib
Lib
iex(3)> Lib.foo()
"foo"

You can see that after the module was defined, we can see a tuple with some keywords, the module symbol, and other stuff. If we try to examine Lib in the shell it just prints Lib. Yet, we can call the function foo from this module via the dot syntax: Lib.foo()

If we try to inspect what Lib actually is, we can see that it has a bunch of information but, at least to me, this tells almost nothing of substance:

iex(4)> i Lib
Term
  Lib
Data type
  Atom
Module bytecode
  []
Source
  iex
Version
  [249975233796634777878553557963106734802]
Compile options
  [:no_spawn_compiler_process, :from_core, :no_core_prepare, :no_auto_import]
Description
  Call Lib.module_info() to access metadata.
Raw representation
  :"Elixir.Lib"
Reference modules
  Module, Atom
Implemented protocols
  IEx.Info, Inspect, List.Chars, String.Chars

Let’s look at another language, that I don’t often tinker with, but a lot of other people do - Python. This time I’ll put this into a file, as I don’t know the shell capabilities of Python, and if it is possible to do stuff like in Elixir. Here’s the same module:

# lib.py
def foo ():
    return "foo"

def bar ():
    return "bar"

Now, if we import it, we can at least see some indication of what lib is when we try to inspect it:

>>> import lib
>>> lib
<module 'lib' from '/tmp/lib.py'>
>>> lib.foo()
'foo'

What is this <module 'lib' from '/tmp/lib.py'> thing actually? Python defines different keywords for working with modules, like from, import, as, and maybe others, as well as a special set of functions to query the module, like dir. But the module itself is probably some kind of an object:

>>> type(lib)
<class 'module'>

Here’s another language, that I know better than Python or Elixir - Clojure. In Clojure, similar to Elixir, you can define modules in the REPL, so let’s do this:

user> (ns lib)
nil
lib> (defn foo [] "foo")
#'lib/foo
lib> (defn bar [] "bar")
#'lib/bar
lib> (in-ns 'user)
#namespace[user]
user> (require 'lib)
nil
user> (lib/foo)
"foo"

Once again, we can see that the namespace is some kind of #namespace[user] thingy. If we try to reference the namespace as is in the REPL, we get an error:

user> lib
Syntax error compiling at (*cider-repl blog/posts:localhost:44133(clj)*:0:0).
Unable to resolve symbol: lib in this context

However, Clojure provides us with a set of ns-* functions to work with namespaces. For instance, we can get all interned symbols of the namespace as a map:

user> (ns-interns 'lib)
{bar #'lib/bar, foo #'lib/foo}

We can even go back to the lib module, define a private function, return to the user module, and inspect the lib module again:

user> (in-ns 'lib)
#namespace[lib]
lib> (defn ^:private baz [] "baz")
#'lib/baz
lib> (in-ns 'user)
#namespace[user]
user> (ns-interns 'lib)
{baz #'lib/baz, bar #'lib/bar, foo #'lib/foo}
user> (ns-publics 'lib)
{bar #'lib/bar, foo #'lib/foo}

Yet, modules are something hidden from us, we usually don’t operate on them directly.

So that brings us to Fennel, and therefore Lua. I will demonstrate things in Fennel just for better clarity, as it provides better inspection capabilities.

What separates Lua runtime from the rest is that its modules are just tables. When we’re writing a library, or just a module in our application, the module is just a file on the file system, and at the end of that file the last statement returns a table (or a function):

;; lib.fnl
(fn foo [] "foo")
(fn bar [] "bar")
(fn baz [] "baz")

{:foo foo
 :bar bar}

In this case, the table holds references to local functions, and thus the baz function is completely private. When we require this module with the require function it somehow finds the file of this module on disc, loads it, evaluates the top level, and returns the table, which we just then store in a variable, same as any other value:

>> (local lib (require "lib"))
>> (lib.foo)
"foo"

If we try to reference lib here, we’ll see that it is just a table, nothing special:

>> lib
{:bar #<function: 0x55f4cfecd350>
 :foo #<function: 0x55f4cff20610>}
>> (getmetatable lib)
nil

We can check if it has a metatable that hides some module-related trickery, but it’s not. It’s a plain ol’ Lua table.

Lua module system

But Lua doesn’t stop there. I mentioned that require evaluates the top level, so if we put something like (print "hi!") into a module, and require it, we’ll see the text being printed:

;; qux.fnl
(fn qux [] "qux")
(print "hi!")
{:qux qux}

Here’s how it looks in the Fennel REPL:

>> (require :qux)
hi!
{:qux #<function: 0x55f4cfe7a1e0>}	"./qux.fnl"

But if we do a second require the print won’t happen:

>> (require :qux)
{:qux #<function: 0x55f4cfe7a1e0>}

This is because while the modules are plain tables, the require function is actually smarter than just read+eval. Module loading process in Lua ensures that modules are only loaded once, and are cached in a special package table.

`package.loaded` - caching module definitions

>> package.loaded.qux
{:qux #<function: 0x55f4cfe7a1e0>}

So each time we call (require :something) it first looks if package.loaded.something exists, and if it is, it just returns the value from there. This way, we can unload any module, just by setting a key in the package.loaded to nil, and then require will re-evaluate our module. Fennel actually provides a REPL command for that, called reload.

However, it’s still not that simple. Lua’s require is actually extensible. If the module was not found in the package.loaded the next place require checks is package.preload.

`package.preload` - preloading modules via anonymous functions

This table stores modules in a similar way, except instead of storing tables, it stores functions that when called will load a module.

We can utilize this in the REPL:

>> (fn package.preload.my-lib []
     (print "loading my-lib")
     (fn hello [name]
       (.. "hi " name "!"))
     {:hello hello})
#<function: 0x55f4d4a23d50>
>> package.preload.my-lib
#<function: 0x55f4d4a23d50>

I’ve created a function in that table, that when called will signal that the module is loaded, define a local function, and return a table, referencing that function. So it’s essentially the same as the file approach, except it is stored in a function, which execution is delayed until someone tries to require the module. Lua was meant to be able to work in filesystem-less environments, and this is one of the solutions for storing libraries.

When we require the my-lib module, we see the loading my-lib message, but instead of providing a second value with the file from where the module was read, the value is :preload::

>> (require :my-lib)
loading my-lib
{:hello #<function: 0x55f4d4a4ecf0>}	":preload:"

This indicates that the library was indeed loaded from the package.preload table. We can then use it as normal:

>> (local my-lib (require :my-lib))
nil
>> (my-lib.hello "Bob")
"hi Bob!"

This already gives us some power, and Fennel utilizes package.preload to provide fully self-contained AOT-compiled Lua files that can be shipped without any external dependencies. Yet, there’s a final step in this system that we can discuss.

`package.searchers` - extending the module system

If the module was found neither in the package.loaded nor in the package.preload, the require function starts using special package.searchers:

>> package.searchers
[#<function: 0x55f4cfc8b930>
 #<function: 0x55f4cfc8b970>
 #<function: 0x55f4cfc8b9b0>
 #<function: 0x55f4cfc8b9f0>
 #<function: 0x55f4cff32450>]

It’s another table in the package table, that stores a list of functions, that will try to find and load the given module by their own means. Let’s write our own:

>> (table.insert
    package.searchers
    (fn [mod]
      (io.write "searching " mod " with a custom searcher\n")
      (fn loader []
        (fn bye [name]
          (: "bye %s" :format name))
        {:bye bye})
      (values loader mod)))

This searcher function in our case will work for any module regardless of its existence, so by doing this we kinda broke the loading mechanism. Still, this might be useful in some sandboxing scenarios. Alternatively, this mechanism can be used if the environment you run Lua in doesn’t have a real file system, so finding a module requires some trickery, and thus you can extend the general require mechanism. What our function does is not as complicated - it’s similar to what we did in the preload example, except what makes it different is that it basically allows us to do whatever we want before this loader function is given back to the runtime.

We can see it works:

>> (require :asdf)
searching asdf with a custom searcher
{:bye #<function: 0x55ec3e592ae0>}	"asdf"
>> (require :qwer)
searching qwer with a custom searcher
{:bye #<function: 0x55ec3e529460>}	"qwer"
>> (require :asdf)
{:bye #<function: 0x55ec3e592ae0>}
>>

And that it obeys the same caching rules.

This package.searchers table is the sole reason why Fennel works as seamlessly as it is. Setting up Fennel’s package searcher means that when we’re trying to require some module if there is a module.fnl file on the FENNEL_PATH, Fennel will compile this module, and then load it as a normal Lua module. If you want to implement your own Lua-based language, this is definitely the place to embed your compiler.

Now that you have an idea of how modules work in Lua, let’s discuss the main topic of this post - single-file Fennel libraries, and complications related to it.

Fennel libraries with macros

A library in Fennel obeys the exact same rules as a Lua library, except the Fennel compiler adds some steps to the whole process of library loading. As I explained in the searchers topic, the compiler is invoked on the module file before it is given to the Lua runtime. This means, that when we’re loading a Fennel file, it first gets compiled to Lua, and only then it is loaded. This works well up until we introduce compile-time features into our modules - mainly macros.

If we look back at Clojure and Elixir examples, I didn’t use macros in the modules - let’s fix this:

lib> (defmacro unless [test & body] `(when-not ~test ~@body))
#'lib/my-do
lib> unless
Syntax error compiling at (*cider-repl blog/posts:localhost:44133(clj)*:0:0).
Can't take value of a macro: #'lib/unless
lib> (ns-interns 'lib)
{baz #'lib/baz, bar #'lib/bar, unless #'lib/unless, foo #'lib/foo}

If we try to take a value of the macro, we get an error, as macros are a compile-time construct. Yet, it is referenced in the ns-intern as a var. And we can go back to the user module, and use our macro from there:

lib> (in-ns 'user)
#namespace[user]
user> (lib/unless true (println "this should never be printed"))
nil

In Elixir, it is similar, as you also can provide a macro in your module, and upon requiring the module you can use a macro from it. Here’s an example macro from the docs:

iex(2)> defmodule Unless do
...(2)>   defmacro unless(clause, do: expression) do
...(2)>     quote do
...(2)>       if(!unquote(clause), do: unquote(expression))
...(2)>     end
...(2)>   end
...(2)> end
{:module, Unless,
 <<70, 79, 82, 49, 0, 0, 6, 72, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 191,
   0, 0, 0, 20, 13, 69, 108, 105, 120, 105, 114, 46, 85, 110, 108, 101, 115,
   115, 8, 95, 95, 105, 110, 102, 111, 95, 95, ...>>, {:unless, 2}}
iex(3)> Unless.unless true, do: IO.puts "this should never be printed"
this should never be printed
** (UndefinedFunctionError) function Unless.macro_unless/2 is undefined or private.
      However, there is a macro with the same name and arity.
      Be sure to require Unless if you intend to invoke this macro
    Unless.unless(true, [do: :ok])
    iex:3: (file)

Except, we can’t use this macro right away - unlike in Clojure, we need to require the module:

iex(3)> require Unless
Unless
iex(4)> Unless.unless true, do: IO.puts "this should never be printed"
nil

Because Elixir’s macros are entirely Elixir’s concept, BEAM knows nothing about them, and thus you can use them only in Elixir, and it explains why the shell requires some additional steps. Perhaps these can be automated, as Clojure can do it without any additional require calls, but I don’t know for sure how it is done in Clojure or in Elixir.

I’m bringing Elixir and Clojure here because both languages are hosted on their respective runtimes - Erlang’s BEAM and JVM. Fennel is similarly hosted on Lua, but on the other hand, is different from these two when it comes to macros.

Let’s add the unless macro to the lib.fnl module:

;; lib.fnl
(fn foo [] "foo")
(fn bar [] "bar")
(macro unless [test ...]
  `(when (not ,test) ,...))
{:foo foo :bar bar :unless unless}

Trying to load this module, however, will result in a compile error:

(local lib (require :lib))
./lib.fnl:5:27 Compile error: tried to reference a macro without calling it

{:foo foo :bar bar :unless unless}
* Try renaming the macro so as not to conflict with locals.

We can’t simply export our macro alongside our functions. What should we do then?

The fennel language reference suggests that macros can’t be referenced and exported alongside functions And that macro is supposed to be used only as file-local macros, and thus these can’t be exported. Additionally to that, when writing macros, we can’t access any surrounding code. The reference section then says that in order to export macros we can put them into a separate module and use import-macros to import them to the arbitrary code.

The whole macro system in Fennel is not as advanced as in other lisps or as in Elixir, but it gets the job done, it just requires some squats to be done.

Macro modules

A macro module is exactly the same as an ordinary Fennel module - it shares the same file extension, same syntax, but is meant to be used in conjunction with import-macros or require-macros specials. To write a macro module we can simply put our unless macro into a separate file, replacing macro with fn:

;; unless.fnl
(fn unless [test ...]
  `(when (not ,test) ,...))

{:unless unless}

That’s right, in Fennel, macros are ordinary functions running in a compiler environment, and now we actually can export them as an ordinary table. The compiler environment defines additional functions that we can use in macros, such as quote and unquote which we’re using here in the forms of ` and ,. Now we can require this module with import-macros:

>> (import-macros Unless :unless)
nil
>> (Unless.unless true (print "this should never be printed"))
nil
>> (Unless.unless false (print "this should be printed"))
this should be printed
nil

There’s also another special that can import macros from a module: require-macros. Unlike import-macros it doesn’t put macros in a binding, instead, it pollutes the current scope with all of the macros from the module:

>> (require-macros :unless)
nil
>> (unless true (print "this should be printed"))
nil

So (import-macros name module) is much like (local name (require module)) but for macro modules, and it uses require-macros underneath.

`import-macros` and `require`

Speaking of the relationship between import-macros and require, they indeed work similarly. Fennel’s developers understood that Lua’s module system is very flexible and decided to make a similar interface for macros.

You couldn’t see it, but in the previous example, when I called require-macros on the unless module again, it didn’t re-read the module from disk. Instead, it used the same technique of caching, much like package.loaded but instead, it uses fennel.macro-loaded to store macros. We can see that by requiring Fennel in the REPL and inspecting the table:

>> (local fennel (require :fennel))
nil
>> fennel.macro-loaded
{:unless {:unless #<function: 0x56224adc40a0>}}

We can even do the same thing as with the package.loaded, and set the unless module to nil with (set fennel.macro-loaded.unless nil) If we then add some side-effecting code, like (print "macros!") to the module, and require it again, we’ll see that Fennel will re-read our module:

>> (set fennel.macro-loaded.unless nil)
nil
>> (import-macros Unless :unless)
macros!
nil
>> (import-macros Unless :unless)
nil

But only once, much like with ordinary require.

Similarly to Lua, Fennel provides its own table with searcher functions for macros:

>> fennel.macro-searchers
[#<function: 0x56224ac5a740> #<function: 0x56224ac5a7f0>]

So the system is extremely similar, the only thing it lacks is something akin to package.preload but for macros. With that in mind let me introduce you to the library problem.

The library problem

Let’s imagine, we want to write a library that provides you with a lot of functions, and with some convenience macros that remove some boilerplate. In reality, almost anything can be done without macros, but macros are great for writing code that you don’t want to write yourself. So let’s say we want to add polymorphic dispatch to Fennel by adding multimethods.

Let’s write the functions first:

(fn multi [dispatch-fn]
  (let [methods {}]
    (setmetatable
     {:add-method (fn [dispatch-val fn1]
                    (tset methods dispatch-val fn1))
      :get-method (fn [dispatch-val]
                    (. methods dispatch-val))
      :remove-method (fn [dispatch-val]
                       (tset methods dispatch-val nil))}
     {:__call (fn [self ...]
                (let [method (self.get-method (dispatch-fn ...))]
                  (method ...)))})))

(fn add-method [multi dispatch-val method]
  (multi.add-method dispatch-val method))

(fn get-method [multi dispatch-val]
  (multi.get-method dispatch-val))

(fn remove-method [multi dispatch-val]
  (multi.remove-method dispatch-val))

{:multi multi
 :add-method add-method
 :get-method get-method
 :remove-method remove-method}

The idea is simple. We create an object with the function multi and give it a dispatch-fn - the function it will invoke with the arguments to determine what method it needs to call. If there is a method, it will be called with the same arguments as the dispatch-fn functions. I also provided convenience functions that provide some abstraction over the inner implementation of the multimethod object. Let’s try it:

>> (local mult (require :multimethods))
nil
>> (local interact (mult.multi (fn [a b] (.. a.type " " b.type))))
nil
>> (mult.add-method interact "bunny lion" (fn [bunny lion] "bunny runs away"))
nil
>> (mult.add-method interact "lion bunny" (fn [lion bunny] "lion eats bunny"))
nil
>> (mult.add-method interact "lion lion" (fn [lion1 lion2] "lions fight"))
nil
>> (mult.add-method interact "bunny bunny" (fn [bunny1 bunny2] "bunnies mate"))
nil
>> (interact {:type "bunny"} {:type "lion"})
"bunny runs away"
>> (interact {:type "lion"} {:type "bunny"})
"lion eats bunny"

I opted for a simple example, where I just get the type field of each object and concatenate them into a string. Then we dispatch on that string.

The library is now complete, but it’s a bit tedious to type all of these anonymous functions every time, let’s add an interface similar to how Clojure does this. We’ll add two macros defmulti, which given a name and a dispatch fn will create a local variable by itself, and defmethod which will act the same as add-method except it will construct the anonymous function for us. We start by creating the macro module:

(fn defmulti [name dispatch-fn]
  `(local ,name (mult.multi ,dispatch-fn)))

(fn defmethod [name dispatch-val ...]
  `(mult.add-method ,name ,dispatch-val (fn ,...)))

{:defmulti defmulti
 :defmethod defmethod}

We already have the multimethod itself, so let’s try our defmethod macro:

>> (require-macros :multimethod-macros)
nil
>> (defmethod interact "ship rock" [ship _rock] (.. ship.name " sinks"))
nil
>> (interact {:type "ship" :name "Wasa"} {:type "rock"})
"Wasa sinks"

It works, but did you notice the problem? Let’s try this again, but instead, let’s call our library something else in the REPL:

>> (local mmethods (require :multimethods))
nil
>> (require-macros :multimethod-macros)
nil
>> (defmulti interact (fn [a b] (.. a.type " " b.type)))
unknown:2:? Compile error: unknown identifier: mult

Looking into the macroexpansion of defumilti we see that it does exactly what we told it:

>> (macrodebug (defmulti interact (fn [a b] (.. a.type " " b.type))))
(local interact (mult.multi (fn [a b] (.. a.type " " b.type))))

Except, the mult is nowhere to be found.

So, here’s the problem - we need to reuse functions from our function module in the macro module. Unfortunately, there’s literally no way to do it without making some assumptions about the module name. Instead of using available binding in the macro, we can require the library because it’s cached:

(fn defmulti [name dispatch-fn]
  `(local ,name (let [mmethods# (require :multimethods)]
                  (mmethods#.multi ,dispatch-fn))))

(fn defmethod [name dispatch-val ...]
  `(let [mmethods# (require :multimethods)]
     (mmethods#.add-method ,name ,dispatch-val (fn ,...))))

{:defmulti defmulti
 :defmethod defmethod}

With this change, everything starts working:

>> (local mmethods (require :multimethods))
nil
>> (require-macros :multimethod-macros)
nil
>> (defmethod interact "ship rock" [ship _rock] (.. ship.name " sinks"))
nil
>> (interact {:type "ship" :name "Wasa"} {:type "rock"})
"Wasa sinks"

But there’s another problem - what if the user of our library doesn’t like that the module file name is multimethods.fnl and wants to rename it? Suppose they already have their own multimethods module and want to try out ours alongside it, so they rename it to aorst-multimethods.fnl. Required macros will suddenly stop working because in our macros we hardcoded the multimethods module name.

`init.fnl` and `init-macros.fnl`

One solution to this problem is to use Lua’s other hidden module mechanic, related to how the searchers work.

In Lua, we can load a file by giving require a module name that matches that file, like require "foo" will probably load the foo.lua file. However, we can also load a directory in the same way - instead of creating foo.lua we can create a directory foo and write the contents of foo.lua into foo/init.lua. Then, calling require "foo" will seamlessly load code from init.lua, but it will look like we’re loaded a directory.

Fennel supports the same thing for ordinary modules with init.fnl, but in addition to that, it also supports this for macro modules with init-macros.fnl. We can re-implement our multimethod library this way, but we’ll change how we require the library in the macros:

;; multimethods-lib/init-macros.fnl
(local lib-module ...) ;; for relative require

(fn defmulti [name dispatch-fn]
  `(local ,name (let [mmethods# (require ,lib-module)]
                  (mmethods#.multi ,dispatch-fn))))

(fn defmethod [name dispatch-val ...]
  `(let [mmethods# (require ,lib-module)]
     (mmethods#.add-method ,name ,dispatch-val (fn ,...))))

{:defmulti defmulti
 :defmethod defmethod}

This trick is called relative-require. It’s quite common in Lua, but I decided to use it when I was trying to ship huge libraries with lots of macros and inner modules.

When the module is loaded, Lua passes the module name to the module as an argument. This (local lib-module ...) captures it in the lib-module variable, and then we can refer to it in our macros. If we now look at what our macros expand to, we’ll see that they use the module name correctly:

>> (local mmethods (require :multimethods-lib))
nil
>> (require-macros :multimethods-lib)
nil
>> (macrodebug (defmulti interact (fn [a b] (.. a.type " " b.type))))
(local interact
  (let [mmethods_2_auto (require "multimethods-lib")]
    (mmethods_2_auto.multi (fn [a b] (.. a.type " " b.type)))))
nil

Now our macros are unbound from the library name itself. However, shipping library in this way is not that great. For example, here’s how one of my libraries that uses this technique looks if you peek into the library root:

.
├── CHANGELOG.md
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── doc/
├── fennel-test/
├── impl/
├── init.fnl
├── init-macros.fnl
├── LICENSE
├── Makefile
├── README.md
└── tests/

Basically, it’s the project’s root, and if you want to use this library, the easiest way is to just git-clone the project and require it. But it means that there are lots of unrelated files, and there’s no way to know what files are related unless the project has some kind of manifest, which is not the case for Fennel projects. It would be much better if you could just grab only the init.fnl file, name it however you want, and require both macros and functions from it.

Require-as-include

Fennel has a feature that can do almost that. For multi-file libraries, we can AOT compile all of the Fennel code into Lua, and the compiler will inject all of the dependencies in an interesting way. It does so by tracking what modules we require and splicing them into the code by using package.preload trick I’ve described way earlier in this post:

(fn foo []
  "some-lib's foo")

{: foo}

(local some-lib (require :some-lib))

(fn my-foo []
  (some-lib.foo))

{:my-foo my-foo}

If we call fennel --require-as-include -c my-lib.fnl, we’ll get the following code:

local some_lib
package.preload["some-lib"] = package.preload["some-lib"] or function(...)
  local function foo()
    return "some-lib's foo"
  end
  return {foo = foo}
end
some_lib = require("some-lib")
local function my_foo()
  return some_lib.foo()
end
return {["my-foo"] = my_foo}

As can be seen, the whole module definition of some-lib was included in the file as an entry in the package.preload. In the later code, it is required and used as normal.

However, this doesn’t work with macros, as there is no fennel.macro-preloaded or something similar. And for a long enough time, I wasn’t sure if it was possible to mimic this behavior at compile time. But then I remembered that we indeed have the module name at compile time, as clearly shown in the relative-require example.

Mimicking preload behavior for macros

Here’s what I came up with:

(eval-compiler
  (local lib-module ...)

  (fn defmulti [name dispatch-fn]
    `(local ,name
       (let [mmethods# (require ,lib-module)]
         (mmethods#.multi ,dispatch-fn))))

  (fn defmethod [name dispatch-val ...]
    `(let [mmethods# (require ,lib-module)]
       (mmethods#.add-method ,name ,dispatch-val (fn ,...))))

  (tset macro-loaded lib-module
        {:defmulti defmulti
         :defmethod defmethod}))

(fn multi [dispatch-fn]
  (let [methods {}]
    (setmetatable
     {:add-method (fn [dispatch-val fn1]
                    (tset methods dispatch-val fn1))
      :get-method (fn [dispatch-val]
                    (. methods dispatch-val))
      :remove-method (fn [dispatch-val]
                       (tset methods dispatch-val nil))}
     {:__call (fn [self ...]
                (let [method (self.get-method (dispatch-fn ...))]
                  (method ...)))})))

(fn add-method [multi dispatch-val method]
  (multi.add-method dispatch-val method))

(fn get-method [multi dispatch-val]
  (multi.get-method dispatch-val))

(fn remove-method [multi dispatch-val]
  (multi.remove-method dispatch-val))

{:multi multi
 :add-method add-method
 :get-method get-method
 :remove-method remove-method}

Finally, we can require our library and macros from the same module, and it doesn’t have to be a directory! Here’s how it looks:

>> (local mm (require :combined-multimethods))
nil
>> (require-macros :combined-multimethods)
nil
>> (defmulti interact (fn [a b] (.. a.type " " b.type)))
nil
>> (defmethod interact "ship rock" [ship _rock] (.. ship.name " sinks"))
nil
>> (interact {:type "ship" :name "Wasa"} {:type "rock"})
"Wasa sinks"

So, it’s a combination of techniques, but what exactly is eval-compiler doing here?

If you look closely at evel-compiler body, you’ll see that instead of exporting the table at the end of the scope we simply set it to the macro-loaded table. We use the module name which was set to combined-multimethods when the module was required, and thus the macro-loaded.combined-multimethods now stores the table with macro definitions. Then, when require-macros looks up the macro module combined-multimethods, it sees that it was already loaded, and simply returns it without trying to find a file with these macros. Alternatively, we could define a macro-searcher that, given a specific module-name would do the same thing, loading macros while avoiding trying to find a file, but it’s more work for almost no benefit over what we have here already.

Preprocessor-as-include

There’s still a problem though. What if we want to write a library, that uses another library, that uses macros in this way?

One such example is my cljlib library I keep re-implementing every few months or so. In that library I also wanted to ship macros and functions in the same module, but here’s a problem - this library depends on other libraries, namely lazy-seq and itable. While itable doesn’t provide any macros, lazy-seq does, and cljlib re-uses these macros.

Before some recent changes I made to both lazy-seq and cljlib you had to fiddle with FENNEL_PATH and FENNEL_MACRO_PATH in order to use cljlib, and you had to check out the full repository with submodules. I feel like this is way too much work for most people, and I don’t want anyone to suffer that much, so I changed it to use the trick above and embedded other libraries into it by manually writing the package.preload trick instead of relying on --require-as-include flag. Here’s how it looks in the src/cljlib.fnl file:

;;; itable
(set package.preload.itable
  (or package.preload.itable
      (fn [...]
;;;###include itable/src/itable.fnl
        )))

;;; lazy-seq
(set package.preload.lazy-seq
  (or package.preload.lazy-seq
      (fn [...]
;;;###include lazy-seq/lazy-seq.fnl
        )))

;;; cljlib
(eval-compiler
  (local lib-name (or ... :cljlib))
  ;; macros ...
  )
;; functions ...

These ;;;###include path comments are nothing special from the perspective of Fennel, but I made a small build.fnl script for the library that takes care of them:

(fn spit-lib [path to]
  (with-open [lib (io.open path)]
    (each [line (lib:lines)]
      ;; patching compile-time variable used to store macro module
      ;; name because when loafing the combined file it will always
      ;; equal the main module and will break macros in vendored
      ;; libraries.
      (case (line:match "%(local lib%-name %(or %.%.%. (.*)")
        name (to:write (.. "(local lib-name (or " name "\n"))
        _ (to:write line "\n")))))

(with-open [cljlib (io.open "./cljlib.fnl" :w)]
  (let [main (io.open "src/cljlib.fnl")]
    (each [line (main:lines)]
      (case (line:match ";;;###include (.*)")
        (path) (spit-lib path cljlib)
        _ (cljlib:write line "\n")))))

It is meant to be used only for development, the library is pre-built and ready for anyone to grab. What’s interesting about this script is that it also patches the library it includes. This was a pain to figure out.

You see, in the lazy-seq library I’m using the same eval-compiler trick, and it in turn uses the (local lib-name (or ... :lazy-seq)) trick. However, when this library is spliced into our main file as is, when we load this file as (require :cljlib), the ... becomes "cljlib", and the lazy-seq library, included in this file will write its macros into macro-loaded.cljlib instead of supposed macro-loaded.lazy-seq. This can’t be worked around, as this happens way before package.preload.lazy-seq runs, so the ... provided by the function in that table is never used by eval-compiler. The only way to delay the eval-compiler is to put it into the file or use the searcher that will evaluate this block of code by itself which is way too tricky.

So I made a hack, that while splicing the library code scans for any occurrences of the (local lib-name (or ... pattern, and replaces it with the name used in the or expression, without checking .... This way, the (local lib-name (or ... :lazy-seq)) becomes (local lib-name (or :lazy-seq)) in the final version of cljlib.fnl. Of course, it is extremely finicky, as even an introduction of a newline between lib-name and (or ...) expression will break this. Changing lib-name to something else will also break this down.

But even with this kind of monkey-patching it works well enough, as I control what code goes in both libraries, and the user doesn’t need to think about it at all. All that is needed to do is just to grab the cljlib.fnl from the repo root and put it into the project. I’ve updated Fenneldoc to use the newer version of cljlib and was pleased to see that it still works.

Fixing macros in Fennel

Even with the ability to ship macros in the same file as functions, I find it weird that Fennel’s macro system has all sorts of other limitations.

Firstly, the inability to use functions defined alongside macros is really bad. This is one of the reasons we have to do all these squats with relative-require stuff and require the library itself from the macro. Which in turn makes us more error-prone, if some weird environment will not set the module args (...) properly.

Another thing is that macros can’t use other macros unless they’re defined in a separate module, or in the eval-compiler trick I was featuring in this post. If you define a macro with the macro special, it must be self-contained. Otherwise, you have to move it somewhere. I mean, it’s weird that Fennel has a proper implementation of macros sitting in the compiler, but for some reason macro and macros specials don’t use it.

What I think would fix all this mess is this:

First, we need to ditch macro modules completely. They’re inferior to ordinary fennel modules and only make things complicated.

Next, we need to make it so the macro special sets the macro in the macro-loaded entry of the current module, much like I manually do in the eval-compiler block.

And lastly, when compiling the module to Lua, just load the module at compile time as a whole! This way not only macros will be able to use other macros, but they’ll also be able to use other ordinary functions at compile time!

The only problem left is how to make it so the generated code can use definitions from the library. And I think I know the way, and it’s similar to how Clojure handles things.

So basically, same as in Clojure, macros can only use public definitions. What this means is that these definitions are in the packagr.loaded by the time we’ve required the library. Therefore, the backquote can safely fully qualify any unqualified symbol in the calling position!

In other words, the macro bar from this file:

(fn foo [val] val)

(macro bar [expr]
  `(foo ,expr))

{:foo foo}

After macroexpanding will become:

>> (require-macros (doto :lib require))
nil
>> (macrodebug (bar (+ 1 2 3)))
(package.loaded.lib.foo (+ 1 2 3))

And will work just fine, as it can reach foo, as it is already in the package.loaded by that time. Which is similar to what Clojure does:

user> (defn foo [val] val)
#'user/foo
user> (defmacro bar [expr] `(foo ~expr))
#'user/bar
user> (macroexpand '(bar (+ 1 2 3)))
(user/foo (+ 1 2 3))

And as the final change, we need to make that the fennel.searcher properly sets up macro-loaded in order to ditch the awkward (doto :lib require) thing.

Anyhow, that’s all I got for today. I know that I’m writing a lot of devlog posts about my small game dev marathon, so I try to write some posts in-between so that my blog doesn’t become too focused on one topic. Hope that this was an interesting read!

Comment via email

async.fnl enchancements

Wed, 02 Aug 2023 09:10:00 +0300

Lately, I’ve been working on async.fnl in my spare time and realized that the previous code that I used to schedule timers was terrible. Well, realized isn’t an appropriate word here, because I knew it back when I first implemented it, I just didn’t bother to make it better until I had a fully working library. So, a few weeks ago I had an idea on how to make things both more efficient and prevent excessive CPU usage when there’s no need to.

Lua `debug` hooks

Lua comes with the debug library. This library provides a set of relatively low-level facilities to implement your own debugger, and one such things are hooks.

All hook manipulations are done with debug.sethook and debug.gethook functions - one to set/unset the hook, and the other to retrieve existing hook, respectively. A hook is a simple Lua function, that accepts the event which triggered it. Lua gives us five types of events we can check for in our hook: "call", "tail call", "return", "line", and "count", but we’re only really interested in one - the "count".

But before we discuss hooks, I’d like to do a small detour into how my old async library handled things. It will both allow you to understand why I decided to move away from the old implementation, and how we can use hooks to achieve similar facilities later. Feel free to skip if you’re not interested.

Scheduler

Here’s how asynchronous threads were handled in an older implementation of my other async library.

The library had a special scheduler object, that kept track of all tasks to be run.
Each task that wanted to stop itself for some time is called the async.park function, usually indirectly via something like async.take or async.put.
- additionally, If a task wants to sleep for some time, it can call async.sleep, which, when called on the asynchronous thread, parks it with a special condition, and tells the scheduler that this task should not be touched for the specified amount of time.
Then, once the internal scheduler.run is called, all tasks are examined, and the shortest amount of sleep is chosen, if any.
- If there are tasks that aren’t sleeping, the system resumes those tasks;
- If, however, all tasks want to sleep, the system determines the minimum amount of time it needs to sleep and puts the main thread to sleep.
Once all tasks ran, the scheduler.sleep exits unless the user requested it to run until all tasks are fully complete.

This worked reasonably fine but required someone to call the scheduler.run periodically. The library tried to make it transparent for the program, as any operation that created threads, pushed or polled for values, and so on, called shceduler.run internally, to give an illusion that the system actually works in parallel. Thus, if you made a blocking take on some channel, the scheduler would run until that channel is ready to return the value (though, this might never happen).

The main downside of this was that the whole system depends on a central scheduling mechanism, and the more tasks you have spawned, the less performant the scheduler becomes. For example, here’s a function, that tracks the amount of time it took for value to be put to a channel and then read back in another thread:

(local {: gettime} (require :socket))
(local {: put : close! : take : chan &as async}
  (require :fennel-async.async))

(fn bench []
  (let [c (chan 1)
        lock (chan)]
    (var put-time 0)
    ;; a thread that loops putting the values one by one, and tracks
    ;; the time when it completed each put
    (async
     (fn []
       (for [i 1 10]
         (set put-time (gettime))
         (put c i))
       (close! c)))
    ;; a thread that takes values, and measures the amount of time it took
    ;; for value to be transferred.  Loops for more until the channel
    ;; is closed
    (async
     (fn loop []
       (let [val (take c)]
         (if val
             (do (print (- (gettime) put-time))
                 (loop))
             (close! lock)))))
    ;; waits til both threads are done
    (take lock)))

Let’s call it:

>> (bench)
6.0796737670898e-05
1.1205673217773e-05
2.8848648071289e-05
9.0599060058594e-06
7.8678131103516e-06
9.0599060058594e-06
8.1062316894531e-06
9.0599060058594e-06
7.8678131103516e-06
2.6941299438477e-05
nil

Pretty fast. Now, let’s add a few tasks that will immediately park because there’s no one to produce values, and will simply exist in a scheduler:

;; create 10000 channels
(local channels (fcollect [i 1 10000] (async.chan)))

;; spawn 10000 tasks waiting on these channels
(each [_ chan (ipairs channels)]
  (async (fn [] (async.take chan))))

This seemingly innocent each loop took a whopping 148 seconds to execute on my machine! It takes so much time because the library re-runs the scheduler over an ever-increasing number of tasks on every single call to async. So, it’s basically O(n²) loop even though it looks like O(n).

Now, let’s call our function again:

>> (bench)
0.033475875854492
0.040903806686401
0.044415950775146
0.043467998504639
0.039731979370117
0.039532899856567
0.039582014083862
0.039039134979248
0.039515972137451
0.039596080780029
nil

So, even though all of the threads were parked, and our bench function doesn’t interact with any of them directly, its execution was slowed down severely. For a good measure, let’s release all those tasks by putting some values onto the channels with another loop (each [_ c (ipairs channels)] (async.put c 42)) (another 163 seconds to complete), and then re-run the bench function, to check that it is fast again.

>> (bench)
0.00029706954956055
8.8214874267578e-06
9.7751617431641e-06
9.0599060058594e-06
9.0599060058594e-06
9.0599060058594e-06
6.1988830566406e-06
8.8214874267578e-06
1.5974044799805e-05
2.4080276489258e-05
nil

And we’re back to being fast again. Yay!..

This is a major downside of the system that keeps track of all tasks - there’s no way of optimizing it because it has to check for all tasks every time - what if some tasks are no longer blocked and can advance? We can’t even partition those tasks, because, as in the example above, the task is not related to timed events at all. Because of that, I decided to re-implement the library from scratch, but I already blogged about it.

Fake it til you actually need it

In the new version of the library, I avoided the scheduler altogether. Well, there is a scheduler still, but unless you’re using timeout channels, you can live completely without it (for the most part, that is). BTW, I shouldn’t call it a new version of the library, it is simply a different library, but the facilities are largely the same for both, so we can compare them.

First things first, let’s test the code from the above with the new library:

(import-macros {: go : go-loop} (doto :async require))
(local {: >! : <! : <!! : close! : chan : take!} (require :async))
(local {: gettime} (require :socket))

(fn bench []
  (let [c (chan)
        lock (chan)]
    (var put-time 0)
    ;; a thread that loops putting the values one by one, and tracks
    ;; the time when it completed each put
    (go
      (for [i 1 10]
        (set put-time (gettime))
        (>! c i))
      (close! c))
    ;; a thread that takes values, and measures the amount of time it took
    ;; for value to be transferred.  Loops for more until the channel
    ;; is closed
    (go-loop [val (<! c)]
      (if val
          (do (print (- (gettime) put-time))
              (recur (<! c)))
          (close! lock)))
    ;; waits til both threads are done
    (<!! lock)))

>> (bench)
8.9883804321289e-05
4.6968460083008e-05
0.00019192695617676
4.6014785766602e-05
0.00013494491577148
1.8835067749023e-05
9.7990036010742e-05
2.0027160644531e-05
9.8943710327148e-05
3.0994415283203e-05
nil

The overall speed is more or less the same but now, let’s repeat the same trick, again creating 10000 channels, and registering a task on each of them:

;; create 10000 channels
(local channels (fcollect [i 1 10000] (chan)))

;; spawn 10000 tasks waiting on these channels
(each [_ c (ipairs channels)]
  (go (<! c)))

Not only this code executes almost immediately, taking about a second, but this amount of channels also don’t affect our bench function at all:

>> (bench)
0.00028419494628906
9.2029571533203e-05
0.00049304962158203
6.6041946411133e-05
0.00028800964355469
5.1021575927734e-05
0.00026488304138184
5.6982040405273e-05
0.00023293495178223
7.9870223999023e-05
nil

So, what’s the trick?

If we take a look at the internals of the library, there’s also a queue of dispatched tasks, called, well dispatched-tasks. We can load the library as a whole into the REPL and poke around:

>> (local channels (fcollect [i 1 10000] (chan)))
nil
>> (each [_ c (ipairs channels)] (go (<! c)))
nil
>> dispatched-tasks
{}
>> (length channels)
10000
>> (. channels 1)
#<ManyToManyChannel: 0x55fbd1d328d0>
>> (. channels 1 :takes)
[#<reify: 0x55fbd2cbb690: Handler>]

So, even though we’ve actually created 10000 channels plus a 10000 of go blocks, each of which is also a channel, and registered a pending take (<!) on each one, no tasks were actually added to the dispatched-tasks at all. A task is only added to the scheduler when the channel is used! In other words, once we put something onto a channel, that has a pending take, only then the task will be scheduled to execute:

>> (put! (. channels 1) 42)
true
>> dispatched-tasks
{#<function: 0x55dcaf30ecb0> true}

This means that the work done by the scheduler is limited to the amount of resources really used by the program at a given point in time. So our partitioning problem is gone. There’s still another problem though - we need to run the scheduler somehow.

Now we can talk about hooks.

Hooks (the wrong way)

Initially, when implementing this system I realized, that while the scheduler actually isn’t necessary for plain channels at all, because you can always delay the execution of a pending operation until the channel is ready this won’t work if I add timeout channels. And actually, before I introduced timers to the library, there wasn’t any dispatched-tasks kind of thing at all, so the library was truly scheduler-less. But after I ported the whole thing from ClojureScript, I had to add timeout channels to be fully compatible, thus the scheduling had to be introduced too.

For a brief note, a timeout function returns a channel that closes automatically after the specified period of time:

>> (let [start (time)]
     (<!! (timeout 300))
     (print (.. "Elapsed: " (- (time) start) " s")))
Elapsed: 0.3021879196167 s
nil

But how exactly it knows when to close? While we can block our main thread with <!! OP and inspect the timeout object for the time it needs to sleep, thus putting the main thread to sleep for that time it will also stop all other background tasks from happening. And if the timeout is awaited asynchronously in a go block, we can’t do even that.

So, we do need a scheduler that will run regularly and will check for any timeouts that are ready to be released. My initial thought was that I once again need some kind of scheduler.run function that will need to be manually called by each function in the library, but this approach has other problems too.

For example, what if the user will write a tight loop that will run for too long and we’ll miss the time we need to release a timeout? Well, that isn’t that big of a deal, but I’d like not to have a system that is this unreliable. The user can even forget to call scheduler.run in their loop, wondering why the timer doesn’t even count. So I started searching for other approaches.

The solution was to use debug.sethook with some options, to run the hook regularly enough that it will not cause serious performance hit for the system as a whole, and will update timers with enough precision. And that’s exactly what the ClojureScript version of the library does! Except it uses the goog.async thing, which I have no idea how it works, but the solution is simple and effective. As far as I know, JavaScript has some facilities to register callbacks to be run after a certain amount of time, so maybe it’s something along the line there.

So I decided to try this out and made a simple hook that runs every time we return from functions, and every one thousand VM instructions:

;; ----8<---- piece of old code ahead -----8<-----
(local -timeouts {})

(fn -advance-timeouts []
  "Close any timeout channels whose closing time is less than or equal
to current time."
  (when (next -timeouts)
    (let [to-remove (icollect [t ch (pairs -timeouts)]
                      (when (> 0 (-difftime t (-time)))
                        (ch:close)
                        t))]
      (each [_ t (ipairs to-remove)]
        (tset -timeouts t nil)))))

(fn -register-hook [f]
  "Run function `f` on each function return and every 1000 instructions.
Appends the hook to existing one, if found, unless the hook is `f`."
  (when (and gethook sethook)
    (match (gethook)
      f nil
      nil (sethook f :r 1000)
      (hook mask count)
      (sethook
       (fn [...] (hook ...) (f ...))
       mask
       count))))

(-register-hook -advance-timeouts)
;; ----8<----

And it worked! Satisfied with the result, and not paying any attention to the slowdown in the REPL, I moved on. But the bitter taste of this code still followed me, as I knew that this isn’t a proper solution by any means. This had many problems.

The first and obvious one is that the hook runs too frequently, but I didn’t think about it too hard because I had other stuff to port to the library.
Second, and a bit less obvious, the hook runs all the time, even if there are no timeout channels at all. Unlike the ClojureScript solution, which schedules the hook once and only when needed, this one runs til the end of time. Well, if there are no timeout channels it does nothing but the VM still stops to execute it each time the condition is met, which is bad.
And third, I tried to keep the old hooks around, by using composition and thus made a foreign hook run possibly too often or even not often enough depending on the original settings.

Later, when I added the mentioned dispatched-tasks table to the code, I started noticing the slowdown even more. So I knew I need to do something.

Hooks (the less wrong way)

So, I’m pleased to say that all these problems were rectified recently.

One morning, or a late night, I don’t really remember, I thought that it would be nice to actually measure how much time it takes for Lua to execute these 1000 instructions. I then could adjust the parameter to only run the hook approximately 10ms after it was registered. Ten milliseconds feels like a decent enough precision for timeout channels. I mean no one in a sane mind will try to implement a hard real-time system with my library, or in Lua for that matter. Even a firm one.

Plus, to save up some performance and avoid keeping track of too many timers, I made an optimization to only create a new timer if the time passed since the previous one was created is greater than 10ms. Thus, the timers at minimum are 10 milliseconds apart from each other.

After a bit of testing, it turned out that a million and a half instructions approximately take 10 ms to complete on my machine. And while 1 million felt like a more appropriate amount to use for advancing timers, I knew that it would change on a per-machine basis, so I decided to tweak the system once more.

Inspired by the ClojureScript approach, I thought that there was no need to run the hook every N instructions. Instead, I can run it after N instructions, and then disable it from within the hook itself. Moreover, I can calculate how much time it took to execute this amount of instructions, and recalculate N to set the hook to run 10ms in the future.

Here’s the code behind that:

And the main interface for hook scheduling.

(var (n-instr register-time orig-hook orig-mask orig-n) 1_000_000)

(fn schedule-hook [hook n]
  (when (and gethook sethook)
    (let [(hook* mask n*) (gethook)]
      (when (not= hook hook*)
        (set (register-time orig-hook orig-mask orig-n)
          (values (os/clock) hook* mask n*))
        (sethook main-thread hook "" n)))))

(fn cancel-hook [hook]
  (when (and gethook sethook)
    (match (gethook main-thread)
      (hook ?mask ?n)
      (do (sethook main-thread orig-hook orig-mask orig-n)
          (values ?mask ?n)))))

Code Snippet 1: Hook cancellation. Returns the hook's settings for future analysis, and resets to the old hook if specified.

(fn process-messages [event]
  (let [took (- (os/clock) register-time)
        (_ n) (cancel-hook process-messages)]
    (set n-instr
      (if (not= event :count) n
          (// 0.01 (/ took n))))
    (faccumulate [done nil _ 1 1024 :until done]
      (case (next dispatched-tasks)
        f (tset dispatched-tasks (doto f pcall) nil)
        nil true))
    (each [t ch (pairs timeouts)]
      (when (>= 0 (difftime t (time)))
        (tset timeouts t (ch:close!))))
    (when (or (next dispatched-tasks) (next timeouts))
      (schedule-hook process-messages n-instr))))

Code Snippet 2: The main body of the hook, that processes the messages. Re-registers itself if there's more work to be done. Calculates the amount of time it takes 1 instruction to complete, and uses it to reschedule itself 10ms after the execution.

In addition to that I opted out of combining several hooks into a single one. This feels like a Lua problem - I don’t really understand why the VM only allows for a single hook to be registered. Lua has five different events to hook up to, so why not allow registering five hooks? And why isn’t it possible to register more than one hook for a single event type? I mean, if there were many hooks registered, just run them in succession.

I guess I can implement my own sethook and replace the one that comes with the debug library with my own version after someone loads my library. Then register a handler that will keep track of all other hooks registered and run them accordingly. Maybe I’ll try it.

Two problems though - there’s no way of removing a specific hook in stock Lua, and I will need to somehow determine what hook to call on count events if more than one was registered. Especially when each specifies a different amount of instructions. I guess this is why there’s this kind of limitation in the stock debug library, but I don’t think it’s impossible to do it in the VM itself with an ever-growing instruction counter. Probably it’s just an optimization.

In any case, the new system balances itself much better than before keeping 10ms intervals between tasks, and the slowdowns are gone.

And while I was on it, I realized, that given the fact that I have both scheduked-tasks and timeouts storages, instead of checking for them in a hook, I can also do the check when I do a blocking operation. This means that if the blocking operation only needs to wait for a timeout to close, it can actually sleep instead of doing a busy loop:

>> (let [start (time)
         cpu-start (os.clock)]
     (<!! (timeout 1000))
     (print (.. "CPU time: " (- (os.clock) cpu-start) " s"))
     (print (.. "Elapsed: " (- (time) start) " s")))
CPU time: 0.169319 s
Elapsed: 1.0089449882507 s
nil

Another realization came in that there is no point in doing any waiting if there are no active timeouts or tasks dispatched. Because blocking the only thread means that no other thread can dispatch its task. And a task can only be dispatched by another task, thus we can exit the waiting loop and throw an error without blocking the execution forever. This doesn’t mean that we’re free from data races, far from it, but at least we can detect that the data will never arrive on a channel if we block. In practice this shouldn’t happen too often.

Probably, there are more clean ways of handling hooks than the one I’ve came up with, so feel free to suggest how to change the approach in the project’s repository.

Comment via email

Game1 - Results

Sun, 30 Jul 2023 15:29:00 +0300

Well, this was fun! A bit exhausting, actually. The first of five months of the challenge has ended and here are the results:

Play Game1 on itch.io

The game isn’t really complete, but I did my best to make it feel as complete as possible in the time constraints I had. Technically, I have 1 day left, but given that I need to start working on the next game immediately I decided to give myself a day off. It only features one level, and cuts out to the “you won” screen, but I still consider it a success.

Here’s the last bit of devlog for this particular game outlining some things I did since the last post in no particular order.

Graphic overhaul

I’ve added parallax scrolling to the background, making the game happen inside a cave. Well, at least, now the background is not that boring to look at, but it’s still plain.

Then, I added a second layer and started experimenting with different colors:

The problem with limited colors is that I can’t find a way of having foreground and background properly separated without using outlines on sprites, which I wanted to avoid for this game at least. Thus, slimes are now blue instead of green. But I like it more this way because it pops from the green ground a bit better. And now that the background is also green it’s a lot more sensible.

I’ve spent some time drawing foliage, like mushrooms, trees, grass, water, and so on. None of this is animated though - time is once again against me.

Also, added the menus, title screen, cover art, and some fade-in/out animations for smoother transitions:

Enemy AI

While I wanted to add more enemy types to the game I simply ran out of time, so the game only features basic slimes. Their AI is extremely simple too - just roam forward until there’s no more ground, a wall, or another entity, then turns around.

Still, I had some trouble implementing this at first, as I tried to use the world:check method to try and move the slime forward and below to check if there’s ground to traverse on. But this didn’t work for some reason, so I opted for advanced bump API world:queryRect and queried the rectangle below the slime instead. It worked nicely, and perhaps I’ll use this to make a more robust AI in future games.

I was toying for a little bit with flying enemies, but it’s hard to make them work right with the environment. I didn’t want something like Medusa heads from Castlevania games, but that’s exactly what I got during testing.

I also finally added the death mechanic, as previously the game lacked it completely. Death doesn’t take away your score, as the game turned out to be hard, and there are not enough coins in the only level to get 100 of them and earn an extra life.

Level design

Probably shouldn’t have left this to the last possible moment, but designing interesting levels turned out to be a lot harder than I thought initially. I think I’ve managed to at least do a somewhat challenging one, but you’re the judge here. Probably most enemies and traps are cheap.

As I mentioned previously, I’ve used the map editor to construct the level:

It’s a shame though that Tiled, a tilemap editor, doesn’t support TIC’s maps. Well, there’s a converter but it didn’t work for me. Map editor in TIC isn’t bad, but it isn’t as good as something like Tiled. For example, Tiled has randomization from a selected set of tiles when you paint with them, which makes things like ground feel less repetitive. In TIC I had to do it by hand, and I may have forgotten to do it somewhere in the game, so excuse me if you’ll see something like that.

I used the bottom of the map to draw menus, backgrounds, transitions, and so on, as can be seen here:

My initial idea was to implement the first few levels like this, one below another, and then switch the memory bank, draw a new set of tiles, and make more levels. But not only I ran out of time, I also ran out of ideas. Maybe next time.

Overall, I’d say making levels isn’t my least favorite thing so far but it’s close. I’ve played in a fair bit of games with level editors and even published some online, but it never grew on me. I guess I’m not a level designer at heart.

Music and SFX

This was probably the hardest part. I can’t compose music and TIC’s tracker is alien to me. Sound design is a bit easier, but I don’t know a lot of tricks, so sounds in this game are far and few between. Still was worth a shot.

Up next

The next game in the challenge is a roguelike. This kind of game poses different problems, like pathfinding, more complex enemy AI in general, randomization of levels, items, etc. I kinda feel exhausted right now but I’ll still try not to repeat the mistake I made and not procrastinate too much in the first weeks. Though the next month is packed with various life-related events, much more than this one, so we’ll see.

As for this game, here’s the cartridge if you want to play it yourself:

Though, because the original code requires a more recent version of Fennel than the one TIC provides, the code in the cart above was AOT compiled to Lua and isn’t very readable. Also note that the last stable release of TIC still contains the #1904 bug, which means that my game will run ~10 times slower than it should. So you either need to play the game on the itch.io page, as it features a more recent version, build a standalone version yourself, or use an older release with the Lua version above. You can read the original source code at the itch.io page linked above, or in the game repository.

Comment via email

Game1 W4/4 progress

Wed, 26 Jul 2023 20:09:00 +0300

This post is midway through the last week I have to work on the game in a platforming genre. And it’s a bit of a shame because currently, I’m having a blast - now that the physics and camera are in place the game already feels like a playable thing. So I decided to boost this feel and make some graphics so the world won’t look too boring to traverse.

I haven’t decided on what the world be like yet, but I’m running out of time, and also, I’m not that experienced in pixel art. A few posts ago I wrote how I tried to do pixel art on the phone by using a stylus, and it worked pretty well for me. So I took out the app again and started thinking about how would I use the colors that are still available to me.

I’ve briefly considered changing the palette too. The default TIC-80 palette is a bit weird for my taste, and it doesn’t include some colors that I’d like to have, e.g. brown. I don’t know how, but every time I see some pixel art that uses this particular palette I swear it has brown, but upon closer inspection it’s an illusion caused by interplay with the other colors. Unable to find a new palette that works, and also that doesn’t make every other sprite I drew up to this point use some weird colors, I gave up for now. Maybe I’ll just look at what colors I don’t use at all or use too infrequently and replace them with the colors I need for my purposes.

For now, I drew some grass on stone type of terrain (again, no brown for dirt!), and some construction site-type red bars:

Note, these are tiles, e.g. what goes on the map, but there are some things that are not part of the map, for example, the coin, or the slime enemy. I’ll explain why they’re here shortly.

And here are the sprites:

As you can see, I’m storing these sprites mostly in rows, and tall sprites, like the player character look naturally in the editor. Except for the ones in the third and fourth rows first half - these are horizontal sprites, but I still store them this way, because of how I do animations. This reminds me that I haven’t explained anything regarding how my game works! I guess I was too busy implementing it and didn’t have any spare time to write a proper post.

So the animations are implemented in terms of sprites, obviously, but the way I set up them is a bit weird. And I know, I definitively could save some space and re-use some sprites, because clearly there are doubles or even triples of the same sprite. But it was easier this way, so I did it anyway. I have this function, called sprites that takes an id of the first sprite, the last sprite, and an optional loop? parameter:

(fn sprites [from to loop?]
  (let [sprites (fcollect [id from (or to from)] id)]
    (doto sprites
      (tset :n (length sprites))
      (tset :loop? loop?))))

The result of this function is a table, that simply holds a range of IDs for all the sprites in the animation. Does the n key hold the total count of sprites, and loop?, well, we’ll get to it in a moment.

The other component of animation is how I choose when to change the frame. I’m sure there’s a different, and probably more robust way of doing this, but what I’ve settled on is this function:

(fn next-frame [obj dt]
  (let [dt (+ dt (or obj.last-draw-time 0))]
    (if (>= dt 60)
        (let [sprites (obj:sprites)
              n (m/floor (/ dt 60))]
          (set obj.last-draw-time 0)
          (if (= obj.frame sprites.n)
              (case sprites.loop?
                n (if (= :number (type n))
                      (set obj.frame n)
                      (set obj.frame 1)))
              (set obj.frame
                (clamp 1 (+ n obj.frame) (length sprites)))))
        (set obj.last-draw-time dt))))

As you can see, this function receives an object obj and a delta time dt. The object stores its own draw time, and when the sum of dt and this time is greater than 60 I change the frame. The loop? field is used in a special way - when the next frame exceeds the total sprite count, we check if the animation should loop. If the field is true the animation loops from the first frame. However, if the field is a number, we restart the animation from that number. This way the animation can have a set of frames that act as preparation or transition, and then loop only the necessary part.

Here’s how I draw coins:

(local coin-sprites (sprites 330 335 :loop))
(local coin-collect-sprites (sprites 346 351))

(fn draw-coin [{: x : y : frame &as self} {:x cam-x :y cam-y}]
  (match (self:sprites)
    coin-sprites
    (let [s (or (. coin-sprites frame) 511)]
      (spr s (+ cam-x -120 x) (+ cam-y -64 y) 0 1)
      (next-frame self (/ dt 2)))
    coin-collect-sprites
    (let [s (or (. coin-collect-sprites frame) 511)]
      (spr s (+ cam-x -120 x) (+ cam-y -64 y) 0 1)
      (next-frame self dt)
      (when (= frame collect-sprites.n)
        (tset coins self nil)))))

There are some wonky numbers in here to adjust to the center of the screen and camera position in the world, but the code should be straightforward. The (self:sprites) call simply returns the object’s current set of sprites, which I change in other functions depending on the situation. E.g. if the coin was collected, the object’s sprite is changed to the coin-collect-sprites which doesn’t loop, and the last sprite of that animation is invisible. The coin itself is deleted from the world once the last frame of the animation was played out.

Your browser does not support the video tag.

This is the basis for all animation in the game. Each object has it’s :draw method, and each such method calls to next-frame to understand what to display next time.

Interaction with entities is done similarly - each entity has an :interact method that decides what to do when a collision is detected. For example, for the slime enemy, the method accepts the y normal of the collision, and if it is not negative 1, which means that the collision happened from the side or from below, the player is damaged. Otherwise, the slime sprite set is replaced with the stomp animation, and the object is deleted from the world:

Your browser does not support the video tag.

By the way, I haven’t talked about how I structure the world yet. You may know, that TIC has an inbuilt map editor:

This is basically a level designer for me because I’ve implemented a small function that upon loading the cart walks through each tile on the map and populates the world based on it. I still use map for drawing the level, but before that, all entity tiles are removed from the map, like in the image above coins and slimes, and the player’s head, which acts as a spawn point. This way I can simply put objects in the map editor, and re-run the game, getting an updated world, complete with collectibles, power-ups, enemies, etc.:

Your browser does not support the video tag.

Here I’m obviously testing things out, but the game feels complete enough to actually start designing some actual levels. I’m running out of time though, so perhaps I’ll just make a single course to complete, and that’s it. But it’s already more than I’ve anticipated at the beginning!

I’ve also started working on the menu, but there’s nothing to show yet.

From now, I’ll probably stop doing any in-game stuff, and concentrate on the level itself and I also need some audio to play when the player collects coins, takes damage, jumps, and so on. Maybe I’ll even include some music if I’ll manage to compose something not too annoying to listen to.

Comment via email

Game1 W3/4 (again)

Sat, 22 Jul 2023 21:03:00 +0300

Who knew that writing a post about how I’ve procrastinated for two whole weeks instead of following my own challenge would be so motivating? Immediately after I posted the previous post on Monday, I regained interest and started working on physics integration. Well, making a platformer without experience turned out to be harder than I thought, though I got stuck on things that are not platformer specific.

For the first half of the week, I’ve been trying to implement collision resolution myself. This… didn’t go well, as can be seen here:

Figure 1: The green rectangles depict the area in which I check for possible collisions, and red rectangles mark tiles that collided with the player

I’ve read some tutorials on TIC’s wiki, mainly this one. It mentions how to track collisions by checking nearby squarest cells on the map grid. That was exactly my idea, but it didn’t work well, as was shown in the GIF above.

I suspect the main reason for that is that the player object moves at irregular intervals because the distance is determined by velocity, and not constant as in the tutorial. Thus, the object often overshoots or undershoots the wall and the floor, which led to the player stuck in walls, or going completely through them. It reminded me of problems I had with my previous game, where the ball often got through the platform and bricks, so after failing to do physics again I decided to use a library.

Using a library in a TIC game means that the library has to be included in the game cart because there’s no other way of shipping games for this platform - a cart is a self-contained program without external dependencies. TIC’s carts also have limited capacity so the amount of code I can include is also limited. Fortunately, there’s a small enough library called bump.lua that does AABB collision detection, which is exactly what I need for this type of game. TIC games are tile-based, so axis-aligned rectangles are exactly what I need to handle. With some trickery, it is even possible to implement slopes by using special kinds of boxes that define a slope offset based on the X coordinate.

There was a problem though. bump.lua is a Lua library, and while TIC supports Lua, I’m doing this game in Fennel. Usually, it’s not a problem, because projects written in Fennel can use Lua libraries without any issues, and technically I could have used bump.lua as a library from TIC-80 too. But, as I mentioned, TIC carts must be self-contained, and thus can’t load an external library. So I had to decide - either I port bump.lua to Fennel, or compile my Fennel code to Lua and continue the development this way. I also considered re-implementing a smaller subset of bump.lua specifically for using it in TIC with map tiles, which would probably make the library smaller and easier to handle, but I decided that I want to actually see some results of my work at the end of this month.

There was another compelling way of handling this situation though. I could have used Fennel’s -require-as-include compile flag to, well, include bump.lua as an embedded dependency, and after the development is finished compile the cart to Lua right before shipping. This would be the easiest way of doing things, as it would mean that I can do a completely normal development and use any libraries I want, but the downside is that the cart will be a hot mess of obfuscated Lua code, and I wanted the cart to actually contain something readable. Maybe I’ll try this in the future, who knows?

So, after some consideration of continuing in Lua and just copying bump.fnl code into the cart, I decided to translate bump.lua to Fennel. As much as I like Lua, Fennel is more pleasant to work with. It wasn’t that hard to do, given that there’s an excellent Lua-to-Fennel decompiler, that did most of the job. It does generate weird code here and there, calling set-forcibly! every now and then, and inserting (lua "return ") in places where Lua code had a sole return, or doing things like this:

-- original Lua code
local function sortByTiAndDistance(a,b)
  if a.ti == b.ti then
    local ir, ar, br = a.itemRect, a.otherRect, b.otherRect
    local ad = rect_getSquareDistance(ir.x,ir.y,ir.w,ir.h, ar.x,ar.y,ar.w,ar.h)
    local bd = rect_getSquareDistance(ir.x,ir.y,ir.w,ir.h, br.x,br.y,br.w,br.h)
    return ad < bd
  end
  return a.ti < b.ti
end

;; Lua decompiler result
(fn sort-by-ti-and-distance [a b]
  (when (= a.ti b.ti)
    (local (ir ar br) (values a.itemRect a.otherRect b.otherRect))
    (local ad
           (rect-get-square-distance ir.x ir.y ir.w ir.h ar.x ar.y ar.w ar.h))
    (local bd
           (rect-get-square-distance ir.x ir.y ir.w ir.h br.x br.y br.w br.h))
    (let [___antifnl_rtn_1___ (< ad bd)] (lua "return ___antifnl_rtn_1___")))
  (< a.ti b.ti))


;; cleaned version
(fn sort-by-ti-and-distance [a b]
  (if (= a.ti b.ti)
      (let [ir a.itemRect
            ar a.otherRect
            br b.otherRect
            ad (rect-get-square-distance
                ir.x ir.y ir.w ir.h ar.x ar.y ar.w ar.h)
            bd (rect-get-square-distance
                ir.x ir.y ir.w ir.h br.x br.y br.w br.h)]
        (< ad bd))
      (< a.ti b.ti)))

Code Snippet 1: e.g. here, the decompiler transformed a single-branch if into when and generated an explicit return, but a smarter decompiler could transform this into two branched if without any early returns. Conceptually these two variants are the same, except one is more like the guard-return style and another is more like actual lisp-style.

So I took some extra time to clean all set-forcibly! calls, restored the original comments which were lost during translation, and removed all instances of explicit early returns, by converting the code to use branches instead. After compiling the library back to Lua, I tested it with some demos listed in the library’s readme and it worked fine, so I hope I didn’t introduce any new bugs during the cleanup state. The next thing was to integrate it into the game.

It wasn’t that hard to do, as I just copied the whole library code into the cart, but I was worried if it will be able to handle lots of squares. My idea was to iterate over every map cell, check if it is solid or not, and if so add it to the world. Thus, each non-empty tile on the map exists in the world and acts as a collideable object. As a result, I can use the map editor basically as a level editor, as adding a tile basically is like adding a world object.

The resulting performance seems to be fine on my PC, though I’m unsure about the browser which I’ll target to make the game playable on my itch.io web page. I’m still exploring the idea of using special tile flags to make rooms longer than they seem on the map, as described in the platformer tutorial that I’ve mentioned. But this probably requires special handling when it comes to integrating it with the bump library. For now, the result (with some debug info) looks like this:

Figure 2: bump.lua in TIC-80 with some debug visualisations of collisions

The next thing to do was the camera. Now, since each object in the world has the same coordinate as the map, the only thing I need to do is implement the translation of world coordinates to screen coordinates based on player position with some slight interpolation. Fortunately enough, there’s a handy tutorial on camera movement, and while I’d like to try and implement this myself, I decided to rely on external source of wisdom.

Unfortunately, for some weird reason, the camera described in this tutorial worked differently for me. For some reason, the player sprite was drawn in the top left corner of the screen, and the camera was also positioned there. I mean, it moves correctly but is unplayable in this state. After I’ve experimented with some random numbers I’ve managed to get it to work:

Figure 3: The camera moves with a slight delay to add a bit more dynamic feel, but I didn’t go for a more proper camera implementation that tries to show more in the direction of movement yet. Also, what’s up with these GIFs generated by TIC, their size is around 10MB each, so I had to heavily compress them myself, yet they look entirely the same afterwards.

With that in place, now I need to design some levels and add more graphics, and sounds. As well, as implement actual environment interactions, such as roaming and collideable enemies, stationary hazards like spikes, health, score, and respawn systems, and maybe some other intractable objects, like coins to collect. Not sure what I’ll do first, probably the latter, as it will teach me how to add collideable items to the world that should not be treated as platforms.

Comment via email

Game1 W3/4

Mon, 17 Jul 2023 23:07:00 +0300

Long story short, no progress so far. I’m starting to think that the way I’ve set this challenge up actually mostly demotivates me to continue rather than encourages me to press forward. To say that I’ve been working on Game1 will be both a lie to myself and to you. Well, I did, but in a tinkerer’s fashion - rewriting one piece of code til I’m satisfied (except I never am).

So far I’ve managed to put sprites into the TIC-80 engine and created a simple controller, that actually doesn’t work how I expected it to, and that’s it:

Mostly because for the past two weeks I was either distracted by work or by working on a new Emacs window manager described in the last post (that, probably, no one needs, even me).

I have two weeks left (including this one), and I don’t think I’ll be able to finish this one in time, at least in its full shape. Maybe I’ll settle on a simpler thing, like a demo in which I just explore how to write a platformer, it’ll still be a good start.

Comment via email

Clojure's core.async port for the Fennel language

Mon, 15 May 2023 22:44:00 +0300

After I explored asynchronous programming in my first naive implementation a few years ago, I was hooked on the idea of asynchronous communication in programs. Motivated to take this concept further I’ve created the fennel-async library that I’ve shown at FennelConf 2022. While this library works, and I’ve managed to write a simple socket-based REPL and integrate it into Emacs as an experiment, I wasn’t satisfied with the implementation. Mainly because the library is pull-based, which means that the tasks are pulled by the continuously running scheduler. It poses some difficulties in Lua runtime, as there’s no support for multithreading, and the scheduler can’t be offloaded to a separate thread. I soon realized that a proper asynchronous system is push-based, meaning that pushing to the task triggers its activation. With that in mind I’ve started working on a newer version of the library: async.fnl

The library started as a small channel-based communication system, where each channel had its own queue of puts, a queue of takes, and an optional buffer, much like core.async from Clojure. I was planning to only implement put!, take! and alts! operations, as these are the bare minimum that is needed for the library to be functional, as everything else is bootstrapped on them. However, it was hard to test the library this way, so I started adding more and more operations, like merge, into, eventually added the support for transducers, and before I knew it I was thinking about how to port Mult, Mix, and Pub. At that moment I decided that I will simply port everything from the core.async, including tests, and that’s where my implementation of channels started failing. So I ported the channels too.

Now this library is a mostly complete port of the ClojureScript version of the library. The reason I’ve chosen ClojureScript as a base is that it is also a single-threaded system, so it should behave similarly to Lua, which Fennel uses as its target for compilation. Interestingly enough, and you don’t have to believe me¹, but up until I decided to port the tests, and later the channel implementation, I was writing this library without looking at the core.async source code and the resulting Fennel code was quite similar. Well, this is largely due to the fact that channels are well covered by Rich Hickey in his talk, and I’ve already made three wrong implementations of them, so I knew what I need to do now for the most part. I’m also a bit familiar with Clojure’s core.async code, but only with the public API. I’ve never looked into the implementation of channels, as my tinkering with the library only needed to go as far as the implementation of pipelines, which I’ve covered in my post about fixing the off-by-two error.

So when the tests started to fail, I knew that something wasn’t right, so after some hours of debugging I ditched my own implementation of channels and ported the code from ClojureScript into Fennel. This was a fun experience, and I’ve even “ported” reify and defprotocol just not to think too much about how to write it manually in every function. I’ve put ported in quotes because it’s not a true port, just a similar macro that expands to the runtime definition of an object that conforms to the defprotocol spec:

(macro defprotocol [name ...]
  `(local ,name
     ,(faccumulate [methods {} i 1 (select :# ...)]
        (let [method (select i ...)]
          (assert-compile (list? method) "expected method declaration")
          (let [[name arglist body] method]
            (assert-compile (sym? name) "expected named method" name)
            (assert-compile (sequence? arglist) (.. "expected arglist for method " (tostring name)) arglist)
            (assert-compile (or (= :string (type body)) (= nil body)) (.. "expected no body for method " (tostring name)) body)
            (doto methods
              (tset (tostring name) `#(<= $ ,(length arglist)))))))))

(macro reify [...]
  (let [index (gensym)
        protocols []
        actions '(do)]
    (var current nil)
    ((fn loop [x ...]
       (assert-compile (or (sym? x) (list? x)) "expected symbol or fnspec" x)
       (if (sym? x)
           (do (set current x)
               (table.insert protocols (tostring x)))
           (list? x)
           (let [[name & [arglist &as fnspec]] x]
             (assert-compile (sym? name) "expected method name" name)
             (assert-compile (sequence? arglist) "expected method arglist" arglist)
             (table.insert
              actions
              `(case (. ,current ,(tostring name))
                 f# (if (f# ,(length arglist))
                        (tset ,index ,(tostring name) (fn ,(unpack fnspec)))
                        (error ,(.. "arity mismatch for method " (tostring name))))
                 ,(sym :_) (error ,(.. "Protocol " (tostring current) " doesn't define method " (tostring name)))))))
       (when (not= 0 (select :# ...))
         (loop ...)))
     ...)
    `(let [,index {}]
       ,actions
       (setmetatable
        {}
        {:__index ,index
         :name "reify"
         :__fennelview
         #(.. "#<" (: (tostring $) :gsub "table:" "reify:")
              ": " ,(table.concat protocols ", ") ">")}))))

It’s a little messy but works fine. I added it when I was porting Mult, which looks a lot like the ClojureScript version:

(defprotocol Mux
  (muxch [_]))

(defprotocol Mult
  (tap [_ ch close?])
  (untap [_ ch])
  (untap-all [_]))

(fn mult [ch]
  "Creates and returns a mult(iple) of the supplied channel
`ch`.  Channels containing copies of the channel can be created with
'tap', and detached with 'untap'.

Each item is distributed to all taps in parallel and synchronously,
i.e. each tap must accept before the next item is distributed.  Use
buffering/windowing to prevent slow taps from holding up the mult.

Items received when there are no taps get dropped.

If a tap puts to a closed channel, it will be removed from the mult."
  (var dctr nil)
  (let [atom {:cs {}}
        m (reify
           Mux
           (muxch [_] ch)
           Mult
           (tap [_ ch close?] (tset atom :cs ch close?) nil)
           (untap [_ ch] (tset atom :cs ch nil) nil)
           (untap-all [_] (tset atom :cs {}) nil))
        dchan (chan 1)
        done (fn [_]
               (set dctr (- dctr 1))
               (when (= 0 dctr)
                 (put! dchan true)))]
    (go-loop []
      (let [val (<! ch)]
        (if (= nil val)
            (each [c close? (pairs atom.cs)]
              (when close? (close! c)))
            (let [chs (icollect [k (pairs atom.cs)] k)]
              (set dctr (length chs))
              (each [_ c (ipairs chs)]
                (when (not (put! c val done))
                  (m:untap c)))
              (when (next chs)
                (<! dchan))
              (recur)))))
    m))

Sure, fennel doesn’t have sequences and atoms by default, so this uses mutable tables, but the semantics are the same.

In general, Fennel isn’t exactly compatible with Clojure, as it is a different language. There’s no arity overloading, data structures aren’t persistent, and there are no lazy sequences, only iterators over tables. All these features can be implemented as libraries, and I’ve already made a bunch that should make a Clojure programmer feel more at home. Well, at least the strings are immutable by default, which is good!

I’ve chosen the same versioning scheme as the Clojure version of the library, but it is still in an experimental state. More tests should be added, I also need to test on different runtimes, as this is only tested to work on Lua 5.4 at the moment.

Differences from the Clojure version

One tricky thing this library requires is the presence of the debug Lua library. It is included by default in most Lua implementations, but it may be absent in an embedded Lua.

The debug library is required to run timers. By using the debug.sethook function, when the library is loaded, a hook that checks if any of the timeout channels have expired runs every 1000 Lua instructions, and on each function return. This needs a bit more profiling to get an optimal amount of instructions between runs, but the tests show a nice accuracy with the error of 10ms, which is largely due to the library trying to minimize the number of timeout channels by rounding their duration to these 10ms. It’s similar to what the Clojure version does, e.g. the expression (= (timeout 1000) (timeout 1000)) is true because only one timeout channel is created under the hood. When the debug library is unavailable, the timeout function throws an error, but the rest of the library should work fine, as timeout channels aren’t used anywhere.

Another tricky thing is that Lua by default doesn’t support sub-second precision for measuring time. The default os.time reports the number of seconds since the epoch, and there’s no way to get the number of milliseconds. The solution is to check if the luasocket or luaposix libraries are present on the LUA_PATH. If so, their implementations of the gettime function are used, which gives millisecond precision. Otherwise, the number of milliseconds passed to the timeout function is rounded up to the next whole second. Not ideal, but works, especially since the asynchronous code shouldn’t be too dependent on specific timings. The library will warn you if you don’t have any of these installed.

And the final difference from Clojure is that there is no inversion of control (IOC) transformations involved. Lua, unlike the JVM, supports sackful coroutines, which means that any function can be turned into coroutines that can be stopped and resumed with the coroutine.yield call. Thus, all operations that would result in a blocking behavior, like <! or >!, actually just capture the running thread with the coroutine.running put it to the respective queue on the channel where the operation occurs, and the channel calls coroutine.yield. Next, when this channel receives another put or takes if there’s a such pending thread, it will be resumed with the value of the operation. Stackful coroutines make the implementation much simpler, without the need to write complex macros that transform the code into state machines. I wish more languages had them, as it is a great feature that can be used in a variety of ways. Generators are cool, but Lua coroutines are much cooler. But I digress.

Future plans

Overall, this was a fun experiment, and while I really like asynchronous programming, I feel that I need to stop myself here. There are a lot of other topics I want to get into, like the bytecode VM design, gamedev, and other cool stuff. So hopefully there will be no more async shenanigans from me. Though, maybe I’ll try to port the TCP module from fennel-async into this library, but we’ll see. That may be difficult, as there’s no way to register a callback on a socket object with the luasocket library, which almost defeats the purpose of being push-based.

If you’re using Fennel, and you’re familiar with core.async do give this library a test. I’m curious how it will work in other programs because I’m making all these libraries, but I don’t really have any time to use them as parts of other projects, besides maybe the fenneldoc, which uses the cljlib library.

Anyway, thanks for reading, and I hope this library will be useful!

You don’t have to, but if you read my blog, you may remember that that’s the way I usually like to do stuff. Diving in without any or very little amount of prior knowledge on the topic and thinking about how would I implement that. And if the result is similar to the reference implementation or differs from it, that’s how I know if I made any mistakes, or if my assumptions were correct. This library started the same way. ↩︎

Comment via email

New Fennel Proto REPL and call for testing

Sat, 08 Apr 2023 10:25:00 +0300

In the previous post I’ve described how to define a simple protocol, upgrade the stock Fennel REPL with it, and create a simple client that works with this setup. And at the end, I mentioned that I was working on a proper client implementation and a more robust protocol as part of the fennel-mode package. So today I’m presenting the new fennel-proto-repl module!

It’s an experimental implementation of the new Fennel REPL client, that is based on the message-callback system, and it provides more robust ways of communicating with the REPL process. We’re not going to remove the old REPL integration support, and it going to remain the default one since I expect that some rough edges still need to be refined, and we still need to wait until Fennel will release the next stable version anyway. But if you’re using the development version of Fennel or don’t mind trying things, you can update fennel-mode and try the new REPL implementation by adding the following hook to your Emacs configuration:

(add-hook 'fennel-mode-hook 'fennel-proto-repl-minor-mode)

This new minor mode enables the integration with the new REPL implementation, meaning that all usual commands, such as Ctrl+Alt+x (eval-defun) will use the new REPL implementation.

At the end of the previous post I mentioned that the protocol-based REPL may not work in all cases, but this limitation should now be lifted in the latest Fennel (that’s why the Fennel update is required). I’ve sent a patch that makes inbuilt Fennel REPL more dynamic, and now the currently running REPL can be changed at runtime. Thus, if your application uses a custom REPL, like the one in the min-love2d-fennel repo, you should be able to use new Proto REPL, you’ll just need to update the Fennel dependency in your application.

So this post is a call for testing! If you experience hiccups or problems with the stock fennel-repl mode in Emacs, give fennel-proto-repl a try!

What’s new

Here I’ll briefly list enhancements that are now possible thanks to the protocol. There aren’t a lot of new features, the new client mostly provides stability-related improvements, yet, there are some things I’d like to note.

Programmatic API

First of all, now it is possible to send both synchronous and asynchronous messages to the REPL process and use a system of callbacks to get back the results. Previously it was possible but it was quite finicky because of how comint¹ works, and thus all requests sent outside of the REPL were synchronous (like completions or documentation fetching). Again, I’ve described it in detail in the previous post. The REPL itself is still synchronous - all messages are processed sequentially, but the client is now decoupled and doesn’t have to proactively wait for each message to be processed.

The new API includes these two functions: fennel-proto-repl-send-message, and fennel-proto-repl-send-message-sync. The first one will never block Emacs and should be used whenever you want to send some code to the REPL but you don’t require the result immediately. Here’s an example:

(fennel-proto-repl-send-message
 :eval "(+ 1 2 3)"
 (lambda (vals) (message "%S" vals)))

In this case, the result will be shown when the REPL is ready to process the message. You can additionally provide error callback and printing callback to capture error messages or data that the REPL requested to print.

(fennel-proto-repl-send-message-sync :eval "(+ 1 2 3)") ;; => ("6")

This will block Emacs until the REPL is ready to process the message, so it should be used with care. The message, however, can timeout after a configurable amount of time (5 seconds by default), so the block will not be indefinite.

Most of the features in the new client are implemented via these two functions, so it should be enough for other packages to integrate with the new client if necessary. I’m already thinking about updating my ob-fennel package to use this new REPL implementation, as it will fix a lot of bugs and problems in general.

Multiple REPLs

Support for multiple REPL sessions received some improvements. Previously, with the comint-based REPL, in order to have more than one REPL active at a time, it was required to rename the existing REPL buffer, and then start a new one. Otherwise, the function that starts the REPL just jumped to the already running one. This worked, however, the in-buffer commands, such as eval-defun or eval-last-sexp started working with the new REPL buffer unless it also was renamed. If there wasn’t a REPL buffer with the default name these commands didn’t work.

Fennel Proto REPL introduces a way to link a certain buffer to a certain REPL with the fennel-proto-repl-link-buffer function. This function provides a list of currently running REPLs to choose from, and after choosing one all in-buffer commands start working with that REPL. Additionally to that, it’s no longer necessary to manually rename the REPL buffer to start a new one - the fennel-proto-repl function always starts a new REPL and automatically links the current buffer to it.

Improved error reporting

The protocol changes how the errors are reported to the client, so the client could better analyze the stack traces and make them clickable. Now, when the error occurs, a special buffer will pop up, displaying the detailed message and a clickable stack trace:

The REPL itself shows a shortened message to avoid too much visual noise.

Improved Xref, Eldoc, and completion-at-point support

Fennel REPL has a command that can provide you with the target file and line of the definition you’re interested in, but it’s a bit tedious to copy it from the REPL every time. So the new Xref integration uses a special protocol OP that will automate most of the things.

Eldoc support was already part of the fennel-mode, but it was implemented poorly. The new implementation is asynchronous, and should never block Emacs if the REPL is busy, like what’s happened with an older implementation. We will remove eldoc support from basic fennel-repl in the future as it causes lots of problems. Unfortunately, asynchronous Eldoc is a feature of Emacs 28+. I’ve decided not to implement support for older versions of Emacs because in the case of Fennel, there’s no way of querying for documentation concurrently with the currently running task, like can be done in other languages, and blocking Emacs is not fun.

Figure 1: Argument list hint by Eldoc

Figure 2: Documentation hint by Eldoc

Completion at point also received some enhancements and should be much faster now, because everything is now processed in batches. The previous implementation requested a list of completions with one command and then sent a request for each item to get its kind. The new implementation does an initial request, and then a second request that gets kinds for all results of previous requests and caches them until the next completion happens.

Figure 3: Completions, complete with candidate kinds, and documentation

Overall, all these features should be more robust, because we no longer require parsing the output of the REPL, as we had to with comint.

Rough edges

The implementation is not perfect, so there are some things to be concerned about.

The client code is still can be improved performance-wise. Nothing crazy, but if the process starts spitting out lots of output, it can start piling up and Emacs will chug on it. This happens when printing huge tables, like _G, or when a tight loop produces a lot of output, like (for [i 1 1000000] (print i)). Disabling process logging can help, and this is why the logging is disabled by default. Though it should not be a major problem because large amounts of output are rare, and data is usually small.

The buffer linking mechanism is not as user-friendly as something more automatic, like linking REPLs based on project structure, I agree. But I hope it is not as bad, and we can always improve this part later as we gather more feedback from the users.

Lastly, the overall complexity of the client may seem daunting, and there’s a lot of code but the architecture is quite simple, it’s just that there’s a lot of code for various features.

I hope that it will be good enough to be the default REPL one day. However, as I’ve mentioned, there may be problems with embedded REPLs or just the fact that you can’t update the old Fennel version due to some circumstances. For these cases, there’s always the old implementation of the REPL.

The Protocol library

The protocol behind the new client is in fact editor-agnostic and can be used by other editors. Because of that, I’ve released the protocol as a library: fennel-proto-repl-protocol, and I’m looking forward to others trying it and implement support in their editors.

The protocol setup is quite simple - all you have to do is either require the protocol library from the REPL and call the provided function with an editor-specific format function. Or you can send the whole protocol code to a running REPL without using require as the fennel-proto-repl module does.

Here’s an example for JSON based messaging:

>> (fn format-json [env data]
     (: "{%s}" :format
        (table.concat
         (icollect [_ [k v] (ipairs data)]
           (: "%s: %s" :format (env.fennel.view k)
              (: (case v
                   {:list data} (.. "[" (table.concat data " ") "]")
                   {:string data} (env.fennel.view data)
                   {:sym data} (tostring data)
                   _ (env.protocol.internal-error "Wrong data kind" (env.fennel.view v)))
                 :gsub "\n" "\\\\n"))) ", ")))
#<function: 0x563fec938af0>
>> ((require :protocol) format-json)
{"id": 0, "op": "init", "status": "done", "protocol": "0.1.0", "fennel": "1.3.1-dev", "lua": "PUC Lua 5.4"}

For more info, see the project’s readme.

All in all, I hope this will help make better tooling for Fennel in the future, and will bring the programming experience closer to what’s available for other Lisps.

Comint is the generic mechanism for implementing interactive shells in Emacs. It works by setting up a process output filter and convenience functions for sending input to the process, but it’s hard to use it as a machine-to-machine interface. ↩︎

Comment via email

Implementing a protocol-based Fennel REPL and Emacs client

Sat, 25 Mar 2023 19:47:00 +0300

Recently I read a post by @nikitonsky about writing a custom REPL for Clojure and Sublime Text. What got my attention was a way of implementing a protocol over a plain Clojure REPL. In the post, Nikita describes a way to “upgrade” the connection by sending code that basically implements a new REPL to the main REPL process. I liked the idea and decided that two can play this game, and started working on a similar thing, but for Fennel.

A few months ago I’ve already proposed the idea of a simple protocol for the Fennel REPL. The idea was met with a certain amount of skepticism because there’s already an implementation of a such protocol that is language agnostic (in theory) and battle-tested. The protocol in question is called nREPL and it is widely used in the Clojure ecosystem. There are several client implementations for various editors, and the protocol is extensible enough to support other languages and various clients.

In fact, Fennel already has an implementation of the nREPL protocol, called jeejah. However, it has problems.

First of all, nREPL is mainly designed for Clojure, tested against Clojure, and implements features that Clojure needs in a pretty specific Clojure way. It’s not impossible to implement support for other languages, and there were attempts of various completeness.

Another problem is that nREPL has “n” in its name for a reason. It’s a Network REPL, which means that the protocol is designed around network communication pipelines. In principle, it is possible to implement a standard-input/-output or a pipe-based version, but no clients will probably support it. And Fennel, running on Lua, has some problems with networking. The luasocket library becomes a dependency and there’s no clear way of implementing asynchronous communication. Again, not impossible, JeeJah does it, but it’s hard to do it properly.

There’s also bencode. It’s an encoding that is used by torrents. Encoding and decoding messages may be slow, especially in a single-threaded REPL. And we’ll need to re-implement half of fennel.view the serialization function to support all kinds of Lua quirks.

Nikita also has some thoughts on nREPL:

You need to add nREPL server dependency to your app. It also has a noticeable startup cost (~500ms on my machine).

Which is a good point, although it’s not completely true. Some existing nREPL clients, such as CIDER, can inject the nREPL dependency into your project, so you don’t need to actually ship it with your application. You develop with nREPL, and use all of its goodies, yet your app has no nREPL once it’s shipped. So you only need to include the nREPL dependency if you want to ship your application with capabilities for remote code execution.

This is not really possible with Fennel, however. There’s no package manager for fennel, and neither there is a way to dynamically inject an entire nREPL library from the client. But what about injecting a smaller and simpler protocol?

Why, yes! That’s exactly what Nikita is doing in their Clojure-Sublimed plugin, and I’ve decided to do the same in Emacs.

The protocol

If you’ve read the mailing list discussion I linked earlier, you’ve seen that I envisioned line-based, tab-separated messages. I’ve scraped the idea and went for data-based communication: table in, plist out. Tables can be interpreted by fennel, we can pattern-match on them, and store arbitrary amounts of data. Plists are natively understood by Emacs Lisp, which is an implementation language of our client.

So my idea was that I send something like that to Fennel REPL:

{:id 1 :eval "(+ 1 2 3)"}

And get back something like:

(:id 1 :op "accepted" :data t)
(:id 1 :op "eval" :values ("6"))
(:id 1 :op "done")

I’ll explain why there are three outgoing messages for one incoming, but a bit later. Right now we need to address the main problem: how do we teach Fennel to work in terms of our protocol?

When I thought about the protocol for the first time, I thought that it will be part of the Fennel REPL by default. However, Fennel is quite minimalist, and the choice was not to include anything like that. The REPL is already better than the Lua one. This is true when we’re talking about interactive usage, but not when we’re talking about machine interaction.

So the answer is - REPL upgrade.

Upgrading the REPL

This technique is using the ability to evaluate in the existing REPL, basically implementing a different REPL at runtime.

For example, here’s the simplest REPL:

(let [fennel (require :fennel)]
  (while true
    (io.write "repl>> ")
    (io.flush)
    (print (fennel.eval (io.read)))))

Sending it to Fennel REPL starts our own REPL inside of it:

Welcome to Fennel 1.3.0 on PUC Lua 5.4!
Use ,help to see available commands.
Try installing readline via luarocks for a better repl experience.
>> (let [fennel (require :fennel)] (while true (io.write "repl>> ") (io.flush) (print (fennel.eval (io.read)))))
repl>> (+ 1 2 3)
6

However, it’s more of a downgrade, rather than an upgrade. Our prompt doesn’t support unfinished expressions, comma-commands no longer work, and so on. The purpose of this example is to give you a general idea of how the upgrade process works.

Reimplementing the REPL from scratch is a tough task, but we don’t even have to do it! Fennel comes with an extensible REPL already: the fennel.repl function accepts a table with the following callbacks:

readChunk - poll for the user input.
onValues - called when the REPL returns the result of the evaluation,
onError - called when the error occurs,
pp - pretty-printer function

If we implement these functions we’ll get a REPL that works the way we need it to.

Implementing the Proto(col|type) REPL

First things first, we need fennel the library. We’ll use some of its public API, later on, but the feature we’re interested in the most right now is fennel.repl:

(local {: repl : view : eval : parser &as fennel} (require :fennel))
(local {:read io/read :write io/write} io)
(local protocol {})

I’m caching fennel and io functions to avoid some repeated table lookups. It’s only part of the reason though, but more on that later.

(fn proto-repl []
  (protocol.message [[:id 0] [:op "init"] [:data "done"]])
  (repl {:readChunk protocol.read-chunk
         :onValues protocol.on-values
         :onError protocol.on-error
         :pp (fn [data] (view (view data)))}))

I’ll go through the implementation of each callback but I will not implement the whole protocol as part of this post. The protocol while small, still has a lot of code, so I’ll only do parts that pose interesting challenges. It will still be a functional protocol, just not as feature-full as the one I’ve actually implemented for Emacs integration.

First things first, let’s make the read-chunk callback. In the built-in REPL, it is used to print the prompt and handle user input. We don’t need the prompt, as this will not be an interactive REPL, but we do need to poll for user input. Not just that, our input comes in a form of a message, so we also need to parse it.

(fn protocol.read-chunk [parser-state]
  (io/write ">> ") (io.flush)
  (let [message (io/read :l)
        (ok? message) (pcall eval message)]
    (if ok?
        (case message
            {: id :eval data} (protocol.accept id :eval data)
            _ (error "message did not conform to protocol"))
        (error (.. "malformed input: " (tostring message))))))

In the read-chunk function we’re reading a line from the client and eval it. I could have used just the parser here, but you’ll see why having eval here is useful. If the evaluation was successful, we call case and do pattern matching on the message. If the message matches any pattern (right now the only one) we know how to process it. The processing is handled by the accept function, so let’s implement it:

(set protocol.current-id -1)

(fn protocol.accept [id op data]
  (protocol.message [[:id id] [:op "accept"]])
  (set protocol.current-id id)
  (case op
    :eval (.. data "\n")
    _ (error (.. "unsupported op: " op))))

The code is quite simple. The message function is what sends a message (:id 1 :op "accepted") to the client. Here’s how we implement it:

(fn protocol.message [kvs]
  (io/write (protocol.format kvs) "\n")
  (io.flush))

(fn protocol.format [kvs]
  (.. "("
      (table.concat
       (icollect [_ [k v] (ipairs kvs)]
         (: ":%s %s" :format k
            (if (= :table (type v))
                (.. "(" (table.concat v " ") ")")
                (view v))))
       " ")
      ")"))

The message function simply prints messages to standard out, while format handles transforming our sequence of pairs to Emacs Lisp plist.

Believe it or not, we’re almost done. What’s left are on-values and on-error:

(fn protocol.on-values [data]
  (protocol.data protocol.current-id data)
  (protocol.done protocol.current-id))

(fn protocol.data [id data]
  (protocol.message [[:id id] [:op "eval"] [:values data]]))

(fn protocol.on-error [_ data]
  (protocol.error protocol.current-id data)
  (protocol.done protocol.current-id))

(fn protocol.error [id data]
  (protocol.message [[:id id] [:op "error"] [:message data]]))

The done function is the final piece of the puzzle:

(fn protocol.done [id]
  (protocol.message [[:id id] [:op "done"]]))

With all of this, we should get our protocol-based REPL up and running after calling (proto-repl):

>> (proto-repl)
(:id 0 :op "init" :data "done")
>> {:id 1 :eval "(+ 1 2 3)"}
(:id 1 :op "accept")
(:id 1 :op "eval" :values ("6"))
(:id 1 :op "done")

Nice! Now, obviously, this omits a lot of plumbing, error handling, other operations, and so forth, but it should give you the idea. Also, there are a lot of unnecessary table lookups, I just wrote the code in such a way so you could evaluate them one after another without forward declarations. But, as a result, sending each of those code blocks to a plain Fennel REPL gives us a new REPL that acts in terms of a specified protocol. Now we’ve really upgraded the REPL.

Unfortunately, this will not work as is, not because we’re missing ops or error handling though. There’s an elephant in the room, consider this message: {:id 1 :eval "(print :foo)"}. What would happen if we send this message to our REPL?

>> {:id 1 :eval "(io.write :foo)"}
(:id 1 :op "accept")
foo(:id 1 :op "eval" :values ("#<file (0x7feff913f780)>"))
(:id 1 :op "done")

Right, the output is mixed with the protocol messages. Here’s another one:

>> {:id 1 :eval "(io.read :l)"}
(:id 1 :op "accept")

Now our REPL suddenly waits for user input midway. So if another protocol message comes, it will be consumed by this read. So we have to handle this too.

Handling IO

Unfortunately, Lua doesn’t have any way of redirecting output from standard out to something else. Clojure kinda does:

(with-out-str
  (.println System/out "foo"))

Even though we call the println method of the System/out class, this expression still returns a string "foo\n". It is possible because in the JVM there are ways to configure that. I’ve tried using io.output in Fennel, but it can only be set to a file, we can’t pass it a table that implements the necessary methods:

;; naive approach
(let [orig-out (io.output)
      fake-file {:data ""
                 :write (fn [t s] (set t.data (.. t.data s)))}]
  (io.output fake-file) ; runtime error: bad argument #1 to 'output' (FILE* expected, got table)
  (print :foo)
  (io.output orig-out)
  fake-file.data)

So we need to set up IO in such a way that it works as expected when we communicate through a protocol, yet wraps the IO inside of the user’s code. Fortunately, we can do that:

(fn set-io [env]
  (let [{: stdin : stderr : stdout} io
        {:write fd/write
         :read fd/read
         &as fd} (. (getmetatable stdin) :__index)]
    (fn env.print [...]
      (env.io.write (.. (table.concat [...] "\t") "\n"))
      nil)
    (fn env.io.write [...]
      (: (env.io.output) :write ...))
    (fn env.io.read [mode]
      (let [input (env.io.input)]
        (if (= input stdin)
            (protocol.read mode)
            (input:read mode))))
    (fn fd.write [fd ...]
      (if (or (= fd stdout) (= fd stderr))
          (protocol.message [[:id protocol.current-id]
                             [:op "print"]
                             [:data (table.concat [...] "")]])
          (fd/write fd ...))
      fd)
    (fn fd.read [fd ...]
      (if (= fd stdin)
          (env.io.read ...)
          (fd/read fd ...)))))

It’s a bit of code, but what it essentially does is this:

store the original values of stdin, stdout, stderr,
define a bunch of replacement functions, like env.print, env.io.write,
capture original *FILE metatable in fd and override it’s metamethods.

Unlike tables, all file objects in Lua share the same metatable, so we only need to redefine metatable entries for the stdin file. Though it may depend on the implementation of the Lua runtime being used.

The env argument here is a table that will be used in the fennel.repl, so we need to change that bit of code. Because we don’t want to actually modify the real _G table, we’ll need to copy it first:

(fn copy-table [t]
  (collect [k v (pairs t)]
    k v))

Here’s our new REPL:

(fn proto-repl []
  (protocol.message [[:id 0] [:op "init"] [:data "done"]])
  (repl {:env (doto (copy-table _G) set-io)
         :readChunk protocol.read-chunk
         :onValues protocol.on-values
         :onError protocol.on-error
         :pp (fn [data] (view (view data)))}))

Now, when we send the {:id 1 :eval "(io.write :foo)"} message, we get proper IO handling:

>> (proto-repl)
(:id 0 :op "init" :data "done")
>> {:id 1 :eval "(io.write :foo)"}
(:id 1 :op "accept")
(:id 1 :op "print" :data "foo")
(:id 1 :op "eval" :values ("#<file (0x7f0e234e7780)>"))
(:id 1 :op "done")

Great, now we get a message with the print OP, and the data to be printed. The client then can detect such message and act accordingly. We can even add support for different descriptors, hide stderr messages, or color them differently. But what about the opposite operation? How do we read input from the user?

This is a bit tricky. The general idea is the same, we need to tell the client that the REPL is awaiting input, but we also need to wait in such way that other messages don’t get confused with user input. You can see that in the set-io function, the io.read calls protocol.read if the descriptor is stdin. Let’s implement protocol.read now:

(fn protocol.read [mode]
  (let [tmpname (with-open [p (io.popen "mktemp -u /tmp/proto-repl.XXXXXXXX 2>/dev/null")]
                  (p:read :l))]
    (: (io.popen (.. "mkfifo '" tmpname "' 2>/dev/null")) :close)
    (protocol.message  [[:id protocol.current-id] [:op "read"] [:data tmpname]])
    (let [data (with-open [f (io.open tmpname)]
                 (f:read mode))]
      (: (io.popen (.. "rm -f '" tmpname "'")) :close)
      data)))

Now, if we call io.read as part of our code, we should get back a message:

>> {:id 1 :eval "(io.read :n)"}
(:id 1 :op "accept")
(:id 1 :op "read" :data "/tmp/proto-repl.adTbqVI2")

If we write data, in this case a number to that file, we should get it processed by the REPL:

echo 27 > "/tmp/proto-repl.adTbqVI2"

And indeed we get the usual messages we get from eval OP:

(:id 1 :op "eval" :values ("27"))
(:id 1 :op "done")

Unfortunately, Lua doesn’t have an inbuilt filesystem library, so this method is not cross-platform. It will probably work under BSD systems, and Macs, but not under Windows, unless you’re using WSL, but still not guaranteed. I don’t have a Windows PC, so I can’t test, but if you know a cross-platform, or even just a special case for Windows, for creating a named FIFO pipe, and attaching to it, it’d be great if you reach me.

This pretty much concludes our very basic, yet fully functional protocol!

The IO part is the trickiest part because there may be specifics to how files are read, but again, we can’t expect that the target application will have luafilesystem installed. But there’s a way to work this around. So we do have a final small change to make in our read-chunk function.

Making the protocol extensible at runtime

You may remember that I’ve chosen eval over parser in the REPL to work with incoming messages. There was a specific reason for it - the message can actually be not for the target application, but for the REPL itself! And we don’t even need any kind of special-casing for it, only a nop OP:

(fn protocol.read-chunk [parser-state]
  (io/write ">> ") (io.flush)
  (let [message (io/read :l)
        (ok? message) (pcall eval message)]
    (if ok?
        (case message
          {: id :eval data} (protocol.accept id :eval data)
          {: id :nop ""} "\n"
          _ (error "message did not conform to protocol"))
        (error (.. "malformed input: " (tostring message))))))

We also need to expose the protocol table in the _G table, and fennel for convenience:

(set _G.proto-repl protocol)
(set _G.fennel fennel)

With these changes, we can send a message like (do (print "hi!") {:id 1 :nop ""}) and it will be processed by the eval, and the print call will be evaluated outside of the main protocol handling. Thus, we can get fancy! Here, for example, how we can make our protocol respond in JSON:

((fn to-json-protocol []
   (fn _G.proto-repl.format [data]
     (.. "{" (table.concat
              (icollect [_ [k v] (ipairs data)]
                (let [v (if (= :table (type v))
                            (.. "[" (table.concat v ", ") "]")
                            (fennel.view v))]
                  (.. (fennel.view k) ": " v)))
              ", ")
         "}"))
   {:id 15 :nop ""}))

Now, if we send this to the REPL, we will get nothing back, since the OP in question is nop, however, the format function will be redefined. So if we send one more message, the usual eval one, we’ll get back JSON messages in response:

>> {:id 1 :eval "(+ 1 2 3)"}
{"id": 1, "op": "accept"}
{"id": 1, "op": "values", "data": ["6"]}
{"id": 1, "op": "done"}

This is why languages with the capability to run any code at runtime are cool in my opinion. We can build our application while it is running. Any protocol method, that we’ve defined can be re-implemented at runtime. So if you’re running Windows, you can redefine protocol.read to support your particular platform without even asking me to update the code. (But please, if you know a more portable way of handling input, send me a message).

As a final change, let’s remove the >> prompt from the read-chunk, and we’re done. The prompt is not needed, it was only to help you differentiate what messages we’ve sent, and what messages we receive. For the machine the prompt will only get in the way:

(fn protocol.read-chunk [parser-state]
  (let [message (io/read :l)
        (ok? message) (pcall eval message)]
    (if ok?
        (case message
          {: id :eval data} (protocol.accept id :eval data)
          {: id :nop ""} "\n"
          _ (error "message did not conform to protocol"))
        (error (.. "malformed input: " (tostring message))))))

Finally, we can add the call to proto-repl to our protocol code, so right after we’ve sent it the new REPL is started automatically:

(proto-repl)

Now, for real, this concludes the basic protocol implementation. This is, in fact, what I started with, and then gradually made it more mature by extending the number of operations the protocol supports, and making it more robust. Not every protocol method is exposed to be changed in my implementation though. This was more like a demo that you can follow, but in the real version of the protocol, it is a single giant function that gets sent to the REPL in one go.

Now, we can talk about the client, because without the client our protocol has no value.

The client

My editor of choice is Emacs for many reasons. And this is one of them - we can make all kinds of applications in Emacs, including a custom client for a custom protocol. We’re like full-stack developers now, but our backend is written in Fennel and our frontend will be written in Emacs Lisp.

As I mentioned at the beginning of the post, I’ve chosen plists because Emacs understands them natively. Still, our messages come in as strings, so we’ll need to parse them. But how will we organize our client to begin with?

Comint

The current implementation of the Fennel REPL in the fennel-mode package uses the inbuilt comint package to do all of the heavy lifting. Comint is Emacs’ generic interface for providing user interaction to asynchronous processes. The selling point is that you can start almost any interactive program in Comint, set the prompt regexp and it will work. However, the problems start when we begin building a machine-to-machine interface over a human-to-machine interface.

As with any kind of interface it involves parsing the output. Our protocol is no different here, we’re still going to parse output, so what’s wrong with doing it via comint, especially since it already knows how to do the majority of things? Comint even has the redirection facility to deal with such tasks specifically.

The answer is - there’s no message queue that spans over both comint and the target process.

For example, you can send a long-running code to the REPL that will print something after a long period of time, like this one:

(do (for [i 1 1000000] nil) (print "foo" "bar" "baz"))

Once the loop is completed the stdout of the process will contain a string like this one: foo\tbar\tbaz. Comint will grab it and print it to the REPL. All good.

But if we set up a redirect while this loop runs, that, say, queries for the completions from the REPL, we can get into a funny situation. Imagine, user typed f and pressed the Tab key. The current implementation of completion support in fennel-mode package will use a comint redirect mechanism, and for this case, it will send the following expression ,complete f to the REPL. The output of the ,complete f command will be fn\tfor\tfcollect\tfaccumulate, another tab-separated line, but our REPL is busy at the moment, running the loop. What can go wrong?

What happens is that the ,complete f is buffered by the process stdin, and not processed unless the REPL is able to read the message. The comint redirect, however, is waiting for the output from the process and grabs what’s first to appear there. So it grabs the output from the print function and uses it as the data for the completion engine. While the REPL gets the output from the ,complete command:

This is a race, and the last request from comint wins the first output from the process. It just so happens that both messages used a format similar enough for completion to work. In most cases, completion will silently fail, and the REPL will just lose the results of the expression.

Comint is not suited for working with a protocol like ours, but we can still reuse a fair bit of comint features in our client. Instead, we’ll have two separate processes - one for the server that implements the protocol, and one for the REPL that acts as a fake process with all comint goodies.

Here’s what our architecture will look like:

By separating input from output, and effects from parsing, we avoid all problems with comint racing over process output and misparsing different commands. And the input doesn’t have to come from comint, we can send messages via the input sender from anywhere, so code interaction in the buffer is easy to implement too.

Notice how ID comes into play here - we read user input, format it as a message, and assign it an ID, which we send to the server. While doing so, we register a callback for the message in a hash table, that uses message IDs as keys, and callback as values. Once the server has processed the message, it responds with several messages of its own, all of which include a callback. We then parse these messages in the output filter, which looks for the callback in the hash table. If the callback was found, it is being called with the data from the message according to the protocol. And the callback can print the result back to the REPL, to a completion engine, or to any other target buffer.

This is also the reason why our protocol answers with special messages accept and done before and after the data-oriented ones, like values, print, read, and such. All protocol operations require a callback to reach the user, but once the message is processed fully its callback can be released. This is what the done OP is for in our protocol, but what about accept?

Right now in my design, accept is a noop, but I can see some potential later use cases. One of them is for implementing asynchronous timeouts. For example, when we register a callback we can store the maximum amount of time that is meaningful for such a callback to exist. Once the server answers with the accepted OP, we can check if the time since callback registration exceeds the lifespan of the callback. If not, we continue waiting for other messages, otherwise, we unassign it.

Another possible use for the accept OP is to cancel previous callbacks. This is not particularly useful in the context of a single-threaded Lua application, as it will always process messages sequentially, but nothing prevents us from implementing an asynchronous REPL. In an asynchronous context, it is quite possible that a message will be accepted before the previous one was fully processed. And in cases like when we query for completions, we are interested only in the latest results, so we can use accept to cancel the previous completion callback.

But enough talk, let’s write some code!

Server process

Now, I must say, that in the same way that we didn’t implement a full-blown protocol, we won’t implement a full-blown client. However, I will try to provide a complete enough example that you’ll be able to make your own client like this in the future if you wanted to.

We start simple - we need a way to start the server process that will act as a pipe:

(defvar proto-repl--upgrade-code "...whole protocol code...")
(defvar proto-repl-process " *proto-repl*")
(defvar proto-repl--message-callbacks nil)
(defvar proto-repl--message-id 0)

(defun proto-repl--start-server (command)
  "Start the Fennel REPL process.
COMMAND is used to start the Fennel REPL.  Sends the upgrade code
to the REPL and waits for completion via a callback."
  (let ((proc (make-process
               :name proto-repl-process
               :buffer proto-repl-process
               :command (split-string-shell-command command)
               :connection-type 'pipe
               :filter #'proto-repl--process-filter)))
    (buffer-disable-undo proto-repl-process)
    (with-current-buffer proto-repl-process
      (setq mode-line-process '(":%s")))
    (message "Waiting for Fennel REPL initialization...")
    (setq proto-repl--message-callbacks nil)
    (setq proto-repl--message-id 0)
    (proto-repl--assign-callback #'proto-repl--start-repl)
    (send-string proc (format "%s\n" proto-repl--upgrade-code))))

First of all, let’s address the elephant in the room - the protocol code. Our package needs to somehow obtain the protocol code and send it to the REPL process. I’ve chosen to store it as a string, even though it is a very long string. The reasoning behind this is that I want this file to be both self-contained and because package management in Emacs is a mess, and there’s no reliable way to obtain a file that is located in the same package unless they’re both in the same directory, which may not be the case¹. Not the most elegant solution, but it works.

Next up, we define two more variables, one for process buffer name, and another one for storing callbacks. For simplicity’s sake, I’ll use an associative list for callbacks, but in the real client, a hash table with fast equality function should be used.

And the main piece of code - the function that starts the process. Code should be mostly self-explanatory, but in case it’s not, the main idea is to start the process and assign a process filter to it. The process filter will be the next thing we’ll implement, so I’ll save the explanation for later. After that, we do a few setup steps and send the message to the process with the upgrade code. Sending messages is another bit part of the client, and we’ll look into them as well.

Process filtering

The process filter is a function that accepts two arguments, the process being filtered and the data that was read from it. By specifying a custom process filter we actually prevent any output from the process in the process buffer, so some kind of logging should be implemented. And here’s our first challenge.

Our protocol works on a one message per line basis, yet the output from the process is received in hunks instead of lines. It is quite possible to receive a message that contains the start of the next message without its end. In a case like this, a custom encoding would be a much better choice than using a line-based protocol, but we can workaround the problem by implementing buffering.

(defvar proto-repl--message-buf nil)

(defun proto-repl--buffered-split-string (string)
  "Split STRING on newlines.
If the string doesn't end with a newline character, the last (or
the only) line of the string is buffered and excluded from the
result."
  (let ((strings (string-lines string t)))
    (when proto-repl--message-buf
      (setcar strings (concat proto-repl--message-buf (car strings)))
      (setq proto-repl--message-buf nil))
    (if (string-suffix-p "\n" string)
        strings
      (setq proto-repl--message-buf (car (last strings)))
      (nbutlast strings))))

First, we split the string on newlines, removing empty lines, as they’re irrelevant to us. Next, if we have something in our buffer, we concatenate it with the first line, as it is a leftover from the previous hunk. Then we check if the string ends with a newline, if it is we return lines as is. If not, we set the buffer to the last incomplete line and return everything except for the last line.

This way, the protocol is much more resilient to process buffering, and the fact that Emacs doesn’t read line by line on most systems. Yes, our protocol is line-based and it’s not exactly machine-friendly, but it is much easier than using some kind of encoding, like bencode.

Once we have a list of lines, we can begin processing them. Here’s our process filter:

(defun proto-repl--process-filter (_ message)
  "Parse the MESSAGE and process it with the callback handler."
  (dolist (message (proto-repl--buffered-split-string message))
    (let ((message (substring message (string-match-p "(:id " message))))
      (when-let ((data (condition-case nil
                           (car (read-from-string message))
                         (error nil))))
        (when (plistp data)
          (proto-repl--handle-protocol-op data)))))
  (with-current-buffer proto-repl-process
    (goto-char (point-max))
    (insert message)))

In this function, we’re going to work on a line-by-line basis because the first thing we do is split the incoming message. However, the input massaging doesn’t stop there - for each message we want to strip everything that is not part of it. If you remember, before we had our IO wrapped, the output from io.write was right before one of the messages. This is partly why we do it. In fact, this should never happen, but we can’t control it before the protocol is initialized, especially because by default Fennel REPL is quite verbose. If we send a partially complete expression to the base Fennel REPL it will display a .. prompt, so while sending our protocol line by line we’ll get back a lot of dots preceding the initialization message:

...............................................(:id 0 :op "initialized")

So we have to deal with it. This function is also the right place to add logging, but I’m omitting it for the sake of simplicity and just emitting everything to the process buffer.

After we’ve split the input, and truncated it we parse it with the read-form-string function. This again is where some errors might lurk - if our protocol ever would return two messages in a single line, we’ll lose the last one. So you might want to check for that. If the message was read successfully, we also check if it is a plist with the plistp function, and pass it to the protocol handler.

The protocol handler is the meat of our client. It manages callbacks for each message and handles all supported protocol OPs.

(defun proto-repl--handle-protocol-op (message)
  "Handle protocol MESSAGE.
Message contains an id, operation to execute, and any additional
data related to the operation."
  (let ((id (plist-get message :id))
        (op (plist-get message :op)))
    (when-let ((callback (proto-repl--get-callback id)))
      (pcase op
        ("accept" nil)
        ("done"
         (proto-repl--unassign-callback id)
         (unless (proto-repl--callbacks-pending?)
           (proto-repl--display-prompt)))
        ("eval"
         (let ((values (plist-get message :values)))
           (funcall callback (format "%s\n" (string-join values "\t")))))
        ("print" (proto-repl--print (plist-get message :data)))
        ("read"
         (let ((inhibit-message t))
           (write-region
            (read-string "stdin: ") nil
            (plist-get message :data))))
        ("error"
         (proto-repl--display-error
          (plist-get message :type)
          (plist-get message :data)
          (plist-get message :traceback)))
        ("init"
         (proto-repl--unassign-callback 0)
         (funcall callback nil))))))

Even though our protocol is quite small, we do have a few OPs to support. As I’ve mentioned the accept OP is a nop, but the done OP is quite important. You can see, that we unassign the callback for the message, and then check if there are pending callbacks. This allows us to avoid drawing the prompt in the REPL because pending callbacks mean that the REPL is still busy.

Another special OP is the init one. It has a special callback with the ID 0, and this callback is strictly for the REPL initialization. Other callbacks should be pretty much self-explanatory.

Let’s write functions for dealing with callbacks:

(defun proto-repl--get-callback (id)
  "Get a callback for a message with this ID."
  (cdr (assoc id proto-repl--message-callbacks)))

(defun proto-repl--unassign-callback (id)
  "Remove callback assigned to a message with this ID."
  (setq proto-repl--message-callbacks
        (assoc-delete-all id proto-repl--message-callbacks)))

(defun proto-repl--assign-callback (callback)
  "Assign CALLBACK and return the id it was assigned to."
  (let ((id proto-repl--message-id))
    (add-to-list 'proto-repl--message-callbacks (cons id callback))
    (setq proto-repl--message-id (1+ id))
    id))

(defun proto-repl--callbacks-pending? ()
  "Check for callbacks that still waiting for the DONE message."
  proto-repl--message-callbacks)

With these, we can do all operations on callbacks that our client requires. The proto-repl--assign-callback is an interesting one. It increments the ID after it assigned the callback, and returns the ID of the callback itself. This is our main interface for sending messages to the REPL.

User interaction

With the server part mostly done, we can make the comint part that is an actual user interface to our protocol-based REPL. Let’s start with the mode for the REPL:

(require 'fennel-mode) ;; for font-lock

(defvar proto-repl-buffer "*Fennel Proto REPL*")
(defvar proto-repl-prompt ">> ")

(define-derived-mode proto-repl-mode comint-mode "Fennel Proto REPL"
  "Major mode for Fennel REPL.

\\{proto-repl-mode-map}"
  (setq comint-prompt-regexp (format "^%s" proto-repl-prompt))
  (setq comint-prompt-read-only t)
  (setq comint-input-sender 'proto-repl--input-sender)
  (setq mode-line-process '(":%s"))
  (setq-local comment-end "")
  (fennel-font-lock-setup)
  (set-syntax-table fennel-mode-syntax-table)
  (unless (comint-check-proc (current-buffer))
    (let ((proc (start-process proto-repl-buffer (current-buffer) nil)))
      (add-hook 'kill-buffer-hook
                (lambda ()
                  (when-let ((proc (get-buffer-process proto-repl-process)))
                    (delete-process proc)))
                nil t)
      (insert ";; Welcome to the Fennel Proto REPL\n")
      (set-marker (process-mark proc) (point))
      (proto-repl--display-prompt))))

There’s a lot to dig in, but the main part is where we start a new process with the start-process function. This process is what the comint will use for it’s internal implementation of input handling and such. Let’s look at the fennel-repl--input-sender:

(defun proto-repl--input-sender (_ input)
  "Sender for INPUT from the REPL buffer to REPL process."
  (let* ((id (proto-repl--assign-callback #'proto-repl--print))
         (mesg (format "{:id %s :%s %S}\n" id "eval" (substring-no-properties input))))
    (send-string proto-repl-process mesg)))

This function is responsible for sending the input to the server as a message, so it formats user input like one. Before sending the message it assigns the callback, and then the rest of our system should just work.

The proto-repl--display-prompt is a simple function that just prints the prompt to the REPL buffer:

(defun proto-repl--display-prompt ()
  "Display prompt."
  (let ((proc (get-buffer-process proto-repl-buffer)))
    (comint-output-filter proc proto-repl-prompt)))

And the proto-repl--print is a default callback for printing anything to the REPL buffer:

(defun proto-repl--print (message)
  (let ((proc (get-buffer-process proto-repl-buffer)))
    (comint-output-filter proc message)))

This function is almost identical to the prompt displaying one, but it is mostly for the simplicity’s sake. In fact the real one is a bit more complicated because we want to print stuff before the prompt if we’re sending code not from the REPL but from the buffer. I’ve left this out as it’s not that hard to implement.

So the last thing we need is to actually start the REPL in a buffer:

(defun proto-repl--start-repl (_)
  "Start the REPL."
  (message "Fennel Proto REPL initialized")
  (with-current-buffer (get-buffer-create proto-repl-buffer)
    (proto-repl-mode)
    (pop-to-buffer (current-buffer))))

This one is a callback we’ve used when starting a server. And this one is the funtion for the end user:

(defun proto-repl ()
  "Fennel Proto REPL."
  (interactive)
  (if-let ((repl-buffer (get-buffer proto-repl-buffer)))
      (if (process-live-p (get-buffer-process repl-buffer))
          (pop-to-buffer repl-buffer)
        (proto-repl--start-server "fennel --repl"))
    (proto-repl--start-server "fennel --repl"))
  (get-buffer proto-repl-buffer))

And the client is done! Let’s try it:

On the left is the REPL buffer, which is responsible for input handling, and sending messages. On the right is the server buffer, which shows the process log. As you can see, we’ve got the usual messages for all our expressions.

New Fennel REPL integration for Emacs

Sure, the implementation of the client and the protocol has a lot of room for improvement, but it is enough for the purpose of this post. As a matter of fact, I’m almost done working on a proper implementation, and most of the code here is greatly simplified version of what I’ve already made for the fennel-mode package. For now it lives in a separate module, but I have high hopes on completely replacing inbuilt comint-based client with this protocol based one, once I test it more. It will probably still live in a separate module for a while even after I will consider it mostly complete, so users would be able to try it in their environments, but my hopes are high.

There is one problem though - this protocol implementation may not work in some contexts. For instance, if you’re already providing a custom REPL in your application, that is based on the fennel.repl and implements its own readChunk, that, for example, doesn’t read from stdin. One such example is the LÖVE game engine, there’s an implementation of the REPL that polls for input on a separate thread. If we send our protocol-based REPL to such a custom REPL, we’ll negate all the efforts made to make the REPL work in a non-blocking way. For Clojure, this works because the REPL already lives in its own thread, but there are no threads in Lua apart from what coroutines provide.

So I’m still thinking about how to handle the situation when the upgrade is impossible. This is why the official protocol is much better than such a custom one, but alas.

You can already try this REPL if you check out the proto-repl branch in the fennel-mode project. If you’re using straight.el, or Quelpa, or Emacs version recent enough to have the package-vc-install function, you can try installing fennel-mode and supplying the proto-repl branch to the recipe.

If you want to experiment with the code that is provided in this article for example for the purpose of building a similar protocol for another language, this very article is written in the literate style, and you can grab it here. Run the org-babel-tangle on it, and you should get the protocol.fnl file with all the necessary functions, and proto-repl.el file with the Emacs Lisp code. You’ll need to put the protocol code to the proto-repl--upgrade-code variable, as I’ve excluded it from the article’s text since it was making that particular code block too long.

Thank you for reading! I hope this was interesting and useful. As always, if you have any thoughts you want to discuss on the matter, feel free to contact me via one of the ways mentioned on the about page.

Package managers like straight.el or quelpa use recipes to specify what files to use during the build process. It is possible that users of the package may not notice that the recipe needs to be changed. This can be handled by the package archive like MELPA, but I’d rather not bother with it right now. ↩︎

Comment via email

Small languages with little tooling

Wed, 02 Nov 2022 20:31:00 +0300

When it comes to software I prefer things that are simple and small, even though I’m using Emacs. This is mainly the reason why my favorite languages are Clojure and Fennel. However, it doesn’t end on programming languages themselves, I like small tools in general. One such example would be Kakoune - a text editor, inspired by Vim, which focuses on multiple selections.

Kakoune doesn’t have a scripting language, and when I started using it, the collection of plugins was relatively small, and it didn’t have a plugin manager at all. With a limited amount of options, it’s easier to start using particular software, because you don’t have the problem of choice, especially when you know nothing about the things that you’re choosing from. You pick it up, and you’re good to go. This was a major problem for me to get into Emacs, as, as opposed to Kakoune, not only does it has a pretty huge core full of features, but it also has a ton of packages written by users over the years, and several package managers to install those.

Eventually, I figured things out, and now I’m a happy Emacs user, but its enormous pack of inbuilt features still bothers me, and Kakoune made me understand why I like small ecosystems. It’s because in a small ecosystem it’s easier to see what is needed by people, and you can participate in making things better not just for yourself but for everyone.

Don’t quote me out on this, but I think plug.kak was the first fully featured plugin and configuration manager for Kakoune, which I made mostly for myself, but to my surprise, it got to be pretty popular. There even were plugins (written by other people) that suggested using plug.kak as a preferred way of installing them. Before plug.kak people had to use the default mechanism of loading, which used an autoload directory to specify what Kakoune should load at startup, and creating a such directory in your config disabled system-wide autoload directory, making Kakoune featureless. This was one of my motivations behind plug.kak, because I saw issues in Kakoune’s repository titled mentioned that the creation of the auotload dir broke their config in an unexpected way of not loading any of the inbuilt features. My approach was to create a, hopefully, small system that knows where to look for external stuff and how to load and configure it. And people seemed to like it.

I also wrote a bunch of other plugins for Kakoune, all of which are listed here, if you’re interested. This helped me shape Kakoune into a thing that I liked to use, having features that I need, and also igniting my interest in programming in general, because the tools I built were used by other Kakoune users, and it was inspiring. Later on alternatives to plug.kak appeared, and I’m glad because they were simpler and more efficient. I no longer maintain plug.kak, as I no longer use Kakoune, but it still has a place in my heart.

Before Kakoune I made plugins for Vim, but given that the plugin ecosystem was already saturated, my plugins were made mostly for myself and I wasn’t hoping that anybody else would prefer them to alternatives. It was fun, still.

So my point is - I like to participate in small ecosystems, mostly because the points where an improvement is welcome are more visible. This leads us back to programming languages, and Fennel in particular.

The Fennel Programming Language

If you read my blog, you may have seen posts about Fennel before, and likely know what it is. If not, here’s a quick introduction:

Fennel is a small surface language implemented on top of the Lua language. It has a Clojure-inspired syntax, and Lisp macro system, allowing users to extend the language with new constructs pretty easily. Fennel compiles down to Lua, and doesn’t have its own semantics, besides mentioned macro system and being a bit more strict. And because Fennel is actually just a compiler, it includes almost no runtime features. Its only runtime components are REPL (Read Eval Print Loop), which is a common thing for Lisps, and the pretty-printer - a small library that prints data structures in a way that can be read back by Fennel compiler, or just examined by us, humans.

I’ve been using fennel for almost 2 years now, and I’ve been reflecting on what has changed for me personally. What’s improved, what things I made for myself, and so forth. And looking at my project workflow in Fennel, and seeing a lot of stuff happening, I get the impression of working in a mature language, with a mature ecosystem. So I decided to write about this topic, highlighting interesting changes, that happened in the past two years of my Fennel experience.

Pretty Printer

When I started using Fennel, the pretty-printer module was in a bit sad state. It worked, but its output was not formatted as one would expect from a language with Clojure-like syntax. Instead, it was formatted more like Lua code, and because of that, it was difficult to inspect deeply nested structures.

I have a post describing existing problems and my approach in detail, but in short, I had to rewrite the fennelview.fnl module from the ground up. And as far as I can tell, the community liked the new pretty printer, so the effort was well worth it. It needed some tweaks here and there afterward made by other people, but I think this is still my biggest contribution to Fennel, especially since we’re seeing results of the pretty printer every time we send something to the REPL. Which made Fennel much more pleasant to work with (for me, at least).

The `fennel-cljlib` library

With the pretty printer in place, I started working on various libraries that were just interesting to make. The first one was fennel-cljlib, which was aimed at Clojure programmers that wanted to try out Fennel but didn’t want to give up some of the features they love in Clojure. The library was crappy and had a lot of problems, which I fixed over the years, but right now I would say that it is solid. Of course, there’s still a lot of room for improvement, but I’ve already utilized this library in some projects, and even seen its use in projects from other people!

One thing that this library made me good at is writing macros. Fennel has a similar syntax compared to Clojure, but it’s not the same, and some features are missing. For instance, in Clojure, there is such thing as multi-arity functions, which are extremely handy. Here’s how you might define such a function in Clojure:

(defn add
  "Sum arbitrary amount of numbers."
  ([] 0)
  ([a] a)
  ([a b] (+ a b))
  ([a b c] (+ a b c))
  ([a b c d] (+ a b c d))
  ([a b c d & rest] (apply add (+ a b c d) rest)))

So if you call this function with zero arguments given, the ([] 0) arity is selected, and 0 is the return value. If you call it with three arguments, the ([a b c] (+ a b c)) branch is selected, and the return is the sum of three arguments. However, if you call it with five-or-more arguments, the last arity ([a b c d & rest] (apply add (+ a b c d) rest)) is selected, which will recursively call itself with less and fewer arguments, until it reaches another arity. So the call stack is (add 1 2 3 4 5 6 7 8 9 10) to (add 10 5 6 7 8 9 10) followed by (add 28 8 9 10) and the result value is 55. That’s three tail-recursive calls, to sum 10 numbers, and we do as much work to avoid unnecessary recursive calls.

Of course, math functions, such as this can be written in a more performant way without any recursion, this just illustrates the idea, that you can write functions in a pretty much declarative style: given this amount of arguments do this thing. And with the cljlib library we can define such functions But I’ve lied to you, that’s actually not Clojure up there, but Fennel code, defining one of the functions in cljlib. And if you scroll through the linked code, you’ll see that pretty much every function utilizes this feature of the language. Even more so, if you take a look at Clojure sources, you’ll notice that a lot of code was ported almost directly, which some changes needed for it to work on the Lua runtime.

But I’m going a bit off-topic here, this post is about the projects related to the language ecosystem, not just random projects of mine. However, this library motivated me to make a contribution to the ecosystem, because I felt that there’s a point for improvement in the current workflow. In the end, all I really want is a comfortable environment so I could enjoy programming a bit more, and maybe make it more enjoyable for other people as a side effect. And I’m glad to see other people using the next project I’m going to talk about.

Fenneldoc

Fenneldoc is the tooling I wanted for Fennel so badly that I had to make one myself.

This project is a rather simple documentation generation tool, which extracts documentation strings from functions’ metadata, and creates a set of Markdown formatted files. I wanted this tool because, already mentioned, cljlib project had 60+ functions and 10+ macros, and I had to maintain documentation for all of that by hand. And while I was working on this library and adding new functions, or changing existing ones, I had to keep the documentation in sync. At some point, I was tired of noticing that I forgot to update some function’s documentation, or forgot to mention one entirely. So I wrote fenneldoc and replaced, all of the manually written documentation with auto-generated one.

I know, a lot of people seem to agree on the topic that auto-generated documentation is bad, however, I don’t think it’s entirely true. There’s a popular post by Jacob Kaplan-Moss, which describes what to write in the documentation. It has a paragraph regarding auto-generated documentation:

Don’t.

Auto-generated documentation is almost worthless. At best it’s a slightly improved version of simply browsing through the source, but most of the time it’s easier just to read the source than to navigate the bullshit that these autodoc tools produce. About the only thing auto-generated documentation is good for is filling printed pages when contracts dictate delivery of a certain number of pages of documentation. I feel a particularly deep form of rage every time I click on a “documentation” link and see auto-generated documentation.

There’s no substitute for documentation written, organized, and edited by hand.

I’ll even go further and say that auto-generated documentation is worse than useless: it lets maintainers fool themselves into thinking they have documentation, thus putting off actually writing good reference by hand. If you don’t have documentation just admit to it. Maybe a volunteer will offer to write some! But don’t lie and give me that auto-documentation crap.

Jacob Kaplan-Moss: Writing great documentation: What to write

I’m not sure exactly what kind of documentation here was in mind, and I can agree if we’re talking about documentation that was generated from the code alone. I should also note that I have written a lot of documentation by hand before, and I would say that not all documentation is better when written like that. And documentation can be hard to write in general, it certainly was for me.

When working on a library, handwriting the documentation becomes very hard. Not only do you have to sync every documentation entry with every update in your code base, but you’ll also have to remember what’s in the documentation and what’s missing. And if you’re covering only the public API of your library it’s one thing, but when you also cover the private APIs for other developers to understand it’s a different kind of beast. Doing all of this by hand can leave you no time to work on features as a result.

The point is - it entirely depends on how documentation is generated, how it was written, and also for who you write it. Your handwritten documentation may as well be complete garbage, useless for the end user if you have written it in a bad style or with a great lack of detail. The same goes for auto-generated documentation - it can be good, if the tooling is good, and if programmers took their time in writing meaningful things in the documentation strings/comments.

I’m getting a bit ranty here.

So I think I found a compromise - I hand-write all documentation on a function level and generate the markdown files which enrich the documentation text with links between items automatically. Paired with the fine-tuning of documentation order fenneldoc provides, and the ability to add a general notice on top, it is possible to create documentation that reads as a normal document, yet is still usable when you found yourself in the sources. And it is never out of date, unless you forgot to update it, which also happens without auto-generating.

But there’s more. What I also wanted from Fenneldoc was documentation testing.

When I write documentation, I also include a lot of examples of how to use certain functions, or how they can be combined. But after I change these functions, some examples might stop working. And what’s good about having examples that don’t work when you try them?

So Fenneldoc runs all examples and ensures that they at least compile. You can include tests right into them, so they break when you’ve introduced an incompatible change and forgot to update the documentation. It also checks if you’ve mentioned all of the function’s arguments in the documentation, and warns if it couldn’t find a cross-link to an item.

For example, here’s another function from cljlib:

(defn into
  "Returns a new coll consisting of `to` with all of the items of `from`
conjoined. A transducer `xform` may be supplied.

# Examples

Insert items of one collection into another collection:

```fennel
(assert-eq [1 2 3 :a :b :c] (into [1 2 3] \"abcd\"))
(assert-eq {:a 2 :b 3} (into {:a 1} {:a 2 :b 3}))
```

Transform a hash-map into a sequence of key-value pairs:

``` fennel
(assert-eq [[:a 1]] (into (vector) {:a 1}))
```

You can also construct a hash-map from a sequence of key-value pairs:

``` fennel
(assert-eq {:a 1 :b 2 :c 3}
           (into (hash-map) [[:a 1] [:b 2] [:c 3]]))
```"
  ([] (vector))
  ([to] to)
  ([to from]
   (match (getmetatable to)
     {:cljlib/editable true}
     (persistent! (reduce conj! (transient to) from))
     _ (reduce conj to from)))
  ([to xform from]
   (match (getmetatable to)
     {:cljlib/editable true}
     (persistent! (transduce xform conj! (transient to) from))
     _ (transduce xform conj to from))))

You can see that it has a basic description of what it does, and a few examples of what happens when you use it. Arguments to the function exist in the documentation, and they’re using a Markdown notation for monospaced text by being enclosed via the backticks, for instance, the `xform` argument. Here the documentation is just a string of Markdown formatted text in general.

Below is the example section, which features code blocks denoted with the triple backticks, following a language name: ``` fennel ... ```. In each example, there is a test, and you can see that all invocations are wrapped in assert-eq calls, which is an assertion macro from my fennel-test library. In this particular example, I made a mistake and forgot to add the letter d into the result of the very first call to into. When I try to export this documentation to Markdown, I get an error:

In file: 'init.fnl'
Error in docstring for: 'into'
In test:
``` fennel
(assert-eq [1 2 3 :a :b :c] (into [1 2 3] "abcd"))
(assert-eq {:a 2 :b 3} (into {:a 1} {:a 2 :b 3}))
```
Error:
assertion failed for expression:
(eq [1 2 3 "a" "b" "c"] (into [1 2 3] "abcd"))
 Left: [1 2 3 "a" "b" "c"]
Right: [1 2 3 "a" "b" "c" "d"]

The error message comes from the assert-eq macro, which tests whether the left expression is equal to the right one. Fenneldoc just makes it possible to see the test itself before it. A simple test, but can catch some mistakes in the documentation!

I find this tool quite usable and a big part of my project workflow - every project I do has documentation generated with Fenneldoc, and I even read it from time to time, despite the fact that I was the one writing it.

Fennel test

Speaking of testing, I also created a fennel-test library because I didn’t like libraries aimed at testing Lua. My library provides its own test runner, and tests are structured similarly to how they are in Clojure. And if your project uses this library, it is possible to include it in the documentation testing pipeline as well, as I’ve shown before.

Now, including huge tests in the documentation is really bad - in my opinion it should hold only small examples that also act as tests. Also, I think that everything you write in the docstring should be displayed as is, so Fenneldoc doesn’t try to hide any stuff from the code blocks - the reader should see the whole picture. The Rust programming language took a different approach, even has hidden documentation tests, that are there only when you test the documentation, not when you export it. I don’t think this is the correct way, but the paragraph mentions that it has something to do with the internal limitations of their testing framework.

So you should write tests that are a lot more complicated than your documentation tests, but since it is reserved for the test suite it’s not a problem. Here’s the test for the into function I’ve been talking about before:

(deftest test-into
  (testing "into"
    (assert-eq (core.into [] nil) [])
    (assert-eq (core.into nil nil) nil)
    (assert-eq (core.into nil [1 2 3]) (core.list 3 2 1))
    (assert-eq (core.into [] []) [])
    (assert-eq (core.into [1 2 3] []) [1 2 3])
    (assert-eq (core.into [1 2 3] [4 5 6]) [1 2 3 4 5 6])
    (assert-eq (core.into {} {}) {})
    (assert-eq (core.into {:a 1} {}) {:a 1})
    (assert-eq (core.into {:a 1} {:b 2}) {:a 1 :b 2})
    (assert-eq (core.into [] {}) [])
    (assert-eq (core.into {} []) [])
    (assert-eq (. (getmetatable (core.into (core.vector) {})) :cljlib/type) :vector)
    (assert-eq (. (getmetatable (core.into (core.hash-map) [])) :cljlib/type) :hash-map)
    ;; ... more various cases ...
    ))

Here, the deftest is a macro, that defines the test function which is then executed by the test runner. The runner can shuffle tests, capture errors, and form the log afterward. I like how it works, and it’s fine for my use cases. There also are other facilities, like fixtures, which you can use to wrap each test, or all of the tests with additional code, but they only work when using the test runner.

OK, hopefully, this is all great and can help us write more understandable and maintainable programs in Fennel. What about support from text editors then?

Editor tooling

And text-editor support is another thing that often requires improvements when we’re dealing with young languages. I’ve created Fennel syntax highlighting support for Kakoune, and I also currently maintain the fennel-mode package for Emacs. I know that Visual Studio Code has some support for Fennel, and NeoVim has great support with interactive features, and even the ability to configure the editor in Fennel itself. Similarly to NeoVim, there’s also a text editor, called Lite, which is written in Lua, and there are projects that make it extendable via Fennel, which is great! Still, Emacs as of this moment has the best Fennel support (IMO), as it has good support for Lisps in general. However, there are always places to improve tooling, and I improved Fennel’s support in Emacs by providing automatic completions, documentation popups, and a scratch buffer for a different approach for interactive sessions. I have some more plans on improving the REPL support and dynamic highlighting of loaded macro forms.

In addition to that, I’ve made a small package to add support for Fennel to Org Mode, called ob-fennel. I have an extensive post that goes into detail about how it can be used, so if you’re interested you can read it.

There’s an interesting project on the horizon: fennel-ls - a language server for Fennel. A very welcome addition to the language ecosystem, and as far as I can tell after speaking with the author - a passionate project. And, as far as I can see, every mature language eventually gets a language server, so this is a good sign!

What’s next

There’s one point I want to replace in my current workflow with my Fennel project - coverage statistics. Currently, I use the luacov project, which analyzes Lua code specifically. This is not a huge problem for me - I can read Lua, and I learned to read Lua that Fennel generates, but it is still a point of inconvenience.

Ideally, I would like to have a tool similar to cloverage, which is a Clojure library for generating code coverage reports. I like it, because not only it works with Clojure code, and not some Java decompiled byte code, but it also shows coverage in terms of forms, not just lines. For Fennel, it is a huge deal, because like in many lisps, forms may sit on a single line. Though it is not a problem because I opt out of the --correlate flag when I compile the project for coverage testing, so all one-liners are transformed into multiple lines, and I get pretty accurate coverage results. Still inconvenient.

But all of the above is only a small fraction of what has happened with Fennel overall - it’s just things that I’ve experienced in my project workflow. There’s a lot of cool stuff happening even right now, and I welcome everybody to the upcoming FennelConf 2022.

From my point of view, Fennel grew a lot during the last two years. We had the first major release and extended the language with new macros, and there are a lot of projects in the works regarding making the compiler more general via a plugin system. I’m very excited about the future of Fennel and want to thank everybody in the community, which I’m glad I’m a part of.

And I believe this situation is not unique to Fennel. I know people who participate in other programming language-related communities, and I hear great things about them too. Being in a small community is also beneficial in that you get to know a lot of people, and when someone new comes with some crazy ideas it’s always great to see what they can create with the project you are also passionate about.

So keep hacking! I’m looking forward to seeing more projects that help make developing ecosystem better, and I do my best for the ones I participate in!

Comment via email

Reproducible Research with Org Mode, Fennel, and LÖVE

Mon, 26 Sep 2022 00:57:00 +0300

Today I would like to talk about three separate tools, and how we can combine them with the power of Emacs.

Our tools for today are:

Fennel - a Lisp-like surface syntax for Lua that makes programming in Lua fun and more robust.
Org Mode - a markup language and an application inside Emacs, which allows its users to create documents that include interactive pieces of code.
LÖVE - a game engine that allows programming games in Lua.

This post is itself a program, describing how you could create similar Org documents and explore literate programming/reproducible research with Fennel, or other languages for that matter. It’s possible thanks to Org Mode and my new package ob-fennel, which implements necessary functions for sending Fennel code blocks to the REPL process. For a better experience, you should read this in Emacs, and you can get the original .org file from here. If you want to follow the instructions, make sure to install Fennel and LÖVE for this to work.

Bootstrapping LÖVE

First, we need to bootstrap LÖVE so it could understand Fennel code:

local fennel = require "fennel"
table.insert(package.loaders or package.searchers, fennel.searcher)

We’ve required fennel the library and registered the fennel.searcher function as one of the functions responsible to loading files. With that, theoretically, we no longer need to write any more Lua code, and instead, we can write in Fennel, and then use plain Lua’s require to load the files. Let’s create a REPL (Read Eval Print Loop) that will allow us to do interactive programming in LÖVE via its thread and event systems.

First, let’s require fennel once again, and also require love.event. Next, we’ll do a little trick that I’ll explain later, but for now, we’ll only need to capture the module-scope value of ... into variables event and channel:

(local fennel (require :fennel))
(require :love.event)

This will make sense in a moment. Now, we need a REPL itself. REPL consists of the prompt that asks for more input, and the loop that calls the prompt which reads some input and sends it to a channel. Let’s define the prompt function first:

(fn prompt [next-line?]
  (io.write (if next-line? ".. " ">> "))
  (io.flush)
  (.. (tostring (io.read)) "\n"))

Now, we can create the infinite loop, that will read from stdin and push the read value into LÖVE’s event system. That’s where we’ll use the captured event variable, however, we’ll need to run this loop only in a LÖVE thread, so we’ll check if the channel was bound:

(match ...
  (event channel)
  (while true
    (match (channel:demand)
      [:data output] (print (table.concat output "\t"))
      [:read cont?] (love.event.push event (prompt cont?))
      [:error kind & data]
      (print (string.format
              "%s error: %s" kind
              (accumulate [msg "" _ message (ipairs data)]
                (.. msg (tostring message))))))))

Once data appears in the channel, this function exterminates what pattern it received. It is always a table, where the first element means the kind of data, and the rest of the table is the data itself. That’s the protocol we’re going to write in a moment.

The loop check if the event-type is either :data or :error and acts accordingly. In addition to that, there’s a third event-type :next-line which represents the situation when the REPL receives incomplete input and waits for more lines. Let’s implement this protocol:

(let [thread (-> "repl.fnl"
                 love.filesystem.read
                 (fennel.compileString nil)
                 (love.filesystem.newFileData :repl)
                 love.thread.newThread)
      io-channel (love.thread.newChannel)
      coro (coroutine.create #(fennel.repl $...))]
  (->> {:readChunk (fn [{: stack-size &as vaiv}]
                     (io-channel:push [:read (> stack-size 0)])
                     (coroutine.yield))
        :onValues (fn [data]
                    (io-channel:push [:data data]))
        :onError (fn [kind ...]
                   (io-channel:push [:error kind ...]))}
       (coroutine.resume coro))
  (thread:start :eval io-channel)
  (set love.handlers.eval #(coroutine.resume coro $)))

This is an asynchronous REPL, made with Lua’s coroutines. You may wonder, how it will start the loop function if it was under the when channel guard, but that’s exactly what this piece of code does.

First, it loads itself via love.filesystem.read and compiles Fennel code to Lua via fennel.compileString function. Next, this compiled representation is used to create a FileData object, that is used to start a LÖVE thread with the love.thread.newThread function. So this module will be executed twice - first when main.lua loads the file, and second, when LÖVE spawns the thread, but were not there, yet.

After we’ve created the thread we create an io-channel and the REPL coroutine. Next, we start the coroutine by passing it a table of functions, each of which will push data to the io-channel. Importantly, the readChunk function pauses the coroutine after every read, allowing it to run asynchronously within the main thread.

Finally, we start the thread by passing :eval which will be bound to the event variable, and io-channel which will be bound to channel. This thread spins the loop function, which blocks the thread waiting for the input, while the main LÖVE thread continues to run.

With all of that, we can finally require the repl.fnl file in the main.lua like this:

require "repl"

Also, let’s disable vsync while we’re at it. For some reason, it makes the REPL slow on my machine. We won’t be needing it anyway:

function love.conf(t)
  t.window.vsync = 0
end

Now we can start hacking in real time!

Drawing plots

Since we’re done with the literate programming example, let’s look at how the reproducible research can be organized. I’m sure you should be familiar with the concept of a notebook, like R Markdonw or Jupiter. We’re going to achieve something like that.

If you’re reading this file in Emacs, which you should, press the C-c C-v t shortcut, which will run the org-babel-tangle function. After executing it, Emacs will create three files main.lua, conf.lua, and repl.fnl in the same directory as this file. If you’re not reading this file in Emacs, then, well, get Emacs, or just believe me - it works.

Let’s plot a function via LÖVE2D and Fennel. I’m not sure if there’s a library for drawing plots with LÖVE, but it’s ain’t fun to use something like that anyways, so let’s build our own! First, we’ll need a Canvas object to draw to:

(local canvas (love.graphics.newCanvas 640 480))

Put the cursor in this source code block and press C-c C-c shortcut (or call the org-babel-execute-src-block function with M-x) The empty LÖVE window will appear, and Emacs will display a message in the prompt area, with the contents something like “Please re-evaluate once Fennel is initialized”. We won’t be needing this window, so you can minimize it - we’ll be drawing off-screen onto a canvas and then saving it as an image. Org-mode provides us with all we’ll need to preview results without leaving Emacs.

Once Fennel is loaded, and all threads were spawned, hit C-c C-c again, and you should see "nil" in the echo area. This means that the code was executed successfully and the canvas variable was created.

Now, let’s write a function that will draw a grid to the active canvas. First, we’ll need a function that will calculate the grid step based on the zoom level. Our zoom level is represented in percentages, which we’ll later convert to values by dividing by 100. However, for our grid, we will be doing an infinite zoom with finite details.

This function computes the step between each grid line based on current zoom level.

(fn calc-step [zoom]
  (let [gscale (^ 10 (bit.bor (math.log zoom 10) 0))]
    (* zoom 100 (/ gscale))))

It’s a bit convoluted, but the key here is the bitwise or operator - that’s what makes our grid cycle between zoom levels properly. We now can define the function that computes and draws the grid itself:

(fn draw-grid [zoom]
  (let [(w h) (: (love.graphics.getCanvas) :getDimensions)
        step (calc-step zoom)]
    (love.graphics.setColor [1 1 1 1])
    (love.graphics.rectangle :fill 0 0 w h)
    (love.graphics.setLineWidth 1)
    (each [_ [step color]
           (ipairs [[(/ step 10) [0.7 0.7 0.7 (/ (% step 1000) 1000)]]
                    [(* step 1) [0.9 0.9 0.9 (- 1 (/ (% step 1000) 1000))]]])]
      (let [grid []]
        (fcollect [x (+ (/ w 2) step) w step     :into grid] [x 0 x h])
        (fcollect [x (- (/ w 2) step) 0 (- step) :into grid] [x 0 x h])
        (fcollect [y (+ (/ h 2) step) h step     :into grid] [0 y w y])
        (fcollect [y (- (/ h 2) step) 0 (- step) :into grid] [0 y w y])
        (love.graphics.setColor color)
        (each [_ line (ipairs grid)]
          (love.graphics.line line))))))

This function actually draws two grids - one for unit size, and one for 1/10 of the unit size, and applies dynamic transparency based of the zoom level. This way, we can create an illusion of the infinite zoom level, that It’s a bit verbose, but all it really does is create a table of line segments from the center of the canvas in all four directions with the calculated step interval.

This only draws the grid, though, we also need to draw the X and Y axis, with some markings indicating the zoom level:

(fn draw-axis [zoom]
  (let [(w h) (: (love.graphics.getCanvas) :getDimensions)
        step (calc-step zoom)]
    (love.graphics.setLineWidth 1)
    (love.graphics.setColor [0.3 0.3 0.3 1])
    (love.graphics.line [(/ w 2) 0 (/ w 2) h])
    (love.graphics.line [0 (/ h 2) w (/ h 2)])
    (let [h (/ h 2) w (/ w 2)]
      (for [x step w step]
        (love.graphics.line [(+ x w) h (+ x w) (+ h 2)])
        (love.graphics.print (string.format "%.1f" (/ x zoom)) (+ x w) (+ h 5)))
      (for [x (- step) (- w) (- step)]
        (love.graphics.line [(+ x w) h (+ x w) (+ h 2)])
        (love.graphics.print (string.format "%.1f" (/ x zoom)) (+ x w) (+ h 5)))
      (for [y step h step]
        (love.graphics.line [w (+ y h) (+ w 2) (+ y h)])
        (love.graphics.print (string.format "%.1f" (- (/ y zoom))) (+ w 5) (+ y h)))
      (for [y (- step) (- h) (- step)]
        (love.graphics.line [w (+ y h) (+ w 2) (+ y h)])
        (love.graphics.print (string.format "%.1f" (- (/ y zoom))) (+ w 5) (+ y h)))
      (love.graphics.print 0 (+ w 5) (+ 5 h)))))

This function works similarly to the draw-grid one, except it only draws two lines intersecting at the canvas’ center, and the numbers, indicating units of measure. Each number also has a small mark at the axis for better readability.

Finally, let’s write a function plot that will accept a function fun which it will plot on the canvas, and some additional arguments, like from, to, step. Also, did I mention that you should press C-c C-c on each code block in this section? It will send the code to the running LÖVE process, and we will be able to see the results dynamically. But before we do that, let’s write a function that transforms so-called world coordinates to screen coordinates:

(fn world->screen-coordinates [[x y] zoom]
  (let [(w h) (: (love.graphics.getCanvas) :getDimensions)]
    [(+ (* x zoom) (/ w 2)) (+ (- (* y zoom)) (/ h 2))]))

Oh, and we also need a function for generating a unique file name for the output image:

(fn unique-fname [name suffix]
  (let [(base ext) (string.match name "(.*)%.(.-)$")]
    (if (love.filesystem.getInfo
       (.. base (or suffix "") "." ext))
      (unique-fname name (+ (or suffix 0) 1))
      (.. base (or suffix "") "." ext))))

Alright, here’s the plot function:

(fn plot [{:fun f : from : to : step : zoom}]
  (let [image (unique-fname "result.png")]
    (canvas:renderTo
     #(let [zoom (/ (or zoom 100) 100)
            (w h) (: (love.graphics.getCanvas) :getDimensions)]
        (draw-grid zoom)
        (draw-axis zoom)
        (love.graphics.setColor [1 0 0 1])
        (love.graphics.setLineWidth 1)
        (let [points (fcollect [x from to (or step 0.1)]
                       (world->screen-coordinates [x (f x)] zoom))]
          (each [i [x1 y1] (ipairs points)]
            (match (. points (+ i 1))
              [x2 y2] (love.graphics.line x1 y1 x2 y2))))))
    (: (canvas:newImageData) :encode :png image)
    (.. (love.filesystem.getSaveDirectory) "/" image)))

This function uses the renderTo method of the canvas object. It accepts an anonymous function, which draws the gird, the axis, sets the color of the plot to red, and finally draws the plot with love.graphics.points. We can execute it like that, by pressing C-c C-c on the next code block:

(plot {:fun #(* $ $) :from -50 :to 50 :step 0.01 :zoom 5000})

This produces an image, which you should see if you call the org-toggle-inline-images function. As you can see, it plots a simple function, but we can plot any other, like :

(plot {:fun #(math.tan (math.sin $)) :from -10 :to 10 :zoom 5000})

Or a more complex example would be generating a square wave via this formula :

(plot {:fun #(accumulate [res (math.sin $)
                          _ k (ipairs (fcollect [k 3 29 2] k))]
               (+ res (/ (math.sin (* $ k)) k)))
       :from -500 :to 500 :step 0.01 :zoom 8000})

You get the idea.

Additionally, we can create a function that plots Org tables, like this one:

x	y
0	0
2	2
5	7
6	3
8	12
15	20
17	-2
19	-7
25	0

The function is mostly similar to plot except instead of calling a function it just goes through each table row, and builds line segments to draw:

(fn plot-table [{: data : zoom}]
  (let [image (unique-fname "result.png")]
    (canvas:renderTo
     #(let [zoom (/ (or zoom 1000) 100)
            (w h) (: (love.graphics.getCanvas) :getDimensions)]
        (draw-grid zoom)
        (draw-axis zoom)
        (love.graphics.setColor [1 0 0 1])
        (love.graphics.setLineWidth 1)
        (each [i row (ipairs data)]
          (let [[x1 y1] (world->screen-coordinates row zoom)]
            (match (. data (+ i 1))
              next-row
              (let [[x2 y2] (world->screen-coordinates next-row zoom)]
                (love.graphics.line x1 y1 x2 y2)))))))
    (: (canvas:newImageData) :encode :png (string.format image))
    (.. (love.filesystem.getSaveDirectory) "/" image)))

We can plot it like this, passing the table via the :var data=lines header property:

(plot-table {: data :zoom 1000})

You can build more complex functions, that, for example, draw several graphs on the same canvas, or draw entirely different types of graphs altogether. Though this requires defining functions for all these kinds of visualizations, and I guess this is why people tend to use other languages for the task, like R.

Inline evaluation

Well, graphs are cool and all, but what about supplying values into the text directly? We can do that too, and although, we could do this in our LÖVE REPL, that’s a great opportunity to showcase another ability of Org Babel - run multiple instances of Fennel, using completely different environments!

For example, suppose we have some kind of a formula, that, for example, calculates the Ackermann function:

(fn A [m n]
  (if (= n 0) 0
      (= m 0) (* 2 n)
      (= n 1) 2
      (A (- m 1)
         (A m (- n 1)))))

We can define the function using the (fn square [n] (A 1 n)), and use it like (square 4) producing 16 as a result. Or, we can define a function that computes like this: (fn two⇈n [n] (A 2 n)). Calling it as (two⇈n 4) gives us 65536.

These code blocks are running in a separate session, using a regular Lua interpreter, instead of our LÖVE REPL. This is done by specifying the :session header argument, and giving it a name, which instructs ob-fennel to spawn a new process. Upon exporting, the results of the inline evaluation would update automatically which eliminates the problem of stale data in the paper. The results always reflect the actual code they were computed with.

The power of Org

The examples above are just the tip of the iceberg of what you can do with the Org package. Back when I wasn’t an Emacs user, I often found threads on the topic “why should I use Emacs?” and each of these threads contained at least one answer which just said “Org Mode”. I never understood why - what’s so good about Org that I can’t get from, say, Markdown?

Since then, I’ve switched to Emacs and started this blog, using Org Mode, Hugo, and the ox-hugo package. Org mode has a built-in exporting engine, which can export .org files to PDF, ODT, LaTeX, Markdown, or HTML files. The ox-hugo package adds another backend for this engine, which exports to Hugo compatible markdown, which is then used by Hugo to build this website.

And when I was starting this blog, I wasn’t sure that I made the right investment - I’ve just switched to Emacs a couple of months before, and I’m using this Org thing for writing. I was uneasy because if I would stop enjoying writing in this setup I might lose all motivation for writing altogether. But thankfully, I’ve liked the Org+Hugo combination, and it became the reason why I haven’t left Emacs since - it’s just too good as a writing experience.

Literate programming was the way I learned Org. I’ve used it to configure Emacs because Org Mode provides a way to load .org files as Emacs Lisp files, and you can replace your init.el with init.org if you want to. But to be honest, it’s a fun experiment, but I don’t find literate programming to be that practical. It’s even harder to consider, given that there aren’t a lot of tools besides Emacs, that allows writing programs in this style using any language. And you can’t just make your coworkers use Emacs all of a sudden. Yes, there are solutions specific to a particular language, but they lack usability in my opinion.

Reproducible research was a hot topic lately, and the growing popularity of Jupiter notebooks is a great indication. Org Mode might not provide all the fancy UI elements, that Jupiter notebook has, but theoretically speaking, nothing prevents you from writing a UI in a standalone tool, like the game engine, and using it as a rendering frontend from the comfort of Emacs. This post partly shows this, although, I went a more static route with embedded images. But still, anything is possible if you have the dedication to do it.

I hope this post was an interesting read, and you’ve become interested in Emacs and Org Mode, or even in Fennel and LÖVE combo. Feel free to contact me if you have any questions on the topic. See you soon!

Comment via email

Fennel Game Jam 2022

Sun, 12 Jun 2022 12:37:00 +0300

Quite recently a Fennel game jam happened on itch.io and I’ve decided to participate. I’ve been part of the Fennel community for some years, and every once in a while a lot of people from this community participated in a lisp game jam, but I’ve never made a game before, so I’ve skipped these events. However, when the Fennel-specific game jam was announced I decided that I must finally put myself together and participate.

The rules of the jam were quite relaxed - you basically had to use fennel for game programming, and release the game under an OSI approved license. No theme, no engine restrictions, no size requirement - an ideal situation for a programmer like me, who doesn’t know how to make a game, at all. When I was young, and I mean, like almost 20 years ago, I only toyed with Macromedia Flash and ActionScript 2.0 at my school computer class, making interactive cartoons, which can hardly be considered games. It’s not like I’m a complete stranger to game programming though, we had a very brief introduction to game programming at the university, except we never actually programmed games during these courses. So, for example, I know what delta time is, how it should be used, or how to do vector math needed for basic movement, but this doesn’t count as experience. If you read my blog, you may remember my small projects like raycasting and raymarching, which feature movement, camera programming, basic collision detection, and a basic game-loop. So actually I felt prepared when I decided to enter the jam.

I’ve already tried LÖVE2D in the mentioned raymarching example, so I’ve decided to change things up a bit since I’ve wanted to explore another project - TIC-80. The change from LÖVE2D to TIC-80 actually made it easier for me, since I could do the sprite-work and sound effects inside tic’s inbuilt editors. So I’ve begun to work on it.

I’ll try to structure this post as a series of days based on the notes I made during the event. There probably will not be any code though, as I have no early versions, since I’ve opted out of using version control systems for this project, to keep the retro vibe of the fantasy computer.

Day 1

The hardest day in my opinion because it is the day when I have to start working on a project I haven’t had any thoughts about beforehand. I’ve sat down and thought about what kind of game I can make in the limited time of the event which was going from May 25 to Jun 1st. Given 5 days and almost no game development experience, I’ve decided to play it safe and make a Breakout-like game. Though a plain breakout clone would be a bit boring, so I’ve decided that I’ll put a small twist once you destroy the final block which would shake things a bit and hopefully be a welcome surprise.

Now, since the theme was chosen I at least knew what I needed to start - a ball, a platform, and a box to bounce the ball to prevent it from flying away to infinity. But I didn’t know how to do anything in tic, so I’ve opened up a manual. The manual is minimalist, to say the least, and doesn’t provide a lot of info beyond essential things. So I’ve picked up the circ rect and rectb functions and started working. This was my first result:

Your browser does not support the video tag.

This has very basic hard-coded physics, meaning that I’ve literally defined a vector of all ball velocities and just cycled through it when the ball collided with any world object. But it already has some interesting stuff done, which I haven’t changed much afterward.

The first thing is that It has a proper world to screen coordinate transformation, which I’ve kinda never done before. It’s a simple thing, I just have a world X and Y offsets that made the game world, e.g. the box rest, not in the top left corner but 10 pixels to the right and down. All game physics was calculated with the top left corner in 0,0 and then translated before drawing.

The platform tracks its position and makes sure not to go outside the world boundaries. And the ball is checked for collisions with everything but the lower boundary. There, instead of a collision a basic game-over? flag is set to interrupt the game loop.

I was pretty satisfied for day one, especially since it wasn’t really a full day of me programming this, but only 3 hours after work. The jam started at 10:00 AM in my time zone, and my work day also starts at the same time, so I had to begin working on a game 9 hours after the start of the jam. I’m sure this is a common situation for most other participants, though perhaps they have had time before work too.

Day 2

I knew that I’ll not be able to move forward with the current physics system I’ve added to the game, so I spent most of the second day fighting with the collision system. This was the result I got:

Your browser does not support the video tag.

This had really wonky surface normal calculations, and you can see that the ball isn’t particularly happy with the results, as it just decides to leave the game. I wasn’t happy with it either, but it was a move in the right direction.

In addition to improved physics, I’ve added destructible bricks with a random amount of health represented by a color. The color was an accident, as I used brick health in place of the brick color when bricks were drawn on the screen, but I actually liked it. The health was distributed randomly, but I’ve been focused on physics, so I’ve left it as is.

There were two main problems. First, I’ve re-used some linear algebra code from my raymarching project, mainly the calculation of the distance to the box and the calculation of the normal. However, the calculation of a normal was done for a 3D world and wasn’t as meaningful for 2D space. I’m not sure whether I’ll be able to explain the general method, but the idea is that I’ve found an intersection point, and picked two more points one below the surface and to the left and another above the surface and to the right. Then I used some vector math to find a normal based on distance estimation from these points to the intersection point. This works quite well for 3D, given that all reflections in the raymarching demo were correct, but in 2D it resulted in weird unexpected reflections (not in the video).

Unfortunately, I have this kind of problem that once I’ve fixated on some solution I can’t see another one until I step away from the problem for a bit. So I’ve kept the current physics and ended the day 2 session.

Day 3

This day was mostly spent polishing and rewriting the collision system. I’ve almost fully fixed bugs with unexpected angles, but the ball still managed to find its way inside the platform. I couldn’t spend too much time on it, so I’ve decided to leave it as is, and maybe fix it after everything else is done.

In addition to that, I’ve made it such that the block health depends on the row it was spawned at. So the lower blocks had 1 health, the next row above it had 2 health, and so on. This can be seen with this nice color transition from purple to orange.

Your browser does not support the video tag.

I’ve also done some basic point systems that added 100 points when you destroy a brick, and 10 points upon collision. There are now lives, so you can lose the ball several times before the “game over”. Satisfied with that I’ve ended the third day, realizing that theirs not that much time left, but also not too far til the game will be complete.

For the next two days, I was planning to add graphics and a small twist that will hopefully make the game a bit more interesting.

Day 4

Up until this day I’ve drawn everything with basic rectangles and this wasn’t going to cut for the jam in my opinion. So for the fourth day, I’ve decided to focus on some graphics and sounds.

The inbuilt sprite editor is quite good, actually. I’ve only tried the LibreSprite, an open-source fork of aseprite sprite editor, and it was a bit sluggish, perhaps because it was using a software cursor, which felt unresponsive. But with TIC’s sprite editor it wasn’t a problem and I quickly drew basic sprites for bricks, the platform, and the ball:

Figure 1: Sprites

Then I decided to add power-ups.

There are three classic power-ups that affect the size of the ball or the platform - the big ball, the long platform, and the short platform. All these powers are timer-based, with a small exception for the big ball - if you lose the ball while it’s big, the power-up is also lost. In addition to that, there are two speed-related power-ups, one that increases the ball’s speed, and one that decreases it.

Speaking of speed, I’ve also made it so that the speed of the ball gradually grows during the game up to twice the initial speed. And there’s a mentioned power-up to increase and decrease the value by a quarter.

Here’s a demonstration of various power-ups:

Your browser does not support the video tag.

As can be seen, the ball still can act weirdly, almost spinning around the platform, but I’ve already spent most of my time on graphics and sounds, so I’ve decided to left it for the last day.

Day 5

The last day of the jam! I’ve added more graphics, as I wasn’t able to figure out how to measure things in seconds, as the values of the time function were kinda weird. So I’ve opted out of colored bars that gradually decrease in length. I’ve also decided to add some effects, like screen shake when the ball is big. Here’s how it looks:

Your browser does not support the video tag.

At this point, I’ve started working on the twist and polishing things out. For the twist I’ve decided to do a boss battle at the end of the game, essentially turning the Breakout game into Pong when the last brick is destroyed:

Your browser does not support the video tag.

The boss has a fixed amount of health, and each hit with the ball removes some amount of it. If the big ball power-up is active the ball deals twice the damage. I’ve actually made it that this damage multiplier also affects bricks, so bricks are destroyed a bit faster now too. And if you score the boss, it’ll lose 5 times hp of a single hit, although it’s almost impossible, unless you picked a lot of speed power-ups.

Boss’ health is represented with another bar that changes color depending on the health range, going from green at full to red near the end. To win the game you must defeat the boss, which is actually a bit tricky and I can’t do it consistently myself either. Speaking of power-ups, there’s a random chance to spawn a power-up in a random location during the boss battle. The chance of spawning is very low though.

I’ve uploaded the game to itch as a web build, and noticed a weird problem. All power-ups were wearing off too fast! I couldn’t understand what the problem is, because to my knowledge I did a proper way of handling power-up times - I obtained the current game-time with the call to the time function, added a fixed amount to it, and checked on each frame if the value returned by time surpassed it. To simplify, upon activation of a timer based power-up, I did this end = time() + powerup_duration, and checked for end < time() to disable the power-up. Given that it is based on time and not on some ticking mechanism, I’ve thought that it should work equally well regardless of frames per second. Even more weird was that the ball speed was the same as on the desktop. So in short - power-ups ended a lot faster, yet the rest of the speed was the same.

I’ve decided to do it “properly” and calculated the delta time. Then I multiplied ball movement by newly calculated dt and reworked power-ups to work based on ticks, meaning they weren’t tied to time directly. Checked that all works well on my desktop I’ve made a fresh web build, uploaded it to itch, and it was way too fast! The ball speed now was superfast, power-ups ended almost immediately. After a ton of debugging, and trying to make any sense out of the phrase that the main TIC function is called 60 times per second, when it clearly wasn’t, as the game speed was different between the web and Linux builds. Then I made a Windows build, and it worked with the same speed as the web build, and I thought that maybe it was a bug in the Linux build. I’ve checked what value of dt was calculated on each platform, and the Linux build had a 10-fold difference from other platforms:

Figure 2: Delta time differences between platforms

The Linux build had a delta time value of approximately 0.001 while the web and Windows build had 0.02. I’ve tried an older version of TIC, and it had no such issue but didn’t support the Fennel version I was using for the game. So I had to distribute the game only as a web build until the issue is fixed. I’ve tweaked the speed by adding a game_speed parameter that I could tweak around inside the web build to make the speed feel right. After that, I pushed a final web build with these tweaks and submitted the game.

Jam results

Because of my time zone, I was the first who submitted a game, but I couldn’t do any more work because the last hours of the jam were at nighttime here. Out of 21 participants, only seven games were submitted, which actually a pretty great result, in my opinion, it’s a solid third! Funnily enough, we have about 60 people in our #fennel IRC channel on Libera.Chat, so 21 people is a third too. The results can be viewed here, and I suggest you try all games, as there are pretty cool entries!

I managed to score third place, which actually surprised me a bit, given how simple the concept of my game was. But I’m glad, and I think it’s a solid result for the first game.

In retrospect, I think I could do some things better, especially in the user interface department. Adding a proper menu, maybe an intro, making the interface look more like an arcade cabinet, which I’ve originally attempted, but my pixel-art skills are not that great yet. But I still like the result, I feel that for a first game it turned out pretty great. And the process was quite fun, which is the most important thing for me.

I hope I’ll participate in more lisp-themed jams, maybe in a lisp game jam or the next Fennel jam. I’ve always wanted to make games, and I had some ideas about the game I want to make in the Metroidvania genre. Game jams are perfect for experiments, and you can learn a lot by doing things you never did before. I’m not going to make a whole game during the jam as I want it to be a more complete thing, but the jam can help me to flesh out some main mechanics, be it the movement, or world-building practice. So I’m looking forward to the upcoming events!

This was a lot of fun, and I hope more people will join next time! Thanks to everyone who participated, Matt Roelle for organizing the Jam, and everyone involved in Fennel and TIC-80 development.

Comment via email

Naive async implementation in Fennel

Wed, 27 Oct 2021 09:58:00 +0300

Lately, I’ve been into understanding asynchronous programming. Since my background is mostly bare metal C, which has no asynchronous programming whatsoever (apart from running on multiple chips, and communicating via shared memory), and I did only a little bit of C++ and Rust, it’s fair to say, that async is pretty new for me. And because I now work in Clojure, I sometimes have to use asynchronous programming libraries like core.async, and manifold. Both are great libraries, providing ways to write asynchronous applications and stay sane.

Clojure has really good facilities for writing concurrent programs, as it was designed with that in mind. Fennel, on the other hand, wasn’t, as it targets Lua, and Lua doesn’t have multithreading capabilities at all. Because of that, there are no concurrency primitives in the Lua language, besides coroutines. Coroutines, however, provide a certain degree of multitasking but run on a single thread. It is possible to implement some kind of multitasking via coroutines, but it requires writing a program in a certain way.

I’ve decided to try implementing something similar to Clojure’s core.async for Fennel, but probably naive and much simpler, and see where I can get with it. This article will go into detail on how to implement a rather simple event loop, with a way of spawning concurrent tasks, which rely only on coroutines.

Writing an event loop

The event loop in our case will be an infinite loop, that cycles through all tasks over and over, executing each in a blocking way. The only way of writing a non-blocking code in Lua is to use coroutine.create and then coroutine.yield to temporarily stop the execution. Let’s start with a task queue:

(local queue [])

For our purposes, just a sequential table will do. The queue will hold coroutines, and we need to check their statuses in the loop. Let’s write the loop itself:

(fn worker []
  (while true
    (each [i task (ipairs queue)]
      (match (coroutine.status task)
        :suspended (coroutine.resume task)
        _ (table.remove queue i)))))

If the coroutine is suspended, we resume it, and go to the next one, once it yields. If the coroutine has died, we remove it from the task queue to keep the queue clean.

Now we need a way to spawn coroutines. Let’s write a function, that accepts a function, creates the coroutine, and puts it in our queue:

(fn spawn [task]
  (let [task (coroutine.create task)]
    (table.insert queue task)
    task))

That’s pretty much it! Upon invocation, this function would return something like #<thread: 0x564c2c5e2ae8>, which is a newly created thread object. Not to be confused with real threads, this is just the way Lua refers to coroutines.

We now have a very, very simple event loop, that we can use to spawn coroutines like this:

(spawn (fn [] (for [i 1 5] (print "foo" i) (coroutine.yield))))
;; #(...) is the same as (fn [] ...) but shorter
(spawn       #(for [i 1 5] (print "bar" i) (coroutine.yield)))

Right now nothing happens, because we need to start the worker to see the result.

(worker)

We should see something like this:

foo 1
bar 1
foo 2
bar 2
foo 3
bar 3
foo 4
bar 4
foo 5
bar 5

As can be seen, both tasks were executed simultaneously, because each task pauses itself on each iteration step, so other tasks had a chance to run. It is an example of collaborative (preemptive) multithreading. We can’t spawn real threads in stock Lua runtime without special libraries that provide process-based multithreading.

But these libraries spawn separate lua instances that can communicate via message-passing. Coroutines on the other hand can share states via global variables, which is a bit more performant, as you don’t have to serialize and deserialize data. Although, because everything runs in a single process, there’s no real performance gain, when compared to real threading, so where the performance matters, message-passing between real threads is the way to go.

Let’s leave a message passing out for now, as right now we don’t have a nice way of working with these tasks interactively. Right now, since we’ve started the worker, our tasks are completed, and the worker now loops in an infinite empty cycle, blocking our REPL. Because of this, it is quite cumbersome to work, so let’s write a REPL that will work asynchronously in our event loop alongside other tasks.

Writing an asynchronous REPL

The REPL code below is based on Fennel’s REPL test code but slightly simplified for the purpose of this article. For those unfamiliar with Lisps, REPL stands for Read Eval Print Loop and is something like an interactive shell for working with a running Lisp process. Fennel has an inbuilt REPL, and we can reuse it, but we need to make sure, that its looping aspect doesn’t block our task queue when we wait for input.

Let’s start by declaring prompt variables:

(local prompt-ready "repl>> ")
(local prompt-multi ".. ")
(var prompt prompt-ready)

I’m deliberately changing the standard ">> " prompt to "repl>> ", so we could see that we’re in our special kind of REPL a bit easier. There are two separate prompts here, and we will change the type of our REPL receives an unfinished expression, indicating that the REPL still waits for input. Next, let’s set up the REPL itself:

(local fennel (require :fennel))
(fn repl [options]
  (fn send []
    (var output []) ; ➊
    (fennel.repl
     {:readChunk (fn [{: stack-size}]
                   (set prompt (if (< 0 stack-size) ; ➋
                                   prompt-multi
                                   prompt-ready))
                   (when (> (length output) 0)
                     (io.write (table.concat output "\t") "\n"))
                   (let [chunk (coroutine.yield)] ; ➌
                     (set output [])
                     (and chunk (.. chunk "\n"))))
      :onValues (fn [x]
                  (table.insert output (table.concat x "\t")))
      :onError (fn [_ e]
                 (table.insert output (.. "error: " e)))
      :pp (fn [x]
            (fennel.view x))}))
  (let [send (coroutine.wrap send)]
    (send) ; ➍
    send))

Let’s look at it step by step.

First, we define an output variable ➊. It will store the current output, which we will use in the readChunk method, which we can see in the table, being passed to fennel.repl call.

This method accepts a table with REPL’s state, and we’re interested in the stack-size value. If it is greater than 0, we set the prompt to multiline prompt .., and to repl>> otherwise ➋. Next, we check if the output table contains anything. If it does, we concatenate the elements and print it to standard out. After which we yield ➌ from the REPL until new input arrives. This yield right there is what allows us to use the REPL in a non-blocking way.

The rest methods handle values and errors that happen in the REPL, and pp pretty-prints values for display.

Lastly, we wrap our function, execute it to start the REPL ➍ and return the wrapped function. The resulting function is called send, which sends the input to the REPL for evaluation. But how can we get input? Especially since we don’t want to block other tasks from running.

Input polling

We can get the input with the io.read function, and luckily for us, this function works with the stdin buffer:

>> (io.read 0)
1234
""
>> (io.read "l*")
"1234"

We’ve called io.read(0) which reads 0 characters from standard input. It returned an empty string because that’s what zero characters are. Next, we called it again but with the "l*" argument, which means to read a complete line, and the result was "1234" - the data that we’ve previously sent via stdin, and we’ve successfully read it back.

But this has a problem - we don’t know if stdin already has something in the buffer, so if the buffer is empty, the (io.read "l*") call will block until the input is supplied. Unfortunately, Lua runtime doesn’t have any kind of input polling, and its io module is very minimalist. So we need a way to check if stdin has anything in the buffer. For this task we can use luaposix library which can be installed via luarocks:

$ luarocks install --local luaposix

This library provides the rpoll function, which can be used to check if a file descriptor contains anything ready to be read. Let’s write a task for our worker that will first set up a REPL, and then go into an infinite loop, polling for the input:

(local p (require :posix))

(fn async-repl []
  (let [send (repl)]
    (io.write prompt)
    (io.flush)
    (while true
      (when (< 0 (p.rpoll p.STDIN_FILENO 0))
        (send (io.read "l*"))
        (io.write "\r" prompt)
        (io.flush))
      (coroutine.yield))))

(spawn async-repl)
(worker)

The p.rpool function checks if p.STDIN_FILENO has any data in the buffer. When this function returns a non-zero value, we read the whole line from stdio, and send it to the REPL. Then we display the prompt, flush, and finally yield so other spawned processes could cooperate.

Let’s try it out. The code up to this point can be saved to a file, called async.fnl so we could easier run it from the shell:

$ fennel async.fnl
repl>> (+ 1 2 3)
6
repl>> (+ 1
..  2
..  3)
6
repl>>

Now, when we have the REPL, and it’s working we can spawn some processes!

repl>> (spawn #(for [i 1 5] (print i) (coroutine.yield)))
error: global 'spawn' is not callable (a nil value)

Uh oh, the spawn function is not known by our REPL. But that’s easy to fix. As you may remember, coroutines can access the global state, and we just need to make spawn function and the queue globals:

(global queue [])
(fn _G.spawn [task]
  (let [task (coroutine.create task)]
    (table.insert queue task)
    task))

With this change we can start spawning processes from our REPL:

$ fennel async.fnl
repl>> (spawn #(for [i 1 5] (print i) (for [i 1 1000000] (coroutine.yield))))
#<thread: 0x56055ef88ae8>
repl>> 1
2
3
(print :foo)
foo
repl>> 4
5
repl>>

As you can see, I’ve added an empty loop with coroutine.yield call to slow down the process, so I could input the (print :foo) expression. Because our system toggles between processes, we were able to input the expression and see the result without waiting until the first task completes. This basically means that our non-blocking REPL works.

Though, unfortunately, the prompt gets in the way when printing from different threads. As a workaround, the print function can be redefined to clear the line before printing, but it’s not a proper solution either.

Communicating between threads

The last missing point for our system is a way to communicate between tasks/threads. While global variables can be used by coroutines as a way of sharing the state, it is a really poor choice. Global variables don’t have any concurrency semantics and are especially very hard to debug. Another problem is that we will have to implement synchronization primitives ourselves each time we interact with globals, which is not good.

There are two things that can help us solve this problem. First are so-called promises, which can be blocked upon reading if they weren’t realized, and the other one is a channel. Promises are easy to implement, and we can actually write one like this:

(fn _G.promise []
  {:val nil
   :deliver (fn [self val]
              (if (= self.val nil)
                  (do (set self.val val)
                      true)
                  false))
   :deref (fn [self]
            (while (= nil self.val)
              (coroutine.yield))
            self.val)})

It’s a really simple implementation, but we can see that it works as expected:

repl>> (local p (promise))
nil
repl>> (spawn #(print (.. "promise: " (p:deref))))
#<thread: 0x561ae3b49568>
repl>> (p:deliver 42)
true
repl>> promise: 42
repl>>

As we can see, we’ve created a thread, that parked itself when it tried to dereference the promise, and the moment we delivered the promise, we saw the message from that thread.

Note: Once again, repl>> prompt got in the way, and it will be a common thing in all following examples. I’ve thought about removing the prompt from output manually but decided not to, as if someone will try this code, they might think that something is wrong, as the output will differ from the one in this article.

This is a handy construct when we need to deliver value from one thread to another, without blocking our program, but after the delivery is complete, the promise becomes useless. Because we can’t deliver a promise twice - once it was delivered, the value is set in stone¹. What if we need to pass several values from one thread to another? Of course, we can deliver a table with values to our promise, but what if we don’t have all the necessary data to produce these values right now?

Channels are really good for this - we can put as many values as we want to a named channel and take from it in another thread, parking the reading thread, if the channel is empty, or putting a thread if the channel is full in case of backpressure. Let’s create the chan, take and put functions:

(fn _G.chan [size]
  {:size (or size math.huge)
   :buf []})

For now, chan will simply create a table with our channel buffer buf stored as a sequential table, and optional size, indicating channel size and acting as back pressure. Now we can write a non-blocking put and take:

(fn _G.put [{: buf : size} value]
  (assert (not= nil value) "value must not be nil")
  (while (>= (length buf) size)
    (coroutine.yield))
  (table.insert buf value)
  true)

(fn _G.take [{: buf}]
  (while (= 0 (length buf))
    (coroutine.yield))
  (table.remove buf 1))

You can see, that our put will wait until the length of the channel’s buffer is less than its maximum size. Similarly, take will wait for elements to appear. These operations are non-blocking, which means, that they must be used only in the processes we’re created via the spawn function. Let’s try this out:

repl>> (local c (chan 3))
nil
repl>> (spawn #(for [i 1 10] (put c i) (print (.. "put: " i))))
#<thread: 0x564b48408608>
repl>> put: 1
put: 2
put: 3

First, we’ve created a channel c with a size of 3. Next, we’ve spawned a numeric for loop, that will try putting indexes to the channel, and printing the values afterward. You can see that right after that the put: 1 to put: 3 messages appeared, indicating that we’ve successfully put 3 values to our channel c. However, the values 4 to 10 were not put into the channel, because of the backpressure.

Let’s spawn a single take from the channel:

repl>> (spawn #(print (.. "take: " (take c))))
#<thread: 0x564b483f0ae8>
repl>> take: 1
put: 4

As can be seen, we’ve taken the value 1 from the channel, and immediately the next value was put to it, as shown by put: 4 message. Let’s spawn another loop, that repeatedly takes values from our channel and prints them:

repl>> (spawn #(while true (print "take: " (take c))))
#<thread: 0x564b483dbeb8>
repl>> take: 2
take: 3
take: 4
put: 5
put: 6
put: 7
take: 5
take: 6
take: 7
put: 8
put: 9
put: 10
take: 8
take: 9
take: 10

Aha! Now we can see that this spawned task took 3 elements from the channel and parked itself, waiting for elements. Then our other loop inserted values 5, 6, and 7 into the channel, and parked itself because of the back pressure. And the cycle repeated itself until the putting loop was exhausted.

We can spawn another put, and we’ll immediately see take:

repl>> (spawn #(put c 42))
#<thread: 0x564b48408608>
repl>> take: 42
repl>>

Because the size of the channel was set to 3, we saw puts and takes in groups of three. If we create a channel with unlimited buffer size, we can potentially block our program if some process would decide to put an infinite amount of values to the channel. So it is better to have channels with some buffering, to prevent that from happening.

We can create two more channels to illustrate that:

repl>> (local c-no-bp (chan))
nil
repl>> (do (spawn #(for [i 1 5] (put c-no-bp i) (print (.. "put: " i))))
..         (spawn #(while true (print (.. "take: " (take c-no-bp))))))
#<thread: 0x5607d13a0eb8>
repl>> put: 1
put: 2
put: 3
put: 4
put: 5
take: 1
take: 2
take: 3
take: 4
take: 5
repl>>

As can be seen, the first for loop puts all of its elements into a channel c-no-bp, which has no backpressure, and only then the next loop takes each of these elements. On the other hand, if we create a channel with the size of 1, we’ll get this output:

repl>> (local c1 (chan 1))
nil
repl>> (do (spawn #(for [i 1 5] (put c1 i) (print (.. "put: " i))))
..         (spawn #(while true (print (.. "take: " (take c1))))))
#<thread: 0x5607d1365f78>
repl>> put: 1
take: 1
put: 2
take: 2
put: 3
take: 3
put: 4
take: 4
put: 5
take: 5
repl>>

Channel with a fixed size can act as a synchronization primitive to your program. Alternatively, if you really need an unlimited buffer, you could use coroutine.yield directly in the putting loop, to put one value at a time.

Practical example

Now, when our system is mostly ready, let’s create a practical example with our channels!

For this, I’ve decided to use Love2d - a game engine, that can run Lua directly. Love2d actually has its own implementation of threads, that work similarly to what I’ve described at the beginning of this article - by passing messages, serializing, and deserializing data. The real reason is that I want to draw something using collaborative multitasking, and I could as well take Fengari, but I don’t want to mess with the web until necessary.

So let’s put all our code to the file, and set up the LÖVE project, we’ll only need one file main.fnl for this.

First let’s add our spawn, worker, and channel-related code to the main.fnl file:

(local queue [])

(fn spawn [task]
  (let [task (coroutine.create task)]
    (table.insert queue task)
    task))

(fn chan [size]
  {:size (or size math.huge)
   :buf []})

(fn put [{: buf : size} value]
  (assert (not= nil value) "value must not be nil")
  (while (>= (length buf) size)
    (coroutine.yield))
  (table.insert buf value)
  true)

(fn take [{: buf}]
  (while (= 0 (length buf))
    (coroutine.yield))
  (table.remove buf 1))

(fn worker []
  (each [i task (ipairs queue)]
    (match (coroutine.status task)
      :suspended (coroutine.resume task)
      _ (table.remove queue i))))

Note that the worker no longer has the while loop in it, because it will be handled by love.draw function.

Now, let’s set up our channel, and some threads:

(local c (chan 1))

(let [(width height) (love.graphics.getDimensions)]
  (for [i 0 (/ width 8)]
    (for [j 0 (/ height 8)]
      (spawn #(while true
                (put c [(math.random)
                        (math.random)
                        (math.random)])))
      (spawn #(while true
                (love.graphics.setColor (take c))
                (love.graphics.rectangle :fill i j 1 1))))))

We’ve created a channel with a buffer of size 1. Next, for each point on the canvas, we spawn a thread, that will try to put a random color to the channel. Our channel will refuse to take more than 1 color at a time, so these threads will park once it’s full. The next thread reads one color from the channel and draws a rectangle with this color.

Next, we need to draw all of this:

(fn love.load []
  (love.window.setTitle "Async")
  (love.window.setMode 640 480 {:resizable false :vsync false}))

(fn love.draw []
  (love.graphics.scale 8 8)
  (worker))

The love.load simply sets the window parameters and love.draw sets the scaling and calls our worker, which handles our threads. Compiling this code with fennel -c main.fnl > main.lua and running love . in this directory should result in something like this:

The color for each square is set by a separate thread, and the squares are also drawn by separate threads. As you can see here, each thread is a while loop, that is being parked by calls to put and take, each time our channel c is full. Thus, we’ve almost fully abstracted coroutines away.

Further up

This was a very basic implementation that can be expanded upon if one wants to. Obviously, it’s far from being complete, but even in the current state, you can write code with some sort of multitasking in mind. One thing to improve is our implementation of the worker function, which loops over all threads. Right now it’s just an each loop, but a proper scheduler might reorder tasks in a more efficient way, depending on how tasks perform.

The key problem with this system is that your main application must be running inside this worker as well as one of the threads. Because if you want to block on waiting for a promise, you can’t use a conventional while loop, as it will stop worker from running, and your program will hang. This is something that just needed to be kept in mind though, and I believe it is possible to write programs in such a way.

Additionally to parking on channel or promise operations, a set of macros can be created to replace while, for, each, and other blocking things with non-blocking variants. Such macros will allow running loops in threads without blocking other threads:

(macro pwhile [condition ...]
  `(while ,condition
     (do ,...)
     (coroutine.yield)))

(macro pfor [bindings ...]
  `(for ,bindings
     (do ,...)
     (coroutine.yield)))

(macro peach [bindings ...]
  `(each ,bindings
     (do ,...)
     (coroutine.yield)))

Alternatively, a single macro, that wraps user code can be created, that will analyze code for iteration constructs, and upgrade them to use coroutine.yield. This is somewhat similar to what Clojure’s core.async does with its go macro, which transforms the code.

Overall, I think it is an interesting topic to explore, and I might create a library like this if there will be enough interest.

Nothing is really set in stone, as this is a mutable object, and you can break this guarantee by mutating it. This can be avoided by using proxy tables with special __newindex metamethod, but I’ve left this out for the purpose of this article. ↩︎

Comment via email

Lazy sequences and iterators

Sat, 09 Oct 2021 21:34:00 +0300

Today’s topic will be about lazy sequences and how these are different from iterators. I’ve wanted to make an article on this topic for some time, but unfortunately, there was no good way to show the differences using a single language (that I know), because usually, languages stick to one of those things. But since I’ve been into library writing lately, I wrote a lazy sequence library for the Fennel language, and because Fennel compiles to Lua, it makes Lua a great candidate for this article.

Lua has first-class support for iterators, as it is kind of a central paradigm very closely related to tables. And because iterators are so common in Lua, some libraries even treat them as sequences. However, iterators are not sequences, even though they can be used similarly. And proper sequences also can be created from tables much like iterators and would have similar use cases, without some problems that iterators have. So today we’ll look into Lua iterators, and lazy sequences, and will try to understand their differences.

Iterators in Lua

Let’s begin by understanding what iterators are, and what key features they provide.

In Lua, iterators are usually higher-order functions, produced by another function, based on its input. A standard way to get an iterator over a table is to call pairs function:

get_next, t, k = pairs({a = 1, b = 2, c = 3})

Lua supports multiple return values, so calling pairs returns us the iterator function as the first value, a table (our original table in this case), that we’re going to iterate as the second value, and the starting key, which in our case is nil.

Calling this get_next function, and passing it our table t, and the key k, we get back a pair of values, representing the key and a value associated with that key:

get_next(t, k) -- b	2

We then can use the returned key as an input to the next call to get_next and get the next key-value pair, repeating the process until get_next returns nil:

get_next(t, "b") -- c	3
get_next(t, "c") -- a	1
get_next(t, "a") -- nil

nil means our table is exhausted, and no more keys are present in it. The order may be different for you, as it depends on hashing.

This get_next is a stateless iterator - it doesn’t have any hidden state inside it, which means, that it can be reused. For example, we can pass a different table to the get_next function, and it will work on it without any problem:

get_next({f = 42, g = 27}, nil) -- g	27

This pattern is quite common, so Lua has such a function in its standard library, it is called next and is a basis for all table iterators created with pairs. In addition to pairs, Lua provides another iterator constructor called ipairs, which is used to iterate over sequential tables, and it works similarly to pairs except uses integers as input keys.

While stateless iterators are good, it’s not always efficient, and sometimes not even possible to create a stateless iterator. For example, opening a file in Lua and calling lines method on it creates an iterator over this file, which is a stateful iterator that can be exhausted, i.e. you can call it only so many times as there are lines in the file. Let’s create a file example.txt with the following contents, and create an iterator over it:

line 1
line 2
line 3

file = io.open("example.txt", "r")
get_next_line = file:lines()
get_next_line() -- line 1
get_next_line() -- line 2
get_next_line() -- line 3
get_next_line() -- nil

Note that even though we’re calling the get_next_line function without any arguments, like the file, or the line number, it always returns the next line of the file. This is an example of a stateful iterator, which we can only use once - even if we call the lines method on the file object again, the returned iterator will be exhausted:

new_get_next_line = file:lines()
new_get_next_line() -- nil

This happens because the state is held in the file handle itself, so we need to reopen the file to get a new iterator. Because of this you usually want to open the file, walk it, use or cache needed lines, and close the resource, ideally doing all of this in a protected call.

And lastly, let’s create our own stateful iterator, that keeps its state in the closure. For the purposes of this example, let’s create a range iterator, that can create various ranges based on initial input:

function range(lower, upper, step)
  local lower, upper, step = lower or 0, upper or math.huge, step or 1
  return function ()
    local tmp = lower
    lower = lower + step
    if tmp < upper then
      return tmp
    end
  end
end

To keep it simple I’ve left some corner cases out, but this is basically the idea for a range iterator. When we call range, local variables lower, upper, and step are created based on given arguments or their default values. Then an anonymous function is returned, which refers to these local variables as closures, and sets the lower variable to a new value each time it is called:

r = range(1, 4)
r() -- 1
r() -- 2
r() -- 3
r() -- nil

As you can see, when we exhaust our iterator it returns nil, similarly to iterators created with pairs or file iterators. Thanks to this, we can use it in Lua’s for statement:

for x in range(1, 10, 3) do
  print(x)
end
-- 1
-- 4
-- 7

So now, when we more or less know what iterators are, and how they’re defined, let’s discuss their key properties:

Iterators are lazy,
Stateless iterators accept some kind of state (usually an object and a key) as arguments to produce the next key-value pair, and hence can be reused,
Stateful iterators are like cursors, and can’t be reused unless created anew.

Laziness is a very good property of iterators - it allows us to create actual infinite streams of data, computed on demand. For example, we can open really huge, or even infinite files, and read those line by line without consuming more memory than is actually needed to hold a single line. Alternatively, as was shown with the range iterator above, we can create an infinite range, by not supplying an upper border for it, and it will use the math.huge value, which is inf in Lua. So if Lua had arbitrary precision math, the range would grow to infinity, but in the case of this function, it will just overflow again and again.

However, while laziness is an extremely handy and important part of iterators, their benefits pretty much end on it. Iterator is basically a one-shot thing - you create it, use it until it is exhausted, or you’ve accomplished what you wanted to do with them, and then you never use the same iterator again. This is due to the fact, that iterators were not meant to be shared - you can’t tell if an iterator is stateless or stateful, and if it is a stateful iterator, there’s no safe way to share it. Someone can exhaust it, and later users will not get any values. Stateless iterators are less dangerous to share, but the usefulness of this is also significantly lower, as you need to pass the context along with an iterator.

All of this can be summarized to one point - iterators are not a data structure. And while this is not a bad thing, the usefulness of the iterator is limited because of this. So what if we wanted to share an iterator, and also make sure, that it doesn’t have problems with stateful iterators, and can be used as a data structure? That’s where lazy sequences come in handy!

Lazy Sequences

What are sequences? Think of a lazy sequence as a singly-linked list, with the difference that the tail may not be an actual cell, but a special kind of lazily evaluated cell. In order to create a sequence, we need to implement a so-called cons cell:

function cons (head, tail)
  return function (ht)
    if ht then
      return head
    else
      return tail
    end
  end
end

Conses are an old term that comes from the LISP language and basically represents a pair. It’s a data structure that holds two elements - a value and a pointer to the next value or a cons cell. This is usually illustrated with these boxes:

These stacked boxes represent a single cons cell - a pair of boxes, each referring to a value. The first box refers to 1, and the second box refers to another cons cell, which in turn refers to 2, and finally to nothing, indicating the end of the list. This data structure suits our task quite well, but you may notice that our implementation isn’t exactly a data structure, but a function. So how do we work with it then?

We need two primitive operations. In LISP these are historically called car and cdr, but it’s not meaningful to use these names in the context of Lua, so let’s call them first and rest:

function first (cell)
  return cell(true)
end

function rest (cell)
  return cell(false)
end

Since our cons cell is a function of one argument with if statement in it, first will pass true to it, to get the first element, and rest will pass false to get the other element of a cons cell. Here’s a quick demonstration:

list = cons(1, cons(2, cons(3, nil))) -- 1 -> 2 -> 3 -> nil
first(list) -- 1
first(rest(rest(list))) -- 3

We’ve created a list variable, that holds a single-linked list with values 1, 2, 3. And the final value is nil, which is needed to indicate the end of the list since the last cons cell must refer to nothing. However, this list is not lazy just yet.

Let’s recap how iterators achieve laziness. Iterators produce the next value only when you call them, which makes them very similar to generators by nature. However, the key difference from lazy sequences is that all previous results are thrown away and not kept with the iterator.

So how we can achieve laziness in our sequences? We can try a naive approach of wrapping arguments into functions, and realizing them when a value is requested:

function lazy_cons (headf, tailf)
  return function (ht)
    if ht then
      return headf()
    else
      return tailf()
    end
  end
end

Now, when constructing a list, instead of passing the elements directly, we need to pass some functions that will return the list’s elements:

lazy_list = lazy_cons(
  function () print "realize head" return 1 end,
  function () print "realize tail" return 2 end
)

Because the interface is the same, we can use first and rest to get values:

first(lazy_list)
-- realize head
-- 1
rest(lazy_list)
-- realize tail
-- 2

You can see, that when we’re calling first on our lazy_list it realizes the value, and then returns it. This is very close to our goal, but we’re not there yet. The problem is, that if we call first on lazy_list again, it will call the realization function again and again without caching the result, which is not right:

first(lazy_list)
-- realize head
-- 1
first(lazy_list)
-- realize head
-- 1

To solve this issue, we’ll need to rewrite lazy_cons in a such way, that it will realize our cell only once, and cache the result somehow. This can be done by creating mutable closures, but I think there’s a better way, that will also allow us to do more interesting stuff later. And it is also kinda hard to differentiate if the closure is uninitialized or the realization set it to nil. So instead we can use a clever metatable trick to realize cons cell contents upon access:

function lazy_cons (headf, tailf)
  local cell = {}
  local function realize ()
    local head, tail = headf(), tailf()
    local function realized (_, ht)
      if ht then
        return head
      else
        return tail
      end
    end
    return setmetatable(cell, {__call = realized})
  end
  return setmetatable(cell, {__call = function (_, ht) return realize()(ht) end})
end

Basically, we create a table cell, to which we set a metamethod __call that works similarly to the function that is returned by cons, except not quite. Note the realize()(ht) call in this metamethod - it calls the realize function first, then invokes the resulting function. The realize function is defined above, and this function resets the __call metamethod to another function, that uses cached values. Now using first on a such cell will work as expected:

lazy_list = lazy_cons(
  function () print "realize head" return 1 end,
  function () print "realize tail" return 2 end
)

first(lazy_list)
-- realize head
-- realize tail
-- 1
first(lazy_list)
-- 1
rest(lazy_list)
-- 2

It realizes the cell and then caches the results in a closure, so when we call first on this cell later, we get the same values every time. Now we can create and manipulate lazy sequences with only these three basic functions! Here’s our list example from above, but now lazy:

lazy_list = lazy_cons(
  function () print "first" return 1 end,
  function ()
    return lazy_cons(
      function () print "second" return 2 end,
      function ()
        return lazy_cons (
          function () print "third" return 3 end,
          function () return nil end
        )
      end
    )
  end
)

And we can inspect it with the same first and rest functions:

first(lazy_list)
-- first
-- 1
first(rest(rest(lazy_list)))
-- second
-- third
-- 3
first(lazy_list)
-- 1

That’s pretty much it! We have a way of constructing lazy sequences via a very primitive lazy_cons function, and anonymous functions as arguments. The usefulness of this may not be obvious, so let’s see some more examples of how this can be utilized before we move on.

More examples

Let’s create a lazy sequence that works similarly to the range iterator we did earlier:

function range_seq(lower, upper, step)
  local lower, upper, step = lower or 0, upper or math.huge, step or 1
  return lazy_cons(
    function () return lower end,
    function ()
      local tmp = lower + step
      if tmp < upper then
        return range_seq(tmp, upper, step)
      else
        return cons(nil, nil)
      end
    end
  )
end

A bit more verbose, but notice that this is no longer a stateful thing! Instead, it is a recursive function, which returns a self-referencing cons cell, which calls itself with changed arguments.

A recursion? How’s it lazy then?

The thing is, that it’s not even a recursion! The next recursion step will only happen when a cell is realized, and it will simply produce another lazy cell, which will not go to the next recursion step immediately. And even if Lua had no tail call optimization it would still work just fine, as it acts more like a trampoline, and the call stack never actually grows. Here’s how we can see that our range_seq is working:

r = range_seq(1, 4)
first(rest(rest(r)))
-- 3
first(rest(rest(rest(r))))
-- nil

And for infinite ranges we can drop as many values before working with the range:

r = range_seq()
for i=1,1000000 do
  -- let's drop some elements
  r = rest(r)
end
first(r) -- 1000000
first(rest(r)) -- 1000001

It’s a quite common pattern when working with infinite sequences to drop some values before working with them. So if we want to create an infinite range, that starts from a given value, we can write a drop function, which will produce a new lazy sequence without first n elements in it.

Another thing to note is that it is a bit tedious to write nested rest calls every time, so let’s iterate over this range instead, with the power of recursion:

function loop (seq, f)
  local head, tail = first(seq), rest(seq)
  f(head)
  if tail then
    loop(tail, f)
  end
end

loop(range_seq(0, 10, 2), print)
-- 0
-- 2
-- 4
-- 6
-- 8

This loop function accepts a sequence and calls a supplied f function on each element for side effects. A very similar concept exists in most functional languages, and it is called mapping, so let’s create a lazy map function:

function map (seq, f)
  if first(seq) then
    return lazy_cons(
      function () return f(first(seq)) end,
      function () return map(rest(seq), f) end
    )
  else
    return cons(nil, nil)
  end
end

Now with this map function, we can not only apply a function to every element of the sequence, but we can also collect the results into another sequence. Let’s write a function that prints every element of the given sequence, and pass the result of the map call to it:

function print_seq (seq)
  io.stdout:write("[")
  local function print_seq (seq)
    if first(seq) then
      io.stdout:write(first(seq), ", ")
      print_seq(rest(seq))
    end
  end
  print_seq(seq)
  io.stdout:write("\b\b]\n")
end

function square (x) return x * x end

print_seq(map(range_seq(0, 10), square))
-- [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Now, to demonstrate that this is really a lazy map, let’s add a print call to square, and store the resulting sequence to a variable:

function square (x)
  print("square of "..x.." is "..x * x)
  return x * x
end

squares = map(range_seq(0, 10), square)

Nothing is printed until we realize some elements of our sequence:

first(squares)
-- prints: square of 0 is 0
first(rest(rest(squares)))
-- prints: square of 2 is 4
first(rest(rest(squares)))
-- prints nothing, as the result was already realized

Note that while this is still just a (kinda special) function, it behaves just like a real linked list, and thus represents a proper data structure. Although it’s a bit clumsy to use directly, as shown with the map example, we can build a library around this concept, that will abstract the construction of a lazy sequence away from the user. And that’s exactly what I did.

Lazy-seq library

It should be pointed out that the code above is a very rough example of a lazy sequence implementation. I think it’s fine for explaining the idea, but there are some edge cases that are not handled properly. For example, the map function we’ve written is not fully lazy, as it actually realizes its first element, and the range_seq will not properly work for negative ranges. And the lazy_cons is not really the most useful abstraction - instead of using it directly, a much more handy lazy_seq function should be created, that would accept another function, which in turn would return any kind of sequence. To address these shortcomings I’ve created a lazy-seq library, which has a lot of functions to create and manipulate lazy sequences.

Now, it’s a Fennel library, written in Fennel, and aimed toward it first. That is, mainly due to the fact that this library mimics Clojure’s vision of lazy sequences and ports a lot of functions from its core namespace. But the core principles are exactly the same as described above, and you still can use the library from Lua, just like we did earlier!

Except the semantics are expanded a bit - there are handy constructors for creating lazy sequences from tables or strings, supporting both sequential and associative tables. And, as you may remember from the previous section since our cons cells are secretly tables underneath, sequences defined by this library also implement various metamethods, like __len, __index, and so on, for more convenient usage.

Proper versions of map and range already exist in this library, along with other functions, which can be used as building blocks for creating all kinds of sequences. For example, here’s how one would create an infinite Fibonacci sequence using the lazy variant of the map function in Lua:

lazy = require "lazy-seq"
first, rest, map = lazy.first, lazy.rest, lazy.map
concat, drop, lazy_seq = lazy.concat, lazy.drop, lazy["lazy-seq"]
bc = require "bc"
function add (a, b) return a + b end
fib = concat(
  {bc.new(0), bc.new(1)},
  lazy_seq(function () return map(add, rest(fib), fib) end)
)
first(drop(400, fib))
-- 176023680645013966468226945392411250770384383304492191886725992896575345044216019675

In the example above I’m also using the lbc library which provides arbitrary precision math for Lua to illustrate that you can compute as many Fibonacci numbers as you want (try dropping the first 100000 elements for example) That’s a bit verbose still, but not as verbose as the manually created lazy linked list from previous sections. And note that the concat function, which is used to concatenate sequences, also works with tables, as shown in the example above. All sequence manipulating functions in the lazy-seq library work with tables, as tables are implicitly converted to sequences.

In Fennel, the same code is even more compact because we can use the lazy-cat macro instead of using concat and lazy-seq, and thanks to anonymous function shorthand:

(local {: first : rest : map : drop} (require "lazy-seq"))
(import-macros {: lazy-cat} "lazy-seq")
(local bc (require "bc"))
(global fib (lazy-cat [(bc.new 0) (bc.new 1)] (map #(+ $1 $2) (rest fib) fib)))
(first (drop 400 fib))
;; 176023680645013966468226945392411250770384383304492191886725992896575345044216019675

Note however that macros are a Fennel-only feature, not available in Lua, so where in Fennel you would write:

(import-macros {: lazy-cat : lazy-seq} "lazy-seq")

(lazy-cat [1 2 3] [4 5 6])
(lazy-seq (do (print "I'm lazy") [1 2 3]))

In Lua it would be:

lazy = require "lazy-seq"
concat, lazy_seq = lazy.concat, lazy["lazy-seq"]

concat(lazy_seq(function () return {1, 2, 3} end), lazy_seq(function () return {4, 5, 6} end))
lazy_seq(function () print "I'm lazy" {1, 2, 3} end)

This code is fully equivalent to fennel code, except that macros write all these anonymous functions for you.

But macros are not the only unique feature in Fennel, the other one is destructuring. I’ll not go into details here, but destructuring is kinda similar to pattern matching, and it allows specifying data structure shape and binding values of the data structure to variables defined by that shape. So similarly to how you would destructure a table, thanks to __len and __index metamethods, destructuring works on lazy sequences:

(local {: range} (require "lazy-seq"))
(let [[a b c & rest] (range 10)] (print a b c) rest)
;; 0 1 2
;; [3 4 5 6 7 8 9]

Just make sure you don’t use & destructuring with infinite sequences¹. And more than that, sequences define the __pairs metamethod, and thus support creating stateless iterators created with pairs!

fennel = require "fennel" -- for pretty printer
lazy = require "lazy-seq"
squares = lazy.map(function (x) return x * x end, {1, 2, 3, 4, 5})
for k,v in pairs(squares) do
  print(fennel.view(k), v)
end
-- @seq(4 9 16 25) 1
-- @seq(9 16 25)   4
-- @seq(16 25)     9
-- @seq(25)        16
-- @seq()          25

We’ve come a full circle.

I’ve required the fennel library, to provide the pretty-printed representation of a sequence for a better illustration. You can see that on each step of iteration we see the shorter and shorter sequence. Sequence automatically pretty-printed in the REPL like this, but that’s one of Fennel’s features, that I’ve previously written about, and implemented a very similar linked list there, so if you’re interested in that, you can find some more info there. And viewing such a sequence in this way really shows us that it is nothing else but a data structure!

Unlike tables, a stateless iterator over a sequence uses the tail of the sequence as the key to getting the next element, because the iterator is simply built on top of first and rest. It ends when it reaches the empty cons cell, which is at the end of each sequence. The only exceptions are infinite sequences, which don’t have empty cons at the end as, well, they’re infinite. So calling pairs and iterating over infinite sequences should have another guard for terminating iteration. Or, you can limit the size of the sequence with take, take-while, and other filtering functions.

There are also various functions like line-seq which accepts a file handle, so here’s what happens when we call it on a file:

(local {: line-seq : doall} (require "lazy-seq"))
(with-open [f (io.open "example.txt" :r)] (doall (line-seq f)))
;; @seq("line 1" "line 2" "line 3")

We have to realize the whole sequence before with-open automatically closes the file handle, but you can lazily work with this sequence while you’re inside of the with-open block without any issues. In contrary to the iterator, we can re-use this sequence of lines, as no results have been thrown away.

Finally, as I’ve mentioned, lazy sequences can be generated from tables with the seq function:

(local {: seq} (require "lazy-seq"))
(seq [1 2 3])
;; @seq(1 2 3)
(seq "abc")
;; @seq("a" "b" "c")

If we pass in an associative table we get a bit different representation:

(seq {:a 1 :b 2 :c 3})
;; @seq(["a" 1] ["c" 3] ["b" 2])

These sequences are all lazy because seq creates a wrapper around an iterator over the table. This means, that if you change the table before the sequence is fully realized, your resulting sequence will have updated elements much like an iterator would produce them. This is not recommended, and instead immutable tables a better be used to produce sequences, and in general. And since sequences themselves are immutable and persistent this is a win-win.

The lazy-seq library also provides a lot of other functions, like filtering, restructuring, modifying, and combining infinite sequences. It’ll take a lot of time to list all of them here, so I’d recommend you to read the documentation if you’re interested. But as an example of what you can do, here we create an infinite sequence of random numbers, filter out only those which are perfect squares, and group those by 2:

(local {: take : filter : repeatedly : partition} (require "lazy-seq"))
(local rands (repeatedly math.random 0 64))
(local perfect-squares (filter #(= 0 (% (math.sqrt $) 2)) rands))
(local rand-square-pairs (partition 2 perfect-squares))
(take 10 rand-square-pairs)
;; @seq(@seq(64 16)
;;      @seq(4 0)
;;      @seq(64 4)
;;      @seq(4 0)
;;      @seq(0 4)
;;      @seq(4 64)
;;      @seq(16 0)
;;      @seq(0 64)
;;      @seq(16 4)
;;      @seq(16 36))

Note that we did all operations on an infinite sequence, and then only used take to get the first 10 elements of it. This is a common pattern when working with infinite sequences - create something infinite, filter or modify it, and take the needed amount of elements.

Conclusion

That’s all I wanted to say about lazy sequences and their relationship with iterators. But of course, there’s much more to it, as there are a lot more functions for sequence manipulation I’ve left uncovered because it would simply be too long for an article. The main point I’ve wanted to share is that sequences are similar to iterators, but don’t have some downsides that iterators have. Someone can argue that these downsides are features, but I think sequences still have their uses.

So let’s recap what we have learned about sequences:

Sequences are a data structure, with very important properties of persistence and immutability.
Sequences are lazy, much like iterators, but they cache their values for later re-use.
Things that required stateful iterators, can be done with stateless sequences by using laziness and recursion.
Sequences, implemented by lazy-seq library, support common table operations, making it easy to integrate lazy computation into existing Lua code.
Operations on infinite sequences can be easily composed.

I hope you got a good understanding of lazy sequences, and I got you interested in at least trying them out! Despite being a Fennel library, lazy-seq can be used from Lua just fine, you only need to compile it to Lua first, which is not hard to do, given that Fennel itself is also just a library. But I suggest using this library with Fennel, as it adds a bit more facilities, like pretty-printing of your data structures, and provides macros for a more concise code.

: As of Fennel v1.0.0 the __fennelrest metamethod was added, and & destructuring works on infinite sequences. ↩︎

Comment via email

Condition system for Fennel language and Lua runtime

Sun, 08 Aug 2021 18:47:00 +0300

Not long ago I posted a small article on a Condition System in Clojure language. In that article, I was mostly trying to understand what a condition system is, and how it can enhance error handling in the code I write. Since that time, I’ve understood this system a lot better, by actually trying it in the Common Lisp language, the place where it came from, as far as I know. And you know what it means (because you’ve read the title) - it’s time to implement such a system for Fennel!

I like Fennel a lot, it’s a really cool language to tinker with - it’s small, slick, Lua runtime is actually really versatile, and can be bent to your needs without a lot of extra work. But the reason I’ve decided to implement this library for Fennel, and not for Lua itself, is because of a feature that is not available in Lua but is available in Fennel - lisp macros. The library is already available for everyone to use: fennel-conditions, and in this article, I’ll do a deep dive into the details behind the implementation.

Condition system?

First, let’s actually explain what the condition system is.

Condition system is a feature of the Common Lisp language, that can be very roughly described as resumable exceptions. Though, while this isn’t the only way you can use the condition system, it is the main reason I’ve decided to port this system into Fennel, as Lua error handling is not the best, and we can do better. You still can use a condition system for basic control flow to some extent, though not as versatile as in Common Lisp, I don’t think that Fennel really needs these features.

In Common Lisp, a condition is an object, which represents a certain event. You can bind handlers for particular events, and trigger these handlers by raising a condition. This may sound similar to the classic exception model found in languages like Java, but it has one major difference. Exceptions are a two-part system, while the condition system is three-part.

It’s really hard to come up with a small example to show what condition the system has to offer that exceptions don’t, and I don’t really know Common Lisp well enough to write it. So I encourage you to read Chapter 19 from the Practical Common Lisp book which covers this topic in detail before continuing reading. There’s also a good article on the wiki.c2.com. If you’ve read that before or just now, and didn’t understand what the condition system is, hopefully, you’ll get the idea in this post, but I can’t guarantee that.

If I were to describe the main difference between this system from classic exceptions, I would say that when you write a code that throws an exception, you have to deal with this exception somewhere else. And the place where you handle that exception is the place where the program would continue running after the handling is done.

In the condition system, the place where you’ve thrown such an exception can be the place where the program continues running if you use some special constructs, that allow you to return to that place.

An example

In the spirit of the Practical Common Lisp book example mentioned above, here’s a rather naive Clojure code that parses a log file, and uses exceptions to indicate that parsing failed:

(ns log-parser
  (:require [clojure.java.io :as io])
  (:import [clojure.lang ExceptionInfo]))

(defn well-formed?
  "Checks if the line contains basic date-time, a service name, and a message
separated by colons."
  [line]
  (let [date-re #"([12]\d\d\d-[01]\d-[0-3]\d)"
        time-re #"([01]\d:[0-5]\d:[0-5]\d\.\d+)"
        service-re #"([^:]+)"
        message-re #"(.*)"]
    (re-matches (re-pattern (str date-re "T" time-re ":" service-re ":" message-re))
                line)))

(defn parse-line
  "Extract `date`, `time`, `service` name, and a `message` from a `line`."
  [line]
  (if-let [[_ date time service message] (well-formed? line)]
    {:date date
     :time time
     :service service
     :message message}
    (throw (ex-info "malformed log entry" {:line line}))))

(defn parse
  "Return a vector of parsed lines, each element being a map with the
  following keys: `:date`, `:time`, `:service`, `:message`."
  [lines]
  (try
    (mapv parse-line lines)
    (catch ExceptionInfo e
      (binding [*out* *err*]
        (prn (format "%s:%s" (ex-message e) (:line (ex-data e))))))))

(defn parse-log [file]
  (with-open [f (io/reader file)]
    (parse (line-seq f))))

Note: Yes, there are more idiomatic ways of doing this task in Clojure, but for the purposes of demonstrating the difference between exceptions and conditions this will do.

So this code checks if the line has a valid format, and if not it will throw an exception. And here’s a problem - if a single line is invalid, we will get no results at all. We can avoid that by moving try into our iteration:

(defn parse
  "Return a vector of parsed lines, each element being a map with the
  following keys: `:date`, `:time`, `:service`, `:message`."
  [lines]
  (mapv #(try
           (parse-line %)
           (catch ExceptionInfo e
             (binding [*out* *err*]
               (prn (format "%s: %s" (ex-message e) (:line (ex-data e)))))))
        lines))

This will correctly parse our log, but we will have nil in place of each malformed line. This can once again be fixed by using keep instead of mapv, but what if we can’t afford to lose any lines? Maybe we can fix the line and parse it one more time? This again can be done by modifying our anonymous function, so it would catch the exception, try to fix the line, and then call parse-line again. And if we can’t fix the line, we will log it as before, and don’t add it to the resulting vector:

(defn fix-line
  "We don't care when this happened, just want to know what happened."
  [line]
  (when-let [[_ service message] (re-matches #".*:([^:]+):([^:]+)$" line)]
    (str "1970-01-01T00:00:00.000:" service ":" message)))

(defn parse
  "Return a vector of parsed lines, each element being a map with the
  following keys: `:date`, `:time`, `:service`, `:message`."
  [lines]
  (into []
        (keep #(try
                 (parse-line %)
                 (catch ExceptionInfo e
                   (if-let [fixed-line (fix-line (:line (ex-data e)))]
                     (parse-line fixed-line)
                     (binding [*out* *err*]
                       (prn (format "%s: %s" (ex-message e) (:line (ex-data e)))))))))
        lines))

But then again, when we will use parse-line in some other part of the program we will need to do all this stuff over and over. The main problem here is that we’re building concrete ways to recover from error as a part of the calling function that is meaningful only in the context of the parse-line and the responsibility for recovering should really be on it. Condition system can help us write more generic code, that knows how it can recover from the error, but let the recovery happen on the calling side. Consider the following Fennel code that uses fennel-conditions library:

(local {: error : make-condition : invoke-restart} (require :fennel-conditions))
(import-macros {: handler-bind : restart-case : define-condition} :fennel-conditions)

(define-condition MalformedLogEntry)

(fn well-formed? [line]
  "Checks if the line starts with basic date-time followed by a colon and a word."
  (let [date-pat "([12]%d%d%d%-[01]%d%-[0-3]%d)"
        time-pat "([01]%d:[0-5]%d:[0-5]%d%.%d%d%d)"
        service-pat "([^:]+)"
        message-pat "(.*)"]
    (string.match line (.. date-pat "T" time-pat ":" service-pat ":" message-pat))))

(fn parse-line [line]
  "Extract `date`, `time`, `service` name, and a `message` from a `line`."
  (restart-case
      (match (well-formed? line)
        (date time service message) {:date date
                                     :time time
                                     :service service
                                     :message message}
        _ (do (print line) (error (make-condition MalformedLogEntry {:line line}))))
    (:use-value [x] x)
    (:reparse-line [fixed-line] (parse-line fixed-line))
    (:ignore [ignore-val] ignore-val)))

(fn fix-line [line]
  "We don't care when this happens, just want to know what happened."
  (match (string.match line ":([^:]+):([^:]+)$")
    (service message) (.. "1970-01-01T00:00:00.000:" service ":" message)))

(fn parse [lines]
  "Return a vector of parsed lines, each element being a map with the
following keys: `:date`, `:time`, `:service`, `:message`."
  (handler-bind [MalformedLogEntry
                 (fn [e {: line}]
                   (match (fix-line line)
                     fixed-line (invoke-restart :reparse-line fixed-line)))
                 MalformedLogEntry
                 (fn [e {: line}]
                   (io.stderr:write (.. "malformed log entry: " line "\n"))
                   (invoke-restart :ignore nil))]
    (icollect [_ line (ipairs lines)]
      (parse-line line))))

(fn parse-log [file]
  (with-open [f (io.open file :r)]
    (when f
      (parse (icollect [line (f:lines)] line)))))

For those unfamiliar with condition system special forms or Fennel itself, handler-bind is something similar to try with the difference that the only way to recover from the error is by calling invoke-restart ¹. with-open is the same thing as in Clojure or Common Lisp, and icollect is something similar to mapv in Clojure, but it’s a macro, that accepts body form instead of function.

So what’s different about this code from the code in Clojure’s example? Let’s look at parse-line functions in Clojure and in Fennel:

;; Clojure
(defn parse-line
  "Extract `date`, `time`, `service` name, and a `message` from a `line`."
  [line]
  (if-let [[_ date time service message] (well-formed? line)]
    {:date date
     :time time
     :service service
     :message message}
    (throw (ex-info "malformed log entry" {:line line}))))

;; Fennel
(fn parse-line [line]
  "Extract `date`, `time`, `service` name, and a `message` from a `line`."
  (restart-case
      (match (well-formed? line)
        (date time service message) {:date date
                                     :time time
                                     :service service
                                     :message message}
        _ (error (make-condition MalformedLogEntry {:line line})))
    (:use-value [x] x)
    (:reparse-line [fixed-line] (parse-line fixed-line))
    (:ignore [ignore-val] ignore-val)))

Apart from some syntactic differences, you can see that both functions are throwing an error - by using throw in Clojure, and error in Fennel. However, while the Clojure version only throws the error, the Fennel version does some additional things. What we’re interested in here is the restart-case macro, which is a special macro from the condition system, that allows us to return to its location when we want to handle the error.

So when parse-line throws an error in Clojure, JWM unwinds the stack, and we basically leave this function completely. We then catch the exception in the parse-log function, which can’t do anything about it but handle it right there or let it fall through.

However, when parse-line throws an error in Fennel, we don’t actually unwind the stack at all. Instead, we travel up the dynamic scope, in search of a handler bound to this error by the handler-bind macro in the parse function:

;; ---- 8< ----
(handler-bind [MalformedLogEntry
               (fn [e {: line}]
                 (match (fix-line line)
                   fixed-line (invoke-restart :reparse-line fixed-line)))]
  ;; ---- 8< ----
  )

This handler tries to fix the line, and calls (invoke-restart :reparse-line fixed-line). invoke-restart is another special function in condition system - it once again travels through the second dynamic scope, and finds that there’s a restart named :reparse-line bound by restart-case in the parse-line function. This restart is executed right there, despite the fact that we’ve “left” the lexical scope of that function to find a handler, and its value is used in the original parse-line call.

So the order in which all functions are called looks something like that:

In other words, when we wrote parse-line in Clojure, we only defined how to detect malformed lines and how to signal an error. When we wrote parse-line in Fennel, we not only specified what to do if the line is malformed but also provided concrete ways to recover, that are meaningful in the context of this function.

When we caught the exception in Clojure, we had no choice but to handle it right there. In Fennel, when we caught the exception we could decide whether we can handle the error, and how we can handle it without leaving the original call.

I hope this illustration is clear enough to demonstrate the main difference between exception and condition. Again, yes, there are better ways of dealing with this particular error in Clojure, but what I wanted to illustrate is that there’s no way of achieving the same flexibility of the condition system with just exceptions. I’ve covered possible ways of implementing a rather basic condition system in pure Clojure in my previous post Condition System in Clojure language, so it’s not entirely impossible, but has its own shortcomings. With that in mind let’s talk about the implementation of this system.

The implementation

As I’ve mentioned previously, the dynamic scope is what allows us to travel through our code without transferring the control. However, Lua doesn’t have a dynamic scope builtin into the language. So how is this possible?

Well, as you might know, Lua has tables, and these tables are dynamic and mutable. So theoretically, we could create a table, that is available globally, properly encapsulate it, and use it as our dynamic scope. And that’s exactly what the fennel-conditions library does.

However, implementing dynamic scope via tables is not enough. First of all, we need to manage the dynamic scope in a sensible way. Second, we need some way of actually traveling through the lexical scope to the point where the dynamic scope was created, once we know where we need to move after all handling has been done. And finally, the scope should be thread-local, and even though Lua doesn’t have threads, it has coroutines, which can act as threads, so we should maintain dynamic scope with respect to this aspect of the language.

Let’s begin by listing things that will be implemented in the condition system:

condition objects - conditions are objects, that can have inheritance, and can embed additional information inside to convey more info about a conditional situation when needed. Condition objects are defined with define-condition macro, and a particular instance is created from a base condition object with make-condition function.
handler-case, handler-bind, restart-case - macros for creating new frame of the dynamic scope, in which the handlers, and restarts are bound to condition objects.

As well as:

cerror - a macro for raising a condition that can be ignored by invoking special continue restart.
ignore-errors - a macro for suppressing all errors, similarly to try with a catchall clause that does nothing.
unwind-protect - a macro for catching all errors, similar to try with just a finally clause.

All these macros share the very common part about setting up and cleaning the dynamic scope, as well as some pcall trickery to make it all work, which makes it a good entry point to understand what’s happening under the hood.

The dynamic scope

Beware, this is going to get really messy. Dynamic scoping is a foreign concept for Lua, so we have to deal with it ourselves, and ensure that it is correct. But before we begin, I should illustrate the structure of the library:

fennel-conditions/
├─ impl/
│  ├─ condition-system.fnl
│  ├─ debugger.fnl
│  └─ utils.fnl
├─ init.fnl
└─ init-macros.fnl

The init.fnl, and init-macros.fnl at the top level are the main entry points for runtime functions, and macros. These modules depend on the impl/condition-system.fnl module - the library’s private API. Which in turn uses impl/debugger.fnl that holds implementation of the interactive debugger, and impl/utils.fnl which stores common functions, used by condition system and debugger. The main things we’re interested in are init.fnl, init-macros.fnl, and impl/condition-system.fnl - the meat and bones of the library.

The dynamic scope is stored in the impl.condition-system module and this module is required every time we need to access the dynamic scope. Lua has a system for caching required modules, so this is a rather cheap operation and this way we don’t need to expose dynamic scope in the global scope.

This is how the dynamic scope table is defined in impl.utils:

(local {: metadata : view} (require :fennel))

(local dynamic-scope
  (metadata:set
   {}
   :fnl/docstring
   "Dynamic scope for the condition system.

Dynamic scope is a maintained table where handlers and restarts are
stored thread-locally.  Thread name is obtained with
`coroutine.running` call and each thread holds a table with the
following keys `:handlers`, `:restarts`, and `:current-context`.
Handlers and restarts itselves are tables."))

It’s nothing special really, just an ordinary table, but the documentation string mentions several interesting points. First, that this is a maintained table, meaning that it should not be modified manually. Second, that it stores scopes in thread-local contexts, meaning that each thread will have its own dynamic scope table. Each such table in turn has a table for handlers, a separate table for restarts, and some current-context field, which we will cover a bit later.

So, to illustrate how the dynamic scope may look at some point of the program running:

{"thread: 0x564cab9a12a8"
 {:handlers {:handler {#<condition MalformedLogEntry: 0x564caba2c6e0>
                       #<function: 0x564cabb01500>}
             :handler-type "handler-bind"
             :parent {:handler {#<condition MalformedLogEntry: 0x564caba2c6e0>
                                #<function: 0x564caba5ccf0>}
                      :handler-type "handler-bind"
                      :parent {}
                      :target {}}
             :target {}}
  :restarts {:parent {:parent {:parent {}
                               :restart {:args ["ignore-val"]
                                         :name "ignore"
                                         :restart #<function: 0x564cabb01580>}
                               :target {}}
                      :restart {:args ["fixed-line"]
                                :name "reparse-line"
                                :restart #<function: 0x564cabaa7a10>}
                      :target {}}
             :restart {:args ["x"]
                       :name "use-value"
                       :restart #<function: 0x564caba5cd30>}
             :target {}}}}

This is the state of the dynamic scope when we enter the restart-case macro and do all setups, in the fennel code from the example section. There’s quite a lot happening here - lots of nested tables with similar internal structures, and even some empty tables.

First thing you can spot is that the outermost table has a key "thread: 0x564cab9a12a8" which holds the table with handlers and restarts. This is the thread-local dynamic scope, representing the main thread. Inside of it in the table under the handlers key, we see a table with a handler key holding a table with a condition, (which is actually just another table²), and a function:

{:handler {#<condition MalformedLogEntry: 0x564caba2c6e0>
           #<function: 0x564cabb01500>}
 :handler-type "handler-bind"
 :parent {:handler {#<condition MalformedLogEntry: 0x564caba2c6e0>
                    #<function: 0x564caba5ccf0>}
          :handler-type "handler-bind"
          :parent {}
          :target {}}
 :target {}}

This table corresponds to all dynamic scopes set up with handler-bind macros. But as you can see here, it has a parent key, that holds another very similar table, but as you might remember, there was only one handler-bind instance in the example code - why there are two tables? Here’s a quick refresher of how handler-bind was used:

(handler-bind [MalformedLogEntry
               (fn [e {: line}]
                 (match (fix-line line)
                   fixed-line (invoke-restart :reparse-line fixed-line)))
               MalformedLogEntry
               (fn [e {: line}]
                 (io.stderr:write (.. "malformed log entry: " line "\n"))
                 (invoke-restart :ignore nil))]
  ;; ---- 8< ----
  )

Note that there are two separate entries for MalformedLogEntry condition. Each such entry creates a separate frame in the dynamic scope, which results in two nested tables as shown above. This is done this way because the handlers are bound sequentially, and their order in the handler bind determines the order of how they will be called. We directly use this property, by writing the first handler in such a way, that if the line can’t be fixed with fix-line function, we are not calling invoke-restart and the condition falls through to the next handler. This handler in turn calls :ignore restart, and successfully recovers from the error. Now let’s look at the dynamic scope stored under :restarts:

{:parent {:parent {:parent {}
                   :restart {:args ["ignore-val"]
                             :name "ignore"
                             :restart #<function: 0x564cabb01580>}
                   :target {}}
          :restart {:args ["fixed-line"]
                    :name "reparse-line"
                    :restart #<function: 0x564cabaa7a10>}
          :target {}}
 :restart {:args ["x"]
           :name "use-value"
           :restart #<function: 0x564caba5cd30>}
 :target {}}

Very similarly, each level stores a :restart table, with an argument list, name, and the function itself. Under the :parent key, the very similar table is stored, holding the next restart. Since our code defined three restarts, we get a table of the depth of three. One might wonder, why this is necessary, restarts are selected and not tried in order. While this is true, nothing prevents you from creating two restarts with the same name:

(restart-case
    (/ 1 nil)
  (:use-value [x] x)
  (:use-value [x] x))

This will create the following table:

{:parent {:parent {}
          :restart {:args ["x"]
                    :name "use-value"
                    :restart #<function: 0x564cabb01380>}
          :target {}}
 :restart {:args ["x"]
           :name "use-value"
           :restart #<function: 0x564cabaa5b40>}
 :target {}}

You then will be able to choose which restart to call via the interactive debugger interface. But more about that later.

Now that we have a rough understanding of how dynamic scopes are composed inside, let’s look at how those are really constructed.

Constructing dynamic scope for handlers

Let’s look at the handler-bind macro invocation once again:

(handler-bind [MalformedLogEntry
               (fn [e {: line}]
                 (match (fix-line line)
                   fixed-line (invoke-restart :reparse-line fixed-line)))
               MalformedLogEntry
               (fn [e {: line}]
                 (io.stderr:write (.. "malformed log entry: " line "\n"))
                 (invoke-restart :ignore nil))]
  ;; ---- 8< ----
  )

Much like let, it accepts a vector of bindings - a condition object, and a function that will be called. Similarly to let it binds sequentially, but doesn’t override bindings, but creates a set of nested scope frames to store each binding. Then it executes the body in the context of this dynamic scope.

Let’s look at the actual implementation and break it down:

(fn handler-bind [binding-vec ...]
  (let [target (gensym :target)
        scope (gensym :scope)
        binding-len (length binding-vec)
        setup '(do)]
    (assert-compile (= (% binding-len 2) 0)
                    "expected even number of signal/handler bindings"
                    binding-vec)
    (for [i binding-len 1 -2]
      (let [condition-object (. binding-vec (- i 1))
            handler (. binding-vec i)]
        (assert-compile (or (sym? handler) (function-form? handler))
                        "handler must be a function"
                        handler)
        (table.insert setup `(assert (not= nil ,condition-object)
                                     "condition object must not be nil"))
        (table.insert
         setup
         `(tset ,scope :handlers {:parent (. ,scope :handlers)
                                  :target ,target
                                  :handler-type :handler-bind
                                  :handler {,condition-object
                                            ,handler}}))))
    `(let [,target {}
           {:pack pack#} (require ,utils)
           {:pcall-handler-bind pcall-handler-bind#} (require ,condition-system)
           ,scope ,(current-scope)
           orig-handlers# (. ,scope :handlers)]
       ,setup
       (pcall-handler-bind#
        #(pack# (do ,...)) ,scope ,target orig-handlers#))))

This is the entirety of handler-bind macro, minus the docstring and some comments, and while the actual body that gets spliced into the program’s source code is not very big, the setup looks somewhat complicated. So let’s break it down.

(fn handler-bind [binding-vec ...]
  (let [target (gensym :target)
        scope (gensym :scope)
        binding-len (length binding-vec)
        setup '(do)]
    (assert-compile (= (% binding-len 2) 0)
                    "expected even number of signal/handler bindings"
                    binding-vec)
    ;; ---- 8< ----
    ))

First, we create some unique names that we will use both at compile-time, and then at runtime. You might remember the :target key in tables, that represented the dynamic scope. This key is holding this target symbol, which we define at run time, but we need a way of putting it into the dynamic scope at compile-time, that’s why it’s defined here. I’ll explain why it’s an important key soon.

Next up is the scope, this is an instance of dynamic scope we’re going to construct at compile time. binding-len is just to avoid small pieces of code duplication, and the last one - setup is the code that will fill the missing bits of our dynamic scope at runtime.

After let, we’re doing a check, that the binding-vec, a vector that holds condition objects and their handlers, has an even amount of values. We can’t properly construct the dynamic scope if a handler is missing, so it’s a compile-time error.

Now for the main part - the for loop, which creates a set of instructions that will construct the dynamic scope:

;; ---- 8< ----
(for [i binding-len 1 -2]
  (let [condition-object (. binding-vec (- i 1))
        handler (. binding-vec i)]
    (assert-compile (or (sym? handler) (function-form? handler))
                    "handler must be a function"
                    handler)
    (table.insert setup `(assert (not= nil ,condition-object)
                                 "condition object must not be nil"))
    (table.insert
     setup
     `(tset ,scope :handlers {:parent (. ,scope :handlers)
                              :target ,target
                              :handler-type :handler-bind
                              :handler {,condition-object
                                        ,handler}}))))
;; ---- 8< ----

It does several things - first, it checks that the handler is either a symbol or a function form, i.e. a list that will evaluate a function definition. Next it inserts the runtime assertion, that verifies that the condition-object is not nil. And finally, the tset call is inserted, that will populate current scope’s :handlers table with a new table, that refers to the original table via the :parent key.

You’ve might notice that the loop actually goes backward. This is done to preserve the correct order when constructing nested tables. As the dynamic scope is constructed as a set of nested tables, the order of construction has to be reversed, so the travel order gets preserved. But this code doesn’t actually construct the scope, it just describes how to construct it. The last portion of the macro is what gets spliced into the user’s code, so let’s look at it:

;; ---- 8< ----
`(let [,target {}
       {:pack pack#} (require ,utils)
       {:pcall-handler-bind pcall-handler-bind#} (require ,condition-system)
       ,scope ,(current-scope)
       orig-handlers# (. ,scope :handlers)]
   ,setup
   (pcall-handler-bind#
    #(pack# (do ,...)) ,scope ,target orig-handlers#))

We’re finally setting some of those symbols, we’ve defined before! The ,target {} thing, is a way to make sure we get a unique object address that we can later compare, to determine if we’ve reached the correct destination when “unwinding”. Next, we import some helper functions, like pack# and pcall-handler-bind# - the first one is a portable table.pack implementation, and the last one is a runtime part of handler-bind.

After that, we’re calling something like current-scope. It is a commonly used part of all macros, so I’ve put it into another macro. In short, it just returns the current thread’s dynamic scope or creates it if it doesn’t exist yet. And right after that, you can see that we’re caching original handlers right before ,setup. This ,setup will splice in all that setup we’ve been constructing at compile, that will populate the current scope with new tables. And then we’re calling pcall-handler-bind# function with all necessary info.

So this code from the example:

(handler-bind [MalformedLogEntry
               (fn [e {: line}]
                 (match (fix-line line)
                   fixed-line (invoke-restart :reparse-line fixed-line)))
               MalformedLogEntry
               (fn [e {: line}]
                 (io.stderr:write (.. "malformed log entry: " line "\n"))
                 (invoke-restart :ignore nil))]
  (icollect [line (f:lines)]
    (parse-line line)))

uses handler-bind macro we’ve just examined. Let’s look at what this code expands to:

(let [target_1_ {}
      {:pack pack_10_auto} (require "fennel-conditions.impl.utils")
      {:pcall-handler-bind pcall-handler-bind_11_auto} (require "fennel-conditions.impl.condition-system")
      scope_2_ (let [cs_3_auto (require "fennel-conditions.impl.condition-system")
                     thread_4_auto (or (and coroutine
                                            coroutine.running
                                            (tostring (coroutine.running)))
                                       "main")]
                 (match (. cs_3_auto "dynamic-scope" thread_4_auto)
                   scope_5_auto scope_5_auto
                   _ (let [scope_5_auto {:handlers {} :restarts {}}]
                       (tset cs_3_auto "dynamic-scope" thread_4_auto scope_5_auto)
                       scope_5_auto)))
      orig-handlers_12_auto (. scope_2_ "handlers")]
  (do (assert (not= nil MalformedLogEntry) "condition object must not be nil")
      (tset scope_2_ "handlers"
            {:handler {MalformedLogEntry
                       (fn [e {:line line}]
                         (io.stderr:write (.. "malformed log entry: " line "\n"))
                         (invoke-restart "ignore" nil))}
             :handler-type "handler-bind"
             :parent (. scope_2_ "handlers")
             :target target_1_})
      (assert (not= nil MalformedLogEntry) "condition object must not be nil")
      (tset scope_2_ "handlers"
            {:handler {MalformedLogEntry
                       (fn [e {:line line}]
                         (match (fix-line line)
                           fixed-line (invoke-restart "reparse-line" fixed-line)))}
             :handler-type "handler-bind"
             :parent (. scope_2_ "handlers")
             :target target_1_}))
  (pcall-handler-bind_11_auto
   (hashfn (pack_10_auto (do (icollect [line (f:lines)]
                               (parse-line line)))))
   scope_2_ target_1_ orig-handlers_12_auto))

Quite a lot of code, and there are even more things going on when we’re calling the pcall-handler-bind function! But you can see that it does exactly what I’ve described above - requires some API functions, creates a thread-local scope, sets it up, and wraps our code with the pack function, for later execution. Let’s now look at what pcall-handler-bind does.

Implementing control transfer via protected calls

The pcall-handler-bind function is defined in the library’s private API:

(fn pcall-handler-bind [f scope target orig-handlers]
  "Call `f` in given `scope` and pass result up to `target`.
Restore dynamic scope handlers to `orig-handlers`."
  (let [(ok res) (pcall f)]
    (doto scope
      (tset :current-context nil)
      (tset :handlers orig-handlers))
    (if ok (unpack res)
        (match res
          {:state :handled : target}
          (raise res.type res.condition-object)
          {:state :error :message msg}
          (if (or scope.handlers.parent
                  scope.restarts.parent)
              (error res)
              (error msg 2))
          _ (error res)))))

Looks pretty simple, but this is because we delegate most of the work to other functions.

It starts with calling f in a protected call and immediately removes current-context from the scope, and sets handlers to the original handlers, which we’ve stored back at the macro side. Then, if the function exited successfully, we just return the results by unpacking those (for keeping nils in multiple value returns). But when the function didn’t succeed, we expect a set of certain things as a return.

For those unfamiliar with match - it’s a pattern-matching macro from Fennel standard library. It accepts an expression, and a pair of patterns and their respective bodies to run, if the result of an expression matches this pattern. The first pattern here is {:state :handled :target target}, meaning that the function f returned a table with a key :state holding a :handled value, and a non-nil :target key that is equal to the value of target, passed as function argument. The table returned in this case usually looks like this:

{:state :handled
 :target {}
 :type :error
 :condition-object #<condition MalformedLogEntry: 0x564caba2c6e0>}

In this case, we raise the condition object stored in the result from calling f, with a correct type, meaning that the type is :error, we raise it as an error, or if it’s a :warning, as a warning and so on. We’ll look at raise in a moment, but for now, all you need to know is that it will throw the condition object further up.

If the pattern contained :state :error we check if we have more handlers up the dynamic scope, and if we do, we throw the result as is, so it could be handled. If not, that means that we’re already at a top level, so we convert this to Lua error with a message.

In all other cases this is an error that we want to throw up, as we can’t do anything about it because it is either not a conditional situation or target doesn’t match the expected one.

So the basic idea here is that we’re ensuring that the function will either succeed or fail as we expect, and if it fails in an unexpected way, we just rethrow the error. Lua doesn’t have a way to rethrow the stack traces, but it’s a small price to pay here.

Let’s look at raise now:

(fn raise [condition-type condition-object]
  "Raises `condition-object' as a condition of `condition-type'.
`condition-object' must not be `nil'."
  (assert (not= nil condition-object)
          "condition must not be nil")
  (let [thread (current-thread)
        thread-scope (do (when (not (. dynamic-scope thread))
                           (tset dynamic-scope thread {:handlers {} :restarts {}}))
                         (. dynamic-scope thread))]
    (when (= thread-scope.handlers.handler-type :handler-bind)
      (match thread-scope.current-context
             scope (let [target scope.target]
                     (var scope scope.parent)
                     (while (and scope (= target scope.target))
                       (set scope scope.parent))
                     (tset thread-scope :handlers scope))))
    (match condition-type
           :condition (raise-condition condition-object)
           :warning (raise-warning condition-object)
           :error (raise-error condition-object))))

This function does something interesting with current-context. As you saw in pcall-handler-bind, we’ve set the current-context to nil, but raise actually check it. Why?

This is because the condition can be raided inside the handler. In such a case, we need to unwind the stack to the point where we were in the handler. So each invocation of the handle function sets the current-context field, and this field is cleared when we exit the handler. This stack unwinding needs to be done in order to avoid infinite recursion, which will result in a stack overflow error.

After that is done, we branch on the condition type and raise the condition accordingly. For the purposes of this article let’s look at conditions of error type only, by looking at the raise-error function:

(fn raise-error [condition-object]
  (match (raise-condition condition-object :error)
    nil (if _G.condition-system-use-debugger?
            (invoke-debugger condition-object)
            (error (compose-error-message condition-object) 2))))

In case the condition is not handled it’s raised as a Lua error unless the debugger feature was enabled. The condition itself is raised with the raise-condition function:

(fn raise-condition [condition-object type*]
  (match (handle condition-object (or type* :condition))
    (where (or {:state :handled &as res}
               {:state :restarted &as res}))
    (error res 2)
    _ nil))

Here’s the handle innovation, mentioned above. The result can be either a table with :handled or :restarted states, each of which tells the condition system what to do next. Even though the condition is handled this result gets thrown up as an error because we actually need to tell the condition system that the condition was handled, and differentiate it from a successful return from the function.

So let’s look at how conditions are handled.

Handling conditions

The handle function looks like this:

(fn handle [condition-object type* ?scope]
  "Handle the `condition-object' of `type*' and optional `?scope`.

Finds the `condition-object' handler in the dynamic scope.  If found,
calls the handler, and returns a table with `:state` set to
`:handled`, and `:handler` bound to an anonymous function that calls
the restart."
  (let [thread (current-thread)
        thread-scope (. dynamic-scope thread)]
    (match (find-handler
            condition-object
            type*
            (or ?scope thread-scope.handlers))
      {: handler : scope}
      (do (tset thread-scope :current-context scope)
          (match scope.handler-type
            :handler-case {:state :handled
                           :handler #(handler condition-object (unpack (get-data condition-object)))
                           :target scope.target
                           :condition-object condition-object
                           :type type*}
            :handler-bind (do (handler condition-object (unpack (get-data condition-object)))
                              (handle condition-object type* scope.parent))
            _ {:state :error
               :message (.. "wrong handler-type: " (view _))
               :condition condition-object}))
      _ {:state :error
         :message (.. "no handler bound for condition: "
                      (get-name condition-object))
         :condition condition-object})))

Let’s dive into it.

After getting the current thread and the scope, we call find-handler. For now, it’s enough to know, that it takes a condition object, and walks the dynamic scope comparing this object to handlers. Next up, if the handler were found, we set the current-context to the handler’s scope. This will ensure that we know where we last were before actually executing a handler, so we could unwind the stack to this point in case an error happens inside the handler. This was a very painful thing to figure out when I worked on the implementation.

After that, we branch on the handler type. The handler-case macro allows handling without calling invoke-restart, so we just return a table with :state set to :handled, and a handler that actually does the handling.

The handler-bind is different - it requires handling with invoke-restart, which, as you might remember throws an error to transfer control flow. So if the handler function exited normally, we go to the next handle invocation, which will propagate the condition object to another handler up the stack. This ensures that we can decline to handle a condition.

If no handler is found the error is returned.

A similar pattern occurs for other condition types, with the difference being that some may not return an actual error when unhandled.

So how handlers are found? The find-handler function, which accepts a condition object and a handler type, will walk the scope comparing the given condition object to bound handlers. However, condition objects actually support inheritance, so this gets a bit more complicated. We need to look for a handler in the current scope, and if no handler is found, we need to get the parent from the condition object and search its handler in the same scope. If no handler and no parent handler are to be found, we go to the next scope frame.

Additionally, conditions may not be of the condition-object type, but be an arbitrary Lua value:

(handler-case (signal (math.random 5))
  (0 [] :zero)
  (1 [] :one)
  (2 [] :two)
  (Condition [] :else))

;; an unnecessarily complicated equivalent of

(match (math.random 5)
  0 :zero
  1 :one
  2 :two
  _ :else)

Notice, that while I’m handling conditions as numbers, the catch-all condition object is specified as Condition. Let’s talk more about condition inheritance.

Condition inheritance

As was already mentioned, condition objects can have parent condition objects. A parent can be explicitly set by using the :parent key when defining a condition:

(define-condition MathError)
(define-condition DivideByZero :parent MathError)
(define-condition NegSquareRoot :parent MathError)

(fn sqrt [x]
  (if (< x 0)
      (error (make-condition NegSquareRoot x))
      (math.sqrt x)))

(fn div [a b]
  (if (= b 0)
      (error (make-condition DivideByZero))
      (/ a b)))

(fn main []
  (handler-case (sqrt -1)
    (MathError [_ x] (io.stderr:write "Can't take square root of " x "\n")))
  (handler-case (div 2 0)
    (MathError [_ x] (io.stderr:write "Can't divide by zero\n"))))

Here’s an example of handling different errors through their common parent. You can create hierarchical errors, which then can be handled with more granularity, or as a whole class of errors.

But there are a bit more rules regarding inheritance. All conditions raised with error function will by default inherit from the Error condition. All conditions raised with warn will inherit from a Warning condition. Both Warning and Error are inherited from the Condition condition.

This also applies to any arbitrary data you can throw with error, warn, or signal:

(fn raise-random-condition []
  (match (math.random 2)
    0 (error "an error")
    1 (signal "a signal")
    2 (warn "a warning")))

(handler-case
    (raise-random-condition)
  (Error [msg] (print "caught an error" msg))
  (Warning [msg] (print "caught a warning" msg))
  (Condition [msg] (print "caught a condition" msg)))

Here all conditions are just Lua strings, but we still have some hierarchy. Note that the order of handlers here is very important - placing the (Condition [msg] (print "caught a condition" msg)) as the first handler will prevent Error and Warning handlers from ever running. It is very similar to match in that regard - placing _ as the first pattern will make all other patterns unreachable.

Another thing to point out is that Lua errors also inherit from Error condition:

(handler-case (/ 1 nil)
  (Error [] 0))

This is done for seamless integration with libraries that don’t use the condition system, or if your code did something wrong by mistake.

However, the integration is not as seamless as it could be. The point where it get’s not as good as it could be is when interfacing with the interactive debugger, provided by the library.

The debugger

Speaking of which, the debugger is not as featureful as in Common Lisp, but still provides a way to recover from an error, when the current scope allows for it. Consider this:

(restart-case (cs.error 10)
  (:use-value [x] x))

Executing this code will raise a condition, which is not handled, so it gets propagated to a Lua error:

runtime error: condition 10 was raised

However, if we set _G.condition-system-use-debugger? to true, we’ll get this instead:

Debugger was invoked on unhandled condition: 10
restarts (invokable by number or by name):
  1: [use-value] use-value
  2: [throw    ] Throw condition as a Lua error
debugger>>

Now every unhandled condition will invoke the debugger, which will provide you with available recovery options. In the case of this example, there are two restarts bound - use-value, bound by the restart-case macro, and the other one is an automatically bound throw restart, that will exit the debugger.

By typing 1 or use-value in the prompt, we’ll enter the next prompt, that asks for a value to use for a selected restart:

debugger>> 1
Provide inputs for use-value (args: [x]) (^D to cancel)
debugger:use-value>> 5
5

Thus making the result of the expression to be 5, accordingly to the restart’s body.

This debugger has no runtime penalty unless the condition was not handled. So only When an unhandled condition pops to the top of the dynamic scope stack, the debugger begins its work.

Debugger again traverses the scope, gathering all available restarts, and forms this interactive prompt, with each restart in order of their definition:

(restart-case
    (restart-case
        (restart-case (/ 1 nil)
          (:use-value [x] x)
          (:use-value [x] "use-value (impostor)" (- x)))
      (:do-something [] (+ 1 2 3)))
  (:do-something-else [] (- 3 2 1)))

Note that the use-value restart was specified twice. I’ve mentioned that this is possible, but it is meaningless unless we use a debugger. Here’s what will be prompted to the user, if this code is executed:

Debugger was invoked on unhandled condition: attempt to perform arithmetic on a nil value"
restarts (invokable by number or by name):
  1: [use-value        ] use-value
  2:                     use-value (impostor)
  3: [do-something     ] do-something
  4: [do-something-else] do-something-else
  5: [throw            ] Throw condition as a Lua error
debugger>>

Note that the first restart is indeed named use-value, but the second one has no name, only its docstring is displayed. This restart can be manually called by typing 2 in the prompt - it will ask for the argument similarly to use-value restart, but will negate it. Other restarts are self-explanatory.

Ending thoughts

This should provide some information about this library’s internals.

Seeing how many pcall calls it does to transfer control flow around, and how it manages the dynamic stack, can make you think that it has not a very good performance. And in fact, I would agree, the performance could be better. But it’s not that bad. For example, here’s a 10000000 invocations of Lua’s error function inside a pcall:

(macro time [expr times]
  `(let [c# os.clock
         s# (c#)
         r# ,expr
         e# (c#)]
     (print (.. "Average time elapsed " (/ (* 1000 (- e# s#)) ,(or times 1)) " ms"))
     r#))

(time (for [i 1 10000000]
        (pcall error 10))
      10000000)

;; Average time elapsed 0.0002025262 ms

And here’s an equivalent using condition system’s error and ignore-errors macro:

(local {: error} (require :condition-system))
(time (for [i 1 10000000]
        (ignore-errors (error 10)))
      10000000)
;; Average time elapsed 0.0051841138 ms

There’s one order of magnitude difference in performance here, meaning that ignore-errors is ~10 times slower than pure pcall. But the benefits, that the condition system provides, out-weight this cost, in my opinion.

Also, a note on why I didn’t port the rest of Common Lisp’s condition system. While this library is not a full port, I think it has all the necessary parts in the context of the Fennel language. What I mean by that is, that since Fennel, for example, deliberately chose to avoid Lua’s return special, I don’t think a condition system should provide block and return, or go specials. I don’t have strong reasoning behind it, but I feel that the error handling part in Lua is what’s needed to be improved, thus this library is only aimed at fixing this part of the runtime.

In Common Lisp there are more ways to recover from an error from the handler bound with handler-bind, but fennel-conditions library doesn’t implement all those ways. ↩︎
Tables, tables, tables, tables, tables… tables, and tables - that’s Lua at its best. ↩︎

Comment via email

Andrey Listopadov

Clojure on Fennel part four: Parsing (again)

Old parser and compiler

New parser reader

Further work

Better slopes in AABB collision systems

Slopes in bump.lua

Slope response function for bump.lua

Spring Lisp Game Jam 2026

Day 1

Day 2

Day 3

Days 4-7

SFX

Day 10 - Music

The game

Clojure on Fennel part three: parsing

Manual single-pass parser and compiler

Clojure parsers

Edamame

Rewrite-clj

Parcera

Grammar-based parser

The parser

Clojure on Fennel part two: immutable.fnl optimizations

Update

PUC Lua 5.5

LuaJIT

Conclusion

Clojure on Fennel part one: Persistent Data Structures

Proper persistent data structures

Persistent Hash Map

Persistent Vector

Persistent Red-Black Tree

Persistent List

Persistent Queue

ClojureFnl

Implementing SSE with fnl-http and async.fnl

SSE stream

Reading the event stream and triggering events

Conclusion1

Implementing dynamic scope for Fennel and Lua

Implementing dynamic scope in Fennel

Usage example

So why not just set the globals temporarily?

Dynamic font-lock for Fennel

deps.fnl 0.2.2 released

Major changes

The deps.fnl file format

Conflict resolution

Conflict resolution is now module-based

Luarocks (transient) dependencies

Runtime library manipulation with deps.add-lib, and sync-deps

New flags

--tree

--fennel-ls

--profiles

Upcoming work

deps.fnl - a new dependency manager for Fennel projects

A few notes on why I decided to make a dependency manager

deps.fnl

Dependency types

I need feedback

Extending Emacs with Fennel

Running Fennel from Emacs Lisp

Data conversion

More examples

Running ELisp from Fennel?..

Fennelmacs

Extending JSON encoder with custom types

Custom JSON encoders

Custom decoders

Foontones

Streaming and receiving chunked multipart data with fnl-http

HTTP Server

Multipart requests

Performance

Interactive game development with LÖVE

Adding the REPL

Changing code at runtime

Slopes in `bump.lua`

Slope response function for `bump.lua`

Conclusion¹

The `deps.fnl` file format

Runtime library manipulation with `deps.add-lib`, and `sync-deps`

`--tree`

`--fennel-ls`

`--profiles`

`deps.fnl`

`package.loaded` - caching module definitions

`package.preload` - preloading modules via anonymous functions

`package.searchers` - extending the module system

`import-macros` and `require`

`init.fnl` and `init-macros.fnl`

Lua `debug` hooks