Andrey Listopadov

Clojure on Fennel part one: Persistent Data Structures

Tue, 07 Apr 2026 02:47:00 +0300

Somewhere in 2019 I started a project that aimed to bring some of Clojure features to Lua runtime - fennel-cljlib. It was a library for Fennel that implemented a basic subset of clojure.core namespace functions and macros. My goal was simple - I enjoy working with Clojure, but I don’t use it for hobby projects, so I wanted Fennel to feel more Clojure-like, besides what it already provides for that.

This library grew over the years, I implemented lazy sequences, added immutability, made a testing library, inspired by clojure.test and kaocha, and even made a port of clojure.core.async. It was a passion project, I almost never used it to write actual software. One notable exception is fenneldoc - a tool for documentation generation for Fennel libraries. And I haven’t seen anyone else use it for a serious project.

The reason for that is simple - it was an experiment. Corners were cut, and Fennel, being a Clojure-inspired lisp is not associated with functional programming the same way Clojure is. As a matter of fact, I wouldn’t recommend using this library for anything serious… yet.

Recently, however, I started a new project: ClojureFnl. This is a Clojure-to-Fennel compiler that uses fennel-cljlib as a foundation. It’s still in early days of development, but I’ve been working on it for a few months in private until I found a suitable way to make things work in March. As of this moment, it is capable of compiling most of .cljc files I threw at it, but running the compiled code is a different matter. I mean, it works to some degree, but the support for standard library is far from done.

;; Welcome to ClojureFnl REPL
;; ClojureFnl v0.0.1
;; Fennel 1.6.1 on PUC Lua 5.5
user=> (defn prime? [n]
         (not (some zero? (map #(rem n %) (range 2 n)))))
#<function: 0x89ba7c550>
user=> (for [x (range 3 33 2)
             :when (prime? x)]
         x)
(3 5 7 11 13 17 19 23 29 31)
user=>

However, there was a problem.

My initial implementation of immutable data structures in the itable library had a serious flaw. The whole library was a simple hack based on the copy-on-write approach and a bunch of Lua metatables to enforce immutability. As a result, all operations were extremely slow. It was fine as an experiment, but if I wanted to go further with ClojureFnl, I had to replace it. The same problem plagued immutableredblacktree.lua, an implementation of a copy-on-write red-black tree I made for sorted maps. It did a full copy of the tree each time it was modified.

For associative tables it wasn’t that big of a deal - usually maps contain a small amount of keys, and itable only copied levels that needed to be changed. So, if you had a map with, say, ten keys, and each of those keys contained another map with ten keys, adding, removing or updating a key in the outer map meant only copying these ten keys - not the whole nested map. I could do that reliably, because inner maps were immutable too.

But for arrays the story is usually quite different. Arrays often store a lot of indices, and rarely are nested (or at least not as often as maps). And copying arrays on every change quickly becomes expensive. I’ve mitigated some of the performance problems by implementing my version of transients, however the beauty of Clojure’s data structures is that they’re quite fast even without this optimization.

Proper persistent data structures

Clojure uses Persistent HAMT as a base for its hash maps and sets, and a bit-partitioned trie for vectors. For sorted maps and sets, Clojure uses an immutable red-black tree implementation, but as far as I know it’s not doing a full copy of the tree, and it also has structural sharing properties.

I started looking into existing implementations of HAMT for Lua:

hamt.lua (based on mattbierner/hamt)
- seemed incomplete
ltrie
- no transients
- no hashset
- no ordered map (expectable, different algorithm)
- no compound vector/hash
Michael-Keith Bernard’s gist
- no custom hashing

I could use one of those, notably ltrie seemed the most appropriate one, but given that I’m working on a fennel library that I want later to embed into my Clojure compiler I needed a library implemented in Fennel.

So I made my own library: immutable.fnl. This library features HAMT-hash maps, hash-sets, and vectors, as well as a better implementation of a persistent red-black tree, and lazy linked lists.

Persistent Hash Map

I started the implementation with a Persistent HAMT with native Lua hashing. The data structure itself is a Hash Array Mapped Trie (HAMT) with 16-factor branching. Thus all operations are O(Log16 N), which is effectively O(1) for a practical amount of keys.

As far as I know, Clojure uses branching factor of 32, but for a Lua runtime this would mean that the popcount would be more expensive, and despite a shallower tree, each mutation would need to copy a larger sparse array. With branching factor of 16 a map with 50K entries is ~4 levels deep, which would be ~3 with 32 branching factor. So my logic was that it’ll be a compromise, especially since Lua is not JVM when it comes to performance.

Of course, it’s not as fast as a pure Lua table, which is to be expected. Lua tables are implemented in C, use efficient hashing, and dynamically re-allocated based on key count. So for my implementation most operations are a lot slower, but the total time for an operation is still usable.

Here are some benchmarks:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Regular operations are notably slower, when compared to Lua:

Operation	Lua table	Persistent HashMap	Ratio	per op
insert 50000 random keys	2.05 ms	164.80 ms	80.3x slower	3.3 us
lookup 50000 random keys	0.83 ms	92.51 ms	110.8x slower	1.9 us
delete all	0.78 ms	170.78 ms	219.8x slower	3.4 us
delete 10%	0.14 ms	19.50 ms	136.4x slower	3.9 us
iterate 50000 entries	1.74 ms	6.64 ms	3.8x slower	0.133 us

For transients the situation is a bit better, but not by much:

Operation	Lua table	Transient HashMap	Ratio	per op
insert 50000 random keys	2.05 ms	89.17 ms	43.5x slower	1.8 us
delete all	0.76 ms	104.31 ms	138.0x slower	2.1 us
delete 10%	0.16 ms	12.71 ms	82.0x slower	2.5 us

On LuaJIT numbers may seem worse, but per-operation cost is much lower, it’s just that native table operations are so much faster:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	Persistent HashMap	Ratio	per op
insert 50000 random keys	0.86 ms	49.05 ms	56.8x slower	0.981 us
lookup 50000 random keys	0.27 ms	14.21 ms	53.4x slower	0.284 us
delete all	0.13 ms	48.63 ms	374.1x slower	0.973 us
delete 10%	0.05 ms	6.49 ms	138.1x slower	1.3 us
iterate 50000 entries	0.07 ms	1.80 ms	27.7x slower	0.036 us

Operation	Lua table	Transient HashMap	Ratio	per op
insert 50000 random keys	0.76 ms	22.43 ms	29.6x slower	0.449 us
delete all	0.15 ms	34.16 ms	232.4x slower	0.683 us
delete 10%	0.04 ms	5.02 ms	132.1x slower	1.0 us

With a branching factor of 32 the situation gets worse on PUC Lua, but is slightly better on LuaJIT. So there’s still space for fine-tuning.

For hashing strings and objects I decided to use djb2 algorithm. I am almost as old as this hash function, so seemed like a good fit. JK. The main reason to use it was that it can be implemented even if we don’t have any bit-wise operators, and Lua doesn’t have them in all of the versions. It only uses +, *, and % arithmetic operators, so can be done on any Lua version. It’s prone to collisions, and I try to mitigate that by randomizing it when the library is loaded.

Still, collisions do happen, but HAMT core ensures that they will still resolve correctly by implementing a deep equality function for most objects.

However, when first working on this, I noticed this:

>> (local hash-map (require :io.gitlab.andreyorst.immutable.PersistentHashMap))
nil
>> (local {: hash} (require :io.gitlab.andreyorst.immutable.impl.hash))
nil
>> (hash (hash-map :foo 1 :bar 2))
161272824
>> (hash {:foo 1 :bar 2})
161272824
>> (hash-map (hash-map :foo 1 :bar 2) 1 {:foo 1 :bar 2} 2)
{{:foo 1 :bar 2} 2}

This is an interesting loophole. What object ended up in our hash map as a key - our persistent map or plain Lua table? Well, that depends on insertion order:

>> (each [_ k (pairs (hash-map (hash-map :foo 1 :bar 2) 1 {:foo 1 :bar 2} 2))]
     (print (getmetatable k)))
IPersistentHashMap: 0x824d9b570
nil
>> (each [_ k (pairs (hash-map {:foo 1 :bar 2} 2 (hash-map :foo 1 :bar 2) 1))]
     (print (getmetatable k)))
nil
nil

To reiterate, I’m creating a hash map, with a key set to another persistent hash map, and then insert a plain Lua table with the same content. The Lua table hashes to exactly the same hash, and goes into the same bucket, but there’s no collision, because objects are equal by value. But equality of mutable collections is very loosely defined - it may be equal right now, but the next time you look at it, it’s different. So a different hashing was needed for persistent collections, to avoid these kinds of collision. I ended up salting persistent collections with their prototype address in memory.

Other than that, the HAMT implementation is by the book, and the rest is the interface for interacting with maps.

Main operations:

new - construct a new map of key value pairs
assoc - associate a key with a value
dissoc - remove key from the map
conj - universal method for association, much like in Clojure
contains - check if key is in the map
count - map size, constant time
get - get a key value from a map
keys - get a lazy list of keys
vals - get a lazy list of values
transient - convert a map to a transient

Coercion/conversion:

from - create a map from another object
to-table - convert a map to a Lua table
iterator - get an iterator to use in Lua loops

Transient operations:

assoc! - mutable assoc
dissoc! - mutable dissoc
persistent - convert back to persistent variant, and mark transient as completed

This covers most of the needs in my fennel-cljlib library, as anything besides it I can implement myself, or just adapt existing implementations.

A Persistent Hash Set is also available as a thin wrapper around PersistentHashMap with a few method changes.

A note on PersistentArrayMap.

In Clojure there is a second kind of maps that are ordered, not sorted, called a Persistent Array Map. They are used by default when defining a map with eight keys or less, like {:foo 1 :bar 2}. The idea is simple - for such a small map, a linear search through all keys is faster than with a HAMT-based map.

However, in my testing on the Lua runtime, there’s no benefit in this kind of a data structure, apart from it being an ordered variant. Lookup is slower, because of a custom equality function, which does deep comparison.

Persistent Vector

Persistent Vectors came next, and while the trie structure is similar to hash maps, vectors use direct index-based navigation instead of hashing, with a branching factor of 32. Unlike maps, vector arrays in the HAMT are more densely packed, and therefore a higher branching factor is better for performance. So lookup, update, and pop are O(log32 N), append can be considered O(1) amortized.

Still, compared to plain Lua sequential tables the performance is not as good:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Operation	Lua table	Persistent Vector	Ratio	per op
insert 50000 elements	0.19 ms	21.07 ms	109.7x slower	0.421 us
lookup 50000 random indices	0.47 ms	14.05 ms	29.7x slower	0.281 us
update 50000 random indices	0.32 ms	70.04 ms	221.6x slower	1.4 us
pop all 50000 elements	0.25 ms	24.34 ms	96.2x slower	0.487 us
iterate 50000 elements	0.63 ms	10.16 ms	16.2x slower	0.203 us

Operation	Lua table	Transient Vector	Ratio	per op
insert 50000 elements	0.19 ms	7.81 ms	40.3x slower	0.156 us
update 50000 random indices	0.33 ms	20.76 ms	62.4x slower	0.415 us
pop all 50000 elements	0.25 ms	11.14 ms	44.4x slower	0.223 us

On LuaJIT:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	Persistent Vector	Ratio	per op
insert 50000 elements	0.10 ms	7.62 ms	74.0x slower	0.152 us
lookup 50000 random indices	0.06 ms	0.67 ms	11.8x slower	0.013 us
update 50000 random indices	0.04 ms	29.13 ms	710.4x slower	0.583 us
pop all 50000 elements	0.02 ms	8.62 ms	410.4x slower	0.172 us
iterate 50000 elements	0.02 ms	0.57 ms	28.7x slower	0.011 us

Operation	Lua table	Transient Vector	Ratio	per op
insert 50000 elements	0.05 ms	0.59 ms	11.6x slower	0.012 us
update 50000 random indices	0.04 ms	2.06 ms	51.6x slower	0.041 us
pop all 50000 elements	0.02 ms	0.84 ms	46.7x slower	0.017 us

I think this is an OK performance still. Vectors don’t use hashing, instead it is a direct index traversal via bit-shifting, so there’s no hashing, just index math.

Operations on vectors include:

new - constructor
conj - append to the tail
assoc - change a value at given index
count - element count (constant time)
get - get value at given index
pop - remove last
transient - convert to a transient
subvec - create a slice of the vector in constant time

Transient operations:

assoc! - mutable assoc
conj! - mutable conj
pop! - mutable pop
persistent - convert back to persistent and finalize

Interop:

from - creates a vector from any other collection
iterator - returns an iterator for use in Lua loops
to-table - converts to a sequential Lua table

One notable difference in both vector and hash-map is that it allows nil to be used as a value (and as a key, in case of the hash-map). Vectors don’t have the same problem that Lua sequential tables have, where length is not well-defined if the table has holes in it.

It’s a debate for another time, whether allowing nil as a value (and especially as a key) is a good decision to make, but Clojure already made it for me. So for this project I decided to support it.

Persistent Red-Black Tree

For sorted maps and sorted sets I chose Okasaki’s insertion and Germane & Might’s deletion algorithms. Most of the knowledge I got from this amazing blog post by Matt Might.

I believe the operations are O(Log N), as for any binary tree, but given that the deletion algorithm is tricky, I’m not exactly sure:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Operation	Lua table	PersistentTreeMap	Ratio	per op
insert 50000 random keys	2.10 ms	209.23 ms	99.8x slower	4.2 us
lookup 50000 random keys	0.88 ms	82.97 ms	94.2x slower	1.7 us
delete all	0.74 ms	173.76 ms	234.8x slower	3.5 us

On LuaJIT:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	PersistentTreeMap	Ratio	per op
insert 50000 random keys	0.72 ms	101.08 ms	140.4x slower	2.0 us
lookup 50000 random keys	0.25 ms	12.67 ms	49.9x slower	0.253 us
delete all	0.14 ms	56.14 ms	403.9x slower	1.1 us

The API for sorted maps and sets is the same as to their hash counterparts with a small difference - no transients. Clojure doesn’t do them, and I’m not doing them too.

That’s all for benchmarks. I know that there are many problems with this kind of benchmarking, so take it with a grain of salt. Still, the results are far, far better than what I had with itable.

But there are two more data structures to talk about.

Persistent List

As I mentioned, I made a lazy persistent list implementation a while ago but it had its problems and I couldn’t integrate that library with the current one well enough.

The main problem was that this library uses a single shared metatable per data structure, and the old implementation of lazy lists didn’t. This difference makes it hard to check whether the object is a table, hash-map, list, vector, set, etc. So I reimplemented them.

The reason for old implementation to use different metatables was because I decided to try the approach described in Reversing the technical interview post by Kyle Kingsbury (Aphyr). I know this post is more of a fun joke, but it actually makes sense to define linked lists like that in Lua.

See, tables are mutable, and you can’t do much about it. Closures, on the other hand are much harder to mutate - you can still do it via the debug module, but it’s hard, and it’s not always present. So storing head and tail in function closures was a deliberate choice.

However, it meant that I needed to somehow attach metadata to the function, to make it act like a data structure, and you can’t just use setmetatable on a function. Again, you can do debug.setmetatable but all function objects share the same metadata table. So, while you can do fancy things like this:

>> (fn comp [f g] (fn [...] (f (g ...))))
#<function: 0x7bdb320a0>
>> (debug.setmetatable (fn []) {:__add comp})
#<function: 0x7bd17f040>
>> ((+ string.reverse string.upper) "foo")
"OOF"

You can also notice, that our + overload applied to functions in the string module.

So instead, we use a table, and wrap it with a metatable that has a __call metamethod, essentially making our table act like a function. This, in turn means, that we have to create two tables per list node - one to give to the user, the other to set our __call and use it as a meta-table.

Convoluted, I know. It’s all in the past now - current implementation is a simple {:head 42 :tail {...}} table. Not sure what is worse.

But that meant that I had to rework how lazy lists worked, because previously it was just a metatable swap. Now list stores a “thunk”, that when called replaces itself in the node with the :head and :tail keys. Unless it’s an empty list, of course - in that case we swap the metatable to an empty list one.

So Lists have three metatables now:

IPersistentList
IPersistentList$Empty
IPersistentList$Lazy

Instead of god knows how many in the old implementation.

The list interface is also better now. Previously it was hardcoded how to construct a list from a data structure. Current implementation also hardcodes it, but also allows to build a list in a lazy way from an iterator.

This is better, because now a custom data structure that has weird iteration schema (like maps and sets in this library), we still can convert it to a list. A general case is just:

(PersistentList.from-iterator #(pairs data) (fn [_ v] v))

Meaning that we pass a function that will produce the iterator, and a function to capture values from that iterator. Reminds me of clojure transducers in some way.

Persistent Queue

And the final data structure - a persistent queue. Fast append at the end, and also fast remove from the front.

It’s done by holding two collections - a linked list at the front, and a persistent vector for the rear. So removing from the list is O(1), and appending to the vector is also pretty much O(1).

Interesting things start to happen when we exhaust the list part - we need to move vector’s contents into the list. It is done by calling PersistentList.from on the rear. And building a list out of a persistent vector is an O(1) operation as well! Well, because nothing happens, we simply create an iterator, and build the list in a lazy way. But since indexing the vector is essentially ~O(1), we can say that we still retain this property.

Or at least that’s how I reasoned about this - I’m not that good with time-complexity stuff.

ClojureFnl

That concludes part one about ClojureFnl.

I know that this post was not about ClojureFnl at all, but I had to fix my underlying implementation first. Now, that I have better data structures to build from, I can get back working on the compiler itself. So the next post will hopefully be about the compiler itself.

Unless I get distracted again.

Comment via email

Slopes in AABB collision systems

Fri, 09 Jan 2026 04:11:00 +0300

Recently (again, bored on a vacation), I started working on a game I’ve planned for a long time. I wasn’t satisfied with my existing implementations of a player controller¹, so I started working on a new one. After a bit of fiddling around, I came to something I’m satisfied with, for now, at least, but while working on it, I wanted to add something I haven’t done in any of my projects yet - I decided to add slopes to my game.

I’m no game developer, to be honest, it’s not even my hobby, although I enjoy the process to some extent. Game development offers immediate visual and interactive feedback, unlike many other areas in programming, and I like that. So, while working on this player controller and figuring out how to implement slopes, I’ve searched the web to see if someone already did something similar in a system I’m working with, but I didn’t manage to find anything. Maybe I simply wasn’t specific enough in my search requests, or didn’t search deep enough, and there’s actually a guide on how to do that somewhere on the web, I decided to write this post anyway.

AABB collisions

First, let’s discuss what AABB actually means.

For those unfamiliar with collision detection, there are many ways of checking if two objects collide. A lot of them, actually, each with its pros and cons. Some give very accurate collisions for objects of any shape, but are not real-time viable, others are real-time, but far less accurate. It’s an ongoing research, and new ways of detecting complex collisions as fast as possible appear every now and then.

I’m mainly developing games in either TIC-80 or LÖVE2D. LÖVE2D actually comes with a collision detection library, called Box2D. My games are sprite-based, and since I often use TIC-80, I can’t really commit to Box2D, so instead I opt for a simpler collision detection logic.

That’s where AABB comes - it’s an Axis-Sligned Bounding Box collision detection system:

Figure 1: Axis-aligned rectangles are quite easy to check for intersections, since all we need to know is their origin and dimensions.

The general idea is that if the bounding box of object A is completely to the left or to the right of object B, the objects don’t collide. Similarly, if the object A is above or below object B, there’s no collision either. And when all four of these situations are not true, then there’s a collision.

It’s a rather simple solution, but it can yield satisfying results. Implementing this by hand is trivial, but there’s a lot more to checking collisions - we need to make it quite optimized, and handle object interactions.

Thankfully, there’s an amazing pure Lua library called bump.lua, that implements these AABB checks and more. And it’s near perfect for sprite-based games, since everything is aligned to the grid, so that’s what I’m using in TIC-80 and Love2D. Perfect, until we want something other than a rectangle. For example, slopes.

Implementing slopes with `bump.lua`

To be honest, there’s not that much to it for implementing slopes in an AABB collision system. Actually, I could just give you the formula and be done with it. Even more, actually - I’m sure you can figure it out if you think about it for just a bit. But that’s not fun! So instead, I’ll walk you through from the beginning - creating a TIC project, adding bump.lua, loading entities to the world, adding collisions, simple physics, and finally, slopes.

But, if you want, you can skip directly to the slope implementation section, no worries!

TIC-80

TIC-80 is a small game engine that supports Lua as one of its main languages to develop games, along with other ones, like Fennel. I primarily do my projects in Fennel, as I like it a little bit more than Lua², but for this post, I chose Lua, as I want this to be a more general tutorial, applicable to a wider audience.

Even though TIC-80 is a fantasy computer, I believe it is capable of being an engine that you can use to make full-fledged games with. Two years ago, Balatro again proved that Love2D is fully suited for making massively successful games, without the need for something more, like Godot or Unity. I think TIC-80 can shine similarly, but it seems that far fewer people consider it as a viable alternative, even fewer than with Love2D.

The other thing I often think about is that games don’t have to be implemented with the best technology to be good games. I’ve played a few games on Game Boy Color and especially Game Boy Advance, and they were amazing even today. And these consoles were quite limited even for the time. Yet, people are still making games for them.

And that’s another reason why I want to write this post more as a tutorial on TIC-80, as it can maybe make you want to try it!

So let’s start by firing up TIC-80:

By default, TIC-80 starts you in a console-like environment. It’s a fantasy computer, after all.

By pressing Esc, you can go into the code editor:

You can also switch to a built-in sprite editor by pressing F2 or clicking on a ghost icon in the top bar:

There’s also a map editor, which we’ll look into a bit later, you can create sound effects with a built-in synthesizer and write music in the tracker! It’s a fully self-contained system that you can develop your games in, and not just games - people write music in it, create beautiful entries for Demo Scene contests, and more.

I won’t be using the inbuilt code editor, though, as I’m much more comfortable writing in Emacs, and you can use any external editor with TIC, but we’ll be using the sprite and map editors. So, let’s start!

Creating a project and adding `bump.lua`

Actually, if you did start the TIC-80 with me, you don’t have to create anything - by default, TIC creates a sample cart that you can edit right away. That’s what we’ll do. First, let’s edit the metadata at the top of the cart:

-- title:   AABB Slopes
-- author:  Andrey Listopadov
-- desc:    Making slopes wiht AABB collisions
-- site:    https://andreyor.st
-- license: MIT License
-- version: 0.1
-- script:  lua

We could also delete default sprites right now, but we’ll do it later. Right now, we need to add bump.lua to the cart.

TIC-80 doesn’t really provide access to your filesystem - it is expected for the cart to contain everything, much like game cartridges did in the early days of home consoles. Because of that, it’s not possible to install bump.lua via LuaRocks, or download it and place the file near your cart. We’ll have to embed it directly.

Luckily for us, it’s really easy to do so in Lua! All we need is to set package.preload.bump to a function that contains all of the library code. And even greater, that bump.lua is a simple, single-file, self-contained Lua library.

So we can do it like so:

package.preload.bump = function()
    -- Copy the bump.lua file contens here.
    -- Yes, as is.
end

Now, we can use require to bring this library to our game, and create the world:

local bump = require "bump"

local world = bump.newWorld(24)

Great! Now let’s draw some sprites we’ll use!

Creating sprites and using the map editor

There’s nothing much to talk about here, as TIC features a rather simplistic sprite editor:

However, we’ll be using some of its advanced features, which can be revealed by pressing this switch in the top left corner of the editor:

Figure 2: Advanced mode switch (red arrow), flags (green arrow)

One of such features is the flags we can set on a sprite. These flags act as a bit mask, so you can set up to 256 combinations of flags, which is enough, since TIC has space for exactly 256 sprites. Did I mention how cool the design of TIC is yet?

I’ve set bit 1 here to indicate that this sprite is a ball, and I set bit 0 for all of the solid objects drawn near it. This way, when we load these sprites to the game world, we’ve created with bump.lua, we will be able determine what this object should be.

Now, let’s use these sprites in the map editor:

In the map editor, you can choose from either 256 tiles or 256 sprites to draw your map with. TIC actually has two sprite-sets: tiles and sprites. The only distinction between them is that you’re limited to either of those sets when you design your map. Other than that, they’re the same.

I’ve created a small test scene, with our ball and some walls:

Now we can start implementing our game logic, as right now if we run this cart, nothing would happen yet.

Loading entities to the world

We’ll start by editing the TIC function in our standard cart:

function TIC()
    cls(0)
    map(0, 0, 31, 18, 0, 0, 0, 1)
end

Now, if we run our cart, we’ll see the same thing we’ve seen in the map editor:

However, nothing moves yet, as we haven’t implemented any of the game’s logic. Before we can do that, we’ll need to populate our world with rectangles, so let’s do that.

local function load_map()
    for x=0, 240 * 8, 8 do
        for y=0, 136 * 8, 8 do
            local obj = is_obj(x, y)
            if obj then
                world:add(obj, obj.x, obj.y, obj.w, obj.h)
end end end end

Here, I’m loading the entire map to the world. It’s not necessary to do that, since we’re only going to use one map screen, but it doesn’t hurt. The key function here is the is_obj predicate, which we’re going to write next:

local balls = {}

local function is_obj(x, y)
    local id = id_at(x, y)
    if is_solid(id) then
        return {x = x, y = y, h = 8, w = 8}
    elseif is_ball(id) then
        local ball = {
            x = x,
            y = y,
            h = 8,
            w = 8,
            xvel = 0,
            yvel = 0,
            is_ball = true
        }
        balls[ball] = true
        return ball
end end

First things first, we need to get the ID of the tile at coordinates x and y. We do so by using TIC’s built-in function mget:

local function id_at(x, y)
    return mget(x // 8, y // 8)
end

Next, we’ll define a set of predicates that, given an id will tell if the object we’re inspecting is a wall, or a ball:

local function is_solid(id)
    return fget(id, 0)
end

local function is_ball(id)
    return fget(id, 1)
end

So the is_obj function we’ve defined above gets the tile id, and checks if it is a ball, or a wall tile. If so, it creates an object that we can add to the bump.lua world. Each object has a set amount of required properties, but you can also have additional ones, like the is_ball field, in the case of the ball object. We also add a ball to the list of balls, in case we might want to add more than one ball later.

Now, we can load the map. We do so in the BOOT function, which executes when the cart is being loaded. You can also write the call at the top level if you want, and it may make the code a little bit more portable, as older versions of TIC didn’t have BOOT, but I’m not targeting those, so:

function BOOT()
    load_map()
end

With that in place, we can implement some simple physics for the ball.

Working with `bump.lua`

Since we’ve loaded everything into the world we’ve created with bump.lua, we can now manipulate things in it. We’ll be mainly manipulating balls, so let’s add some methods to the balls table:

setmetatable(
    balls,
    {
        __index = {
            update = function(self)
                for ball in pairs(self) do
                    ball.yvel = lerp(ball.yvel, 5, 0.1)
                    ball.xvel = lerp(ball.xvel, 0, 0.1)
                    ball.x, ball.y, cols, len = world:move(ball, ball.x + ball.xvel, ball.y + ball.yvel)
                    for i = 1, len do
                        local col = cols[i]
                        if col.normal.y == -1 then
                            ball.yvel = -(ball.yvel * 0.9)
                        end
                        if col.normal.x ~= 0 then
                            ball.xvel = -(ball.xvel * 0.9)
            end end end end,
            draw = function(self)
                for ball in pairs(self) do
                    spr(1, ball.x, ball.y, 0)
            end end
        }
    }
)

I’ve added two methods: draw and update. It’s not necessary to separate these, you could do everything in one call, but it’s a bit easier to reason about them this way. Yeah, I know, looping over all of the balls two times just to update each, and then draw each is bad design, since we could do it in one loop, but that’s not the main focus here. But it’s great that you’ve pointed this out! Ten points to the TIC house!

Anyway, we’re looping over all of the balls in the table, which we don’t have that many yet, only one, and updating each one. The important part here is that we compute the ball’s X and Y velocity, and then use them to move the ball in the world, by calling world:move(). I’m using linear interpolation, defined as follows:

local function lerp(a, b, t)
    return ((1 - t) * a) + (t * b)
end

We don’t care that much about delta-time here, as TIC runs at a consistent 60 frames per second even when you push it to the limits. You can, of course, slow it down with bad code, and we already have two loops instead of one, which is concerning, but we’ll be fine, don’t worry. Lua is quite fast.

ball.x, ball.y, cols, len = world:move(ball, ball.x + ball.xvel, ball.y + ball.yvel)

So by calling world:move() with the ball object, we receive new X and Y positions of that ball after it was moved with respect to collisions. We’re not going to build a sophisticated physics engine here, but we will need to handle some collisions. bump.lua actually has different collision types, like bounce, that we could use here, but for our purposes the default slide is better.

In addition to the new X and Y positions, this function returns a list of collisions that happened during the move. We can loop over those and update our ball’s properties, such as its velocity. We do so in the following loop³:

for i = 1, len do
    local col = cols[i]
    if col.normal.y == -1 then
        ball.yvel = -(ball.yvel * 0.9)
    end
    if col.normal.x ~= 0 then
        ball.xvel = -(ball.xvel * 0.9)
end end

We’re slowly decreasing the ball’s velocity, also switching directions on contact. That’s pretty much what the bounce collision would do for us.

For the draw method, we simply draw the ball sprite using another TIC’s builtin function spr.

Having these implemented, we can add them to the TIC function, like so:

function TIC()
    cls(0)
    map(0, 0, 31, 18, 0, 0, 0, 1)
    balls:update()
    balls:draw()
end

Now, if we run our game, we’ll see our ball bouncing:

Your browser does not support the tag

Uh oh, why are there two balls, and one floating over the ground? Did we forget to add it to the balls table, so it gets updated? If we did, why is it even drawn - we loop over balls in both functions.

If you look closer, you’ll see the problem. We’ve loaded the map to the world, and we’re drawing the ball using the spr function. But we’re still drawing the original map in the TIC function by calling map! And our map still has the ball.

We can go two ways here. One would be to use the mset function and replace the ball tile with nothing in our map when we’re loading it. Or, we can hide the ball at runtime using map function’s special remapping feature. Let’s do that instead, as it is a bit cleaner:

local function remap(id)
    if is_ball(id) then
        return 0
    end
    return id
end

Now, we update our TIC function to use this remap function in our map call:

function TIC()
    cls(0)
    map(0, 0, 31, 18, 0, 0, 0, 1, remap)
    balls:update()
    balls:draw()
end

And, presto!

Your browser does not support the tag

Handling mouse events

Our introduction to TIC isn’t over yet, because I always wanted to try using the mouse integration. Our ball is currently pretty stiff, so we’ll tweak its physics along the way. Having mouse support will make it easier to test things out, as doing so by updating the code is a bit cumbersome. And I don’t know how to make sensible controls for a ball with arrow keys - it’s just a bouncing ball, what do you expect it to do, roll by itself when you press a magic key?? Let’s drag the balls instead.

We’ll start by creating a small storage variable that we’ll assign to the currently grabbed ball:

local grabbed = nil

It’s uninitialized by default, we’ll set it later in the update method:

local mx, my, is_pressed = mouse()
if is_pressed and not grabbed then
    local items = world:queryPoint(mx, my, function(obj) return obj.is_ball end)
    if next(items) then
        grabbed = items[1]
    end
elseif not is_pressed then
    if grabbed then
        grabbed.xvel = (grabbed.x - grabbed.prev_x)
        grabbed.yvel = (grabbed.y - grabbed.prev_y)
        grabbed.prev_x, grabbed.prev_y = nil, nil
    end
    grabbed = nil
end

We add this before the main loop over all balls in the update function.

The mouse function in TIC returns a set of values, X and Y coordinates, and states of each mouse button. We’re only interested in the left mouse button, so that’s what we store.

Then, if the button is pressed, we use a method from the Intermediate API section of bump.lua. It returns a list of items, and since we don’t have anything that overlaps, we can safely take the first item in the list. We’re also using a similar feature to the map function’s remap here - we’re providing a filtering function that will make sure that we can only grab ball objects, not other tiles.

When the button is released, its state will change, and we need to release the ball. If there was a ball grabbed, we set its velocity to conserve the momentum from moving it by the mouse, which we’ll handle later. Then we unset the grabbed storage variable.

Next, in the for loop responsible for updating the balls, we check if the current ball is the one we’ve grabbed, and we override its movement:

for ball in pairs(self) do
    if ball == grabbed then
        local x, y = lerp(grabbed.x, mx-4, 0.3), lerp(grabbed.y, my-4, 0.3)
        grabbed.prev_x, grabbed.prev_y = grabbed.x, grabbed.y
        ball.x, ball.y = world:move(ball, x, y)
    else
        -- no changes to the old logic
end end

So, as you can see, we’re moving the ball to the mouse pointer coordinates, still with respect to collisions. We also store additional information in the grabbed object, mainly the previous X and Y positions that we use in that momentum conservation code a bit earlier. And with that, we can drag the ball around:

Your browser does not support the tag

This is as basic as it gets - you’ll notice that if we add more balls, we can’t move other balls with the one we’re dragging. You can, of course, make a more sophisticated physics simulation - handle collisions better, so it would move other balls as well, transfer the momentum to other objects, make balls roll off each other, as right now they’re basically squares, and so on.

So you can do something like that if you try experimenting with collision resolution a bit more:

Your browser does not support the tag

Now, we’re pretty much set for working towards the main topic of this post - slopes.

Implementing slopes with AABB collisions

Phew, FINALLY, all that just to tell you that to handle slopes you just need to do…

Figure 3: …is this.

OK, this is anticlimactic, so let’s implement this. But first, what is a slope in our case?

Because we can only have axis-aligned rectangles, we need to create a special kind of object in our world that will still be a rectangle, but handle collisions differently. Since our game is tile-based, we don’t have a lot of meaningful angles to support. The most common slope in games is a segment with dimensions of 16x8. It spans exactly two tiles and takes you one tile height up. Let’s draw it:

You can see that I’ve set bit 2 to the slope tile here, so we can correctly handle it while loading our map to the world. If we add these to the world right now, they won’t work, obviously, since these are regular rectangles:

Your browser does not support the tag

So let’s change that:

local function is_up_slope(id)
    return fget(id, 2)
end

local function is_down_slope(id)
    return fget(id, 3)
end

We’ll start by defining two more predicates, and then use them in our is_obj function:

local function is_obj(x, y)
    local id = id_at(x, y)
    if is_solid(id) then
        return {x = x, y = y, h = 8, w = 8}
    elseif is_up_slope(id) then
        return {x = x, y = y, h = 8, w = 16, slope = slope, dir = 1}
    elseif is_down_slope(id) then
        return {x = x, y = y, h = 8, w = 16, slope = slope, dir = -1}
    elseif is_ball(id) then
        local ball = {
            x = x,
            y = y,
            h = 8,
            w = 8,
            xvel = 0,
            yvel = 0,
            is_ball = true
        }
        balls[ball] = true
        return ball
end end

As you can see, these are regular rectangles, except in our world, the 8x8 tile creates a 16x8 rectangle. The other half of the slope is just decoration.

We also have the slope and dir fields here. The slope is a function responsible for handling collisions that we’ll implement shortly. The dir field determines if this is an up-slope, raising from left to right, or a down-slope, lowering from left to right (or raising from right to left). We need to know the direction, otherwise, we won’t be able to calculate normals and properly map coordinates to new values.

Calculating slope collision coordinates

Here’s the slope function:

local function slope(obj)
    if obj.yvel >= 0 then
        local _, _, cols, len = world:check(obj, obj.x, obj.y + obj.yvel, slope_filter)
        for i=1, len do
            local col = cols[i]
            if col.normal.y ~= 1 then -- -1 or 0 will do
                local slope_floor = col.other.y - col.other.h
                local delta = clamp(0, (obj.x + obj.w / 2) - col.other.x, col.other.w) / col.other.w
                local y_off = nil
                if col.other.dir > 0 then
                    y_off = lerp(col.other.h, 0, delta)
                else
                    y_off = lerp(0, col.other.h, delta)
                end
                local newy = slope_floor + y_off
                return math.floor(newy), slope_normal(col.other)
end end end end

Now, that’s a lot to take in. Let’s go through step by step.

First, we check if the object’s velocity is going at least downwards. We don’t want to trigger slope collisions when objects move away from the slope, so that’s pretty simple.

Next, as usual, we need to handle collisions. However, there’s a catch - since our slopes are rectangles in disguise, we can’t just use the normal world:move() and be done. Instead, we need to kinda implement the world:move() in terms of world:check() and world:update().

The check method does the same stuff as move except it doesn’t move anything. It pretends that the object has moved, and returns a list of collisions and coordinates where it would end up if it actually moved. That’s perfect for our needs. The update method can move the object around without checking for any collisions, and that’s the other piece of the puzzle.

But right now, the ball collides with the slope rectangle, and it can’t go further, what can we do? We use filtering, of course.

The world:check() also accepts the collision filter, as world:move() does. We’ll need to add two new filter functions, one for checking slopes specifically, and the other to allow balls to move through slope rectangles freely:

local function slope_filter(_, other)
    if other.slope then
        return "cross"
end end

local function ball_filter(_, other)
    if other.slope then
        return "cross"
    else
        return "slide"
end end

With that out of the way, we can go into collision checking.

As usual, we iterate over a list of collisions. We then check if the collision’s Y component of the normal vector points at least up. If so, we have our collision, but we are just getting started.

Next, we calculate the slope floor Y position. This is the lowest coordinate our collision could possibly have:

local slope_floor = col.other.y - col.other.h

Next, we calculate the delta:

local delta = clamp(0, (obj.x + obj.w / 2) - col.other.x, col.other.w) / col.other.w

This is how far from the left edge of the slope rectangle the collision is going to happen. We try to measure in the center of the colliding tile, and normalize it to be a value from 0 to 1.

Next, we need to determine the Y coordinate on that slope:

local y_off = nil
if col.other.dir > 0 then
    y_off = lerp(col.other.h, 0, delta)
else
    y_off = lerp(0, col.other.h, delta)
end

We do so by checking the slope direction, and do a linear interpolation between 0 and slope height, using our delta value. Linear interpolation will move us in a straight diagonal line from the bottom corner of the slope to the top corner of the slope. Direction is simply used to flip the interpolation.

Finally, we’re calculating a ney Y position, and normals to return:

local newy = slope_floor + y_off
return math.floor(newy), slope_normal(col.other)

Our Y position from lerp is relative to the slope, so we add back the slope_floor position to translate it back to world coordinates.

And that’s pretty much just a small part of it! Wait…

Handling slope collisions

We still need to adjust the object’s position to that coordinate - why didn’t we do so in the slope function itself? Well, because reasons…

I mean, I tried to do that, but it didn’t work well, there are many outside factors that we may need to consider, and it’s not the best way to move it to this function. So let’s change our update method of the balls table.

First, let’s handle normal ball collisions. Mouse collisions will come later:

for ball in pairs(self) do
    if ball == grabbed then
        --- 8< ---
    else
        ball.yvel = lerp(ball.yvel, 5, 0.075)
        ball.xvel = lerp(ball.xvel, 0, 0.03)
        ball.x, ball.y, cols, len = world:move(ball, ball.x + ball.xvel, ball.y + ball.yvel, ball_filter)
        for i = 1, len do
            local col = cols[i]
            if col.other.slope then
                local newy, xnorm, ynorm = col.other.slope(ball)
                if newy and ball.y >= newy then
                    world:update(ball, ball.x, newy, ball.w, ball.h)
                    ball.y = newy
                    ball.xvel = ball.xvel + xnorm
                    ball.yvel = (- ball.yvel - ynorm) * 0.5
                    break
                end
            else
                if col.normal.y ~= 0 then
                    ball.yvel = -(ball.yvel * 0.9)
                    ball.xvel = ball.xvel * 0.75
                end
                if col.normal.x ~= 0 then
                    ball.xvel = -(ball.xvel * 0.9)
end end end end end

Code Snippet 1: Imagine if I didn't stack the end keywords on the same line? This post would be even longer! It's already a bit too long…

Main differences from before: we added ball_filter to the world:move() call, and split collision handling logic into two separate branches. The first branch checks if the collision happens with a slope and handles it. The second branch is the same as before.

The collision handling is pretty simple, really - we check if slope actually returned a new coordinate, and if the ball’s current Y position is lower than that, we need to handle the collision. The ball_filter sets slope collisions to cross, and it allows balls to move through slope blocks as if they’re not there, except we still get collision information. So if the ball’s Y coordinate is less than the slope collision coordinate, it means that we need to push the ball out of the slope. We do it straight up, without concerns about the normal vector to the slope, but it is possible to do so (I’m just lazy).

Calling world:update() sets the ball to a new position, and we also update its fields to set the new coordinate, and update velocities. We also break from this loop, as we don’t really want to handle any other collision, since it could get messy.

Now, let’s do the same for mouse movement:

for ball in pairs(self) do
    if ball == grabbed then
        local x, y = lerp(ball.x, mx-4, 0.3), lerp(ball.y, my-4, 0.3)
        ball.prev_x, ball.prev_y = ball.x, ball.y
        ball.x, ball.y, cols, len = world:move(ball, x, y, ball_filter)
        for i = 1, len do
            local col = cols[i]
            if col.other.slope then
                local yvel = ball.yvel
                ball.yvel = 1 -- needed for the slope check to happen
                local newy = col.other.slope(ball)
                ball.yvel = yvel -- restore the original velocity
                if newy and ball.y >= newy then
                    world:update(ball, ball.x, newy, ball.w, ball.h)
                    ball.y = newy
        end end end
    else
        --- 8< ---
end end

It’s pretty similar to the one above, except we need to do a small trick.

Notice how we set the ball’s velocity to 1 right before the check? Since the ball is moved by the mouse, its velocity is not really updated. Don’t ask me how long I was debugging why the collision doesn’t happen…

And that’s pretty much it!

Your browser does not support the tag

We can also add a different kind of slope to show that this approach is generic enough:

Your browser does not support the tag

Conclusion

This is still, by no means, a proper physics simulation of slopes. Balls bounce off slopes in the wrong direction at certain angles, normals are ignored when handling collisions, but hey, this might not be a problem in your game at all!

For instance, in my original case, the player is the main object I need to be able to interact with slopes, and I have a pretty simplistic game-oriented physics. So I don’t need the player to jump in a weird way when it’s on a slope - jumps should remain predictable. There are no projectiles that bounce off walls either, and thus, this implementation works fine.

You can, of course, use these principles to implement a more sophisticated engine. And there’s still a lot of room for improvement, too.

For instance, the way we define slopes is quite limited. Sure, we can adjust the slope angle, but because we’re building large slopes from small building blocks, there’s a wonky collision at these blocks’ corners. Instead, we could define two points and programmatically generate map tiles along the slope angle up to its end, using predefined sprites. And for non-sprite-based games, it doesn’t make sense to limit slopes to certain angles - you can define arbitrary-sized rectangles, making slopes as steep as needed.

Alternatively, you can refactor this code and define slopes as mathematical functions. Thus, you’ll be able to create splines and have curved slopes. Normal vector calculation will be a bit tougher, though.

And actually, if you happen to know some blog posts on the matter, please send them! I’m quite interested.

If you want to experiment with the code, here’s a cart that you can load with TIC-80, or you can experiment online on TIC’s website:

Figure 4: That’s an actual game cartridge you can load into TIC-80!

That’s all from me. Hope you’ve enjoyed this write-up as much as I did, and see you next time!

Code responsible for controlling the player object - movement, abilities, general game logic ↩︎
Fennel is Lua, but it adds a few features on top, like macros that allow additional metaprogramming capabilities, and fixes a bunch of Lua’s shortcomings. ↩︎

This looks so much tighter in Fennel, gosh:

       (each [_ col ipairs(cols)]
         (when (= col.normal.y -1)
           (set ball.yvel = (- (* ball.yvel 0.9))))
         (when (not= col.normal.x 0)
           (set ball.xvel = (- (* ball.xvel 0.9)))))

↩︎

Comment via email

2025 Recap

Sun, 28 Dec 2025 12:28:00 +0300

This sure was a long year.

Work stuff

January began with bad news - the product I worked on was about to be closed. We weren’t sure how things would go, and a lot of people started to worry about their job security. And rightfully so, a few months later, the decision was made to reduce staff, and our team was on the list. A lot of people got laid off, only a few remain to support the project in a dormant state.

I was transferred to a different project, thankfully, but the company I was working for experienced a lot of management-related issues and transformations, so ultimately, I decided to leave this company in July. Not that I’m complaining, though, I worked there for five whole years, and was thinking about it for some time already - the events just motivated me to do so. I think five years is a lot of time, and I’ve achieved enough and learned a lot during that time. It was still kinda sad to leave, as there are still a lot of amazing people working there, but I needed a change.

So now I’m working at Health Samurai! Happy to continue my growth as a Clojure developer, and tap into other technologies I wasn’t able to on my previous job.

Music

In March, I released my very first original music:

I have never written music before, and I don’t have any musical training (besides two months in a music school when I was about ten years old), so I’m excited to dive deeper into this art field. I’ve also recorded two more covers.

Vicarious by TOOL:

CRAWL by VELTPUNCH:

Looking back, I think my music output this year wasn’t really that great - only three recordings, compared to five in 2024, but I was busy most of the time. Still a lot more than in 2023 (zero output that year, and any year before 2024 to be honest).

I’ve pretty much completed my pedalboard setup and started using it in place of digital effects when recording. This somewhat limits my sound, compared to the endless variety of effects available in the DAW, but I like to work with limitations. Here’s a video on that:

Gaming

I got a Game Boy Advance as a birthday gift this year! I have a vivid memory, how one of my classmates back in 2003 said that I’ll never get one - well well well, how the turntables…?

Played and completed on GBA:

Kirby: Nightmare in the Dream Land
Kirby and the Amazing Mirror
Rayman Advance
Final Fantasy VI (in progress)

Anyhow, I like retro gaming - I’ve played a bunch of games on my 3DS in the past, and they were great. Now I can play even older games, and let me tell you - playing them on original hardware is a completely different experience. Stripped of the convenience of emulation, like save-states and ergonomic controllers, these games do feel different.

I’m also thinking about trying game development for GBA. No plans on this yet, but this style and format of games is probably my favorite.

This year, I also played a lot of Switch games, namely Tunic, Moonscars, World of Goo 2, Dark Souls, Astral Chain, completed Kirby and the Forgotten Land with my now fiancée, and the Crypt of the Necrodancer. Also played Fire Emblem: Awakening and Super Mario World on my 3DS. Tunic is probably my game of the year this time, it’s purely amazing.

New hobbies

I’ve always said that programming is my hobby. However, this year was weird - I had almost no output as a programmer outside of my day job.

Sure, I worked on deps.fnl a bit, but that’s it. Looking at the list of published posts, the only other programming-related ones are about fennel-mode and string interpolation in Lua. Most of the others are just random thoughts and yapping.

Well, music is my other hobby, but I already talked about it here.

However, this year was interesting in that I started attending various workshops. Last year I tried forging, this year I tried pottery, mosaic, and Tiffany glass making:

Figure 1: Flourless panCakes. Text says: “Pancakes without damn flour is an omelette.”

Figure 2: The goose is made by me, and the kitty is made by my fiancée

Figure 3: I’ve tried to make a tall Japanese-like teacup. My fiancée made a small bowl

Figure 4: Tiffany glass. Mine on the right side, my fiancée’s on the left

I hope to see myself attending more such workshops in the next year!

Trips

Managed to take two trips this year. One to Thailand at the start of the year, and one to Dubai this month.

Both trips went smoothly, although we were fearing that we would have trouble leaving Dubai, because a flood occurred a day before our flight. But we’ve managed to fly back home with no major issues.

Thailand was fun, since it’s a first trip in a long time where we flew as a company of four people, and accidentally met a few more people who were in Thailand at the same time, as a pure coincidence. We’ve rented a car and gone to different places on our own. It was a blast.

My usual vacation routine is: sleep, eat, swim, eat, swim, eat, sleep. After seven days or so, I begin to feel extremely bored, so I start to want to return home. This Dubai trip was not like that at all. It’s already a bit too cold to swim in December, so instead we were going to different places, similar to how we did in Thailand.

Dubai is an interesting country. They have a lot of money and ambition, and manage to pull off a lot of interesting ideas regarding their infrastructure. Sadly, not all is great, as mentioned, there are occasional floods when heavy rain comes, rooftops are leaking, and traffic jams are a major problem even when it’s not raining. The weather is nice, the sites are amazing, and it was, in general, an interesting experience to visit a country with such high ambitions. The feats they managed to pull off with the construction of their human-made islands are nothing but astonishing.

Plans for 2026

I plan to continue my music journey, already have a few more covers planned, so stay tuned! I also want to write more original music, so I’m looking forward to experimenting with it.

As for this blog - I’m not really sure. Each year, as it seems, I have less and less to write about here.

This year was particularly slow for me because I didn’t work on that many projects, and thus, I don’t have any material to write about. I don’t see this as a problem, though - this blog is mainly for collecting my thoughts, and since I don’t have that many right now, I’m not feeling like forcing myself. However, if this trend continues, I may stop paying for my own domain and go back to using the GitLab pages domain directly. No concrete thoughts on this yet, but maybe something like that would happen eventually.

As for long-term projects, I continue experimenting with Fennel and Lua, so maybe there will be more posts on these topics. Currently, I’m trying to work out a better model for immutable tables in Lua, given that my current implementation has poor performance characteristics. Maybe, I’ll rework my fennel-cljlib project to be a bit more robust.

That’s all from me, I wish you a happy new year, and hope to see you here in 2026!

Comment via email

MacOS Music Players

Wed, 10 Dec 2025 06:04:00 +0300

A sequel to Linux Music Players.

I’ve been using Linux for the past 15 years and managed to solve almost all my problems with it during that time. But I had to switch to a Mac due to reasons related both to my work and my hobbies, so now I have to solve the same kind of problems, but in macOS.

I like music. No, I love music.

Maybe I’m not the biggest music listener in the world, and my tastes and music knowledge may seem quite limited. For someone who listens to almost everything, I still think that I love music. I enjoy listening to albums in many genres, but with all honesty, I don’t really research anything about people behind the music I’m listening to. This backfired at me a few times when I was interested in the artist, and after reading about them made me a lot less into their music. So I try to avoid this.

But that’s beside the point. The main aim of this post is again to compare various music players, but now for macOS. Since I don’t want to use two laptops at once, just to listen to music from the other one, I had to find a suitable alternative to my Linux music player of choice: Strawberry.

Since the previous post, I’ve pretty much settled on Strawberry and have been using it since. After a bit of tuning, the user experience is at least tolerable, and I was able to stop thinking about it much after I got accustomed to the way you’re supposed to interact with your music library.

Strawberry

Figure 1: Strawberry screenshot from the project web page

So let’s start with Strawberry. Unfortunately, Strawberry isn’t available on macOS, at least not through brew - the search for “strawberry” on brew points to a browser named “Strawberry”. I was looking into Strawberry’s GitHub page, but there’s no release artifact for macOS. Their official webpage has a link to macOS release, but it is behind a paywall. And I would gladly pay for it, but unfortunately, I can’t.

Clementine, which is what strawberry was forked from, is available through brew, but it is marked as “disabled”.

Audacious

Figure 2: Audacious screenshot from the project web page

I’ve tried Audacious before. It’s a neat little player, but I feel like I’m a bit too dumb to use it.

Again, I don’t really like playlists. I said this before, I say it again. So music players that are designed around playlists are not my cup of tea.

I understand that an album is essentially a playlist, but it’s so clumsy.

But I feel like repeating myself, so let’s look at some music players I didn’t talk about before.

Cog

Figure 3: Cog screenshot from the project GitHub repo

OK, so Cog is the first native macOS music player that I’ll talk about today. By native, I mean that this is not a Linux application running on macOS - instead it’s built for macOS specifically. I have my thoughts on this, mainly - if I like some os-specific software, and in some years I’ll have to switch from macOS, it’ll be a shame if I won’t be able to use it. So I tend to avoid stuff that is not cross-platform. Yes, there are benefits for supporting a single platform, don’t get me wrong, that’s just how I lived for the past 15 years.

So, Cog. It’s a nice music player.

Again, playlist-oriented. Not my cup of tea, but I can work with that in the absence of alternatives.

Unlike Audacious, Cog has a library tree, like Strawberry did. This is good, because I can just click on an album in the tree and listen to it. I would prefer if albums were represented as a grid, with beautiful album covers and the ability to group them by artist, but again, that’s for some reason not really popular in most standalone music players. Maybe I’m crazy to want this, but so few music players do that that I wonder if I’m in the wrong. At least, I can organize my library as a tree of directories, and the player doesn’t get in the way.

One thing that bothers me is that there seems to be some problem with detecting track numbers from the metadata. More often than not, tracks are ordered the same way files in the directory are. This is not bad - if anything it’s even better, as with this the file system fully controls the library structure. The problem is, not all of my library is that well organized, although I try to update it if I find these kinds of problems. And the library search could be better.

Another small hiccup is that there’s no integration with Lastfm. I’m not the biggest scrobbler, but I try to do so, and Strawberry was able to do that.

Music

Figure 4: Music screenshot from the guide page

The inbuilt music player.

Looking at the screenshot you can see that it has a grid of albums, I’ve talked about in the previous section on Cog. Good, right? I finally found it!

Well, no. This app sucks.

First of all, it doesn’t play FLAC, even though macOS can play FLAC files with the Preview program. So not having support for it in the default music player is bonkers.

Secondly, it’s really weird in how it treats your music library. I’ve mentioned how Cog, and most other music players for that matter, read your library from a directory and can use the directory structure to organize the library. Music app doesn’t do that. Instead, it organizes your library via file tags.

I’m having my library backed up on an external drive, so I don’t want a music player to touch my directory structure in any way. So using Music is out of the question.

VLC

VLC is one of the greatest things that happened in free software. It’s mainly a video player, but it actually plays music just fine too. Can’t say that this is a music player with a good user experience though.

I’ve tried VLC many times, and every time I couldn’t wrap my head around the UX. There were always some weird things.

For example, when I open VLC and navigate to my music library, it is presented as a tree view. However, there’s no way to expand a single node of that tree - you can either expand all, or double-click on it, and start playing. It plays the first album it finds, and it’s not what I usually want. Maybe that’s a bug, but what can I say?

On Android, for example, VLC is quite different - it has a grid view with albums, so I wonder why the desktop version doesn’t have that.

MPV

Figure 5: MPC screenshot from the project web page

Another video player.

I’ve used MPV as a music player for a long time in my Linux days, before finally switching to Strawberry. It does the job - you just drag a folder into its main window, and it plays the thing.

Minimal UI, basic controls, and good sound. What more can I ask for?

Well, less minimal UI would be nice.

I was considering using it, but after using a proper music player for some time, I don’t want to go back. It’s a hassle to open Finder, navigate to the library, then open MPV and drag a folder there.

Museeks

Figure 6: Museeks screenshot from the project GitHub repo

Like Cog, this player is playlist-oriented, but it also has an album view, although not as a grid, but as a list. I liked it, but it doesn’t have any settings presented to the user, as it seems, or at least I couldn’t find the settings.

Unfortunately, it didn’t manage to import all of my library. A lot of albums are missing in the album view, and it seems that there are issues with support for CUE files. Seriously, why is it so hard to support CUE? I’ve talked about it in my previous post on the matter, and it’s wild that this continues here.

Others

I’ll go over some more music players that I tried, or tried to try but couldn’t.

5K Player - initially was scared off by its webpage, but tried it through brew. Didn’t like it.
Musique - not available through brew, and I can’t install it from the App Store.
Vox - Paid subscription, playlist-oriented, audiophile-oriented. Not my cup of tea.
Swinsian 3 - Paid, doesn’t look that much different from Museeks.
Audirvana - Paid subscription, audiophile-oriented. Looks high-end, but I’m not really into that.

What’s interesting is that there are a lot of articles on the internet (this post will add one more to the pile) that feature various music players for macOS. A lot of them suggest video players, like Elmedia, iina, and the like. I found almost no music players with a grid view of your library and a simple experience, like pick an album, hit play, enjoy - most are playlist-oriented. I guess streaming services won?

I mean, in Linux, people usually tend to use local-first tools, so it’s more expectable to see music players that are either inbuilt, like GNOME Music, or external but in high variety. On macOS, it seems that the target audience is a bit different, so maybe this explains the far fewer macOS-native music players, as I imagine most of the users used iTunes before and then migrated to Apple Music. I can be wrong though, but my experience with macOS so far tells me that there are a lot more things people want your money for, as opposed to how things are on Linux.

I could, of course, try out more cross-platform players, but I covered many of them in my previous post. I didn’t want to go over all of them again, and my thoughts on them didn’t change for the most part, so you can read about them there if you want. This post turned out to be a lot shorter, but there wasn’t that much to talk about, unfortunately.

If you happen to know anything that is similar to GNOME Music UX-wise, but built native for macOS, with FLAC and CUE support - please let me know! That’d be a dream. For now, I settled on Cog - it was the best of the bunch.

Thanks for reading, and see you next year!

Comment via email

GNOME is better macOS than macOS

Sun, 23 Nov 2025 02:57:00 +0300

Recently, I bought my first-ever MacBook. I’ve spent some time with it, and I gotta say - despite all that hot garbage that is thrown at GNOME for being an OSX clone, GNOME does the job better than I’ve expected, and certainly better than Apple. In some areas, that is.

Good old days of Linux

A bit of a backstory.

I’ve been using GNUplusSlashLinux for more than fifteen years. Most of the time, I used GNOME, starting from GNOME2, moving to Unity maybe for two years, then GNOME Shell, then KDE Plasma 5 for another two years, and switched back to GNOME Shell again. I’m not mentioning some of my at most month-long endeavors to other DEs, like XFCE, or tiling WMs, because they never stuck with me. So I’ve been there for most releases of GNOME Shell, followed them closely, even used to run Ubuntu GNOME when GNOME Shell became a thing, until it became the default in Ubuntu once again. Though by that time, I had already moved from Ubuntu to a different distribution for a variety of reasons.

I wasn’t always satisfied by GNOME, and was a fair bit vocal about it in the past - even got myself banned from r/gnome subreddit, for shitting on it too much. That’s why I experimented with other desktop environments, particularly Unity and KDE.

Unity felt like a breath of fresh air after GNOME2, mostly because Canonical took years of people trying to make GNOME2 more OSX-like, and made decent steps in that direction. A global menu, HUD, blur, buttons on the left - you name it, it was there. Granted, it wasn’t an OSX clone - rather, it was its own thing, and I still remember Unity days fondly.

Figure 1: Honestly, this was the best desktop

I switched to GNOME Shell almost instantly as it was released as Ubuntu GNOME spin, and while it was a bit janky, Unity started to lose steam, and GNOME looked like a hot new thing.

Figure 2: Anyone else remember that wallpaper? GNOME sure has come a long way

I did, however, run Unity on my older PCs, as it was far less taxing on resources than early versions of GNOME3, but then it was discontinued, and long-awaited Unity 8 with Mir never became a thing. So, when I was fed up with GNOME being a resource hog, often crashing, and moving towards Wayland, which didn’t work as good as it was advertised, I decided to try KDE somewhere around 2018.

And boy, how wrong was I, thinking that KDE is a buggy mess that would drain my resources even worse than GNOME. KDE Plasma 5 was nothing like I imagined - it was fast, lightweight, and slick. As a fan of old Unity design and OSX looks, I’ve configured my plasma, but it wasn’t a full clone, as other people often did. Instead, I tried to keep Plasma look like Plasma, but feel like a mix between Unity and OSX:

Figure 3: To this day, I still think this was beautiful. I’ve spent a lot of time fiddling with icon spacings in the top panel, combining widgets, writing my own in QML, just to throw it all away, a few years later

I’ve run Plasma 5 for about two years, and while I was enjoying it, there still were a lot of jank, bugs, occasional crashes (although Plasma recovered from them beautifully), but GNOME also became a lot better. So I switched back to GNOME, and I’m still running it on my old laptop, where I’m writing this part of the text right now:

Figure 4: GNOME 42 (Note, I have a 2K display and use GNOME without scaling with enlarged fonts. Helps a lot with too big GNOME UI elements taking up space)

As you can see, there are no fancy extensions - everything is the same as in a stock Fedora Workstation, because it is a stock Fedora Workstation. And I love it.

Over the years, I had a love-hate relationship with GNOME, but after a while, I came to accept it as it is, and now it feels like a perfect desktop environment for me personally. That, of course, is thanks to GNOME developers actually trying to make GNOME better, and me buying into their vision of what an OS should look and feel like. There is still some jankiness to it, but I’m willing to overlook it at this point, because after years of searching, I couldn’t find anything that suits my workflow better. And I’m fine with that.

A disclaimer

Before I go crazy on bashing macOS, I need to make a disclosure - I AM BIASED. I’ve been using GNOME for a lot of time, and a lot of things that feel logical to me may not feel logical at all to others. Especially for macOS users.

I’m fully aware of that.

Now, with that outta the way, let’s begin.

A new laptop

As I mentioned, I’ve just got myself a MacBook. I did it because today’s laptop market is a bit odd. It’s hard to find a decent non-gaming laptop with good specs. I mean, there are Lenovo ThinkPads, ThinkBooks, and even IdeaPads that are decent, but I already owned an IdeaPad, and while it served me for six years and I liked it, I don’t know if I want another one. Other brands may offer similar hardware, but I didn’t investigate much.

One of the reasons is OS choices. You see, I have run Linux as my main OS since around 2008. But I still have to use Windows from time to time. During university years, most of the non-programming software was Windows-only. After university, I had to use Windows to do media stuff, like music recording and video production, or design.

Design is probably the only task Linux can handle without much of a hassle, given that there are decent programs for that. For instance, I don’t mind GIMP, Krita, or Inkscape for most of my image-related tasks. I can even do video editing in Blender (and I do), but for some reason, it just works better under Windows. Audio production - not a chance. You’ll find me dead in the ground before I figure out how to set up a realtime kernel and configure Jack. Not to mention, VSTs just don’t work because many are relying on Windows APIs, and patching them through Wine is not a path I would like to walk.

So I had to use Windows, constantly rebooting from Linux to it every time I wanted to record myself, or work on some video for my channel. And I also had to use the one that came with my laptop - 6 years ago, it was Windows 10.

My parents recently got a new laptop with Windows 11 on it, and my god, it is horrible. After using Linux for more than fifteen years, I can’t even imagine how Microsoft is still able to get away with it. Ads are everywhere - on the lockscreen, in the start menu, in notifications, etc. You can’t even activate it without connecting to the network and signing into a Microsoft account.

Oh, and updates. Fucking updates are being forced, while I’m waiting for the next 5 hours until my video finishes rendering, because my laptop is too old to render in 4K.

Yes, it’s possible to “debloat” Windows, remove most of the jank they added, disable automatic updates, and have a decent, clean experience. I just don’t want to do that. The whole point of an operating system, to me, is to stay out of the way while managing resources, processes, and making sure my shit is done without major hiccups.

Now, when Windows 10 support is ending, all we’re left with is Windows 11, and god knows what bullshit awaits us in Windows 12.

So I thought to myself: “Ok, I like Linux, but I want to be able to do my media-related stuff, and I don’t want to invest in making Linux be able to do so. What options do I have other than that?” I just applied to a new job, and almost everyone there uses a Mac, and many of them are musicians, either hobbyists like myself or even professionals. After asking around, it appeared to me that switching to macOS could be the solution I’m looking for. It’s a Unix, so I guess my Linux habits won’t have to go through WSL hoops, and it has world-class support for media tasks. I think macOS can even be considered media-first when it comes to production, unlike Linux or Windows.

So I saved some money and got myself a new MacBook Pro, with M4, and decent specs. While the laptop itself is nice - I like the aluminium body, keyboard is nice (although I had an opportunity of using their butterfly one, and liked it better), the screen is gorgeous, and even the laptop speakers are great, but…

…but the OS…

macOS

This is the most counter-intuitive, user-unfriendly, confusing piece of software that I’ve used in my life. And I worked as a consultant in a cellphone store and used software for cashiers, yet it still was not as horrid.

I’m sure nothing I will write here will be new, and I’m committing an “internet crime” of “beating a dead horse” here, while it’s probably already reduced to vapors, but I still want to get it out of my system.

The Desktop

My laptop came with macOS 15 Sequoia preinstalled, so I haven’t used previous versions much, although I had used a Mac a bunch of times in the past when I was at the university ten plus years ago. However, I decided to immediately upgrade to the 26.1 Tahoe, so I wouldn’t get too used to the good-looking interface, and instead learn to get comfortable with this liquid glass crap. So I didn’t touch macOS for some years, and some things have changed, but the majority of annoyances I remember are still here.

A note on different desktops/WMs for macOS

I’ve started receiving some comments on this post, and I’d like to clarify that I’m aware of other window manager alternatives for macOS, like AeroSpace, Yabai and the like.

First, I’m not a fan of tiling window managers. Personally, I almost always have one window per desktop. Tiling never stuck with me. I’ve tried it several times during these fifteen plus years, and I don’t really feel the need for a keyboard-driven environment besides my Emacs. I mostly use touchpad gestures and the overview/mission control feature when I need to switch windows.

Secondly, I don’t really have that many issues with the current macOS WM, and I don’t really want to replace it. I was a GNOME user, remember?

Virtual desktops

As an ex-GNOME user, I like my virtual desktops. MacOS has had virtual desktops for a long time, but I think GNOME handled those better.

First of all, switching with the touchpad gesture feels a lot slower. In GNOME, it wasn’t an animation - the speed was tied directly to how fast you move your fingers, with easing after you release them. I got used to switching between desktops with a lot of speed, and on macOS, it’s not like that.

It seems it is also tied to the speed of the gesture, but it feels like there’s a cap on the maximum possible speed:

Your browser does not support the video tag.

An additional feature of GNOME is that it will automatically create additional virtual desktops once you’ve filled all of them, and there’s no limit to how many you can have. I usually have 4-5 desktops, but when I’m not working on something, it’s nice that I can only have one or two.

In macOS, you can have as many desktops as you want, however, they’re not added automatically. Or at least I couldn’t find a setting for that. Because of that, even if I’m not using all of them, sometimes a stray window is sitting on the last desktop, and I have to go through all of them to get to it.

Fullscreen and maximize button

For some reason, the developers decided that the maximize button should instead act like a fullscreen button in macOS. When you maximize a window, it is moved to its own, dedicated virtual desktop - and I like it. I have used this pattern in GNOME for a long time by installing an extension that brings this feature into GNOME.

However, and it is a big however, in GNOME, when I maximize a window, it creates a virtual desktop next to the one I’m on. In macOS, it also creates a virtual desktop, but it moves it to the far right.

So, imagine I’m browsing the web, seeing something interesting, and I want to open Emacs to take a note. I open Emacs, maximize it, and on the desktop to the left of Emacs, I still have my browser, and I can switch between these back and forth. In macOS, if I do this, the window is moved to a desktop at the far end of the list of desktops. You can rearrange those manually, but it’s clumsy. I would much prefer if the desktop were created to the right of the current one.

Figure 5: Emacs was opend on Desktop 1, but after maximizing it is now essentially at Desktop 5

You can prevent this, in some form, by turning on the “Automatically rearrange Spaces based on most recent use” in the settings. Then, a new fullscreen desktop is created to the right of the one you’re currently on, which is nice, however, then the spaces get rearranged all the time. Like, when you’re opening a link from the messenger, you switch to a desktop that has the browser open. The spaces get rearranged, such that the desktop with the messenger is next to the one with the browser, so you have to find the other one that previously was inbetween. It’s so bizarre that we can’t have fullscreen spaces created next to the current one without automatic rearrangement.

And, because it is a fullscreen mode, it also makes things go a bit haywire for some applications. Emacs doesn’t like this mode in particular.

Right now, I’m using four virtual desktops and manually organizing those, but maybe I need to adjust my usage pattern. Speaking of virtual desktops, let’s look at the Mission Control thing.

Mission Control

That’s another pain point for me. You see, GNOME also has something akin to Mission Control - you swipe up with three fingers, and you see all your open windows:

You can then move your windows between desktops or switch between them.

MacOS also has this, with the exact same gesture:

However, it is far less useful.

In GNOME, I can close windows in the Overview:

Figure 6: This close button is part of the Overview, not the window manager

I can tweak the dock:

Figure 7: Actually, the dock is only visible in Overview

I can start typing, and the search appears:

I can even swipe up again and bring up the applications menu:

Figure 8: Not that I use it that much though

There’s a lot of control! Wanna know how much you can control in the Mission Control?

You can move windows and desktops around.

…

That’s it!

What mission? What control? You can’t do anything.

Sure, one advantage macOS’ mission control has is that you can move entire desktops around - in GNOME, you can’t rearrange desktops themselves, only windows. And it plays nicely, with all this full-screen nonsense, but I still much prefer the GNOME approach here.

The Dock

Dock is a weird concept. It’s cool to have it, and I remember in my early Linux days I wanted a macOS-like dock so badly that I tried to make it out of existing panels and custom widgets. I installed various dock plugins in my desktop environments just to have it zoom in on icons when I hover over them. These days are long past.

Figure 9: remember Cairo dock?

Figure 10: remember Plank?

GNOME made a smart decision to only show the dock when you’re in Overview - their “Mission Control” variant. You don’t need the dock constantly taking vertical space from your windows, and you don’t have to deal with it showing up accidentally when you simply hover the mouse pointer at the bottom of the screen. Or at the side of the screen if it is your thing.

When it is only inside the Overview/Mission control, it makes sense as a favorites bar and a way of switching between applications. I haven’t used Alt+Tab (or ⌘ command+⭾ Tab) in years because of that. I would much prefer for the dock to be only visible in Mission Control, but sadly, there’s no way of doing it in macOS, as far as I can see. Moreover, Dock becomes useless in Mission Control specifically, so it won’t work that well in macOS.

What baffles me is that when you close an app, it stays in the dock, taking up space. Again, there seems to be no way of disabling it. Disabling the “Show suggested and recent apps in Dock” in system settings doesn’t affect this behavior.

You have to either use ⌘ Q, or right-click on the dock item and choose “close”. What’s the red button in the window for then? Usually, the app simply continues working in the background, as if minimized.

Speaking of which, so far only the minimize button seems to work in a sane manner.

Sure, there’s little point in actually closing applications, given that this laptop has a lot of RAM and CPU power, but it always felt weird to me that apps want to stay open all the time. I’m done with you - why do you think that you’re so important that you shouldn’t be closed? Do what you’ve been told.

The Finder

Linux has always had weird file managers. To me, nothing beats Microsoft’s Windows 7 file explorer - it was the least BS one in my opinion. GNOME’s Nautilus, or Files, as it is called nowadays, had a few long-standing problems, some of which date back ten plus years. KDE’s Dolphin was nice, but too had some weird quirks. However, it’s nothing in comparison to macOS Finder.

See the screenshot?

A normal file view, right? Now, what should happen if I shrink the window?

Your browser does not support the video tag.

The what? Who thought that this was a good idea? For those who didn’t get it, the grid of items remains the same despite the size of the window! Like, seriously?

The oddities don’t end there. Where’s the current path? For some reason, you can only see it when holding the ⌥ option, it appears at the bottom of the window. You can, in fact, toggle it in the settings so it is always shown, and while you can click on it to navigate, you can’t edit it, as it seems.

What really baffles me though is that it seems impossible to shift-select multiple files in grid view - only in list view. Even though, if you do shift-select files in list view, and switch back to the grid view, files stay selected. Seems to be related to how the media viewer works.

Another oddity is that when moving directories around, there’s no option to merge directories with the same name by default. I mean, when I move a directory to a different place, where another directory exists with the same name, I expect them to be merged, but the pop-up only says this:

When the ⌥ option key is pressed preemptively and held during the drag, the option to merge directories appears:

Like, you need to know, before the hand, that the target location you’ve chosen contains a folder with the same name you’re moving right now. And if you didn’t, you need to repeat the drag. Why not show it by default?

For the record, when the same thing is done in the GNOME file manager, it asks you if you want to merge directories, but you can’t replace them, so you can’t ever lose data. And if it contains files with the same names, it asks for each file if you want to replace it, or keep both. Of course, you can choose to replace all files, so it wouldn’t drag:

Cutting files is also super weird. You first copy the file as normal, then proceed to the directory you want to move the file, and press ⌘ command+⌥ option+v. Not only does the shortcut itself make your hand curl up as if you were a lobster, but also the semantics of this operation feel extremely weird - you copied the file, after all, why would it move afterwards? I guess, it’ handy in cases when you thought you wanted to copy the file, but you actually decided to move it, so you don’t need to input a whole new shortcut again, but it’s a minor win in my book.

Thankfully, I use graphical file managers rarely, so I can continue living in my comfort zone of Emacs’ DIRED.

Files and folders

But, while we’re at it, let’s talk about files in general.

I have some local music stored in the Music folder. Some videos and films are stored in the Videos folder. I also have pictures stored in the Pictures folder.

Crazy, right? Imagine using the file system to store files systematically?

I also try to keep things organized, so I manage all these with some directories, like band-name/album-name/track-name in case of music files, or year/place-name/photo in case of photos. I like it because if I want to listen to something, I can just drop the folder into the player, or open the image folder and relive those moments. Organizing these with folders is also handy for backing these things up - just drop the folder to the external drive and you’re good to go.

In macOS, as it seems, tags are the way the system wants to work with files. It’s a different approach, and I’m not entirely opposed to that, since filesystems are an illusion, after all.

But every once in a while, I open the Music folder, and I see a second Music folder inside it. Apparently, it is created when opening the Music app. I can’t use the Music app because it can’t play FLAC, which is what the majority of my local library is encoded in.

When I open the Videos folder, I often find a TV folder inside. I’m not sure what app creates it, but it wasn’t done by me, as I never launched the TV app. And it appears there far more regularly than the Music/Music one.

Other apps often create their own folders anywhere they want. The OrbStack app creates an OrbStack folder in $HOME. Some other app I installed created a folder in the Music folder for some reason, even though it didn’t store music in it, just some audio files.

It feels like that this is not my computer, but one I share with everyone else, and watch they do as they please with my filesystem. Never once was this a problem in Linux.

Why can’t apps keep their files to themselves? I’m tempted to create things like Music/My Music, or Pictures/My Pictures, and use them specifically, forgetting about the default Music, Pictures, Videos thing, but it is such a cumbersome solution. It feels like, when you invite friends to a party, and they go to your bookshelf, and decide that they’ll put some of their own stuff on it for the time being. And when they leave, they forget to take their stuff, but you don’t want to touch it, as it is not exactly yours. And you can’t throw it out, as your friends will be mad.

I don’t know.

The media viewer

Oh, the image viewer.

Let me ask you a question:

You have a directory with some photos. You want to go through these photos in the order taken and view them. What do you do?

In Linux, or Windows for that matter, the answer is simple: you double-click on the image you want to start with, the image viewer is opened, and you can use the arrow keys to go back and forth. Simple and effective!

What happens when you do it in macOS Finder? Well, the image viewer still opens, but the arrow keys do nothing! No, in order to go through images, you need to select the items you want to view, hover over the File menu in the top bar, press the ⌥ option key, and select “Start slideshow N items”:

HUH? Slideshow is cumbersome, as it is a slideshow - it’s not meant for going through images manually.

Alternatively, you can use a different viewer, called “Quick Look”, which supports arrow keys. Horray! But what is it? If you’re in grid view, why are the arrow keys following the grid too?

Your browser does not support the video tag.

HUUUUUH?

Who thought that this was a good idea? Hopefully, I won’t repeat this question all too many times.

The only reasonable way of using it is by switching to the list view and using the up and down arrow keys. Which is still not great, because Quick View is meant to be viewed temporarily. Any accidental click outside of it will close it, and you’ll have to find where you were and start from there.

You can put it into fullscreen mode, but then the arrow keys stop working. This seems like such a basic feature for an image viewer, but somehow they’ve managed to make it unusable. Not only that, but if this folder has any non-image files, like text files or PDFs, they’ll be included in the quick view too.

I understand that the purpose of Quick View is to view files quickly, and often it is a great thing to have, but at the same time, it’s super weird in behavior.

Hardware accessibility

OK, I can go on and on about the software part of the os, but to be honest, all of this is covered by other people rather well. And, in truth, I couldn’t be bothered to try all other preinstalled software, because it’ll take a lot of time to put it into this already long article, but also because I don’t want to. For instance, the Music app is also weird, and I already have an extensive post about how Linux music players are weird, so it goes without saying that I’ll need to replace it with something else. Like, the Quick View can play FLAC without any issues, but the Music app (formerly iTunes, I believe) can’t. Even GNOME’s player could do that, and it kinda copies Apple’s Music app in some aspects. Reencoding all my library into ALAC - Apple’s equivalent of FLAC for lossless audio is not an option for me. I did it once before when I was a happy owner of an old 4th-gen iPod touch, but never again - it is pointless, there are better music players out there.

So to me, the silver lining is that GNOME’s inbuilt software is in a lot of ways better than one in macOS. And KDE has even better software than GNOME in a lot of cases. So instead, let’s talk about hardware oddities and their operating system counterparts.

The Microphone

So, while I do care about my privacy, I can’t call myself a privacy maniac. However, one thing I like to have in my devices is the ability to mute the microphone. It’s best when there’s a hardware switch for that, but it’s so rare that I’ve only seen it once in my life on some Lenovo laptop.

I mean, it’s simply more convenient to do that than switch to a specific application that currently uses the microphone and toggle it there.

So, on Linux, I used the microphone key found on my laptop’s keyboard. It muted the microphone in the system settings.

When I got my Mac, I examined the keyboard, and found almost all usual shortcuts - you can change screen brightness on F1 and F2, you can change volume on F11 and F12, and there’s the microphone icon on F5. So I naturally assumed that this was a button to mute the microphone, after all, my old laptop had it too.

I assumed wrong.

There’s no way to disable the microphone in macOS via inbuilt means. The only thing you can do is to manually set the volume input gain level to 0, effectively making the microphone deaf.

Wanna know what the microphone button does then? I’ll show you:

Yup, it asks you if you want to use the speech-to-text feature. Cool, right?

And you can’t change what this button does.

You can set a different shortcut for dictation in the settings, but look what happens when you press the microphone key:

Your browser does not support the video tag.

It changes itself back to this button if you click “don’t ask again”! And then it asks you again once you press it! So what was the point of providing “don’t ask again” then?

Though I’ve managed to defeat this beast of a problem by writing some AppleScript. This script can be executed in the Shortcuts app, and by analyzing its output, we can rename the shortcut accordingly:

After adding this shortcut to the “Controls” menu, it acts as a mute button:

Your browser does not support the video tag.

Works well so far, and even remembers the current input level value, so I can still adjust it if needed. Even more, it works better than it did on Linux, because it constantly forgot what level the mic was set to and chose a default value of 20%, which was too low.

Here’s a text version of the script, if anyone needs it:

on run {input, parameters}
	try
		set currentVolume to input volume of (get volume settings) as number
		if currentVolume is 0 then
			try
				set storedVolume to do shell script "defaults read com.andreyorst.mictoggle volume"
			on error
				set storedVolume to 50
				do shell script "defaults write com.andreyorst.mictoggle volume -int " & storedVolume
			end try
			set volume input volume storedVolume
			return "Unmute"
		else
			do shell script "defaults write com.andreyorst.mictoggle volume -int " & currentVolume
			set volume input volume 0
			return "Mute"
		end if
	end try
	return "Error"
end run

I can’t share the whole script because no way I’m making an iCloud account for that.

The keyboard layout

Not strictly about the hardware part, since the keyboard itself is pretty good. It’s a bit of a shame that Apple stopped using their Butterfly switches, as I did like them more than the current scissor ones, but the keyboard is still better than the one on my old laptop.

One thing that bothers me, and which I got very accustomed to, is the fact that GNOME can remember keyboard layout per window:

MacOS can’t do that - and I don’t know what is the reason for that. The settings page for layouts has a setting called “Automatically switch to a document’s input source”:

However, I’m not sure what it does. It seems to work sometimes, but other times it does the opposite of what I want to. I only have two layouts. Is it that hard to keep track of that amount of data per window?

The keyboard layout (physical)

Though let’s touch the keyboard one more time. I know that Apple has at least two keyboard layouts for different markets - one for the US and one for Europe.

Figure 11: US layout

Figure 12: European layout

I specifically wanted the US layout, because all other laptops and standalone keyboards I had in my entire life had this kind of layout. Never once in my life have I seen the European layout on any device.

But, for some unknown reason, Apple sells its laptops with a European layout in my area, and when my wife had a Mac, it had that. Every time I used it, I was fighting my muscle memory.

Sure, it has more keys, and these contain some characters otherwise inaccessible without switching the layout, so it can look like Apple does a good thing. However, literally no one except Apple users had seen this layout, and it makes no sense to them. In my particular case, some already existing keys in the European layout are placed differently, like . and ?.

So, since I didn’t want to deal with that, I got myself a version with a US layout. Thankfully, there’s a “PC” layout in the system settings, so I don’t have to relearn anything. Using the “PC” layout on a European keyboard, however, is even more cumbersome.

No middle click on the touchpad

For some reason, there’s no such thing as a middle-click on a touchpad. In Linux, it is usually mapped to clicking with three fingers at the same time. I’ve been using it in Firefox a lot, and to paste stuff into the terminal once in a while.

However, in macOS, my muscle memory is now against me. I click with three fingers in Firefox, and usually it brings up the right-click menu, which is fine at least. But sometimes it is registered as a left-click instead, and it is super annoying. Re-learning to use @@html:<kbd>⌘ command</kbd> click will take some time, I guess

The camera cutout

Same as with the microphone, I’d like to have a physical way of blocking the camera, but because Apple moved towards a custom-shaped display matrix, it can’t be done as easily as before. It has a lot of sensors there, and blocking them with a huge plastic cover will make it impossible to close the laptop lid.

Again, not strictly about the hardware side of things, but the display in general feels weird because of this notch - it’s right in the middle of the screen, and I’m used to having the clock there, after years of using GNOME. Some say it’s more logical to have them at the far right side, as it is commonly done on phones, but phones have a much smaller and narrower screen. The same goes for notifications - having them in the middle feels more natural to me, because most of the content I’m working with is somewhere in the middle. But your opinion may vary (as if anything else I said here can be universally agreed on).

The good parts

OK, enough bitching around, let’s discuss some good parts.

The screen itself is gorgeous, colors are vivid yet natural, and the 14-inch model is actually both a bit bigger and a bit taller than my old 13-inch laptop. I like the screen a lot. I didn’t go for the nano-texture display, as I don’t work in environments where reflections are problematic, so I can’t say how it affects image quality. Maybe if I did, it’d be in the bad section.

The speakers are pretty good. And for a laptop, they’re actually amazing. They have depth, the upmp, and are loud enough to watch films with comfort - can’t say this about any other laptop I’ve had before.

Touchpad - oh, the touchpad. Simply put, the fact that it is a lot bigger (in fact, they had even bigger ones before), and I can press on it absolutely everywhere, is amazing. I was also surprised to learn that they aren’t actually pressed in physically. Instead, it is a glass surface, which has haptic feedback that simulates the click. Purely amazing. I’ve spent a long time trying to find anything remotely close to these touchpads on other laptops, but I never could find one.

M4 Pro CPU is a beast. Recently, I forgot a passphrase to one of my GPG keys and tried to brute-force it. My old laptop had AMD Ryzen™ 7 3750H, which isn’t fast by today’s standards, but it wasn’t a slow CPU either - it handled all of my tasks without major hiccups. I tried it first on the old laptop, and it could try about ~45 passphrases per second. The M4 could try ~750 passphrases per second. Everything is so fast I can’t believe it.

The macOS itself isn’t all that bad. As a matter of fact, writing this post helped me find how to fix a lot of problems I initially had - this post was a lot longer before that, but I had to cut stuff from it, as it was no longer applicable. There are still some oddities - it constantly bothers me with asking permission to run apps for the first time, because I install them through brew, or downloaded from the official website (again, I’m not making an Apple account, no, no). But I finally have a machine on which I can do both my programming tasks and media tasks without needing to keep two different operating systems to handle each task specifically. Finally, I can record my music with my right hand while writing code with the left - a dream.

Even the liquid glass thing is not as bad as I thought it would be. Thankfully, Apple added a tinted style to it, so it is less transparent, which helps readability.

And overall, I really like the device - it feels solid, and that it will last me another five years at minimum. Hopefully I didn’t jinx myself here, though.

Anyhow, thanks for reading, and please do share your experience with me, or if you know ways of solving the problems I’ve listed, I would be glad to know about them!

Comment via email

Replacing clojure-lsp with clj-kondo and Refactor-nREPL

Sun, 21 Sep 2025 21:52:00 +0300

I’ve been programming in Clojure for the last five years. I don’t write much about it here, largely because I use Clojure at work and rarely for hobby projects, so I don’t have much to share. Even today, the post will be more about Clojure tooling, rather than Clojure itself.

Today, most developers expect the language they work with to have a certain amount of specific tools, like a language server, build system, etc. These tools usually help work on a project with less friction. For instance, most languages require a language server to provide things like autocomplete, jumping to definition, and linting. However, when I do work on a hobby project, I usually prefer a more distraction-free environment. Count me old-school, but I kinda like the simplicity of just having you, your text editor, and the code. I don’t really use these things even with other languages.

Of course, Clojure itself doesn’t need such tooling that much, because there’s the REPL, which already acts like your language-specific tooling. I use Emacs, and it has excellent Clojure support thanks to nREPL and CIDER - it provides most of the features that a language server can, albeit in a bit different way. However, even with nREPL and CIDER, I don’t really use most of the features, maybe only the goto definition thing. Another thing provided by language servers is linting, but since Clojure is a dynamic language and it has a REPL, I already evaluate code all the time, so the REPL usually gives me all errors, and I don’t really need static linting that much either.

That goes for hobby projects. However, when working in a team on a big project, that’s a different story. For the last five years, I’ve been working on medium-sized projects, and the main difference for me is that there’s a lot of code I didn’t write myself. Language server helps here, because not only does it provide linting of such code, it also allows me to avoid loading all of the code into the REPL for CIDER-specific features to kick in. Also, not everyone in those projects used a language server, so every now and then the linter pointed out some potential problems.

Recently, I switched jobs, and now I work on a different team, on a much bigger project. My work setup didn’t change - basically, my Emacs config was ready as is to start working on a new project, or that’s what I thought. However, I noticed that things didn’t work out as planned.

I’m a firm believer that developers should use the slowest hardware they can get their hands on for developing. If your hardware is slow, you’re urged to write optimal code and, subsequently, create tools that work on that hardware. This is where things started to go haywire.

You see, this is a big project, with a lot of files. Usually, that’s not a problem, however, this time it was. In Clojure, we have Clojure-LSP as the language server to use. I have a pretty old laptop. Here are my specs:

Model	RAM	CPU	GPU
Lenovo IdeaPad S540	16 GB, 8GB ZRAM SWAP	AMD Ryzen™ 7 3750H	AMD Radeon™ Vega 10

It was plenty for anything I did in the past, for hobby projects, that is. At my previous workplace, I had a work laptop that had 24GB of RAM and a slightly faster CPU. Not that I really need an extra eight GB of RAM, I never saw it go above 16 while working on a single project. Sometimes I did work on several projects at the same time, and then it surely helped, but it was rarely the case.

At the current job, I hit SWAP all the time, and it started to bother me a lot lately. The reason is, as you might have guessed is Clojure-LSP. When started in the project I’m working on, it alone takes up around 8-11 GB of RAM. Right now, as I write this post, my editor, Firefox, and a messenger already take up 4.5 GB of RAM, so adding Clojure-LSP to this mix will by itself approach my limit. And then, I start the REPL, which took another 2-3 GB, and we’re in the SWAP territory.

I think it’s obvious that there’s no way my laptop can handle this load without becoming sluggish. RAM isn’t the only bottleneck here, CPU usage spikes up a lot, too, and the temperature is around 75 degrees and up constantly. So I decided to change this.

I’m not alone at this, unfortunately, my colleagues also suffer from Clojure-LSP being a resource hog. And the real problem is that they have even faster machines than mine with more RAM. So even if I upgrade my laptop or buy a new one, it won’t help that much. So, as an experiment, we decided to disable Clojure-LSP and go back to a simpler setup.

Disabling Clojure-LSP

I use the lsp-mode package, so disabling Clojure-LSP was as easy as commenting out the hook that starts the language server, but I went further and removed lsp-mode completely, as I don’t have use for it other than for Clojure. But now, I have no linting, which I’d like to have since this is a complex project. The go-to linter in Clojure world is clj-kondo, so I added flymake-kondor¹, since I use flymake and not flycheck. Fortunately, Clojure-LSP uses clj-kondo internally, so the linting configuration is the same.

I was expecting that linting such a big project would still eat a lot of RAM, however, for some reason, there wasn’t any major spike in RAM usage when just using clj-kondo. Linting works fast, and Emacs no longer freezes every now and then. I guess communication with the language server is much more taxing than simple parsing of stdout.

However, linting isn’t the only thing provided by Clojure-LSP.

Bringing back refactoring

Two main things I noticed I rely on with a language server are symbol renaming and finding references. Both of these tasks can be handled with refactor-nrepl, however, it’s a bit finicky.

Symbol renaming

Before we begin, let me tell you why you might want this feature as part of the tool that does code analysis. Sure, it’s one thing to rename a symbol inside a single namespace, but when you want to rename it across the project, it gets tricky.

One way of doing it is to use grep, and utilize the Emacs capabilities to edit the buffer created by grep directly. While it works, it’s not as precise as with a language server, because the symbol can be different depending on a file. For instance, when renaming a namespaced keyword, you can encounter a problem that the keyword is written as ::foo in the file you’re editing, but as :fully.qualified.namespace.name/foo or ::namespace-alias/foo depending on how the namespace was used. Sure, you can write a regular expression for grep in the first case, but not really for the second one.

Here’s an example: imagine we have several namespaces in our project (can be in different files, can be in a single file like here):

;;; src/project/multimethods.clj
(ns project.multimethods)

(defmulti m ::foo)

;;; src/project/utils.clj
(ns project.utils)

(defn foo [data]
  ,,,)

;;; src/project/other_ns.clj
(ns project.other-ns
  (:require project.multimethods
            [project.utils :refer [foo]]))

(defmethod project.multimethods/a :project.multimethods/foo [data]
  (foo data))

;;; src/project/another_ns.clj
(ns project.another-ns
  (:require [project.multimethods :as mm]))

(defmethod mm/b ::mm/foo [data]
  ,,,)

;;; src/project/unrelated.clj
(ns project.unrelated)

(defn foo [data]
  ,,,)

;;; src/project/yet_another_ns.clj
(ns project.yet-another-ns
  (:require [project.unrelated :as unrelated]))

(defn bar [data]
  (::unrelated/foo data))

It’s a bit verbose, but I tried to make as short of an example that shows all possible ways to use a symbol. Here, if you’re going to rename the ::foo keyword, you can’t really grep with "::foo|:project.multimethods/foo|::[^/]+/foo", because it will also find ::unrelated/foo, which is unrelated. Sure, you can write a pretty generic regular expression, and then meticulously find all relevant symbols in the grep buffer, and rename them using multiple cursors, or the query-replace feature, but it’s a bit much.

Same goes for renaming functions - if you’re trying to rename foo from project.utils, grepping can find foo in the project.unrelated namespace. So grep is not a suitable alternative to Clojure-LSP, as it isn’t capable of doing semantic analysis.

Since we’ve disabled Clojure LSP, we need a different tool to handle this task. Thankfully, there’s the refactor-nrepl project that provides refactoring features via the nREPL integration that we’re using. It has a lot of features, and it works well enough for our project. Well enough, because refactor-nrepl is a bit finicky.

Problems with `refactor-nrelp`

First of all, refactor-nrepl works. Most of the time, that is.

I think one of the reasons why Clojure LSP used so much RAM was that it did all of the possible analysis in the background. The reason I think it’s true is that it did the renaming almost instantaneously.

When renaming a symbol with refactor-nrepl, I often get a timeout error with CIDER - renaming is a blocking operation, and CIDER tries not to block the editor for too long in some cases. On a large project, renaming a symbol takes a lot of time to fetch all symbol occurrences alone. I guess that’s the trade-off. Perhaps I can configure the timeout to be a bit longer, but we’ll see.

Another feature of Clojure LSP I relied on was finding usages. Refactor nREPL gives that in the form of finding references, which, again, works, but is susceptible to the same timeout problem. And I’m not sure if it is as precise as Clojure LSP’s one. CIDER itself also has a feature for finding usages, but it is also finicky in its own way.

I suppose, because Clojure-LSP appeared, fewer people use refactor-nrepl today than it was before, and the overall advancement in CIDER development may have slowed down because of it. Looking at GitHub graphs, refactor-nrepl development started around 2015, and major activity stopped around 2019, precisely when Clojure-LSP was created.

Figure 1: refactor-nrepl

Figure 2: clojure-lsp

Which is a bit of a shame, if Clojure-LSP was responsible for slowing down advancement in refactor-nrepl, but thankfully it is still maintained and works. It’s good to have alternatives, and putting all eggs into one basket was never a good way of doing things.

With refactor-nrepl, most of the things I used Clojure LSP are again available to me, but I have to do further testing, because I found some instances where it fails. Maybe I need to configure a bunch of settings. Or maybe you’ll see another post with me going back to Clojure LSP in a few months/weeks. Who knows!

Not a bashing on Clojure LSP

Contrary to what this post may seem like, this was not intended as bashing on Clojure LSP developers. Clojure LSP is a good piece of tech, and certainly helps Clojure developers around the world.

The reason it is problematic to use on this particular project can be due to a lot of factors. First, the project is enormous, lsp-mode reports that it wants to “watch” for around 2300 directories. Disabling file watchers helps, but by a small margin.

Second, the tech stack. Clojure LSP is written in Clojure, which, while it makes sense, might not be the best choice. Clojure is known to be not the best tool for writing utilities. And while Clojure LSP is a server, and not a CLI utility with a fast lifecycle, it may still be sub-optimal. Maybe, once Jank is ready, Clojure LSP could be rewritten in it, making it faster. I don’t know.

A completely unrelated reason, in my opinion, is that Clojure itself is not the best target for LSP. It’s a dynamic language, where we do a lot at runtime in the REPL. Doing static analysis in such a system can be difficult, as there’s no longer one source of truth. nREPL, bridging runtime and source code, while also having a lot of LSP features, seems like a much better fit for languages like Clojure. And, like, nREPL has appeared almost six years before the Language Server Protocol, and in my opinion, it could have been a far better protocol for developing language tooling, especially since it is also language-agnostic.

Anyway, replacing Clojure LSP with plain clj-kondo brought back the joy of writing Clojure, so I’m happy again.

What’s up with the name? Why not just flymake-kondo? ↩︎

Comment via email

Implementing SSE with fnl-http and async.fnl

Wed, 30 Jul 2025 01:05:00 +0300

Some time ago, I was working on an HTTP library for Fennel. As a proof of concept, I added a module that implements a simple web server and wanted to experiment with it. The server can serve files from a directory using this simple handler:

(fn handler [{: method : path &as request}]
  (case method
    :GET (let [file (io.open (path:gsub "^/" "") :r)]
           (or file {:status 404 :body "not found"}))
    _ {:status 405
       :body "Method not allowed"}))

This handler is simple - when the client sends a GET request with a given path, for example GET http://localhost:8080/some/file.txt, the handler simply removes the leading / and tries to open the given path as a file handle. If the file is present, we return it as is, and the server will handle the closure of this file once the client disconnects. Otherwise, we return a table with the 404 status code, indicating that the file wasn’t found. Other methods are not allowed, and we respond with a 405 error by matching on method with case.

Now we can require the server module and start a small file-serving server:

(local server
  (require :io.gitlab.andreyorst.fnl-http.server))

(local s
  (server.start handler {:port 8080}))

(s:wait)

The (s:wait) call simply locks the current thread because, without it, the process will just exit.

This is, of course, the simplest web server there is; it just accesses files in the current directory, returning them as application/octet-stream. The handler function can be more sophisticated, handling different methods, parsing arguments, etc. For our purposes today, we won’t need most of these features, but I thought it would be better to familiarize you with how the server is set up.

The topic of today’s post is Server-Sent Events, or SSE for short. It is a protocol for communicating between a server and multiple clients that uses a persistent connection and pushes messages from the server to the client. The client can then process these messages at their own pace and react accordingly.

We’ll need an asynchronous library to handle SSE for multiple clients, so let’s add it to the mix:

(import-macros {: go : go-loop}
  :io.gitlab.andreyorst.async)

(local {: <! : >! : timeout : alts! : pub : sub : chan : close! &as async}
  (require :io.gitlab.andreyorst.async))

(local json
  (require :io.gitlab.andreyorst.json))

I’ve also included a JSON library; both are already used by fnl-http. Now, let’s begin!

SSE stream

We’ll send events to our clients via a channel:

(local event-ch (chan))

This event channel will be our main way to send events to the clients, but we won’t read from it directly. Instead, we can create a “publication” or pub that will allow clients to subscribe to the messages:

(local event-pub
  (pub event-ch #:sse #(async.sliding-buffer 100)))

For the purposes of this post, there’s only one type of message in this event-pub thing, but in a more complex scenario, you can separate messages and subscribe to different topics altogether. Speaking of subscribing, here’s a helper function:

(fn subscribe-to-events [client-ch]
  (sub event-pub :sse client-ch))

Finally, we need a function to register an event:

(fn register-event [event]
  (>! event-ch event))

Now the fun part. Right now, our event system is ready to be accessed by clients, but the events themselves don’t make any sense, as they’re just some objects. We’ll define events as tables with the following structure:

{:event "optional event type"
 :id    "optional event id"
 :data  "event data"}

Since our register-event function simply writes to the event-ch, we need a way to intercept that. We can do so by writing middleware that accepts a channel with events as tables and returns a new channel that will have all of the events formatted:

(fn wrap-format-sse-event [ch]
  (let [out (chan)]
    (go-loop [e (<! ch)]
      (case e
        {: event : id : data}
        (doto out
          (>! (.. "event: " (tostring event) "\n"))
          (>! (.. "id: " (tostring id) "\n"))
          (>! (.. "data: " (tostring data) "\n")))
        {: event : data}
        (doto out
          (>! (.. "event: " (tostring event) "\n"))
          (>! (.. "data: " (tostring data) "\n")))
        {: data}
        (doto out
          (>! (.. "data: " (tostring data) "\n"))))
      (if e
          (recur (<! ch))
          (close! out)))
    out))

Now, the general mechanism is as follows:

When a client connects, we create a channel and subscribe to events.
We wrap this channel with our formatter and some other middleware if needed.
Finally, we return this channel as the response body.

Before we write an HTTP handler that does this, we need to consider one other thing: the connection. If the connection is not used for a long time, it can be closed automatically. Because of that, it is common to send meaningless heartbeat events every now and then. Let’s write another wrapper that will track if the client’s event stream doesn’t have any events for some time and return a heartbeat:

(fn wrap-heartbeat [ch timeout-ms]
  (let [out (chan)]
    (go-loop []
      (let [tout (timeout timeout-ms)]
        (match (alts! [tout ch])
          [_ tout]
          (do (>! out {:event "heartbeat" :data 0})
              (recur))
          [event ch]
          (do (>! out event)
              (recur))
          [nil ch] (close! out))))
    out))

Similarly to our previous wrapper, we accept a channel with events and return a new channel. The asynchronous go-loop, which I didn’t mention before, will run in a separate virtual thread, trying to read from both the given channel and a timeout channel. Whichever returns first will determine what event will be sent to the client. If a normal event comes through, we send it to the out channel as is and restart the timer. If no events come and our timer has been exhausted, we send a heartbeat and restart the timer again. If our events channel returns nil, it means that it was closed, so we also close the out channel.

Finally, we can implement our handler:

(fn handler [{: path : method : headers : content}]
  (case path
    (where "/events" (= :GET method))
    (let [events (chan)]
      (subscribe-to-events events)
      {:status 200
       :headers {:content-type "text/event-stream"}
       :body (-> events
                 (wrap-heartbeat 10_000)
                 wrap-format-sse-event)})
    (where "/notify" (= :POST method))
    (case (tonumber headers.Content-Length)
      len (let [event (json.decode (content:read len))]
            (register-event event)
            {:status 200
             :body "OK"})
      _ (error "malformed Content-Lenght"))
    _
    {:status 405
     :headers {:content-type "application/json"}
     :body (json.encode {:error "Method not allowed"})}))

This handler has two main entry points: /events and /notify. The /events entry is for subscribing to the events stream with a GET request. The /notify entry is for sending events from the client to the server (because right now we don’t want to make the server too complicated to generate events on its own).

Same as before, we can start a server:

(local s
  (server.start handler {:port 12345}))

(s:wait)

Reading the event stream and triggering events

With the server up and running, we can connect a pair of clients.

The first client will subscribe to the event stream by executing a GET request:

(local http
  (require :io.gitlab.andreyorst.fnl-http.client))

(local resp
  (http.get "http://localhost:12345/events" {:as :stream}))

If we look at the response, we’ll see the headers containing text/event-stream as the content type, as per the response from our server, and the body as a stream, as requested by the client:

{:body #<Reader: 0x564ae85ca7d0>
 :headers {:Connection "keep-alive"
           :Content-Type "text/event-stream"
           :Transfer-Encoding "chunked"}
 :http-client #<tcp-client: 0x564ae812fa10>
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 38
 :status 200
 :trace-redirects {}}

We can consume events with a helper function:

(fn consume-events [stream]
  (fn read-line [stream buf]
    (let [c (stream:read 1)]
      (when (not= "\n" c)
        (table.insert buf c)
        (read-line stream buf))))
  (while true
    (let [buf []]
      (read-line stream buf)
      (io.write "[" (os.date) "]\t" (table.concat buf) "\n"))))

(consume-events resp.body)

In the process output, we’ll be able to see heartbeat events appearing every ten seconds:

[Wed Jul 30 00:34:35 2025]	event: heartbeat
[Wed Jul 30 00:34:45 2025]	data: 0
[Wed Jul 30 00:34:45 2025]	event: heartbeat
[Wed Jul 30 00:34:55 2025]	data: 0

Next, we can send a POST request from another process, sending an event:

(local http
  (require :io.gitlab.andreyorst.fnl-http.client))
(local json
  (require :io.gitlab.andreyorst.json))

(http.post "http://localhost:12345/notify"
           {:body (json.encode {:event "message" :id "client-2" :data "Hi from client-2"})})

We immediately see the event in the first client’s output, followed by heartbeat events:

[Wed Jul 30 00:35:03 2025]	event: message
[Wed Jul 30 00:35:03 2025]	id: client-2
[Wed Jul 30 00:35:03 2025]	data: Hi from client-2
[Wed Jul 30 00:35:03 2025]	event: heartbeat
[Wed Jul 30 00:35:13 2025]	data: 0

As you can see, the time when the heartbeat event arrived is exactly ten seconds after the last non-heartbeat event. Of course, we can connect multiple clients subscribed to the event stream, and they will all receive events.

Conclusion¹

This should give you an idea of how to implement an SSE stream in your server using the fnl-http library. Of course, this implementation is only loosely based on the specification, so I wouldn’t suggest it as a general implementation guide, but I hope it provides a clear enough explanation of how things work.

The server module is still very much in a proof-of-concept stage, but as this article shows, it is working quite well. All in all, I believe the ability to return an asynchronous channel as a body from the server is a nice way to handle this kind of event system. The code can be written as usual, and no specific handling for returning asynchronous data via HTTP is required.

Thanks for reading, and I hope you’ll find fnl-http interesting enough to try!

As much as I don’t like “Conclusion” as the last heading in an article, I can’t think of a different one here. ↩︎

Comment via email

Effects of Slow and Fast systems

Sat, 14 Jun 2025 04:22:00 +0300

Modern-day computing is fast. Incredibly fast! For most tasks, the feedback can be considered instantaneous. Even the slowest languages used in production, like Python, are considered fast enough for the majority of tasks, and if not, there are almost always faster alternatives you could choose from.

Modern-day CPUs are speedy boys indeed, but it’s not just CPUs that are great. RAM too is no longer a major bottleneck both in terms of size and speed, and even storage is incredibly fast. GPUs can do millions of tasks under a millisecond in parallel. Hardware these days is purely amazing.

Because it’s so great, software engineers can do their job much more comfortably, choosing languages with nicer abstractions, and quality-of-life features like automatic garbage collection. Sometimes, a small trade-off in execution speed is massively outweighed by the robustness that managed languages provide. I myself am a big fan of advanced virtual machines like the JVM and LuaJIT - they can be both incredibly fast and give you a lot of freedom of expression.

However, this wasn’t always the case. In the old days, computers were slower. Incredibly slower! Today’s CPU can process lots of instructions per second. The execution speed is usually measured in MIPS - Million Instructions Per Second. For instance, 2021 Intel Core i7 346,350 MIPS - three hundred forty-six billion three hundred fifty million instructions per second, running at 4.92 GHz frequency. In 1978 Intel 8086 could only process 330,000 instructions per second running at 5.000 MHz^source.

We all know this story, about Moore’s law, that the number of transistors in an integrated circuit doubles about every two years, and how it is slowed in around 2010. But while it was going strong, computers became faster and faster every few years. Computer efficiency is another thing that improved over the years:

Figure 1: Computer processing efficiency, measured as the power needed per million instructions per second (watts per MIPS)

Perhaps, you already understand, where I’m going with this. Hardware has gotten so much faster over the years we no longer have to wait for the majority of tasks to be completed. Consequently, the feedback loop for software development shortened dramatically. So today I want to talk about the implications of working with horribly slow and incredibly fast systems.

Fast systems

First, a word about the term “system” I’m using for this post. I’m not talking about operating systems, rather I use this term more broadly. A particular language can be a system in the context of this text. Or an entire environment starting from your personal machine, network, remote server, and your tooling of choice and workflow revolving around all of this can also be considered a system here. Or more broadly - a system is something you interact with systematically.

So let’s look at a few fast systems and implications of their use.

Interactive coding

I’m a Clojure programmer. So I have a high value for fast feedback during the development process - I can run code in the REPL at any time and immediately see the effect applied to my running application. This makes fixing most of the bugs a breeze.

I’m a backend developer, so most of my code runs on a server, not on my machine. If I were using something else, like Scala or Java, my typical workflow for fixing a bug would probably be:

Identify a possible wrong place in the codebase
Change it
Compile the project
Deploy it to the remote server¹
Restart the service
Try to reproduce the bug on a running system
If the bug is not fixed Goto 1
You’re good

With Clojure, my workflow is a bit different:

Set up SSH port forwarding to the remote server
Connect to the running application via remote REPL
Identify a possible wrong place in the codebase
Change it
Try to reproduce the bug on a running system
If the bug is not fixed Goto 3
Compile the project
Deploy it to the remote server
Restart the service
You’re good

While it is a bit longer overall, the feedback loop itself has only three steps, compared to six in the former case. Thus, I would say that Clojure is a faster system than Scala or plain Java thanks to the REPL² and the ability to change running programs remotely. The fact that I don’t have to wait for compilation, deployment, service restart, and possibly manually setting up some state for bug reproduction after each restart, makes feedback much more immediate. Thus, I can make almost arbitrary and quite small changes and experiment with real data on live applications until I get things right. That’s what fast systems are good for.

Notebooks

Ok, so REPL, while being interactive and fast is not the pinnacle of interactivity. Notebooks, for example, also give you immediate feedback, while also providing a UI for changing the parameters of your computation:

Your browser does not support the video tag.

While this concrete example is kind of slow to respond, it is still a nicer way to get your results faster by dragging a slider instead of changing code directly. I like the idea of reproducible research papers, where the document itself is the program, I even made something like that with Org Mode in Emacs and Lua via LÖVE2D. It wasn’t as interactive, as all of the images are static and require changing the parameters and re-evaluation in the document, but I think it would be possible to make a web version of it by rendering to HTML and connecting to the LÖVE via REPL.

But why stop there? Bret Victor years ago showed us an interactive notebook where not only code is interactive, but the results of it are as well! You can see it for yourself in the talk Inventing on Principle.

LLM-assisted coding

Finally, AI.

Yes, we’re going to talk about LLM-assisted programming, or so-called vibe coding. I consider it another fast system, because you can get your code changes pretty fast by giving your code to some LLM, and it will spit out a solution for you. Either incremental changes to your code or additional code written based on your program’s context.

This also makes your feedback loop shorter, because you don’t have to write the code yourself. You can think in higher-order terms, and LLM will hopefully give you what you need. In the best-case scenario, you won’t even touch the code by yourself.

I think combining Bret Victor’s notebook-like system with LLM can be an interesting approach. Actually, the code for the square wave example above was written by a Claude 3 Haiku model³. I have mixed thoughts on AI and LLM in general, but we’re gonna leave it out of the scope of this topic.

Fast knowledge system

Another thing about LLMs is that it is a very compelling search engine that can find information really fast. Want to learn a new thing? Ask the LLM about it and it will spit out the essential information. The quality of such information is a subject for another time, but let’s assume that it doesn’t hallucinate too much.

You can learn things fast! If the thing you’re trying to learn is well covered in the training data, it can be much faster than learning from the books. Especially because a well-trained LLM can answer your questions and provide more examples, unlike any plain ol’ book.

Slow systems

Now let’s talk about slow systems. Slow here can mean many things, from the amount of time it takes to do something to the general approach to doing something.

Slow hardware

I started this post by talking about hardware advancement for a reason.

At the university, my computer science teacher always said that programmers should get the slowest hardware they can find, and do all of the development on it. He reasoned that if an application can be developed and tested on slow hardware, the end-user experience will be great because their machines would certainly be faster than the ones that were used for development.

Developing on slower hardware limits you in a way that is hard to get otherwise. Today’s computers can process a lot of data fast, but three decades ago it wasn’t possible. Yet, we could go to the cinema in 1995 and watch such films as Toy Story - a film that was fully CGI. It took four years to make the film, and a lot of it was just rendering. Two years earlier, in 1993 Jurassic Park came out, and while it contained a very small amount of computer graphics, a test scene for the film was made before featuring fully computer-generated dinosaurs. The thing is, each frame took around 12 hours to render back then. And today we get video games that can look almost photo-real, outputting more than 60 frames in a single second.

What a horrible time the 90s were for CGI, you might say, and while you’d be right, slow hardware also had a positive effect - people had to write extremely fast software to even get barely usable results. And this software had to be robust too. When computing is this slow, you can’t avoid making a mistake, ruining days or even months of work.

But today it’s not a problem, right? Hardware is always fast, and we can upgrade it any time.

Well, not quite. While computers have become faster, we still have slow hardware that people have to optimize hard for. Want to try and take a guess which is it?

Game consoles.

Yeah, right, the thing that is supposed to be a pinnacle of gaming and graphics on release date slowly becomes obsolete with each year of its life. New CPUs and GPUs come out every few years, but game developers still have to optimize games for a console that is already a few years old.

Take a look at early games for Super Nintendo and the late games released for it. The later games are much more graphically rich than the early ones because while the hardware got older, the ambitions for games only got higher. And developers had to optimize, even use dirty tricks to pump out graphics like in Super Metroid, for example.

Or take PlayStation 3. Some of its late exclusives looked even better than the ones released for PlayStation 4 at launch. Why? Because their development started when PlayStation4 was not available, and gamers were getting better and better graphics on their PCs, so PlayStation 3 had to compete with them, not just other consoles. And slow hardware meant that they couldn’t just slap a new Unreal Engine on and expect it to just work - they had to optimize their games for a specific, and somewhat arcane hardware. And how good these late games looked, especially compared to the launch titles!

Today we have a problem with a lot of games that release on several platforms at once because game developers often use the top tier hardware for their development process, and then games can’t run smoothly on hardware with lower specs that it wasn’t optimized for specifically. Had the developers restricted their hardware limit to the lowest common denominator platform they’re targeting,f and thus the urge to optimize their code, maybe consumers could upgrade their hardware less often too, and be happier overall.

And this isn’t just games. Ordinary software today is as slow as ****. Like, some time ago the GNOME project replaced their old image viewer with a new one, and its startup time is seemingly ten times slower. And once it started it couldn’t even switch pictures as fast as the old one. I don’t know what kind of computers GNOME developers use, but it seems that lower-spec hardware is not their target. And don’t get me started comparing modern GNOME’s speed with GNOME2⁴.

Microsoft Windows sometimes takes a few seconds to show me the menu when I right-click on the desktop. This is the most basic action you can do, and yet it is so slow. Maybe if I had a gaming PC with a top-tier CPU, GPU, RAM, and SSD, it would not be so slow, but all I have is a late 2019 laptop with an eight-core AMD Ryzen 7 3750H. And it packs enough power for almost everything I do, even though it is six years old.

Slow tools and workflows

When your hardware is slow, all of your tools are as fast as your hardware allows them to be. And when you have a slow tool, you don’t want to run it more times than is necessary, because you have to wait every time.

This also applies to workflows. If your workflow is slow, you don’t want to take unnecessary steps, as it would take your precious time. Going back to the Interactive coding section of this post, let’s again look at the workflow for Scala/Java software development:

Identify the correct place in your codebase to start working
Write code
Compile the project
Deploy it to the remote server
Restart the service
Try to see if it works
If it doesn’t work right goto 2
You’re good

Again, compared to a language with interactive development, like Clojure, it is several steps longer. The implication of this is that you should, in theory, spend more time thinking about how to approach the task at hand. You don’t want to repeat this scenario over and over making small incremental changes, because compilation, deployment, restarting, and testing take too much time. So in theory, you should always try to write your code as best as you possibly can.

This, indeed is what happened in the early days of computing. Back then, computers operated via punch cards, and people had to submit their code on paper to the card puncher operator, then wait in the queue of other researchers, finally receiving the results printed out by the machine on another piece of paper. The cost of a mistake in the code was so high, that programmers had to spend a lot of time verifying their code in their head and praying that the operator wouldn’t make a typo while transferring your code to the punch card. The process demanded quality.

Only later we got machines that could do this in a fraction of time, but it was still slow. You may be familiar with “The Story of Mel, a Real Programmer”. The gist of it was that the machine was using a drum memory, which was very slow, but instead of accepting it as fact, Melvin Kaye, the protagonist of this story wrote its code in such a way that the program self-modified itself in the time it takes for the drum to revolve, to immediately continue. This is what basically an optimizing compiler would do, but Mel didn’t like the fact that this compiler would produce ugly code, and was still slower than a hand-optimized one. Somehow, we’ve lost this art of coding, but that’s another story.

Going back to slow workflow, we no longer suffer from waiting in the queue to use the computer, but maybe we should? There’s a saying that urgent things are rarely of great importance, and those that are important are seldom of great urgency. Sometimes it’s good to take your time and go and take a seat, thinking about the problem you’re solving, and how to solve it well, or even leave it to the subconscious thinking and step away from it entirely⁵. The dread of waiting in the queue for several days can easily force you into such behavior.

Slow knowledge system

In the AI section I’ve touched on the learning aspect of using LLMs. While you indeed can get a lot of information fast, sometimes it’s better to take it slow. Slow learning is not bad, if you’re digging deep.

Sometimes, when I’m learning a new thing, I find myself reading books, sometimes even several times. By the time I’ve finished, I have a deep understanding of the subject and can reason about it much more thoroughly than I ever could if I just did a quick search or used an LLM to explain stuff to me. I like the education model at the university - you have lectures that go over the subject in some depth, but then you’re given time to go over that same subject at your own pace with a book at home. That first quick overview gave you enough context to understand a longer-form explanation usually reserved for books so that you have an easier time understanding it. It takes a lot more time because you have to go over the same material basically twice, but your understanding is far more deep as a result. If you’re a good student, that is. I wasn’t, and now I’m sad about it.

Before the internet, the only way of acquiring knowledge was by going to a library. If it had the book, then great, but that’s all you get. If not - tough time, maybe you could afford to go to another town with a bigger library, or maybe you could request for your library to acquire a specific book, but that’s it. Today we can get our hands almost at any book we want.

But this isn’t an all-good thing either. Too many books in our reach can be overwhelming by itself. Sometimes, I think, it’s best when our choice is limited, and if we want to go out of our way to get another book, it is usually because we know that we need it, not because we’re bored with the one we have.

Effects of Slow and Fast systems

Times of slow computing are long gone. But the effects of slow systems are not limited to computing - it’s everywhere. The term system, I used in this article, is a lot wider, and applies to a lot of things. But today, more faster and faster systems are appearing around us. This already had an effect on our life.

For example, long-form content became a rare thing, because our brains have adjusted to the fast form of content consumption, and a lot of people today can’t hold their attention for longer than a minute, sometimes less. A book is a slow system, a course is a faster system, and an article is an even faster one. They all serve the purpose of providing information.

I write pretty long posts myself, and I noticed that because it takes a lot of time to write everyone, my attention span improved over these five years of writing. My workflow is also slow, as I don’t have things like live preview of my blog posts, I don’t use any automation tools or LLMs, just a basic spellchecker. Because I re-read my posts a lot, it improved my ability to concentrate - I became able to concentrate more, and now I can read more than I could earlier as a result.

Old paper mail is a very slow system, engaging longer and more thoughtful forms of conversation. You wouldn’t write “Hi! I have a question” in a mail, and expect a “Hi! What’s your question?” as a response. Email is a faster system, but still a rather slow one, when compared to more modern chats. But a lot of people today have forgotten how to write long and concise messages because faster systems have been around for so long.

I have high respect for communities, where people can respond to each other with thoughtful thought heavy messages. That’s also the reason I’m using email as a comment system for my blog - I doubt many people would want to write a long comment to my post if it was a web form, but when given an email as the feedback system, they subconsciously try to give it more thought.

Don’t get me wrong though, I’m not advocating that we should abandon advancements in hardware and infrastructure and go to simpler times. No, of course not! I too enjoy faster systems, I like interactive programming, fast feedback, and such. I have a lot of fond memories of late-night chatting in fast-paced group chats, and I definitively don’t want all of my communication to be done via email only. Sometimes I want to ask about things I know nothing of, instead of searching through books or papers. I just think that sometimes slowing down is the thing you need, but we rarely think about it. When everyone around you is all about “Go! Go! Go!”, it is tough to rise and say “Hold on, I need to take it slow”.

It’s not a production, but a test server. ↩︎
I know that Scala REPL is a thing, but nobody in our team uses it like this. ↩︎
Not that I couldn’t write it myself, I just don’t know how to use Jupiter and am not familiar enough with Python’s matplotlib. ↩︎
Even though GNOME got a lot faster over the years since the original GNOME Shell release. ↩︎
Hammock Driven Development ↩︎

Comment via email

Implementing dynamic scope for Fennel and Lua

Mon, 09 Jun 2025 22:31:00 +0300

I’m continuing my work on fennel-cljlib, my port of clojure.core and some other core libraries, focusing on porting missing functions and features to it. One such feature, which I sometimes miss in Lua and Fennel, is dynamic binding.

The Lua VM doesn’t provide dynamic scoping as a language feature, and Fennel itself doesn’t introduce any concepts like Clojure’s Var. However, we can still implement dynamic scoping that works similarly to Clojure and other Lisps using the debug library. Most of the ideas are based on information from the guides at leafo.net. There’s even a “Dynamic scoping in Lua” guide that implements a slightly different version of this feature, requiring variables to be referenced by name via dynamic("var_name") call. While this approach is feasible, I wanted something more in line with how other Lisps work, so let’s explore advancing it further. Luckily for us, Leafo already has all the necessary guides!

But first things first. I wanted to delay working on dynamic scoping as much as possible because it is a feature that’s hard to get right. I already have some experience with implementing dynamic scoping for one of my older libraries that implemented a condition system from Common Lisp in Fennel. This library, however, required special syntax to access all of the dynamically bound symbols and thus did not actually require anything fancy for it to work.

So what does dynamic binding/scoping mean in a language? If you know about lexical and dynamic scoping and wish to skip this tangent, feel free to do so.

In short, lexically scoped variables exist only where their lexical scope allows them to. For example, a variable defined in a block of code will only exist in that block because it is its lexical scope.

When working with languages that have higher-order functions, I often find myself in a situation where I want to refactor some code that uses anonymous functions by moving them out and giving them a name. Sometimes it’s possible; sometimes it’s not.

For example, imagine I wanted to move out this function from map:

(fn some-func [messages]
  (let [extra-data (other-func)]
    (map
     (fn [message]
       (do-stuff message extra-data))
     messages)))

If I were to do so, we would have a problem:

(fn process-message [message]
  ;; oops, extra-data is now an unknown variable
  (do-stuff message extra-data))

(fn some-func [messages]
  (let [extra-data (other-func)]
    (map process-message messages)))

So extra-data is bound lexically, and thus if we look at the lexical scope of the process-messages function, we’ll see that it tries to use extra-data while it’s not defined there. If extra-data were a global variable, it wouldn’t be problematic, but it is a local variable with a lexical scope. We could move the entire let that binds extra-data to the result of calling other-func, but let’s say we don’t want to call it on each iteration of a map because it’s slow and will do the same work repeatedly. So, what are our options here?

Well, we can make it a closure!

(fn make-message-processor [extra-data]
  (fn process-message [message]
    (do-stuff message extra-data)))

(fn some-func [messages]
  (let [extra-data (other-func)]
    (map (make-message-processor extra-data) messages)))

Now, we pass extra-data only once to the make-message-processor function, and it returns a function that has this variable stored in a closure. However, this is still a lexical scope because, as you can see, extra-data is present there.

In a language that uses dynamic scoping, this could be a whole different story. Let’s look at Clojure; although I wouldn’t recommend doing it this way, it is possible to do it this way¹:

(defn process-message [message]
  (do-stuff message extra-data))

(defn some-func [messages]
  (binding [extra-data (other-func)]
    (mapv process-message messages)))

Here, I assume that extra-data is a dynamic variable that obeys the rules of dynamic scoping. The binding call introduces a dynamic scope within which extra-data is set to the value of (other-func). It acts more like a scoped global variable, or at least you can think of it that way.

To introduce a dynamic scope, Clojure uses binding. Let’s look at it briefly:

(defmacro binding
  "binding => var-symbol init-expr

  Creates new bindings for the (already-existing) vars, with the
  supplied initial values, executes the exprs in an implicit do, then
  re-establishes the bindings that existed before.  The new bindings
  are made in parallel (unlike let); all init-exprs are evaluated
  before the vars are bound to their new values."
  {:added "1.0"}
  [bindings & body]
  (assert-args
    (vector? bindings) "a vector for its binding"
    (even? (count bindings)) "an even number of forms in binding vector")
  (let [var-ize (fn [var-vals]
                  (loop [ret [] vvs (seq var-vals)]
                    (if vvs
                      (recur  (conj (conj ret `(var ~(first vvs))) (second vvs))
                             (next (next vvs)))
                      (seq ret))))]
    `(let []
       (push-thread-bindings (hash-map ~@(var-ize bindings)))
       (try
         ~@body
         (finally
           (pop-thread-bindings))))))

It’s a simple idea: a try block without catch statements, only with a finally clause. Before we enter the try block, we set all mentioned variables to their values, and after we’re done with the body, we restore those values.

In Clojure, dynamic bindings work really well, but this is due to a combination of factors. First, the try support in the JVM is excellent, ensuring that finally will perform its intended function. Additionally, the JVM supports thread-local bindings, so even in a multithreaded context, binding still works. Finally, heh, Clojure has Vars, which makes it all possible.

Now, let’s try to implement the same concept in Fennel!

Implementing dynamic scope in Fennel

Before I descend into madness, I would say that we could do the same thing as in Clojure: set some variables, run code in a protected call, and reset the variables afterward. While this approach would indeed work, I wanted to tidy up my understanding of function environments in Lua. It’s a neat concept that Lua and a few other languages have, but Lua is one of the few languages that actually allows users to manipulate function environments. So, let’s explore this idea.

First, we need a way to forcefully set a function’s environment. This could be done in Lua 5.1 via setfenv; however, it was removed starting from Lua 5.2 onward. It can be implemented like this:

(local setfenv
  (or _G.setfenv
      (fn setfenv [f env i]
        (let [i (or i 1)]
          (case (debug.getupvalue f i)
            :_ENV (doto f (debug.upvaluejoin i (fn [] env) 1))
            nil f
            _ (setfenv f env (+ i 1)))))))

Now, we can set an environment for any function.

But what exactly is this function environment? I’ve realized that I never explained that, so here we go.

In Lua, the environment is a table² that stores the names of the variables. Starting from Lua 5.2, the environment is represented by a variable called _ENV, which is what we’re testing for in setfenv above. By default, _ENV has the same value as _G, a table that contains all global variables. However, we can change the function’s environment by modifying the value of _ENV.

For instance, in Lua, we can set _ENV to a table, and all global definitions would end up in that table:

a = 0

local function f (t)
    _ENV = t
    a = 42
    b = 322
    return t
end

local t = {}

f(t)

print(a, b) -- 0, nil
print(t.a, t.b) -- 42, 322

This is a cool feature, and we can actually use environments for sandboxing code, but that’s a story for another time. Let’s return to dynamic scoping.

Looking at this, you might get the idea that if we change the function’s environment and set our dynamic variables in it specifically, once we leave the lexical scope of that _ENV, all changes revert to normal because they never happened in a global environment!

Unfortunately, it’s not that simple. Yes, we can change the function’s environment, but it will only affect that specific function. Moreover, this change is permanent, meaning that we’ll have to reset the function back to its original environment. So, it’s not as straightforward as just changing _ENV around the code we want to run. Of course, we could write getfenv, then wrap the entire thing in a pcall, and safely restore the environment once the work is done.

However, we can’t set the environment of just the function we’re calling. _ENV is stored in a closure, so we’ll need to change all of the functions called by the function we wish to invoke with a custom environment. This makes undoing changes trickier to implement.

Luckily for us, we can bypass the need to roll back the changes to the function’s environment completely! Instead, we can simply clone the function and set its environment as we wish! Here’s an implementation:

(fn clone-function-with-env [f env]
  "Recursively clones the function `f`, and any subsequent functions that it
might call via upvalues.  Sets `env` as environment for the cloned function."
  (let [dumped (string.dump f)
        cloned (load dumped)]
    (var (done? i) (values false 1))
    (while (not done?)
      (case (debug.getupvalue f i)
        (where (name val) (= :function (type val)))
        (let [subf (clone-function-with-env val env)]
          (debug.setupvalue cloned i subf))
        name
        (debug.upvaluejoin cloned i f i)
        nil (set done? true))
      (set i (+ i 1)))
    (setfenv cloned env)))

Finally, we write the function that will call a given function f in a context where the given bindings are dynamically bound:

(fn dynamic-call [bindings f ...]
  "Calls `f` with `bindings` as its root environment."
  (let [new-env (setmetatable bindings {:__index _ENV})
        f* (clone-function-with-env f new-env)]
    (f* ...)))

As well, as a convenience macro for using it like a let but with dynamic binding:

(macro binding [bindings ...]
  (assert-compile (sequence? bindings) "expected a sequence of bindings" bindings)
  (assert-compile
   (= 0 (% (length bindings) 2))
   "expected an even number of forms in binding sequence"
   bindings)
  `(dynamic-call
    ,(faccumulate [res {} i 1 (length bindings) 2]
       (doto res
         (tset (tostring (. bindings i)) (. bindings (+ i 1)))))
    (fn [] ,...)))

Now we can use dynamic binding in Fennel!

Usage example

To illustrate, let’s create some variables that we wish to treat dynamically:

(global foo 21)
(global bar 73)

(print foo bar) ;; 21	73

With globals in place, we can try our binding macro:

(binding [foo 42]
  (print foo bar))
;; 42 73

As can be seen, foo no longer refers to 21, but now it is 42. However, if we try to print foo outside of binding’s scope, we will again get 21:

(binding [foo 42]
  (print foo))
;; 42
(print foo)
;; 21

A keen reader would mention that this example is not so different from using a plain let:

(let [foo 42]
  (print foo))
;; 42
(print foo)
;; 21

And you’d be right! However, where it’s going to be different is when we put functions into the mix:

(fn f []
  (print "f:" foo bar))

How, if we were to call it inside of let, the bindings introduced by it won’t affect the function, because both foo and bar are not lexically present here:

(let [foo 42
      bar 1337]
  (f))
;; still prints:
;; f: 21 73

This is where binding jumps in. Instead of following lexical binding rules, that are natural for most languages, we now introduce dynamic binding of foo and bar:

(binding [foo 42
          bar 1337]
  (f))
;; prints:
;; f: 42 1337
(f)
;; prints:
;; f: 21 73

And, as can be seen above, inside binding’s scope, f sees foo as 42, and bar as 1337, while outside of it, the values are still 21 and 73. So, we never actually changed the values of foo and bar, they’re still 21 and 73, respectfully. Instead, in the scope of binding we changed how f accesses these variables.

This also works with functions that call other functions:

(fn f []
  (print "f:" foo))

(fn g []
  (f)
  (print "g:" bar))

(fn h []
  ((fn [] (print "h:" foo bar))))

(binding [foo 42
          bar 322]
  (f)
  ;; prints:
  ;; f: 42
  (g)
  ;; prints:
  ;; f: 42
  ;; g: 322
  (h)
  ;; prints:
  ;; h: 42 322
  )

That’s pretty much it! This approach has a lot of flaws though.

First, it will only work on ordinary functions, so no tricks with __call metamethod, or native functions are supported.

Second, it won’t work with coroutines either. You can’t use string.dump on something like coroutine.resume directly, so we won’t be able to do (dynamic-call {:foo 42} coroutine.resume some-coroutine). It won’t even work if we wrap coroutine.resume into an anonymous function like (dynamic-call {:foo 42} (fn [] (coroutine.resume coro))), because coro here, while being an upvalue, is not a function, so we can’t clone it.

Finally, it relies on the debug library, and recursive function dumping, which itself is already pretty crazy.

There are probably more things that can go wrong with this.

So why not just set the globals temporarily?

First of all, yes, we could just set the globals temporarily, and restore their values later:

(fn set-globals [globals]
  (collect [name new-val (pairs globals)]
    (let [old-val (. _G name)]
      (set (. _G name) new-val)
      (values name old-val))))

(fn close-handler [old-vals ok? ...]
  (each [name val (pairs old-vals)]
    (set (. _G name) val))
  (if ok?
      ...
      (error ... 0)))

(fn call-with-temp-globals [globals f ...]
  (-> globals
      set-globals
      (close-handler (pcall f ...))))

(call-with-temp-globals {:foo 123 :bar 456} g)
;; prints:
;; f: 123
;; g: 456
(g)
;; prints:
;; f: 21
;; g: 73

While this works, I don’t like the idea that we’re actually changing the values instead of shadowing them in the environment, though this is more akin to the original Clojure implementation. Since Lua is single-threaded it should not be problematic, however, I think it can still mess things up if we were to introduce some kind of an asynchronous scheduler, like in my async.fnl library. This also messes up stacktraces in cases where an error occurs as a result of calling f because you can’t re-throw errors in Lua like in other languages. There’s also a potential to use <close> marker that came in Lua 5.4 to avoid pcall altogether:

local function set_globals(globals)
    local old_values = {}
    for name, new_val in pairs(globals) do
        old_values[name] = _G[name]
        _G[name] = new_val
    end
    return old_values
end

local function close_handler(old_values)
    for name, val in pairs(old_values) do
        _G[name] = val
    end
end

local function dynamic_call_close(globals, f, ...)
    local old_values <close> =
        setmetatable(set_globals(globals), {__close = close_handler})
    return f(...)
end

This would keep stack traces intact, and values would be restored right when we exit dynamic_call_close but will only work in Lua 5.4.

Thus, while doing it with pcall is more generic, I wanted to explore the environment approach first since Lua already provides a mechanism for working with function environments. But for now, I think I’ll leave dynamic scoping out of cljlib as I’m not really sold on any of the ways of doing it that I’ve come up with so far.

I really don’t like this way of refactoring this code, but sometimes it is exactly what you need. Not this time, though. The approach with closures is what I’d use in this case in Clojure too. ↩︎
Everything in Lua is a table, because of course it is. ↩︎

Comment via email

Am I becoming obsolete or just older?

Tue, 22 Apr 2025 00:36:00 +0300

Let’s set the tone for this post right away, I just want to get this out of my system: I despise the rise of AI in tech. Coding assistants, chat-bots, vibe-coding, agents. Oh, the vibe coding… WTF?

Figure 1: This will be an “old man yells at cloud” type of post.

I don’t like AI in general, although I know that neural networks can be used for good. Before you accuse me that I’m completely anti-AI, let me explain where I see this technology applicable.

One such example is modeling. I play guitar, and it’s a pretty expensive hobby. A good guitar itself can cost hundreds, or thousands of dollars, but then there’s guitar gear. Tube amplifiers and pedals are expensive. Not everyone has money to have several amplifiers for achieving versatile sound, and a bunch of effect pedals to work with these amplifiers.

The industry noticed that, and a lot of effort was put into modeling these circuits in software. Guitar pedals and amplifiers are basic electronics, for the most part - cascading transistors, condensers, operational amplifiers, etc. It’s possible to model them accurately in software, and I did so myself, implementing Maxon’s OD808 overdrive pedal in C++ as my diploma project for graduation. The diploma itself was about precise software modeling of analog systems, and the guitar pedal was a practical example I could show. You should have seen the faces of the diploma committee when I pulled out a guitar for a project demonstration. Better, you should have seen MY face, when the head of department took my guitar and started blasting Iron Maiden riffs.

However, modeling these schematics accurately and working in real-time is hard. Especially, when we’re talking about emulating tubes or tape machines. People still did, but the results were not as close to the real thing, as one might like, and sound engineers are picky for the details. But then there was another way of modeling - feed the raw signal through your equipment of choice, be it a tube amp, or some weird pedal schematic, capture the result, and train a neural network so it could reproduce it flawlessly. This is the technology behind Neural Amp Modeler and other such projects.

This is where neural network technology shines in my opinion - you get accurate results, even though accurately emulating entire circuits with tubes in real time is still a hard task for programmers. It sure can be done manually, and there are successful examples, where individual components of the circuit are programmed as separate modules, and then the whole circuit is constructed from them. That’s the approach I took with my diploma project as well, but it still takes much more time than just training a model with a pair of .WAV files. And the neural network method comes with another real benefit allowing you to capture your sound exactly as is for later use. For example, if you’re recording at the studio, and you want to patch things later or re-record certain parts, you can capture your studio sound, and then reproduce it later.

There are other fields, where neural networks can be helpful, I won’t go listing all such cases, I’m just going into this tangent so that you understand that I’m not against neural networks in general. However.

When ChatGPT3 was first announced, people started going crazy. We had no real breakthroughs for quite a few years, GPT2 wasn’t really good. Image and video generation models weren’t good either. Then suddenly, GPT3 talks close to being confused with a real person, and images generated by other models start looking correct, and not like a mush of pixels. People notice that GPT3 can generate code in most programming languages, and make attempts at problem-solving and such.

Fast forward a few months, and we have first coding assistants and IDEs with a primary focus on AI for development and refactoring. The internet is becoming worse by the day - AI-generated articles pollute search results, and you don’t know if the thing is real or not when looking at “drawings”. AI-generated music pollutes streaming services, often sounding awfully lot like some other artist, without acknowledging that it was trained on their music. Some streaming services even announced that now you can remix the music with AI if you have a paid plan. People accuse one another of the use of AI to pretend that they know how to do what they do. The worst.

Figure 2: A comment to the Generative AI is a Parasitic Cancer video by Freya Holmér

Then, there’s the new hot thing - vibe coding. I’m living under a rock, I know that, because I don’t follow news, tech hype reaches me mostly via gossip in real life. So when I first heard the term vibe coding, I naturally assumed that it’s another one of these “Lofi girl”-type work setups where you have a comfy environment for coding with good vibes.

Figure 3: this kind of stuff, just imagine she’s programming. I guess, I could generate an image where she’s programming for real, but then this post would be hypocrisy.

This is pretty much what I do when I have free time to work on my personal projects. I prepare myself a nice cup of coffee with steamed milk and peanut butter¹, sit down at my desk, put on headphones, and enjoy my coding time in the late evening. Vibes, right? Nope! Apparently, vibe coding is when you explain to your AI-based editor what the heck you wanted to do, except you’re much more into debugging code written for you. WTF it is calling vibe coding? I wouldn’t mind it being called AI coding, but vibe?

I get it, AI is a hot new thing, and people want to use new shiny things because it triggers a dopamine hit. But toying with something for fun, or being passionate about things should still remain reasonable. It’s fine when you do this for fun, or sparingly, but when all you do is now done by AI, it starts getting a bit too far in my book.

I see AI-generated images in presentations and I cringe.

I see how people brag about how they used AI to do “this and that”, even though the task at hand was easy and I cringe.

I see how people misuse AI for harder tasks that can still be done without AI perfectly fine, with a much lower error rate, and I cringe again.

I see posts about how vibe coding brings back joy to programming, and I only have one question - what joy? If you did enjoy programming, but felt down recently, I think AI is going to make you feel better in the short term, but burndown will hit you again soon enough when the novelty wears off. And if you didn’t enjoy programming in the first place, how AI is going to help you enjoy it? Because you don’t have to do it manually?

If you don’t enjoy the process, AI won’t make it more enjoyable. You will simply offload the unenjoyable part to AI, and then gaslight yourself.

Phew.

These are the kinds of thoughts I had in my head lately. And when I’m annoyed at something for no apparent reason I usually start reflecting on it in hopes to understand the source of the problem for me. Worrying over nothing is the worst thing you can do for your mental health. Like, If I don’t use AI, I can choose not to care about others who use AI, and continue to live my life like I did before. That’s what I did before, like when blockchain and NFT were the hot new topics in town.

However, AI-assisted coding practically hurts me even without me participating in it. First, I chose to ignore AI assistants, heck I don’t even use non-AI assistants such as LSP that much. So when I talk to people, who use AI, I get weird looks, like: “Man, you’re gonna be obsolete in 5 years if you don’t learn how to use AIs, it makes you much more efficient”. Am I?

I don’t think efficiency is in speed only - it doesn’t really matter how fast you can produce code. Programming is mostly thinking a lot, and then writing a little. It sure does matter to a business, because today’s main tactic is TTM optimization, but that’s another topic.

I think relying on AIs for coding is going to hurt us in the long run. Count me old-school, but I believe that today a lot of people in the programming world don’t know how to code. You’d be surprised how little people know these days. Ask them the difference between an array and a vector, and they won’t be able to tell the difference. Don’t get me started on advanced topics like asynchronous/parallel programming, or even more basic features like closures.

I’ve seen a lot of people who think that they know what they’re doing, but when asked about certain internal characteristics of the language/framework/library they use, that did matter to the problem at hand, they stumbled and couldn’t answer. And I’m no better, I was there too - I’ve been writing code for bare-metal in C during the first two years at my first-ever job and didn’t understand anything. Only after I delved deeply into the language and SOC stuff, I started understanding what and why. It improved my results at work and tremendously helped me beyond just C later. So I believe, that people can learn, but would they if everything is done by AI?

I can confidently say, that I know Lua deeply enough to understand how its bytecode VM works. I also can confidently say, that I know how most of Clojure works internally. But I still can’t confidently say that I’m a great programmer. A decent one, maybe, but I still don’t know a lot of stuff behind the scenes. This is why I constantly delve deeper into the technology I’m using, trying to understand the inner workings of it.

And now we have AI assistants, practically writing programs for these people, and they can’t reason about the code written by hand, let alone generated code. I’ve seen people applying for a job, and failing miserably on the simplest questions on the coding interview. They can output the solution for tricky Leetcode tasks, but can’t explain them. Some are even proud that they can make AI assistants solve the problem for them without them even trying to understand the problem. I, myself, can’t solve these Leetcode problems, but I still can write robust code for the real-world task at hand.

Recently, I encountered a message in a group chat, that pretty much triggered me to write this post, with the following contents:

There are orders of magnitude fewer code examples for ClojureDart, which makes it very difficult to train LLMs. Examples from regular Clojure will probably be pulled up, but the libraries won’t work - you’ll need to port them first.

For a beginner, this is a very large and painful entry threshold, especially if there is no desire to write the code yourself and delve deeply into the language.

Oh, how did people even learn to code before LLMs existed? I imagine it was so hard and painful! Imagine - learning the language by writing code and understanding what you’re doing, am I right? This is so wrong on so many levels.

First of all, if there is no desire to write the code yourself, why the heck do you want to be a programmer? This code is going to run on a machine, so it should be well understood, otherwise it’s a time bomb waiting to explode. And with LLMs and vibe coding, the whole point is that you don’t look into the code that much. Or at all.

And, I’m not talking about using LLMs - even without them hallucinating a solution today we can’t write reliable software that works properly all of the time. I get frequent crashes when doing mundane tasks - if I were to take a screenshot every time a program hangs or crashes, making me lose my work, my screenshot folder would fill up my hard drive. Most programs today can’t achieve even “four nines”, let alone five or six nines of uptime. Your Windows laptop will suddenly reboot to install updates, so even if your application can theoretically achieve “five nines”, the platform it is running on can’t. I lost work due to Windows updating the .NET Framework, why the heck does it require a reboot?

Today’s software is in a miserable state. We have a lot of over-engineered stuff at the base of our software. Legacy code creeps in, making writing new code harder. Bugs are today’s norm - I’m still amazed that everything is holding up, instead of falling apart.

If engineers were as bad as programmers today, we would all be dead, lying flattened under collapsed buildings. Oh, wait…²

So am I becoming obsolete or just getting older and more grumpy? I guess it’s both.

I always wondered when would I become one of those old people who don’t understand modern technology. I was born in the 90s, and saw firsthand how technology became more and more advanced - I had access to a computer since the age of three. My folks don’t understand modern social networks that well, let alone all this AI stuff. Now I’ve become a part of the club, I guess - I also don’t understand modern social networks and all this AI stuff. Heck, my dad is probably more advanced in tech than me, as he’s been doing this smart-home thing for a while now, and I’m not understanding any of it.

Programming is both a job and a hobby of mine, I like writing code to solve problems I often impose it on myself for the fun of it. I like to think about solutions, and if I can’t do it myself, I search online like in the good old days, learning in the process.

You can argue, that an AI chatbot is a glorified search interface, except it’s not exactly that in my opinion. Today’s AI chatbots don’t just present you with existing information, instead, they reconstruct it. LLMs are more or less a decompression algorithm, except not precise in most cases. They can hallucinate a solution, and still sound as confident as if it was true. Keep in mind, that an AI chatbot doesn’t understand what it’s outputting, for now at least. People do that too, unfortunately, but at least they don’t sound as confident when they clearly don’t know what they talking about (yeah, sure).

Sometimes, beginner programmers who only know how to code with LLMs face the harsh reality, that LLMs can’t solve all problems. For now. Maybe in the upcoming years, large language models will become so sophisticated that it would be possible. I don’t know. And If such a beginner realizes, that they still need to learn things, the internet is already full of articles generated with AIs by people who don’t understand the topic, making it much harder for them to learn properly. I’m facing this problem too, each time I want to learn something new, figuring out halfway through the article that it is an AI slop.

Maybe one day I will have to use AI because my job will demand it, and I will change my opinion on this whole thing. Or maybe I’ll change the job instead, depending on the circumstances. Blacksmithing looks interesting.

Figure 4: A poker I made

Or maybe woodworking? Or pottery? Unless AI would do these things too by that time. Of course not.

Of course not.

BTW, amazing recipe: 12 grams of coffee beans, grounded for double shot espresso, about 35-40 grams of salty peanut butter, and steamed milk. Nice texture and taste. ↩︎
AI is apparently used in engineering too now. ↩︎

Comment via email

String interpolation in Lua

Mon, 07 Apr 2025 19:03:00 +0300

If you know Python or do a lot of shell scripting (or even write in Perl), you’re probably familiar with the ability of these languages to reference variables or even expressions from string literals.

In a shell, it is possible to do this:

echo "Hello, $user"
# prints: Hello, <your username here>
echo "2 + 2 = $(echo 2 + 2 | bc)"
# 2 + 2 = 4

A similar feature exists for Python with f-strings:

f"{2 + 2 = }"
# '2 + 2 = 4'

That’s a cool feature! I wish I could do this in Lua!

Unfortunately, Lua doesn’t have such a feature, and we’re stuck with plain ol’ string.format(). Which is fine, but it has some quirks.

For instance, when you need to repeat the same thing several times in the resulting string, you have to repeat it the same amount of times in the format string, and in the arguments:

function provoke_thought (what)
    print(string.format("to %s or not to %s", what what))
end

provoke_thought("be") -- to be or not to be
provoke_thought("bee") -- to bee or not to bee

With f-strings in Python, it can be done like this:

def provoke_thought (what):
    print(f"to {what} or not to {what}")

provoke_thought("bee") # to bee or not to bee

It’s a bit nicer, as you don’t have to repeat both format specifiers and the variable two times. Another bit of convenience is when you have a lot of arguments:

print(string.format("this string has %d amount of arguments, which are %d (a %s), %q (a %s), and %p (a pointer)",
                    n,
                    answer,
                    type(answer),
                    greeting,
                    type(greeting),
                    string.format))

Which prints, assuming these variables are defined somewhere, the following message:

this string has 4 amount of arguments, which are 42 (a number), “hi mom!” (a string), and 0x7f0d5fb7f1b0 (a pointer)

Now, when you have this many variables, it’s not as fun to maintain. Especially, when you need to modify this message, possibly reordering format strings, or removing/adding some.

Let’s look how this would look with f-string in Lua:

print(f"this string has {n} amount of arguments, which are {answer} (a {type(answer)}), {greeting:%q} (a {type(greeting)}), and {string.format:%p} (a pointer)")
-- this string has 4 amount of arguments, which are 42 (a number), "hi mom!" (a string), and 0x7f0d5fb7f1b0 (a pointer)

Wait, didn’t I just say, that Lua doesn’t have f-strings?

That’s where I present you interpolation.lua.

`interpolation.lua`

A clever name, I know.

This is a small library I made, that handles string interpolation in Lua in a similar way to Python’s f-strings. There are some differences, of course, but in general, it should be familiar to you, if you ever used one or another variant of this feature.

Let’s look at some of its capabilities.

You require this library like this:

local f = require "interpolation"

And now you have the function f, which accepts a string, parses it, and spits out the result, formatted accordingly to the expansions in {}. When there are no expansions, the string is returned as-is:

f"hi, mom"

hi, mom

However, when there are expansions, you can do various things. For example, you can define some variables, and reference them in the string:

local x, y = 4, 2
f"{x}{y}"

You can even evaluate expressions:

local x, y = 40, 2
f"The Answer is: {x + y}"

The Answer is: 42

One cool feature is the = modifier, which adds the original expression to the output string:

local x, y = 40, 2
f"{x + y = }"

x + y = 42

It’s like when I’m doing literate programming in Org-mode, where I can do a similar thing by writing src_lua[:exports both :post format-result(result=*this*) :wrap]{return 1 + 2}, and it will expand to return 1 + 2 = 3. Except, much less verbose.

As I have shown before, the same format specifiers are available, as per string.format:

local x, y = 40, 2
f"{x + y = :%X}"

x + y = 2A

Now, let’s look at how this is implemented because there are some tricks.

The parser

First things first, this is not a mere string substitution, where I look for something like {([^}]*)} and replace the entire thing with some value. Instead, there’s a parser, which reads the string byte-by-byte until it encounters the { character.

To read a string byte-by-byte I had to port a string-reader from my reader.fnl project. After passing a string to the string-reader function, we get back a reader object with the following methods: read, lines, peek, length, and close. This library doesn’t really use anything but read, peek, and close, but it’s good to have the rest still.

Next, the parser itself:

local function parse_fstring(str)
    local rdr, chars, exprs = string_reader(str), {}, {n=0}
    return parse(rdr, chars, exprs)
end

First, we prepare some storage objects, and the reader, and call the main parse function:

local function parse(rdr, chars, exprs)
    local char = rdr:peek(1)
    if char == "{" then
        rdr:read(1)
        parse_expansion(rdr, chars, exprs)
        return parse(rdr, chars, exprs)
    elseif char ~= nil then
        chars[#chars+1] = rdr:read(1)
        return parse(rdr, chars, exprs)
    else
        return table.concat(chars), exprs
end end

It looks like a fairly straightforward parser, but the meat and bones are in the parse_expansion function. We call it when we encounter a { in the string. It’s pretty big because it has to handle several cases.

Once we’ve found { we enter expansion parsing mode. In this mode, we look for these cases:

We’ve found another {
We’ve found a " or '
We’ve found an equal sign: =
We’ve found a colon followed by percent: :%
Finally, we’ve found the closing }

If we find another opening curly, that means that the expansion contains a table literal. In this case, we enter another parsing mode, that parses a table, until it finds the matching amount of closing curly braces. So any level of table nesting is supported.

If we find a " or ', it means that the expansion is holding a literal string, so we enter a string parsing mode. This means a few things to us, because inside strings we should not treat both special characters described below, and table delimiters in the table parser as well. So the table parser also uses the string parser.

Otherwise, if we’re outside of a string or a table, we process the rest of the special characters.

If we find the = somewhere before the format sequence, we inject the current expression into the resulting string, and then append a format specifier, %s by default, unless changed via :% sequence.

If we find the :% in a string, we assume that this starts the format sequence of the expansion. So we enter another parsing mode, where we parse and remember the format specifier.

After that, we scan for a closing }. If there’s something else after the format sequence, we signal an error, as there should not be anything but spaces.

I glossed over this, so if you’re interested in more details of how this works, I welcome you to read the code. Next, let’s see how variables and expressions are turned into values.

Expansion substitution

At the start of the f function, we create an environment:

env = setmetatable(locals(), { __index = setmetatable(upvalues(), { __index = _G })})

This environment mimics what the Lua VM can see when it needs to get the value of a variable onto the stack.

First, we obtain all locals from the current scope, right outside of the function call:

local function locals()
    local idx, variables = 1, {}
    while true do
        local ln, lv = debug.getlocal(3, idx)
        if ln ~= nil then
            variables[ln] = lv
        else
            break
        end
        idx = 1 + idx
    end
    return variables
end

It’s a rather simple function, that relies on debug.getlocal to obtain all visible local variables - their names and values. So we store it in the table.

Next are upvalues. Upvalues are variables in outer scopes, so we need to get them separately:

local function upvalues()
    local variables, idx, func = {}, 1, debug.getinfo(3, "f").func
    while true do
        local ln, lv = debug.getupvalue(func, idx)
        if ln ~= nil then
            variables[ln] = lv
        else
            break
        end
        idx = 1 + idx
    end
    return variables
end

It’s similar to locals, except it uses debug.getupvalue, ans stores variable names and their values into a table.

We then compose these tables via the __index metamethod, maintaining the scope hierarchy. If some variable x exists as a local but also exists as an upvalue, we will correctly see the local x value. Otherwise, if x is not in the table returned by locals, we automatically look it in locals’ __index metatable. And if it’s not there, it might be in the _G table for globals.

Expressionn evaluation

However, not all things can be looked up the way described above. For example f"{math.pi}" would fail, despite math being in _G, because we would look for _G["math.pi"], and there’s no such thing there. And we can’t really assume that any sequence of characters with . in the middle is a table lookup, so we can’t translate {math.pi} to _G["math"]["pi"]. Even if we could, it wouldn’t be a good solution, because we can also have expressions like f"{1 + 2}".

Instead of implementing a small interpreted language for the expansions, I decided to use Lua’s load function (and loadstring for Lua 5.1):

local function eval(code, env)
    local env = env or rawget(_G, "_ENV") or _G
    local setfenv, loadstring, f = rawget(_G, "setfenv"), rawget(_G, "loadstring")
    if (nil ~= setfenv) and (nil ~= loadstring) then
        f = assert(loadstring(code))
        setfenv(f, env)
    elseif assert(nil ~= load, "neither load nor loadstring are available") then
        f = assert(load(code, nil, "t", env))
    end
    return f()
end

So our lookup mechanism is as follows.

First, we try to look at the variable name directly in the env table. Then, if nothing was found, we try to eval an expression. However, you can’t simply do eval("1 + 2") in Lua, because Lua requires explicit return even on the top level. So our code looks like this:

local value = env[name] or eval("return " .. name, env)

Bypassing variable lookup

There’s one quirk about this implementation, that I’m not sure how to fix. I noticed it randomly when I was writing tests.

What’s the output of this:

local function string_adder(x)
    return function (y)
        return f"{x + y = }"
end end

local g = string_adder(40)
g(2)

If your guess is a string "40 + 2 = 42", then your understanding of this library is correct, but your understanding of the underlying Lua runtime is a bit incomplete. It’s not at all a problem, this stuff is not required to use Lua, and the problem is subtle.

The resulting string won’t even be returned, because it will raise an error attempt to perform arithmetic on a nil value. Why is that?

Before I explain, let’s fix this function:

local function string_adder(x)
    return function (y)
        return f"{x + y = }", x
end end

local g = string_adder(40)
(g(2))

Now the result is indeed "40 + 2 = 42", so what gives?

Apparently, if the code never actually references the variable, it won’t be visible by debug.getupvalue. By code, I mean the string of the actual code that is being compiled by the Lua VM into bytecode. It’s logically sound, though - why would the compiler even bother about storing an upvalue if it is never used? And the compiler doesn’t know that we will in fact use this upvalue in our eval function, by extracting it from the string in this special way.

So if this fails, and all you need is to return a formatted string, what do you do?

The f function got you covered - you don’t need to rely on local lookup, instead, you can provide your own env:

local function string_adder(x)
    return function (y)
        return f("{x + y = }", {x = x, y = y})
end end

local g = string_adder(40)
g(2)

This looks like string.format, again, except it allows providing variables in any order and without duplicates. I mean, f("Agent {x}{x}{y}", {x = 0, y = 6-1}) is still a bit more concise in my book than repeating x several times.

Passing your own env prevents any other kind of lookup, so no more globals, upvalues, or locals will be found. This also makes this library usable without the debug library, although it still requires load for evaluation. Again, I don’t want to write a Lua interpreter as a fallback for that.

A cool feature?

For me, sure yes. I like this to be in the language I work with, as it makes some specific cases less of a burden to write and maintain. Of course, not all strings should become f-strings, there’s still a place both for string.format and plain string concatenation.

Now, why do I make this library for Lua in Lua, and not for Fennel, as I mainly do my hobby project in it? First of all, I enjoy occasionally writing in pure Lua. I like the language, it has its charm. Fennel is objectively a better dialect of Lua, but sometimes I want to be reminded why it is.

Second, Lua allows for calling functions on strings without parentheses: print "foo". This feature also applies to tables, meaning you can create small DSLs in your program. For example, one of my libraries implements a JSON encoder. In Lua, I can call json{x = 1, y = 2} and get back a string '{"x": 1, "y": 2}'. This way I don’t need to write JSON in strings manually, and I can reuse Lua’s table syntax and my editor’s formatting features.

This might be a silly reason to make this in Lua, but I wanted to make this library because it sounded fun and easy. In the end, it wasn’t easy but was still pretty fun. It might be a better fit for Fennel still because all these f-strings could be generated at compile time, additionally solving the upvalue problem, because variables will be referenced, and there would be no need for a custom evaluation step.

This is why I think this feature should be a part of the language and implemented in the compiler itself. I have a toy language that implements a bytecode VM, so perhaps I could try to make it there to see if it would work well. But I hope to see f-strings in Lua someday. Maybe in Lua 12.4, who knows?

Comment via email

Dynamic font-lock for Fennel

Mon, 10 Mar 2025 01:26:00 +0300

I’ve been working on adding a dynamic font-locking for fennel-mode for a while now. The first attempt started years ago, in July 2022, and were quite crude. It consisted of sending a special code to the REPL that would return me a list of macro names, and then I created a regular expression that matched all these names for syntax highlighting. This had several problems.

First, this meant that any macro known to the REPL would be highlighted. This is not great, as macros in Fennel can be loaded from any file that you require, but it doesn’t mean that they’re available to your file in particular.

For example, imagine you have this code:

Figure 1: All code in this post will be provided as images to better show code highlighting

As you can see, this code requires the module :foo, defines a few functions, and finally calls them. Ignore the fact that the module foo is unused for now - turn your attention to the colors instead.

As you can see, fennel-mode uses colors to tell the user a few things. Special forms and macros are highlighted in a blueish color. Inbuilt functions are highlighted in purple. Names of declared functions are also in purple, but not their calls.

However, you can see that for some reason, bar is highlighted blue, as if it was a special or a macro. This is because I’m using my old implementation of dynamic highlighting. If we look into the foo module, we’ll see that it requires a macro module, that defines bar as a macro. It’s not a problem - macros in Fennel are only available in the module they’re required, so there’s no name conflict here, as you would expect. However, highlighting is not very smart.

When syntax highlighting is applied, a request is sent to the REPL to get all available macros. Then no distinction is made for where these macros are coming from, hence the wrong highlighting.

Another tricky case, that I don’t see implemented around is scoping:

Figure 2: Like, imagine, otherwise I’d have to do this highlighting in CSS manually

In Fennel, the macro special, which is used to define macros, obeys the scoping rules. Thus when we create a macro inside of the do’s scope, this macro should not be highlighted outside of it. But again, because the initial implementation of dynamic highlighting didn’t care about anything but macro names, the macro was incorrectly highlighted in this case as well.

Because of these problems, Phil and I decided that this won’t be a good feature to bring into fennel-mode in this state. The branch rested for almost three years, up until recently, when I started working on fennel-mode again.

Recently, I added support for Flymake via the fennel-ls language server. That was another old branch of mine, where I added linting via Fennel’s own linter plugin, but Phil mentioned that the plugin was never supposed to be used like this. It was also suggested that since fennel-ls already existed back then, and it was working with eglot there was no real need for using Flymake with Fennel’s linter plugin.

I gave this branch a rest, and after a while, when I started using fennel-ls myself, I decided to contribute a small change so that it could be used from Flymake. Personally, I don’t need a full-blown language server for Fennel, because everything I need is provided by the Fennel Proto REPL, but I do want some linting capabilities. So after merging the Flymake branch into the main fennel-mode branch, I noticed this old dynamic-font-lock branch, and decided that “it’s time to do this”.

Thus, I’m happy to announce, that all the problems listed above were fixed! Take a look at this:

Of course, when we just load the file, nothing is highlighted yet, as the syntax highlighting is dynamic. However, when we load the module into the REPL, the highlighting is applied correctly:

Once the module is loaded, highlighting is cached, and no further requests to the REPL are sent. This doesn’t, however, mean that when we change the code, the highlighting won’t react:

Your browser does not support the video tag.

I wanted to make this feature for quite some time. I’m a big fan of CIDER’s dynamic font-locking and have been using it for some years. It never occurred to me, however, that CIDER doesn’t do scoping:

But Clojure macros are not scoped, like in Fennel, so that’s probably fine. What’s not as fine is that it will highlight the macro before it was declared:

Which would be an error if we were to try to reload this file. But, for what it’s worth, I think CIDER’s solution is fine, because again - no special scoping rules, and unless the file loads without errors, no dynamic syntax highlighting would be applied. For Fennel, however, I wanted a more robust solution.

For now, this is still available in a feature branch, but I’m positive that I’ll merge this soon, after I give it a bit more testing, and gather some feedback from the users. If you’re a fennel-mode user, and you use my fennel-proto-repl module, I encourage you to try out dynamic font-locking and see if it works for you!

Comment via email

Blog - fifth anniversary

Sun, 16 Feb 2025 08:52:00 +0300

Somehow I’ve missed the fifth anniversary of my blog! Oh well. Well, it’s not that big of a deal, but I want to reflect on my blogging a bit.

I started this blog on January 27th, 2020 with the following line:

This is the first public post that I wrote because I’ve finally thought that I’m clever enough to do so.

And that’s one of the most important decisions I made in the last five years.

In fact, I wasn’t smart enough to make that post. As I’m looking at my first set of articles about GRScheme, it is obvious that these were baby steps. But this kick-started my interest in writing and more importantly - learning.

Every time I write a post on this blog is usually because I learned something new. And every time I learn something new I want to make a post on my blog. And more often than not I want to learn something new to later make a post about it.

The only thing I regret is not starting this blog sooner in my life - I’m positive that it would help me grow much quicker and I wouldn’t make a lot of mistakes that I did.

2020

In 2020 most of my posts were all over the place. I was still figuring out what I wanted to write about and because I only had one recent project I was working on, I decided to use it as a starting point. That project was GRScheme - an implementation of an interpreter of a Scheme-like language written in Rust.

I was learning Rust because I felt that I couldn’t do C for a living and enjoy it at the same time. Don’t get me wrong, I like C, it has its charm, but I was working for almost five years with it at my day job, and it started to wear out. I didn’t want to switch to C++ full-time, so I started looking at alternatives. Rust was an obvious one - a hot new language with a lot of projects circulating over the web, the whole “rewrite it in Rust” shtick, etc.

Rust hit me at a weird time - I was discovering Lisps, and because I liked what I saw in both languages at the time, I decided that writing a Lisp from scratch in Rust would be a fun project. And a fun project it was! The result is horrible too!

I was also messing with Emacs a lot at the time, and Emacs was another reason for my passion for blogging. You see, I started this blog with Org Mode and Hugo, and I still use them for this blog. I would probably do so even after Hugo fades away and will be replaced with a new hot thing. Maybe it already was - I never checked. Hugo + Org Mode has everything I ever needed.

Emacs also played a role in my career, because it is configured in Lisp. And Lisps were an interest of mine at the time so all three things came in nicely. I configured Emacs with Lisp to use Rust, wrote Lisp in Rust, and blogged about it from Emacs.

After finishing the project and failing a few interviews for a junior Rust developer position I thought to myself: if I like Lisp that much, maybe I should learn a proper Lisp, instead of toying with a weird implementation of a thing I know nothing of? Yes, I knew nothing about lisps and still managed to make an interpreter. A horrible, yet working one, with a lot of bugs, of course. So I set myself on a journey to find the Lisp I want to learn.

After examining alternatives, and briefly considering Racket, I stumped on the talk by this guy Rich Hickey about simplicity. And kinda fell down the rabbit hole with this Clojure language that I never heard of before.

So after blogging a bit about Rust and Emacs, Clojure was the next thing I wanted to learn. And what better way to learn something than to blog about it?

I made a simple ray casting demo in ClojureScript, and was pretty confident in my knowledge of the language after just two months of messing with it. Sure, learning some Scheme beforehand helped, but only to make me feel easier about all these parentheses. Which I blogged about later.

So I applied for the junior Clojure developer into an organization I never heard of before. This company specializes in information security, and they were looking for a middle-to-senior Clojure developer. I still don’t know how I got this job, but I’m extremely glad that I did.

I feel that I need to address this thing though. I was applying for a junior developer despite having five years of experience in software development. Why?

Simply put - the stack is so different that I might as well don’t know anything. At my previous job, I hadn’t written any servers, never talked to a database, and so on. So junior position was the only thing I could really go for. Another five years passed since then, and I’m no longer a junior developer, but if I ever have to drastically change the stack again I don’t mind applying to the junior position again at all. I learn fast, so I won’t be a junior for long.

2020 was a year I learned about Fennel - another lisp language. And I fell in löve. You can kinda see that, given how many there are posts tagged fennel.

2021

This year was mostly Fennel-focused. I became a Fennel contributor by re-implementing the pretty-printer. This code is still used today, either because it is terribly complex and nobody is brave enough to rewrite it again, or because it works well enough. I’m on the latter side, although I feel that the former one is not entirely incorrect.

I further experimented with Fennel. Macros were a hot thing to learn, so I made a condition system implementation. This was another attempt at making something I know very little of, by just observing how it behaves. Turned out pretty well, and completely useless too!

On the brighter side, I made another library that year that implemented lazy sequences. It was fun to make, and I learned a lot about how lazy sequences work in Clojure along the way.

The next thing I tried was asynchronous programming. You see, at work, I started working with asynchronous code, and literally nobody understands asynchronous code. So the only way to understand async is to write a library for it - I thought to myself. And so I did.

This library was horrible, but I didn’t realize it until I understood how async works. Which was way after I made this library.

This year also was a year when I decided to participate in the Advent of Code event. I made a post each day, and by day 25 I was so exhausted that I decided that I would never do it again, and hidden all of the posts. They are still there if you know how to look, but I won’t give you any links.

Also, I grew to hate these AoC posts in the feed. Being bombarded each December in the RSS feed by “Advent of Code - day N shenanigans” type of posts kinda ruined AoC for me.

2022

2022 wasn’t as productive when it came to my personal projects, so I shifted my focus to writing posts in the “misc” and “random-thoughts” categories.

I also changed the theme of this blog to the current one, and used it ever since. It is a custom-made theme, and I purposely don’t share it. Don’t ask.

Because this year was more about thinking, rather than making, it might feel less productive, but I made a lot more posts than in the previous year.

One of my favorite posts from this year is about code coverage. I still think that 100% is the only acceptable coverage if you know why you measure coverage at all.

More Fennel stuff this year though. I’ve participated in my first ever game jam, and made a decently-looking game (at least I think so). I also made a library for evaluating Fennel from Org Mode source code blocks, and turned it into a kind of a notebook.

Finally, I made a small library for Clojure, that fixes a long-standing bug in the core.async library. Sadly, this patch wasn’t accepted by Clojure maintainers, but I was expecting that.

Overall, not a bad year. I only had three major projects, but I kinda wanted to go for quality over quantity.

2023

And I went back to quantity over quality pretty quickly.

For some reason in 2023, I felt a huge urge to write. Forty-four posts, including the hidden ones!

Another bad async implementation, now in Emacs Lisp.

A new REPL implementation for Fennel.

Clojure’s core.async port to Fennel! (Finally, a decent one. All it took was to actually look inside one of the libraries actually used in the industry.)

And of course, gamedev.

This year I decided to really get into game development, so I set myself a goal - make five games, each in one month.

The first game was a basic platformer. I was pumped up and worked on this project a lot. Probably, the longest I ever worked on a game before or to this day. After finishing it (stopping would be a more accurate word), I realized how hard it is to pump games out each month.

The second game didn’t go as planned at all. I was so burned out from the previous game, that I couldn’t bring myself to work on the second one. So the only obvious solution was to start working on the third one, right?

The third game I actually managed to complete to the point of having some kind of a game loop. Not a good one, but give me some slack - I have been programming non-stop day and night for three full months now.

The fatigue was real though. I decided to abandon this self-assigned challenge, after making a portion of a game and deleting everything in frustration. I’ve later written a postmortem on all this marathon thing, and to this day I’m recovering and it’s really hard for me to get back into game development. Maybe this year, who knows?

Apart from gamedev, I decided to revisit the language development field. So I started working on language by following the Crafting Interpreters book. This was the project I spent most of my time remaining in 2023.

2024

The year started with me and my wife getting divorced. I’ve finished the Crafting Interpreters book, but this was the last push - I haven’t any energy left to continue with the blog. I made a post reflecting on this blog but later hidden it away. I wasn’t planning on feeling down for long, just needed a break from things. It’s still there to remind me of my past self.

So I took a break for a full month until I felt the urge to write again. It started slow, mostly with posts about thinking on various programming topics. And hobbies.

And I rediscovered my passion for a different hobby I had years ago - playing guitar and recording videos about it!

Your browser does not support the video tag.

Code Snippet 1: Intro cover of A Martian Winter by Angel Vivaldi (Warning - it's a bit loud)

Then an inspiration struck again and I started working like a crazy man on a new passionate project. I started working on an asynchronous HTTP client for Fennel, actually applying all of my knowledge and previous projects that I made. I worked on it almost til the end of the year.

I dipped my toes into game development for a brief moment again.

One of the things I reflected upon was doing things blindly. That was the style of learning I applied til the last year. This year was different, I was reading specs, different implementations, and learning from stuff people made instead of reinventing the wheel. Hopefully, this method will stay with me and evolve further.

Another spontaneous project was about bringing Fennel into Emacs as a second extension language. While working on it, and preparing for the FennelConf 2024 where I wanted to talk about fnl-http I suddenly found myself working on yet another project that I already made several attempts at - the dependency manager. So, somehow by the end of the year, I was working on three projects at once again.

2025 and beyond

And that’s what brings us to 2025! I’m still working on this dependency manager project, and I have high hopes that it will be useful to the Fennel community. Not sure what other projects this year will bring, but I hope they will produce exciting and interesting posts!

All in all, starting this blog five years ago made me who I am today. My only advice to you - if you want to start doing something, but you’re waiting til you’re ready, then do not wait! Act, and you will see that you’re probably already capable of doing what you always wanted to do. And even if you’re not, you will clearly see what parts you have to work on. And others will help you with that.

If you’ve been here for the past five years, I gladly appreciate that a lot. A lot of comments were sent to me in 2024, helping me make this blog better, and move through life. And if you’re just passing by, I’m still glad that you made it here!

Thank you for reading!

Comment via email

deps.fnl 0.2.2 released

Sun, 16 Feb 2025 06:30:00 +0300

I just released a new version of the deps.fnl project!

After getting some feedback on the 0.1.0 version and addressing most of it, I’m ready to present a new version with improved dependency and conflict handling.

Major changes

The `deps.fnl` file format

The :deps field was changed back to use maps, as it was during the prototyping stage:

{:deps {"https://gitlab.com/andreyorst/async.fnl"
        {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}
        "https://gitlab.com/andreyorst/json.fnl"
        {:type :git :sha "eebcb40750d6f41ed03fed04373f944dcf297383"}
        "https://gitlab.com/andreyorst/reader.fnl"
        {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}
        "luasocket"
        {:type :rock :version "3.1.0-1"}}
 :paths
 {:fennel ["src/?.fnl" "src/?/init.fnl"]}}

Before this change the deps.fnl file would have looked like this:

{:deps [["https://gitlab.com/andreyorst/async.fnl"
         {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}]
        ["https://gitlab.com/andreyorst/json.fnl"
         {:type :git :sha "eebcb40750d6f41ed03fed04373f944dcf297383"}]
        ["https://gitlab.com/andreyorst/reader.fnl"
         {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}]
        ["luasocket"
         {:type :rock :version "3.1.0-1"}]]
 :paths
 {:fennel ["src/?.fnl" "src/?/init.fnl"]}}

The main reason for a change to the use of vectors was the fact that all of the Luarocks dependencies were using the same tree. Because conflict resolution was made during dependency installation, and maps are unordered, and if any of the libraries had a Luarocks library as its transient dependency, Luarocks would silently override the dependency. This made conflict checking nondeterministic, and thus error-prone.

This is no longer an issue, and thus the format can be changed back to maps.

Conflict resolution

Previously, deps would use a deterministic order of dependencies and the --allow-conflicts flag to resolve conflicts. This meant, that if your project had two dependencies, that had the same transient dependency but of a different version, --allow-conflicts would honor dependencies that appeared earlier in the order. For example, if you had dependencies A, and B, both of which depended on C of versions v1 and v2 like this:

;; project deps.fnl
{:deps [["A" {:version "1"}]
        ["B" {:version "1"}]]}

;; A deps.fnl

{:deps [["C" {:version "1"}]]}

;; B deps.fnl

{:deps [["C" {:version "2"}]]}

The deps script would throw an error:

[A:1]->[C:1] conflicts with [B:1]->[C:2]

Previously, this could be resolved by using the --allow-conflicts flag, meaning that deps would see the conflict, but ignore it, and use the C library from the first one encountered in the deps.fnl. So for the following deps.fnl:

{:deps [["A" {:version "1"}]
        ["B" {:version "1"}]]}

The libraries used would be A version 1, transient C version 1 from A, and B version 1.

However, for the following deps.fnl (A and B appear in the reverse order):

{:deps [["B" {:version "1"}]
        ["A" {:version "1"}]]}

The libraries used would be A version 1, transient C version 2 from B, and B version 1.

This wasn’t clear enough, and could introduce unexpected results when editing deps.fnl.

For example, adding another dependency before A that (unknowingly) also depends on yet another version of C would override the version for the project.

At first, I was thinking about using :exlusions key to specify that we want to exclude certain transient dependencies, but it also introduced the need to do this at arbitrary depth:

{:deps [["A" {:version "1"}]
        ["B" {:version "1"
              :exclusions ["C"]}]]}

And we would still not know what version of C we’re using.

So instead, conflict resolution works by specifying the specific version of dependency C at the same level as A and B:

{:deps {"A" {:version "1"}
        "B" {:version "1"}
        "C" {:version "1"}}}

This way the project author is in full control of what versions will be used.

The --allow-conflicts flag is no more. By default, conflicts are checked, and the only way to resolve conflicts is either to update the library in question, or override it at the root level.

Conflict resolution is now module-based

Another new feature I’m pretty excited about is module-based conflict resolution. In the 0.1.0 version of deps it was not possible to detect conflicts when something is a fork.

What I mean by that is, say, I have a library: async.fnl and someone decides to fork it to change something or fix a bug. They add the following to their project deps.fnl:

{:deps {"https://gitlab.com/some-fork/async.fnl"
        {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}
        "https://gitlab.com/andreyorst/async.fnl"
        {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}}}

The main question thus was - what async.fnl implementation would actually be loaded? We could answer this question if :deps was a list instead of a table because the ordering would rule out and (require :io.gitlab.andreyorst.async) would load the first library. However, with deps stored as a map, we would have gotten different implementations each time because key hashing is randomized.

The problem is - previously deps didn’t see this as a conflict. These two libraries are the same library but different projects.

Now, instead of comparing just library names (URLs in this case) and versions (SHAs for Git dependencies), deps also compares exported modules. So we can detect the conflict because both libraries export io.gitlab.andreyorst.async module. If the person who forked the library chooses to change the module name to io.gitlab.some-fork.async instead, there won’t be a conflict, as this is now a properly different library.

The example above is unlikely to happen as is, but if some library uses a fork as a transient dependency this might happen. This was pointed out to me by Ambrose Bonnaire-Sergeant off the mailing list, and we’ve come up with this solution.

Luarocks (transient) dependencies

Another problem with transient dependencies is the fact that Luaorcks dependency manifests are very lax about versions of the transient dependencies. Usually, you would see just library names, and sometimes version ranges. For example, here’s a fragment from the lapis library manifest describing dependencies:

dependencies = {
  "lua",

  "ansicolors",
  "argparse",
  "date",
  "etlua",
  "loadkit",
  "lpeg",
  "lua-cjson",
  "luaossl",
  "luasocket",
  "pgmoon",
}

The pgmoon dependency itself depends on lpeg. So Luarocks default strategy is to pull the latest available version of the library.

This makes conflict resolution harder, because deps.fnl requires specifying a concrete version of any dependency. Paired with the fact that Luarocks silently overrides already installed dependencies when you give it a specific version to install, and the fact that deps.fnl is now unordered, the only way to get deterministic versions from Luarocks was to ask users to specify all of the transient dependencies:

{:deps {"lapis"
        {:type :rock :version "1.16.0-1"
         :dependencies
         {"loadkit" {:type :rock :version "1.1.0-1"}
          "etlua" {:type :rock :version "1.3.0-1"}
          "luasocket" {:type :rock :version "3.1.0-1"}
          "lua-cjson" {:type :rock :version "2.1.0.10-1"}
          "argparse" {:type :rock :version "0.7.1-1"}
          "lpeg" {:type :rock :version "1.1.0-2"}
          "ansicolors" {:type :rock :version "1.0.2-3"}
          "date" {:type :rock :version "2.2.1-1"}
          "pgmoon" {:type :rock :version "1.16.0-1"
                    :dependencies {"lpeg" {:type :rock :version "1.1.0-2"}}}
          "luaossl" {:type :rock :version "20220711-0"}}}}}

The :dependencies key is akin to the dependencies map in the rock manifest.

The conflict resolution works exactly the same.

Runtime library manipulation with `deps.add-lib`, and `sync-deps`

Something that escaped my previous announcement, because it was developed a bit later is that it is now possible to add libraries while working on the application without restarting the REPL. When the REPL is started via the deps --repl command, a new deps module is available:

>> (require :deps)
{:_VERSION "0.2.0"
 :add-lib #<function: 0x560a243c2990>
 :add-libs #<function: 0x560a245b5820>
 :sync-deps #<function: 0x560a246d8d50>}

Thus, if you’re working on some project and in need of a new dependency, or want to compare a few and choose which one to use, you can quickly do so in the REPL:

>> (deps.add-libs {"bump" {:type :rock :version "3.1.7-1"}
                   "anim8" {:type :rock :version "v2.3.1-1"}})
["bump" "anim8"]
>> (deps.add-lib
    "penlight"
    {:type :rock :version "1.14.0-2"
     :dependencies {"luafilesystem" {:type :rock :version "1.8.0-1"}}})
["pl.input" "pl.path" "pl.array2d" "pl.compat" "pl.luabalanced" "pl.pretty"
 "pl.import_into" "pl.dir" "pl.func" "pl.Map" "pl.comprehension" "pl.app" "pl.seq"
 "pl.sip" "pl.file" "pl.stringx" "pl.class" "pl.Date" "pl.strict" "pl.template"
 "pl.types" "pl.permute" "pl.MultiMap" "pl.text" "pl.utils" "pl.OrderedMap" "pl.xml"
 "pl.url" "pl.tablex" "pl.test" "pl.lapp" "pl.List" "pl.Set" "pl.operator" "pl.data"
 "pl.lexer" "pl.config" "pl.init" "pl.stringio" "lfs"]

The returned value is a table of module names that are now available to require.

Alternatively, the deps.fnl file can be edited, and changes can be synced into the running REPL via the sync-deps call.

New flags

Several new flags were added.

`--tree`

Prints the dependency tree, outlining what dependencies are on your path, and what transient dependencies are brought by each dependency:

[["lapis" "1.16.0-1"
  [["argparse" "0.7.1-1"]
   ["luaossl" "20220711-0"]
   ["pgmoon" "1.16.0-1"
    [["lpeg" "1.1.0-2"]]]
   ["date" "2.2.1-1"]
   ["luasocket" "3.1.0-1"]
   ["loadkit" "1.1.0-1"]
   ["ansicolors" "1.0.2-3"]
   ["lpeg" "1.1.0-2"]
   ["etlua" "1.3.0-1"]
   ["lua-cjson" "2.1.0.10-1"]]]
 ["andreyorst/fennel-test"
  "9aae4dd52be1f5dd1ee97e61c639e6c5a2f9afee"]]

`--fennel-ls`

The deps script is now able to create or update an existing configuration for the fennel-ls project.

When calling deps --fennel-ls a flsproject.fnl file will be created if it doesn’t exist already. This file will contain all of the PATH-related data:

{:fennel-path "test/?.fnl;.deps/git/andreyorst/fennel-test/9aae4dd52be1f5dd1ee97e61c639e6c5a2f9afee/src/?/init.fnl"
 :lua-version "lua54"
 :macro-path ".deps/git/andreyorst/fennel-test/9aae4dd52be1f5dd1ee97e61c639e6c5a2f9afee/src/?/init.fnlm"}

`--profiles`

Another new feature - profiles! Before this flag existed profiles were kind of possible by using --merge and additional deps.fnl files:

;; deps.fnl
{:deps {"https://gitlab.com/andreyorst/reader.fnl"
        {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}}
 :paths
 {:fennel ["src/?.fnl"]}}

;; dev-deps.fnl
{:deps {"https://gitlab.com/andreyorst/fennel-test"
        {:type :git :sha "72a74394f89fdaf15abf11f33752a07e9e0bee91"}}
 :paths {:fennel ["tests/?.fnl"]}}

Having these two files, a final deps map can be obtained with deps --merge dev-deps.fnl:

;; output of `deps --merge dev-deps.fnl --show`
{:deps {"https://gitlab.com/andreyorst/fennel-test"
        {:sha "72a74394f89fdaf15abf11f33752a07e9e0bee91" :type "git"}
        "https://gitlab.com/andreyorst/reader.fnl"
        {:sha "252ea2474cb7399020e6922f700a5190373e6f98" :type "git"}}
 :paths {:clua {} :fennel ["src/?.fnl" "tests/?.fnl"] :lua {} :macro {}}}

This works, but semantically it is a bit unclear. To know that the development “profile” exists you need to actively look for it in the project files. Multiple such profiles also pile up in the project, which is not great.

Now, this can be done a bit more consistently with:

;; deps.fnl
{:deps
 {"https://gitlab.com/andreyorst/reader.fnl"
  {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}}

 :paths
 {:fennel ["src/?.fnl"]}

 :profiles
 {:dev
  {:deps {"https://gitlab.com/andreyorst/fennel-test"
          {:type :git :sha "72a74394f89fdaf15abf11f33752a07e9e0bee91"}}
   :paths {:fennel ["tests/?.fnl"]}}}}

And the same final deps map can be obtained with deps --profiles dev:

;; output of `deps --profiles dev --show`
{:deps {"https://gitlab.com/andreyorst/fennel-test"
        {:sha "72a74394f89fdaf15abf11f33752a07e9e0bee91" :type "git"}
        "https://gitlab.com/andreyorst/reader.fnl"
        {:sha "252ea2474cb7399020e6922f700a5190373e6f98" :type "git"}}
 :paths {:clua {} :fennel ["src/?.fnl" "tests/?.fnl"] :lua {} :macro {}}}

The --merge flag is here to stay for compatibility. Maybe there are other use cases for it too, so it won’t be removed unless deemed unnecessary. Moreover, --profiles actually supports multiple comma-separated profile names, so it is much more compact compared to --merge:

$ deps --profiles dev,test,doc ...
$ deps --merge dev-deps.fnl --merge test-deps.fnl --merge doc-deps.fnl ...

Upcoming work

I’ll probably take a short break from working on deps for a while, or at least I won’t work on it as actively. Both to let it settle a bit, give it time to gather some feedback, and let me rest a bit too. I’m feeling good about what I did so far, and deps in my opinion is out of the alpha stage for sure. So I think it can be safely used in other projects, and I’m still open for feedback!

As for plans, I have something I want to experiment with.

I’ve started outlining an interface for custom backend implementations. Right now there are only two backends: Git and Luarocks, but if the community desires, I’ll extend deps in such a way that custom backends could be added via some sort of a plugin system. Thus other ways to download and add dependencies would be possible. For example, recently I learned that some Lua libraries exist only in SVN. There may be others that are only available through GNU Bazaar, the new hot Jujutsu VCS, or maybe some use Mercurial, I don’t know!

Additionally, I think it is safe to say, that using version control systems for libraries is not a great idea, so I might start working on a dependency packaging system for Fennel libraries that we could later upload to some package archive. Thinking of a name - Fennel Bulbs sounds fun!

Dependency building needs to be changed to be a bit more robust. Right now I think it should work fine, but probably only on *nix-based systems. Windows support for deps.fnl is there, but completely untested. So if any Fennel users happen to do their stuff on Windows - let me know if it works, please!

That’s all for now. Huge thanks to Phil Hagelberg, Ambrose Bonnaire-Sergeant, and Emma B. for the feedback and code reviews!

Comment via email

deps.fnl - a new dependency manager for Fennel projects

Fri, 10 Jan 2025 00:47:00 +0300

I’d like to present a new project, aimed at one of the areas where the Fennel ecosystem can be improved - project dependency management.

Other programming languages have tools to pull in, build, and load external dependencies. Java has Maven, Clojure has Leiningen and other projects like this, Python has Pip and Poetry, and Lua has Luarocks. All these tools provide ways for pulling in dependencies concerning how the language is organized.

Since Fennel runs on Lua, we could use Luarocks, but, well, it’s not really good. It has a lot of design problems and doesn’t fit for Fennel for a number of reasons, like:

It is tailored for Lua dependencies only, so shipping Fennel libraries would require AOT compiling them to Lua;
Luarocks only knows how to build LUA_PATH and LUA_CPATH, but Fennel requires FENNEL_PATH and more importantly FENNEL_MACRO_PATH;
Shipping macros is hard or impossible.

A few notes on why I decided to make a dependency manager

I’ve made a lot of libraries for Fennel and encountered various problems with shipping these libraries to users.

Most of the Fennel libraries I write feature user-exposed macros, which means that the library can’t be shipped as a single file most of the time. I did find a workaround for that eventually, but it still poses some interesting challenges for library writers.

Another problem I encountered are libraries that need other libraries to work. Some of my libraries depend on other libraries I made, and thus I either need to include them in the source code directly for shipping or do some other trickery. And libraries depending on other libraries (transient dependencies) could potentially conflict with other dependencies in the project.

Finally, multifile libraries that can’t be compiled into a single file, since they have multiple modules that serve as entry points are hard to ship because users will have to properly set up PATH in their project.

So I made deps.fnl to solve these problems.

`deps.fnl`

This project consists of a single script, called deps, that works as a drop-in replacement for the fennel executable. This means that you can call deps with any arguments that you specify to Fennel, and it will work transparently. In other words:

deps --require-as-include --compile file.fnl > file.lua

But the main thing that deps was designed to do is to set up the environment for your project. When the deps.fnl file is present in the project, the deps script analyzes this file and pulls in all of the specified libraries, sets all required PATH variables, like LUA_PATH, LUA_CPATH, FENNEL_PATH, and FENNEL_MACRO_PATH, and launches fennel.

For example, here’s a deps.fnl for my fnl-http project:

{:deps
 [["https://gitlab.com/andreyorst/async.fnl"
   {:type "git" :sha "a83b13b397fdfab3ebf7f17a16d21a7ec2674ceb"}]
  ["https://gitlab.com/andreyorst/json.fnl"
   {:type "git" :sha "bfb8d7d03c26619768eefaa47782a3e95dbe5156"}]
  ["https://gitlab.com/andreyorst/reader.fnl"
   {:type "git" :sha "3ff2bc790c8b7922267af5712a64a14572a172cf"}]
  ["luasocket"
   {:type "rock" :version "3.0rc1-2"}]]
 :paths
 {:fennel ["src/?.fnl" "src/?/init.fnl"]}}

As can be seen, this project depends on four libraries: async.fnl, json.fnl, reader.fnl, and luasocket. Previously I had to vendor these dependencies in the repository, and instruct people how to install fnl-http and use it as a library.

Moreover, fnl-http is a multi-file library, so manually setting up PATH for it and all of its dependencies is not a pleasant task. But with deps you can use this library in your project by placing this deps.fnl file at your project root:

{:deps [["https://gitlab.com/andreyorst/fnl-http"
         {:type "git" :sha "ea885a8c767a627206126cc36f1befb1a2d71850"}]]
 :paths {:fennel ["?.fnl"]}}

Then just call deps --repl, and you’re good to go:

$ deps --repl
processing git repo: https://gitlab.com/andreyorst/fnl-http
processing git repo: https://gitlab.com/andreyorst/async.fnl
processing git repo: https://gitlab.com/andreyorst/json.fnl
processing git repo: https://gitlab.com/andreyorst/reader.fnl
processing rock: luasocket 3.0rc1-2
Welcome to Fennel 1.5.1-dev on PUC Lua 5.4!
Use ,help to see available commands.
>> (require :io.gitlab.andreyorst.fnl-http.client)
{:connect #<function: 0x5605682104e0>
 :delete #<function: 0x560568bc64c0>
 :get #<function: 0x5605685315c0>
 :head #<function: 0x5605689c9ca0>
 :options #<function: 0x5605682b0a20>
 :patch #<function: 0x5605687d8f40>
 :post #<function: 0x560568c40270>
 :put #<function: 0x5605681d9d80>
 :request #<function: 0x560568b37450>
 :trace #<function: 0x560568580320>}

In the output above after analyzing the project’s deps.fnl the deps script proceeds to download dependencies. The project’s only dependency is fnl-http, but after downloading it, the deps script check if there’s a deps.fnl in the root of the repository. Since there is, it proceeds to download all transient dependencies. Calling deps --repl the second time would not download anything, unless new dependencies were added to the deps.fnl file or hashes/version were changed.

After processing all of the dependencies the REPL is started as requested, and the library is ready to be used.

Dependency types

Dependencies can be of two types - git and rock.

Git dependencies are just a link to the repository and a commit hash to checkout. Until we have some kind of a registry and a way to package libraries for downloading this is the best we can do. But as seen above, this already makes using libraries a lot simpler.

Rock dependencies are processed with Luarocks. Yes, deps wraps not only the Fennel executable but Luarocks as well. Luarocks is fine enough to be used like this and provides enough command-line arguments for deps needs.

All of the dependencies are stored in the project’s root unless requested elsewhere. I decided not to go the .m2 route. One reason for that is the fact, that Luarocks can’t install two versions of the same library into a single “tree”, as they call it. Instead, it overrides any library already present in the tree. This is probably related to how the LUA_PATH is constructed by Luarocks.

Speaking of PATH=⁣s. The =deps.fnl file contains a mandatory field :paths, that contains a table. This table specifies the project’s path configuration. Each library may require a different path configuration, and when deps script processes a said deps.fnl file, it reads the path table and constructs the final PATH for your project.

You can analyze the path generated for the project with deps --path. It prints the path to standard out in a format similar to luarocks path.

Finally, some Git repositories may not use deps.fnl, but it is still possible to manage them with deps.fnl. For example, my library fennel-test doesn’t yet feature a deps.fnl file, but the deps script itself uses it for testing. To make this work, you can specify :paths field for each dependency:

{:deps [["https://gitlab.com/andreyorst/fennel-test"
         {:type :git
          :sha "01ea080dc8176512c5890462252c5bf214baf137"
          :paths {:fennel ["?.fnl"] :macro ["?.fnl"]}}]]
 :paths {}}

Thus, we can even pull in pure Lua dependencies, and specify the :paths table with something like {:lua ["?.lua"]}, and deps will set everything up.

You can read more information on the deps.fnl file format, and deps script usage in the project’s readme.

I need feedback

I have high hopes for this project, and I’m looking for feedback. If you’re working on some application that requires external dependencies, give deps.fnl a shot. If you’re a library maker like me, consider using deps.fnl so your users have an easier way of using your work. You can submit feedback either via email, or GitLab issues.

If the project gets enough traction and is to community’s satisfaction, Phil and I are discussing the possibility of including this into Fennel itself, so no installation would be required. Since deps is already a drop-in replacement for fennel, once deps is a part of the fennel executable, all commands from the above would work with just fennel itself.

Thanks for reading, and I hope this project will be useful to you!

Comment via email

Tiny OOP in Lua

Tue, 07 Jan 2025 23:39:00 +0300

Recently I was participating in a conversation about popular programming languages and I brought up Lua. I was met with a response like this:

Lua? Can you even do anything with this language?

The conversation then shifted towards discussing modern dynamic languages, mostly Python, and how it gives you everything Java does without being as complicated as Java.

I’m not a Python fan - in fact, I think Python is a badly designed language and should be avoided. But people tend to like it for some reason. I also think that Lua is a much better language and it’s worth considering it before jumping to Python or alternatives.

I do agree that Lua is much smaller than Python, and it can do less by itself, as there are no batteries included in the language. But it also means that you can bring in only what you need for the project.

And Lua is extremely flexible too. When the conversation shifted towards discussing Python, one particular thing was discussed the most - Python is object-oriented. Lua isn’t, as there are no classes, no inheritance, or such. I brought up Lua again in the conversation, saying that it’s quite easy to add object orientation to Lua if you need it for your project. People were skeptical so I decided to write this post.

Most of the code here I wrote on my phone while guys were arguing about Python, so I wasn’t participating in the conversation much. After I was done, I showed them the following example, but the only response I got was a yet again skeptical “neat”.

Neat.

So here’s a basic OOP-style code in Lua.

First, we need a class:

local Shape = {}

It’s just a table, but now let’s add a new method to it and do some metatable trickery:

function Shape:new(obj)
    self.__index = self
    return setmetatable(obj or {}, self)
end

Now we can create an object like this:

local s = Shape:new{}

It’s still a table, but it has Shape as it’s metatable. It means that we can add methods directly to a class, and all objects would share them:

function Shape:setOrigin(x, y)
    self.x, self.y = x, y
end

function Shape:getOrigin()
    return self.x, self.y
end

We can set the shape’s origin, and then obtain it with these methods:

s:setOrigin(1, 1)
print(s:getOrigin()) -- 1, 1

Note on self: in Lua self is an ordinary variable name, that is implicitly declared when we use method declaration syntax. In other words, these two function definitions are the same thing:
function Shape:getOrigin() -- the colon after Shape
    return self.x, self.y
end

function Shape.getOrigin(self) -- the dot after Shape
    return self.x, self.y
end
It’s just a bit more convenient to write Class:method(...) than Class.method(self, ...) when defining a function. And when we’re calling a method with : instead of . it automatically passes the object as the first argument. So again, these are the same:
s:setOrigin(1, 1)

s.setOrigin(s, 1, 1)

But what makes objects special is the ability to build upon objects using inheritance. So let’s create a new class that inherits all of the methods of Shape and provides some new on its own:

local Rectangle = Shape:new{}

function Rectangle:setWidth(width)
    self.width = width
end

function Rectangle:setHeight(height)
    self.height = height
end

function Rectangle:area()
    return self.width * self.height
end

We’ve defined a new class Rectangle that inherits all of the Shape methods:

local rect = Rectangle:new{width = 2, height = 4}

rect:setOrigin(3, 4)

print(rect:getOrigin()) -- 3, 4
print(rect:area())      -- 8

Here’s how this works:

Rectangle inherits new from Shape like any other method. This time, however, when new executes, the self parameter will refer to Rectangle. Therefore, the metatable of rect will be Rectangle, whose value at __index is also Rectangle. So, rect inherits from Rectangle, which inherits from Shape.

We can go even further, defining a class that would inherit from Rectangle but first, let’s add two helper functions:

local function implements(obj, class)
    local mt = getmetatable(obj)
    if nil ~= mt then
        if class == mt.__index then
            return true
        else
            return implements(mt.__index, class)
    end end
    return false
end

local function super (class)
    return (getmetatable(class) or {}).__index
end

The first function implements is a predicate that checks if a given object implements a given class:

print(implements(rect, Shape))     -- true
print(implements(rect, Rectangle)) -- true

The second function super allows us to access a superclass of a class. For example, to call a superclass method, like a constructor.

We can create a new class that inherits from Rectangle and use it to alter the inherited constructor:

local Square = Rectangle:new{}

function Square:new(obj)
    self.__index = self
    local obj = super(self):new{width = obj.width, height = obj.width}
    return setmetatable(obj, self)
end

You can see that Square implements all of the methods:

local square = Square:new{width = 10}

print(implements(square, Shape))     -- true
print(implements(square, Rectangle)) -- true
print(implements(square, Square))    -- true

square:setOrigin(5, 6)
print(square:getOrigin()) -- 5, 6
print(square:area()) -- 100

This isn’t a new kind of trick though. All of this was greatly described in the Programming In Lua book. In fact, I straight up took the explanation from this page and adjusted class names in it to represent the code on this page.

Now, of course, this leaves out a lot of stuff like encapsulation, and multiple inheritance, but you can build on this simple concept further if you really need to. Moreover, there are multiple ways of achieving object orientation in Lua. You can define the object’s methods in five different ways. But in my practice even what I did here is way more than I ever needed.

What I wanted to show here is the flexibility that Lua provides you with. If you need something that’s missing in pure Lua, the same pure Lua probably has a way to add that to the language. Over the years I’ve created lazy sequences, immutable data structures, and asynchronous channels all in pure Lua. With C modules you can add even crazier stuff.

Lua is a very capable language. The community made a lot of libraries. If you want so-called “batteries” there’s Penlight. If you’re making games, you have LÖVE2D that gives you its own batteries and a plethora of small Lua libraries for game-making are available too.

If you’re not a fan of the syntax or want more features in the base language you can choose amongst other languages compile to Lua. For example, there’s MoonScript which is whitespace-based like Python. Or there are Fennel and Urn if you’re into lisps. And probably other languages.

Lua is fast, has a small footprint, is easily embeddable, and is very flexible. It’s a shame it’s not taken seriously most of the time.

Comment via email

2024 Recap

Sun, 29 Dec 2024 21:17:00 +0300

What a year!

Not as productive as last year, with only 18 posts this time (19 including this one), but I decided to post less for several reasons.

The first one, is that right at the start of this year I had some events in my personal life that had an impact on my passion for writing. I’ve pretty much recovered since, but this explains some quietness in the blog during February.

The second one is that I felt a strong burnout from programming. I do programming for a living, it’s also my hobby, and they say “Find A Job You Love And You’ll Never Work A Day In Your Life”, but it doesn’t really work. I love programming, really do, but it takes a lot of mental energy, and when you’re writing code for your whole day at work, and then come home and write code for your projects fatigue tends to pile up. So I took a break from Fennel, and pretty much focused on Clojure and work stuff. I also became a software architect this year and had to adjust my way of thinking a bit, so programming is no longer the main task at my job, and after a bit of rest, I was finally ready to go back to coding in my spare time.

Before that happened though I bought myself a new guitar, and finally put up my first pedalboard! Look at this beauty:

Figure 1: Signal chain: Fender Telecaster → Polytune mini 3 → Yellow Comp → Pitch Box → Blues Driver → Mojomojo→ Flashback 2 → Hall of Fame 2 → Orange Crush 20 RT

Now I have three guitars - two electric guitars and a bass. Getting a new guitar (the red one in the photo) was a great way to kick-start my interest in playing again, as I hadn’t done so for the last five or so years. I bought a Japanese-manufactured Fender Telecaster made in 1993 for a moderate price, and I love it! Plays and feels really great - I started to record myself again and made a small Telegram channel to post stuff about it, as I don’t really want to overwhelm GitLab with video files. But here’s a short one:

Your browser does not support the video tag.

Code Snippet 1: Intro cover of A Martian Winter by Angel Vivaldi (Warning - it's a bit loud)

I have used modelers for most of my time playing guitar, meaning that I would just plug my guitar into an audio interface, and add virtual implementations of various gear to shape my sound. Like virtual overdrive and delay pedals, virtual amps, cab sims, etc. So getting myself a real pedalboard was a step up from that in some sense, as I love analog interfaces with knobs. VSTs just didn’t do that for me. However, I still use emulation when recording.

Haven’t read much this year. Mostly manga, but again, I was kinda overwhelmed with life stuff, so it was a small escape route for me.

I had a lot of comments this year! 20 unique emails, not including replies and continued discussions. That’s even more than there were posts! I’m extremely glad that everyone commenting, it makes me feel much better about this blog being read, and not just existing for no one.

Managed to participate in FennelConf 2024. Prepared a talk about dependency management and my latest crazy package for Emacs require-fennel.el. It was cool as always, and I’m happy to be a part of this amazing community.

Now, for the next year, I have some plans.

I finally decided to learn some music theory, so that’s one thing I’m excited about. I’ve been playing guitar for more than ten years now, but only recently decided to “up my game” a bit, and actually figure out what I’m doing.

As for this blog, and my projects - not so long ago I started working on a game in LÖVE2D, so maybe expect some dev-logs next year. I don’t know though, making games is hard, and I only work on this project when I have the passion to, so it may take a long time for anything to be made public. Maybe not even in the next year. We’ll see.

And that’s it! I wish you a happy new year, and excited to see what the next year will bring us! See you in 2025!

Comment via email

Extending Emacs with Fennel

Fri, 20 Dec 2024 03:23:00 +0300

After watching this year’s EmacsConf and seeing Guile Emacs being resurrected I thought to myself - why limit ourselves to Guile? Sure, Guile isn’t just a Scheme implementation, thanks to its compiler-tower-based design. Other languages exist for Guile VM, such as Emacs Lisp, and Guile manual lists the following languages with various stages of completeness:

ECMAScript
Brainfuck
Lua
Ruby
Python

Sure, it would be nice, if Emacs could natively run all of these, but we have to understand, that Guile Lua is not PUC Lua. Even LuaJIT has difficulties maintaining full compatibility with modern releases of Lua, having us stuck in the realm somewhere between Lua 5.1 and 5.2, while PUC Lua is already 5.4. Future releases would bring more differences, making PUC Lua and Luajit diverge even more.

And Lua is a simple language, unlike something like JS, Ruby, or Python. So I wouldn’t bet on the fact that Guile Ruby would be exactly the same as Ruby. Maybe it will, but it’ll probably take years.

So I thought, why not just use Lua as is? The only thing we can’t do is to run Fennel functions from Emacs Lisp. There were attempts at bringing Lua VM into Emacs, running in the same process, however, as I know, it isn’t stable enough.

However, Emacs can connect to a Fennel REPL, like it does for many other languages, such as Common Lisp or Clojure. Sure, the language is running in a separate process, and this comes with a lot of nuances, but it’s still a possible route.

So I made a small package: require-fennel.el. It’s capable of loading Fennel (or Lua) modules, and defining a set of functions on the Emacs side for every Fennel function in the module.

Running Fennel from Emacs Lisp

For example, we can load Fennel itself into Emacs like this:

(require-fennel fennel)

Now we have all sorts of functions available, for example, here’s fennel.eval:

(fennel.eval "(fcollect [i 1 10] (* i i))")
;; => [1 4 9 16 25 36 49 64 81 100]

Or, we can create a file greet.fnl:

(fn greet [name]
  "Greets NAME with a message."
  (print (.. "Hello, " name " from Fennel!")))

and load it:

(require-fennel greet)

(greet "Andrey")
(require-fennel foo)
(require-fennel vec-add)

Code Snippet 1: "Hello, Andrey from Fennel!" will be displayed in the echo area of Emacs

Let’s ask Emacs to describe the greet function:

greet is a interpreted-function.

(greet NAME)

Greets NAME with a message.

As can be seen, Emacs can show us the function signature and its documentation string obtained from Fennel.

Unlike Emacs Lisp, Fennel functions support destructuring with special syntax based on how data literals are written in Fennel. For example, let’s create a function that accepts a map, and destructures its keys:

(fn foo [{:x x :y y}]
  "Accepts a map with the X and Y keys, and adds them together."
  (+ x y))

After loading it in Emacs we’ll see the following documentation:

foo is a interpreted-function.

(foo ((:y . y) (:x . x)))

Accepts a map with the X and Y keys, and adds them together.

I took some creative liberties and made it so Fennel hash tables are represented as association lists in Emacs Lisp. So the ((:x . 1) (:y . 2)) in Elisp is equal to {:x 1 :y 2} in Fennel. The argument list on the Emacs Lisp side is a bit unconventional, but I think that’s OK.

The same goes for vector destructuring:

(fn vec-add [[x0 y0] [x1 y1]]
  "Adds two 2D vectors."
  [(+ x0 x1) (+ y0 y1)])

Emacs’ documentation uses lists:

vec-add is a interpreted-function.

(vec-add (X0 y0) (X1 y1))

Adds two 2D vectors.

I couldn’t make it work with vectors for some reason, and as can be seen, Emacs tries to upcase parameters, but it doesn’t recognize that y0 is also a parameter. Probably fine, as such documentation would never be generated for Emacs Lisp functions.

We can call such a function with both vectors and lists:

(vec-add [1 2] [1 2])   ;=> [2 4]
(vec-add '(1 2) '(1 2)) ;=> [2 4]

That’s because Fennel doesn’t have a list datatype, it only has associative and sequential tables. Thus I’m using Elisp vectors as the result type.

Data conversion

Here’s how conversion table when passing data from Elisp to Fennel:

Elisp	Fennel
`[1 2 3]`	`[1 2 3]`
`(1 2 3)`	`[1 2 3]`
`((:foo . 1) (:bar . 2))`	`{:foo 1 :bar 2}`
`#s(hash-table test equal data ("foo" 1 "bar" 2))`	`{:foo 1 :bar 2}`
`:foo`	`"foo"`
`'false`	`false`
`"foo"`	`"foo"`

And here’s the conversion table when receiving data from Fennel in Emacs:

Fennel	Elisp
`[1 2 3]`	`[1 2 3]`
`{:foo 1 :bar 2}`	`((:foo . 1) (:bar . 2))`
`(values 1 2 3)`	`(1 2 3)`
`(fn [])`	`(lambda (&rest args) ...)`
`userdata`	`#<udata: 0x55b66bd6ac28>`
`true`	`t`
`nil`, `false`	`nil`

A small note about hash tables and multiple value returns. As can be seen, both are returned as lists. I specifically chose the cons notation for association lists, i.e. (a . b) because this way we can tell apart hash tables and mustivalues. There are no cons cells in Fennel, so no other data structure will be represented like that.

Because multiple values are returned as lists, we have to call apply, if we wish to compose two Fennel functions. In other words, in Fennel we can do this:

(fn rest [_ ...] ...)

(fn first [x _] x)

(first (rest 1 2 3)) ;=> 2

{: first : rest}

Code Snippet 2: fr.fnl

In Elisp, however, we would have to do:

(require-fennel fr)

(apply #'fr.first (fr.rest 1 2 3)) ;=> 2

If we were to call it as (fr.first (fr.rest 1 2 3)) the result would be [2 3], because fr.rest returned a list of return values, and lists are converted to vectors or hash maps depending on their structure.

That’s the basic gist of this package.

More examples

After implementing it I was like a kid in a toy store. With a childish grin, I started loading all my crazy Fennel libraries that I made over the years.

For example, here’s cljlib:

(require-fennel cljlib)

(cljlib.conj [1] 2) ;=> [1 2]

Or, here’s a JSON parser I made:

(require fennel json)

(json.decode "{\"foo\": [1, 2, 3]}")
;; (("foo" . [1 2 3]))

(json.encode '(("foo" . [1 2 3])))
;; "{\"foo\": [1, 2, 3]}"

Here’s a crazy one - I’m loading my Fennel port of a Clojure library clj-http into Emacs, and calling it from Elisp:

(require-fennel io.gitlab.andreyorst.fnl-http)
(require-fennel io.gitlab.andreyorst.fnl-http.client :as client) ; nested module

(client.get "http://httpbin.org/get" '((:as . :json)))

;; => (("headers" . (("Content-Length" . "198")
;;                   ("Access-Control-Allow-Credentials" . t)
;;                   ("Access-Control-Allow-Origin" . "*")
;;                   ("Content-Type" . "application/json")
;;                   ("Connection" . "keep-alive")
;;                   ("Server" . "gunicorn/19.9.0")
;;                   ("Date" . "Mon, 16 Dec 2024 22:56:35 GMT")))
;;     ("reason-phrase" . "OK")
;;     ("length" . 198)
;;     ("protocol-version" . (("name" . "HTTP")
;;                            ("minor" . 1)
;;                            ("major" . 1)))
;;     ("request-time" . 214)
;;     ("trace-redirects" . [])
;;     ("status" . 200)
;;     ("body" . (("args" . [])
;;                ("headers" . (("Host" . "httpbin.org")
;;                              ("X-Amzn-Trace-Id" . "Root=1-6760b023-39ca8e413573df5d14d09486")))
;;                ("url" . "http://httpbin.org/get"))))

Code Snippet 3: I've formatted nested alists to use dot notation for better reading clarity. In other words changed (a (b . c) (d . e)) into (a . ((b . c) (d . e))) which are the same thing.

The client made an HTTP request from the Fennel side and passed the response back to Emacs. The response body was in JSON, but specifying ((:as . :json)) as options map forced parsing the response body into Fennel tables. Which, upon arriving in Emacs Lisp were converted into association lists.

So now we can call Fennel from Elisp. But why stop there? Let’s call Emacs Lisp from Fennel!

Running ELisp from Fennel?..

Since this package uses the Fennel Proto REPL, which is a custom protocol that can be extended at runtime, we can create a set of functions that will allow us to call back to Elisp. Here’s how we extend the protocol:

(let [format-elisp _G.___repl___.pp]
  (protocol.env-set!
   :emacs {:call (->> {:__index
                       (fn [_ k]
                         (fn [...]
                           (protocol.message
                            [[:id {:sym protocol.id}]
                             [:op {:string :require-fennel/call}]
                             [:fun {:sym k}]
                             [:arguments {:list (fcollect [n 1 (select "#" ...)]
                                                  (format-elisp (pick-values 1 (select n ...))))}]])
                           (. (protocol.receive) :data)))}
                      (setmetatable {}))
           :var (->> {:__index
                      (fn [_ k]
                        (protocol.message [[:id {:sym protocol.id}]
                                           [:op {:string :require-fennel/var}]
                                           [:var {:sym k}]])
                        (. (protocol.receive) :data))}
                     (setmetatable {}))
           :eval (fn [expr]
                   (protocol.message [[:id {:sym protocol.id}]
                                      [:op {:string :require-fennel/eval}]
                                      [:expr {:sym expr}]])
                   (. (protocol.receive) :data))})
  {:id 1000 :nop ""})

I had to update the protocol core a bit, adding the protocol.env-set! and protocol.receive functions. The env-set! allows us to create user-visible definitions. So we add the emacs table that holds three fields.

The call field is a table with some metatable trickery. Since it’s an empty table, any key would trigger the __index metamethod. This metamethod can be another table or a function - in our case, it is the latter. This function returns a handler that calls into protocol.message that sends a custom OP require-fennel/call that we can handle in our package.

The same goes for the var field, except it doesn’t return a function, it just returns a value.

The last field, eval, is a combination of both, kinda. It is a function, not a table, and it accepts one argument - an expression to evaluate in Emacs.

However, we can’t use these just yet. The second part of the trick comes from the new ability to extend the main protocol handler with custom handlers:

(cl-defgeneric fennel-proto-repl-handle-custom-op (_op _message _callbacks)
  "Handler for a custom protocol OP.
The first argument OP is used for dispatching.  Accepts the whole
MESSAGE, and its CALLBACKS.  The default implementation of unknown OP is
a nop."
  nil)

With this generic function, we can define new handlers in separate packages. These are the methods I’ve added in require-fennel.el:

(cl-defmethod fennel-proto-repl-handle-custom-op ((op (eql :require-fennel/call)) message callbacks)
  "Custom handler for the require-fennel/call OP.
Accepts a MESSAGE and its CALLBACKS.
The MESSAGE contains a function to evaluate and its arguments."
  (fennel-proto-repl-send-message
   nil
   (format "{:id %s :data %s}"
           (plist-get message :id)
           (apply (plist-get message :fun) (plist-get message :arguments)))
   nil))

(cl-defmethod fennel-proto-repl-handle-custom-op ((op (eql :require-fennel/var)) message callbacks)
  "Custom handler for the require-fennel/var OP.
Accepts a MESSAGE and its CALLBACKS.
The MESSAGE contains a var which value is then returned to Fennel."
  (fennel-proto-repl-send-message
   nil
   (format "{:id %s :data %s}"
           (plist-get message :id)
           (require-fennel--elisp-to-fennel (eval (plist-get message :var))))
   nil))

(cl-defmethod fennel-proto-repl-handle-custom-op ((op (eql :require-fennel/eval)) message callbacks)
  "Custom handler for the require-fennel/var OP.
Accepts a MESSAGE and its CALLBACKS.
The MESSAGE contains an expression which is evaluated and its result is returned to Fennel."
  (fennel-proto-repl-send-message
   nil
   (format "{:id %s :data %s}"
           (plist-get message :id)
           (require-fennel--elisp-to-fennel (eval (plist-get message :expr))))
   nil))

These three fields and methods allow us to do anything with Emacs, and it happens in the same process, so we can access the full Emacs state. For example:

;; Welcome to Fennel Proto REPL 0.6.0
;; Fennel version: 1.5.1-dev
;; Lua version: PUC Lua 5.4
>> (emacs.call.emacs-uptime)
"19 minutes, 17 seconds"
>> emacs.var.emacs-version
"31.0.50"
>> (emacs.eval "(with-current-buffer \" *fennel-elisp*\" (buffer-substring-no-properties (point-min) (point-max)))")
";; Welcome to Fennel Proto REPL 0.6.0
;; Fennel version: 1.5.1-dev
;; Lua version: PUC Lua 5.4
>> (emacs.call.emacs-uptime)
\"19 minutes, 17 seconds\"
>> emacs.var.emacs-version
\"31.0.50\"
>> (emacs.eval \"(with-current-buffer \\\" *fennel-elisp*\\\" (buffer-substring-no-properties (point-min) (point-max)))\")
>> "
>>

Crazy!

Fennelmacs

So now we can write Fennel scripts that can obtain data from Emacs, process it, and pass back the results. Thus, our goal is achieved - we’ve added a second language to Emacs!

Of course, it’s not the same as what Guile Emacs does. We’re still running code in a separate process, adding communication between the two via some protocol. It’s a neat trick, nothing more. Guile Emacs, if it ever becomes the thing, will be more powerful, as we would be able to leverage all these different languages in the same process, possibly sharing the memory, calling functions interchangeably, etc. Some may even write Emacs packages in Brainfuck, who knows.

Still, this is really cool, in my opinion, and shows the power of Emacs pretty well. If you’re into Fennel, like me, I hope you’ll enjoy using this package as I enjoyed making it. See you at Fennel Conf!

Comment via email

Doing things blindly

Tue, 10 Dec 2024 23:45:00 +0300

During the course of my life, I was mainly doing things blindly. Not that I’m actually blind, what I mean is that I didn’t study that much. Instead, I always preferred to rediscover things by myself. For the most part.

When I was little, I didn’t have any particular hobbies. My childhood mainly consisted of playing outside with friends, watching cartoons on TV, and playing various games on my PC. I wasn’t good with sports, due to some health conditions, and did not participate in any of the activity clubs, like handicraft clubs, drawing clubs, etc.

My parents tried though. Once they suggested that I try joining a pottery class, but I couldn’t stay because I was allergic to the paint they used. Well, I did go to the English club for maybe 8 years, but it wasn’t my choice, my parents thought that it would be beneficial if I learned a second language, that is widely used. They were totally right, but I didn’t understand that up until I was twenty-ish or so. Still glad they did.

Going back to my childhood, things changed when I discovered Macromedia Flash. I would consider drawing animations my first serious hobby. However I never went to the arts school, and I just did things the way I did them. I wasn’t very good at animating, though I did learn a lot by myself, and If needed I can animate. If you’ve seen the games I made recently, I’d say that animation skills obtained when I was toying with Flash were beneficial to me today.

Later I asked my parents and went to the arts school. I was studying there for two years before going to the university, as I was hoping to become a professional animator. That didn’t really work out, I was told that I was too old for them to accept me to the full program, so I took courses that would provide me with some kind of certificate that should be enough for some universities, or so I was told. However, most universities in our country hold an entry contest, and you have to be able to draw pretty well to pass. So I didn’t bother with that and never went to become an animator.

Though, I think that’s good because after drawing small cartoons on my own for maybe 8 years straight I was kinda bored of animation. The results weren’t so good, and a mere 20 seconds of animation usually took a whole day of work. I was mainly doing frame-by-frame style animation, without much shortcuts, and at 24FPS so…

My second hobby was historical fencing. Yes, I said I wasn’t good with sports, but it wasn’t an official club, so they didn’t have any strict requirements. I just told the trainer that I can’t run for too long, and may need an extra break or three. They were fine with pretty much anyone, as the club was dying.

I did fencing for three years or so, and in my last year of high school, I decided to stop. I was going to move to a different city anyway, so I couldn’t continue here, and I wasn’t sure that I wanted to continue there. Again, the club didn’t teach us that much, as it was just a bunch of high schoolers swinging swords around. The club did participate in some local tournaments though, but I don’t remember us winning any. So my sword skills are something I developed mostly on my own, and I can’t say I’m that good.

Right at the time when I dropped fencing, I picked up a guitar. For the second time, though, as my parents tried to put me into the music school, as they noticed that I was able to hear some pitch.

I went to music school for two months, but couldn’t continue, as I was basically going into three full-blown schools now. The English club was serious, and I had to do a lot of homework for it. The ordinary school had a lot of homework as well. And now the music school that demanded hours of homework on top of that. So we decided to stop.

Also, I can’t sing. It’s funny because I hear that I can’t hit a single note with my voice and people often think that I have no musical ear, but that’s not true. I can tell when my guitar is out of tune, tune it by ear, I even can tune to E without a tuner, though I just memorized how the E string should sound, I don’t have perfect pitch. My inability to sing simply comes from the fact that I don’t know how to control my voice. It was a problem in the music school because teachers didn’t like the fact that when we were doing choirs I just pretended to sing by opening my mouth, as I was tired of them constantly yelling at me. At solfeggio, when the teacher asked us to sing the note they’d played at the piano I couldn’t do that, and I couldn’t really explain to them why. All I said was - I know it’s not the right note, I just can’t sing it. They were disappointed, but I got by sometimes by finding that note on the keyboard from memory when they let me.

When I was in the last year of high school I picked up guitar once more, but I didn’t go to the teacher. Instead, I decided to learn it on my own. So I did, by watching YouTube videos back in 2010. After grasping most of the concepts I began to learn tabulatures, as they didn’t require any knowledge of music theory besides some basic stuff. Like the signs for stops or articulation.

Now, here comes programming. When I got to the university, the department I entered was unique, if I may say so. It was unfocused, there was little specialization, and the way they were teaching us was… not great. After five years, I can’t say that I knew how to do anything. I wasn’t a bad student, I attended all the classes. Still, after graduating, I wasn’t sure what kind of job I should get.

Linux became another hobby for me. I have been using Linux as my main OS since 2008 or so. So naturally, I thought that I could find a job as a Linux administrator or something like that. After going to several interviews it was obvious that I knew nothing about it either.

Somehow I managed to find a job as a C programmer. We had a whole semester of C++ at the university - surely it’s enough to learn C++, right? What were they thinking?

Anyway, I started working as a C programmer, writing tests for microcontrollers and SOCs. I wasn’t good at that for maybe two years, and then something happened. First, I discovered Vim, and Vimscript took my interest for quite a while. I started making plugins and enjoying programming for the first time. Vimscript was nothing like C, and the only other languages I saw before were Delphi, taught to us at the university during the first year, and ActionScript 2 when I was tinkering with Macromedia Flash.

Vimscript ignited my interest in programming in general, and I gradually became more and more proficient with C. After four years I had a Senior title and decided to change my stack because C started to become tiring. So I studied Clojure for about two months and became a junior Clojure developer in a different company.

I think the fact that they weren’t really teaching us programming at the university affected me in a bad way because if they did, I would grow much faster than it took me. A lot of my friends from school started working when they were in their third year of the university, while I started another three years after them. By accident, even.

After that, I did a lot of experiments with other languages and expanded my knowledge, and now I feel pretty confident in my skills, but I still think that I could have learned most of the stuff at the university if the program had been better. The program was awful, really, trust me.

Going back to guitar for a second, I could say that I played guitar for 14 years now, but I did drop about five years ago, and just picked it up again recently. However, I didn’t improve much from what I could do, say, after three years of learning. I’ve plateaued.

I can feel it now because I felt it so many times.

With animation.

With fencing.

With programming.

And now with music.

With animation, I did quit because it wasn’t getting anywhere. If I did recognize that I’ve reached a plateau, and I won’t be able to improve unless I take it seriously and study, I would stay at that plateau for the rest of my life. I was too young to understand this, so I just quit.

With fencing, I quit not because I’ve reached a plateau, but for a different reason, however, if I were to continue it in the same way, as I were, it would end up like that. I still train occasionally, and I know my limits, and that I’m not really improving. Learning proper fencing is hard, and there’s a lot of theory that goes into it, but I never studied anything, so hence the result.

With programming, I’ve reached a plateau after two years of working, I couldn’t improve my C knowledge just by doing tasks. I was mindlessly hitting a wall most of the time. Luckily for me, I’ve managed to gain interest in studying programming - I read a few books and did a few side projects, and it was enough to break from the plateau and rise. I feel that I’m reaching another plateau, though, maybe this one is a different one, I still have to reflect on this.

With guitar, I’ve plateaued because I only played tabs. It’s a mere act of repeating what the sheet tells you. Yes, some musicians do just that for a living, but they still have a lot more knowledge than I am. Lately, I started studying music theory, bit by bit - reading some hints here, picking up some tricks there. Once I have a bit more understanding, I might take some actual lessons.

What I’m saying is - I did most of the stuff in my life without actually knowing what I was doing. I still do, occasionally. Sometimes it’s fun, sometimes it’s just a waste of time. And time is not a resource to spare. I’m in my thirties now, and I think it’s time to start taking things more seriously.

Comment via email

Extending JSON encoder with custom types

Mon, 04 Nov 2024 06:39:00 +0300

One thing I like about Lua is that it has a limited amount of built-in types and data structures. There are no objects, just tables. And if you need a custom behavior, a metatable can be attached to any table to extend it with custom methods that integrate with the rest of Lua runtime.

However, this is both a blessing and a curse. And unfortunately, Lua developers made a huge mistake by breaking this API¹ once already, in a minor release even.

As you may noticed, fnl-http has a lot of stuff. It has asynchronous Channels, and various Readers, which act like objects, but are still ordinary tables underneath. And this is true for a lot of Lua projects, tables are flexible enough to implement many cool things, so people do.

However, this poses an interesting problem - because objects are not a thing in Lua, and a set of default metamethods is quite limited, there’s no default way of extending objects with custom behavior. In Java, you would specify what interfaces are required for an object to have, and it often means that if something doesn’t implement them, a wrapper object is needed. In Clojure, we can get away with protocols and multimethods, but Lua has neither of those. In Lua, and subsequently Fennel, we have to be creative, because even though we can add stuff through metatables, not every value can have one.

While working on fnl-http, I added a basic JSON parser and encoder. The parser part is quite simple, as JSON has a specification and a fixed amount of data types to worry about. It’s not like EDN or XML, for example, which are both extensible data notations that can represent pretty much anything. In JSON you get only these data types:

Number
String
Array
Object
Boolean
Null

The only difference from Lua here is that null is nil and Lua doesn’t have arrays as a separate type. So implementing a parser was easy, and everything pretty much maps to Lua data structures one-to-one.

Implementing JSON encoder, however, was not as easy. For basic data types, it is easy enough, but once we touch custom objects created with metatables things get messy.

For example, let’s say we need a set data structure. A set is similar to a table, except the values are the same as the keys. But, we may want to distinguish them from ordinary tables, if, say, we’re making a library that has a set of functions for working with mathematical sets. Like intersection, difference, etc. Here’s a set constructor:

(fn Set [...]
  (let [data (faccumulate [res {}
                           i 1 (select :# ...)]
               (let [k (select i ...)]
                 (doto res (tset k k))))]
    (setmetatable
     {}
     {:__index data
      :__newindex (fn [_ k _] (tset data k k))})))

As you can see, we create an empty table and set its metatable to a bunch of functions that close over the local data. This technique is quite common in Lua and is called proxy tables. By using the empty table as a base, we ensure that any table access triggers the __index or __newindex metamethod, depending on the action.

However, there’s a problem. If we try to print such an object in the REPL, we don’t get any information about it, even though it has values:

>> (local s (Set :a :b :c))
nil
>> (. s :a)
"a"
>> (set (. s :d) 42)
nil
>> (. s :d)
"d"
>> s
{}

Additionally to that, we have no means of iteration over this data structure, as the default next function will just return nil:

>> (next s)
nil

We can fix this by adding a __pairs metamethod:

(fn Set [...]
  (let [data (faccumulate [res {}
                           i 1 (select :# ...)]
               (let [k (select i ...)]
                 (doto res (tset k k))))]
    (setmetatable
     {}
     {:__index data
      :__newindex (fn [_ k _] (tset data k k))
      :__pairs (fn [_] #(pick-values 1 (next data $2)))})))

Now, printing works, albeit in a bit strange way:

>> (Set :a :b :c)
{:a nil :b nil :c nil}

A few years ago, I remade the pretty-printer for Fennel, and support for custom objects was a huge part of its design. In short, we can fix our object textual representation by using the __fennelview metamethod.

Now, __fennelview is fine and all, but it is not exactly meant for extending existing objects. Of course, given a Lua library, that implements these proxy tables you can add support for Fennel, but it’s more of an ad-hoc solution. So it’s meant for library authors to make the Fennel REPL experience more user-friendly, and make sure that the printed representation can be read back by Fennel.

And it’s not always possible to add __fennelview to existing objects. For example, one common complaint in Lua is that arrays start from 1 and not 0. We can “fix” that by implementing our own Array type:

(fn Array [...]
  (let [vals [...]]
    (setmetatable
     []
     {:__index (fn [_ i]
                 (. vals (+ i 1)))
      :__newindex (fn [i val]
                    (tset vals (- i 1) val))
      :__len #(length vals)
      :__pairs (fn [_] #(next vals $2))})))

Right now, the object above prints like this:

>> (Array 1 2 3)
[1 2 3]

Let’s assume this came from the library, and we wish to print this differently, e.g. as (Array 1 2 3) instead of [1 2 3] to make sure that such arrays are read back as 0-based arrays, and not as ordinary Lua tables. We can do that, but it is tricky because we don’t have any means of accessing the data closure:

(fn wrap [arr]
  (tset (getmetatable arr)
      :__fennelview
    (fn pp-doc-example [t view options indent]
      (let [lines (icollect [i v (pairs t)]
                    (let [v (view v options (+ 7 indent))]
                      (if (= i 1) v
                          (.. "       " v))))]
        (doto lines
          (tset 1 (.. "(Array " (or (. lines 1) "")))
          (tset (length lines) (.. (. lines (length lines)) ")"))))))
  arr)

However, as you can see, we have to use wrap on every Array, because each instance comes with its own metatable, and setting it once wouldn’t work. And if we try to read back the printed representation we’ll get the old representation:

>> (wrap (Array 1 2 3))
(Array 1 2 3)
>> (Array 1 2 3)
[1 2 3]

So we can make another wrapper:

(local Array* Array)

(fn Array [...]
  (wrap (Array* ...)))

>> (Array 1 2 3)
(Array 1 2 3)

So you can see, how this is a bit problematic.

Custom JSON encoders

Now, this was a bit of a tangent, so let’s get back to the main topic - JSON encoding. Because I can’t ask everyone to do this kind of wrapping, I decided to implement a different way of extending the json.encode function. Thus, the library now features two more functions: register-encoder and unregister-encoder. The second one is for fixing mistakes, and won’t probably be used too much.

So, let’s see how we can add support for custom types. First, we need a function that will tell if the given object is an instance of something we want to encode:

(fn Array? [x]
  (and (= :table (type x))
       (not= nil (. x 0))
       Array))

A library may provide a custom predicate to check if the given object is of the same type, like chan? for asynchronous channels. However, it can’t be used as is, because to register an encoder, such a function needs to return a singleton, that can be compared with =. So we return Array from our example function.

Next, we can implement the encoder:

(fn encode-array [arr encode]
  (.. "["
      (-> (fcollect [i 0 (- (length arr) 1)]
            (encode (. arr i)))
          (table.concat ", "))
      "]"))

Finally, we can register it:

(json.register-encoder object object? encode-object)

The first argument is the object itself, and the second one is a function to obtain the object’s type. Internally, register-encoder calls the object? function on the first argument to provide a bit more flexibility, in case object? returns different values for different objects or based on the object’s properties. The third argument is a function that provides the encoding. It accepts a callback to encode nested values as a second argument.

With that, we can register the encoder and encode our array as JSON by calling (json (Array 1 2 3)):

>> (json.register-encoder (Array) Array? encode-array)
nil
>> (json (Array 1 2 3))
"[1, 2, 3]"

This should provide enough flexibility to support any proxy-based objects.

Custom decoders

I decided not to add custom decoder support to this library, yet. JSON doesn’t have any custom types, so it doesn’t make sense to parse its values to anything but default Lua types.

One could argue, that it would be nice to have a way to parse arrays to zero-based ones, given that we can convert zero-based arrays to JSON. While true, I think this only adds unnecessary complexity to the parser.

One reason, registering an encoder requires a singleton as type designator is to ensure that objects can’t be mixed up. If it was allowed to register encoders via string types, it would be possible to register two "Array" types, making ambiguous what encoder would be used or would require to change the name in case of conflict.

JSON has a fixed amount of types, so registering a decoder for, say, Array, would mean that all arrays would be parsed to this custom type in all of the project parts. Meaning, that if you use a different library, that also uses this JSON parser, and internally expects one-based arrays it will break.

I’m open to changing my mind on this, but for now, I feel that custom encoders are sufficient enough to make this library more usable. As a part of this change, I’m moving this library out from the fnl-http repo into its own library, making fnl-http depend on it instead, making it possible for other projects to use this parser more easily. You can find the code here.

Thanks for reading, and I hope this library will be useful!

Foontones

The deprecation of __ipairs metamethod in Lua 5.3 resulted in an inability to implement custom iteration over custom sequential data structures. For example, the zero-indexed array example would benefit from __ipairs because we could specify that the iteration starts from 0. With the default ipairs we would always miss the first value. It’s not hard to fix manually, by overriding ipairs with a custom function that checks for __ipairs metamethod:
```
       (local ipairs* ipairs)
       (fn _G.ipairs [tbl ...]
         (case (getmetatable tbl)
           {:__ipairs ip} (ip tbl ...)
           _ (ipairs* tbl ...)))
```
However, it’s not practical to intrude into Lua’s standard library in such a way. ↩︎

Comment via email

Streaming and receiving chunked multipart data with fnl-http

Sat, 02 Nov 2024 04:46:00 +0300

I’ve been absent for a while - you may have noticed that compared to the previous year, I posted a lot less this time. There are two closely related reasons for that. First, I felt burned out from programming. Second, I finally picked up a guitar after five or more years and started recording again. Thus, now I only do hobby programming when I feel passionate or challenged in an entertaining way. This works for me, but I have less material to write about.

One such passionate project for me this year is fnl-http. It’s an HTTP/1.1 client and server implementation for Fennel made using luasocket and async.fnl - another library of mine.

Last time I posted, I was mostly talking about testing. And I’m glad that I wrote all these tests, because the change I’m going to talk about this time broke a lot of them, and I was able to diagnose problems much quicker. But the main changes are related to the post I made before that: fnl-http Improvements. In that post, the most exciting things for me were chunked transfer encoding and support for multipart requests. I’ve implemented support for reading chunked responses (the client already could send chunked requests) and moved onto implementing multipart requests.

HTTP Server

For some time, the implementation was sufficient enough, so I started working on an POC asynchronous HTTP server. The server is pretty much bare-bones right now, though it features everything you need to implement a proper one. Here’s how it can be used:

(local server (require :io.gitlab.andreyorst.fnl-http.server))
(local json (require :io.gitlab.andreyorst.fnl-http.json))

(fn handler [{: headers &as request}]
  {:status 200
   :headers {:connection (or headers.Connection :keep-alive)
             :content-type :application/json}
   :body (json request)})

(with-open [s (server.start handler {:port 3000})]
  (s:wait))

It’s a rather simple echo server, that sends back the parsed request as JSON. Let’s try it:

>> (local http (require :io.gitlab.andreyorst.fnl-http.client))
nil
>> (http.get "localhost:3000" {:as :json})
{:body {:headers {:Host "localhost:3000"}
        :method "GET"
        :path "/"
        :protocol-version {:major 1 :minor 1 :name "HTTP"}}
 :headers {:Connection "keep-alive"
           :Content-Length "131"
           :Content-Type "application/json"}
 :http-client #<tcp-client: 0x561839afe390>
 :length 131
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 13
 :status 200
 :trace-redirects {}}

Now, of course, this won’t work as is, because not all Lua values can be encoded as JSON. Such examples are functions, and custom objects, like Readers. We can cheat, though, and encode it as a Fennel table and send it back as a string:

(local {: view} (require :fennel))

(fn handler [{: headers &as request}]
  {:status 200
   :headers {:connection (or headers.Connection :keep-alive)
             :content-type :application/json}
   :body (view request)})

Now we can try a POST request:

>> (http.post "localhost:3000" {:body "Hello HTTP"})
{:body "{:content #<Reader: 0x55eeb3016fc0>
 :headers {:Content-Length \"10\" :Host \"localhost:3000\"}
 :length 10
 :method \"POST\"
 :path \"/\"
 :protocol-version {:major 1 :minor 1 :name \"HTTP\"}}"
 :headers {:Connection "keep-alive"
           :Content-Length "183"
           :Content-Type "application/json"}
 :http-client #<tcp-client: 0x55eeb3059470>
 :length 183
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 20
 :status 200
 :trace-redirects {}}

This should give you an idea of how to implement a proper handler.

The reason the content is a Reader is simple - we don’t want to process any more data than we need when parsing the request. Once we obtain the HTTP-related data, such as headers, method, path, and version, we wrap the rest of the data in a Reader object, so the handler function can work on it later or even discard it. In this particular case, it’s as simple as doing (request.content:read (tonumber request.headers.Content-Length)) but it may vary depending on what was sent, and what path the client used. So I’m leaving this open. Using a Reader enables asynchronous and lazy processing of the request’s body.

Multipart requests

What’s interesting, however, is that there may be multiple bodies, if the request is a multipart one. We can send a multipart request like this:

(http.post "localhost:3000"
           {:multipart [{:name "foo" :content "some string"}
                        {:name "file" :filename "some-data.json"
                         :content (io.open "some-data.json")}]})

Ideally, the server should be able to first process the foo part, and then a file part. Additionally, a file can be really big, so we might want to avoid reading it fully into memory. For example, the JSON module of fnl-http can parse files without fully reading them into memory, as it is based around readers.

The question, however, is how do we give this to a handler without reading the request in full first? We don’t know how many parts there will be, and we might not know the sizes of each part. Let’s look at how the handler encodes this request:

>> (http.post "localhost:3000"
           {:multipart [{:name "foo" :content "some string"}
                        {:name "file" :filename "some-data.json"
                         :content (io.open "some-data.json")}]})
{:body "{:headers {:Content-Type \"multipart/form-data; boundary=------------aaec41d0-1c75-4037-a039-91cc2b31584c\"
           :Host \"localhost:3000\"}
 :method \"POST\"
 :parts #<function: 0x55eeb28e79c0>
 :path \"/\"
 :protocol-version {:major 1 :minor 1 :name \"HTTP\"}}"
 :headers {:Connection "keep-alive"
           :Content-Length "256"
           :Content-Type "application/json"}
 :http-client #<tcp-client: 0x55eeb31df9e0>
 :length 256
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 80
 :status 200
 :trace-redirects {}}

You can see, that there’s no content field anymore. Instead, there’s a parts field, and it is a function.

This function is a standard Lua iterator, that you can call until it returns nil or plug it in any of the iteration forms. Let’s change our handler to support both content and parts:

(fn handler [request]
  (case request
    {: content}
    (set request.content
      (if request.headers.Content-Length
          (content:read (tonumber request.headers.Content-Length))
          (= request.headers.Transfer-Encoding "chunked")
          (content:read :*a)))
    {: parts}
    (set request.parts
      (icollect [part parts]
        (doto part (tset :content (part.content:read :*a))))))
  {:status 200
   :body (json request)})

Let’s look at this a bit closer. The case has two branches - one for content and one for parts. The content branch is pretty straightforward - we replace request’s content field value with the data obtained from Reader. If there was a Content-Length header, we used it to determine how much to read. If not, we (rather crudely) check if the chunked Transfer-Encoding was used, and read all of the data from the Reader. We can do so, because chunked readers know when to stop, based on the final chunk header.

The parts branch is similar, except we have to go one level deeper, and it’s a bit simpler to work with the part’s contents. First, the main function of icollect is to produce a sequential table from the iterator values. We do so by replacing the content field of each part. The content field is again a Reader, but it’s a sized reader, so we can almost always read it in full. Even if content length was not specified, parts are separated by boundaries, so we always know when to stop.

Each part is a table of the following structure:

{:content #<Reader>
 :headers {...}
 :length N ;; optional field, if headers had a Content-Length specified
 :name "part name"
 :filename "name of the uploaded file"
 ;; alternatively :filename* can be present instead
 :type "part kind"}

As much information is given to make working with parts easier. And because parts are provided by an iterator, we don’t have to know in advance how many parts are coming.

However, there’s a caveat - we can’t obtain information about each part first, and process the contents later. Moving to the next part consumes the request body until the next boundary is given, so we can’t reverse the Reader back to its content. Thus, doing this won’t work:

(let [all-parts (icollect [part parts] part)]
  (each [_ part (ipairs all-parts)]
    (part.content:read :*a)))

You’ll have to either copy reader contents into another in-memory Reader, or process parts as they appear.

Now, we can try our request again:

>> (http.post "localhost:3000"
              {:multipart [{:name "foo" :content "some string"}
                           {:name "file" :filename "some-data.json"
                            :content (io.open "some-data.json")}]
               :as :json})
{:body
 {:headers
  {:Content-Type
   "multipart/form-data; boundary=------------a03996f2-0d1c-4aa4-97f5-ac90b06cb6ea"
   :Host "localhost:3000"}
  :method "POST"
  :parts [{:content "some string"
           :headers {:Content-Disposition "form-data; name=\"foo\""
                     :Content-Length "11"
                     :Content-Transfer-Encoding "8bit"
                     :Content-Type "text/plain; charset=UTF-8"}
           :length 11
           :name "foo"
           :type "form-data"}
          {:content "{\"value\": \"JSON data\"}\n"
           :filename "some-data.json"
           :headers {:Content-Disposition "form-data; name=\"file\"; filename=\"some-data.json\""
                     :Content-Transfer-Encoding "binary"
                     :Content-Type "application/octet-stream"
                     :Transfer-Encoding "chunked"}
           :name "file"
           :type "form-data"}]
  :path "/"
  :protocol-version {:major 1 :minor 1 :name "HTTP"}}
 :headers {:Connection "keep-alive" :Content-Length "808"}
 :http-client #<tcp-client: 0x55cf2d942450>
 :length 808
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 13
 :status 200
 :trace-redirects {}}

As you can see, the foo part had length of 12, and the file part had a chunked transfer encoding.

Before this post, there was no way of sending multipart with chunked encoding, so this is a pretty good change in my opinion. Definitively expands the possibilities. Still need to test this with some real HTTP servers, as I don’t see this feature used too often though.

Performance

Now, let’s talk performance. I used the wrk tool to measure the server’s RPS with a simple handler that does 50 millisecond delay on each request:

(local async (require :io.gitlab.andreyorst.async))

(fn handler [{: headers &as request}]
  (async.<! (async.timeout 50))
  {:status 200
   :headers {:connection (or headers.Connection "keep-alive")
             :content-length 11
             :content-type "text/plain"}
   :body "hello world"})

The handler function is always running in an asynchronous context, so we can use parking operations such as <! inside of it.

Running it with 12 threads and 400 connections for 30 seconds yields about 2k requests per second:

Running 30s test @ http://localhost:3000
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   146.42ms   94.13ms   2.00s    98.54%
    Req/Sec   205.07     90.83   343.00     62.36%
  69835 requests in 30.09s, 8.00MB read
  Socket errors: connect 0, read 0, write 0, timeout 151
Requests/sec:   2321.21
Transfer/sec:    272.37KB

Not bad for a single-threaded Lua runtime, I would say!

Now, of course, this kind of benchmark is more synthetic than real, so don’t expect it to work as well as it says here in all cases. With a more complex handler, that deals with routing, and data processing this number will go down. Still, I’m pretty satisfied with these results.

And CPU usage is quite low too. When idling a single CPU core is sitting at about 5%, and during the benchmark, it rises to 70%. Removing the asynchronous sleep, raises CPU usage to 100% (single core), peaking at 67MB of RAM and serving about 4k requests per second. So with an asynchronous sleep, our server process is doing some actual sleeping, leaving some resources to the OS!

And given how async.fnl is implemented, we can actually replace (async.<! (async.timeout 50)) with a blocking loop, like (for [i 1 1_000_000] nil) and still get around 400 RPS. That’s a busy loop with one million iterations per handler invocation on each request. So you shouldn’t worry too much when blocking the handler.

Is it a good result? You be the judge.

Comment via email

Lazy Sequences and Java Streams

Wed, 16 Oct 2024 16:48:00 +0300

When Clojure 1.12.0 was released in September, the release note had a lot of cool features, such as virtual threads, but one feature caught my eye in particular: support for Java Streams and functional interface support. That was just what I needed at work to port one of our internal libraries to Java, to make it easier to reuse from projects written in different JVM languages. Little did I know that Java Streams suck.

The problem

We have a small library, that implements queues with lazy sequences. It has both producer and consumer parts, so several projects can reuse the same library, one being a producer, and the other being a consumer. Today we’re interested in the consumer part.

One of the features this library has is that we can have several separate queues, and we can take items from all of them in a fixed order. Each queue is sorted, so when we have multiple queues, we need to get elements in order and still in a lazy way. Let’s look at some examples:

(def queue1
  (filter odd? (range)))

(def queue2
  (filter even? (range)))

Here, I created two infinite sequences of sorted numbers, just as an example:

user> (take 10 queue1)
(1 3 5 7 9 11 13 15 17 19)
user> (take 10 queue2)
(0 2 4 6 8 10 12 14 16 18)

It’s not that different from actual queues used in our library, as each sequence is potentially infinite and sorted. What we need to get as a result is a single queue with items of both queues combined with retained ordering:

(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ...)

Lazy Seq solution

To achieve this we can implement a merge sort function that will produce a single queue from multiple queues:

(defn merge-sort-sequences [sequences]
  (lazy-seq
   (let [[sequence & sequences] (sort-by first sequences)]
     (cons (first sequence)
           (merge-sort-sequences (conj sequences (rest sequence)))))))

Note, that I’ve omitted some checks for things like empty sequences, and such for the sake of simplicity.

In short, it evaluates like this:

Let’s say, we have [(1 3 5 7...) (0 2 4 6...)] as an input;
We sort it by the first element of each sequence: [(0 2 4 6...) (1 3 5 7...)];
We take the first element from the first queue: 0, and cons it to the recursive call to which we pass back all of the sequences, except we remove the first element from the first queue;
- Since the function returns a lazy sequence the recursive call is effectively trampolined.

And we can see that it works:

user> (take 10 (merge-sort-sequences [queue1 queue2]))
(0 1 2 3 4 5 6 7 8 9)
user> (take 20 (merge-sort-sequences
                [(take-nth 3 (range))          ; (0 3 6 9 12 ...)
                 (take-nth 3 (drop 1 (range))) ; (1 4 7 10 13 ...)
                 (take-nth 3 (drop 2 (range))) ; (2 5 8 11 14 ...)
                ]))
(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19)
user> (take 20 (merge-sort-sequences
                [(filter odd? (range))
                 (filter even? (drop 20 (range)))]))
   (1 3 5 7 9 11 13 15 17 19 20 21 22 23 24 25 26 27 28 29)
;;  \_first_sequence_only_/  \___both_sequences_merged___/

Of course, it won’t work if sequence items are in arbitrary order, but that’s not the case for our implementation.

Java Stream solution

Now, as I mentioned, I had to rewrite this library in Java, for it to be reused in other projects written in other JVM languages. It is possible to use a Clojure library from, say, Scala, but it is quite tedious to do so. We either have to AOT compile the library to a jar, and provide a bunch of methods via genclass. Alternatively, it’s possible to load Clojure, compile the sources, and use it this way, but it introduces way too many hoops to jump through, that rarely anyone would want to do so in our team. And the library in question is small, about 400 LOC with documentation strings and comments, so rewriting it in Java wouldn’t be that hard.

Or so I thought.

I’m not a Java programmer, and I have very limited knowledge of Java. Thankfully, Java is a simple language, unless you wrap everything in an unnecessary amount of classes, use abstract fabric builders, and so on. This library, thankfully, required neither of those cursed patterns - it’s a single static class with no need for instancing, with a bunch of pure methods.

So, knowing that Java has a Stream class, and looking at its interface I thought that I would be able to implement this library. And in truth, it wasn’t a problem, until I got to the lazy merge sort part. That’s when it started looking cursed.

First of all, a Stream is not a data structure - you can’t work with it as if it were data, you have to use pipelines, and then either consume it or pass it around. Moreover, most examples use streams as an intermediate transformation step and return a collection, or suggest passing in a transformer, instead of returning a stream, so I wonder where is this coming from:

Java APIs increasingly return Streams and are hard to consume because they do not implement interfaces that Clojure already supports, and hard to interop with because Clojure doesn’t directly implement Java functional interfaces.

From Clojure 1.12.0 release notes.

Anyhow, I’ve created functions that return streams of items in the queue, much like the ones I showed above. So it was time to implement the merge sort. Let’s look at the skeleton of our function:

public static Stream<Object> mergeSortStreams(Stream<Object>[] streams) {
	return Stream.generate(() -> {
        // implement merge sort somehow...
	});
}

Stream.generate(Supplier) produces an infinite stream, generated by calling the supplier. Basically, it’s what we need here, we can do our sorting inside the supplier. However, there’s a problem - Streams are not data structures. And there’s no way to take one element without consuming the stream. I mean, there’s stream.findFirst() but if we look at the documentation for it, we’ll see that:

Optional<T> findFirst()

Returns an Optional describing the first element of this stream, or an empty Optional if the stream is empty. If the stream has no encounter order, then any element may be returned.

This is a short-circuiting terminal operation.

Returns: an Optional describing the first element of this stream, or an empty Optional if the stream is empty Throws: NullPointerException - if the element selected is null

What is a short-circuiting terminal operation you ask? A terminal operation may traverse the stream to produce a result or a side effect. A short-circuiting terminal operation does the same, but even if presented with an infinite stream, it can finish in a finite time. And after the terminal operation is performed, the stream is considered consumed, and can no longer be used.

But even if we could use findFirst without closing the stream, it wouldn’t be useful to us, because remember - we need to take the first element from each stream and sort the streams themselves. But findFirst is a destructive operation, it removes the element from the stream.

In Clojure, sequences are immutable - all we can do is construct a new sequence if we wish to add items or take its tail if we need fewer items. Thus first does nothing to the sequence in question, we can freely call it on any sequence, obtain an element, do stuff with it, and be done. You can think of first like of an iterator peek, where you look at what the next element is in the iterator without advancing it.

Thankfully, we can convert a Stream to an Iterator, with it staying lazy and potentially infinite. Only, there’s no peek method in the base Iterator class in Java. Oh well.

Well, we can always implement our own wrapper for the Iterator class:

package org.example;

import java.util.Iterator;

public class PeekingIterator<T> implements Iterator<T> {
    Iterator<T> iterator;
    private boolean peeked = false;
    private T peeked_item;

    public PeekingIterator(Iterator<T> it) { iterator = it; }

    public T next() {
        if (peeked) {
            peeked = false;
            T tmp = peeked_item;
            peeked_item = null;
            return tmp;
        } else
            return iterator.next();
    }

    public boolean hasNext() { return iterator.hasNext(); }

    public T peek() {
        if (!peeked && iterator.hasNext()) {
            peeked = true;
            peeked_item = iterator.next();
        }
        return peeked_item;
    }
}

Of course, we could use a dependency, but due to circumstances, we have to keep the amount of dependencies as low as possible. Now, we can get back and implement our merge sort:

public static Stream<Object> mergeSortStreams(Stream<Object>[] streams) {
	List<PeekingIterator<Object>> iterators = new ArrayList<>();
	for (Stream<Object> s : streams) {
		iterators.add(new PeekingIterator<>(s.iterator()));
	}
	return Stream.generate(() -> {
		iterators.sort(Comparator.comparingInt(a -> (Integer) a.peek()));
		return iterators.getFirst().next();
	});
}

Testing it with Clojure examples from above reveals that it works as expected:

System.out.println(
		mergeSortStreams(
				new Stream[]{
						Stream.iterate(1, i -> i + 2),
						Stream.iterate(0, i -> i + 2)
				})
				.limit(20)
				.collect(Collectors.toList())
);
// => [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
System.out.println(
		mergeSortStreams(
				new Stream[]{
						Stream.iterate(0, i -> i + 3),
						Stream.iterate(1, i -> i + 3),
						Stream.iterate(2, i -> i + 3)
				})
				.limit(20)
				.collect(Collectors.toList())
);
// => [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
System.out.println(
		mergeSortStreams(
				new Stream[]{
						Stream.iterate(1, i -> i + 2),
						Stream.iterate(20, i -> i + 2)
				})
				.limit(20)
				.collect(Collectors.toList())
);
// => [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]

Obviously, I’m omitting most of the things in the code here, but this should be enough to give you an idea of what I had to do.

Sequences vs Streams

Let’s compare our versions.

Clojure:

(defn merge-sort-sequences [sequences]
  (lazy-seq
    (let [[sequence & sequences] (sort-by first sequences)]
      (cons (first sequence)
            (merge-sort-sequences (conj sequences (rest sequence)))))))

A pretty straightforward way of doing this kind of operation, in my opinion.

Java:

public static Stream<Object> mergeSortStreams(Stream<Object>[] streams) {
	List<PeekingIterator<Object>> iterators = new ArrayList<>();
	for (Stream<Object> s : streams) {
		iterators.add(new PeekingIterator<>(s.iterator()));
	}
	return Stream.generate(() -> {
		iterators.sort(Comparator.comparingInt(a -> (Integer) a.peek()));
		return iterators.getFirst().next();
	});
}

The general idea is the same but the fact that we had to create a peeking iterator, and store it in an array is disturbing. In Clojure, we manipulate lazy sequences as if they were ordinary data. In Java, we don’t have any real data, so we have to make our own way of accessing it. We have to create an intermediate array list to sort it when every item in the stream is generated. The same happens in Clojure, of course when we call sort-by, and this is possibly worse, as in Java we only create the array once, and sort it in place, and in Clojure, we create a new list every time. JVM is good at collecting garbage though, and the rate at which this sequence is consumed is far greater than the time to clean up the garbage, but it is a thing to consider. Java streams also don’t specify anything about the order and can be processed in parallel, so I’m not sure how my PeekingIterator would behave.

And all of that is simply because Java streams are a half-backed interface made in a rush, or at least it feels like that. Yes, it supports data pipelines with its own implementation of map, filter, etc., however, it makes me appreciate Clojure even more because it has one implementation of map that works across everything (also, it has transducers). In a more complete version of this Java library, we have to map over streams, transform them into iterators just to do some stuff, that is not implemented for streams, transform iterators back into streams, and so forth. The Clojure version is much more straightforward, and concise.

In hindsight, I wish it was easier to use Clojure from other JVM languages. It would save me the time it took to re-implement everything in Java for sure.

In the end, I hooked the old Clojure implementation to use the Java version as a core and retained the interface by converting streams to sequences via stream-seq!. It passed all of the library tests, so I moved on.

Comment via email

Interactive game development with LÖVE

Sun, 08 Sep 2024 21:00:00 +0300

This is a continuation of the previous post on game development with the LÖVE game engine. I’m slowly appreciating the freedom it gives, compared to the TIC-80 experience. One of such freedoms is the fact, that LÖVE is a well-behaving console application.

When you launch TIC, it boots up into a console:

If you do so from a terminal emulator, you can see that the same console is there too:


 TIC-80 tiny computer
 version 1.1.2837 Pro (be42d6f)
 https://tic80.com (C) 2017-2024

 hello! type help for help

>ls

[tic80.com]
mtd.lua

>

However, writing in this prompt doesn’t do anything - it’s not sent back into TIC. And unfortunately, there’s no way of reading from standard IN, because there’s no io namespace provided. One could, theoretically, fiddle with sockets, if it is even possible to load them into TIC, which I’m not so sure about.

In LÖVE, however, you get full Lua experience and more! As you may know, Lua’s io.read() is blocking, and there’s no real way of making it non-blocking or checking the input buffer. However, that’s not a problem for LÖVE, as it has threads! So we can set up a separate thread that reads from the stdin.

Why would we want that? Well, you see, when it comes to Lua, I usually do my programming in Fennel, and Fennel has a decent REPL for doing interactive development. I love the idea of programming with a REPL, I do it a lot for my hobby projects, and at work with Clojure. Interactive development makes iterations much faster, and, in my experience, minimizes the amount of bugs in the resulting program. And we can leverage that during game development too.

Adding the REPL

The way we’re going to set this up is as follows:

We create a new, asynchronous handler for the REPL, and spawn it on the main thread. Then, we’re going to spawn another thread, that will listen on stdin, and send everything it reads to the main thread. Finally, once the main thread finishes the evaluation, it sends the data back to the IO thread for printing. After that, I’m going to spice things up by using the Fennel Proto REPL package I made for Emacs for a better REPL experience.

Let’s look at the REPL code:

(require :love.event)

(local fennel
  (require :lib.fennel))

(fn read [cont?]
  (io.write (if cont? ".." ">> "))
  (io.flush)
  (.. (tostring (io.read)) "\n"))

(case ...
  (event-type channel)
  (while true
    (match (channel:demand)
      :write (-> (channel:demand)
                 (table.concat "\t")
                 (io.stdout:write "\n"))
      :error (-> (channel:demand)
                 (io.stderr:write "\n"))
      :read (->> (channel:demand)
                 read
                 (love.event.push event-type)))))

(let [chan (love.thread.newChannel)
      repl (coroutine.wrap #(fennel.repl $...))]
  (repl
   {:readChunk (fn [{: stack-size}]
                 (chan:push :read)
                 (chan:push (< 0 stack-size))
                 (coroutine.yield))
    :onValues (fn [vals]
                (chan:push :write)
                (chan:push vals))
    :onError (fn [errtype err]
               (chan:push :err)
               (chan:push err))
    :moduleName "lib.fennel"})
  (-> (love.filesystem.read "lib/repl.fnl")
      (#(fennel.compile-string $))
      (love.filesystem.newFileData "repl")
      love.thread.newThread
      (: :start :eval chan))
  (set love.handlers.eval repl))

Simply requiring this file would start a REPL. You may be confused, about how’s it going to work, the while true may look especially concerning.

When we require this file, the ... is bound to the module name, so the case doesn’t match, as it expects two values an event-type and the channel. Then we fall into the let block, which defines the channel chan, the REPL handler repl, starts the REPL coroutine, which pushes the :read message to the channel. Finally, we load this file again, using love.filesystem.read, compile it to Lua, and start a new thread with :eval for the event-type, and chan for the channel. This makes case match and starts the while loop, whose sole purpose is to take values off the channel and act accordingly.

Finally, we set the love.handlers.eval handler, matching our :eval event type to call the repl as a callback when the event occurs. The event occurs only when we get the :read message from the channel, then we use read to get data from the stdio, and we push it back to the eval handler so the repl callback can be called.

That’s pretty much it! Now, let’s put it to the test!

Changing code at runtime

Here, I have a simple player controller and a bunch of values that correspond to how the player behaves:

(local fall-vel 300)
(local jump-vel -300)
(local run-vel 200)

(local player
  {:name "player"
   :color [0.3 0.7 0.3 1]
   :xvel 0
   :yvel 0
   :x 50
   :y 50
   :flip? false
   :width 16
   :height 24
   :state :idle
   :extra-jump 10
   :draw
   (fn [p]
     ;; omitted
     )
   :update
   (fn [{: xvel : yvel : extra-jump : flip? : x : y &as p} dt]
     ;; omitted
     )})

We can try and change the run-vel and see if it is picked up by the game:

Your browser does not support the tag

Seems it’s not. Perhaps, we need to recompile the game module?

Your browser does not support the tag

Aw, snap!

By recompiling the entire module, we undefined some variables, and the game crashed! By the way, the reason changing the run speed value didn’t apply is that we were actually defining a completely new speed value, that is not part of the module we’re editing. That’s a quirk of how Lua modules work and how Fennel REPL works. So let’s fix this!

Modules and state

While Lua has some lisp characteristics, it is understandably not an actual lisp. So, unfortunately, we can’t expect the same level of interactivity from it.

In Lua, a module is an ordinary table. When you write Lua, you return the value at the bottom of the file, usually a table, or sometimes a function. Any local variables, defined in that module are locals, stored as closures for the functions that reference them in the exported value.

Because of that, there’s no such thing as entering a module like in many other lisps. The REPL has its own kind of module space, where all locals, defined in the REPL are stored. The same goes for any code that we send to the REPL. So, while we do have a REPL, we can’t enter a specific module and change it partially - the only way to do that is to recompile the module and put the updated version in the package.loaded table. Which we can do.

However, there’s another problem lurking around - state. The video above shows that the error message was:

./game.fnl:68: attempt to index upvalue 'world' (a nil value)

When we’ve recompiled the module, world became nil, why? Well, because world is defined as a nil, and later set in the love.load function:

(var world nil)

;; ---- 8< ----

(fn love.load []
  (set world (bump.newWorld 64))
  (each [_ wall (pairs level)]
    (world:add wall wall.x wall.y wall.width wall.height))
  (world:add player player.x player.y player.width player.height))

However, the love.load runs only when the game starts, and hence after recompiling the module, world becomes nil again, and our game doesn’t really expect that. The same would happen to any other state we’re setting up when the game loads.

Unfortunately, Fennel doesn’t have a defonce or anything like it. The defonce macro is available in many lisps, and its purpose is to ensure that when we’re redefining something in the REPL or re-evaluating the whole module/namespace/package (whatever you guys call it) the value doesn’t get redefined. Fennel lacks this because there’s no way of doing this in Lua, because of how locals are stored (as closed variables).

But there’s a way around this, we can move the definitions to another module, and use require to load them. In Lua require caches everything it loads, so when we run require again, it doesn’t really do anything except get the value from the package.loaded table.

Here’s how we can define the world module:

(local bump (require :lib.bump))

(bump.newWorld 64)

Code Snippet 1: state/world.fnl

That’s it. Now, we can bring it to the main game module using require:

(local world (require :state.world))

This way, when we recompile the module, we get the same world as before, unless we were to recompile the world module.

I moved most of the game state in such modules, and now we can use our REPL.

Using the REPL

Here’s an example of tweaking the player’s motion speed again:

Your browser does not support the tag

I’m sending the code to the REPL, instead of reloading the module, just because it’s a bit more convenient. This way tweaking the game is much nicer and frictionless, although it doesn’t reach the same heights as that one video from Bret Victor.

Here’s me using a REPL to make an ad-hoc level editor:

Your browser does not support the tag

Unfortunately, there’s no way of using the REPL on Android, since there’s no way of running LÖVE from the same environment as Emacs, without installing X11 and running everything through Termux. Probably doable, but I don’t want to go this route. Android development is more backup route anyway, since I have access to my PC again.

Comment via email

Boredom and gamedev

Sun, 08 Sep 2024 00:34:00 +0300

I spent the previous ten days on vacation. Usually, I try to go off once or twice a year to somewhere where I can just passively relax - usually, it is some sea resort. This year I decided to go to the Republic of Türkiye and spend my time at the beach without any major attractions. The problem is - I get bored pretty quickly and my brain wants to do something.

I took with me my trusty Nintendo 2DS in case I have an urge to spend some time looking at the screen and to get myself occupied on the plane, and a phone with some books to read while I’m lying on the beach. However, my phone is also my computer, and I do a lot of coding on it in my spare time, usually when I don’t have access to a proper PC.

The last time I went on such a vacation I got so bored, that I decided that I wanted to do some game dev, and started a personal game dev marathon. Originally, I planned to start once I got back from vacation, but I was so eager that I started early, while I was still at the beach. The marathon was a crunch, and I had mixed feelings about it in the end, which ultimately led to burnout, and I put the idea of making games on the shelf for a while. And while the last time I started doing this I only drew some sprites, the only reason for that was that I couldn’t get any game engine on my phone. This time around, I got so bored that I started programming a game while gradually getting sunburns and taking breaks to go and actually swim in the sea.

Making games on Android

Basically, if the only thing you have is a phone you have two main options - TIC-80 and löve2d. tic has everything you need - an integrated code editor, a sprite editor, a map editor, and even a sound editor if you’re that versatile. however, the experience of using TIC for doing everything is not great, especially on the phone. I mean, that’s what you have to deal with:

Not a lot of screen space to view the code, the keyboard is abysmal, and the editor experience itself has a lot to be desired. Yes, it is authentic, I get it, it’s just not what I’m personally after.

On PC you can actually get a PRO version of TIC-80, which allows you to write a game in an external editor, and load it into the engine. Technically, you could do the same on Android, but there’s a problem - I use Android 14, which has storage restrictions for apps. And TIC-80 can’t access anything except files in its directory stored at /sdcard/Android/data/com.nesbox.tic80/. Which can’t be accessed directly on the phone either - you need a PC to access it. So even if you have a separate text editor, you can’t do your game development with TIC on Android. Moreover, you can’t even get your game out of the engine unless you have a PC at hand.

The other option, as I said, is LÖVE2D. It has the same problem as TIC, as you can’t launch the game from anywhere but /sdcard/Android/data/org.love2d.android/, however, newer versions have a separate dedicated launcher, that can load .love files from anywhere on the filesystem, which is good enough. Alternatively, an older version, like 11.3 can be installed that bypasses the sandboxing restrictions. So that was the route I took.

Making games on Android with LÖVE

I have little experience with LÖVE2D. Previously, I did a bunch of tests with it and made a few demos like the one described in my raymarching post, or the F.A.B.R.I.K. algorithm I implemented back in the day. So nothing major. The raymarching demo barely runs on a PC, and there’s no way to control it on the phone without connecting a gamepad. Unfortunately, I had no gamepad with me this time, so if I wanted to make a game on the phone I had to implement one myself. Fortunately, I have a lot of experience playing games with the on-screen gamepad, so I’m at least well accustomed to the idea.

I had no idea how to interact with the touch screen though, so I found a drop-in gamepad implementation for LÖVE2D in the game-tools repository:

Figure 1: lovepad.lua

Still, I wanted this gamepad to be a bit more advanced, with analog stick imitation, and more buttons, so I ported it to Fennel first and then extended it with more features:

tag"/>

It has all of the buttons of a modern controller, so I could share the control scheme between the on-screen gamepad and a real gamepad. I do understand that the on-screen gamepad should, ideally, be tailored around a game, but this will suffice for now. It was fun to implement this too, and I found a small bug in the lovepad.lua file, while I was porting it.

Next, I added the bump.lua library for basic physics (because I don’t want to deal with box2d yet), and made a simple player controller:

As you can see, I’m using Emacs on the phone, with the Google Keyboard set to PC QWERTY layout - a neat feature. This is my phone setup for programming, and I’m actually using the same configuration as on my regular PC.

After adding a camera library from the author of bump.lua, here’s what the player controller looks like at the moment:

Your browser does not support the tag

I want to tweak this player-controller a bit further, making it more fleshed out, but as you can see it already handles a few things:

Direction facing is tracked based on horizontal velocity
Jumping
- Holding the jump button longer increases the jump height
- Bumping against a ceiling resets the vertical velocity
Crouching and crawling
- When crouched, the player’s height is reduced, and when crawling the width is increased
- When crawling into a narrow space, the player changes to the crouched state if no direction input is given.
The camera follows the player
- Camera pans in the direction of movement, showing more of the upcoming level
- Vertical position only changes when the player is grounded, or when the vertical position is lower than the original camera position

I still need to tweak various speed values and add more movement options like dashing, sliding, etc., but it already feels gameish.

Writing a player controller was the most challenging thing before when I made the first demo game in the marathon. I made this one similar to the previous one, as can be seen in further GAME1 demos, but it’s a complete rewrite, as I wanted to make it differently this time. We’ll see how it evolves over time.

Drawing

While I didn’t draw anything for this game, I have a few updates on how I approach drawing on the phone. In short, I ditched the PixelStudio I’ve talked about in the first post about GAME1 mainly because its UX doesn’t work for me personally. It’s still a good app for pixel art, I just don’t like how it works, and it lacks some features I’d like to have.

Fortunately, there exists a better option - Krita which has a port for Android that doesn’t compromise any features, so you get a full Krita experience. It’s probably better suited for Android tablets, but works well enough for me on the phone. I’m slowly learning pixel art, and here’s a timelapse of me drawing a tree on my phone:

Your browser does not support the tag

I made a few mistakes when using various tools, as I was not well accustomed to Krita’s toolset back then.

Here’s the final result after a bit of refinement:

This isn’t a drawing for the game I’m making, just a sketch, following the principles outlined in this amazing video by Brandon James Greer. But one thing that makes a difference when making games for LÖVE2D and not for TIC-80 is the unrestricted palette. It’s still better to restrict the palette when making pixel art, but using only 16 colors for the entirety of game graphics is too restrictive for me. Especially right now.

One thing I haven’t tried yet is animating. PixelStudio had a decent animation pipeline, and it was possible to export the animation as a sprite-sheet. Still need to experience this in Krita - I know it has animation capabilities, but I’m not sure if they’ll be comfortable to use on the phone or not.

I’ll be making more pixel art for this game I’m making in the future, so I’ll post actual images once I have anything more to share!

Comment via email

fnl-http - testing with fennel-test

Thu, 15 Aug 2024 23:01:00 +0300

fnl-http is my current passion project - I spend a lot of free time tinkering with it, and the last week was spent on testing and fixing bugs.

As you may know, I made a testing framework, called fennel-test, which has a dedicated test runner, and a set of macros for writing tests. I use it in all of my projects, and oftentimes, some project requires more features that are currently available in the framework, so I extend it as needed. I think it’s a great way of designing something - you use it, and enhance it as you encounter new needs.

I want programming to be exciting, so I spent a lot of time making my test report nice to look at. Here’s how the report looks by default:

Test run at Tue Aug  6 23:20:41 2024, seed: 1722975641242
(....)(......)(.)(.....)(.)(........)(...)(...)

The default reporter is called dots. Each test namespace is encoded with parentheses (), each successful test is encoded as a dot ., and failed tests are encoded as F. The runner randomizes the order of tests each time, so if we run it again, the result would be different:

Test run at Tue Aug  6 23:24:10 2024, seed: 1722975850235
(......)(........)(.)(...)(...)(....)(.....)(.)

However, this is all you get with a reporter like this. If a test fails, information about the error is printed after all tests have finished, so you know what test did fail.

There’s another inbuilt reporter, called namespaces:

Test run at Tue Aug  6 23:27:33 2024, seed: 1722976053226
test.http-parser-test: PASS
test.client-test: PASS
test.json-test: PASS
test.httpbin-test: PASS
test.body-test: PASS
test.headers-test: PASS
test.readers-test: PASS
test.url-test: PASS

As you can see, it prints each namespace that is currently being tested, followed by the namespace-wide status. If a test fails, you again see its error message, but only after all tests have finished, which isn’t ideal because tests in the fnl-http project take about two minutes to complete.

Fortunately enough, you can create reporters in the .fennel-test configuration file!

Reporters

A reporter is just a table with a set of functions. Each one is called when the respecting stage of testing is being run. The methods are:

ns-start - called when entering a new test namespace.

Used to print the information about the namespace, such as its name.
ns-report - called when exiting a test namespace.

Used to print the overall status of the namespace.
test-start - called when running a single test from a namespace.

Used to print the information about the test, such as the test’s name.
test-report - called after the test is finished.

Used to print the status of the test.
stats-report - called after running all tests in all namespaces.

Used to print overall statistics of tests - how many tests were executed, how many errors were encountered, were there any warnings, etc.

This was enough for me to implement other kind of reporters, like the one in the project’s readme. However, I wanted a bit more for this project - first, I wanted the report to be fun to look at, and second, I wanted some statistics. And third, actually, I noticed the need to skip tests in case something is not available on the machine.

After a bit of work, here’s what I’ve got:

With this reporter, if a specific test fails, I can immediately see it, without waiting for all tests to pass. In the end, you can see some statistics on the longest tests, and average test time of the namespace. For example, the test.httpbin-test namespace took 90 seconds to complete, but on average each test only takes 11 seconds. I’m not entirely sure why this metric would be necessary, but I’ve seen it in another test runner, so I implemented it in my reporter.

Now, here’s a problem - one of the tests takes a whopping 90 seconds. I don’t want to run this test all the time, especially, when I’m fixing a bug found by some other test. So I introduced a skip-test function as a first step.

Originally, this function had two modes - terminating and non-terminating, but I later removed the second mode. The reason for that is the with-open macro.

Here’s an example of one of the fixtures I use in the fnl-http project:

(use-fixtures
 :once
 (fn [t]
   (with-open [proc (io.popen (.. "podman run  -p " port ":80 kennethreitz/httpbin >/dev/null 2>&1 & echo $!"))]
     (let [pid (proc:read :*l)
           attempts 10]
       (if (wait-for-server attempts)
           (do (t)
               (kill pid))
           (do (kill pid)
               (skip-test (.. "coudln't connect to httpbin server after " attempts " attempts"))))))))

The with-open macro ensures that the resource is closed no matter what. Closing the process, however, doesn’t kill the program running in the background, so I had to get its PID with sketchy echo $! part and kill it later. Killing the process doesn’t release the file descriptor created by popen, so with-open is only responsible for releasing the file handle here.

A sudden bug

What with-open also does is catching, and re-throwing errors. It’s something akin to trying with resources in Java, and other languages that support this kind of abstraction. You can think about it as try/finally without the catch part.

Lua, however, doesn’t really have well-defined exceptions. Instead, functions usually throw string messages about the error and that’s it. Sometimes, however, it’s useful to throw a table instead. Tables are like objects, so you can embed some additional information in those, making it easier to match a particular error. It’s better to convert this table to a string if it ever reaches the user uncaught though.

What makes things harder is that in Lua it’s not possible to re-throw an error without losing its previous stack trace. While it is a common practice to use pcall to catch any errors and to re-throw them if you can’t do anything about the error, this is what causes losing the precious stack trace. More precisely, if the error message comes with the stack trace, and you would just re-throw it with error, you wouldn’t add your stack trace to the trace. Sounds convoluted, yeah?

So Lua gives you another option: xpcall. The Lua manual doesn’t have a thorough explanation of it though:

xpcall (f, msgh [, arg1, ···])

This function is similar to pcall, except that it sets a new message handler msgh.

It works the same as pcall, except, it accepts an additional argument - a function that would append a stack trace to the error. Usually it’s the debug.traceback function:

debug.traceback ([thread,] [message [, level]])

If message is present but is neither a string nor nil, this function returns message without further processing. Otherwise, it returns a string with a traceback of the call stack. The optional message string is appended at the beginning of the traceback. An optional level number tells at which level to start the traceback (default is 1, the function calling traceback).

Knowing all that, let’s look at how I’ve implemented the skip-test function:

(local Skip
  (setmetatable
   {}
   {:__tostring #:Skip
    :__fennelview #:Skip}))

(fn skip-test [reason]
  "Calling this function inside a test or a fixture will stop the test
early and mark it as skipped. The optional `reason` argument is a
message to display in the log if the reporter is configured to do so."
  (error [Skip reason]))

Right, skip-test throws an array with a special table object, defined as a singleton, so the test runner could distinguish between ordinary test errors and skipped tests. In theory, everything should have worked, but for some reason, some of the tests were skipped correctly, but others didn’t.

The culprit? It’s with-open, obviously, otherwise, why would I take such a tangent about it?

Internally, with-open uses the xpcall function to do its job. And theoretically, it should work just fine, however, it doesn’t use debug.traceback if Fennel’s own compiler.traceback is available.

Originally, I thought that’s just how it is, and implemented two modes in skip-test. After a bit of digging and consulting the manual, I found out that compiler.traceback mistakenly converted every message to a string. Fixing that made the skip-test function work in all cases.

Another sudden bug!

After the patch landed, and I updated Fennel to the newest commit, something else broke. Now, the JSON parsing test no longer succeeded, complaining that some number has no integer representation. The number in question is 23456789012000000000000000000000000000000000000000000000000000000000000000000 or, as it is written in the JSON test file that I use, 2.3456789012e+76.

Before the update, everything worked fine, so I started digging again. Turns out, recently, the way numbers are compiled by Fennel has changed:

@@ -534,8 +534,9 @@ (fn compile-sym
 ;; We do gsub transformation because some locales use , for
 ;; decimal separators, which will not be accepted by Lua.
 (fn serialize-number [n]
-  (pick-values 1 (-> (tostring n)
-                     (string.gsub "," "."))))
+  (if (= (math.floor n) n)
+      (string.format "%d" n)
+      (string.gsub (tostring n) "," ".")))

And now it is failing, because indeed, you can’t convert 2.3456789012e+76 to a string using %d. Except, you can if you’re using LuaJIT - it happily converts this number to "-9223372036854775808", which is even worse.

After a bit of searching, I found that the safest way to format integers as stings is by using "%.f format specifier. Strangely enough, the Lua manual doesn’t have any information on most number formatting patterns. A tiny bit of information was found on the lua-users.org page:

To make sure integer numbers are properly converted into string (without scientific notation), rather than using tostring(x), use format.string("%.0f",x)

I don’t really get why %.f would format a number as an integer given that %f formats floats. And, even though it works the same way in C, I have never seen this in my life. But OK, we could just replace %d with %.f, and be done with that, however, the number above is then would appear as "23456789012000000697746671432670411048021854502345623879537911769302235611136". I get it, it’s a big number, and Lua has limited precision for integers, but the main problem is that this number is much harder to read than its e-notation brother 2.3456789012e+76.

So I settled on finding a way to fix this bug so that both PUC Lua and LuaJIT would format this number in the same way. The result is a patch, that changes the compiler and pretty-printer parts of the language. The function that does that, looks horrifying:

(fn serialize-number [n]
  (let [val (if (= (math.floor n) n)
                (let [s1 (string.format "%.f" n)]
                  (if (= s1 (tostring n)) s1
                      (or (faccumulate [s nil i 0 99 :until s]
                            (let [s (string.format (.. "%." i "e") n)
                                  n* (tonumber s)]
                              (when (= n n*)
                                (let [exp (s:match "e%+?(%d+)$")]
                                  ;; Lua stops transforming numbers from
                                  ;; e-notation to integers at e+14
                                  (if (and exp (> (tonumber exp) 14))
                                      s
                                      s1)))
                              s1))))
                  (tostring n))]
        (pick-values 1 (string.gsub val "," "."))))

We start by checking if (tostring n) and (string.format "%.f" n) give the same string. If they are, we just return the string. If not, we try the scientific notation for each power from 0 to 99. I probably should’ve tested for more powers than that, but it’s OK for now.

We’re converting the number to a string using the e-notation and parse it back, comparing it with the original number. What this does, is that if the power is less than the original, we would lose precision, and comparison would fail. If we can’t find the exponent up to 99, we bail out and return the %.f representation. Otherwise, if the numbers match, we check if the exponent is greater than 14 as it is the exponent where PUC Lua starts using the e-notation. Otherwise, we again return the %.f representation.

Phew!

But I’m not done yet!

Let’s try to compile a really huge number:

>> 1e+99999
nil

What?

Let’s do it in an expression then, maybe that’s just the pretty printer freaking out:

>> (+ 1e+309 1)
attempt to perform arithmetic on a nil value (global 'inf')

~~What?~~ I, of course, jest, but the reason this happens is pretty peculiar.

You see - when we do (tostring "%.f" n) on a huge number like 1e+309 it returns the string "inf". The compiler then happily includes it in the output Lua code, which is then run by the Lua VM. In Lua, however, inf is not a reserved identifier or a number, so it treats inf like a variable name. The inf variable is unbound, hence the nil in the first example, and an error in the second.

So we need to account for that:

(fn serialize-number [n]
  (let [val (if (= (math.floor n) n)
                (let [s1 (string.format "%.f" n)]
                  (if (= s1 "inf") "(1/0)" ; portable inf
                      (= s1 "-inf") "(-1/0)"
                      (= s1 (tostring n)) s1 ; no precision loss
                      (or (faccumulate [s nil
                                        i 0 308 ; beyond 308 every number turns to inf
                                        :until s]
                            (let [s (string.format (.. "%." i "e") n)]
                              (when (= n (tonumber s))
                                (let [exp (s:match "e%+?(%d+)$")]
                                  ;; Lua keeps numbers in standard notation up to e+14
                                  (if (and exp (> (tonumber exp) 14))
                                      s
                                      s1)))))
                          s1)))
                (tostring n))]
    (pick-values 1 (string.gsub val "," "."))))

I also brought the exponent limit up to the last number representable in this notation without becoming inf.

Now we can go back to testing!

Counting skipped tests and assertions

The other thing I added to fennel-test was more counters. Counting skipped tests required some tinkering, like counting tests in the whole namespace if the tests were skipped from the :once fixture.

Counting assertions was even harder, because tests in fennel-test are implemented through macros, and assertions are macros too. I had to inject some runner state, conditionally, and modify it at runtime from each assertion. A bit shady but it works.

What is much more shady is the coverage reports for this project.

Coverage report

I have a firm belief that 100% is the only acceptable coverage. However, and this is a big however, it’s important that such coverage comes from public API tests only. Achieving 100% coverage this way means that your program doesn’t have any dead code and that your public API test at least touches all of our code at least once. It doesn’t speak about test quality, of course, and it doesn’t guarantee that the program is bug-free, but those other points alone make a huge difference.

Lua has one and only luacov that you can use to test your code. I’ve used it in some other Fennel projects in the past, but it comes with some nuances you have to deal with.

First, luacov itself is imprecise. In the post about 100% coverage I talk about it in more detail, but basically, I often see situations where the side effects happened, but the coverage report showed code responsible for the side effects as untouched. Other times, the coverage percentage is bigger than its actual value due to code formatting.

With the fnl-http project I hit a different problem. Two, actually.

The first one is related to the --correlate option of the Fennel compiler. I had an idea, that if I used --correlate I could substitute Lua sources used to produce the coverage with Fennel sources. And since the line numbers are matched by specifying the --correlate option, the coverage report would be so much easier to read.

Didn’t work out.

Unfortunately, --coverage is imprecise. Many parts of Fennel AST don’t contain any line information. Here’s an example:

(local {: a
        : b}
  (require :c))

{: a
 : b}

And here’s the corresponding Lua code:

local _local_1_ = require("c") local a = _local_1_["a"]
local b = _local_1_["b"]
-- empty line
-- empty line
return {a = a, b = b}
-- empty line

As you can see, the overall amount of lines is the same, but many lines that are not empty in the Fennel example are empty in Lua. It’s not a problem, as long as you’re measuring the coverage and viewing the report with Lua, but if we replace the Lua code with its Fennel counterpart, the report becomes invalid. I guess, when the report is constructed, luacov checks if the line is empty or not, and if it isn’t, and there were no hits registered, it counts it as a miss. I’m using luacov-console to view the report, maybe it’s a bug from there, I don’t know.

The only way to fix this is to write the code accounting for these nuances of --correlate:

(local {: a                ; local _local_1_ = require("c") local a = _local_1_["a"]
        : b} (require :c)) ; local b = _local_1_["b"]
                           ; <empty line>
{: a : b}                  ; return {a = a, b = b}

There are more examples like that in the project’s code.

The other problem is more peculiar. Luacov uses debug.sethook to build its report. And I use the async.fnl library in this project, which also uses debug.sethook to schedule tasks. So every time anything happens asynchronously, and trust me, there’s a lot of that in fnl-http, the Luacov report is no longer properly calculated.

So, Luacov has a lot of problems, but I understand why it is like that. You can’t really instrument the Lua code in some other way.

Fennel, on the other hand, isn’t like that. Being a language where everything is an expression, we can probably make some compiler plugin that injects some side-effecting code that writes the report for every expression in the code. It would be difficult to do with things like tables and destructuring, but it should be doable. Maybe I’ll try to tackle this problem in the future, who knows?

Anyway, that’s all from me for now! As usual, you can check out the project, which contains all of the stuff mentioned above regarding fennel-test and luacov. Thanks for reading!

Comment via email

fnl-http Improvements

Thu, 01 Aug 2024 22:57:00 +0300

In the last post two weeks ago I described the process of making an asynchronous HTTP/1.1 client from scratch (minus the socket part). At the end, I mentioned that there’s a lot more to implement:

Now, of course, that’s not all that needs to be implemented. In its current state, this client doesn’t support responses that do not specify the Content-Length or produce a response body in chunked encoding. HTTPS is out of the question for now too. Most of the code just takes a “happy path” without much thought given to errors or more complicated scenarios. But it’s fine, I can always make it better down the line.

HTTPS is still out of the question - to be honest, I don’t want to deal with binary protocols that much, and get myself involved with implementing SSL in Lua. I could, of course use a library, but I feel that Luasocket alone is already big enough dependency.

So here are improvements so far.

Chunked transfer-encoding

As previously mentioned, the client didn’t understand chunked responses, so I had to fix it with a chunked body reader:

(fn read-chunk-size [src]
  {:private true}
  ;; TODO: needs to process chunk extensions
  (case (src:read :*l)
    "" (read-chunk-size src)
    line
    (case (line:match "%s*([0-9a-fA-F]+)")
      size (tonumber (.. "0x" size))
      _ (error (format "line missing chunk size: %q" line)))))

(fn chunked-body-reader [src initial-chunk]
  "Reads body in chunks, buffering each fully, and requesting the next
chunk, once the buffer is empty."
  {:private true}
  (var chunk-size initial-chunk)
  (var buffer (or (src:read chunk-size) ""))
  (var more? true)
  (fn read-more []
    ;; TODO: needs to process entity headers after the last chunk.
    (when more?
      (set chunk-size (read-chunk-size src))
      (if (> chunk-size 0)
          (set buffer (.. buffer (or (src:read chunk-size) "")))
          (set more? false)))
    (values (> chunk-size 0) (string-reader buffer)))
  (make-reader
   src
   {:read-bytes (fn [_ pattern]
                  (let [rdr (string-reader buffer)]
                    (case pattern
                      (where n (= :number (type n)))
                      (let [buffer-content (rdr:read pattern)
                            len (if buffer-content (length buffer-content) 0)
                            read-more? (< len n)]
                        (set buffer (buffer:sub (+ len 1)))
                        (if read-more?
                            (let [(_ rdr) (read-more)]
                              (if buffer-content
                                  (.. buffer-content (or (rdr:read (- n len)) ""))
                                  (rdr:read (- n len))))
                            buffer-content))
                      (where (or :*l :l))
                      (let [buffer-content (rdr:read :*l)
                            (_ read-more?) (not (buffer:find "\n"))]
                        (when buffer-content
                          (set buffer (buffer:sub (+ (length buffer-content) 2))))
                        (if read-more?
                            (let [rdr (read-more)]
                              (if buffer-content
                                  (.. buffer-content (or (rdr:read :*l) ""))
                                  (rdr:read :*l)))
                            buffer-content))
                      (where (or :*a :a))
                      (let [buffer-content (rdr:read :*a)]
                        (set buffer "")
                        (while (read-more) nil)
                        (let [rdr (string-reader buffer)]
                          (set buffer "")
                          (case (rdr:read :*a)
                            nil (when buffer-content
                                  buffer-content)
                            data (.. (or buffer-content "") data))))
                      _ (error (tostring pattern)))))
    :read-line (fn [src]
                 (let [rdr (string-reader buffer)
                       buffer-content (rdr:read :*l)
                       read-more? (not (buffer:find "\n"))]
                   (when buffer-content
                     (set buffer (buffer:sub (+ (length buffer-content) 2))))
                   (if read-more?
                       (if buffer-content
                           (.. buffer-content (or (src:read :*l) ""))
                           (src:read :*l))
                       buffer-content)))
    :close (fn [src] (src:close))
    :peek (fn [_ bytes]
            (assert (= :number (type bytes)) "expected number of bytes to peek")
            (let [rdr (string-reader buffer)
                  content (or (rdr:read bytes) "")
                  len (length content)]
              (if (= bytes len)
                  content
                  (let [(_ rdr) (read-more)]
                    (let [data (rdr:read (- bytes len))]
                      (set buffer (.. buffer (or data "")))
                      buffer)))))}))

It’s a mess, but it works. Please, refer to the previous post for the explanation of what a reader is.

The main idea here is that, when the chunked transfer encoding is detected, we create a reader with chunked-body-reader instead of a regular body-reader, and set up an internal string buffer. We then process all reading operations in terms of this buffer, until it depletes, or when we request more data than left in the buffer. After that, we download the next chunk.

We still get a stream-like behavior out of this, although it has a somewhat big issue - if the server responds with a chunk of enormous size, we can fill up all available memory. I’ve thought of reworking this function to fill the buffer with smaller chunks, but it complicates the code even further, so I decided not to. For now, at least.

And this now works both ways! We can send a chunked body to the server if we’ve didn’t specify the Content-Length header, and the body is not a string. This happens automatically for files, other readers, and channels. Yes, with this library, you can stream data to a server from a channel:

(let [ch (chan)]
  (go #(with-open [body (io.open "some-file" :r)]
         (each [line (body:lines)]
           (<! (timeout 300))
           (>! ch (.. line "\n")))
         (close! ch)))
  (http.post "http://some-server.org" {:body ch}))

This will stream the file line by line, sending each line every 300 milliseconds. The server will receive each line as a separate chunk.

I won’t go into much detail on how this is implemented, as I would need to go through a lot of code.

Multipart requests

After I finished with chunked requests, I turned my eyes on multipart/form-data. It’s an interesting part of HTTP, and after a closer look at it, I realized that email is just a multipart request with weird headers. I’ve been editing emails in Emacs for quite some time but never noticed that the head of the mail is exactly like the head of an HTTP request. What a revelation!

That aside, implementing multipart came with a challenge. I don’t know if it’s just me, but I couldn’t find anywhere if the total Content-Length header is required for a multipart request or not. Logically speaking, since every part is separated by a boundary, a server should be able to read a request’s body until the final boundary. So theoretically, it doesn’t have to know the total content length of the request.

However, I’ve been testing my client with the httpbin.org service and, for some reason, it doesn’t parse the multipart request unless the Content-Length header is present. And it’s not just the sum of part lengths, it’s also their headers, empty lines that separate the header part from content, and so forth.

So I had to implement a set of functions to format multipart parts, and count their lengths:

(fn format-multipart-part [{: name : filename : filename*
                            : content :length content-length
                            : headers
                            : mime-type} boundary]
  "Format a single multipart entry.
The part starts with the `boundary`, followed by headers, created from
`name`, optional `filename` or `filename*` for files, `mime-type`, and
`content-length` which is either calculated from `content` or provided
explicitly.

Default headers include `content-disposition`, `content-length`,
`content-type`, and `content-transfer-encoding`. Provide `headers` for
additional or to change the default ones."
  {:private true}
  (let [content (wrap-body content)]
    (format
     "--%s\r\n%s\r\n"
     boundary
     (headers->string
      (collect [k v (pairs (or headers {}))
                :into {:content-disposition (format "form-data; name=%q%s%s" name
                                                    (if filename
                                                        (format "; filename=%q" filename)
                                                        "")
                                                    (if filename*
                                                        (format "; filename*=%s" (urlencode-string filename*))
                                                        ""))
                       :content-length (if (= :string (type content))
                                           (length content)
                                           (or content-length (content:length)))
                       :content-type (or mime-type (guess-content-type content))
                       :content-transfer-encoding (guess-transfer-encoding content)}]
        k v)))))

This function just formats a single multipart header, i.e. it doesn’t include the content. The content is streamed later, so we don’t need it here.

But with this function, we can now calculate the total length of our request:

(fn multipart-content-length [multipart boundary]
  "Calculate the total length of `multipart` body.
Needs to know the `boundary`."
  (+ (accumulate [total 0
                  _ {:length content-length
                     : name
                     : content
                     &as part}
                  (ipairs multipart)]
       (let [content (wrap-body content)]
         (+ total
            (length (format-multipart-part part boundary))
            (if (= :string (type content)) (+ (length content) 2)
                (reader? content)
                (+ 2 (or content-length
                         (content:length)
                         (error (format "can't determine length for multipart content %q" name) 2)))
                (not= nil content-length)
                (+ content-length 2)
                (error (format "missing length field on non-string multipart content %q" name) 2)))))
     (length (format "--%s--\r\n" boundary))))

This function doesn’t construct the whole body either - it only calculates the length based on headers, and the length of content fields that it can reasonably calculate. There’s a small bug in the format-multipart-part function though. We can override any header, including content-length, meaning we can break the request, if we send a string, and provide a different content-length for it either by mistake or deliberately. This can be fixed, of course, by either not allowing to change content length for strings, or by wrapping string bodies in a string reader. I’ll have to think about what method is better here.

That aside, we can now stream the multipart request:

(fn stream-multipart [dst multipart boundary]
  "Write `multipart` entries to `dst` separated with the `boundary`."
  (each [_ {: name : filename
            : content :length content-length
            : mime-type
            &as part}
         (ipairs multipart)]
    (assert (not= nil content) "Multipart content cannot be nil")
    (assert name "Multipart body must contain at least content and name")
    (let [content (wrap-body content)]
      (->> (if (= :string (type content)) content "")
           (.. (format-multipart-part part boundary))
           (dst:write))
      (when (not= :string (type content))
        (stream-body dst content {:content-length (or content-length (content:length))})))
    (dst:write "\r\n"))
  (dst:write (format "--%s--\r\n" boundary)))

Again, I’m not going to dive too deep here. If you’re interested in the code, you can find and read it here.

Here’s an example of a multipart request body:

--a25537b6-51d6-491f-b18b-a33d197ae4c6\r
Content-Disposition: form-data; name="json-test"; filename="test.json"\r
Content-Length: 1438\r
Content-Transfer-Encoding: binary\r
Content-Type: application/octet-stream\r
\r
[
    "JSON Test Pattern pass1",
    {"object with 1 member":["array with 1 element"]},
    {},
    [],
    -42,
    true,
    false,
    null,
    {
        "integer": 1234567890,
        "real": -9876.543210,
        "e": 0.123456789e-12,
        "E": 1.234567890E+34,
        "":  23456789012E66,
        "zero": 0,
        "one": 1,
        "space": " ",
        "quote": "\"",
        "backslash": "\\",
        "controls": "\b\f\n\r\t",
        "slash": "/ & /",
        "alpha": "abcdefghijklmnopqrstuvwyz",
        "ALPHA": "ABCDEFGHIJKLMNOPQRSTUVWYZ",
        "digit": "0123456789",
        "0123456789": "digit",
        "special": "`1~!@#$%^&*()_+-={':[,]}|;.</>?",
        "hex": "\u0123\u4567\u89AB\uCDEF\uabcd\uef4A",
        "true": true,
        "false": false,
        "null": null,
        "array":[  ],
        "object":{  },
        "address": "50 St. James Street",
        "url": "http://www.JSON.org/",
        "comment": "// /* <!-- --",
        "# -- --> */": " ",
        " s p a c e d " :[1,2 , 3

,

4 , 5	   ,  		        6           ,7        ],"compact":[1,2,3,4,5,6,7],
        "jsontext": "{\"object with 1 member\":[\"array with 1 element\"]}",
        "quotes": "&#34; \u0022 %22 0x22 034 &#x22;",
        "/\\\"\uCAFE\uBABE\uAB98\uFCDE\ubcda\uef4A\b\f\n\r\t`1~!@#$%^&*()_+-=[]{}|;:',./<>?"
: "A key can be any string"
    },
    0.5 ,98.6
,
99.44
,

1066,
1e1,
0.1e1,
1e-1,
1e00,2e+00,2e-00
,"rosebud"]
\r
--a25537b6-51d6-491f-b18b-a33d197ae4c6\r
Content-Disposition: form-data; name="description"\r
Content-Length: 27\r
Content-Transfer-Encoding: 8bit\r
Content-Type: text/plain; charset=UTF-8\r
\r
a JSON file to test parsers\r
--a25537b6-51d6-491f-b18b-a33d197ae4c6--\r

And here’s this request issued with the client library:

>> (http.post "http://httpbin.org/post"
              {:multipart [{:name "json-test"
                            :content (io.open "test/data/valid.json")
                            :filename "test.json"}
                           {:name "description" :content "a JSON file to test parsers"}]
               :as :json})
{:body {:args {}
        :data ""
        :files {:json-test "..."} ;; contents were excluded to save some vertical space
        :form {:description "a JSON file to test parsers"}
        :headers {:Content-Length "1946"
                  :Content-Type "multipart/form-data; boundary=------------5af9b616-6e9d-4c19-8aa5-cc0c5f52ca11"
                  :Host "httpbin.org"
                  :X-Amzn-Trace-Id "Root=1-66aad04d-5155e53a4023b5726bdd134e"}
        :url "http://httpbin.org/post"}
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "2118"
           :Content-Type "application/json"
           :Date "Thu, 01 Aug 2024 00:01:18 GMT"
           :Server "gunicorn/19.9.0"}
 :http-client #<SocketChannel: 0x559596d0e200>
 :length 2118
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 388
 :status 200}

As you can see, the JSON got into the files field, and the description into the form field.

Redirects and query-params

Honestly, implementing redirection was a pain in the butt. But I’m glad that I did!

First, I now finally understand how redirection works. Second, I had to do a major code refactoring for it, which made some parts more approachable, and robust. That was a lot of changes, though.

Alongside redirects, I improved the URL parser and implemented a query builder, that can merge multiple queries into one. Here’s an example, again with httpbin.org:

>> (http.get "http://httpbin.org/redirect-to?status_code=302"
             {:query-params {:url "http://httpbin.org/get"}
              :as :json})
{:body {:args {}
        :headers {:Host "httpbin.org"
                  :X-Amzn-Trace-Id "Root=1-66aad21a-4c52d9316b88f5d862a4b535"}
        :url "http://httpbin.org/get"}
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "199"
           :Content-Type "application/json"
           :Date "Thu, 01 Aug 2024 00:08:58 GMT"
           :Server "gunicorn/19.9.0"}
 :http-client #<SocketChannel: 0x559596734da0>
 :length 199
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 394
 :status 200}

As can be seen, the query params in the URL string and the query-params field in the options table were merged. During the merge, all values are URL-encoded, so you can pass in URLs that are not entirely correct, and it should work.

Of course, we can change how we deal with redirects, limiting the total amounts of redirects:

>> (http.get "http://httpbin.org/absolute-redirect/4"
             {:max-redirects 3
              :as :json})
too many redirecs

Or, we can choose not to follow redirects altogether:

>> (http.get "http://httpbin.org/absolute-redirect/4"
             {:follow-redirects? false})
{:body "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: <a href=\"http://httpbin.org/absolute-redirect/3\">http://httpbin.org/absolute-redirect/3</a>.  If not click the link."
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "283"
           :Content-Type "text/html; charset=utf-8"
           :Date "Thu, 01 Aug 2024 00:24:48 GMT"
           :Location "http://httpbin.org/absolute-redirect/3"
           :Server "gunicorn/19.9.0"}
 :http-client #<SocketChannel: 0x5589fcb69530>
 :length 283
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "FOUND"
 :request-time 228
 :status 302}

Changes in the asynchronous API

I’ve decided to make it easier for users of this library to work with the client’s asynchronous API, and you should no longer be required to use async.fnl directly. Instead, every request function now accepts callbacks for processing responses, and exceptions:

(http.get "https://lua-users.org"
          {:async? true}
          (fn on-response [resp] (print resp.body))
          (fn on-raise [err] (print err.status)))

This means, that the only need for async.fnl is if you need to wait for certain requests to complete. It also means, that the http.METHOD functions will no longer return a promise channel, as I promised in the previous post.

In addition to that, it is now possible to read the response body, returned as a stream on the main thread, even if the request was asynchronous. This is possible because I’ve cheated:

(fn <!? [port]
  "Takes a value from `port`.  Will return `nil` if closed.  Will block,
if nothing is available and used on the main thread.  Will park if
used in the `(go ...)` block."
  (if (main-thread?)
      (<!! port)
      (<! port)))

(fn >!? [port val]
  "Puts a `val` into `port`.  `nil` values are not allowed.  Will block,
if no buffer space or consumer are available and used on the main
thread.  Will park if used inside a `(go ...)` block.  Returns `true`
unless `port` is already closed."
  (if (main-thread?)
      (>!! port val)
      (>! port val)))

Now try to do something like that in Clojure! You can read why this works in my post on async.fnl, or watch my talk at Fennelconf 2023 where I explain this with some more examples.

Performance check

Since the client got bigger and more complicated, let’s check the performance! In the previous post, the client couldn’t accept a chunked response, but now it can, meaning we can compare it with Luasocket’s HTTP client:

>> (local lhttp (require :socket.http))
nil
>> (= (lhttp.request "http://lua-users.org/wiki/FiltersSourcesAndSinks")
      (. (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks") :body))
true
>>

Let’s measure the time again:

>> (time (do (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks") nil))
Elapsed 501.31988525391 ms
nil
>> (time (do (lhttp.request "http://lua-users.org/wiki/FiltersSourcesAndSinks") nil))
Elapsed 630.47289848328 ms
nil

It’s about the same. However, we can be slightly faster, if the user doesn’t need to process the body immediately:

>> (time (do (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks" {:as :stream}) nil))
Elapsed 317.51012802124 ms
nil

Of course, the time is saved by not reading the body of the response, which shows that reading the body alone is about 200ms. But Luasocket’s HTTP client can’t do that at all - it always processes the body, even if the generic form is used and a sink is supplied. That’s an advantage, I would say!

And, thanks to the asynchronous processing of requests, we can actually issue multiple requests at the same time:

>> (let [n-requests 10
         ch (chan)]
     (print ";;; luasocket")
     (time
      (let [start (gettime)]
        (for [i 1 n-requests]
          ;; issuing a synchronous request
          (lhttp.request "http://lua-users.org/wiki/FiltersSourcesAndSinks")
          (io.write i " response processed in " (- (gettime) start)
                    " ms since the first request\n"))))
     (print ";;; fnl-http")
     (time
      (let [start (gettime)]
        (for [i 1 n-requests]
          ;; issuing multiple requests. The on-success handler
          ;; will put the request number into the channel `ch`
          ;; on completion.
          (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks"
                    {:async? true} #(>! ch [i (- (gettime) start)]) #nil))
        (for [i 1 n-requests]
          ;; avaiting each response
          (let [[i ms] (<!! ch)]
            (io.write i " response processed in " ms
                      " ms since the first request\n"))))))
;;; luasocket
1 response processed in 0.55836296081543 ms since the first request
2 response processed in 1.0487260818481 ms since the first request
3 response processed in 1.4974920749664 ms since the first request
4 response processed in 1.9714229106903 ms since the first request
5 response processed in 2.4879760742188 ms since the first request
6 response processed in 3.010272026062 ms since the first request
7 response processed in 3.6146268844604 ms since the first request
8 response processed in 4.1402359008789 ms since the first request
9 response processed in 4.6472299098969 ms since the first request
10 response processed in 5.1433019638062 ms since the first request
Elapsed 5143.6059474945 ms
;;; fnl-http
5 response processed in 1.3388519287109 ms since the first request
7 response processed in 1.4401869773865 ms since the first request
9 response processed in 1.5485229492188 ms since the first request
4 response processed in 1.6689200401306 ms since the first request
8 response processed in 1.7789130210876 ms since the first request
6 response processed in 1.9086010456085 ms since the first request
1 response processed in 2.0185270309448 ms since the first request
10 response processed in 2.1092948913574 ms since the first request
2 response processed in 2.2185280323029 ms since the first request
3 response processed in 2.3184778690338 ms since the first request
Elapsed 2318.8180923462 ms

Still ‘bout twice as fast!

And, as you can see, the order of request completion in the asynchronous case is different, since multiple requests were processed in parallel. Well, it’s not really parallel execution, since the Lua runtime only has one thread, and all response reading is synchronous, but the moment we’ve written the request into the socket, the operating system, and networking hardware could act independently. Thus, the response was sent, and the server on the other side received a series of requests, and probably processed them in parallel, resulting in a faster overall response. While we wait for the data to come back to our socket, other tasks can be executed, such as making another request or processing a part of the response. Of course, when we process the response, we do it asynchronously as well, creating opportunities for other asynchronous threads to collaborate.

Neat stuff!

Even further development, I guess?

When I started this project, I just wanted to use async.fnl for something more real and see if it works. After working on it for about a month, and reading the source code of the clj-http client, this project gradually became something like a re-implementation of clj-http in Fennel. So I changed the project name to fnl-http. It’s not a one-to-one port like async.fnl where I studied the source code - here all code is written from scratch without looking at other clients, but still, I tried to make it faithful.

There are still a lot of parts of HTTP/1.1 that I haven’t processed, and I still have some refactoring ideas and TODOs, so stay tuned!

Comment via email

Asynchronous HTTP client for Fennel

Mon, 15 Jul 2024 20:08:00 +0300

A while ago, I made a library for asynchronous programming in Fennel. It’s based on Clojure’s core.async vision of asynchronous programming using only channels. As an experiment, I’ve added a TCP support layer in that library, allowing one to create a TCP channel, and use it in the same way as a regular channel. My original plan was to implement a small asynchronous HTTP client using that, and I’ve tried it a bunch of times in the last year.

A bit later, I felt unusually burned out from programming as a hobby. In April, I made a post about hobbies, saying that I’d like to switch my hobbies from computers for some time. Truth be told, I’ve been in this stance since February, and only now do I once again feel some interest in programming as a hobby.

So today, I’ll walk you through why I decided to write an HTTP client almost from scratch, and why Luasocket’s HTTP client didn’t work out.

Luasocket’s HTTP client

So, there’s an HTTP client in the luasocket library. And while it does support streaming the body through the LTN12 module, the responses are always synchronous. Which means, that if you have a big enough body, that you’d like to process on demand, you only get the convenience of a stream that you’ve provided as a sink. I’d like the body to be returned as some kind of stream that you can process later, reading data from the client, not from some kind of buffer.

The first thing I tried was patching http.open function at runtime. Then we can open the connection and change the close method on the socket, to work with our intentions. Here’s a piece of code that does this:

(let [{: open : request &as http} (require :socket.http)]
  (fn http.open [...] ;; replacing `open` with our version
    (let [h (open ...) ;; using the original `open` inside
          close (. (getmetatable h) :__index :close)]
      (set close-handler #(close h))
      (fn h.close [self]
        (if (not got-body)
            (close-handler self)
            1))
      h))
  (let [(ok? code resp) (request opts)] ;; "thankfully" `http.request`
                                        ;; resolves `http.open`
                                        ;; dynamically
    (set http.open open) ;; restoring the original `http.open`
    (if ok?
        {:status code
         :headers resp
         :body out} ;; the body is now a channel
        (error code))))

There are a lot of missing details, for example, the out is an ordinary channel, however, there’s a sink that passed via the opts table that pumps everything to it on demand. Once the sink is drained the connection is closed.

All in all, this is a very stateful and shaky approach, so I didn’t want to rely on that. Furthermore, the sink is being pumped in chunks that can’t be controlled very well, making it harder to work with the data in a meaningful way. So I knew I should do something else.

I decided to write my own HTTP client based on Luasocket sockets, and streams. Also, I wanted a bit more familiar calling interface, because Luasocket’s HTTP client is not pleasant to use in my opinion. So let’s start with the HTTP parser!

Reader

Actually, before parsing HTTP, we need to be able to read data from a stream. Sockets are kinda like streams already, but we’re not there yet, so let’s create a string reader:

>> (string-reader "foo\nbar\nbaz")
#<Reader: 0x55f6ca3475b0>

The string-reader function accepts a string, and returns a Reader object that provides the following methods:

read - returns a specified amount of bytes, determined either by the number of bytes or by a supported read pattern.
lines - returns a function that when called will return the next logical line.
peek - returns a specified amount of bytes, without moving the caret.
close - closes the Reader.

Here’s the implementation of string-reader:

(fn string-reader [string]
  (var (i closed) (values 1 false))
  (let [len (length string)
        try-read-line (fn [s pattern]
                        (case (s:find pattern i)
                          (start end s)
                          (do (set i (+ end 1)) s)))
        read-line (fn [s]
                    (when (<= i len)
                      (or (try-read-line s "(.-)\r?\n")
                          (try-read-line s "(.-)\r?$"))))]
    (make-reader
     string
     {:close (fn [_]
               (when (not closed)
                 (set i (+ len 1))
                 (set closed true)
                 closed))
      :read-bytes (fn [s pattern]
                    (when (<= i len)
                      (case pattern
                        (where (or :*l :l))
                        (read-line s)
                        (where (or :*a :a))
                        (s:sub i)
                        (where bytes (= :number (type bytes)))
                        (let [res (s:sub i (+ i bytes -1))]
                          (set i (+ i bytes))
                          res))))
      :read-line read-line
      :peek (fn [s pattern]
              (when (<= i len)
                (case pattern
                  (where bytes (= :number (type bytes)))
                  (let [res (s:sub i (+ i bytes -1))]
                    res)
                  _ (error "expected number of bytes to peek"))))})))

It’s a bit big, but there’s a reason for that - strings in Lua are immutable, and we have to maintain the state when reading. What makes this more complicated is the peek method, but we’ll get to that later.

The make-reader function is a generic Reader constructor, that accepts a table of optional methods:

(fn ok? [ok? ...] (when ok? ...))

(fn make-reader [source {: read-bytes : read-line : close : peek}]
  (let [close (if close
                  (fn [_ ...]
                    (ok? (pcall close source ...)))
                  #nil)]
    (-> {: close
         :read (if read-bytes
                   (fn [_ pattern ...]
                     (ok? (pcall read-bytes source pattern ...))) #nil)
         :lines (if read-line
                    (fn []
                      (fn [_ ...]
                        (ok? (pcall read-line source ...))))
                    (fn [] #nil))
         :peek (if peek
                   (fn [_ pattern ...]
                     (ok? (pcall peek source pattern ...)))
                   #nil)}
        (setmetatable
         {:__close close
          :__name "Reader"
          :__fennelview #(.. "#<" (: (tostring $) :gsub "table:" "Reader:") ">")}))))

As you can see, it just wraps everything with pcall and passes the specified source to the methods. Perhaps, there’s a more clear way of doing this, but it’s fine for now.

Here’s a file-reader, for example:

(fn file-reader [file]
  (let [file (case (type file)
               :string (io.open file :r)
               _ file)]
      (make-reader file
                   {:close #(: $ :close)
                    :read-bytes (fn [f pattern] (f:read pattern))
                    :read-line (file:lines)
                    :peek (fn [f pattern]
                            (assert (= :number (type pattern)) "expected number of bytes to peek")
                            (let [res (f:read pattern)]
                              (f:seek :cur (- pattern))
                              res))})))

It accepts either a file handle or a string that is treated as a path to open the file. It’s much shorter than the string-reader since files can be seek’d and the file object is already stateful.

In the end, we can do this:

>> (local rdr (reader.string-reader "foo\nbar\nbaz\nqux"))
nil
>> (rdr:read 2)
"fo"
>> (rdr:read :*l)
"o"
>> (rdr:read :*l)
"bar"
>> (rdr:read :*a)
"baz
qux"

We’ll be implementing a socket Reader soon, but before that, let’s start implementing a set of functions to generate HTTP messages.

HTTP generation

We start with the build-http-request function:

(local HTTP-VERSION "HTTP/1.1")

(fn build-http-request [method request-target ?headers ?content]
  "Formaths the HTTP request string as per the HTTP/1.1 spec."
  (string.format
   "%s %s %s\r\n%s\r\n%s"
   (string.upper method)
   request-target
   HTTP-VERSION
   (or (headers->string ?headers) "")
   (or ?content "")))

As can be seen, it’s just a string.format call, which we fill in with specific parts. The method and request-target are mandatory arguments, as nothing will work without them. ?headers and ?content on the other hand are optional. Let’s implement the headers->string function now:

(fn header->string [header value]
  "Converts `header` and `value` arguments into a valid HTTP header
string."
  (.. (utils.capitalize-header header) ": " (tostring value) "\r\n"))

(fn headers->string [headers]
  "Converts a `headers` table into a multiline string of HTTP headers."
  (when (and headers (next headers))
    (-> (icollect [header value (pairs headers)]
          (header->string header value))
        table.concat)))

We simply check whether headers is not nil, and if it has any keys. If so, we iterate over the headers table, and convert each pair to a string, then concatenating the resulting table:

>> (headers->string {:connection "close" :content-type "application/json"})
"Content-Type: application/json\r
Connection: close\r
"

The utils.capitalize-header function accepts a string in any case, and transforms it to an HTTP case, meaning each word is capitalized and separated with a dash:

>> (utils.capitalize-header "foo_bar")
"Foo-Bar"
>> (utils.capitalize-header "FooBar")
"Foo-Bar"
>> (utils.capitalize-header "FOO BAR")
"Foo-Bar"

I’ll let you think about implementation, as I’m not particularly fond of my own :)

With that, we can generate a request:

>> (build-http-request
    :get "/"
    {:host "localhost:1234"
     :connection "close"
     :content-type "text/plain"}
    "foobar")
"GET / HTTP/1.1\r
Content-Type: text/plain\r
Host: localhost:1234\r
Connection: close\r
\r
foobar"

While we’re at it, let’s make the build-http-response function too:

(fn build-http-response [status reason ?headers ?content]
  "Formats the HTTP response string as per the HTTP/1.1 spec."
  (string.format
   "%s %s %s\r\n%s\r\n%s"
   HTTP-VERSION
   (tostring status)
   reason
   (or (headers->string ?headers) "")
   (or ?content "")))

Same as before, string.format, filling missing parts, no new functions used:

>> (build-http-response
    200 "OK"
    {:connection "close"
     :content-type "application/json"}
    "{\"foo\": \"bar\"}")
"HTTP/1.1 200 OK\r
Content-Type: application/json\r
Connection: close\r
\r
{\"foo\": \"bar\"}"

Now, we can parse these using our Reader objects.

HTTP Parsing

Let’s start with parse-http-response, as we’re more likely to parse responses, rather than requests, as this library is mainly a client. We’ll write parse-http-request as well, but just for fun.

(fn parse-http-response [src read-fn]
  (let [status (read-response-status-line src read-fn)
        headers (read-headers src read-fn)]
    (doto status
      (tset :headers headers)
      (tset :body src))))

Obviously, this isn’t all we really need, but it’s a good start. First, let’s parse the status line, i.e. the first line of the response: HTTP/1.1 200 OK

(fn parse-response-status-line [status]
  ((fn loop [reader fields res]
     (case fields
       [field & fields]
       (let [part (reader)]
         (loop reader fields
               (case field
                 :protocol-version
                 (let [(name major minor) (part:match "([^/]+)/(%d).(%d)")]
                   (doto res
                     (tset field {: name :major (tonumber major) :minor (tonumber minor)})))
                 _ (doto res
                     (tset field (utils.as-data part))))
               ))
       _
       (let [reason (-> "%s/%s.%s +%s +"
                        (string.format
                         res.protocol-version.name
                         res.protocol-version.major
                         res.protocol-version.minor
                         res.status)
                        (status:gsub ""))]
         (doto res
           (tset :reason-phrase reason)))))
   (status:gmatch "([^ ]+)")
   [:protocol-version :status]
   {}))

(fn read-response-status-line [src read-fn]
  (parse-response-status-line (read-fn src :*l)))

You may be wondering, why these functions accept read-fn. Well, this is something we need to do because we’ll later pass in a socket-channel as a stream.

With these functions, we can read and parse the status line:

(read-response-status-line
 (reader.string-reader "HTTP/1.1 200 OK\r\n")
 (fn [src pattern] (src:read pattern)))
{:protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

You may also wonder, why the loop doesn’t just continue to read the reason-phrase, instead relying on status:gsub. Well, reason can include spaces:

>> (read-response-status-line
    (reader.string-reader "HTTP/1.1 404 NOT FOUND\r\n")
    (fn [src pattern] (src:read pattern)))
{:protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "NOT FOUND"
 :status 404}

Onto headers:

(fn parse-header [line]
  (case (line:match " *([^:]+) *: *(.*)")
    (header value) (values header value)))

(fn read-headers [src read-fn ?headers]
  (let [headers (or ?headers {})
        line (read-fn src :*l)]
    (case line
      (where (or "\r" ""))
      headers
      _ (read-headers
         src
         read-fn
         (case (parse-header (or line ""))
           (header value)
           (doto headers (tset header value)))))))

This one is tail-recursive, accumulating parsed headers in the ?headers table:

>> (read-headers
    (reader.string-reader "Content-Type: application/json\r\nConnection: close\r\n\n\r")
    (fn [src pattern] (src:read pattern)))
{:Connection "close" :Content-Type "application/json"}

Finally, for the body, we just return the reader as it is, as we’ve already read from it all the necessary data, and the rest are the contents of the request. Together, we get a fully parsed response:

>> (parse-http-response
    (reader.string-reader "HTTP/1.1 200 OK\r
Content-Type: application/json\r
Connection: close\r
\r
{\"foo\": \"bar\"}")
    (fn [src pattern] (src:read pattern))
    )
{:body #<Reader: 0x55a54832b810>
 :headers {:Connection "close" :Content-Type "application/json"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

We can ensure that the parsed response can be encoded back:

>> (let [req (build-http-response 200 "OK" {:connection :close} "foobar")
         rdr (reader.string-reader req)
         {: body : headers : reason-phrase : status}
         (parse-http-response rdr (fn [src pattern] (src:read pattern)) {:as :raw})]
     (= req (build-http-response status reason-phrase headers body)))
true

Now, let’s parse the request for completeness:

(fn parse-request-status-line [status]
  ((fn loop [reader fields res]
     (case fields
       [field & fields]
       (let [part (reader)]
         (loop reader fields
               (doto res
                 (tset field (utils.as-data part)))))
       _ res))
   (status:gmatch "([^ ]+)")
   [:method :path :http-version]
   {}))

(fn read-request-status-line [src read-fn]
  (parse-request-status-line (read-fn src :*l)))

(fn parse-http-request [src read-fn]
  (let [status (read-request-status-line src read-fn)
        headers (read-headers src read-fn)]
    (doto status
      (tset :headers headers)
      (tset :content (read-fn src :*a)))))

It’s similar, but we need to parse the status line a bit differently. Same as before, let’s ensure that we can do both:

>> (let [req (build-http-request :get "/" {:connection :close} "foobar")
       rdr (reader.string-reader req)
       {: headers : method : path : content}
       (parse-http-request rdr (fn [src pattern] (src:read pattern)) rdr)]
   (= req (build-http-request method path headers content)))
true

Now, we can start implementing the client itself!

Async.fnl

For communication, we’ll use already existing TCP channels, that enclose socket objects. Although, we’ll need a small modification to the async.fnl library, as currently it always reads data in chunks of fixed size.

Here’s the function that returns a socket channel:

(fn set-chunk-size [self pattern-or-size]
  (set self.chunk-size pattern-or-size))

(fn socket-channel [client xform err-handler]
  (let [recv (chan 1024 xform err-handler)
        resp (chan 1024 xform err-handler)
        ready (chan)
        close (fn [self] (recv:close!) (resp:close!) (set self.closed true))
        c (-> {:puts recv.puts
               :takes resp.takes
               :put! (fn [_  val handler enqueue?]
                       (recv:put! val handler enqueue?))
               :take! (fn [_ handler enqueue?]
                        (offer! ready :ready)
                        (resp:take! handler enqueue?))
               :close! close
               :close close
               :chunk-size 1024
               :set-chunk-size set-chunk-size}
              (setmetatable
               {:__index Channel
                :__name "SocketChannel"
                :__fennelview
                #(.. "#<" (: (tostring $) :gsub "table:" "SocketChannel:") ">")}))]
    (go-loop [data (<! recv) i 0]
      (when (not= nil data)
        (case (socket.select nil [client] 0)
          (_ [s])
          (case (s:send data i)
            (nil :timeout j)
            (do (<! (timeout 10)) (recur data j))
            (nil :closed)
            (do (s:close) (close! c))
            _ (recur (<! recv) 0))
          _ (do (<! (timeout 10))
                (recur data i)))))
    (go-loop [wait? true]
      (when wait?
        (<! ready))
      (case (client:receive c.chunk-size)
        data
        (do (>! resp data) (recur true))
        (nil :closed "")
        (do (client:close) (close! c))
        (nil :closed data)
        (do (client:close) (>! resp data) (close! c))
        (nil :timeout "")
        (do (<! (timeout 10)) (recur false))
        (nil :timeout data)
        (do (>! resp data) (<! (timeout 10)) (recur true))))
    c))

It’s a lot of code, but in reality, it isn’t that complicated. The socket-channel function is responsible for three things:

Creating a channel, that has separate queues for puts and takes, such that when we put something to a channel, we won’t be able to read it back immediately.
Creating an asynchronous loop that waits for data from the socket, and puts it into the receive queue
Creating another asynchronous loop, that waits data on the channel, and puts it into the socket.

These loops account for timeouts, as we’re going to set the socket timeout to 0, to ensure non-blocking behavior. Both asynchronous loops employ their own timeout handling.

The resulting object is a channel, but with some custom methods to account for its duplex nature. And, there are also other properties, such as chunk-size and set-chunk-size which we’re going to use with our streams.

Notice, how the second go-loop has a wait? argument. Because the channel is buffered, it will happily read everything until the buffer is full, which is not ideal, since we may want to change the chunk-size later. So we need to delay the actual read, which we do in the take! method using offer!.

Finally, the tcp.chan function acts as a constructor for TCP channels:

(fn tcp.chan [{: host : port} xform err-handler]
  "Creates a channel that connects to a socket via `host` and `port`.
Optionally accepts a transducer `xform`, and an error handler.
`err-handler` must be a fn of one argument - if an exception occurs
during transformation it will be called with the thrown value as an
argument, and any non-nil return value will be placed in the channel.
The read pattern f a socket can be controlled with the
`set-chunk-size` method."
  (assert socket "tcp module requires luasocket")
  (let [host (or host :localhost)]
    (match-try (socket.connect host port)
      client (client:settimeout 0)
      _ (socket-channel client xform err-handler)
      (catch (nil err) (error err)))))

This is all we need on the async.fnl side of things. No more patching Luasocket’s HTTP client.

Now we can start implementing our own client.

HTTP client

We already have most of the things we need, but we need to change the parse-http-response a bit. Remember, our function returns the body as the Reader, but our source is actually a channel. We can account for that by passing in an appropriate read-fn function but our users shouldn’t deal with that. Instead, the body should provide proper Reader methods, so let’s create a body-reader:

(fn body-reader [src read-fn]
  (var buffer "")
  (make-reader
   src
   {:read-bytes (fn [src pattern]
                  (let [rdr (reader.string-reader buffer)
                        buffer-content (rdr:read pattern)]
                    (case pattern
                      (where n (= :number (type n)))
                      (let [len (if buffer-content (length buffer-content) 0)
                            read-more? (< len n)]
                        (set buffer (string.sub buffer (+ len 1)))
                        (if read-more?
                            (if buffer-content
                                (.. buffer-content (or (read-fn src (- n len)) ""))
                                (read-fn src (- n len)))
                            buffer-content))
                      (where (or :*l :l))
                      (let [read-more? (not (buffer:find "\n"))]
                        (when buffer-content
                          (set buffer (string.sub buffer (+ (length buffer-content) 2))))
                        (if read-more?
                            (if buffer-content
                                (.. buffer-content (or (read-fn src pattern) ""))
                                (read-fn src pattern))
                            buffer-content))
                      (where (or :*a :a))
                      (do (set buffer "")
                          (case (read-fn src pattern)
                            nil (when buffer-content
                                  buffer-content)
                            data (.. (or buffer-content "") data)))
                      _ (error (tostring pattern)))))
    :read-line (fn [src]
                 (let [rdr (reader.string-reader buffer)
                       buffer-content (rdr:read :*l)
                       read-more? (not (buffer:find "\n"))]
                   (when buffer-content
                     (set buffer (string.sub buffer (+ (length buffer-content) 2))))
                   (if read-more?
                       (if buffer-content
                           (.. buffer-content (or (read-fn src :*l) ""))
                           (read-fn src :*l))
                       buffer-content)))
    :close (fn [src] (src:close))
    :peek (fn [src bytes]
            (assert (= :number (type bytes)) "expected number of bytes to peek")
            (let [rdr (reader.string-reader buffer)
                  content (or (rdr:read bytes) "")
                  len (length content)]
              (if (= bytes len)
                  content
                  (let [data (read-fn src (- bytes len))]
                    (set buffer (.. buffer (or data "")))
                    buffer))))}))

Now, that’s a lot of code again. The reason for that is the peek method - same as for strings, we need to implement a way to peek at data without modifying the reader, or at least hide the modification from the user. This is essentially what this function does - it creates a string buffer that we can act upon, accounting for possible data in it before actually reading from the channel itself.

With that function, we can update parse-http-response:

(fn parse-http-response [src read-fn {: as}]
  (let [status (read-response-status-line src read-fn)
        headers (read-headers src read-fn)
        stream (body-reader src read-fn)]
    (doto status
      (tset :headers headers)
      (tset :body
            (case as
              :raw (stream:read (or headers.Content-Length :*a))
              :stream stream
              _ (error (string.format "unsupported coersion method '%s'" as)))))))

As you can see, I’ve added another parameter with options. Right now, there’s only one option as that will choose how to return the body. Specifying {:as :raw} to this function will return the contents of the reader. And {:as :stream} will return the stream itself.

Now, we can write the request function:

(local http {})

(fn http.request [method url ?opts]
  "Makes a `method` request to the `url`, returns the parsed response,
containing a stream data of the response. The `method` is a string,
describing the HTTP method per the HTTP/1.1 spec. The `opts` is a
table containing the following keys:

- `:async?` - a boolean, whether the request should be asynchronous.
  The result is an instance of a `promise-chan`, and the body must
  be read inside of a `go` block.
- `:headers` - a table with the HTTP headers for the request
- `:body` - an optional string body.
- `:as` - how to coerce the output.

Several options available for the `as` key:

- `:stream` - the body will be a stream object with a `read` method.
- `:raw` - the body will be a string.
  This is the default value for `as`.

When supplying a non-string body, headers should contain a
\"content-length\" key. For a string body, if the \"content-length\"
header is missing it is automatically determined by calling the
`length` function, ohterwise no attempts at detecting content-length
are made."
  (let [{: host : port &as parsed} (http-parser.parse-url url)
        opts (collect [k v (pairs (or ?opts {}))
                       :into {:as :raw
                              :async? false}]
               k v)
        headers (collect [k v (pairs (or opts.headers {}))
                          :into {:host
                                 (.. host (if port (.. ":" port) ""))
                                 :content-length
                                 (case opts.body
                                   (where body (= :string (type body)))
                                   (length body))}]
                  k v)
        path (utils.format-path parsed)
        req (build-http-request method path headers opts.body)
        chan (tcp.chan parsed)]
    (if opts.async?
        (let [res (promise-chan)]
          (go (>! chan req)
              (>! res (http-parser.parse-http-response
                       chan
                       (make-read-fn <!)
                       opts)))
          res)
        (do (>!! chan req)
            (http-parser.parse-http-response
             chan
             (make-read-fn <!!)
             opts)))))

We’re still missing some important parts, but we’ll get to those in a bit. This function is long, but it’s not that complicated.

First, we parse the given url into its components, mainly the host and port. Then, we prepare the default opts table to include as key and async? key if none were passed.

Next, we prepare headers. The HTTP spec requires the Host header, so we put it in based on the parsed URL. The path part of the request is then formatted from the rest data from the parsed URL.

Finally, we build-http-request, using the method, path, headers and body if supplied. And create the tcp.chan using the parsed object that contains host and port needed for the constructor.

Then, if opts.async? is true, we create a promise-chan, which is like Clojure’s promise, which is only computed once, and then returns the same result over and over. We start the go thread, in which we asynchronously put the request into the socket channel, and then we asynchronously parse the response supplying our read-fn with the <! function to take from the channel asynchronously. And we return res - the promise channel I mentioned earlier.

If the opts.async? wasn’t true, we do all the same thing, but in a blocking way, using >!! and <!! to work with the channel.

Let’s look at parse-url:

(fn parse-authority [authority]
  (let [userinfo (authority:match "([^@]+)@")
        port (authority:match ":(%d+)")
        host (if userinfo
                 (authority:match (.. "@([^:]+)" (if port ":" "")))
                 (authority:match (.. "([^:]+)" (if port ":" ""))))]
    {: userinfo : port : host}))

(fn parse-url [url]
  (let [scheme (url:match "^([^:]+)://")
        {: host : port : userinfo}
        (parse-authority
         (if scheme
             (url:match "//([^/]+)/")
             (url:match "^([^/]+)/")))
        scheme (or scheme "http")
        port (or port (case scheme :https 443 :http 80))
        path (url:match "//[^/]+/([^?#]+)")
        query (url:match "%?([^#]+)#?")
        fragment (url:match "#([^?]+)%??")]
    {: scheme : host : port : userinfo : path : query : fragment}))

Nothing special, just some patterns to match components of the URL:

>> (parse-url "https://lua-users.org/?some-query-param=some-value")
{:host "lua-users.org"
 :port 443
 :query "some-query-param=some-value"
 :scheme "https"}

And the format-path function constructs the path back:

(fn format-path [{: path : query : fragment}]
  (.. "/" (or path "") (if query (.. "?" query) "") (if fragment (.. "?" fragment) "")))

Here’s what it does on the previously parsed URL:

>> (format-path {:host "lua-users.org"
 :port 443
 :query "some-query-param=some-value"
 :scheme "https"})
"/?some-query-param=some-value"

Finally, the mysterious make-read-fn function creates a function that will work with our socket-channel:

(fn make-read-fn [receive]
  "Returns a function that receives data from a socket by a given
`pattern`.  The `pattern` can be either `\"*l\"`, `\"*a\"`, or a
number of bytes to read."
  (fn [src pattern]
    (src:set-chunk-size pattern)
    (receive src)))

Since our channel does reading on demand, we can’t pass the chunk-size/pattern in, so we use the set-chunk-size method instead. Paired with delayed read, I’ve explained earlier, this works really well.

With all of that, we’re ready to do some requests!

>> (http.request :get "http://lua-users.org/" {:headers {:connection "close"} :as :raw})
{:body "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">

<html>

<head>
<title>lua-users.org</title>
<meta http-equiv=\"content-type\" content=\"text/html; charset=ISO-8859-1\">
<LINK TYPE=\"text/css\" REL=\"stylesheet\" HREF=\"/styles/main.css\">
</head>

<body>

<img src=\"/images/lua-users-org.png\" alt=\"lua-users.org logo\">

<br clear=all>
<p>
<b>lua-users.org</b> is an internet site for and by users of the programming language Lua.
</p>
<p>
To find out what Lua is and for documentation, downloads, mailing list info, etc. please visit the <a href=\"http://www.lua.org/\">official Lua web site</a>.  The <b>lua-users.org</b> web is not affiliated with Tecgraf.
</p>
<p>
Currently available resources:
</p>
<ul>
    <li> <a href=\"/wiki/\">lua-users wiki</a> - collaborative web site
    <li> <a href=\"/lists/lua-l/\">lua-l archive</a> - searchable web archive of
        the official Lua mailing list
    <li> <a href=\"MiniCharter.html\">lua-users.org mini charter</a>
    <li> <a href=\"Acknowledgements.html\">acknowledgements</a>
</ul>

</body>

</html>
"
 :headers {:Accept-Ranges "bytes"
           :Connection "close"
           :Content-Length "1055"
           :Content-Type "text/html"
           :Date "Mon, 15 Jul 2024 12:03:00 GMT"
           :ETag "\"61acad05-41f\""
           :Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
           :Server "nginx/1.20.1"
           :Strict-Transport-Security "max-age=0;"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

Or, we can use :as :stream, and process the stream later:

(local resp (http.request :get "http://lua-users.org/" {:headers {:connection "close"} :as :stream}))
nil
>> resp
{:body #<Reader: 0x55d7090707c0>
 :headers {:Accept-Ranges "bytes"
           :Connection "close"
           :Content-Length "1055"
           :Content-Type "text/html"
           :Date "Mon, 15 Jul 2024 12:03:47 GMT"
           :ETag "\"61acad05-41f\""
           :Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
           :Server "nginx/1.20.1"
           :Strict-Transport-Security "max-age=0;"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}
>> (resp.body:read :*l)
"<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">"

However, writing (http.request :get ...) all the time, is a bit tedious. Let’s instead define functions for each HTTP method. And, to do so, we’ll use macros, because I’m too lazy, and I don’t want to write documentation for each function:

(macro define-http-method [method]
  "Defines an HTTP method for the given `method`."
  `(fn ,(sym (.. :http. (tostring method)))
     [url# opts#]
     {:fnl/arglist [url opts]
      :fnl/docstring ,(.. "Makes a `" (string.upper (tostring method))
                          "` request to the `url`, returns the parsed response,
containing a stream data of the response. The `method` is a string,
describing the HTTP method per the HTTP/1.1 spec. The `opts` is a
table containing the following keys:

- `:async?` - a boolean, whether the request should be asynchronous.
  The result is an instance of a `promise-chan`, and the body must
  be read inside of a `go` block.
- `:headers` - a table with the HTTP headers for the request
- `:body` - an optional string body.
- `:as` - how to coerce the output.

Several options available for the `as` key:

- `:stream` - the body will be a stream object with a `read` method.
- `:raw` - the body will be a string.
  This is the default value for `as`.

When supplying a non-string body, headers should contain a
\"content-length\" key. For a string body, if the \"content-length\"
header is missing it is automatically determined by calling the
`length` function, ohterwise no attempts at detecting content-length
are made.")}
     (http.request ,(tostring method) url# opts#)))

(define-http-method get)
(define-http-method post)
(define-http-method put)
(define-http-method patch)
(define-http-method options)
(define-http-method trace)
(define-http-method head)
(define-http-method delete)
(define-http-method connect)

Now, we can call any HTTP method by just writing (http.method-name ...):

>> (http.get "http://lua-users.org/" {:headers {:connection "close"} :as :stream})
{:body #<Reader: 0x55d7095dbb20>
 :headers {:Accept-Ranges "bytes"
           :Connection "close"
           :Content-Length "1055"
           :Content-Type "text/html"
           :Date "Mon, 15 Jul 2024 12:08:28 GMT"
           :ETag "\"61acad05-41f\""
           :Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
           :Server "nginx/1.20.1"
           :Strict-Transport-Security "max-age=0;"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

And we can even see the documentation for each method in our editor:

Now, that body-reader, with a custom buffer, seemed a bit out of the way, so let’s use it for good.

JSON parsing

Let’s extend our parse-http-response function with another as method:

;; ---- 8< ----
      (tset :body
            (case as
              :raw (stream:read (or parsed-headers.Content-Length :*a))
              :json (json.parse stream)
              :stream stream
              _ (error (string.format "unsupported coersion method '%s'" as))))
;; ---- 8< ----

Now, we can implement a simple JSON parser:

(fn parse [rdr]
  "Accepts a reader object `rdr`, that supports `peek`, and `read`
methods.  Parses the contents to a Lua table."
  ((fn loop []
     (case (rdr:peek 1)
       "{" (parse-obj rdr loop)
       "[" (parse-arr rdr loop)
       "\"" (parse-string rdr)
       (where "t" (= "true" (rdr:peek 4))) (do (rdr:read 4) true)
       (where "f" (= "false" (rdr:peek 5))) (do (rdr:read 5) false)
       (where "n" (= "null" (rdr:peek 4))) (do (rdr:read 4) nil)
       (where c (c:match "[ \t\n]")) (loop (skip-space rdr))
       (where n (n:match "[-0-9]")) (parse-num rdr)
       nil (error "JSON parse error: end of stream")
       _ (error (string.format "JSON parse error: unexpected token ('%s' (code %d))" _ (_:byte)))))))

First, let’s look at parse-string:

(fn parse-string [rdr]
  (rdr:read 1)
  ((fn loop [chars escaped?]
     (case (rdr:read 1)
       "\\" (loop chars (not escaped?))
       "\"" (if escaped?
                (loop (.. chars "\"") false)
                chars)
       _ (loop (.. chars _) false)
       ))
   "" false))

This function handles strings, and escaped patterns withing set strings, like nested strings:

>> (parse (reader.string-reader "\"foo \\\"bar\\\"\""))
"foo \"bar\""

Now, let’s look at the parse-arr

(fn parse-arr [rdr parse]
  (rdr:read 1)
  ((fn loop [arr]
     (skip-space rdr)
     (case (rdr:peek 1)
       "]" (do (rdr:read 1) arr)
       _ (let [val (parse)]
           (tset arr (+ 1 (length arr)) val)
           (skip-space rdr)
           (case (rdr:peek 1)
             "," (do (rdr:read 1) (loop arr))
             "]" (do (rdr:read 1) arr)
             _ (error (.. "JSON parse error: expected ',' or ']' after the value: " (json val)))))))
   []))

This function accepts the parse callback to parse nested values, so if we try to parse an array of strings, we’ll call parse on each value, and then expect either a colon or the closing bracket to mark the end of the array.

>> (parse (reader.string-reader "[\"a\", \"b\"]"))
["a" "b"]

The parse-obj works for objects in a similar way:

(fn parse-obj [rdr parse]
  (rdr:read 1)
  ((fn loop [obj]
     (skip-space rdr)
     (case (rdr:peek 1)
       "}" (do (rdr:read 1) obj)
       _ (let [key (parse)]
           (skip-space rdr)
           (case (rdr:peek 1)
             ":" (let [_ (rdr:read 1)
                       value (parse)]
                   (tset obj key value)
                   (skip-space rdr)
                   (case (rdr:peek 1)
                     "," (do (rdr:read 1) (loop obj))
                     "}" (do (rdr:read 1) obj)
                     _ (error (.. "JSON parse error: expected ',' or '}' after the value: " (json value)))))
             _ (error (.. "JSON parse error: expected colon after the key: " (json key)))))))
   {}))

Except, it waits for the : after each key, and then the value:

>> (parse (reader.string-reader "{\"arr\": [\"a\", \"b\"]}"))
{:arr ["a" "b"]}

Finally, there’s the parse-num and code to parse Booleans, and null:

(fn parse-num [rdr]
  ((fn loop [numbers]
     (case (rdr:peek 1)
       (where n (n:match "[-0-9.eE+]"))
       (do (rdr:read 1) (loop (.. numbers n)))
       _ (tonumber numbers)))
   (rdr:read 1)))

With that, we can call http.get passing it {:as :json} as opts:

>> (http.get "http://httpbin.org/get" {:as :json})
{:body {:args {}
        :headers {:Host "httpbin.org"
                  :X-Amzn-Trace-Id "Root=1-6695183d-27c59aff535ae9f11df8c98d"}
        :origin "185.169.167.87"
        :url "http://httpbin.org/get"}
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "199"
           :Content-Type "application/json"
           :Date "Mon, 15 Jul 2024 12:38:21 GMT"
           :Server "gunicorn/19.9.0"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

And our body was successfully parsed.

Asynchronous HTTP calls

Now, this client is asynchronous, meaning we can download lots of resources simultaneously. I’m not saying in parallel, because there are no threads in Lua, but still, we can process several requests at once using our asynchronous library.

Let’s define a simple time macro first:

(macro time [expr]
  `(let [clock# socket.gettime
         start# (clock#)
         res# ,expr
         end# (clock#)]
     (print (.. "Elapsed " (* (- end# start#) 1000) " ms"))
     res#))

With it, we can measure the execution time of a body. Let’s make three asynchronous requests, that upon completion will count the lengths of the body for each, and return their sum:

>> (time
    (let [headers {:connection "close"}
          a (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
          b (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
          c (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
          lengths (chan 3)]
      (go (let [resp (<! a)]
            (>! lengths (length resp.body))))
      (go (let [resp (<! b)]
            (>! lengths (length resp.body))))
      (go (let [resp (<! c)]
            (>! lengths (length resp.body))))
      (+ (<!! lengths) (<!! lengths) (<!! lengths))))
Elapsed 530.24077415466 ms
597

In comparison, three synchronous requests, take almost twice as much time:

>> (time
    (let [headers {:connection "close"}
          a (http.get "http://httpbin.org/get" {:headers headers :as :raw})
          b (http.get "http://httpbin.org/get" {:headers headers :as :raw})
          c (http.get "http://httpbin.org/get" {:headers headers :as :raw})]
      (+ (length a.body) (length b.body) (length c.body))))
Elapsed 1047.5831031799 ms
597

For the reference, here’s Luasocket’s HTTP client:

>> (local lhttp (require :socket.http))
nil
>> (time
    (let [a (lhttp.request "http://httpbin.org/get")
          b (lhttp.request "http://httpbin.org/get")
          c (lhttp.request "http://httpbin.org/get")]
      (+ (length a) (length b) (length c))))
Elapsed 1919.6438789368 ms
717

(Note, the resulting length is different, because Luasocket specifies its user-agent, and httpbin includes it in the answer.)

Further development

Now, of course, that’s not all that needs to be implemented. In its current state, this client doesn’t support responses that do not specify the Content-Length or produce a response body in chunked encoding. HTTPS is out of the question for now too. Most of the code just takes a “happy path” without much thought given to errors or more complicated scenarios. But it’s fine, I can always make it better down the line.

But all in all, it was a fun experiment, and I’m feeling better, as I finally have some use for async.fnl. You can find the source code for this project on my GitLab profile. And by the way, even though it is written in Fennel and uses some Fennel libraries, like async.fnl, this library works with ordinary Lua:

Lua 5.4.4  Copyright (C) 1994-2022 Lua.org, PUC-Rio
> http = require "http"
> http.get("http://httpbin.org/get", {headers = {connection = "close"}}).body
{
  "args": {},
  "headers": {
    "Host": "httpbin.org",
    "X-Amzn-Trace-Id": "Root=1-6695424f-01e0510f2ab49b9f46bd51dc"
  },
  "origin": "185.165.163.78",
  "url": "http://httpbin.org/get"
}
> http.get("http://httpbin.org/get", {headers = {connection = "close"}, as = "stream"}).body
Reader: 0x55a6c0eff710

So if you like what you see, but you don’t use Fennel, you can still try out this library. It just needs to be compiled first.

Anyway, that’s all from me, so… thanks for reading!

Comment via email

Why today's phones are so boooooring?

Tue, 11 Jun 2024 04:10:00 +0300

So, here’s a question - when were you last excited for a new phone? Well, I mean really excited?

Here’s the thing. Phones are so boring today that most retail stores show you the backside of the phone. Because from the front they all look exactly the same.

Do you remember the old days when phone manufacturers fought for your money by trying to make cool-looking phones? Nokia and Siemens were legendary rivals, at least in my region. And then Sony Ericsson came to the party and nuked Siemens out of the competition for a few years - no one talked about Siemens. And then Motorola came with their Razor series and woooh! Just look up old phone magazines - they’re filled with amazing designs.

I still have my old Nokia and Sony Ericsson phones:

Figure 1: Both of the phones feature buttons above the screen. Nokia had them cleverly hidden with the ability to slide the screen in both ways which were used both as media controls and for some games. You could comfortably game on these phones without the main keyboard!

Yes, they’re busted, but I still love them - they look rad.

Now look at the typical online marketplace for phones:

Figure 2: Current best-sellers at amazon.com

Apart from the wallpaper at the front, these phones all look similar - it’s a black rectangle. The backside is where the design is supposed to happen today, yet most phones are just some variation of mat glass, leather, or just some colored plastic for cheaper ones.

*duh*
Smartphones are a pretty much a figured-out tech since 2007. It has to be a black rectangle, because the screen is the most important part, so it has to cover all of the available area and it was refined over the years until no borders were left to spare. And the backside is pretty much what is left for creativity, but you can’t do much with how big camera modules are today, because people demand good cameras even in cheaper segments.

Well, you could say that. But do you know that other tech, that was figured out and for a much longer time than the Smartphones? I’ll give you a hint, it contains the latter part of the word “smartphone”.

Headphones.

Headphones haven’t really advanced much since the 80s. I mean, they did, and quite a bit in the IEM field, but not radically overall. One of the most balanced studio headphones as of today are Sennheiser HD 600 were released in 1997.

I know, there are all these different technologies for producing sound, like dynamic, planar-magnetic, electrostatic drivers, MEMs, and possibly more. But we have had great audio since the 80s, you don’t have to buy new headphones every few years - the technology is already pretty much maxed out our hearing capabilities.

We approach the same scenario with smartphone screens - ~500 ppi on a 7-inch screen is already far beyond the required amount to not-see the individual pixels. It’s like drivers in headphones - the sound resolution is way beyond human hearing, so there’s no need for more advancements to be made. Different drivers have different response times, sure, but it’s the same with screens if you think about refresh rates.

But I digress. Have a look at modern headphones:

Figure 3: headphones.com catalog

Yes, these all are premium products, but even consumer-grade ones are competing in design and ergonomics:

Figure 4: headphones.com catalog

Think about it - headphones, essentially, are just ovals or circles - you don’t need all these fancy earcup designs, materials, and whatnot. Again, one of the best-sounding headphones is just an oval with a perforated mesh on the outer side:

Figure 5: A product photo from sennheiser.com

You don’t need all of these cool-looking shapes on the earcup, yet headphone manufacturers do it because it is a form of expression. Art, even.

And if you look at premium smartphones, apart from a back rectangle on the front, what do you get on the back usually? A mat glass with too many cameras and a logo that you gonna cover with a phone case anyway.

When Samsung showed up their new foldable phones, I got excited for a bit, but the foldable screen tech is still not ready for prime time. And again, these phones all pretty much look the same. They do have a different hinge design, and often the outer shell has a more stylized look, but still, do you remember the old foldable phones?

Why today’s phones are so boooooring?

Comment via email

Lazy state machines

Wed, 15 May 2024 04:26:00 +0300

I’m not sure if this is a new thing or not, and I’m too lazy to look it up as it’s 3 AM right now, so here it is.

I’ve been thinking about state machines lately, and how Clojure’s multimethods are a cool way to implement a state machine. Multimethods are somewhat like pattern matching, in a way, and you can use them to do some declarative programming in Clojure, like this:

(defmulti factorial identity)

(defmethod factorial 0 [_] 1)

(defmethod factorial 1 [_] 1)

(defmethod factorial :default [n]
  (*' n (factorial (dec n))))

(factorial 5) ;; => 120

So, here, we’re declaring the factorial multimethod that dispatches on numbers. Let’s ignore negative numbers for now - pretend we’re ancient Greeks, and there is no such thing as a negative number.

So, when factorial sees a value of 0, it simply returns 1. The same thing happens if it sees 1. Otherwise, we’ll multiply the number with the result of calling factorial with the said number decreased by 1.

In reality, there’s and if hiding in here, as we might have just written factorial the conventional way:

(defn factorial [n]
  (cond (= n 0) 1
        (= n 1) 1
        :else   (*' n (factorial (dec n)))))

(factorial 5) ;; => 120

It’s the same thing, except we’ve avoided concretion of our function, allowing it to be extended with arbitrary dispatch values from outside, without modifying existing code.

I say it’s declarative because we’re declaring how a function should behave, given a specific input. I say it’s like pattern matching because we kinda match a pattern with the dispatching function.

E.g. we can write a pattern matching function that can do stuff based on a pattern of arguments given:

(defmulti foo (fn [& args] args))

(defmethod foo [0 0] [& _] "draw")
(defmethod foo [1 0] [& _] "win")
(defmethod foo [0 1] [& _] "loose")
(defmethod foo [1 1] [& _] "snake eyes")

(foo 1 1) ;; => "snake eyes"

It’s not quite pattern matching, as it doesn’t do binding, and there’s no match-all/I-don’t-care clause, but it can be used like that, and we certainly do.

Now, the factorial example is also a kind of a state machine. When it sees the number 1 or 0 its action is simple - return 1 and halt. If it sees anything else, it changes the state by decreasing a number, and proceeds. After the state reaches 1 or 0, the stack of actions is reduced, and we get the final result.

However, there’s a problem. Such a state machine can’t run for an arbitrary amount of time - it will consume a lot of the stack, and eventually fail. How can we avoid that?

We know, there’s a way of doing recursion in Clojure, using the recur special. It isn’t like a proper recursion though, rather, it’s a jump to the beginning of the function which also rebinds its arguments. It’s a handy thing, but it can’t handle mutual recursion.

The other way of doing recursion in Clojure is the trampoline function. Instead of calling the function you want to recurse into, you return it as a callback, and trampoline executes it instead. Because of that the stack is constantly incremented and decremented, but we can’t have non-tail mutual recursion.

Yet, there’s a way of having non-tail mutual recursion that doesn’t blow up the stack - lazy sequences!

Here’s a great example, a lazy sequence of Fibonacci numbers:

(defn fib-seq [a b]
  (cons a (lazy-seq (fib-seq b (+' a b)))))

(defn fib [n]
  (first (drop (inc n) (fib-seq 0 1))))

(fib 9) ;; => 55

(map fib (range 10)) ;; => (1 1 2 3 5 8 13 21 34 55)

(take 10 (fib-seq 0 1)) ;; => (0 1 1 2 3 5 8 13 21 34)

As you can see, fib-seq creates a lazy sequence by constantly calling itself. Its call is delayed until we consume the sequence, so it acts mostly like a trampoline, except we can retain all previous results.

Thinking of this, I came up with this kind of state machine:

(defmulti machine first)

(defmethod machine :a [[state & states]]
  (cons "A" (lazy-seq (machine (cons :c states)))))

(defmethod machine :b [[state & states]]
  (cons "B" (lazy-seq (machine states))))

(defmethod machine :c [[state & states]]
  (cons "C" (lazy-seq (machine (cons :b states)))))

(defmethod machine nil [[state & states]]
  ["done"])

This machine is simple - when it sees the state :a it will produce the "A" as a result, and change to the state :c. In state :c it produces the result "C" and changes to the state :b. Finally, in the state :b the machine produces the result "B" and doesn’t change its state. Then it halts. So, if we call (machine [:a]) we’ll get the following list ("A" "C" "B" "done").

One important observation would be that, unlike the fibbonaci, this state machine accepts a sequence of states, and modifies it to change state. This means that we can provide more tasks in a sequence, like:

lazy-state-machines
> (machine [:a :a])
("A" "C" "B" "A" "C" "B" "done")
lazy-state-machines
> (machine [:a :c])
("A" "C" "B" "C" "B" "done")
lazy-state-machines
> (machine [:b :b])
("B" "B" "done")
lazy-state-machines
> (machine [:b :b :a :c])
("B" "B" "A" "C" "B" "C" "B" "done")

And, since this state machine is lazy, we don’t have to consume all of its results, as it might never actually halt, say there’s one more state :d:

(defmethod machine :d [[state & states]]
  (cons "D" (lazy-seq (machine (cons :d states)))))

No other state triggers this one, so the machine would always halt if we only gave it states :a, :b, and :c. However, if we give it the state :d manually, we would get this:

lazy-state-machines
> (take 10 (machine [:a :d]))
("A" "C" "B" "D" "D" "D" "D" "D" "D" "D")
lazy-state-machines
> (first (drop 1000000000 (machine [:a :d])))
"D"

Unless I used take we would’ve stuck on the :d state forever. However, it doesn’t concern us much, as we lazily take the results out of the state machine, and even if we were to process an extreme amount of steps, the stack would not overflow. The memory footprint would also not be that big unless we retained the head of the produced list somewhere.

Now, onto mutual recursion. Here are two more state machines:

(defmulti dog first)
(defmulti kid first)

(defmethod dog :play [[state & states]]
  (cons "dog: runs back with a ball"
        (lazy-seq (kid (cons :throw states)))))

(defmethod kid :throw [[state & states]]
  (cons "kid: throws the ball"
        (lazy-seq (dog (cons :ball states)))))

(defmethod dog :ball [[state & states]]
  (cons "dog: runs for the ball"
        (lazy-seq (dog (cons :play states)))))

(defmethod kid nil [[state & states]]
  (cons "kid: does nothing" (lazy-seq (dog states))))

(defmethod dog nil [[state & states]]
  ["dog: stops playing"])

Hopefully it’s not too complicated.

There are two communicating state machines the dog and the kid. The kid can :throw the ball, and the dog can :play which means that it brings the ball back to the kid asking to throw again. As can be seen, there’s a mutual recursion going on, and there’s no state that will make these machines halt. The kid and the dog are doomed to play til the end of time, and the StackOverflow won’t save them. Thankfully, they’re lazy, so they won’t unless we check on them.

Now, you may be wondering, apart from being able to specify many states upfront, what is the reason for having states as a sequence? Most of the time we’re just consing a new state, and discarding the previous one, so we could just use a singular state instead.

That’s right, but lazy sequences also give us one more advantage - such state machines can influence its future!

Let’s introduce a random chance for the dog to get tired:

(defmethod dog :ball [[state & states]]
  (cons "dog: runs for the ball"
        (lazy-seq (if (> (rand) 0.75)
                    (dog (lazy-cat [:tired] states [:rest]))
                    (dog (cons :play states))))))

(defmethod dog :rest [[state & states]]
  (cons "dog: rests" (lazy-seq (kid states))))

(defmethod dog :tired [states]
  (cons "dog: walks back with a ball"
        (lazy-seq (kid states))))

(defmethod kid :tired [[state & states]]
  (cons "kid: sees dog is tired"
        (lazy-seq (dog states))))

As you can see, we’ve changed the :ball state for the dog. If the random number is higher than 0.75, the dog changes its state to :tired and appends a task for later to :rest.

Then, in the :tired state, the dog doesn’t want to do anything yet, so it looks at the kid with the same :tired state. The kid notices that the dog is tired, and stops playing, allowing dog to check if it wants to do anything else. Earlier the dog appended the :rest task, so it decides to rest, and let’s kid check if it has any more tasks to do.

Since no more states are present, both stop playing:

lazy-state-machines
> (run! println (kid [:throw]))
kid: throws the ball
dog: runs for the ball
dog: runs back with a ball
kid: throws the ball
dog: runs for the ball
dog: walks back with a ball
kid: sees dog is tired
dog: rests
kid: does nothing
dog: stops playing

If the kid is extremely unlucky, this machine can still loop forever though.

We can, however, force the dog to play, by giving the kid two throwing tasks. This would require implementing a :default method for the dog, so if it doesn’t know what to do with the state, it asks the kid to check if it can. The same applies to the kid:

(defmethod kid :default [states]
  (lazy-seq (dog states)))

(defmethod dog :default [states]
  (lazy-seq (kid states)))

Now, we can make the kid more ignorant:

lazy-state-machines
> (run! println (kid [:throw :throw]))
kid: throws the ball
dog: runs for the ball
dog: walks back with a ball
kid: sees dog is tired
kid: throws the ball
dog: runs for the ball
dog: runs back with a ball
kid: throws the ball
dog: runs for the ball
dog: runs back with a ball
kid: throws the ball
dog: runs for the ball
dog: walks back with a ball
kid: sees dog is tired
dog: rests
dog: rests
kid: does nothing
dog: stops playing

The dog needs double rest now though.

Anyway, that’s it! I hope you’ve found this entertaining, and it would be cool to hear from you if you have ever used this kind of lazy state machine in the past, or if you can think of other cool applications for them. Thank you for reading!

Comment via email

Hobbies

Tue, 02 Apr 2024 21:59:00 +0300

For quite a long time programming was my main hobby. I enjoyed it, as I felt like I was creating something (hopefully) useful, and the problems I tried to solve were making my brain-cogs turn. However, recently it seemed to change, or at least, I’m feeling my usual burnout a bit harder than usual.

You may remember, recently I made a (now unlisted) post about taking a break from writing and thinking about what I want to do in the future with this blog. Most of the stuff there is still relevant, though I felt that the way I put it down was a bit off my usual style, so I unlisted it. This post is, somewhat, of a correction of these thoughts.

Not so long ago I had a change in my career. I’m still a software engineer, though now I will probably write less code, as I’m becoming a software architect. This has a lot of implications, because I have to learn a lot of new stuff, and my work now consists of lots and lots of thinking on all levels. It also means that the way I’m thinking has to change.

My programming, usually, consisted of throwing my brain against a problem and seeing what sticks. Rarely did I dig into the internals of stuff I was doing, research any prior art, or use known methodologies. It was fun, but it is not the proper way of doing things unless you’re working on something completely new. I used to think about it as my way of learning things, but honestly, it was more of a way to procrastinate.

After reading a few books on how to think properly, I decided that I needed to change the approach.

This resulted in an unusual amount of burnout in the last month, because now I have to think more about how things are done properly, and I have enough of it at work. My mental capacity is not as huge as it may seem, so after-work programming sessions started to feel like a burden, rather than fun activities. Well, for now at least.

So I decided that I should switch my hobbies from computers for a while. I’m now re-learning how to play guitar, and I finally went to the rock climbing gym. Gotta say, that these activities are as brain-intense as programming, or at least they feel like it. Though, I’m not sure if this blog is a fitting place to talk about both of these activities.

I’m still not giving up hope on gamedev though, so maybe after I’m more proficient with my current tasks I’ll return to programming, but I still think that I should limit my scope to one thing. Gamedev looks like the most fun one - I have enough data-processing tasks at work, so something interactive is more encouraging. Anyway, we’ll see. Maybe instead I’ll become a rock-climbing rock star, who knows?

Comment via email

Accepting your own solutions to your own problems

Tue, 05 Mar 2024 03:42:00 +0300

There was a weird thought going over and over in my head, regarding my Emacs configuration, and it extends to the other projects I do both at home and at work. You see, my configuration is riddled with custom code, and up until recently I had mixed feelings about that.

For example, I have a custom piece of code to automatically switch themes, based on the dark mode setting in my desktop environment. I do so via the DBUS interface:

(use-package dbus
  :when (featurep 'dbusbind)
  :requires (functions local-config)
  :commands (dbus-register-signal dbus-call-method)
  :preface
  (defun color-scheme-changed (path var value)
    "DBus handler to detect when the color-scheme has changed."
    (when (and (string-equal path "org.freedesktop.appearance")
               (string-equal var "color-scheme"))
      (if (equal (car value) '1)
          (load-theme local-config-dark-theme t)
        (load-theme local-config-light-theme t))))
  (defun dbus-color-theme-dark-p ()
    (equal '1 (caar (condition-case nil
                        (dbus-call-method
                         :session
                         "org.freedesktop.portal.Desktop"
                         "/org/freedesktop/portal/desktop"
                         "org.freedesktop.portal.Settings"
                         "Read"
                         "org.freedesktop.appearance"
                         "color-scheme")
                      (error nil)))))
  :init
  (dbus-register-signal :session
                        "org.freedesktop.portal.Desktop"
                        "/org/freedesktop/portal/desktop"
                        "org.freedesktop.portal.Settings"
                        "SettingChanged"
                        #'color-scheme-changed))

It works perfectly fine for me, yet there’s a certain feeling I experience every time I see it in my configuration.

Or, here’s another example, that kinda motivated me to write this post.

With the introduction of Tree-Sitter, Emacs now can use different parsers to provide syntax highlighting for certain modes. I use it with several modes, but the default experience is a bit rough because while there are functions to install grammars, they’re a bit inconvenient for general use. So to mitigate that, I made this function:

(use-package treesit
  :when (treesit-p)
  :preface
  (defun treesit-p ()
    "Check if Emacs was built with treesiter in a protable way."
    (and (fboundp 'treesit-available-p)
         (treesit-available-p)))
  (cl-defun treesit-install-and-remap (lang url &key revision source-dir modes remap org-src)
    "Convenience function for installing and enabling a ts-* mode.

LANG is the language symbol.  URL is the Git repository URL for
the grammar.  REVISION is the Git tag or branch of the desired
version, defaulting to the latest default branch.  SOURCE-DIR is
the relative subdirectory in the repository in which the
grammar’s parser.c file resides, defaulting to \"src\".  MODES is
a list of modes to remap to a symbol REMAP."
    (when (and (fboundp 'treesit-available-p)
               (treesit-available-p))
      (unless (treesit-language-available-p lang)
        (add-to-list
         'treesit-language-source-alist
         (list lang url revision source-dir))
        (treesit-install-language-grammar lang))
      (when (and remap (treesit-ready-p lang))
        (dolist (mode modes)
          (add-to-list
           'major-mode-remap-alist
           (cons mode remap))))
      (when (and org-src (treesit-ready-p lang))
        (eval-after-load 'org
          (lambda ()
            (add-to-list 'org-src-lang-modes org-src))))))
  :custom
  (treesit-font-lock-level 2))

I use it like this:

(use-package js
  :defer t
  :when (treesit-p)
  :init
  (treesit-install-and-remap
   'javascript
   "https://github.com/tree-sitter/tree-sitter-javascript"
   :revision "master" :source-dir "src"
   :modes '(js-mode javascript-mode js2-mode)
   :remap 'js-ts-mode
   :org-src '("js" . js-ts)))

(use-package json-ts-mode
  :defer t
  :after json
  :when (treesit-p)
  :init
  (treesit-install-and-remap
   'json "https://github.com/tree-sitter/tree-sitter-json"
   :modes '(js-json-mode)
   :remap 'json-ts-mode
   :org-src '("json" . json-ts)))

(use-package elixir-ts-mode
  :defer t
  :when (treesit-p)
  :init
  (treesit-install-and-remap
   'elixir "https://github.com/elixir-lang/tree-sitter-elixir"
   :org-src '("elixir" . elixir-ts)))

As you can see, this function handles everything from installing the grammar to remapping the major modes to use tree-sitter-backed ones, and also handles Org Mode source block languages.

Finally, I have these packages, that I made over the years of using Emacs:

(use-package isayt
  :vc ( :url "https://gitlab.com/andreyorst/isayt.el.git"
        :branch "main"
        :rev :newest)
  :delight isayt-mode
  :hook (common-lisp-modes-mode . isayt-mode))

(use-package region-bindings
  :vc ( :url "https://gitlab.com/andreyorst/region-bindings.el.git"
        :branch "main"
        :rev :newest)
  :commands (region-bindings-mode)
  :preface
  (defun region-bindings-off ()
    (region-bindings-mode -1))
  :hook ((after-init . global-region-bindings-mode)
         ((elfeed-search-mode magit-mode mu4e-headers-mode)
          . region-bindings-off)))

The isayt package automatically keeps the lisp code indented as I type, and the region-bindings package enables mappings while the region is active. Originally both these packages were just a part of my configuration, but the amount of code became a bit too big for my liking so I moved them into separate files, and uploaded them to GitLab.

Now, why am I telling you all this, you might ask, and what it has to do with the title of this post?

Well, you see, each example that I have shown you has a package that already does something like that.

The DBUS theme switching is handled by the auto-dark-emacs package, however, it also handles macOS, Termux, and Windows, while I only support my operating system of choice. Tree-Sitter is handled by the treesit-auto package, which covers much more languages than I am. For automatic indentation, there’s the aggressive-indent-mode which, unlike isayt, handles not only lisps, but other language modes as well. And the region-bindings package is basically a reimplementation of the region-bindings-mode package.

So why did I create all these things, even if there are solutions already available and used by the community?

Well, I created isayt because I wasn’t satisfied with the timer-based approach of aggressive-indent-mode. It felt laggy. Moreover, I don’t really need this in any modes other than lisps. Lisp is structured, and indentation is semantic, so having it automatically adjust is nice. Other languages either can’t do that properly, while having semantic indentation (e.g. Python) or don’t really need it, because there’s no real benefit to having your code always shift around, as it doesn’t really represent a data structure.

I made region-bindings because the original region-bindings-mode didn’t work properly with turned off transient-mark-mode (TMM). I don’t remember why I disabled it, perhaps after reading Mastering Emacs, or some other power-users note on how not having a visible region makes you more productive. In the end, I enabled TMM back, but the package already performed with fewer bugs than the region-bindings-mode so I decided to keep it.

The DBUS stuff is simple, really. At the time I wrote this code, I don’t think any packages did that. Now, it works fine, and I don’t see any reason to replace it with a package from someone else. It works, so don’t touch it. Moreover, I haven’t changed this piece of code ever since I wrote it (except for that time when the API changed in GNOME), and the auto-dark-mode is somehow constantly updated.

Finally, I just don’t need support for all of the tree-sitter-based modes out there, so it’s fine for me to throw a small use-package form with a single call to my custom function.

There are many more examples of custom weird things I do in my configuration, like custom compilation modes with complicated filename handling. Or the mu4e context managing code based on some private data structures I defined to automate the process of adding new inboxes. I have a custom memoization macro and a piece of code that decides if I can scroll my window to the left. I wrote so much code for automating stuff for this very blog that I decided to move it into a separate package. Yet, there are still pieces spread across my init.el that are related to my blog workflow.

I have a lot of custom code that is oriented to solve my problems and enhance my editing experience. After years, I learned to accept this kind of philosophy in programming - if you can solve your problem, and learn something new along the way, then do it. It doesn’t matter if the solution already exists. More often than not I’m frustrated with small annoyances of how many packages kinda do what I want, but also not exactly how I’d like to.

This also affected how I think about code in general. When faced with a problem I don’t want to reach to a library until I see that implementing a solution is either pointless, too complicated, and error-prone, or if I don’t have time to do that. Only then do I go and look for a library. I guess, I’m lucky to code in Clojure at work, as the language has a lot of useful features, making it often unnecessary to search for a library to do something. In other languages, the situation may be different.

It’s kinda the same in Emacs - the core has so much stuff that most of the time you don’t need any extra packages. You just have to dig a bit, and you’ll probably find a function that solves your problem. Or can help you solve it.

Today’s code is often complicated, and this complexity is justified by achieving so-called “reuse”.

This library is so general that you can reuse it in your very specific context! This code is so abstract that it is basically an art form!

I don’t buy that anymore. The most reusable code is the one you can copy from one project and paste into another one, and just use it. Of course, copy-pasting is just a metaphor, we create a library with that code and use it in different projects. But the code in that library could as well just be copied, meaning there were no changes to that code to make it support multiple scenarios. And if you have to support multiple scenarios - just modify this code!

That was a bit of a tangent, but I feel that it applies to what I do in my Emacs configuration. A lot of Emacs packages are way too complicated because they have to support very specific cases that the author of a package might not even have. We became too afraid to take a piece of code and modify it to our needs ourselves. And Emacs actually gives you tools to do that - the hooks, and the advice system. I advice a lot of stuff in my Emacs configuration, and I’m glad that this system is there and that it works in such a robust way that it allows me to do crazy stuff without fear that it’ll break. And if it breaks, I’ll just fix it.

It’s a fine solution.

Comment via email

Programming languages are stuck in the 80s

Fri, 01 Mar 2024 23:20:00 +0300

Maybe I’m “beating a dead horse” here, but I haven’t thought about programming languages in this particular way before, so I decided to share the thought anyway:

Most programming languages I know are designed like it’s still 80’s, and all we have are textual interfaces, and single-core CPUs.

Well, the last part isn’t entirely correct so to speak - many languages have support for multithreading, it’s just the fact that they often still list it as a feature even though we have multi-core CPUs in the home segment for 25 years now. And even if the support for parallelization is present, you often have to go out of your way to use it. Moreover, we now have GPU computing, and a lot of stuff gets computed on GPUs today, but how many programming languages have native support for that in their standard library? Even if not for GPU computing, how many languages do you know that have a way to display a window without relying on an external library that basically contains bindings to some toolkit written in C?

Look, Java, despite all the hate, can create a window with AWT. You might still wonder, why am I saying this like it’s something special? Well, because if you look at most other languages’ standard libraries, you’ll see that even if they have a lot of useful stuff, like networking, OS stuff, etc - it’s really nothing special as of today. All this stuff is there and has been there since the 80s. Except, some languages from the 80s still have more features.

I know, more languages have GUI facilities. Microsoft’s C# has facilities for GUI programming for the Windows operating system. Apple’s Swift has SwiftUI, which is one of the main ways to do GUI for Apple products. I guess Kotlin has Compose, and there’s Dart with Flutter, but I have mixed feelings about these platforms, so I’ll not delve into them. Linux… we’ll get to that.

Notice how both C# and Swift are basically platform-specific? It would be strange for these languages not to have a GUI library of sorts, because if the vendor language doesn’t provide you with one how else would you do GUI for that system? Similar thing with Kotlin and Flutter but they’re more phone-oriented, and the situation there is a bit different than on the desktop.

Well, C programmers have written multiple toolkits, like GTK, and C++ programmers have Qt, which looks decent, but then, these are not part of the language. And for Linux, this means that there’s no central way to do GUI - the most popular choices are GTK and Qt, but if you choose GTK you’ll have poor integration with Qt-based desktops, and vice versa. In simpler cases, you can use something like SDL, and just create a window, and then draw anything you want. It’s a shame though, that this isn’t just available in the language. A language that is being used for both writing a kernel, the userspace, and the GUI running in that userspace - the language itself doesn’t have the means to create a simple window. I mean, there’s Vala, but c’mon.

Nowadays, Rust people, whose main focus is to rewrite everything in their language of choice ^{(what language is it, btw?)}, are reinventing GUI toolkits all over again. Why doesn’t Rust have its own GUI library that supports all of the major operating systems? Especially given the origin of Rust is a web browser - in other words, rendering graphics should be pretty natural to Rust. Can a language really call itself modern without an inbuilt GUI library, in a world where everything runs with some kind of GUI?

The same goes for pretty much any other language that calls itself “modern” - you can call yourself that all you want, but if you can’t do something that most languages from the ’80s could, then define “modern”. And don’t come to me with all this “modern approach to programming” - garbage-collected bytecode interpreter isn’t modern. If you have a JIT and state-of-the-art GC algorithm, your VM is highly dynamic and modifiable, we can talk, but then again - does your language provide anything more than the other ones apart from subjectively nicer syntax and opinionated standard library? Pretty much all new languages are toy projects - rare few really do bring new things to the table, and deserve being called modern.

There’s an amazing talk, which I realized contains mostly everything I wanted to say in this post but in a better form: Stop Writing Dead Programs. I do recommend watching it and trying out the technologies mentioned in the talk. Many of these had a major influence on how I see most languages these days.

But let’s get back to GUI’s for a bit - what’s funny, actual languages from the 80’s didn’t have this problem.

Take for example Smalltalk - it had a lot of GUI capabilities pretty much since it was developed. Remember Xerox? They had Smalltalk with a GUI interface running on Xerox Alto in 1979! Since then we had all kinds of GUI systems available, yet so few languages took advantage of that.

Symbolics had Lisp Machines running GUI in the 80’s. Lisp itself was about 20 years old by that time, yet it got GUI support pretty much instantly everyone started doing GUI! After a bit of digging, I found that MIT Scheme had graphics support at least since 1995, but definitively since even earlier - I just couldn’t find any documents before version 7.3 to confirm. But I’m sure of it because it was used in the SICP book, and its first edition had the graphics programming section in it since the 1980s. Racket, one of the modern Schemes, has a GUI programming library bundled with the language, if you choose so. And it supports all major operating systems.

Tcl is yet another example. It’s from the 90s, but it also had GUI programming capabilities via the Tk toolkit since 1991.

I mean, sure - it’s obvious now why command line utilities, and TUI libraries are still hot even today, in 2024! Because most languages only know how to talk to standard out! Then we have curses, various TUI toolkits written in Rust or Go of all things, and still using an emulator for VT100 - a piece of tech from 1978. I guess, targeting VT100 is easier, than targeting some specific toolkit/window manager/display server/whatever, yet some languages manage somehow?

And today you don’t have to do even that - we have cross-platform browsers that run everywhere. Take JavaScript - another language many people openly hate. Yes, it has quirks in the design, and yes it has performance issues, yet we use it every day because it can draw on the screen in the browser! And visuals are the main way we perceive the world ^{^{^{Wide Web}}}. And a lot of people I know, fell in love with programming because they can actually see the results of their code immediately!

People realized that we had such poor options for GUI programming with all these system languages, GUI toolkits, static types, yada yada, and made Electron. Kudos to them, now we can build GUIs in JavaScript, without relying on any particular toolkit for better or worse.

I don’t have any false beliefs in browsers being a proper way to do this kind of stuff - if anything, I want a web browser to be a simple thing. A web browser. Not another glorified VT100 that, instead of parsing escape codes, parses JavaScript. I don’t want to run software in a browser. If you need a cross-platform way to write apps with facilities similar to what the current state of the web provides - use something other than the browser. Even Electron would be a better fit.

I know, I know, Electron is just stripped down ^{(maybe not)} Chromium, but I still think it’s better than running in a browser because it’s still a separate thing. Browsers have a specific workflow built in, and when the app tries to override it, it always works horribly (think shortcuts). Recently, with the introduction of WebAssembly and WebGPU now we have the option to draw something in the browser using other languages than JavaScript, and maybe even with more performance. Maybe in a few years, we’ll get something slimmer than Electron, with a better dynamic language that will utilize WA, and thus will become the next application platform. Though I don’t want to be too optimistic.

Speaking of application platforms, Emacs is one such application platform, again from the 80s. Look at it this way, Emacs has a runtime with GC and compilation to machine code, some GUI capabilities, and a language that can change how Emacs works almost without any limits. People have been writing applications for Emacs for many years now.

Like browsers, Emacs runs on multiple systems, and applications developed for Emacs run on different systems, if you don’t use platform-specific stuff too much. However, Emacs can’t really compete with browsers in terms of user interactivity. Well, you can have interactive stuff, it’s just janky. If you see someone saying things like Org Mode is basically Jupiter Notebook on steroids - ask them to create sliders in Org Mode that dynamically update variables in a code that is then plotted in 3D in real-time. You can do 3D plots in Org Mode, sure, interacting with 3D plots is the problem.

I mean, I’m glad I can plot the data and display the graph as an inline image in the REPL while working on some project, but can we have this instead? I’d like to see projects like Glamorous Toolkit for more languages. Now, that’s modern.

I’ve spent a major part of this post talking about GUI specifically because, well, all of my devices have a GUI, yet I can’t pick a language and draw a window with it. Recently I made a post speculating how Emacs could get a GUI library, and while I understand that this is ambitious, I still think having a decent GUI library and arbitrary drawing capabilities as a part of the language can make a great application platform.

OK, I understand, that making something good with these in-house GUI libraries is often a daunting task. Most Java AWT apps I’ve used look horrible. Same for JavaFX, Swing, etc. C# kinda can make a decently looking app, if decent means looking like all other Windows apps in your book. And other GUI libraries, that try to mimic OS GUI guidelines, like Racket’s library are fine, but you can’t really make an appealing app with that either, because you do want to design an app for each respective system, follow their guidelines, and so on. Not only that - OS GUIs are getting worse by the year too. And I think desktop OS GUIs are getting worse partly because of advancements in mobile GUIs.

The smartphone industry gave another shake to GUI programming because it is a different medium with different control schemes, and thus we have new ways to do GUI. And platform languages, as I mentioned, have decent GUI programming support. Yet, people still often just want to bring something like React Native, because it is easier to make an interface in it, and ship it to all platforms, instead of developing several GUIs. So we have the Electron situation on our hands all over again.

But I don’t want you to think that I believe that GUI is the only thing that makes a language modern - no, of course not. Just, let’s put it this way - go and research a bit of programming language history, and you’ll probably find a lot of interesting things. Like the fact, that JIT compilation was a thing in APL of all things in 1979. Forty-five years ago.

Anyway, what I wanted to say with this post is simple. We already have 65+ years of programming language history, yet, most new languages seem to completely ignore most of it, repeating what’s been done in the earliest years, calling it modern.

Comment via email

Using transducers

Wed, 31 Jan 2024 21:57:00 +0300

I’ve been working with Clojure professionally for four years now, and I made some posts about the language in the past. Clojure is a great language, although not without its fair share of things to consider. In other words, I don’t see Clojure as an ideal language by any means, and it’s not suitable for every type of project. Still, the language is my favorite among others, and I enjoy writing and reading code in it.

Now, I already made a post about transducers before. In that post I tried to explain what transducers are, how they work, and more importantly how they compose. I think since then I had a good grasp on what transducers are, and I use them when appropriate. However, I still see some concerns when it comes to transducers in the community, even though I’m almost not involved in it, and I see this even at work sometimes. So, this post isn’t going to be hard on the technical details behind transducers, but more about how and when you should use them.

Don’t be lazy until it matters

Clojure has a weird aspect to it that trips a lot of people - Clojure is lazy by default. Often I see this kind of code:

(clojure.string/join separator (map foo bar))

This is just an example, of course, but the pattern is not uncommon - we build a sequence and consume it. Ideally, though this code should use mapv instead of map because map is lazy. Lazy sequences in Clojure come with a bit of overhead, and in this case we’ll be consuming everything immediately, thus this overhead is not justified. I’ll admit, this particular example can benefit from consuming a lazy sequence, if the sequence is large enough. This will ensure that we don’t store both the string and the sequence itself in the memory, only the string and one of the elements of the sequence. Often, however, programmers just type out map because it’s a common concept, not thinking about it being lazy, and whether if the laziness here is beneficial or not. Now, that’s not about transducers, really, but it’ll make sense in a few minutes why I bring this up.

The spear of lazy poisoning

Clojure has a lot of convenience features in its core library, and I think that’s partly the reason why people, myself included, like the language. One of such conveniences are -> and ->> macros. These are hard to understand at first, but once you get what they do, I think it’s a safe bet that you’ll going to use them pretty much everywhere. It makes the code easier to read. Now, these macros don’t do anything crazy, they just make deeply nested expressions appear unnested.

For example, here’s an expression that maps over the range of ten numbers, incrementing everything, filtering odd numbers, and adding them together:

(reduce + (filter odd? (map inc (range 10))))

Notice how the expression is written pretty much backward from what I’ve described. First comes reduce, then filter, then map, and, finally, range. Lisps are known for this kind of notation, and you often read the code backward. In a more mainstream language, this would probably have been written like this:

[0,1,2,3,4,5,6,7,8,9].map(x=>x+1).filter(x=>x%2!=0).reduce((res,x)=>res+x)

As you can see, the order of operations feels more logical, even though it’s just a syntactic abstraction. Methods, really, are just functions, and their first argument simply is what appeared before the dot. In its essence, the above code is equivalent to this code, even though it’s not a valid JavaScript:

reduce(filter(map([0,1,2,3,4,5,6,7,8,9], x=>x+1), x=>x%2!=0), (res,x)=>res+x)

So we’re back to the start, reduce is executed last, but appears first. The . allows us to express this more sequentially, and that’s what Clojure’s -> and ->> are for. The original code can be rewritten as:

(->> (range 10) (map inc) (filter odd?) (reduce +))

The difference between -> and ->> is only where the previous expression is going to be placed in the next expression. The thread-first (-> )puts it as the first argument, and thread-last (->>) as the last: (-> 42 (+ 1 2)) expands to (+ 42 1 2) and (->> 42 (+ 1 2)) expands to (+ 1 2 42). This repeats for all forms in the threading macro. Now, that was a pretty long tangent, but I felt that this needs explaining, because we’re going to see the use of ->> in the following examples.

Why did I call this section “The spear of lazy poisoning”? Well, I think that ->> looks like a spear, or a harpoon, and it often poisons the code with unnecessary laziness. You can even see that in the example above - both map and filter are lazy, yet the reduce is eager, and all laziness up to that point was unnecessary because the data we were processing is finite. This gives us the laziness overhead, which is mostly computational. We can, of course, use mapv and filterv - the eager versions of map and filter that produce vectors instead, but it would also harm us:

(->> (range 10) (mapv inc) (filterv odd?) (reduce +))

Now we construct a whole vector only to discard half of it in the next step, then discarding the remaining half in the reduce. That’s a memory overhead we’d also like to avoid. So what should we do?

Transducers

Transducers are used by the transduce function which is like reduce except it doesn’t accept just a reducing function, it also accepts a transforming function. Before we go into that, let’s go back to the lazy example for a moment:

(->> (range 10) (map inc) (filter odd?) (reduce +))

I think it’s important to build intuition of how it works.

The operations are lazy, so by the time we get to the reduce, nothing happened yet. What we get in the reduce is a lazy composition of two operations:

Take one item from the range sequence, and process it with inc;
Take one item from sequence built by map, filter it with odd?; a. If the result is not truthly, go to step 1;
Reduce it to the result using +.

This means that in reality, the sequence that the reduce is processing looks kinda like [(odd? (inc 0)), (odd? (inc 1)), (odd (inc 2)), ...], with the exception that filtered items magically disappear from it. These expressions are lazy, and not computed until they get into reduce which realizes them. This, of course, isn’t how it works, but you can use this model for building intuition.

If you want to read more about lazy sequences, I have a post, describing the implementation of a simple lazy sequence. It’s in Lua, but the code should be simple enough to understand even if you don’t know the language.

Now, the key part here is that we’re composing operations by delaying them - we could say that we’re building a recipe. However, each operation is lazy, but, as you can see, we don’t need every operation to be lazy, as once we get a number we can process it eagerly from there. In other words, if we could combine both map and filter into a single step, we could avoid the overhead of multiple lazy compositions.

Now, Clojure has such a function, it’s called keep:

(->> (range 10)
     (keep #(let [x (inc %)]
              (when (odd? x)
                x)))
     (reduce +))

However, keep is also lazy, and in this case, again, there’s no point in being lazy. And we may need to work with something more general, i.e. not always want to combine map and filter, sometimes we need to combine multiple map calls, or even some other things, like partitioning. That’s where transducers come in nicely.

Clojure developers actually wanted to solve a different problem when they introduced transducers, and I’ll get to it a bit later. For now, we can rewrite our code to be both eager and memory-efficient:

(loop [items (seq (range 10))
       result 0]
  (if items
    (let [x (inc (first items))]
      (recur (next items)
             (if (odd? x)
               (+ result x)
               result)))
    result))

Just kidding. That’s a lot of machinery for describing the core concepts of the language. Here’s how to do it with transduce:

(->> (range 10) (transduce (comp (map inc) (filter odd?)) +))

As you can see, it is different from reduce because both map and filter are not involved with the collection processing at all. Instead, they go as a parameter to the transduce function, and we use comp to compose the two into a single step that does everything. Importantly, both map and filter calls are made without any collection supplied to them - this way these functions return transducers. I tried my best explaining how transducer composition works in my previous post on the matter, so read it if you want to know.

However, that’s a lot to take in, as transducers are quite involved, so here’s a simpler intuition for you.

If you want to refactor any transduceible expression into a transducer, first, transform it into the ->> form, and then replace the ->> with comp:

(reduce + (filter odd? (map inc (range 10))))

;; rewrite using ->>:
(->> (range 10)
     (map inc)
     (filter odd?)
     (reduce +))

;; replace ->> with comp
(comp (range 10)
      (map inc)
      (filter odd?)
      (reduce +))

We’re halfway through, though. This code obviously wrong, but what’s left out to do is to understand what it does, and finish the transformation. The first step is always the same though, we need to remove the data that we’re processing from this expression:

(range 10) ; just move it aside for now

(comp (map inc)
      (filter odd?)
      (reduce +))

Finally, reduce isn’t transduceible, so we need to move it out and replace it with something that can accept the transducer, like transduce:

(transduce
 (comp (map inc)
       (filter odd?))
 +
 (range 10) ;; with transduce we know where to put the data
 )

That’s basically it. You can apply this method to pretty much any ->> expression, as long as the calls done there have arities that return transducers.

More examples

I know, the example above was weird, so here’s a different one:

(->> (get-users)
     (filter active-user?)
     (map fetch-capabilities)
     (filter admin?)
     (map fetch-notification-settings)
     (notify))

In this example, we build a sequence of users who have the administrator role on the server and add some additional data while we’re at it. We then notify each such user. This is not a real-world code, though I have seen something like that in production.

The notify, again, consumes all of the users, so we don’t really need the laziness, here, unless get-users doesn’t return an infinite sequence of users, which it is not. But even if it did, we don’t really need all of the steps to be lazy here, so we can go about this two ways.

If the list of users is short, we can store it in a vector. Let’s refactor the code:

;; start with `comp`:
(comp (get-users)
      (filter active-user?)
      (map fetch-capabilities)
      (filter admin?)
      (map fetch-notification-settings)
      (notify))

;; move out data and consumer:

(get-users)
(comp (filter active-user?)
      (map fetch-capabilities)
      (filter admin?)
      (map fetch-notification-settings))
(notify)

;; Since we want a vector, we can use `into`:

(notify
 (into []
       (comp (filter active-user?)
             (map fetch-capabilities)
             (filter admin?)
             (map fetch-notification-settings))
       (get-users)))

;; let's bring back ->> to bring nesting down a bit

(->> (get-users)
     (into []
           (comp (filter active-user?)
                 (map fetch-capabilities)
                 (filter admin?)
                 (map fetch-notification-settings)))
     (notify))

If we still want to consume users lazily, we can use sequence:

(->> (get-users)
     (sequence ;; now the result is a lazy sequence
      (comp (filter active-user?)
            (map fetch-capabilities)
            (filter admin?)
            (map fetch-notification-settings)))
     (notify))

It’s a small change from the previous example, but it has benefits over the original example. In the original example, we had all processing done lazily, however, we only needed to consume users lazily, and the transformation can be done eagerly for each user. We just don’t want to do the transformation in the notify function, because it is not its task really. The sequence function consumes input lazily, applying a composed transformation to every element, and produces a lazy sequence as a result. Thus we only have overhead of one lazy sequence, instead of four. Your profiling flame graphs would look much nicer this way.

The beauty of this is that ->> transforms to comp almost one-to-one. The problem is, that it is counter-intuitive because for ordinary functions the order is the opposite. So keep that in mind.

For ordinary function, the composition works like this:

((comp a  b  c  d) x) ; same as the expression below
                      ; (expressions are aligned for clarity)
      (a (b (c (d  x))))

For transducers, it works exactly the same, except the argument we’re composing with is not the value, it’s a reducing function. The resulting function processes the value in the order specified by the code, so that’s one of the reasons people tend to dislike transducers. My advice is just to remember that the order for transducer composition is the same as in ->>.

Other contexts

Now, the reason why transducers are a thing is that when working on other libraries for the Clojure ecosystem, the authors noticed that while mapping, filtering, and reducing a core concept, it can’t be reused. One such example is the clojure.core.async library. At one point they needed to map a function over a channel, but you can’t really do that. Items appear on channels only when someone puts them, so map can’t pull on the channel. Moreover, map would need an extra channel because it can’t put the item back into the original channel.

Originally, core.async had its own version of map and filter, called map> and filter>. However, these were deprecated in favor of transducers.

Say, get-users doesn’t return you a lazy sequence, but instead returns you a core.async channel. How would we need to change our processing pipeline from the lazy variant?

(def user-pipeline
  (comp (filter active-user?)
        (map fetch-capabilities)
        (filter admin?)
        (map fetch-notification-settings)))

(let [admins (async/chan 1 user-pipeline)]
  (async/pipe (get-users) admins)
  (notify admins))

Imagine we’ve changed notify to accept a channel. Now, we can create a channel with our transformation pipeline as a transducer. As a matter of fact, core.async provides a way of running such actions in parallel:

(let [admins (async/chan)]
  (async/pipeline 10 admins user-pipeline (get-users))
  (notify admins))

Extracted the transformation into a separate binding makes the code a bit more concise, and reusable. And that’s exactly what transducers are about - we can abstract the action away from how the data arrives and how it should be stored. Here’s another example, with the manifold library:

(->> (get-users)
     (manifold.stream/->source)
     (s/transform user-pipeline)
     (notify))

The same transformation pipeline works for manifold.stream because it also accepts transducers with the transform function. We can then modify notify to accept a stream instead of a channel and even have an interop between both libraries.

So, there you have it - we can process, eager collections, lazy sequences, asynchronous channels, and manifold streams, using the same data pipeline:

(->> (get-users-seq)
     (into [] user-pipeline)
     (eager-notify))

(->> (get-users-seq)
     (sequence user-pipeline)
     (lazy-notify))

(let [admins (async/chan 1 user-pipeline)]
  (async/pipe (get-users-chan) admins)
  (chan-notify admins))

(->> (get-users-chan)
     (manifold.stream/->source)
     (s/transform user-pipeline)
     (stream-notify))

Final thoughts

While transducers are great, there are always things to consider.

Let’s start with the elephant in the room - the code using transducers is often ugly. Like, let’s go a bit back and compare these two examples:

(->> (get-users)
     (filter active-user?)
     (map fetch-capabilities)
     (filter admin?)
     (map fetch-notification-settings)
     (notify))

(notify
 (into []
       (comp (filter active-user?)
             (map fetch-capabilities)
             (filter admin?)
             (map fetch-notification-settings))
       (get-users)))

The first one is so slick, yet inefficient. The second one looks like a tool that can do the job efficiently, but you’re going to hurt yourself in the process.

Well, we can refactor the second one a bit more, moving out the comp call, using ->> on the result, but honestly, it’s not often when I see comp moved out and defined as binding. More often it’s written in this crud form. However, this can be fixed for many scenarios with macros. There’s a library that provides x> and x>> macros that automatically use transducers. Here’s what they have in the readme:

(->> (range 10000000)
     (map inc)
     (filter odd?)
     (mapcat #(do [% (dec %)]))
     (partition-by #(= 0 (mod % 5)))
     (map (partial apply +))
     ;; (mapv dec)
     (map (partial + 10))
     (map #(do {:temp-value %}))
     (map :temp-value)
     (filter even?)
     (apply +)
     time)

;; "Elapsed time: 8275.319295 msecs"
;; 5000054999994

(x>> (range 10000000)
     (map inc)
     (filter odd?)
     (mapcat #(do [% (dec %)]))
     (partition-by #(= 0 (mod % 5)))
     (map (partial apply +))
     ;; (mapv dec)
     (map (partial + 10))
     (map #(do {:temp-value %}))
     (map :temp-value)
     (filter even?)
     (apply +)
     time)

;; "Elapsed time: 2913.851103 msecs"
;; 5000054999994

That’s two to three times the speed increase by one character change. If we expand the macro, we’ll see that it simply builds a transducer with some internal machinery:

(->> (range 10000000)
     ((injest.impl/xfn [[clojure.core/map inc]
                        [clojure.core/filter odd?]
                        [clojure.core/mapcat (fn* [p1__8713#] (do [p1__8713# (dec p1__8713#)]))]
                        [clojure.core/partition-by (fn* [p1__8714#] (= 0 (mod p1__8714# 5)))]
                        [clojure.core/map (partial apply +)]
                        [clojure.core/map (partial + 10)]
                        [clojure.core/map (fn* [p1__8715#] (do {:temp-value p1__8715#}))]
                        [clojure.core/map :temp-value]
                        [clojure.core/filter even?]]))
     (clojure.core/apply +)
     time)

The result of calling injest.impl/xfn is a function that builds a transducer and passes it into a sequence call. So there’s no magic, apart from a clever way of transforming the code into this form.

But, there’s another problem with transducers - not every transformation pipeline can be described as one. Notice that there’s a commented-out call to mapv. Unlike map, mapv doesn’t return a transducer, and is always eager. If we were to refactor this code into comp we would get an error. The x>> macro handles this by breaking the transducer into two parts:

(->> (range 10000000)
     ((injest.impl/xfn [[clojure.core/map inc]
                        [clojure.core/filter odd?]
                        [clojure.core/mapcat (fn* [p1__8724#] (do [p1__8724# (dec p1__8724#)]))]
                        [clojure.core/partition-by (fn* [p1__8725#] (= 0 (mod p1__8725# 5)))]
                        [clojure.core/map (partial apply +)]]))
     (clojure.core/mapv dec)
     ((injest.impl/xfn [[clojure.core/map (partial + 10)]
                        [clojure.core/map (fn* [p1__8726#] (do {:temp-value p1__8726#}))]
                        [clojure.core/map :temp-value]
                        [clojure.core/filter even?]]))
     (clojure.core/apply +)
     time)

That’s another clever trick, and it certainly helps.

Now, I don’t use this library in practice. Instead, I tend to write transducers by hand, because I want them to be understandable. It’s easy enough to explain how comp works with transducers to colleagues once, than to close my eyes and believe that the library will do everything for me. Maybe I’m old-fashioned.

However, there’s another difference that we need to remember about - transducers have a bit different laziness semantics. You can create lazy sequences using transducers with the sequence function, however, you should be mindful that any step in your transducer will still be executed eagerly.

Finally, not all functions provide a transducer counterpart to them. There are libraries, again, that provide more transducers, but I’d much rather see more transducers in the Clojure core.

And sometimes, there’s no need to do that at all - lazy transformations can be fine, or exactly what you need in a particular context. So don’t just rush out replacing all consecutive calls to map and filter with a transducer. Although, if you want to, here’s a clj-kondo hook that issues a warning when you use too many lazy transformations in a row:

(ns hooks.thread-macros
  (:require [clj-kondo.hooks-api :as api]))

(def consecutive-threshold 2)
(def transduceable
  #{'map
    'filter
    ;; add more functions if you need
    })

(defn consecutive-threading [{:keys [:node]}]
  (let [[_ & forms] (rest (:children node))]
    (loop [forms forms
           counter 0]
      (when-let [form (first forms)]
        (if (contains? transduceable (some-> form :children first api/sexpr))
          (if (>= counter consecutive-threshold)
            (api/reg-finding!
             (assoc (meta node)
                    :message "prefer transducers to consecutive maps or filters in threading macros"
                    :type :expensive-threading))
            (recur (next forms)
                   (inc counter)))
          (recur (next forms)
                 0))))))

Code Snippet 1: .clj-kondo/hooks/thread_macros.clj

{:linters {:expensive-threading {:level :warning}}
 :hooks {:analyze-call {clojure.core/->> hooks.thread-macros/consecutive-threading}}}

Code Snippet 2: .clj-kondo/config.edn

That’s all from me, and I hope you’ve found this post interesting. Reach out, if you have any interesting examples where transducers were hard to apply, or where they made the code more robust. And I hope transducers will get a better rep in the future.

Comment via email

Thoughts on Crafting Interpreters - Part 3

Thu, 25 Jan 2024 23:23:00 +0300

The unexpected part!

I liked hacking on Lox in Zig a lot, so I decided it would be great to make some changes to the language. It should be good for a better understanding of the book’s material, and probably will be a lot of fun!

Minor changes from the book

I looked back on what we’ve done in the book and decided to do some tweaking.

First, I made print to be a function and not a statement. Making it a statement was understandable, as there was no way to call anything in the language yet, and in order to see some evaluation results print statement was introduced. Changing it to be a function has some benefits.

First, we can pass more than one argument, and automatically concatenate those with a space. It’s not the same as printf in C, and with other functions that accept format strings in various languages, but still handy:

print(1, 2, 3) // prints: 1 2 3\n

Secondly, we can pass it as a callback to other functions, which can’t be done with statements. Lox isn’t a real-world language, so this isn’t something that people will probably want to be able to do, but still, it’s an advantage over print being a statement.

Speaking of passing functions - why are there no lambdas in the book? IIRC, the first half had a “Challenge”, for implementing a support of anonymous functions, and in the second half, there was none. It’s not like it’s hard either. I added two possible ways to define an anonymous function.

The first one has exactly the same syntax as a named function, except you don’t specify the name:

fun (x, y, z) { return x + y + z; }

You can assign this expression to a variable, or call it immediately:

ziglox> var foo = fun (x, y, z) { return x + y + z; };
nil
ziglox> return foo(1, 2, 3);
6
ziglox> return (fun (x, y, z) { return x + y + z; })(4, 5, 6);
15

But it’s too verbose, so I added another syntax using a backslash:

ziglox> var bar = \x,y,z -> x + y + z;
nil
ziglox> return bar(1,2,3);
6
ziglox> return (\x,y,z->x+y+z)(4,5,6);
15

It’s a bit terse, and I’m not sure if I like it or not, but it works.

Speaking of backslashes - the book doesn’t deal with string escaping at all. I understand that dealing with it is kinda tricky - you need to escape quotes, escape the already escaped quotes and escape characters, you need to correctly unescape everything when printing, etc. So it being omitted from the book is OK, I guess, but I decided to implement this in the parser and in the tostring function that I added as one of the book’s “Challenges”:

ziglox> return "\"foo\""; // nested strings can be escaped
"\"foo\""
ziglox> print("\"foo\""); // printing correctly unescapes everything
"foo"

This will play nice with changes I’ll describe a bit later.

As a part of another “Challenge”, I changed the virtual machine in such a way that it avoids consecutive popping and pushing in most of the instructions where possible. This was supposed to make the VM slightly faster, however, it did not have any effect on speed, at least how I measured it.

Finally, I decided to add the type function, so the user will be able to tell what kind of an object the function received. This will play nicely with what I will talk about a bit later.

Objects gone

As you may remember, in my previous post on the topic of this book I said that I don’t want to implement classes, and instead would prefer more general hash tables. And, I also mentioned that the lack of arrays is a strange decision, that limits language capabilities quite heavily. So I decided to implement data structures!

Arrays

The first data structure I added was arrays. Adding arrays was easy enough so I still think it is weird that the book doesn’t mention them at all.

First, I needed a few new instructions:

pub const OPCode = enum(Code) {
    // ...
    Array,
    SetIndex,
    GetIndex,
    InvokeFromArray,
};

I chose to implement the syntax for indexing arrays as in most languages, i.e. array[index]. Similarly to how we did with invoking instances, I added other instructions for calling array indexes directly: array[index](args). It’s a minor optimization, but I wanted to do it to better grasp it. This meant that in the scanner I had to implement two new tokens:

--- a/src/zig/lox/scanner.zig
+++ b/src/zig/lox/scanner.zig
pub const TokenType = enum {
    // ...
    LEFT_BRACKET,
    RIGHT_BRACKET,
    // ...
};

// in the scanner's switch statement:
        switch (c) {
            '[' => return self.makeToken(.LEFT_BRACKET),
            ']' => return self.makeToken(.RIGHT_BRACKET),
            // ...
        }

In the compiler, these tokens receive a prefix and infix parse rules:

fn makeRules() []ParseRule {
    // ...
    r[@intFromEnum(TokenType.LEFT_BRACKET)] = .{ .prefix = Parser.array, .infix = Parser.arrayIndex, .precedence = .CALL };
    r[@intFromEnum(TokenType.RIGHT_BRACKET)] = .{ .prefix = null, .infix = null, .precedence = .NONE };
    // ...
}

Rules themselves are simple:

pub const Parser = struct {
    // ...

    fn arrayIndex(self: *Parser, canAssign: bool) LoxError!void {
        try self.expression();
        self.consume(.RIGHT_BRACKET, "Expect ']' after an index.");
        if (canAssign and self.match(.EQUAL)) {
            try self.expression();
            try self.emitByte(@intFromEnum(OPCode.SetIndex));
        } else if (self.match(.LEFT_PAREN)) {
            const argCount = try self.argumentList();
            try self.emitByte(@intFromEnum(OPCode.InvokeFromArray));
            try self.emitByte(argCount);
        } else {
            try self.emitByte(@intFromEnum(OPCode.GetIndex));
        }
    }

    fn array(self: *Parser, _: bool) LoxError!void {
        var elemCount: Code = 0;
        if (!self.check(.RIGHT_BRACKET) and !self.check(.EOF)) {
            var do = true;
            while (do or self.match(.COMMA)) {
                do = false;
                try self.expression();
                elemCount += 1;
                if (elemCount == 255)
                    self.error_("Can't have more than 255 array elements.", .{});
            }
        }
        self.consume(.RIGHT_BRACKET, "Expect ']' after expressions.");
        const arr = (try Array.new(self.vm, elemCount)).obj.asValue();
        try self.emitBytes(@intFromEnum(OPCode.Array), try self.makeConstant(arr));
        try self.emitByte(elemCount);
    }

    // ...
}

Now, onto the object itself.

First, I added the new struct, called Array:

pub const Array = struct {
    const Self = @This();
    obj: Obj,
    values: ArrayList(Value),

    pub fn new(vm: *VM, capacity: u8) LoxError!*Self {
        const obj = try Obj.allocate(vm, Self);
        const out = obj.as(Self);
        out.* = Self{
            .obj = obj.*,
            .values = ArrayList(Value).initCapacity(vm.allocator, @max(4, @as(usize, @intCast(capacity)))) catch
                return LoxError.OutOfMemoryError,
        };
        return out;
    }

    pub fn destroy(self: *Self, vm: *VM) void {
        self.values.deinit();
        vm.allocator.destroy(self);
    }

    pub fn markValues(self: *Self, vm: *VM) !void {
        for (self.values.items) |value| {
            try value.mark(vm);
        }
    }

    pub fn equal(self: *Self, other: *Self) bool {
        if (self.values.items.len != other.values.items.len) return false;
        for (self.values.items, 0..) |item, i| {
            if (!item.equal(other.values.items[i])) return false;
        }
        return true;
    }
};

As you can see, it’s basically a thin wrapper around the ArrayList from Zig’s standard library.

I also implemented the equality semantics. If array lengths are equal and all elements of both arrays are equal then the arrays are equal. We don’t do the deep comparison if arrays are the same objects. Pretty standard stuff.

In the VM, creating the array is straightforward:

const op: OPCode = @enumFromInt(frame.readByte());
switch (op) {
    // ...
    .Array => {
        const arr = frame.readConstant().as(*Array);
        const valueCount = frame.readByte();
        for (0..valueCount) |i| {
            arr.values.append((self.stackTop - (valueCount - i))[0]) catch
                return self.runtimeError("OOME: can't append to array", .{});
        }
        self.stackTop -= valueCount;
        self.push(arr.obj.asValue());
    },
    // ...
};

Maybe a bit sub-optimal, but we had to do it now, as expressions are not always constant.

Tables

A.K.A. dictionaries, hash maps, key-value storage, etc.

I have a firm belief, that an object is just a special case of a hash map, and not vice-versa. With closures, anonymous functions, and hash maps we can implement objects and most of their properties without resorting to making it a part of the language. Well, if we want this to be really efficient, it is probably better to make it part of the language, but I don’t care about OOP code efficiency, or OOP at all for that matter. So I removed all of the code that was related to instances, classes, and objects, and replaced it with a third of the code needed to implement hash tables.

Same as arrays, hash tables required new syntax. I decided to do it his way:

#{"foo" => "bar", "baz" => ["qux", "wax"]}

Additionally, string keys can be written as .key:

#{.foo => "bar", .baz => ["qux", "wax"]}

This matches the key access dict.key.

Implementing this syntax required adding another character to the scanner:

@@ -253,6 +242,7 @@ pub const Scanner = struct {
+            '#' => return self.makeToken(.HASH),
         }

And a new parse rule in the compiler:

fn makeRules() []ParseRule {
    // ...
    r[@intFromEnum(TokenType.HASH)] = .{ .prefix = Parser.hashTable, .infix = null, .precedence = .NONE };
    // ...
 }

The rule is again quite simple:

    fn hashTable(self: *Parser, _: bool) LoxError!void {
        self.consume(.LEFT_BRACE, "Expect '{' before table body.");

        var kvCount: u32 = 0;
        while (!self.check(.RIGHT_BRACE) and !self.check(.EOF)) : (kvCount += 2) {
            if (self.match(.DOT)) {
                self.consume(.IDENTIFIER, "Expect property name after '.'.");
                const s = try String.copy(self.vm, self.previous.name);
                try self.emitConstant(s.obj.asValue());
            } else {
                try self.expression();
            }
            self.consume(.FAT_ARROW, "Expect '=>' after key expression.");
            try self.expression();
            if (!self.check(.RIGHT_BRACE) and !self.check(.EOF))
                self.consume(.COMMA, "Expect ',' after value expression.");
        }

        self.consume(.RIGHT_BRACE, "Expect '}' after table body.");
        if (kvCount % 2 != 0) self.error_("Missing value expression", .{});
        if (kvCount > 255) self.error_("too many key-value expressions", .{});
        const table = (try HashTable.new(self.vm)).obj.asValue();
        try self.emitBytes(@intFromEnum(OPCode.Table), try self.makeConstant(table));
        try self.emitByte(@intCast(kvCount));
    }

In the VM it’s again, pretty straightforward:

const op: OPCode = @enumFromInt(frame.readByte());
switch (op) {
    // ...
    .Table => {
        const table = frame.readConstant().as(*HashTable);
        const valueCount = frame.readByte();
        var i = valueCount;
        while (i > 0) : (i -= 2) {
            const slot = (self.stackTop - i);
            _ = table.fields.set(slot[0], slot[1]) catch
                return self.runtimeError("OOME: can't append to array", .{});
        }
        self.stackTop -= valueCount;
        self.push(table.obj.asValue());
    },
    // ...
};

Array indexing also replaced the get and set functions for tables:

var tbl = #{"foo bar" => "baz"};
var key = "foo bar";

// before:
get(tbl, key); // "baz"
set(tbl, key, "qux");

// now:
tbl[key]; // "baz"
tbl[key] = "qux";

Perhaps a bit unconventional to be able to use array indexing on tables, but I find this OK.

In addition to hash tables, I added hash sets:

var set = #["foo", "bar", "baz"];

I thought about using #{} syntax as in Clojure, but it meant that I needed to choose what data structure to use once I met any of the keys and decide what was a syntactic error and what wasn’t. For example, is it an incorrect hash set or an incorrect hash map?

#{"foo", "baz" => "qux"}

Perhaps, it was meant to be a hash set, but was confused and added => between the elements, because it has the same syntax. Then the error something like "unexpected token '=>' in the hash set literal" would be the correct one. Or, if the programmer wanted a hash map and just forgot to add a value for the first key, then the error would be confusing. So I chose #[] instead.

Multiple value returns

That was a feature I wanted to add, but couldn’t find a suitable way of doing it.

One thing I don’t like about Lua is how it implemented multiple value returns. It has a lot of gotchas, and when something in the language has a gotcha it’s a bad thing in my opinion. You can read this article for a lot of interesting examples, so I’ll keep it brief here.

For example, here’s a function that returns several values:

function foo ()
   return 1, 2, 3
end

We can call a function that accepts multiple values, and pass it the result of calling this function, for example, print is one such function:

print(foo()) -- prints 1	2	3

We can prepend more arguments before the call to foo:

print(0, foo()) -- prints 0	1	2	3

But we can’t do the same after the call to foo:

print(foo(), 4) -- prints 1	4

That’s a gotcha, and it often hits people when they use table.insert in combination with some function that can return multiple values. For example, string.gsub:

local words = {"hello", "world"}
local capitalized_words = {}

for _, word in pairs(words) do
   table.insert(
      capitalized_words,
      string.gsub(word, "^%l", string.upper)
   )
end

If we execute this code, we’ll get the following error:

bad argument #2 to 'insert' (number expected, got string)

Why? Because string.gsub returns the string and then the index where the replacement occurred. And table insert accepts a table, index, and a new value: table.insert(t, 3, "foo"). You can omit the index, and just supply the value like this table.insert(t, "foo") and the value will be inserted as the last in the table. In other words, it’s the same as table.insert(t, #t, "foo"). When we pass string.gsub as the expression that should produce a value, it actually returns two values, and table.insert thinks that we use it’s three-argument arity and fails.

There are many more gotchas to multiple values than that, and the most annoying one perhaps is that you can’t store them in a variable to create a closure. You have to create a table with your variadic arguments if you want to use them in some other inner function. In other words, this code is invalid:

function a(...)
   return function()
      print(...)
   end
end

We get this error:

cannot use '...' outside a vararg function near '...'

Instead, we need to do this:

function a(...)
   -- or use table.pack(...) in lua 5.2 and above
   local va = {n=select("#", ...) ...}
   return function()
      -- unpack was moved to table.unpack in Lua 5.2
      print((table.unpack or unpack)(va, 1, va.n))
   end
end

This has a performance penalty, it is inconvenient, and if you have to mix variadic args from multiple functions you have to copy them to a single table, and then unpack them.

Sorry for this long tangent, but when I was thinking about how I would implement multiple values in Lox. I’m OK with the syntax of Lua here, e.g. return a, b, c and local a, b, c = foo(). However, this was hard to do properly.

I had to change how the return statement works. It’s not hard, we can read values until we find the ; token. We also count the values, and emit the count with the instruction:

fn returnStatement(self: *Parser) LoxError!void {
    if (self.match(.SEMICOLON)) {
        try self.emitReturn();
    } else {
        var valueCount: u32 = 0;
        var do = true; // do-while when?
        while (do or self.match(.COMMA)) {
            do = false;
            try self.expression();
            if (valueCount == 255)
                self.error_("Can't return more than 255 values.", .{});
            valueCount += 1;
        }
        self.consume(.SEMICOLON, "Expect ';' after return value.");
        try self.emitBytes(@intFromEnum(OPCode.Ret), @intCast(valueCount));
    }
}

OK, we can emit the necessary code, but what do we do with it?

One thing we can do is create a slice from the stack and return it as a single value. Then, when we use it as a function parameter, or in an expression, we do the appropriate thing. However, this isn’t particularly great, because the stack is mutable, and we manipulate it extensibly.

ziglox> var x = (\-> return 1, 2, 3)();
nil
ziglox> print(x)
1 2 3
ziglox> return x + 10;
20

Oops, we’ve pushed to the beginning of the stack, and x became 10. Well, perhaps we can keep the stack slice, and just not decrement it, but we need to decide when to decrement it, i.e. once multiple values are no longer used. Which is hard to track.

The reason I wanted to be able to store multiple values in a single variable is because I want them to behave more like tuples, but that are always on the stack, no matter what. So they remain fast, without additional memory allocations. And to be able to make a closure over the multiple values, so we don’t have to fiddle with them like in Lua. This, however, had problems all over the place.

I also needed to implement a syntax for assigning multiple values to multiple variables, but I felt that at this point I was kinda burned out with the project. I’m satisfied with the data structures that I added, and other stuff that is not available in the book. All in all, I feel that I had a good grasp of what this book tried to cover.

Comment via email

Thoughts on Crafting Interpreters - Part 2

Mon, 15 Jan 2024 07:02:00 +0300

This is a second post about the Crafting Interpreters book by Robert Nystrom.

In the first post, I’ve described my experience with the first half of the book, and the challenges of using a different language with different idioms and practices. This post will be no different, although I have a bit more to discuss, and the contents aren’t actually ~2-year-old weak impressions and remembrances. That being said, this is a second iteration both on this post and on the book so a lot of frustration I had with the book the first time I went through is coming from memory. Well, these memories aren’t as old, as I did the project more recently, but still, I’ll point out when the complaints come from the old version or the new one.

Same as before, I decided to choose a different implementation language, rather than using the one suggested by the author. Please note, that this decision isn’t done for the sake of just using a different language - I have pretty practical reasons to do that. Even more so than in the first half of the book.

For the bytecode VM, Robert chooses the C programming language. It is again, a reasonable choice, and I agree, that for a book on a language runtime, there are probably no other valid choices. For a book, that is.

I’m not writing a book, however, so I felt that C is a rather poor choice for an actual implementation. Not that I’m suggesting that C is a bad language for this kind of task - it has several advantages over most others, but in general, C isn’t the best pick today. C demands a lot of knowledge of how computers work, and it is the least forgiving language when it comes to fixing bugs down the line. We’ve been writing C since the 1970s and haven’t perfected it by any means. Memory errors are everywhere, programs segfault, and a lot of projects dedicated to finding errors in C code are basically required for any modern work done in C because humans are, in fact, humans.

If someone claims that they know C, they are either lying or are extremely rare cases of people who actually know what they’re doing. I don’t think I have ever publicly said that I know C, but I did work with C professionally in the bare metal context, and it’s a whole different story to working with C in the context of the operating system. So even there we can’t say that we really know C if we only use it in one of the many possible contexts where you can use C. Well, at least I worked with ASM as well, so when I saw the C code I could roughly see what ASM would correspond to it.

Yes, C is still good for teaching the concepts behind data structures, used for language implementation, as you do have to implement them for yourself a lot. The context that C is used in often requires this, because libraries might not be available for your specific use case. So implementing stuff in C, especially given that this language just steps aside and says: “go”, not holding your hand, not preventing you from doing crazy shit like many modern languages often do. Oh, so you want to create an array, fill it with random numbers, cast it to the function, and run your machine code you’ve just put in - be my guest, that’s what we call meta-programming! As ridiculous as it is, I have done that in the past.

But, I don’t want to write in C anymore for many reasons. So, what language did I pick?

Well, since you have seen the tags for this post, you already know that it is the Zig programming language. However, I have considered a number of other languages. Namely Nim, Hare, Zig, Ada, and even OCaml.

I have already tried Rust for this kind of project before, so I knew that I didn’t want to do it again with the same language. I realized that I don’t like Rust in general. The idea is nice on paper, but I believe it’s overly fixated. And I have my thoughts on the Rust community, so this language isn’t really for me.

But I wanted a language that can be considered a system language, much like C is. And while Rust is indeed used for Linux kernel, I, myself, can’t consider it a system programming language, much like I can’t consider C++ to be one, even though you can write a system in it, for example Haiku. You can write systems in all kinds of languages, but I think that a system programming language has to have some pretty specific features and these are usually related to the ability to understand code, not to write one. Rust and many others failed this for me.

So, I quickly decided that I didn’t want to use Nim because of its Python-like syntax which I don’t like at all. It may be a decent language, but the other point I don’t like about it in comparison to the other is that it is transpiled to C and then compiled, correct me if I’m wrong and things have changed since when it was like that. I don’t think that this approach is bad, but I think that C is a poor target to compile to, and I can’t be sure that everything I write clearly translates to C without weird shenanigans that can have weird or unexpected behavior. Although more languages do that, namely ECL and Gambit Scheme, it’s not impossible to do this well. Sadly I think Scheme is a better language than Nim.

OCaml is often brought up when discussing a language for writing compilers. There are valid reasons to use it, but I’m quite unfamiliar with the ML family of languages, so I decided to not use it. Also, OCaml is often used for its ability to work with LLVM, which isn’t what we’re doing in the book.

Hare is a relatively new language by Drew DeVault. From what I’ve seen about this language it seems interesting, and it has a nice article listing the advantages over C. However, the documentation was almost non-existent at the moment I started this project, and I didn’t want to dive into a language that was in so early stages of its development. I have vague plans on maybe doing a Forth implementation in Hare in the future, so I’ll get to try it eventually. I would say that Hare was a close second to me.

Ada seems like an interesting language, and I have read the hellscape of a code from one of the projects done by a member of a community in which I hang out. While it seems like a capable language, I didn’t like it that much to try it myself.

There’s also Odin… I have never heard of this language and it’s being used anywhere, but according to their web page, it’s basically everywhere. That seems like a stretch for a project that started in 2016. Zig started in 2015 and had no stable release yet, and Odin claims to be professionally used in so many game studios and other companies, and yet I haven’t heard of it at all! I consider myself a language enthusiast, yet somehow this language completely dodged my field of view. Maybe they’re that good, I don’t know, but I decided that I’ll pass.

Jai, a language by Jonathan Blow, also seemed interesting to try out, but I feel that its aim at being a replacement for C++ in the game programming field is not exactly what I’m looking for. At the time of writing, it’s still in a closed beta test, so it’s hard to get hands on it, and judging from what I have seen in the videos, it’s not like I would use this language for this particular project either.

Before you ask, I haven’t considered Go for this project, because I simply don’t like Go. There’s the interpreterbook book which is oddly similarly named to the book discussed in this post, and it is written in Go, so you don’t even have to go through translating C to Go if you want to learn how to design a language if Go is your thing. Well, I have my reasons to dislike Go, but I don’t want to go into them here. Maybe next time.

And yes, there are many more languages that can be used for this kind of task - obviously, I can’t compare all of them. So I had to stop somewhere.

The Zig programming language caught my eye a few years ago after it was pointed out to me as a response to my rant about C. It is more mature than Hare, and as far as I know, the 1.0.0 release is not as far away. And for what it’s worth, it is closest to C in many ways that Ada or Rust are not. So I decided that it would be a proper pick for the second half of the book.

Zig

So I chose Zig. At the moment of writing this post, the latest release is 0.11.0, which is important to note as Zig is still constantly changing. While this is a good thing, and I understand that Zig had no stable release yet, I am spoiled. I am spoiled by languages with the stable core, like Clojure, where you can find a 13-year-old code and run it with the modern version of Clojure. That’s not how it works with Zig. A lot of code I’ve tried during this experiment did not work because it was written for an older version of Zig.

As you may remember from part 1 - this is a second try on writing the second half of the book in Zig. The first try wasn’t successful, and I’m going to discuss this separately later in this post. During the first iteration, I had to learn many things about the language and its build system. After I hit the point where my implementation did not work (for reasons described later), I decided to delete everything and start over (for reasons described later).

So I took a considerable amount of time to rest from this project and in the meantime, Zig 0.11.0 was released. This is important because the previous run was done using Zig 0.10.0, and I deleted everything, except for the build.zig that I had spent a lot of time creating to fulfill my needs. And now it doesn’t work.

That made me realize that I didn’t actually spend any time in the previous post talking about the build system. So let’s do that right now!

Clojure build system

So, in the Clojure version of the Lox language, I opted to use the deps.edn which is pretty much the standard way to create projects in Clojure. There are other systems that can help manage projects, notably Leiningen, but I like to do things manually, for better understanding, so when I can I use stuff like deps. And unlike Leiningen, deps doesn’t come with the automatic building of your project - you have to make it by yourself.

And that’s actually great! I like the idea that builds are programs because you can write custom builds that are specific to your project’s needs. Of course, most projects don’t need anything fancy, so there is a library that provides the basic stuff you’ll probably need.

Here’s what the build.clj looks like for CljLox:

(ns build
  (:require [clojure.tools.build.api :as b]))

(def lib 'lox)
(def version (format "0.0.%s" (b/git-count-revs nil)))
(def class-dir "target/clojure/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def uber-file (format "target/clojure/%s-%s-standalone.jar" (name lib) version))

(defn clean [_]
  (b/delete {:path "target/clojure"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["src"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :src-dirs ["src"]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'lox.core}))

That’s it!

I like the idea of using the number of commits since the repository creation as the last component of the version, and the build library provides a function for that: b/git-count-revs. We do have to specify some paths, create a basis, whatever it is, and just define the compilation functions. We then execute it like clojure -T:build uber, and it creates the lox-0.0.123-standalone.jar file that we can later run with the java -jar command.

Simple and effective. If my project required additional steps I could just define more functions to do these steps. Let’s compare that to Zig, shall we?

Zig build

Here’s the entire build.zig for the previous version of ZigLox:

const std = @import("std");
const Version = std.builtin.Version;
const Builder = std.build.Builder;
const print = std.debug.print;
const LibExeObjStep = std.build.LibExeObjStep;
const Mode = std.builtin.Mode;
const CrossTarget = std.zig.CrossTarget;
const Step = std.build.Step;
const trimRight = std.mem.trimRight;
const parseInt = std.fmt.parseInt;
const exit = std.process.exit;
const fs = std.fs;

const MAJOR = 0;
const MINOR = 1;

fn gitCountRevs(b: *Builder) u32 {
    var code: u8 = undefined;
    b: {
        const commits_num = trimRight(
            u8,
            b.execAllowFail(
                &[_][]const u8{ "git", "-C", b.build_root, "rev-list", "HEAD", "--count" },
                &code,
                .Ignore,
            ) catch break :b,
            "\n",
        );
        return parseInt(u32, commits_num, 10) catch break :b;
    }
    print("unable to count number of revs\n", .{});
    exit(1);
}

fn getVersion(b: *Builder) Version {
    return Version{
        .major = MAJOR,
        .minor = MINOR,
        .patch = gitCountRevs(b),
    };
}

fn isZigFile(name: []const u8) bool {
    return std.mem.eql(u8, name[name.len - 4 ..], ".zig");
}

/// Automatically resolves packages under the source directory.  Adds
/// packages to the `exe`.
fn resolveAndAddPackages(
    src_dir: []const u8,
    allocator: std.mem.Allocator,
    exe: *LibExeObjStep,
) void {
    const dir = std.fs.cwd().openIterableDir(src_dir, .{}) catch exit(2);
    var iterator = dir.iterate();

    while (iterator.next() catch null) |path| {
        if (isZigFile(path.name)) {
            const package_name = path.name[0 .. path.name.len - 4];
            const src_path = fs.path.join(
                allocator,
                &[_][]const u8{ src_dir, path.name },
            ) catch exit(3);
            exe.addPackagePath(package_name, src_path);
        }
    }
}

/// Automatically finds tests under the test directory, adds the test
/// to the build `step`, and adds all necessary packages for test to
/// run.
fn resolveAndAddTests(
    b: *Builder,
    test_dir: []const u8,
    allocator: std.mem.Allocator,
    mode: Mode,
    target: CrossTarget,
    step: *Step,
) void {
    const dir = std.fs.cwd().openIterableDir(test_dir, .{}) catch exit(2);
    var iterator = dir.iterate();

    while (iterator.next() catch null) |path| {
        if (isZigFile(path.name)) {
            const test_path = fs.path.join(
                allocator,
                &[_][]const u8{ test_dir, path.name },
            ) catch exit(3);
            const tests = b.addTest(test_path);
            resolveAndAddPackages("src/zig/lox", allocator, tests);
            tests.setTarget(target);
            tests.setBuildMode(mode);
            step.dependOn(&tests.step);
        }
    }
}

pub fn build(b: *Builder) void {
    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();
    const allocator = arena.allocator();

    const version = getVersion(b);
    const version_string = b.fmt("{[major]d}.{[minor]d}.{[patch]d}", version);

    const target = b.standardTargetOptions(.{});
    const mode = b.standardReleaseOptions();

    const exe = b.addExecutable(
        b.fmt("lox-{s}", .{version_string}),
        "src/zig/lox/main.zig",
    );
    exe.setTarget(target);
    exe.setBuildMode(mode);
    exe.install();

    const run_cmd = exe.run();
    run_cmd.step.dependOn(b.getInstallStep());
    if (b.args) |args| {
        run_cmd.addArgs(args);
    }

    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);

    const test_step = b.step("test", "Run unit tests");
    const test_dir = fs.path.join(
        allocator,
        &[_][]const u8{ "test", "zig", "lox" },
    ) catch exit(1);

    resolveAndAddTests(b, test_dir, allocator, mode, target, test_step);
}

It’s not that big, but the amount of boilerplate you have to do, even though the language has a build library is far greater than in Clojure. I do understand that these things pursue different goals, but does it really need to be that complicated?

Actually, let me explain why it is so complicated. You see, I have a quite specific project layout in mind:

Lox
├── build.clj
├── build.zig
├── deps.edn
├── src
│   ├── clojure
│   │   └── lox
│   │       └── sources.clj
│   └── zig
│       └── lox
│           └── sources.zig
├── target
│   ├── clojure
│   └── zig
└── test
    ├── clojure
    │   └── lox
    │       └── test_sources.clj
    ├── data
    │   └── various_test_files
    └── zig
        └── lox
            └── test_sources.zig

So the sources are located in the src directory, and the tests are located in the test directory. And each implementation has a language directory in there, so Clojure sources are in the src/clojure and the Zig sources are in src/zig. The same goes for tests.

Unfortunately, I couldn’t find a way to have tests outside of the source directory, so I had to dance around, creating my own solution for this self-introduced problem. Is it that uncommon to put tests into a separate directory, outside of the source directory? Let’s look at other languages.

In Java, this is the norm, same in Clojure. I like this actually, tests are separate entities, they don’t have anything to do with the source code. I mean, the source is for the actual application or the library you ship to the consumer. The tests are for you, the developer, not for the consumer.

In Rust, the tests usually are in the same file as the source code. When I was writing in Rust I kinda liked it, but only, seriously, only because before Rust I was writing C professionally, and tests in C are, well, they’re not standardized, to say the least. So having Rust giving you tools to write and run tests was a huge step up from C, but that was the only step up.

In Go it is possible to put tests in a different directory, and I’ve seen projects doing it, so I guess that’s a fine part of Go.

What else can we look at? Nim? Seems that putting tests into a separate directory is possible there. Hare? From what I can see, the projects have either +test.ha or just test.ha files in the same place where the sources are - not sure if this is the only way it can work, the hare docs are sparse on the details here.

Well, I guess there’s no single answer to this question, but I don’t see why a language would forbid putting tests whenever the project author sees fit. And in Zig, well, while it was hard to make it, and make it fully automatic, remember - the build is a program. You can do whatever you want, as long as the tooling gives you the ability, and in the case of Zig, the build system can do that. Well, at least it could in 0.10.0, but now I have to rewrite it for 0.11.0.

I had a few hiccups and managed to do that, so I’m not going to complain that much here, but I still think it’s too complicated. I mean, it’s not as complicated as it was in 0.10.0, so that’s good, but it is still a lot of manual work. Here’s a diff with the previous version so you can feel the amount of subtle changes I had to make to make this thing compile again:

@@ -1,11 +1,10 @@
 const std = @import("std");
-const Version = std.builtin.Version;
-const Builder = std.build.Builder;
+const Builder = std.Build;
+const Step = Builder.Step;
 const print = std.debug.print;
-const LibExeObjStep = std.build.LibExeObjStep;
+const CompStep = Step.Compile;
 const Mode = std.builtin.Mode;
 const CrossTarget = std.zig.CrossTarget;
-const Step = std.build.Step;
 const trimRight = std.mem.trimRight;
 const parseInt = std.fmt.parseInt;
 const exit = std.process.exit;
@@ -14,6 +13,12 @@ const fs = std.fs;
 const MAJOR = 0;
 const MINOR = 1;

+const Version = struct {
+    major: u32,
+    minor: u32,
+    commits: u32,
+};
+
 fn gitCountRevs(b: *Builder) u32 {
     var code: u8 = undefined;
     b: {
@@ -46,10 +51,10 @@ fn isZigFile(name: []const u8) bool {

 /// Automatically resolves packages under the source directory.  Adds
 /// packages to the `exe`.
-fn resolveAndAddPackages(
+fn resolveAndAddModules(
+    b: *Builder,
     src_dir: []const u8,
-    allocator: std.mem.Allocator,
-    exe: *LibExeObjStep,
+    exe: *CompStep,
 ) void {
     const dir = std.fs.cwd().openIterableDir(src_dir, .{}) catch exit(2);
     var iterator = dir.iterate();
@@ -57,11 +62,8 @@ fn resolveAndAddPackages(
     while (iterator.next() catch null) |path| {
         if (isZigFile(path.name)) {
             const package_name = path.name[0 .. path.name.len - 4];
-            const src_path = fs.path.join(
-                allocator,
-                &[_][]const u8{ src_dir, path.name },
-            ) catch exit(3);
-            exe.addPackagePath(package_name, src_path);
+            const src_path = b.pathJoin(&.{ src_dir, path.name });
+            exe.addModule(package_name, b.createModule(.{ .source_file = .{ .path = src_path } }));
         }
     }
 }
@@ -71,9 +73,9 @@ fn resolveAndAddPackages(
 /// run.
 fn resolveAndAddTests(
     b: *Builder,
+    src_dir: []const u8,
     test_dir: []const u8,
-    allocator: std.mem.Allocator,
-    mode: Mode,
+    optimize: Mode,
     target: CrossTarget,
     step: *Step,
 ) void {
@@ -82,52 +84,41 @@ fn resolveAndAddTests(

     while (iterator.next() catch null) |path| {
         if (isZigFile(path.name)) {
-            const test_path = fs.path.join(
-                allocator,
-                &[_][]const u8{ test_dir, path.name },
-            ) catch exit(3);
-            const tests = b.addTest(test_path);
-            resolveAndAddPackages("src/zig/lox", allocator, tests);
-            tests.setTarget(target);
-            tests.setBuildMode(mode);
-            step.dependOn(&tests.step);
+            const tests = b.addTest(.{
+                .name = path.name,
+                .root_source_file = .{ .path = b.pathJoin(&.{ test_dir, path.name }) },
+                .target = target,
+                .optimize = optimize,
+            });
+            resolveAndAddModules(b, src_dir, tests);
+            step.dependOn(&b.addRunArtifact(tests).step);
         }
     }
 }

 pub fn build(b: *Builder) void {
-    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
-    defer arena.deinit();
-    const allocator = arena.allocator();
-
-    const version = getVersion(b);
-    const version_string = b.fmt("{[major]d}.{[minor]d}.{[patch]d}", version);
+    const version = b.fmt("{[major]d}.{[minor]d}.{[commits]d}", getVersion(b));

     const target = b.standardTargetOptions(.{});
-    const mode = b.standardReleaseOptions();
-
-    const exe = b.addExecutable(
-        b.fmt("lox-{s}", .{version_string}),
-        "src/zig/lox/main.zig",
-    );
-    exe.setTarget(target);
-    exe.setBuildMode(mode);
-    exe.install();
-
-    const run_cmd = exe.run();
-    run_cmd.step.dependOn(b.getInstallStep());
-    if (b.args) |args| {
-        run_cmd.addArgs(args);
-    }
+    const optimize = b.standardOptimizeOption(.{});
+    const src_dir = b.pathJoin(&.{ "src", "zig", "lox" });
+    const main = b.pathJoin(&.{ src_dir, "main.zig" });
+
+    const exe = b.addExecutable(.{
+        .name = b.fmt("lox-{s}", .{version}),
+        .root_source_file = .{ .path = main },
+        .target = target,
+        .optimize = optimize,
+    });
+
+    b.installArtifact(exe);
+    const run_cmd = b.addRunArtifact(exe);

     const run_step = b.step("run", "Run the app");
     run_step.dependOn(&run_cmd.step);

     const test_step = b.step("test", "Run unit tests");
-    const test_dir = fs.path.join(
-        allocator,
-        &[_][]const u8{ "test", "zig", "lox" },
-    ) catch exit(1);
+    const test_dir = b.pathJoin(&.{ "test", "zig", "lox" });

-    resolveAndAddTests(b, test_dir, allocator, mode, target, test_step);
+    resolveAndAddTests(b, src_dir, test_dir, optimize, target, test_step);
 }

Yet, this isn’t a final version of the file¹, just the first working one. Gotta say, I had to spend a few hours figuring out why tests don’t actually run, only to notice a line in the changelog:

addTest No Longer Runs It

Before, addTest created and ran a test. Now you need to use b.addRunArtifact to run your test executable.

That’s on me, of course, cuz I haven’t taken my time to read the whole changelog. But it’s like 24k words without the code, so can you blame me? I just had a stroke of passion to restart this project, and I didn’t really want to kill it by reading a huge changelog of the language I already don’t remember almost completely. So for me, it is not even a change log, it’s just a log.

But it now works and that’s the only thing that matters. Hopefully, I’ll finish this project before 0.12.0 comes out, and I’ll reimplement everything again. Oh well, I can just not update, but the language is being improved so I’d rather not miss on these improvements.

With that covered, I think we can actually start with the book!

Chunks of Bytecode

There’s going to be an ongoing theme with this book: I’m not going to re-implement low-level machinery for the sake of implementing it. It’s something I need to say early on, as the book is full of this, and while I understand the author’s intent, I’m not the type of reader the book really aims for.

What I’m talking about is implementing such things as dynamic arrays. Zig already comes with the dynamic array, called the ArrayList. If you’re curious enough, you can check how it is implemented, and it is almost exactly the same as what is described in the book. I implemented it during the first run, looked at it, looked at the standard library implementation, said to myself “I got it”, and decided not to reinvent the wheel. These tricks, while neat, are only meaningful to C - in Zig, we can and should use the standard library.

Even if we don’t re-implement all this stuff, there are a lot of differences in implementing things with Zig. Let’s have a look at the code for the chunk:

const std = @import("std");
const ArrayList = std.ArrayList;

const common = @import("common.zig");
const Code = common.Code;
const OPCode = common.OPCode;
const Allocator = common.Allocator;
const exit = common.exit;
const print = common.print;
const str = common.str;

pub const Chunk = struct {
    const Self = @This();
    code: ArrayList(Code),

    pub fn init(allocator: Allocator) Chunk {
        return Chunk{
            .code = ArrayList(Code).init(allocator),
        };
    }

    pub fn deinit(self: *Self) void {
        self.code.deinit();
    }

    pub fn write(self: *Self, byte: OPCode) void {
        self.code.append(@intFromEnum(byte)) catch
            exit(1, "can't allocate memory for the instruction");
    }

    fn simpleInstruction(name: str, offset: u32) u32 {
        std.debug.print("{s: <16}\n", .{name});
        return offset + 1;
    }

    fn disassembleInstruction(self: *Self, offset: u32) u32 {
        print("{d:0>4} ", .{offset});

        var opcode = self.code.items[offset];
        return switch (@as(OPCode, @enumFromInt(opcode))) {
            .Ret => |op| simpleInstruction(@tagName(op), offset),
            else => |op| {
                print("unknown opcode {d}\n", .{op});
                return offset + 1;
            },
        };
    }

    pub fn disassemble(self: *Self, name: str) void {
        print("==== {s} ====\n", .{name});
        var offset: u32 = 0;
        while (offset < self.code.items.len) {
            offset = self.disassembleInstruction(offset);
        }
    }
};

Code Snippet 1: src/zig/lox/chunk.zig

Do you see that barrage of const definitions at the top? There will be lots of these things during this post. As you can actually notice, a dozen came from the common.zig module:

const std = @import("std");
pub const Allocator = std.mem.Allocator;

pub const str = []const u8;
pub const Code = u8;
pub const print = std.debug.print;

pub const OPCode = enum(Code) {
    Ret,
    _,
};

pub fn exit(code: u8, msg: str) noreturn {
    print("{s}\n", .{msg});
    std.process.exit(code);
}

Code Snippet 2: src/zig/lox/common.zig

Zig claims that the top-level constants don’t have to be ordered, which while convenient, I don’t think is practically good. I do prefer a top-down approach, where the most specific things appear at the top, and more general stuff is at the bottom. So if this isn’t your cup of tea, well, sorry, I guess?

Anyhow, here’s the main.zig:

const std = @import("std");
const Chunk = @import("chunk.zig").Chunk;
const common = @import("common.zig");
const OPCode = common.OPCode;

pub fn main() !void {
    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();
    const allocator = arena.allocator();

    var chunk = Chunk.init(allocator);
    defer chunk.deinit();

    chunk.write(OPCode.Ret);
    chunk.disassemble("test chunk");
}

Code Snippet 3: src/zig/lox/main.zig

If we run it with zig build run, we get a lovely:

$ zig build run
==== test chunk ====
0000 Ret

So far so good. Note, this is code up to the part where we are supposed to add a value array for constants, so it’s not like it’s the whole code for the chapter. For now, I won’t go into code explanations, it’s not that hard right now, you can get the idea just by reading it, and by referring to the book. But there is a reason why I decided to show it right now.

Adding constants

Let’s have a look at how the book suggests to add the constants.

First, we introduce the addConstant function, which adds a Value type to the dynamic array of constants. Next, we use writeChunk, or in my case chunk.write to write the constant index in the array to the code array of the chunk. Do you see the issue here? Let’s look at the code then.

The first thing we do is add a new function:

pub fn addConstant(self: *Self, value: Value) Code {
    self.constants.append(value) catch
        exit(1, "can't allocate memory for the constant");
    return @truncate(self.constants.items.len - 1);
}

You may wonder, why am I using @truncate on the return value, and the return type is Code? Especially since the code in the book tells us that it should return an int:

int addConstant(Chunk* chunk, Value value) {
  writeValueArray(&chunk->constants, value);
  return chunk->constants.count - 1;
}

Now, remember, our write function (writeChunk in the book) accepts the chunk and an opcode to write. Initially, I’ve made it so that the signature expects the OPCode enum itself:

    pub fn write(self: *Self, byte: OPCode) void {
        self.code.append(@intFromEnum(byte)) catch
            exit(1, "can't allocate memory for the instruction");
    }

However, that would mean, that if we were to pass the result of addConstant to it, we would need to convert it to the enum tag, and then immediately convert it back to the Code type, that the ArrayList uses to store code in the chunk. So, I’ve changed it:

    pub fn write(self: *Self, byte: Code) void {
        self.code.append(byte) catch
            exit(1, "can't allocate memory for the instruction");
    }

Now, we can add constants in the main:

pub fn main() !void {
    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();
    const allocator = arena.allocator();

    var chunk = Chunk.init(allocator);
    defer chunk.deinit();

    const constant = chunk.addConstant(1.2);
    chunk.write(@intFromEnum(OPCode.Constant));
    chunk.write(constant);
    chunk.disassemble("test chunk");
}

Still don’t see the issue? Well, @truncate kinda spoiled it, so I’ll get straight to the point - we can only have 256 constants in the Chunk. The rest will overflow our Code type, which is, in fact, just a u8 in disguise.

The book overlooks this for now, as in C truncation happens automatically, but comes back to it in the “Challenges” section. And, as you may remember I’ve promised that I have a lot to say about the “Challenges”.

Here’s the “Challenge”:

Because OP_CONSTANT uses only a single byte for its operand, a chunk may only contain up to 256 different constants. That’s small enough that people writing real-world code will hit that limit. We could use two or more bytes to store the operand, but that makes every constant instruction take up more space. Most chunks won’t need that many unique constants, so that wastes space and sacrifices some locality in the common case to support the rare case.

To balance those two competing aims, many instruction sets feature multiple instructions that perform the same operation but with operands of different sizes. Leave our existing one-byte OP_CONSTANT instruction alone, and define a second OP_CONSTANT_LONG instruction. It stores the operand as a 24-bit number, which should be plenty.

Implement this function:
void writeConstant(Chunk* chunk, Value value, int line) {
  // Implement me...
}
It adds value to chunk’s constant array and then writes an appropriate instruction to load the constant. Also add support to the disassembler for OP_CONSTANT_LONG instructions.

Defining two instructions seems to be the best of both worlds. What sacrifices, if any, does it force on us?

So here’s the problem - I had implemented this in the first run and regretted it many, many times. So for this run, I’m not going to, sorry Robert.

I mean, the challenge itself isn’t hard, however, the book assumes that you don’t do them because it is a challenge, not homework that you have to do. So the latter chapters continue as if no challenges were even there - the code in the book still uses the naive truncated approach.

I do understand that this book is about teaching how to implement a language and not a step-by-step guide. But later, these challenges made my code so different from what the book gives you that I had several hours-long sessions of looking through the book, figuring out where the particular piece of code came from, and what the intention there was. All because I had a bug somewhere and the reason was usually that differences, introduced by the challenges, piled up and combined introducing a subtle bug that made itself visible only after several chapters.

So, no challenges this time. All I care about this time is to finish the book and have a working language written in Zig.

Line information

Sike! Let’s do a challenge!

The book suggests that we can store lines in a dynamic array, for each index of the chunk:

    chunk->lines[chunk->count] = line;

However, this is quite inefficient in terms of memory. And the book mentions that in the “Challenges” section. It suggests using a so-called Run-length encoding.

After scratching my head for a moment, I came up with this solution:

    fn writeLine(self: *Self, line: u32) void {
        const len = self.lines.items.len;
        if (len > 1 and self.lines.items[len - 2] == line) {
            self.lines.items[len - 1] = self.lines.items[len - 1] + 1;
        } else {
            self.lines.append(line) catch
                exit(1, "can't allocate memory for line information");
            self.lines.append(1) catch
                exit(1, "can't allocate memory for line information");
        }
    }

Code Snippet 4: a method in the Chunk struct

It’s a rather simple algorithm - we check if the last line is the same as the new one, and if it is, we increase the count, that is stored right next to it. Otherwise, we append a new line and set its count to 1.

But now we need to extract line information by uncompressing this data:

    fn getLine(self: *const Self, offset: u32) u32 {
        var cnt: u32 = 0;
        var off: u32 = offset;
        while (off > 0) loop: {
            var n: u32 = self.lines.items[cnt + 1];
            while (n > 0) {
                if (off == 0)
                    break :loop;
                off = off - 1;
                n = n - 1;
            }
            cnt = cnt + 2;
        }
        return self.lines.items[cnt];
    }

Code Snippet 5: a method in the Chunk struct

Simply put, we’re decreasing the counter while the offset is greater than zero while increasing the counter each time we’ve exhausted the line counter n. Sure, it’s not a very fast algorithm, and we can probably do it without using loops if I think hard enough, but I’m OK with this solution. It isn’t used until the chunk gets disassembled, and this only happens when we debug.

This challenge I like. It doesn’t affect the overall code of the VM and is more of a lesson in optimization. We saved some memory, at the expense of a bit slower disassembler, which is fine by me.

However, seriously, this is probably the last challenge I’ll tackle for a long time, so don’t get excited.

Tests

As a final thing in this chapter, the book suggests testing your code. And here’s the thing - it’s a great suggestion, only this book will make you hate your decision to test the code. I’m a bit exaggerating here, but keep in mind, during this chapter we’ve already changed some parts of the Chunk code several times.

First, we introduced the write method, and it accepted the OPCode enum. Next, we introduced the addConstant method, and it required us to change the write to accept the Code instead. Finally, we’ve added support for line information, changing the write method once again.

You may think that that’s fine, as we did it during one chapter, and the tests should be written after we complete it. Well, TDD people may disagree with you. I’m not a TDD person, I usually test things as I go in the REPL, and then, once I’m satisfied I write proper tests to prevent later breakage. But Zig doesn’t have a REPL, so testing it this way won’t work.

So, while Robert claims that: “I wrote a test suite for Lox before I wrote a single word of this book.”, I’d be cautious when writing tests for this book’s code. Probably his claim is true, especially if he knew what the result would be and wrote a test for it. Or, maybe the book was simply written after the implementation of the language became a thing, which then doesn’t matter if the tests were written upfront or not.

I tend to write tests either during the chapter or at the end of it. And at the first run, I did exactly that. But, as we’ll see later, you’ll have to discard or rework a lot of tests due to changes in the code. And often the changes are dramatic, so I’m not sure if it makes sense to write too many tests early on.

A Virtual Machine

This chapter is mostly straightforward, except for a few shenanigans we have to make because Zig is different. The first thing I would like to talk about is pointers in various programming languages.

In C we basically have one kind of pointer:

SomeType * variable;

This really can be anything. It can be a pointer to a single location, or it can be a pointer to an array, or even to a place somewhere in a set array. Well, there’s also a function pointer in C, but we’re not going to look at it now.

Zig, on the other hand, has quite a few types of pointers:

*T - single-item pointer to exactly one item.
- Supports deref syntax: ptr.*
[*]T - many-item pointer to an unknown number of items.
- Supports index syntax: ptr[i]
- Supports slice syntax: ptr[start..end]
- Supports pointer arithmetic: ptr + x, ptr - x
- T must have a known size
*[N]T - pointer to N items, same as a single-item pointer to an array.
- Supports index syntax: array_ptr[i]
- Supports slice syntax: array_ptr[start..end]
- Supports len property: array_ptr.len
[]T - is a slice (a fat pointer, which contains a pointer of type [*]T and a length).
- Supports index syntax: slice[i]
- Supports slice syntax: slice[start..end]
- Supports len property: slice.len

As you can see, we can’t just write variable: *SomeType in Zig and use it like we can in C. I believe this is a good thing, and in practice, it does eliminate bugs related to accessing ordinary pointers as arrays. On the other hand, it is quite tedious to write code with such a system.

To get what I mean, let’s have a look at how the instruction pointer concept is implemented by the book:

typedef struct {
  Chunk* chunk;
  uint8_t* ip;
} VM;

InterpretResult interpret(Chunk* chunk) {
  vm.chunk = chunk;
  vm.ip = vm.chunk->code;
  return run();
}

You’re looking at the VM struct that holds the currently executed chunk and an instruction pointer ip. In the interpret function we set the VM’s current chunk, and its instruction pointer to point to the code from the chunk. As you may remember the Chunk struct contains the code field which is a dynamic array: uint8_t * code. So by saying vm.ip = vm.chunk->code we essentially say that ip now holds the same address that was in the code field. Hopefully, it’s the start of the array!

With that, we can now inspect the run function. For now, all we really care about is how it uses the instruction pointer:

static InterpretResult run() {
#define READ_BYTE() (*vm.ip++)
        // ...
    switch (instruction = READ_BYTE()) {/* ... */}
        // ...
#undef READ_BYTE
}

Aha, C tricks! The READ_BYTE is a substitution macro that simply increments the pointer with ++, dereferencing the resulting one with *. It’s a clever trick - too clever, even, because of the operator precedence rules in C. The ++ has to happen before * because otherwise, we would increment the dereferenced value. But the effect of ++ has to happen after we dereference the pointer, otherwise, we would dereference the next address. This code can be rewritten into two separate expressions, like uint8_t val = *ip; ip = ip + 1; but it’s hard to use that in a macro, that itself is used in the expression context. So the *ptr++ form is a quite common trick.

But here’s the thing - the ip is not an array, it just happens to hold an address to an array. Uh oh.

Well, that’s a common thing in C, so let’s look at Zig. Now, I’m not really experienced in Zig’s pointers but after poking around I think we have several options:

pub const VM = struct {
    chunk: *Chunk, // store a pointer to the chunk
    ip: []Code, // and a fat pointer to the array

    // methods ...
};

Code Snippet 6: Option №1

Here I have the [*]T type of pointer because we’re going to point to an array because the Chunk stores items in a dynamic array. We can implement the interpret and run functions as follows:

    pub fn interpret(self: *VM, chunk: *Chunk) Result {
        self.chunk = chunk;
        self.ip = chunk.code.items;
        return self.run();
    }

    pub fn run(self: *VM) Result {
        // ...
        const instr = self.ip[0]; self.ip += 1;
        switch (instr) {
            // ...
        }
    }

We can put the pointer reading into an inline function, but that’s beside the point. Look at how we have to dereference this kind of pointer: ip[0]. Unlike in C, we can’t use a single-item pointer dereference syntax here, because it’s not a single-item pointer!

Let’s try a single-item pointer then:

pub const VM = struct {
    chunk: *Chunk, // store a pointer to the chunk
    ip: *Code, // and a pointer to a place in the array

    // methods ...
};

Code Snippet 7: Option №2

Now we have a single-item pointer for the instruction pointer (makes sense), but let’s look at how we have to use it:

    pub fn interpret(self: *VM, chunk: *Chunk) Result {
        self.chunk = chunk;
        self.ip = &chunk.code.items[0];
        return self.run();
    }

    pub fn run(self: *VM) Result {
        // ...
        const instr = self.ip.*;
        self.ip = @ptrCast(@as([*]Code, @ptrCast(self.ip)) + 1);
        switch (instr) {
            // ...
        }
    }

Ugh, that’s looking nasty. We have to do two pointer casts to do a simple thing like that. Looking at it now the ip[0] doesn’t look that out of place.

But then the book does this kind of thing:

disassembleInstruction(vm.chunk, (int)(vm.ip - vm.chunk->code));

If you recall the disassembleInstruction signature, the second argument is an integer offset. So we do all this pointer arithmetic here and there just to convert it back to an index in the array. And that’s our option three:

pub const VM = struct {
    chunk: *Chunk, // store a pointer to the chunk
    ip: []Code,
    ic: u32, // and an index!

    // methods ...
};

Code Snippet 8: Option №3

This does require a bit different approach in the rest of the code:

pub fn interpret(self: *VM, chunk: *Chunk) Result {
    self.chunk = chunk;
    self.ip = chunk.code.items;
    self.ic = 0;
    return self.run();
}

pub fn run(self: *VM) Result {
    // ...
    const instr = self.ip[self.ic]; self.ic += 1;
    switch (instr) {
        // ...
    }
}

We can simply use the vm.disassembleInstruction(self.ip) to disassemble the instruction avoiding all of the pointer arithmetic altogether. Now, I’m sure the raw pointer for this kind of task is more appealing, and probably it is slightly faster. But to me, this looks like a premature optimization, at least for now. So that’s the way I decided to do it for my implementation.

Another thing I want to touch here is the return type of the interpret function. In the book, the return type is called InterpretResult and it is a enumeration with simple tags: INTERPRET_OK, INTERPRET_COMPILE_ERROR, INTERPRET_RUNTIME_ERROR. When we reach the OP_RETURN the result of the run function is the INTERPRET_OK tag. At the end of the chapter, after we’ve implemented the stack and basic arithmetic operations we read the last value on the stack and print it before we exit:

      // ...
      case OP_RETURN: {
        printValue(pop());
        printf("\n");
        return INTERPRET_OK;
      }
      // ...

While this is in the spirit of C, we can do better. And in fact, we have to do better, as we need to write tests, as the book suggested. It’s hard to write tests if you don’t have access to the result of interpretation, so my Result type is defined as union(enum):

pub const Result = union(enum) {
    OK: Value,
    CompileError,
    RuntimeError,
};

This way we can return the actual value instead of printing it, and indicate that everything is OK at the same time, kinda like Rust’s Option type:

    // switch(opcode) {
    // ...
           .Ret => return .{ .OK = self.pop() },
    // }

This way we can check the results in our VM tests.

I’ve omitted a bunch of stuff from this chapter, but simply because there aren’t many interesting things there when it comes to Zig vs C. But one thing I would like to return to is metaprogramming, because this book does it frequently enough to be a hassle. We’ve already seen a bunch of #define directives in the C examples above, and the book goes pretty far with them, defining a polymorphic BINARY_OP macro:

#define BINARY_OP(op) \
    do { \
      double b = pop(); \
      double a = pop(); \
      push(a op b); \
    } while (false)

The do {...} while(false) is a common trick in C macros. For those unfamiliar with how do while works, as it is rarely seen in other languages, it executes the body first, then checks the condition and repeats if it is true. Classical while checks the condition first. Most languages omit do while because it is essentially the same as writing the body of the while loop two times, e.g.:

foo();           // do {
while (test()) { //    foo();
    foo();       // } while (test());
}

Of course, there are other ways to rewrite this code, and do while looks clearer, but it’s quite rare. In our case though, it is used as a way to introduce a valid language construct that allows for a semicolon to be inserted after it. Because of that, it uses while(false) i.e. it never loops.

So when you write BINARY_OP(+); (mind the semicolon) it expands to the correct code with a semicolon after while (false); which is required in the case of this language construct, unlike with other loops. Yep, that’s the sole reason it is used here, and compilers are smart enough to detect that the loop will never happen, so they omit the loop entirely. A truly sad state of metaprogramming, and expression statements in C.

However, Zig doesn’t have macros. Instead, it has the comptime keyword, which marks the expression or function argument to be used only at compilation time. Unfortunately, there doesn’t seem to be a way to write an inline comptime function that will accept an operator and return some kind of AST, or other code to be present in the expression. We can work around it, though:

    inline fn binaryOp(op: OPCode, a: Value, b: Value) Value {
        return switch (op) {
            .Add => a + b,
            .Sub => a - b,
            .Mul => a * b,
            .Div => a / b,
            else => unreachable,
        };
    }
   // ---8<---
   //       switch(opcode) {
                .Add, .Sub, .Mul, .Div => |instr| {
                    const b = self.pop();
                    var sp = self.stack_top - 1;
                    sp[0] = binaryOp(instr, sp[0], b);
                },
   // }

Not as pretty, but works, I guess. We do have a cost of doing branching twice, e.g. first in the run function, then in the binaryOp function, even though by the time we’re in binaryOp we already know what operation we’re dealing with. C’s macros are giving us the ability to construct code without explicit copy-pasting and without explicit re-branching. We can avoid re-branching by writing it like this, of course:

    inline fn add(self: *Self) void {
        const b = self.pop();
        var sp = self.stack_top - 1;
        sp[0] = sp[0] + b;
    }

    inline fn sub(self: *Self) void {
        const b = self.pop();
        var sp = self.stack_top - 1;
        sp[0] = sp[0] - b;
    }

    inline fn div(self: *Self) void {
        const b = self.pop();
        var sp = self.stack_top - 1;
        sp[0] = sp[0] / b;
    }

    inline fn mul(self: *Self) void {
        const b = self.pop();
        var sp = self.stack_top - 1;
        sp[0] = sp[0] * b;
    }

    pub fn run(self: *Self) Result {
        while (true) {
            switch (opcode) {
                // ...
                .Add => self.add(),
                .Sub => self.sub(),
                .Mul => self.mul(),
                .Div => self.div(),
                // ...
            }
        }
    }

But now you see clearly what parts of the code can be easily generated, all these functions only differ by their name. Unfortunately, Zig’s comptime isn’t capable of treating operators as values, so we can’t write something like self.binaryOp(-) and handle it at compilation time to produce the body. Oh well, Lisp macros have spoiled me, I guess.

That’s like the only part I have an issue with in this language. The author is very vocal about macros, but after speaking with them on IRC a few times it seems to me that they only saw C macros and, maybe, C++ templates, and were like: “Yikes, macros are bad!”. We’ve been manipulating code as data with macros since the 60s, people did a lot of research creating safer ways to write macros, then the C preprocessor comes along and now everybody shits on macros because text substitutions are bad. Of course, they are! What baffles me the most is that among lispers there are people who are equally vocal about how bad macros are and that you should never write ones. I guess I’ll write another post about that, adding another hit to the horse.

Scanning on Demand

This chapter starts painfully. Well, painfully isn’t the right word really, but it introduces a lot of inconvenience by breaking our main function.

At the start of the chapter, we’re introduced to two basic workflows our executable will support - run a “REPL” or run a file. And that brings me to my pain point - why doesn’t Zig have any kind of a REPL? Again, I’m spoiled by lisps, but it’s so liberating to have a broken program, yet still be able to call functions from the REPL, gradually fixing the program. Here, I’m basically stripped of any way to run any code, until I make the whole source work again. Well, that’s the downside of the edit compile run cycle, but lisps are compiled languages that don’t have such a problem. The compilation² just happens in the REPL.

As a matter of fact, the main function will be broken for the entirety of this chapter.

This nitpick aside, we’re finally starting to work on the parser. Unfortunately, this chapter isn’t interesting on its own, when it comes to implementing this in Zig. Same tricks with pointers replaced by integers, and pretty much the same code otherwise.

Still not sure why the chapter required to break main if we never actually used it. It will make more sense as the part of second chapter.

Compiling Expressions

In the spirit of the previous chapter, the breakage continues. Remember how I said that writing tests during this book will result in lots of test rewriting due to constant refactoring? This is probably the first, but not the last instance of this - we’ve changed the interpret signature to accept the source code instead of a chunk.

This broke all of my VM tests, though, thankfully, I only had just one at this point. Actually, I don’t like this change, to be honest, I’d much rather preserve the old signature, as the VM interprets the bytecode, not the source code. The bytecode doesn’t have to come from the source code, it can come from anywhere, and hence it should be what the function accepts. Instead, we create the chunk inside the interpret function and fill it with the compiler. Oh well, I’m not a language designer proper, so I guess I should not complain.

Anyway, this chapter is interesting. We’re actually starting to compile our language from source code into bytecode, and that’s cool. The parser itself is quite concise, and I like how it is implemented so far. One interesting thing I’d like to point out is how the book defines the rules:

ParseRule rules[] = {
  [TOKEN_LEFT_PAREN]    = {grouping, NULL,   PREC_NONE},
  [TOKEN_RIGHT_PAREN]   = {NULL,     NULL,   PREC_NONE},
  [TOKEN_LEFT_BRACE]    = {NULL,     NULL,   PREC_NONE},
  [TOKEN_RIGHT_BRACE]   = {NULL,     NULL,   PREC_NONE},
  [TOKEN_COMMA]         = {NULL,     NULL,   PREC_NONE},
  [TOKEN_DOT]           = {NULL,     NULL,   PREC_NONE},
  [TOKEN_MINUS]         = {unary,    binary, PREC_TERM},
  [TOKEN_PLUS]          = {NULL,     binary, PREC_TERM},
  [TOKEN_SEMICOLON]     = {NULL,     NULL,   PREC_NONE},
  [TOKEN_SLASH]         = {NULL,     binary, PREC_FACTOR},
  [TOKEN_STAR]          = {NULL,     binary, PREC_FACTOR},
  [TOKEN_BANG]          = {NULL,     NULL,   PREC_NONE},
  [TOKEN_NUMBER]        = {number,   NULL,   PREC_NONE},
  // ...
  [TOKEN_ERROR]         = {NULL,     NULL,   PREC_NONE},
  [TOKEN_EOF]           = {NULL,     NULL,   PREC_NONE},
};

static ParseRule* getRule(TokenType type) {
  return &rules[type];
}

This is when C looks nice and beautiful (if you can say so). The fact that enumerations in C are just numbers has a lot of problems, but sometimes it’s cool because you can use them as array indexes.

As for Zig, I couldn’t find if it has a way to do the same thing. Mainly because you can’t initialize an array using enumeration tags as indexes.

Using zig translate-c on this simplified example:

typedef enum {
    TOKEN_RIGHT_PAREN,
    TOKEN_LEFT_BRACE,
    TOKEN_RIGHT_BRACE,
    TOKEN_COMMA,
    TOKEN_DOT,
} TokenType;

typedef enum {
    PREC_A,
    PREC_B,
    PREC_C,
    PREC_D,
} Precedence;

typedef struct {
    Precedence precedence;
} ParseRule;

ParseRule rules[] = {
    [TOKEN_LEFT_BRACE]    = {PREC_A},
    [TOKEN_DOT]           = {PREC_B},
    [TOKEN_RIGHT_BRACE]   = {PREC_C},
    [TOKEN_RIGHT_PAREN]   = {PREC_D},
};

We get the following Zig code:

// .. tons of definitions
pub const TOKEN_RIGHT_PAREN: c_int = 0;
pub const TOKEN_LEFT_BRACE: c_int = 1;
pub const TOKEN_RIGHT_BRACE: c_int = 2;
pub const TOKEN_COMMA: c_int = 3;
pub const TOKEN_DOT: c_int = 4;
pub const TokenType = c_uint;
pub const PREC_A: c_int = 0;
pub const PREC_B: c_int = 1;
pub const PREC_C: c_int = 2;
pub const PREC_D: c_int = 3;
pub const Precedence = c_uint;
pub const ParseRule = extern struct {
    precedence: Precedence,
};
pub export var rules: [5]ParseRule = [5]ParseRule{
    ParseRule{
        .precedence = @as(c_uint, @bitCast(PREC_D)),
    },
    ParseRule{
        .precedence = @as(c_uint, @bitCast(PREC_A)),
    },
    ParseRule{
        .precedence = @as(c_uint, @bitCast(PREC_C)),
    },
    @import("std").mem.zeroes(ParseRule),
    ParseRule{
        .precedence = @as(c_uint, @bitCast(PREC_B)),
    },
};
// .. tons of more definitions

Looks like Zig doesn’t have a syntax like that, suggested by the fact that it sorted elements of the array based on the enumeration tag values. But that practically removes any benefits of using enumerations in such a way, as we’re forced to keep the order ourselves. Perhaps, there’s a way to do that using comptime, but I’m not that good with Zig. So, I did it like that:

fn makeRules() []ParseRule {
    var r = [_]ParseRule{undefined} ** @typeInfo(TokenType).Enum.fields.len;
    r[@intFromEnum(TokenType.LEFT_PAREN)] = .{ .prefix = Parser.grouping, .infix = null, .precedence = .CALL };
    r[@intFromEnum(TokenType.RIGHT_PAREN)] = .{ .prefix = null, .infix = null, .precedence = .NONE };
    r[@intFromEnum(TokenType.LEFT_BRACE)] = .{ .prefix = null, .infix = null, .precedence = .NONE };
    r[@intFromEnum(TokenType.RIGHT_BRACE)] = .{ .prefix = null, .infix = null, .precedence = .NONE };
    r[@intFromEnum(TokenType.COMMA)] = .{ .prefix = null, .infix = null, .precedence = .NONE };
    r[@intFromEnum(TokenType.DOT)] = .{ .prefix = null, .infix = null, .precedence = .NONE };
    r[@intFromEnum(TokenType.MINUS)] = .{ .prefix = Parser.unary, .infix = Parser.binary, .precedence = .TERM };
    r[@intFromEnum(TokenType.PLUS)] = .{ .prefix = null, .infix = Parser.binary, .precedence = .TERM };
    r[@intFromEnum(TokenType.SEMICOLON)] = .{ .prefix = null, .infix = null, .precedence = .NONE };
    r[@intFromEnum(TokenType.SLASH)] = .{ .prefix = null, .infix = Parser.binary, .precedence = .FACTOR };
    r[@intFromEnum(TokenType.STAR)] = .{ .prefix = null, .infix = Parser.binary, .precedence = .FACTOR };
    r[@intFromEnum(TokenType.BANG)] = .{ .prefix = Parser.unary, .infix = null, .precedence = .NONE };
    r[@intFromEnum(TokenType.NUMBER)] = .{ .prefix = Parser.number, .infix = null, .precedence = .NONE };
    // ...
    r[@intFromEnum(TokenType.ERROR)] = .{ .prefix = null, .infix = null, .precedence = .NONE };
    r[@intFromEnum(TokenType.EOF)] = .{ .prefix = null, .infix = null, .precedence = .NONE };
    return &r;
}

const rules = makeRules();

fn getRule(t: TokenType) ParseRule {
    return rules[@intFromEnum(t)];
}

It works, looks awful.

Apart from that, the chapter is pretty straightforward. Another thing I want to point out is the fact that C supports compilation flags out of the box, and Zig doesn’t.

Well, in C it is a hot mess, but it works reliably enough to see it as a feature in my opinion:

#define DEBUG_PRINT_CODE

// later in the code

#ifdef DEBUG_PRINT_CODE
    // compile-time code injection
#endif

Next, when compiling the code, you can supply -DDEBUG_PRINT_CODE to the compiler and it will set this to something so the #ifdef directive contents will be included in the resulting source code.

In Zig, there’s again comptime and I guess writing if (comptime DEBUG_PRINT_CODE) { ... } would include/eliminate the code at compile time, but how do we supply it to the compiler? The answer is the build system of Zig, which itself is just a program written in Zig. And again, I feel that this is way too complicated than it should be, especially since I have tests in a separate directory, and I have to pass compilation options to tests as well as to the main executable. I didn’t bother with that in the end.

Finally, challenges. This chapter has a decent set of challenges³, except for the last one:

You might be wondering about complex “mixfix” expressions that have more than two operands separated by tokens. C’s conditional or “ternary” operator, ?:, is a widely known one.

Add support for that operator to the compiler. You don’t have to generate any bytecode, just show how you would hook it up to the parser and handle the operands.

First, no - why would you need to implement ?: before we have implemented if itself? Second - why implement ?: at all if we’re implementing the new language and can make if to be an expression? I understand that this is a small and simple language, but the if expressions can be a great way to introduce people to the concept of familiar statements being expressions as well.

Types of Values

This chapter begins with a paragraph I can connect with:

The nice thing about working in C is that we can build our data structures from the raw bits up. The bad thing is that we have to do that. C doesn’t give you much for free at compile time and even less at runtime. As far as C is concerned, the universe is an undifferentiated array of bytes. It’s up to us to decide how many of those bytes to use and what they mean.

Indeed, when I was working with C this was always the case. On one hand, it’s problematic, on the other, it is empowering. But I’m doing this book’s project in Zig and I can’t say that Zig is the same as C in this regard. Instead of doing things the C way, Zig provides more high-level concepts to us, like union enums, and such. So let’s try to use these instead.

Here’s how we implement the Value:

pub const Value = union(enum) {
    Number: f64,
    Bool: bool,
    Nil,

    pub fn print(self: Value) void {
        switch (self) {
            .Number => |value| std.debug.print("{d}", .{value}),
            .Bool => |value| std.debug.print("{}", .{value}),
            .Nil => std.debug.print("nil", .{}),
        }
    }
}

In its essence, it’s similar to the union in the book:

typedef enum {
  VAL_BOOL,
  VAL_NIL,
  VAL_NUMBER,
} ValueType;

typedef struct {
  ValueType type;
  union {
    bool boolean;
    double number;
  } as;
} Value;

However, as an added bonus, we can add methods to our Value type, the same as with other struct’s. As you can see, the print function is a part of the union(enum) and can be invoked as val.print() later in the code.

Once again, we have to change parts of our code that are quite basic, but nothing too breaking this time. All I had to do was wrap numbers into the Value constructor in a few tests. This is just the beginning though, we have to make more Value sub-types for strings, and objects.

Strings

Now, this is where my previous attempt at this book in Zig train-wrecked. Or, should I say, it was the beginning of the end for that implementation. You see, til now, most concepts introduced by the book were easy enough to translate from C to Zig, because most of the time the stuff was either the same or already backed by Zig’s standard library. For example, at the start of the book, I decided not to implement dynamic arrays, because:

I already made my fair share of dynamic arrays in C back when I was a C programmer
After looking at what the book suggests, and the implementation in Zig’s standard library, they’re practically the same, with some small changes that are irrelevant for the matter.

Other stuff was pretty much the same, except for, perhaps, how pointers are used - instead I used offsets most of the time, as it is easier to do. Unions and enumerations are again similar, though with some small differences, like being able to use union(enum) and add methods onto it, but again, C code kinda does the same thing, just the C way. Strings, however, are a different beast. They’re objects, and hence they’re dynamically typed because objects kinda are. E.g. in Lox string is an object, a function is also an object, and a class also is an object, and we have to express this idea somehow in C/Zig. Here’s how the book does it in C:

typedef enum {
  OBJ_STRING,
} ObjType;

struct Obj {
  ObjType type;
};

This is only a struct definition for the object type, but there’s a certain trick that we can do in C by abusing the fact that structure layouts are consistent. We can create a second structure like this one:

struct ObjString {
  Obj obj;
  int length;
  char* chars;
};

Now, here’s an interesting part - we can convert a pointer between these two structs at will! It is possible because the memory layout between the first field in the ObjString and the entirety of Obj struct is the same. Now, since the first field in the struct has the same address as the struct itself, i.e. an offset of 0, and the struct field size is the same as its type, given the pointer to ObjString we can cast it to Obj and read its fields as normal.

But how do we do that in Zig? Zig’s structs are much more than C structs, as they can have methods, we can manipulate field types at compile time, and so on. My first approach was like this:

pub const LoxString = []const u8;

pub const LoxObject = struct {
    next: ?*LoxObject,
    data: union(enum) {
        String: LoxString,
        _,
    },

    pub fn objType(self: LoxObject) Tag(LoxObject) {
        return activeTag(self.data);
    }

    pub fn isString(self: LoxObject) bool {
        return switch (self.data) {
            .String => true,
            else => false,
        };
    }

    pub fn allocString(s: LoxString, vm: *VM(Code), allocator: Allocator) *LoxObject {
        var obj = allocator.create(LoxObject) catch exit(123, "Not enough memory to allocate object");
        switch (obj.*.data) {
            .String => |*value| value.* = s,
            else => unreachable,
        }
        obj.next = vm.objects;
        vm.objects = obj;
        return obj;
    }

    pub fn freeObject(obj: *LoxObject, allocator: Allocator) void {
        switch (obj.*.data) {
            .String => |s| allocator.free(s),
            else => unreachable,
        }
        allocator.destroy(obj);
    }
};

As you can see, I decided to store the underlying data of the object in the union(enum), similarly to how I do that for Value:

pub const Value = union(enum) {
    Number: f64,
    Bool: bool,
    Nil,
    Obj: *Obj,
    // ...
};

Later, when we added more object types, this became a problem. This layout means that I can’t cast the LoxObject to a LoxString, and I can’t really access it without going through the union:

test "string concatenation" {
    var vm = VM(Code).init(allocator);
    defer vm.deinit();

    try switch (interpret("\"a\" + \"b\"", &vm).OK) {
        .Obj => |obj| switch (obj.*.data) {
            .String => |s| expectEqualSlices(u8, s, "ab"),
            else => unreachable,
        },
        else => unreachable,
    };
}

The code was filled with these obj.*.data switches, that extracted the string. And later on, at some point, I had really weird bugs, like when my isString function returns true, but when I try to access the string, I get the error “trying to access the inactive field of enumeration”. Be it a bug in Zig itself or in my code I don’t care.

Since then I changed the implementation to be more like in the book:

pub const Type = enum {
    String,
};

pub const Obj = struct {
    objType: Type,
    next: ?*Obj,

    pub fn allocate(vm: *VM, comptime T: type, objType: Type) *Obj {
        const ptr = vm.allocator.create(T) catch
            Common.exit(200, "OOME: can't allocate object", .{});
        ptr.obj = Obj{ .objType = objType, .next = vm.objects };
        vm.objects = &ptr.obj;
        return &ptr.obj;
    }

    pub fn asString(self: *Obj) *String {
        return @fieldParentPtr(String, "obj", self);
    }

    pub fn destroy(self: *Obj, vm: *VM) void {
        switch (self.objType) {
            .String => self.asString().destroy(vm),
        }
    }
};

pub const String = struct {
    obj: Obj,
    bytes: []const u8,

    pub fn create(vm: *VM, bytes: str) *String {
        const obj = Obj.allocate(vm, String, .String);
        const out = obj.asString();
        out.* = String{ .obj = obj.*, .bytes = bytes };
        return out;
    }

    pub fn copy(vm: *VM, source: str) *String {
        const buffer = vm.allocator.alloc(u8, source.len) catch
            Common.exit(200, "OOME: can't allocate string", .{});
        std.mem.copy(u8, buffer, source);
        return String.create(vm, buffer);
    }

    pub fn destroy(self: *String, vm: *VM) void {
        vm.allocator.free(self.bytes);
        vm.allocator.destroy(self);
    }
};

Now, this @fieldParentPtr feels sketchy with its "obj" parameter. It is supposed to be a field name, and I find string a weird choice as strings can contain arbitrary data, while field names can’t. But that’s beside the point, let’s look at it a bit closer:

17
18
19

    pub fn asString(self: *Obj) *String {
        return @fieldParentPtr(String, "obj", self);
    }

The signature of this builtin is as follows: comptime ParentType: type, comptime field_name: []const u8, field_ptr: *T, and the return type is a pointer to the ParentType. As you can see, the usage is interesting - we pass the desired type String which is a different struct. Then we pass the "obj" field name, and the last parameter should be a pointer to a field, for which we use the object pointer itself.

Presumably, Zig knows its memory layout and can calculate the offset to the base struct given a field pointer. Indeed, knowing the layout and giving a pointer to a field, it is possible to calculate the start of the struct, however, in this case, we do it a bit weird, as we get a pointer to the Obj structure, which doesn’t have the obj field at all. We then use this pointer as a third parameter, i.e. as a pointer to the field of the String structure. Therefore, this built-in calculates the width of the obj field, subtracts it from the pointer address, and returns the pointer to the base struct but as the String type.

Overall, this looks like what we do in C, except we use @fieldParentPtr to do the pointer casting. We can see it being used in the create function:

    pub fn create(vm: *VM, bytes: str) *String {
        const obj = Obj.allocate(vm, String, .String);
        const out = obj.asString();
        out.* = String{ .obj = obj.*, .bytes = bytes };
        return out;
    }

We have the object allocated as a String, ensuring the memory layout, but then we assign the .obj field an Obj structure, finally returning a pointer to this field:

    pub fn allocate(vm: *VM, comptime T: type, objType: Type) *Obj {
        const ptr = vm.allocator.create(T) catch
            Common.exit(200, "OOME: can't allocate object", .{});
        ptr.obj = Obj{ .objType = objType, .next = vm.objects };
        vm.objects = &ptr.obj;
        return &ptr.obj;
    }

Everything should now make sense - we indeed have the pointer to the obj field by the time we get to use the asString function, but we always have the pointer to the Obj at hand.

In other words, it’s not as in C where we literally convert a single pointer between two different layouts, abusing the fact that the memory layout allows for it. Instead, we can have a pointer to anywhere inside the struct and still get the correct parent pointer thanks to the known memory layout. We could do the same in C but it would be more work, and far more error-prone, as we would need to compute this offset manually. Here, Zig has us covered.

Overall, I like this approach more, I guess. C’s way of doing this feels like a clever memory trick, while Zig’s way of doing this is more of a compiler-supported way.

Hash Tables

Now, in my previous attempt I thought to myself: “Well, I skipped the dynamic array, let’s at least implement a hash table”. As a result, I ended up looking at how the hash table is implemented in Zig’s standard library. Now, I’m not sure if my implementation was correct, but it seemed that way.

Looking at it now, I’m not sure if I could use a hash table from Zig’s standard library, as it needs custom hashing. I probably can implement some kind of interface for the objects that would allow me to hash them. So let’s look into that:

/// General purpose hash table.
/// No order is guaranteed and any modification invalidates live iterators.
/// It provides fast operations (lookup, insertion, deletion) with quite high
/// load factors (up to 80% by default) for low memory usage.
/// For a hash map that can be initialized directly that does not store an Allocator
/// field, see `HashMapUnmanaged`.
/// If iterating over the table entries is a strong usecase and needs to be fast,
/// prefer the alternative `std.ArrayHashMap`.
/// Context must be a struct type with two member functions:
///   hash(self, K) u64
///   eql(self, K, K) bool
/// Adapted variants of many functions are provided.  These variants
/// take a pseudo key instead of a key.  Their context must have the functions:
///   hash(self, PseudoKey) u64
///   eql(self, PseudoKey, K) bool
pub fn HashMap(
    comptime K: type,
    comptime V: type,
    comptime Context: type,
    comptime max_load_percentage: u64,
) type {
    // ...
}

OK, so the documentation of std.hash_map.HashMap says that we can pass a Context that has the hash and eql functions. That should do it.

Speaking of reusing the standard library - Zig already has the “FNV-1a” algorithm the book implements as a part of std.hash.Fnv1a_32. So we’re going to reuse it too.

After walking through the chapter, here’s the final table:

const table_max_load = 75;

pub const Table = struct {
    hm: HashMap(*String, Value, Ctx, table_max_load),

    const Ctx = struct {
        pub fn hash(_: Ctx, key: *String) u64 {
            return @intCast(key.*.hash);
        }

        pub fn eql(_: Ctx, key_a: *String, key_b: *String) bool {
            return key_a.*.hash == key_b.*.hash and
                std.mem.eql(u8, key_a.*.bytes, key_b.*.bytes);
        }
    };

    pub fn init(allocator: Allocator) Table {
        return Table{ .hm = HashMap(*String, Value, Ctx, table_max_load).init(allocator) };
    }

    pub fn deinit(self: *Table) void {
        self.hm.deinit();
    }

    pub fn findString(self: *Table, chars: str, hash: u32) ?*String {
        return self.hm.getKey(@constCast(&String{ .obj = undefined, .hash = hash, .bytes = chars }));
    }

    pub fn set(self: *Table, key: *String, value: Value) bool {
        return if (self.hm.fetchPut(key, value) catch
            common.exit(205, "can't put the key to a table", .{})) |_|
            false // key is not new
        else
            true;
    }

    pub fn get(self: *Table, key: *String) ?Value {
        return self.hm.get(key);
    }

    pub fn delete(self: *Table, key: *String) bool {
        self.hm.remove(key);
    }
};

Someone would say that that’s cheating.

While yes, I'm not doing what the book suggests, I already have done that before.

const table_max_load = 0.75;

const Entry = struct {
    key: ?*LoxObject,
    value: Value,
};

inline fn growCapacity(capacity: usize) usize {
    return if ((capacity) < 8) 8 else capacity * 2;
}

pub const Table = struct {
    entries: []Entry,
    capacity: usize,
    allocator: Allocator,
    count: usize,

    pub fn init(allocator: Allocator) Table {
        return Table{
            .allocator = allocator,
            .entries = &[_]Entry{},
            .capacity = 0,
            .count = 0,
        };
    }

    fn allocatedSlice(self: *Table) []Entry {
        return self.entries.ptr[0..self.capacity];
    }

    pub fn deinit(self: *Table) void {
        self.allocator.free(self.allocatedSlice());
    }

    fn findEntry(entries: []Entry, capacity: usize, key: *LoxObject) *Entry {
        const hash = switch (key.data) {
            .String => |s| s.hash,
            else => unreachable,
        };
        var index = hash % capacity;
        var tombstone: ?*Entry = null;

        while (true) {
            const entry = &entries[index];
            if (entry.key == null) {
                if (entry.value.isNil()) {
                    return if (tombstone) |t| t else entry;
                } else {
                    if (tombstone == null) tombstone = entry;
                }
            } else if (entry.key == key) {
                return entry;
            }
            index = (index + 1) % capacity;
        }
    }

    pub fn findString(self: *Table, chars: []const u8, hash: u32) ?*LoxObject {
        if (self.count == 0) return null;

        var index = hash % self.capacity;
        const entries = self.entries;
        while (true) {
            const entry = entries[index];
            if (entry.key) |k| {
                if (switch (k.data) {
                    .String => |s| s.hash == hash and
                        std.mem.eql(u8, s.chars, chars),
                    else => false,
                })
                    return entry.key;
            } else {
                if (entry.value.isNil()) return null;
            }

            index = (index + 1) % self.capacity;
        }
    }

    fn adjustCapacity(self: *Table, capacity: usize) void {
        const new_memory = self.allocator.alloc(Entry, capacity) catch
            common.exit(48, "can't allocate memory for the hash table");

        for (new_memory) |*entry| {
            entry.key = null;
            entry.value = Nil;
        }

        self.count = 0;
        for (self.entries) |*entry| {
            if (entry.key) |key| {
                var dest = findEntry(new_memory, capacity, key);
                dest.key = entry.key;
                dest.value = entry.value;
                self.count += 1;
            }
        }

        self.allocator.free(self.allocatedSlice());

        self.entries = new_memory;
        self.capacity = capacity;
    }

    fn tableAddAll(from: *Table, to: *Table) void {
        var i = 0;
        while (i < from.capacity) : (i += 1) {
            const entry = &from.entries[i];
            if (entry.key != null) {
                to.tableSet(entry.key, entry.value);
            }
        }
    }

    pub fn set(self: *Table, key: *LoxObject, value: Value) bool {
        if (self.count + 1 > @floatToInt(usize, @intToFloat(f32, self.capacity) * table_max_load)) {
            const capacity = growCapacity(self.capacity);
            self.adjustCapacity(capacity);
        }
        var entry = findEntry(self.entries, self.capacity, key);
        const is_new = entry.key == null;
        if (is_new and entry.value.isNil()) self.count += 1;
        entry.key = key;
        entry.value = value;
        return is_new;
    }

    pub fn get(self: *Table, key: *LoxObject) ?Value {
        if (self.count == 0) return null;

        const entry = findEntry(self.entries, self.capacity, key);
        if (entry.key == null) return null;

        return entry.value;
    }

    pub fn delete(self: *Table, key: *LoxObject) bool {
        if (self.count == 0) return false;

        const entry = findEntry(self.entries, self.capacity, key);
        if (entry.key == null) return false;

        entry.key = null;
        entry.value = True;
        return true;
    }
};

I just don’t feel like doing it again.

And, I would like to learn how to use Zig’s own hash tables, so this is a perfect opportunity. The use of Context is interesting - thanks to it we can implement the hashing however we want.

The only thing I don’t like in the new solution is the @constCast:

25
26
27

    pub fn findString(self: *Table, chars: str, hash: u32) ?*String {
        return self.hm.getKey(@constCast(&String{ .obj = undefined, .hash = hash, .bytes = chars }));
    }

This function itself is a bit hacky, as we only mimic the key object by creating it on a stack. Alternatively, I could use the getKeyAdapted, and provide it a context that does a lookup of a fake key:

    // after 'const Ctx = struct { ... };'

    const AdaptedCtx = struct {
        h: u32,
        b: str,

        pub fn init(b: str, h: u32) AdaptedCtx {
            return AdaptedCtx{ .b = b, .h = h };
        }

        pub fn hash(self: AdaptedCtx, comptime _: @TypeOf(null)) u64 {
            return @intCast(self.h);
        }

        pub fn eql(self: AdaptedCtx, comptime _: @TypeOf(null), key: *String) bool {
            return self.h == key.*.hash and
                std.mem.eql(u8, self.b, key.*.bytes);
        }
    };

Now we can change findString to:

43
44
45

    pub fn findString(self: *Table, chars: str, hash: u32) ?*String {
        return self.hm.getKeyAdapted(null, AdaptedCtx.init(chars, hash));
    }

I don’t really like it either, and I don’t know which one is better/more idiomatic. But, other than that, the new version is nice and clear.

Global Variables

Here we go again. By the end of this chapter, all tests will be broken. Again.

I mean, I’m glad I write tests, as Zig privies a test allocator that acts like Valgrind and I had noticed a couple of memory leaks. Not because the book has them, but just because I was careless and forgot to clean up some values.

However, now no VM test is passing. Why is that? Because this chapter introduced statements, and now if you write any expression you must end it with a semicolon, thus turning it into a statement. And statements are known for not returning anything. Hence the interpret function now can’t return anything, and therefore I can’t test for anything. Amazing!

Now, there are ways of working this around but all of these are hacky. I think the proper solution would be to introduce a top-level return statement, so we could return any expression from the top level. And it’s a shame - the book mentions the REPL four times during this chapter, saying how useful something would be in the context of one. However, it completely disregards the fact that REPL stands for Read Eval Print Loop, and in our case, we can’t really print anything in the REPL, because now everything is a statement. Welp.

What I like about Lox is that the syntax for declaring a variable requires a keyword. This is one of the reasons I dislike languages like Python - looking at the assignment you can’t guess if you’re creating a new variable or assigning an existing one. Paired with the absence of scopes, we get this horrible thing:

def foo():
    if True:
        a = 10
    print(a)

foo()

Why is a available outside the scope of the if statement? Because there’s only one scope⁴ - the function scope.

Lua is also guilty of this - assigning a non-existent variable creates a global:

function foo()
   if true then
      a = 10
   end
   print(a)
end

foo()

But at least it is global, and you can avoid this behavior by using a local keyword. Something that all of the Lua code should really do, that’s why Fennel uses local by default.

JavaScript also had this problem, but then they introduced the let keyword. It’s still weird because let variables are “hoisted” - their logical position of definition is the top of their enclosing scope. It’s not as bad as it sounds, e.g. you can’t do this:

function f() {
  {
    console.log(x) // Error: can't access lexical declaration 'x' before initialization
  }
  let x = 1
}

But you can do this:

function f() {
  function g() {
    console.log(x) // a closure? to what?
  }
  let x = 1
  g()
}
f() // prints 1

But I’m getting ahead of the book. The next chapter is on local variables and scope, so let’s see how Lox handles them.

Local Variables

And to be honest, it’s not bad. Lox handles local variables pretty much the same way as most other languages I have used. The only nitpick I have is this:

{
  var a = 20;
  var a = 30; // Error at 'a': Already a variable with this name in this scope.
}

I don’t like this. In Clojure, that’s the norm, because there’s no way to set a new value to an already established binding. So you can overshadow it with a different value.

Rust also allows shadowing, and I think that’s a good feature to have. Sometimes you want to keep the name but change the type. The Rust book has this example:

let spaces = "   ";
let spaces = spaces.len();

Note, this can’t be done using mutability, because of the type system:

let mut spaces = "   ";
spaces = spaces.len();

The error says we’re not allowed to mutate a variable’s type:

$ cargo run
   Compiling variables v0.1.0 (file:///projects/variables)
error[E0308]: mismatched types
 --> src/main.rs:3:14
  |
2 |     let mut spaces = "   ";
  |                      ----- expected due to this value
3 |     spaces = spaces.len();
  |              ^^^^^^^^^^^^ expected `&str`, found `usize`

For more information about this error, try `rustc --explain E0308`.
error: could not compile `variables` due to previous error

Lox, however, is dynamically typed, so this can be done using mutability, and that may be a reason to disallow overshadowing. Still, I would prefer variables that are constant, and overshadowing instead of assignments.

Jumping Back and Forth

Oh, boy.

This is a big chapter. It introduces most of the control flow primitives to the language, in terms of branching, looping, and short-circuiting. I have a lot to say.

If statements

First things first, the book introduces the if statement. To which I ask - why not introduce an if expression instead? if expressions are very useful, and not that hard to implement, especially in a small language like Lox. I guess, it’s a bit late to do so, as almost everything in Lox grammar builds from the statement rule, so if expressions would be out of place. It’s a shame though, as a lot of people don’t even know that if can be an expression, without being a ternary operator ?:.

Anyway, the if expression is implemented in an interesting way. We parse the if and the condition, and emit a jump instruction. Emitting is a bit special we also added a placeholder that we will later patch:

    fn emitJump(self: *Self, instruction: OPCode) u32 {
        self.emitByte(@intFromEnum(instruction));
        self.emitByte(0xff);
        self.emitByte(0xff);
        return @as(u32, @intCast(self.currentChunk().code.items.len)) - 2;
    }

These two emitByte calls with 0xff as an argument are exactly for that.

After the emitJump step, we parse a statement and then emit another jump instruction. And right after that, we patch the original jump:

    fn ifStatement(self: *Self) void {
        self.consume(.LEFT_PAREN, "Expect '(' after 'if'.");
        self.expression();
        self.consume(.RIGHT_PAREN, "Expect ')' after condition.");

        const thenJump = self.emitJump(.JumpIfFalse);
        self.emitByte(@intFromEnum(OPCode.Pop));
        self.statement();

        const elseJump = self.emitJump(.Jump);
        self.patchJump(thenJump);
        self.emitByte(@intFromEnum(OPCode.Pop));

        if (self.match(.ELSE)) self.statement();
        self.patchJump(elseJump);
    }

The same happens for the elseJump at the end of the function, if the else branch is present.

So, how it works? The patchJump function looks like this:

    fn patchJump(self: *Self, offset: u32) void {
        // -2 to adjust for the bytecode for the jump offset itself.
        var jump = self.currentChunk().code.items.len - offset - 2;

        if (jump > std.math.maxInt(u16)) {
            self.error_("Too much code to jump over.", .{});
        }

        self.currentChunk().code.items[offset] = @intCast((jump >> 8) & 0xff);
        self.currentChunk().code.items[offset + 1] = @intCast(jump & 0xff);
    }

Basically, we count how much code we generated by reading the statement after the condition, and patch in the address we need to jump to if the condition was false. It’s a clever trick. Notice, the if statement itself doesn’t know anything about scopes - all that is handled by calling the expression function. It will kinda be important later.

Aside from not being an expression, I have no objections to implementing the if statement this way. Well, the limitation of 65535 opcodes is a bit weird, but if you may remember, the book earlier suggested implementing ways to have more than 256 constants by introducing more instructions that know how to deal with that. Here, we could also introduce a LongJump and LongJumpIfFalse instructions that would handle 4294967295 opcodes instead if the body is longer than 65535 opcodes. But this complicates things, and previously it resulted in a lot of misunderstanding where I should and should not take care of such long instructions while I did a first run of the book. So I won’t do it here.

Now, then there’s a “Challenge”:

In addition to if statements, most C-family languages have a multi-way switch statement. Add one to clox. The grammar is:
   switchStmt     → "switch" "(" expression ")"
                    "{" switchCase* defaultCase? "}" ;
   switchCase     → "case" expression ":" statement* ;
   defaultCase    → "default" ":" statement* ;
To execute a switch statement, first evaluate the parenthesized switch value expression. Then walk the cases. For each case, evaluate its value expression. If the case value is equal to the switch value, execute the statements under the case and then exit the switch statement. Otherwise, try the next case. If no case matches and there is a default clause, execute its statements.

To keep things simpler, we’re omitting fallthrough and break statements. Each case automatically jumps to the end of the switch statement after its statements are done.

Uh, this isn’t how switch is supposed to work. The swich statement is a constant dispatch, meaning cases aren’t tried in order, instead, the right one is picked based on the condition. It can be done with a jump table, or by computing offsets at compile time, and allowing only integer switching and then jumping to the given branch. That’s also a reason why C implements break, without it the program will continue executing to the next label because labels don’t really exist in the program. This allows for clever tricks like the Duff’s device:

send(to, from, count)
register short *to, *from;
register count;
{
    register n = (count + 7) / 8;
    switch (count % 8) {
    case 0: do { *to = *from++;
    case 7:      *to = *from++;
    case 6:      *to = *from++;
    case 5:      *to = *from++;
    case 4:      *to = *from++;
    case 3:      *to = *from++;
    case 2:      *to = *from++;
    case 1:      *to = *from++;
            } while (--n > 0);
    }
}

I do understand that the book tries to keep things easy, but this kind of switch statement is a useless addition in my opinion. If Lox had macros, such switch could be easily implemented in terms of chained if else.

Loops

I’m skipping over logical operators, as I don’t have anything to say about them - they’re OK. Loops, however, are, again, weird.

First, we have a while loop, which is probably the only kind of loop any language needs if it doesn’t implement tail call elimination/optimization technique. It is implemented in pretty much the same way as the if statement, except we emit a Loop instruction instead. Again, the design is OK, it is very bare-bones and clean:

    fn emitLoop(self: *Self, loopStart: u32) void {
        self.emitByte(@intFromEnum(OPCode.Loop));

        const offset = self.currentChunk().code.items.len - loopStart + 2;
        if (offset > std.math.maxInt(u16)) self.error_("Loop body too large.", .{});

        self.emitByte(@intCast((offset >> 8) & 0xff));
        self.emitByte(@intCast(offset & 0xff));
    }

    fn whileStatement(self: *Self) void {
        const loopStart: u32 = @intCast(self.currentChunk().code.items.len);
        self.consume(.LEFT_PAREN, "Expect '(' after 'while'.");
        self.expression();
        self.consume(.RIGHT_PAREN, "Expect ')' after condition.");

        const exitJump = self.emitJump(.JumpIfFalse);
        self.emitByte(@intFromEnum(OPCode.Pop));
        self.statement();
        self.emitLoop(loopStart);

        self.patchJump(exitJump);
        self.emitByte(@intFromEnum(OPCode.Pop));
    }

Then, however, we have a for loop. Not only the implementation is much more involved, as it is implemented in terms of the while loop, but it is also quite useless. Why? Because it is a plain old C-style numeric loop. We already have while, why do we need for? I mean, it’s not too hard to translate this into while loop:

for (var i = 0; i < 10; i = i + 1) {
  print i;
}

Here:

{ var i = 0; while (i < 10) {
    print i;
  i = i + 1; } }

That’s about the same amount of code. Yes, I’ve used weird formatting on the while loop example, but only to show that the actual amount of code is about the same. And that’s exactly what the book does in the implementation of the for loop, except it’s this amount of code:

    fn forStatement(self: *Self) void {
        self.beginScope();
        self.consume(.LEFT_PAREN, "Expect '(' after 'for'.");
        if (self.match(.SEMICOLON)) {
            // No initializer.
        } else if (self.match(.VAR)) {
            self.varDeclaration();
        } else {
            self.expressionStatement();
        }

        var loopStart: u32 = @intCast(self.currentChunk().code.items.len);
        var exitJump: i32 = -1;
        if (!self.match(.SEMICOLON)) {
            self.expression();
            self.consume(.SEMICOLON, "Expect ';' after loop condition.");
            // Jump out of the loop if the condition is false.
            exitJump = @intCast(self.emitJump(.JumpIfFalse));
            self.emitByte(@intFromEnum(OPCode.Pop)); // Condition.
        }

        if (!self.match(.RIGHT_PAREN)) {
            const bodyJump = self.emitJump(.Jump);
            const incrementStart: u32 = @intCast(self.currentChunk().code.items.len);
            self.expression();
            self.emitByte(@intFromEnum(OPCode.Pop));
            self.consume(.RIGHT_PAREN, "Expect ')' after for clauses.");

            self.emitLoop(loopStart);
            loopStart = incrementStart;
            self.patchJump(bodyJump);
        }

        self.statement();
        self.emitLoop(loopStart);

        if (exitJump != -1) {
            self.patchJump(@intCast(exitJump));
            self.emitByte(@intFromEnum(OPCode.Pop)); // Condition.
        }

        self.endScope();
    }

Again, if Lox had macros, the language could define for in terms of while without all this code. Macros are special because they allow your language to operate on your code. which may sound complicated, but in actuality is much easier than fiddling with this bytecode generation, as it is more high-level. Here’s an example of a for macro for Emacs Lisp:

(defmacro for (initializer condition step &rest body)
  `(let (,initializer)
     (while ,condition
       (progn ,@body)
       (setq ,(car initializer) ,step))))

(macroexpand-1
 '(for (a 0) (< a 10) (+ a 1)
    (print a)))
;; (let ((a 0))
;;   (while (< a 10)
;;     (progn (print a))
;;     (setq a (+ a 1))))

Yes, it will get more involved if we would like to support a clause without an initializer, or with no explicit step, but in general, manipulating your language with your language is more concise. Again, I know why the book teaches it this way, and it is good to show that the for loop can be implemented in terms of a while loop, but here’s a second question - what the for loop is for in Lox?

There are no arrays in Lox, and you can’t index strings either. The language will add objects later to teach how to implement object-oriented programming, but that’s a hard sell for me and doesn’t make for more useful. Because for doesn’t support object traversal either. Well, theoretically, we can implement the linked list using objects in Lox, and then traverse them using the for loop like so:

for (var pair = ll.rest(); pair.tail != nil; pair = ll.rest()) {
  var val = pair.head;
  // do stuff
}

But that’s not a great interface for linked lists either.

OK, I am nitpicking here. But wait, there’s one more thing - where’s a break statement? Oh, there’s another “Challenge”:

In jlox, we had a challenge to add support for break statements. This time, let’s do continue:
   continueStmt   → "continue" ";" ;
A continue statement jumps directly to the top of the nearest enclosing loop, skipping the rest of the loop body. Inside a for loop, a continue jumps to the increment clause, if there is one. It’s a compile-time error to have a continue statement not enclosed in a loop.

Make sure to think about scope. What should happen to local variables declared inside the body of the loop or in blocks nested inside the loop when a continue is executed?

Wait, it’s not about implementing break? It’s about implementing continue?! Then, what happened to break?

The break is completely omitted from clox. Well, it was a challenge in jlox, and even back then I considered it to be a weird choice. Loops without the break statement are even more useless. Again, you can overcome this by modifying the condition, but that’s extremely ugly. The book has this example in the “Design Note” section:

bool found = false;
for (int x = 0; x < xSize; x++) {
  for (int y = 0; y < ySize; y++) {
    for (int z = 0; z < zSize; z++) {
      if (matrix[x][y][z] == 0) {
        printf("found");
        found = true;
        break;
      }
    }
    if (found) break;
  }
  if (found) break;
}

They suggest that this can be rewritten using goto:

for (int x = 0; x < xSize; x++) {
  for (int y = 0; y < ySize; y++) {
    for (int z = 0; z < zSize; z++) {
      if (matrix[x][y][z] == 0) {
        printf("found");
        goto done;
      }
    }
  }
}
done:

Alternatively, this can be done like this:

bool found = false;
for (int x = 0; !found && x < xSize; x++) {
  for (int y = 0; !found && y < ySize; y++) {
    for (int z = 0; !found && z < zSize; z++) {
      if (matrix[x][y][z] == 0) {
        printf("found");
        found = true;
      }
    }
  }
}

Each for loop will stop once found is set to true. While works, this is clumsy when you need to do an early break:

for (int x = 0; !done && x < 10; x++) {
  if (x == 7) break;
  // rest of the body sits here
}

bool done = false;
for (int x = 0; !done && x < 10; x++) {
  if (x == 7) {
    done = true;
  } else {
    // rest of the body has to be in the else clause
  }
}

While works, this isn’t particularly great. I work with a language that has neither break nor continue - Fennel. It doesn’t have an early return either. It works, however, because Fennel is a lisp-like language, even though it just compiles to Lua. Its syntax is concise, and that allows us to write things with less indentation and nesting than in most other languages. Still, sometimes it is clumsy to write a certain algorithm without an early break.

`continue` and `break`

I’m not going to implement continue, I just don’t feel like it, and certainly haven’t seen continue used in any of the code in ten years. But let’s just think how we could do that.

Here’s an example code:

for (var i = 0; i < 10; i = i + 1) {
  if (i == 5) continue;
  print i;
}

As you may remember, this loop really compiles to this equivalent while loop:

{
  var i = 0;
  while (i < 10) {
    if (i == 5) continue;
    print i;
    i = i + 1;
  }
}

Well, not quite - I lied to you. In the bytecode this loops looks nothing like that, instead it is structured kinda like this:

{
  var i = 0;

  condition:
  if (i < 10)
    goto body:

  increment:
  i = i + 1;
  goto condition:

  body:
  if (i == 5) continue;
  print i;
  goto increment:
}

Upon initialization, we create a condition: label, followed by the if statement. If the if clause is true we goto (or rather use the JUMP instruction) to the body: label. Then, at the end of the body, we goto to the increment: label, which finally brings us back to the condition: label.

Convoluted, huh? Well, the VM doesn’t really have a goto, instead it uses the jumping instructions, but the idea is the same. Now we need to somehow incorporate continue in that.

Semantically, continue should just go to the increment: label, right? Well, yes, however, consider this loop:

{
  var i = 0;
  while (i < 10) {
    if (i == 5) {
      var random = random_int(10);
      // assume we can concatenate strings with numbers using +
      print "The loop will continue from" + random;
      i = random - 1;
      continue;
    };
    print i;
    i = i + 1;
  }
}

Now, this is no different, we still can jump to the increment: label, but what happens to the random variable? Correct, it leaks because by jumping we don’t properly close the scope. So, we need to track how much scopes we jump over as well.

And, mind that, continue may not be the last expression in the scope, so we can’t simply close the scope on it. It can be nested in multiple scopes too.

Implementing break is even harder, because it has to jump over the loop, but we can’t know the loop size, because our compiler is single-pass. Storing the marker and patching it later made harder because break is part of the expression inside the loop, and so we need to have a stack of break labels somewhere.

On the first run I tried some approaches, but failed. None of my solutions worked, and I honestly don’t know why. I consider this partly a book’s failure - implementing break and continue is non-trivial, and just throwing it as a “Challenge” is cheap. I’ve searched for different implementations of the Lox on the web, and I can’t remember if I found any with the continue statement implemented. Maybe, that’s because continue is pretty much useless. Maybe, because it’s hard and unclear how to do so.

I would appreciate if the book had two versions of CLox implemented - one is a bare minimum by-the-book, the other one with all challenges completed. I wonder, if this would change the contents of the book, because when I did the challenges, a lot of later material became less clear, because the code differed a lot by that time. Anyhow, there’s the third “Challenge” wich would be interesting to see implemented officially:

Control flow constructs have been mostly unchanged since Algol 68. Language evolution since then has focused on making code more declarative and high level, so imperative control flow hasn’t gotten much attention.

For fun, try to invent a useful novel control flow feature for Lox. It can be a refinement of an existing form or something entirely new. In practice, it’s hard to come up with something useful enough at this low expressiveness level to outweigh the cost of forcing a user to learn an unfamiliar notation and behavior, but it’s a good chance to practice your design skills.

What’s the point of this “Challenge”? I don’t know - maybe you would came up with something like COMEFROM, or maybe you’ll be able to create something that will change computing forever. I don’t want to waste time on pretty much useless stuff though. If it’s meant to be a joke, then OK, but I didn’t get it.

Calls and Functions

Finally, some real action. Defining functions is pretty straightforward, and I don’t really have anything to say about this chapter…

EXCEPT:

We’re not totally done, though. The new return statement gives us a new compile error to worry about. Returns are useful for returning from functions but the top level of a Lox program is imperative code too. You shouldn’t be able to return from there.
return "What?!";
We’ve specified that it’s a compile error to have a return statement outside of any function, which we implement like so:
static void returnStatement() {

compiler.c
in returnStatement()

  if (current->type == TYPE_SCRIPT) {
    error("Can't return from top-level code.");
  }

  if (match(TOKEN_SEMICOLON)) {
This is one of the reasons we added that FunctionType enum to the compiler.

Why? I mean, even the book kinda mentions that it can be useful:

Allowing return at the top level isn’t the worst idea in the world. It would give you a natural way to terminate a script early. You could maybe even use a returned number to indicate the process’s exit code.

The author seems to be familiar with Lua, so I don’t really get why they discard top-level = return statements. In my implementation, I left this check out. Especially because it helps with making testing possible again.

As for performance… Well, the book mentions that the current by-the-book implementation of CLox is faster than Python at computing the factorial of a number:

> fun fib(n) { if (n < 2) return n; return fib(n - 2) + fib(n - 1); }
> var start = clock(); print fib(35); print clock() - start;
9.22746e+06
1.35232

I’ve checked this Python implementation:

Python 3.11.5 (main, Aug 28 2023, 00:00:00) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def fib(n):
      if (n < 2):
        return n;
      return fib(n - 2) + fib(n - 1)
... ... ... ...
>>> import time
>>> start = time.time(); print(fib(35)); print(time.time() - start)
9227465
1.7716176509857178

It is certainly slower than CLox, what about my implementation in Zig?

./target/zig/bin/lox-0.1.73
ziglox> fun fib(n) { if (n < 2) return n; return fib(n - 2) + fib(n - 1); }
nil
ziglox> var start = clock(); print fib(35); print clock() - start;
9227465
28.784000158309937
nil

Almost 30 seconds?! I’m not even sure where the problem is, the speed difference is drastic. And I’m unsure how to diagnose this too, perhaps there’s a way to profile Zig code? Apparently, there’s this project, and there are bindings available, but I couldn’t make it build.

Wait… Zig does all these safety checks by default, right? And the default optimization target is “Debug”. Let’s try compiling with other optimization levels:

rm -rf ./target
zig build -Doptimize=ReleaseSafe -p ./target/zig --cache-dir ./target/zig/.cache
./target/zig/bin/lox-0.1.73
ziglox> fun fib(n) { if (n < 2) return n; return fib(n - 2) + fib(n - 1); }
nil
ziglox> var start = clock(); print fib(35); print clock() - start;
9227465
3.015000104904175
nil

OK, ReleaseSafe is ten times faster! Let’s try ReleaseFast then:

rm -rf ./target
zig build -Doptimize=ReleaseFast -p ./target/zig --cache-dir ./target/zig/.cache
./target/zig/bin/lox-0.1.73
ziglox> fun fib(n) { if (n < 2) return n; return fib(n - 2) + fib(n - 1); }
nil
ziglox> var start = clock(); print fib(35); print clock() - start;
9227465
1.7879998683929443
nil

Now we’re as fast as Python, yay… Let’s throw in Lua for comparison:

Lua 5.4.4  Copyright (C) 1994-2022 Lua.org, PUC-Rio
> function fib(n) if (n < 2) then return n end return fib(n - 2) + fib(n - 1) end
> socket = require"socket"
> time = socket.gettime
> start = time(); print(fib(35)); print(time() - start)
9227465
1.0116229057312

Well, Lua is often called one of the fastest scripting languages, so I guess that statement holds up well. A complete CLox implementation by the author of the book is also fast:

> fun fib(n) { if (n < 2) return n; return fib(n - 2) + fib(n - 1); }
> var start = clock(); print fib(35); print clock() - start;
9.22746e+06
1.3271

However, the way the result is printed isn’t particularly helpful.

Well, now, that is sorted out, I think we can conclude that the implementation is fast enough. I will keep the debug optimization target for now, as all these checks are helpful, but I guess I need a separate make rule for a release build.

There’s one “Challenge” in this chapter I’d like to tackle:

Add some more native functions to do things you find useful. Write some programs using those. What did you add? How do they affect the feel of the language and how practical it is?

Here are two functions that I would consider useful.

The first one is length:

fn lengthNative(n: u8, args: [*]Value, vm: *VM) Result {
    if (n > 0) {
        const s = args[0];
        if (s.isString())
            return .{ .OK = Value.from(s.asString().bytes.len) };
    }
    return vm.runtimeError("Expected a string as an argument.", .{});
}

Simply put, it’s handy to be able to know the length of the string. It’s not generic, because there are no other countable data structures in Lox.

As you may noticed, the return value of this function is not of a Value type but an Result one. That’s because we have to signal a runtime error if we get something unexpected as an argument. It plays nicely with the previous Challenge:

Right now, there’s no way for a native function to signal a runtime error. In a real implementation, this is something we’d need to support because native functions live in the statically typed world of C but are called from dynamically typed Lox land. If a user, say, tries to pass a string to sqrt(), that native function needs to report a runtime error.

Extend the native function system to support that. How does this capability affect the performance of native calls?

As for performance - we’re wrapping the result, and Zig is quite good at unwrapping these with the switch statement. Here’s how we handle native calls now:

The other one is tostring:

fn tostringNative(nargs: u8, args: [*]Value, vm: *VM) Result {
    if (nargs == 0)
        return vm.runtimeError("Expected an argument", .{});

    var array_list = std.ArrayList(u8).init(vm.allocator);
    defer array_list.deinit();

    args[0].format("{}", .{}, array_list.writer()) catch
        common.exit(200, "OOME: can't tostring a value", .{});

    const s = array_list.toOwnedSlice() catch
        common.exit(200, "OOME: can't tostring a value", .{});
    defer vm.allocator.free(s);

    return .{ .OK = (try String.copy(vm, s)).obj.asValue() };
}

Lox allows us to concatenate strings using the + operator. However, this is almost useless, because we can’t concatenate anything else to it. Formatting a message is pretty much impossible.

With tostring we can use + with anything Lox supports:

ziglox> fun countdown(n) {
          for (var i = n; i > 0; i = i - 1)
            print tostring(i) + "...";
          print "liftoff!!";
        }
nil
ziglox> countdown(3);
3...
2...
1...
liftoff!!

I would also consider adding sleep to the language, as this function is very useful when you need to wait, but don’t want to waste CPU. However, Lox is more of a toy language, so having sleep in it isn’t necessary. Anyway, here’s sleep for completeness:

fn sleepNative(nargs: u8, args: [*]Value, vm: *VM) Result {
    if (nargs == 0)
        return vm.runtimeError("Expected an argument", .{});

    const arg = args[0];
    if (!arg.is(.Number))
        return vm.runtimeError("Expected an integer argument", .{});

    const f = arg.asNumber();
    const i: i64 = @intFromFloat(f);

    if (f != @as(f64, @floatFromInt(i)))
        return vm.runtimeError("Expected an integer argument", .{});

    if (i > 0) sleep(@intCast(i * 1000000));
    return .{.OK = Nil };
}

It doesn’t have much utility, though, but having both the clock and sleep functions can work wonders.

Closures

This is getting out of hand - we’ve been adding new object types, and each one requires a method to test if it’s an instance of a particular object:

pub const Obj = struct {
    objType: Type,
    next: ?*Obj,

    // ...
    pub fn isString(self: *Obj) bool {
        return self.objType == .String;
    }

    pub fn asString(self: *Obj) *String {
        return @fieldParentPtr(String, "obj", self);
    }

    pub fn isFunction(self: *Obj) bool {
        return self.objType == .Function;
    }

    pub fn asFunction(self: *Obj) *Function {
        return @fieldParentPtr(Function, "obj", self);
    }

    pub fn isNativeFunction(self: *Obj) bool {
        return self.objType == .NativeFunction;
    }

    pub fn asNativeFunction(self: *Obj) *NativeFunction {
        return @fieldParentPtr(NativeFunction, "obj", self);
    }
};

Now we need to add one more for Closure, and then a bunch of others too. You can see that the only difference between these methods is the enumeration tag or type. So, instead of adding new ones, let’s delete all of these and use Zig’s comptime:

pub const Obj = struct {
    objType: Type,
    next: ?*Obj,

    // ...
    pub fn is(self: *Obj, objType: Type) bool {
        return self.objType == objType;
    }

    pub fn as(self: *Obj, comptime T: type) *T {
        return @fieldParentPtr(T, "obj", self);
    }
}

Now we only need to update all of our code from x.isFunction() to x.is(Function) which is a simple enough substitution. Sadly, we can’t do the same for our evergrowing Value struct:

pub const Value = union(enum) {
    Number: f64,
    Bool: bool,
    Nil,
    Obj: *Obj,

    pub inline fn isNumber(self: Value) bool {
        return self == .Number;
    }

    pub inline fn isBool(self: Value) bool {
        return self == .Bool;
    }

    pub inline fn isObj(self: Value) bool {
        return self == .Obj;
    }
    // ...
};

It may seem that we can do the same, except use the enumeration literal as the parameter to is:

    pub inline fn is(self: Value, comptime target: @TypeOf(.enum_literal)) bool {
        return self == target;
    }

However, we also have methods like this:

    pub inline fn isString(self: Value) bool {
        return self == .Obj and self.Obj.objType == .String;
    }

Unfortunately, here we can’t use this kind of signature. We could do:

    pub inline fn is(self: Value, comptime target: @TypeOf(.enum_literal), objType: ObjType) bool {
        if (comptime target == .Obj) {
            return self == target and self.Obj.objType == objType;
        } else {
            return self == target;
        }
    }

But then it would require us to write x.is(.Number, undefined) and y.is(.Obj, .String), which is ugly. So instead, we can do this comptime check:

    pub inline fn is(self: Value, comptime target: @TypeOf(.enum_literal)) bool {
        if (comptime @hasField(ObjType, @tagName(target))) {
            return self == .Obj and self.Obj.objType == target;
        } else {
            return self == target;
        }
    }

This way we can write both x.is(.Number) and y.is(.String) and it will generate an appropriate code at compile time. This is as close as we can get to macros, I guess. It gets harder from there, though.

Now we need to get rid of these methods:

pub const Value = union(enum) {
    Number: f64,
    Bool: bool,
    Nil,
    Obj: *Obj,
    // ...
    pub inline fn asNumber(self: Value) f64 {
        return self.Number;
    }

    pub inline fn asBool(self: Value) bool {
        return self.Bool;
    }

    pub inline fn asObj(self: Value) *Obj {
        return self.Obj;
    }

    pub inline fn asString(self: Value) *String {
        return self.Obj.as(String);
    }

    pub inline fn asFunction(self: Value) *Function {
        return self.Obj.as(Function);
    }

    pub inline fn asNativeFunction(self: Value) *NativeFunction {
        return self.Obj.as(NativeFunction);
    }
};

As you can see, we again have two kinds of actions to perform - one is to return the desired enumeration tag value, and the other is to call a method. The first few are easy to convert by using the same trick with comptime target: @TypeOf(.enum_literal) but we can’t do the same for the last few, because Obj.as expects a type. On the other hand, we know all of the types we’re gonna use with as, so we can do this:

pub const Value = union(enum) {
    Number: f64,
    Bool: bool,
    Nil,
    Obj: *Obj,
    // ...
    fn typeNameUnqualified(comptime T: type) []const u8 {
        const name = @typeName(T);
        return name[if (std.mem.lastIndexOfAny(u8, name, ".")) |i| i + 1 else 0..];
    }

    pub inline fn as(self: Value, comptime T: type) T {
        return if (comptime @hasField(ObjType, typeNameUnqualified(T)))
            self.Obj.as(switch (@typeInfo(T)) {
                .Pointer => |p| p.child,
                else => T,
            })
        else switch (T) {
            f64 => self.Number,
            bool => self.Bool,
            void => self.Nil,
            *Obj => self.Obj,
            else => unreachable,
        };
    }
};

Not pretty, eh? Well, that’s what you get, when you don’t have macros that operate on actual code.

We call this function for basic types like this:

a.as(f64); // Value{.Number=42} => 42

Transforming the Value into an actual number. And for object types, we do it like this:

// s is a `Value{.Obj=&{objType=.String,next=null}}`
s.as(*String); // => &String{.Obj={objType=.String,next=null},.hash=1337,.bytes="ваыв"}

Because we want a pointer to the String type we pass it as *String. Seems logical, but there’s a problem.

First, we need to check if the ObjType enum has the String field, but here we have a type. More than that, it is actually an *src.object.String type, not just String, so we need a way to transform this type into a tag name. The typeNameUnqualified function takes care of it, but then there’s another problem - we need to pass the correct type to the Object.as method, and *String is not a correct type - it needs a String! So we have to do a little dance with @typeInfo and check that we got a Pointer type, that has a child field, containing the type we’re interested in. Then we can pass it to the as method of the object.

Phew! We’re almost done with the refactoring, the only thing left is to remove these methods:

pub const Value = union(enum) {
    Number: f64,
    Bool: bool,
    Nil,
    Obj: *Obj,

    // ..
    pub inline fn fromNumber(x: f64) Value {
        return Value{ .Number = x };
    }

    pub inline fn fromBool(x: bool) Value {
        return Value{ .Bool = x };
    }

    pub inline fn fromObj(x: *Obj) Value {
        return Value{ .Obj = x };
    }
};

Well, that’s easy:

pub inline fn from(x: anytype) Value {
    return switch (@TypeOf(x)) {
        f64, comptime_float => Value{ .Number = x },
        bool => Value{ .Bool = x },
        void => Nil,
        *Obj => Value{ .Obj = x },
        else => unreachable,
    };
}

Now we can do Value.from(12.3) and it will do it’s job smoothly.

Look at the before and after:

|-----------------------------Before------------------------------------+----------------------------------------After----------------------------------------|
|pub const Value = union(enum) {                                        | pub const Value = union(enum) {                                                     |
|    Number: f64,                                                       |     Number: f64,                                                                    |
|    Bool: bool,                                                        |     Bool: bool,                                                                     |
|    Nil,                                                               |     Nil,                                                                            |
|    Obj: *Obj,                                                         |     Obj: *Obj,                                                                      |
|    // ...                                                             |     // ...                                                                          |
|    pub inline fn isNumber(self: Value) bool {                         |     pub inline fn is(self: Value, comptime target: @TypeOf(.enum_literal)) bool {   |
|        return self == .Number;                                        |         if (comptime @hasField(ObjType, @tagName(target))) {                        |
|    }                                                                  |             return self == .Obj and self.Obj.objType == target;                     |
|                                                                       |         } else {                                                                    |
|    pub inline fn fromNumber(x: f64) Value {                           |             return self == target;                                                  |
|        return Value{ .Number = x };                                   |         }                                                                           |
|    }                                                                  |     }                                                                               |
|                                                                       |                                                                                     |
|    pub inline fn asNumber(self: Value) f64 {                          |     fn typeNameUnqualified(comptime T: type) []const u8 {                           |
|        return self.Number;                                            |         const name = @typeName(T);                                                  |
|    }                                                                  |         return name[if (std.mem.lastIndexOfAny(u8, name, ".")) |i| i + 1 else 0..]; |
|                                                                       |     }                                                                               |
|    pub inline fn isBool(self: Value) bool {                           |                                                                                     |
|        return self == .Bool;                                          |     pub inline fn as(self: Value, comptime T: type) T {                             |
|    }                                                                  |         return if (comptime @hasField(ObjType, typeNameUnqualified(T)))             |
|                                                                       |             self.Obj.as(switch (@typeInfo(T)) {                                     |
|    pub inline fn fromBool(x: bool) Value {                            |                 .Pointer => |p| p.child,                                            |
|        return Value{ .Bool = x };                                     |                 else => T,                                                          |
|    }                                                                  |             })                                                                      |
|                                                                       |         else switch (T) {                                                           |
|    pub inline fn asBool(self: Value) bool {                           |             f64 => self.Number,                                                     |
|        return self.Bool;                                              |             bool => self.Bool,                                                      |
|    }                                                                  |             void => self.Nil,                                                       |
|                                                                       |             *Obj => self.Obj,                                                       |
|    pub inline fn isObj(self: Value) bool {                            |             else => unreachable,                                                    |
|        return self == .Obj;                                           |         };                                                                          |
|    }                                                                  |     }                                                                               |
|                                                                       |                                                                                     |
|    pub inline fn fromObj(x: *Obj) Value {                             |     pub inline fn from(x: anytype) Value {                                          |
|        return Value{ .Obj = x };                                      |         return switch (@TypeOf(x)) {                                                |
|    }                                                                  |             f64, comptime_float => Value{ .Number = x },                            |
|                                                                       |             bool => Value{ .Bool = x },                                             |
|    pub inline fn asObj(self: Value) *Obj {                            |             void => Nil,                                                            |
|        return self.Obj;                                               |             *Obj => Value{ .Obj = x },                                              |
|    }                                                                  |             else => unreachable,                                                    |
|                                                                       |         };                                                                          |
|    pub inline fn isString(self: Value) bool {                         |     }                                                                               |
|        return self == .Obj and self.Obj.objType == .String;           | };                                                                                  |
|    }                                                                  |                                                                                     |
|                                                                       |                                                                                     |
|    pub inline fn asString(self: Value) *String {                      |                                                                                     |
|        return self.Obj.asString();                                    |                                                                                     |
|    }                                                                  |                                                                                     |
|                                                                       |                                                                                     |
|    pub inline fn isFunction(self: Value) bool {                       |                                                                                     |
|        return self == .Obj and self.Obj.objType == .Function;         |                                                                                     |
|    }                                                                  |                                                                                     |
|                                                                       |                                                                                     |
|    pub inline fn asFunction(self: Value) *Function {                  |                                                                                     |
|        return self.Obj.asFunction();                                  |                                                                                     |
|    }                                                                  |                                                                                     |
|                                                                       |                                                                                     |
|    pub inline fn isNativeFunction(self: Value) bool {                 |                                                                                     |
|        return self == .Obj and self.Obj.objType == .NativeFunction;   |                                                                                     |
|    }                                                                  |                                                                                     |
|                                                                       |                                                                                     |
|    pub inline fn asNativeFunction(self: Value) *NativeFunction {      |                                                                                     |
|        return self.Obj.asNativeFunction();                            |                                                                                     |
|    }                                                                  |                                                                                     |
|};                                                                     |                                                                                     |
|-----------------------------------------------------------------------+-------------------------------------------------------------------------------------|

I guess, you can do many things that are done with macros in C using the comptime in Zig, although mostly when it is related to types. Can’t say that I like this more than macros, but it gets the job done. Now we can start this chapter proper!

In the last chapter, we added functions and had them working, but you can do with just functions only so much. In this chapter, we’re adding closures, which take our functions to a whole other level.

Honestly, this chapter, while having mostly straightforward stuff, gave me a lot of trouble. You see, there’s a certain problem with how the book gives you the material. It’s subtle, but when the time comes it’s hard to ignore.

The book gives you information bit by bit while trying to have a working interpreter at all times. Yes, each chapter breaks the interpreter in the beginning, but by the end, you have something that works. And often the chapter progressively gives you more examples and explains why they should work and why they don’t right now. On its own, it’s a great way to show you your progress, but on the other hand, if a small mistake sneaks somewhere in the process, you won’t know till much later. That’s exactly what happened to me.

So, during this chapter, I tried most examples the book featured, and all of them seemed to work. Satisfied, I proceeded to the “Challenges” section, and because I despise OOP much more than said “Challenges”, I decided to do this particular one:

A famous koan teaches us that “objects are a poor man’s closure” (and vice versa). Our VM doesn’t support objects yet, but now that we have closures we can approximate them. Using closures, write a Lox program that models two-dimensional vector “objects”. It should:

Define a “constructor” function to create a new vector with the given x and y coordinates.

Provide “methods” to access the x and y coordinates of values returned from that constructor.

Define an addition “method” that adds two vectors and produces a third.

And it threw a weird error - it couldn’t find some closures.

Now, because Lox doesn’t have any data structures except for strings yet, I decided to implement a linked list, as closures are a perfect solution for that. You don’t need any kind of data structures for linked lists as long as your language has closures. Observe:

fun cons(head, tail) {
    fun cell(isHead) {
        if (isHead)
            return head;
        return tail;
    }
    return cell;
}

fun first(cell) {
    return cell(true);
}

fun rest(cell) {
    return cell(false);
}

With these three functions, we should be able to create a list like this:

var list = cons(1, cons(2, cons(3, nil)));

And we should be able to traverse it with for loop, albeit in a bit ugly way:

for (var head = list; head != nil; head = rest(head)) {
	print first(head);
}

Now, if we try to execute this script, we get

$ ziglox list.lox
Undefined variable 'head'.
[line 1] in cell
[line 1] in first
[line 1] in script
Interpret Runtime Error

An error. I then made sure that the head variable is not related to the loop, by trying to get the head of the list with just first(list), but still got the same error. Then I spent a day looking for what I did wrong.

During this, I decided to try out all of the examples in the chapter. I remember that all of them have worked before, and if they are then the problem is much deeper than the book accounts for. So I started, and everything worked until I hit this one:

fun outer() {
  var x = "value";
  fun middle() {
    fun inner() {
      print x;
    }

    print "create inner closure";
    return inner;
  }

  print "return from outer";
  return middle;
}

var mid = outer();
var in = mid();
in();

Strangely enough, all later examples worked fine. I’ll explain why in a moment. For now, let it sink in - I had a working interpreter the whole time, and it could do everything the book suggested I try, yet there was a bug somewhere.

So, here’s the thing. For some reason, I decided to use a statically allocated stack for locals, back in the chapter where we added support for locals. It worked well until we were introduced to the concept of compiler stacks. However, Since I created the stack way back, I kept using it, and it basically meant that every compiler shared the same stack of locals. So when we were searching for upvalues we saw the same upvalues on all levels.

So you should now understand why it worked for all of the examples, except for the one above - it has one more compiler level. After fixing this by using a proper dynamic array, all of the examples in this chapter work as expected.

The linked list, however, still doesn’t work. This time around, I decided that I would automate this process a bit, and put all of the examples as test cases. I had to adjust them to actually return values instead of printing them, and what do you know - I found another broken case:

fun outer() {
  var a = 1;
  var b = 2;
  fun middle() {
    var c = 3;
    var d = 4;
    fun inner() {
      return a + c + b + d;
    }
	return inner;
  }
  return middle;
}

return outer()()();

It should return 10, but instead it returns 12. How? Well, if we print all of the values that went into the inner function we’ll see that a is 3, b is 2, c is 2, and d is 4 and the answer changes to 14. So while only the d upvalue was correct, the introduction of print still changed something.

There’s another bug. And the same happens in my linked list implementation - first returns itself. Perhaps something to do with how we capture upvalues then?

During these days of debugging, I tried everything I could - I moved readByte to CallFrame struct, I changed the instruction counter to the instruction pointer, as the book does, I also tried allocating stacks on the heap instead of the static memory. I poked the reference implementation by Robert - the bytecode is exactly the same as in my implementation, so it has to be something in the VM. I printed pointer addresses, analyzing if there is a different between my implementation and C, and comparing relative offsets - again, they’re the same. Nothing helped. Reverting to my previous implementation revealed that this case wasn’t handled properly as well - I somehow missed it back then too.

And then I found it. After 4 days of debugging the reason was that I SOMEHOW missed this line in the captureUpvalue function:

createdUpvalue.next = upvalue;

Finally, everything works:

// cons, first, rest...

var list = cons(1, cons(2, cons(3, nil)));

for (var head = list; head != nil; head = rest(head)) {
    print first(head);
}

1
2
3

And, guess what? During all this debugging, and trying different things I managed to improve the speed of the interpreter a bit:

ziglox> fun fib(n) { if (n < 2) return n; return fib(n - 2) + fib(n - 1); }
nil
ziglox> var s = clock(); print fib(35); print clock() - s;
9227465
1.4010000228881836

Here’s the same code when checked out to the previous commit at chapter 24:

ziglox> fun fib(n) { if (n < 2) return n; return fib(n - 2) + fib(n - 1); }
nil
ziglox> var s = clock(); print fib(35); print clock() - s;
9227465
1.815000057220459

Nice! Going back to the third “Challenge”, here’s my attempt at an object via closures:

fun vec2(x, y) {
  var _x = x; // storage
  var _y = y;
  fun obtainer(what) {
    if (what == "x") return _x;
    else if (what == "y") return _y;
  }
  fun establisher(what, val) {
    if (what == "x") _x = val;
    else if (what == "y") _y = val;
  }
  fun add(other) {
    var ox = other("obtain")("x");
    var oy = other("obtain")("y");
    _x = _x + ox;
    _y = _y + oy;
  }
  fun clss (action) {
    if (action == "obtain") return obtainer;
    else if (action == "establish") return establisher;
    else if (action == "add") return add;
  }
  return clss;
}

var a = vec2(3, nil);
var b = vec2(22, 10);

a("establish")("y", 7);
a("add")(b);
print a("obtain")("x") + a("obtain")("y"); // 42

It’s a vec2 structure representing this kind of data {.x = 1, .y = 2}. Although it’s a bit rough to use, it is a proper data structure. And yes - I like obtain and establish better than get and set. What?

Garbage Collection

Hooo…

Now we get to the interesting stuff. The book implements this language in C, and C is pretty straightforward when it comes to memory management. Well, straightforward might not be a proper word, but by default it’s quite basic - you have malloc, free, and realloc as a base.

Zig, on the other hand, doesn’t have a default allocator, which, in theory, should make it more appropriate for embedded systems, where there may not be any way to actually allocate memory. This, in turn, affects the language design - every Zig function that wants to allocate some memory, accepts an allocator (or accesses it from an enclosing struct if that’s a method). Now you can understand why all of the code in this post had something like Allocator in function arguments or structures.

This meant that we didn’t have to implement most of the book’s memory.c and memory.h because Zig’s allocators already can do all of that. However, now we can’t just use an existing allocator, as we need to implement garbage collection mid-allocation. That’s why we need a custom allocator.

Thankfully, we don’t really have to do the allocation ourselves. Instead, we can create an allocator that just wraps another allocator, like a decorator. All we need to do is to implement these three functions:

pub const GCAllocator = struct {
    parentAllocator: Allocator,
    bytesAllocated: usize,

    pub fn init(parentAllocator: Allocator) GCAllocator {
        return .{
            .parentAllocator = parentAllocator,
            .bytesAllocated = 0,
        };
    }

    pub fn allocator(self: *GCAllocator) Allocator {
        return .{
            .ptr = self,
            .vtable = &.{ .alloc = alloc, .resize = resize, .free = free },
        };
    }

    fn alloc(ctx: *anyopaque, len: usize, log2_ptr_align: u8, ra: usize) ?[*]u8 {
        const self: *GCAllocator = @ptrCast(@alignCast(ctx));
        const result = self.parentAllocator.rawAlloc(len, log2_ptr_align, ra);
        if (result != null) self.bytesAllocated += len;
        return result;
    }

    fn resize(ctx: *anyopaque, buf: []u8, log2_buf_align: u8, new_len: usize, ra: usize) bool {
        const self: *GCAllocator = @ptrCast(@alignCast(ctx));
        if (self.parentAllocator.rawResize(buf, log2_buf_align, new_len, ra)) {
            if (new_len <= buf.len) {
                self.bytesAllocated -= buf.len - new_len;
            } else {
                self.bytesAllocated += new_len - buf.len;
            }

            return true;
        }
        std.debug.assert(new_len > buf.len);
        return false;
    }

    fn free(ctx: *anyopaque, buf: []u8, log2_buf_align: u8, ra: usize) void {
        const self: *GCAllocator = @ptrCast(@alignCast(ctx));
        self.parentAllocator.rawFree(buf, log2_buf_align, ra);
        self.bytesAllocated -= buf.len;
    }
};

Now we should be able to implement the rest of this chapter in terms of this allocator. And, thanks to the fact that all our functions already accept an allocator, we can just plug this one in, and it should work!

I called it GCAllocator, although right now it is more like a tracking allocator. It tracks how many bytes were allocated and freed, and that’s basically it. With that, we can implement the rest of the interface.

The book suggests to implement the “Mark and Sweep” algorithm. While not great, it is perfect for teaching. Moreover, there’s still software that uses this technique, for example, one that I’m writing this blog in - Emacs! It was always an active debate whether the GC in Emacs is outdated or not, but I can say that Emacs performs fine for my daily tasks, so it’s probably OK. Something like JVM would never settle on something simple like this though.

I won’t go into the source code of the collector itself, as the translation from C is pretty much 1:1 here. One thing to point out though is the use of grayStack in the VM. The book uses an array of pointers to the objects, and we do the same by using ArrayList(*Obj). However, using append to add things to this array can throw an out-of-memory error. So when we do the marking of objects we have to handle that.

Between these two chapters, I’ve refactored the code to use Zig’s errors. So handling allocation failures is simple enough now, but I don’t really know if we should handle out-of-memory errors. On one hand, JVM doesn’t recommend handling the OOME, and the most common practice I know of is to do a heap dump and die. On the other hand, we’re dealing with OOME inside the virtual machine, not inside the language itself, as Lox doesn’t really have exceptions or other kinds of error handling. I chose to kill the VM if the OOME ever occurs, but in a real language, I would probably not do it this way. And I think in the language that runs on the VM it should be possible to handle OOM too.

Classes and Instances

Now, I’m a bit torn apart here. I don’t want to implement classes.

I mean, seriously, this language doesn’t have any data structures yet, and we’re already in classes? Where are arrays, at the very least? Plus, we already did classes in cljlox, so it’s boring to do it again.

Instead, I think it would be better to go the Lua route and give users only a basic set of data structures, like hash tables. Except, actually give users data structures, and not a mixed table/array thingy. I’m not sure, however, if it will affect the last chapter that talks about optimizations.

Yet, there are benefits in the following chapters. For example, if I were to introduce a hash table, accessing keys from it can be made with a special syntax like a.b. Assigning a value to the key can also use this syntax, and we’re going to handle it in the next chapter.

I guess, classes are more involved, and thus a bit better fit for teaching how to make a language, but on the other hand, class systems can be a fully runtime thing. Sparkle some macros on top of it, and you can hack your own object system in a weekend. That’s what a lot of Lisp/Scheme projects did back in the day.

Anyway, after adding the Class opcode and some other stuff around it the book says:

There you have it, our VM supports classes now. You can run this:
class Brioche {}
print Brioche;
Unfortunately, printing is about all you can do with classes, so next is making them more useful.

Unfortunately, we have classes before arrays.

Yes, I’m negative, because come on, arrays are such a basic data structure that not having them is a bummer. How would you implement anything without an array? Well, technically, we can use our closure-based linked lists as a substitute, but it has horrible performance in many algorithms where constant indexing is required.

Anyway, after implementing field access the chapter is suddenly over. That’s it? Well, right now it’s just a hash table with inconvenient syntax. I mean, yes, classes are just a special case of a hash table in some languages, and hash tables are a special case of classes in some other broken languages. But here, why didn’t we introduce a basic hash table with an open set of keys, even though our runtime already supports one?

Onto “Challenges”.

Trying to access a non-existent field on an object immediately aborts the entire VM. The user has no way to recover from this runtime error, nor is there any way to see if a field exists before trying to access it. It’s up to the user to ensure on their own that only valid fields are read.

How do other dynamically typed languages handle missing fields? What do you think Lox should do? Implement your solution.

Yeah, that was a strange decision. I decided to just return nil.

Fields are accessed at runtime by their string name. But that name must always appear directly in the source code as an identifier token. A user program cannot imperatively build a string value and then use that as the name of a field. Do you think they should be able to? Devise a language feature that enables that and implement it.

Well, first of all, a user program cannot imperatively build a string value at all, because Lox has no string library. A few chapters ago I added tostring and with that, it is possible to build strings using +, but not in vanilla Lox.

A user should be able to do that, but most languages that allow that it does require a special syntax, and I don’t want to deal with that now. If I did, though, I would probably introduce something like foo[bar], but instead we can always use a native function:

    fn getNative(nargs: u8, args: [*]Value, vm: *VM) !Value {
        if (nargs == 0)
            return vm.runtimeError("Expected an argument", .{});
        const instance = args[0];
        if (!instance.is(.Instance))
            return vm.runtimeError("Expected a class instance as the first argument", .{});
        const name = args[1];
        if (!name.is(.String))
            return vm.runtimeError("Expected a string as the second argument", .{});

        return instance.as(*Instance).fields.get(name.as(*String)) orelse Nil;
    }

    fn setNative(nargs: u8, args: [*]Value, vm: *VM) !Value {
        if (nargs == 0)
            return vm.runtimeError("Expected an argument", .{});

        const instance = args[0];
        if (!instance.is(.Instance))
            return vm.runtimeError("Expected a class instance as the first argument", .{});

        const name = args[1];
        if (!name.is(.String))
            return vm.runtimeError("Expected a string as the second argument", .{});

        if (nargs >= 3 and !args[2].is(.Nil)) {
            const value = args[2];
            _ = try instance.as(*Instance).fields.set(name.as(*String), value);
            return value;
        } else {
            _ = instance.as(*Instance).fields.delete(name.as(*String));
            return Nil;
        }
    }

With these we can do set(foo, "vaiv", 42) and get(foo, "vaiv"). These native functions are bloating the VM struct, so I should probably move them out.

As you can see, I don’t care if the native function has more arguments than it really needs. The only thing I care about is the required arguments. I like this design more, and in fact, doing things this way should allow variadic functions naturally. Maybe I should implement them in the future, alongside arrays.

Lastly:

Conversely, Lox offers no way to remove a field from an instance. You can set a field’s value to nil, but the entry in the hash table is still there. How do other languages handle this? Choose and implement a strategy for Lox.

But as you may have noticed, I already implemented this. Setting a field to nil removes it from the hash table. This is how it works in Lua, and while I strongly dislike this particular part of Lua, I don’t have any problems with this in Lox. Why? Because Lua has a weird mix of tables and arrays, setting a value in the array to nil also removes it from the array, as if the array were a hash table with integer keys.

So by doing arr[3] = nil you effectively transform {"a", "b", "c", "d"} into {1 = "a", 2 = "b", 4 = "d"}, or rather {"a", "b", 4 = "d"}, as Lua views it. This breaks iteration on arrays, as no the iteration will only see the first two fields (and yes, arrays in Lua start from 1, which is great):

arr = {"a", "b", "c", "d"}
print "-- before set --"
for i,v in ipairs(arr) do
   print(i, v)
end
arr[3] = nil;
print "-- after set --"
for i,v in ipairs(arr) do
   print(i, v)
end

-- before set --
1	a
2	b
3	c
4	d
-- after set --
1	a
2	b

If Lua did not remove things when you set them to nil and instead provided a contains and remove functions to check for key presence, and removal of combined array/table would be fine, but in its current state, it’s too easy to trip on this. I know that because I teach people how to write Lua, and this always makes people angry.

If the language provides separate array and table objects, setting table keys to nil for removal is fine - it should not matter if the key is present and nil or just absent. Arrays, however, should be able to have nil in them, and removal is a whole different operation, that shifts all other elements, so setting array elements to nil for removal doesn’t really make sense. Anyway, that’s my reasoning behind this, I’m sure you can disagree, but it’s OK.

Finally:

Because fields are accessed by name at runtime, working with instance state is slow. It’s technically a constant-time operation—thanks, hash tables—but the constant factors are relatively large. This is a major component of why dynamic languages are slower than statically typed ones.

How do sophisticated implementations of dynamically typed languages cope with and optimize this?

Ah, if only we had arrays - we could easily implement records that have constant time access to fields by field index determined at compile time. Alas.

Methods and Initializers

I don’t have anything to say about this chapter. Mostly, because it is straightforward and concise.

Still, I would prefer not having methods at all, but there were interesting bits to tackle with this, so I guess it was worth it.

Superclasses

// ...return constantInstruction("OP_CLASS", chunk, offset);
    case OP_INHERIT:
      return simpleInstruction("OP_INHERIT", offset);
//  case OP_METHOD: ...

Simple instruction, how ironic.

The chapter proceeds adding tons of new stuff in the compiler, and then in the VM. While straightforward, I can’t help but notice how things got more complicated since we added classes. Honestly, I have never seen a program where inheritance made things easier.

A tenet of object-oriented programming is that a class should ensure new objects are in a valid state. In Lox, that means defining an initializer that populates the instance’s fields. Inheritance complicates invariants because …

If Lox was your language, how would you address this, if at all? If you would change the language, implement your change.

If Lox was my language it wouldn’t have classes at all. And it would have arrays.

Optimization

The final chapter. Honestly, didn’t think that I’d make it through. It’s 4 AM right now, as I’m writing this, and perhaps I should go to sleep, but I want to finish this stuff quicker, so…

The chapter starts with a benchmark:

class Zoo {
  init() {
    this.aardvark = 1;
    this.baboon   = 1;
    this.cat      = 1;
    this.donkey   = 1;
    this.elephant = 1;
    this.fox      = 1;
  }
  ant()    { return this.aardvark; }
  banana() { return this.baboon; }
  tuna()   { return this.cat; }
  hay()    { return this.donkey; }
  grass()  { return this.elephant; }
  mouse()  { return this.fox; }
}

var zoo = Zoo();
var sum = 0;
var start = clock();
while (sum < 100000000) {
  sum = sum + zoo.ant()
            + zoo.banana()
            + zoo.tuna()
            + zoo.hay()
            + zoo.grass()
            + zoo.mouse();
}

print clock() - start;
print sum;

Running this on the reference implementation takes 3.87092 seconds and the result is 1e+08 or 100000000. Running on the release build of my language takes 8.065999984741211, and the result is 100000002. Don’t mind the difference, it’s an IEEE 754 quirk - if we change how clox prints it’s value from scientific notation to plain float, we’ll get the same thing.

Now, my implementation is twice as slow. Thankfully, the loop itself is fast - if we just do this:

ziglox> var sum = 0;
ziglox> var start = clock(); while (sum < 100000000) { sum = sum + 1 + 1 + 1 + 1 + 1 + 1; } print clock() - start;
2.2915e+01
nil
ziglox>  print sum;
1.00000002e+08
nil

The speed is comparable.

Unfortunately, I am lazy. The book proceeds at optimizing table lookups, by using bit-masking instead of the modulo operator. I’m not using the table implementation from the book itself - instead, I’m wrapping Zig’s std.hash_map.HashMap. Fortunately, Zig’s HashMap already uses bit-masking:

/// Find the index containing the data for the given key.
/// Whether this function returns null is almost always
/// branched on after this function returns, and this function
/// returns null/not null from separate code paths.  We
/// want the optimizer to remove that branch and instead directly
/// fuse the basic blocks after the branch to the basic blocks
/// from this function.  To encourage that, this function is
/// marked as inline.
inline fn getIndex(self: Self, key: anytype, ctx: anytype) ?usize {
    comptime verifyContext(@TypeOf(ctx), @TypeOf(key), K, Hash, false);

    if (self.size == 0) {
        return null;
    }

    // If you get a compile error on this line, it means that your generic hash
    // function is invalid for these parameters.
    const hash = ctx.hash(key);
    // verifyContext can't verify the return type of generic hash functions,
    // so we need to double-check it here.
    if (@TypeOf(hash) != Hash) {
        @compileError("Context " ++ @typeName(@TypeOf(ctx)) ++ " has a generic hash function that returns the wrong type! " ++ @typeName(Hash) ++ " was expected, but found " ++ @typeName(@TypeOf(hash)));
    }
    const mask = self.capacity() - 1;
    const fingerprint = Metadata.takeFingerprint(hash);
    // Don't loop indefinitely when there are no empty slots.
    var limit = self.capacity();
    var idx = @as(usize, @truncate(hash & mask));

    var metadata = self.metadata.? + idx;
    while (!metadata[0].isFree() and limit != 0) {
        if (metadata[0].isUsed() and metadata[0].fingerprint == fingerprint) {
            const test_key = &self.keys()[idx];
            // If you get a compile error on this line, it means that your generic eql
            // function is invalid for these parameters.
            const eql = ctx.eql(key, test_key.*);
            // verifyContext can't verify the return type of generic eql functions,
            // so we need to double-check it here.
            if (@TypeOf(eql) != bool) {
                @compileError("Context " ++ @typeName(@TypeOf(ctx)) ++ " has a generic eql function that returns the wrong type! bool was expected, but found " ++ @typeName(@TypeOf(eql)));
            }
            if (eql) {
                return idx;
            }
        }

        limit -= 1;
        idx = (idx + 1) & mask;
        metadata = self.metadata.? + idx;
    }

    return null;
}

Assuming that it is as fast, let’s look at another optimization: NaN Boxing.

Now, this is something completely new to me - I’m not sure if I ever heard of it. Anyway, now we need to re-implement the Value struct:

pub const NanBoxedValue = packed struct {
    data: u64,

    const SIGN_BIT: u64 = 0x8000000000000000;
    const QNAN: u64 = 0x7ffc000000000000;

    pub const Nil = NanBoxedValue{ .data = QNAN | 1 };
    pub const True = NanBoxedValue{ .data = QNAN | 3 };
    pub const False = NanBoxedValue{ .data = QNAN | 2 };
};

We start with defining some constants. Previously, Nil, True, and False were top-level constants, but I’ve moved them into the struct, so then we could get the correct ones via a compile-time alias. I did the same with the UnionValue struct, previously named Value.

Next, we need to reimplement the is, as, and from methods. I’m glad that I made them generic:

pub const NanBoxedValue = packed struct {
    // ...

    pub inline fn is(self: NanBoxedValue, comptime target: @TypeOf(.enum_literal)) bool {
        if (comptime @hasField(ObjType, @tagName(target))) {
            if (self.is(.Obj)) {
                const o = @as(*Obj, @ptrFromInt(@as(usize, @intCast(self.data & ~(SIGN_BIT | QNAN)))));
                return o.objType == target;
            } else return false;
        } else {
            return switch (target) {
                .Bool => (self.data & False.data) == False.data,
                .Nil => self.data == Nil.data,
                .Number => (self.data & QNAN) != QNAN,
                .Obj => (self.data & (QNAN | SIGN_BIT)) == (QNAN | SIGN_BIT),
                else => unreachable,
            };
        }
    }

    pub inline fn as(self: NanBoxedValue, comptime T: type) T {
        return if (comptime @hasField(ObjType, typeNameUnqualified(T)))
            @as(*Obj, @ptrFromInt(@as(usize, @intCast(self.data & ~(SIGN_BIT | QNAN))))).as(switch (@typeInfo(T)) {
                .Pointer => |p| p.child,
                else => T,
            })
        else switch (T) {
            f64 => @as(f64, @bitCast(self.data)),
            bool => self.data == True.data,
            void => Nil.data,
            *Obj => @as(*Obj, @ptrFromInt(@as(usize, @intCast(self.data & ~(SIGN_BIT | QNAN))))),
            else => unreachable,
        };
    }

    pub inline fn from(x: anytype) NanBoxedValue {
        return switch (@TypeOf(x)) {
            i32, comptime_int => NanBoxedValue{ .data = @as(u64, @bitCast(@as(f64, @floatFromInt(x)))) },
            f64, comptime_float => NanBoxedValue{ .data = @as(u64, @bitCast(@as(f64, x))) },
            bool => if (x) True else False,
            void => Nil,
            *Obj => NanBoxedValue{ .data = SIGN_BIT | QNAN | @intFromPtr(x) },
            else => unreachable,
        };
    }
};

Lastly, we need the remaining isFalse, format, equal, and mark:

pub const NanBoxedValue = packed struct {
    // ...

    pub fn isFalsey(self: NanBoxedValue) bool {
        if (self.is(.Bool)) return !self.as(bool);
        if (self.is(.Nil)) return true;
        return false;
    }

    pub fn equal(self: NanBoxedValue, other: NanBoxedValue) bool {
        if (self.is(.Number) and other.is(.Number)) return self.as(f64) == other.as(f64);
        return self.data == other.data;
    }

    pub fn mark(self: NanBoxedValue, vm: *VM) !void {
        if (self.is(.Obj))
            try self.as(*Obj).mark(vm);
    }

    pub fn format(
        self: NanBoxedValue,
        comptime fmt: []const u8,
        options: std.fmt.FormatOptions,
        out_stream: anytype,
    ) !void {
        _ = fmt;
        _ = options;
        if (self.is(.Number)) {
            try out_stream.print("{d}", .{self.as(f64)});
        } else if (self.is(.Bool)) {
            try out_stream.print("{}", .{self.as(bool)});
        } else if (self.is(.Nil)) {
            try out_stream.print("nil", .{});
        } else {
            try self.as(*Obj).print(out_stream);
        }
    }
};

Notice that we’re no longer using a constant dispatch on the enum tag - instead, it’s a chain of if else. Well, that is a small price to pay, as we rarely print things, compared to overall execution.

After implementing this type of Value struct, we can tell the rest of the implementation to use it like this:

pub const Value = if (NAN_BOXING) NanBoxedValue else UnionValue;

And the rest of the code, hopefully, works. I had to make some small adjustments though.

Now, running the benchmark shows takes 5.4649999141693115 seconds. A great improvement!

Thoughts on the book

The book is great. The writing style is engaging, I really liked the diagrams, and the material is properly structured. Would recommend. This is my second read, mind you.

Perhaps, I was too harsh at times, but I just was frustrated with some things and let it out.

Now, I do think that the lack of arrays, and plain hash tables is a bit of a bummer. I get it - it is a learning exercise, but, it would be great to have just a bit more from actual languages. Also, it would be really interesting to look at JIT and FFI, but I guess it deserves its own book already.

All in all, I’m glad that I read this book. It allowed me to tighten my Clojure skills in the past, and learn an awesome language today - Zig.

Thoughts on Zig

I really like Zig.

At times I thought to myself - I am enjoying this way too much. As you may know, I was a C programmer before, and I did some pretty low-level stuff in it. I got tired of C and decided that I didn’t want to write in these kinds of languages. Rust gave me hope, but it fell apart before my eyes.

Zig, however, is a C I always wanted. With a type system that actually helps reading and writing code, and not intended to do some convoluted nonsense. I’ve written all of the code without an LSP, and it was a breeze. The only tooling I used, besides the compiler, was Emacs tags. Generating tags isn’t supported, and their precision is limited, but it was enough for this project. If you’re interested, here’s how I generated the TAGS file.

First, I made these regex rules for etags:

/.*\<fn +\([a-zA-Z0-9_]+\) *(/\1/
/.*\<\(var\|const\) *\([a-zA-Z0-9_]+\) *= *\(extern\|packed\)? *\(struct\|enum\|union\)/\2/
/.*\<error +\([a-zA-Z0-9_]+\)/\1/

Code Snippet 1: zig.etags

And added this to Makefile:

ZIG_STD ?= ~/.zig/lib/std
ZIG_STD_SOURCES := $(shell find $(ZIG_STD) -type f -name '*.zig')
ZIG_SOURCES := $(shell find ./src/zig -type f -name '*.zig')

tags:
	etags --language=none --regex=@zig.etags $(ZIG_SOURCES) $(ZIG_STD_SOURCES)

It works well enough and allows jumping to the standard library. I don’t use automatic completion that much even in other languages, and error checking is fine when I compile code. Like old times.

Sometimes the errors are weird:

zig build -Doptimize=Debug -p ./target/zig --cache-dir ./target/zig/.cache
zig build-exe lox-0.1.77 Debug native: error: the following command failed with 1 compilation errors:
~/Downloads/zig-linux-x86_64-0.11.0/zig build-exe ~/Git/lox/src/zig/lox/main.zig --cache-dir ./target/zig/.cache --global-cache-dir ~/.cache/zig --name lox-0.1.77 --listen=-
Build Summary: 0/3 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
+- install lox-0.1.77 transitive failure
   +- zig build-exe lox-0.1.77 Debug native 1 errors
~/Downloads/zig-linux-x86_64-0.11.0/lib/std/fmt.zig:87:9: error: expected tuple or struct argument, found void

Like, OK, where exactly is the error though?

Here’s the source code that generates the message above. Can you spot the problem?

const std = @import("std");

pub fn main() !void {
    const stdout = std.io.getStdOut().writer();
    try stdout.print("Hello, world!\n", {});
}

Yup, I forgot the ..

There are other such errors, I just didn’t bother to save more.

Zig’s build system is something else. Can’t say I like it, but I don’t dislike it either. I guess I need to spend a bit more time with it to say for sure. The inability to set the cache and target directories from build.zig is a bummer though.

Also, the language name is, well, weird. The Crafting Interpreters book starts by explaining why the language is called Lox. In the design note they mention Nim:

It doesn’t have negative connotations across a number of cultures. This is hard to be on guard for, but it’s worth considering. The designer of Nimrod ended up renaming his language to “Nim” because too many people remember that Bugs Bunny used “Nimrod” as an insult. (Bugs was using it ironically.)

Well, the name “Zig”, in my country at least, sounds a lot like the beginning of a Nazi salute. And when translated and spelled it has literally the same letters. I’m sure it is not intentional, but I had a hard time talking about this language to basically anyone. Literally, everyone was like: “name’s weird”. So I started calling it ZigLang instead.

Apart from that, I really enjoy this language, and maybe will use it for other projects in the future.

Closing thought

Thank you for reading through this insanely enormous flow of consciousness. And if you did not read it fully, I appreciate that you still got to the end. Even if you just scrolled past everything.

This post was about two years in the making. I’m really glad that I’ve finally finished it, and I hope it is interesting and informative.

If you’re interested in the project’s source code, you can find it here: lox. And if you’re a seasoned Zig developer, I would appreciate it if you reviewed the code. I am by no means an expert in Zig, so I probably made a lot of stupid decisions.

Let me know what you think!

because even this didn’t work later and I just symlinked the source directory into tests, because of course ↩︎
Common Lisp is compiled to native code and loaded without requiring application restart. ↩︎
Just call them the homework. ↩︎
Technically, there’s more, but there are no sub-scopes like in other languages. ↩︎

Comment via email

2023 Recap

Sun, 31 Dec 2023 15:52:00 +0300

This year certainly was a productive one for me. I’ve written ~40 posts, have many more in the works, made a few new projects in Fennel and Clojure, and changed more of how I spend my time overall. The last year’s recap I mentioned that I’m no longer available on most social networks - this certainly helps me keep a more healthy mental state. I feel that I can keep my focus for longer, and overall my thoughts feel more structured than before. Perhaps, reading books also had something to do with that, but I want to think that leaving social networks helped with that too.

This year’s events

Here are events and things in general that I think were impactful to me during this year:

LLMs

This is the year we saw the rise of Large Language Models. It was fun playing with ChatGPT for a few hours, but nothing that ultimately caught my interest for long enough. Unfortunately, however, the world took way too much liking for LLMs, and now it polluted so many things that I started thinking about leaving not only the social networks but the internet completely. How often do we see a tech that can do that?

Let me explain what I mean, and why I think the net becomes less usable.

As I browsed the net, I started suspecting that a lot of articles and posts that were previously written by copywriters now is seemingly written by LLMs. A lot of reviews and comparison articles, of all things, are now written by LLMs. Automated mail that I receive is now written by LLMs. Even freakin’ art is now drawn by LLMs.

There were reports that people lost their jobs due to big companies realizing that a single LLM can replace a handful of people, and the quality of the result won’t change as much. Well, I mean, if such a company makes such a move, perhaps their products weren’t worth my time even before that, but still. LLMs are killing the net, and subsequently the software. For now, at least.

AI, in general, is a weird thing. It’s certainly in its infancy, but it is already capable of a lot of things. I just think that we’re applying it the wrong way. Certainly, an AI that can reason about hard-to-grasp problems is an interesting field, but what we’re largely doing with AI today is not that. Maybe there’s a need for another AI winter to better spend the resources we’re given?

Gamedev

I finally tipped my toes into the game dev field. It was fun, but exhausting, though that’s mostly on me - I had a bright idea to impose myself with a 6-month marathon. Looking back on it, the idea wasn’t half bad, I just needed more time. A resource I don’t have as much as I’d like to.

In the next year, I hope to do more game dev stuff, but I won’t probably do it this way. I have two project ideas I wanted to make for a long time. Yet, I know that the dream project shouldn’t be your first project, or else you would probably damage it without any way to repair it afterward. So we’ll see. I’m not sure that I really want to become a full-time game developer hobbyist yet.

Fennel

Fennel, again, was a big part of this year. I haven’t worked on the compiler at all though, I think the only thing I did was metadata-related stuff. However, I did a lot of projects in Fennel, the already-mentioned games, and some new libraries.

However, I noticed that I’m starting to get tired of Fennel - it’s a nice project and makes Lua more fun, but it has a lot of quirks that I simply learned to accept. After trying to teach a friend to program in Fennel I’ve realized how many things you can stumble on, I just don’t notice these anymore. Not because they’re harmless, just because I already know how exactly these things need to be written. Most of these are fennel-specific, actually, so I even tried writing in pure Lua for a kick, but didn’t like it as much. Although I still like Lua as a runtime.

I still need to reflect on this more, I think.

Music

I started working on a proper setup for music listening. I’m not yet ready to buy all these expensive amplifiers, turntables, and stuff, but I finally got myself a great pair of open-back headphones for critical listening. Now I want to build my own music library, and probably leave music streaming services. Or at least to use them only for discovering new artists, not as a general way to listen to music.

I wouldn’t call myself an audiophile, but a lot of releases there have audible compression artifacts when listened to through proper equipment. That’s not an issue when I’m listening to music on the go, but when I want to listen to music at home, I dedicate an hour to it and try to listen to all of the nuances, and sometimes these ruin the vibe. This is also why I’m interested in vinyl - I like the vibes of manually setting up a record, and then listening to it until it ends. But this requires a dedicated place in the house to place both the records and the turntable with all of its equipment, which I don’t yet have.

Books

Not long before the end of the year, I started reading some management-related books. There’s a reason for that - I’m preparing for the lead position at my company, and this requires a lot more skill than just programming language and project knowledge.

Strangely, I’ve noticed that there’s a certain misconception in the tech companies, at least with the ones I had contact with. A lot of times a team leader is someone who’s skills are just higher than others in the team. E.g. you have this kind of career ladder:

Trainee;
Junior;
Middle;
Senior;
Team leader??

I mean, sure, you’re a skilled programmer by the time you’ve reached a Senior position, and you sure know a lot about projects, but how exactly this will help you lead people? Unless you know basic psychology, how to work with people, how to properly give feedback, how to effectively motivate, and how to help people grow you’re not a team leader (in my eyes), you’re just a person with a project map. A good team leader should not necessarily be a great programmer, it’s a whole different field, that requires a separate education.

Unless we’re talking about a technology leader, of course, and some companies mix these two positions into one and hence require that the team leader has great tech skills. But tech skills are simply not enough to be a good team leader. Tell you what - a great team leader makes it so every team member sincerely wants to do their best, and also are generally happier and feel like a real team.

Games

I have almost finished my PlayStation 4 backlog of games I bought during the years. Finally, I mean, FINALLY, managed to beat Isshin Ashina in Sekiro Shadows Die Twice. Took me several days. I haven’t bought more games on this platform though, as newer games are already mostly targeting the PlayStation 5, which I don’t have, and in general, newer games are less interesting to me.

On the other hand, I’ve played a lot of cool games on the Nintendo Switch. Finally got to play Metroid Prime, thanks to the recent remake/remaster. The Legend of Zelda: Tears of the Kingdom was fun at first, but later I felt exhausted by mostly filler content in the depths. Pikmin 3 was wonderful, did not expect that, although it was quite short. Unfortunately, I wasn’t really inspired by Mario Wonder. Even though it had a lot of seemingly crazy wonder stuff, it felt a bit empty and oddly paced. For example, this year I also played DK Tropical Freeze for the first time, and even though I can’t say that I enjoyed every second, most of the playthrough was interesting, and the pacing felt much better, compared to Wonder. Anyway, that’s just my point of view, you don’t have to agree or disagree, it’s not a review.

What I can say, is that with age it becomes harder to invest time into games. With each year I play fewer games, and I think that’s not uncommon. However, the game industry tries to sell as many games as possible, and I see people with a lot of games in their Steam library that they haven’t played at all. There’s even a term for collecting books but not reading them called Antilibrary or sometimes Tsundoku, and that practically what happens with games today for a lot of people.

I stopped buying games a few years ago due to this happening to me too, and the only reason I recently started this again was that I got the Nintendo Switch. Now I try to complete everything I buy, but I often find myself unable to invest in the game’s world. Like, a lot of games now feel tedious, not interesting. Usually, all I care about is the main plot, unless the side characters are interesting enough, and I don’t want a lore-dump every time I speak with a random person in the game. So, like, it’s hard, but I do enjoy good stories in games, it just has to be compelling enough. Maybe that’s the quality of today’s games, maybe it’s age, I don’t know.

Talks

I gave some talks this year, mostly about Fennel. It’s good that people are interested in Fennel, but I’ve already mentioned that once they try it, a lot are turned down by lots of nuances you have to keep in mind.

I also took a small intensive course on giving talks. Hopefully, my future talks will be better structured and more engaging. I’ve set up a dedicated page where all of my talks are available.

Plans for the next year

Now, I have a lot of plans, but it’s hard to decide upfront what exactly I will do. I definitively want to do more game dev, albeit at a slower pace, and maybe in something less limiting than TIC-80. Don’t get me wrong, though, TIC-80 is amazing.

I also want to finish working through the Crafting Interpreters book, as I’m currently in the process of going through it. Until then, no gamedev things will happen, probably.

Speaking of, I thought about implementing my own language that transpiles to Lua. Not that I dislike Fennel, I just want something both simple and more robust, as well as the fact that I simply want to try tackling this task myself. It’s not exactly in the spirit of Crafting Interpreters, but I figured that there’s no point in creating a new runtime, unless you have some really interesting ideas to try. It’s a good task for education, but I feel that today we have way too many languages, and, honestly, I often think that most of these are just rehashes of existing ones, because people did not like one thing or another in the original.

I’ve mentioned that I’m thinking about leaving the net, and in fact, I actually have plans to do that. In fact, I’m already incorporating some changes to how I live - for the most part, now I try to keep my laptop offline. I won’t believe how less of a distraction it becomes this way! So, for the next year, I’m planning to continue keeping it offline, only going to the net when I need to search for something. I will make a proper post on this, once I have some results. Wish me luck though!

Additionally, I’m slowing down my response time by disabling notifications on my smartphone. This means that if someone writes me a message or an email I won’t be notified of it until I decide to check. I feel that this is a far less disturbing, and more productive model to use my phone. Can’t keep it offline, like my laptop, but that alone is already helpful. After all, emails and chats were meant to be asynchronous, so you should not assume that you’ll get the response instantly.

Definitely going to continue reading more books. Lately, I’ve started noticing that my general erudition is lacking, so I don’t know, but maybe alongside to tech and psychology books I will also try to squeeze in some classics. Maybe I’ll even start a series of posts with a theme of reading reports where I list books I read, with some thoughts on each.

Ending thoughts

This post came out longer than usual. Usual for a recap post, I mean. Probably because this year had a lot of stuff happening.

Anyway, I’m interested in your feedback! Not on this particular post, but in general, I mean. Recently I added a ‘Comment’ button at the end of each post, that allows anyone to send me an email. I already get emails from time to time, so I figured I’d help people reach me by providing a dedicated link for each post with the automatic subject so I could categorize these in my inbox.

Such feedback often encouraged me throughout this year, and I’m trying to make this blog better based on your suggestions. Maybe you want me to cover something, or maybe you completely disagree with something I said - I’m open to any kind of feedback. All such conversations are private, so don’t be shy - if you have something to say, you can reach me!

That’s all from me for this year, so… See you next year!

Comment via email

We've become overdependent on the net

Tue, 26 Dec 2023 22:56:00 +0300

The title says it all.

No, really, I’m astonished at how much software is basically useless without an internet connection. Net is no longer something additional to your daily tasks, it is essential for your daily tasks.

Just recently, I installed GSConnect, a GNOME addon that implements the KDE Connect protocol. I use the Fedora Silverblue, and perhaps it doesn’t come with Open SSL, so the GSConnect complains about it:

Note the message: “Click for help troubleshooting”. When clicked, instead of the help dialog, the browser is opened, automatically trying to visit GitHub wiki:

However, it fails, because my PC is not connected to the net. What baffles me is that they know about the problem, they even have it on the Wiki, yet they refused to include a helpful message, instead requiring me to go to the web page. This could have been summarized in the notification itself.

Not so long ago, we bought a new laptop with Windows 11 on board. It was not for my personal use, and as much as you may like to suggest that I should just install Linux on the machine, I can’t, and the person would not want to deal with it either. What’s funny, though, was that the laptop refused to continue through the first boot welcome screen until we connected it to the net.

What’s the problem, you may ask? The problem was again the absence of a network around - the house did not have a PC for maybe five to ten years, so the net bill was pointless. 4G was fine enough on the tablet, and therefore when we bought a PC it could not connect to anything, because there wasn’t anything to connect to. What makes things worse, the mobile provider provides unlimited traffic but forbids tethering it from the device, so we couldn’t even use that to turn the laptop on.

Some people told me that if I press the connection button enough times, Windows gives up and lets you in without the net, but it didn’t work for me. I guess this was either added in the later builds or was removed by that point. The question is - why do I need the network connection to turn the device on at all?

Smartphones are the same - buy a new phone, and it asks for a SIM card, then for WIFI, although it is possible to skip these things. For now, at least. Funnily enough, old cellular phones from the 90s and mid-2000s refused to boot without a SIM cart too. Why? Who knows. Some even offered a “demo mode” - you could see what the phone can do, but could not use it until you put a SIM in. Oh well.

But let’s go back to the modern times. A lot of smartphone apps refuse to work without an internet connection. For example, when I used Spotify, on the premium plan you could download the music for offline listening. However, when the network is unavailable, i.e. you’re offline, the app refuses to play the song. Probably because it tries to connect to the server to check if your local date and server date are in sync, and if your premium plan is still active. Or maybe it is just a poorly written app, but that can not be, I paid every month, and I suppose a lot of this money was spent on the app development. Right?

The subscription model is handy for businesses that want to constantly get a lot of money without that much effort. For example, before the subscription model was a thing for Adobe products, each version of Photoshop featured a lot of additions and improvements justifying purchasing a new license. Now, however, years go by, yet Photoshop still frequently crashes and doesn’t see substantial features, and a lot of different software is catching up feature-wise, further dis-justifying the subscription model. Though, I don’t use Photoshop, so don’t take my word for anything I said - this information is provided by public reviews, and from some friends and coworkers who do have a subscription. And yes, a subscription model is not directly tied to the topic of dependence on the net, however, you usually can’t use your software without periodic checks, and paying also requires the network.

Another thing is the web apps. I don’t use such apps often, but speaking of Photoshop reminded me of Figma. It is not exactly a competitor, but it took away some of the userbase because it is a better fit for the task. Again, probably, I don’t know why people use it, I, myself, don’t. However, being a web application, it doesn’t really have an offline mode at all. All your work is constantly synced with the server, and again, I’m not sure why - local storage is right there.

Web technology has probably the fastest time-to-market so I can understand why it is probably better for a business to quickly publish a web app, instead of doing a proper desktop application. But then again, these apps, even though they are emulating a desktop application, usually don’t stop at local features, and provide features that are dependent on the net. For example, a recent rise in the popularity of Large Language Models resulted in a lot of apps including one as a “feature”. But running these locally comes with a huge performance cost, not everyone would agree to pay, so instead they run by a huge server somewhere, and are provided as a paid API subscription. Now we have subscriptions that the app developer has to pay, and these apps usually use a subscription model too, meaning that in the end, you pay for the API, even if you don’t use it.

I know, I know. This may sound silly. But just for a day, try to use your PC or, if you brave enough, a smartphone without an internet connection, and see how much work can be done. The reason I’m writing this post is that I simply don’t have a network connection right now, so I just noticed this issue. While I mostly do my work from Emacs, some apps still require a network, and I feel that I can’t use my PC to reach its full potential. It’s a shame though, as most of the things I need the network for can be, and were done without it before. Net just makes things easier.

Comment via email

Janet

Wed, 20 Dec 2023 23:17:00 +0300

I decided to give Janet another look - I’ve mentioned Janet before in this blog, and I have my thoughts on it. However, I have never actually interacted with the language that much - I only read its documentation and some code.

Recently, I found this interesting post: My Kind of REPL by Ian Henry. It’s an interesting post, and I really liked the Braille plotting library they mentioned there. So I wanted it for myself.

That’s not strong enough motivation to switch to Janet from Fennel or Clojure, so I decided to port this library to Clojure. Yes, no Fennel this time. To be honest, I noticed that my feelings towards Fennel are becoming a bit rusty. Perhaps Janet’s author felt the same thing and made Janet after Fennel?

Well, after a bit of searching, I found the library they’re talking about in the post: bytemap. The name is a bit confusing, I was expecting something like braille-plot or braillemap at least. But it doesn’t matter - what does matter, is the ability to plot stuff in the REPL. And the library is neat, with cool tricks to render points to the Braille symbols using bit-ops! Kudos to Ian for implementing it!

Now, I have a lot of experience porting Clojure code to Fennel and back, and I would say that the experience is seamless enough. What I mean by that is that most of the forms can be used as is, or some simple text substitutions are required. Most of these are substituting Fennel’s module separator . with Clojure’s /, and substituting , in macros with ~. There are more differences than that, of course, but one of the goals I had with fennel-cljlib and async.fnl libraries was to allow copying code from Clojure to Fennel and back with as little changes as possible. These two libraries provide most of the important Clojure behavior, and some of Clojure syntax to Fennel.

Janet, on the other hand, says that it is not a Clojure port, and lists all differences that come with that. On the other hand, their standard library is far more rich than Fennels, and there are a lot of inspirations taken from Clojure as far as I can see. So that’s good.

I mentioned that apart from small syntactic changes, Fennel code can be ported to Clojure fairly easily. With Janet, the situation is a bit different. Say, here’s some Fennel code:

(fn [x]
  (let [a (foo)
        b (bar)
        b (baz b)]
    (qux a b x)))

Or was it Clojure code? In fact, this code will work in the same way in both languages. Now, let’s look at the same code in Janet:

(fn [x]
  (def a (foo))
  (def b (bar))
  (def b (baz b))
  (qux a b x))

Now, this code is different, however, it will work in Clojure, yet with a subtle difference. What is the difference you ask? Why, in Clojure a and b are global bindings!

Now, you might wonder, why I bring this up. Well, I find this strange, but in Janet, all of the documentation is written using def instead of let which Janet also has. And as a result, a lot of code in the wild is written like that.

That’s another mystery to me - why Janet author, who created Fennel before that, decided that using def is the way to create local variables? I mean, sure, let introduces yet another scope into the code, but is it that bad? Anyhow, you can write this code in Janet in the same way as in Clojure or Fennel, but I’m yet to see let used in Janet code in the wild.

If you’re interested in the full source code of the port, you can find it here.

Porting Janet code to Clojure

Janet uses def to establish immutable lexically-scoped bindings, and var to create mutable bindings. It’s a bit similar to Fennel, where the local (a much better name, TBH) is used for the same purpose as def in Janet, and var does the same thing. However, when porting to Clojure we should remember that we don’t have mutable bindings of any kind. So my first steps were to change all of the code to have correct Clojure syntax and replace all var definitions with the use of atom.

For example, here’s such change illustrated on the plot function:

;; Janet                                                     ;; Clojure
(defn plot [f [w h] x-scale y-scale &named axis]             (defn plot [f [w h] x-scale y-scale
  (default axis true)                                                    & {:keys [axis] :or {axis true}}]
  (def canvas (new w h))                                       (let [canvas (new w h)
  (def [w h] (bounds canvas))                                        [w h] (bounds canvas)]

  (when axis                                                     (when axis
    (for i 0 h                                                     (dotimes [i h]
      (draw-point canvas [(/ w 2) i]))                               (draw-point canvas [(/ w 2) i]))
    (for i 0 w                                                     (dotimes [i w]
      (draw-point canvas [i (/ h 2)])))                              (draw-point canvas [i (/ h 2)])))

  # we don't want to clip the extreme values off the canvas,     ;; we don't want to clip the extreme values off the canvas,
  # so we narrow the y range slightly                            ;; so we narrow the y range slightly
  (def y-scale (* y-scale (/ (+ h 1) h)))                        (let [y-scale (* y-scale (/ (+ h 1) h))

  (var current-point nil)                                              current-point (atom nil)]
  (for i 0 w                                                       (dotimes [i w]
    # x spans -0.5 to 0.5 (inclusive!)                               ;; x spans -0.5 to 0.5 (inclusive!)
    (def x (- (/ i (- w 1)) 0.5))                                    (let [x (- (/ i (- w 1)) 0.5)
    (def y (/ (f (* x 2 x-scale)) y-scale -2))                             y (/ (f (* x 2 x-scale)) y-scale -2)
    (def p [(* (+ x 0.5) (- w 1))                                          p [(* (+ x 0.5) (- w 1))
            (* (+ y 0.5) h)])                                                 (* (+ y 0.5) h)]]
    (when current-point                                                (when @current-point
      (draw-line canvas current-point p))                                (draw-line canvas @current-point p))
    (set current-point p))                                             (reset! current-point p)))
  (print-canvas canvas))                                           (print-canvas canvas))))

Here’s a quick demo:

braille-plot.core> (plot math/sin [40 10] math/PI 1)
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⢀⠤⠖⠊⠒⠒⠤⡀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⢀⠔⠁⠀⠀⠀⠀⠀⠀⠈⠢⡀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⢀⠔⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢆⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⢠⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠱⡀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡷⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⢄
⠹⡉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⢉⠝⡏⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉
⠀⠘⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⠊⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠈⠢⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡰⠁⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠑⢄⠀⠀⠀⠀⠀⠀⠀⠀⡠⠊⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠑⢤⣀⣀⢀⣀⠤⠊⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
nil

Changes in the code so far:

Janet uses a default macro to establish a default value, in case one wasn’t specified. Here it works with the &named attribute in the argument list.

In Clojure, we use rest syntax & together with the map destructuring syntax {} to achieve the same style of argument list. And to supply default values for named arguments, map destructuring supports :or modifier that itself specifies a map of names to default values.

Hence the change from &named axis with (default axis true) to {:keys [axis] :or {axis true}}.
for is a list comprehension in Clojure, so it was changed to dotimes which has the closest semantic.
Comments were changed to use ;; instead of #
All inline def usage was replaced with let binding multiple symbols.

This introduced more nesting, but it’s not a problem.
var usage was replaced with the use of atom.

To be clear, I should use volatile! here, but this is a temporary change, and more people are familiar with atom. This change required changing set to use reset! as a means to update the atom.
I also had to replace the new function name with canvas because new is a special form in Clojure.

So even this small piece of code required a fair amount of changes. The most annoying one was converting def usage to an appropriate let usage, as it involved a lot of manual copying.

By the way, the original Janet code changes a lot of values in place. This happens all over the code, and because of that, I had to use atom in a few more places. However, most of these mutations are not necessary, and we can use things like Clojure’s update, that instead of modifying a collection, return a new one with the modifications. So the second change to this code was to make the canvas immutable,

Now all of the functions that change the canvas accept it, and return a new, modified canvas:

(defn plot
  ([f] (plot f (canvas)))
  ([f {:keys [x-scale y-scale]
       :as canvas}]
   (let [[w h] (bounds canvas)
         y-scale (* y-scale (/ (+ h 1) h))]
     (loop [[x & xs] (range w)
            current-point nil
            canvas canvas]
       (if x
         (let [x (- (/ x (- w 1)) 0.5)
               y (/ (f (* x 2 x-scale)) y-scale -2)
               p [(* (+ x 0.5) (- w 1))
                  (* (+ y 0.5) h)]]
           (recur xs p
                  (cond-> canvas
                    current-point (draw-line current-point p))))
         canvas)))))

I’ve changed plot to accept canvas as the last argument instead of passing all of its options and creating it if it wasn’t passed. So now we can just call (plot math/sin) and get a sensible result. This is another way to provide default arguments in Clojure - we create two bodies for the function, and if the function is called with less than the needed amount of arguments, it has a dedicated body that can call itself with additional arguments. Neat!

I’ve moved out the axis creation code into a separate function draw-axis because it makes more sense to create a blank canvas and draw axies on it afterward. Since we’re passing the canvas object through, gradually creating newer versions of it with each function, we can have it as a separate step.

The final change here is the use of the loop instead of the dotimes. This change is motivated by the fact that we need to change some state between iterations, but we don’t want to use mutable constructs such as atom. So instead, we create a binding current-point and change it with each call to recur. And the same thing happens to the canvas binding. This is literally a recursion, except Clojure has to do it this way because JVM can’t optimize tail calls.

And, it works! We can now pass the canvas through multiple plot calls:

braille-plot.core> (->> (canvas :width 42 :height 20)
                        (plot #(math/sqrt (abs (- 1 (math/pow % 2)))))
                        (plot #(- (math/sqrt (abs (- 1 (math/pow % 2))))))
                        (plot #(/ (math/sqrt (abs (- 1 (math/pow % 2)))) 2))
                        (plot #(/ (- (math/sqrt (abs (- 1 (math/pow % 2))))) 2))
                        draw-axies
                        print-canvas)
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⣠⠤⠔⠒⠒⠒⠊⡗⠒⠒⠒⠢⠤⢄⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⠴⠒⠉⠁⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠉⠒⠤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⢀⠴⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠑⠤⡀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⢀⠔⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠢⡀⠀⠀⠀⠀
⠀⠀⠀⡠⠃⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⢄⠀⠀⠀
⠀⠀⡔⠁⠀⠀⠀⠀⠀⢀⣀⣠⠤⠔⠒⠒⠒⠊⠉⠉⠉⡏⠉⠉⠑⠒⠒⠒⠢⠤⢄⣀⡀⠀⠀⠀⠀⠀⠈⢢⠀⠀
⠀⡰⠁⠀⠀⣀⠤⠒⠉⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠒⠤⣀⠀⠀⠈⢆⠀
⢠⠃⢀⠔⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠢⡀⠘⡄
⢸⠔⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠢⡇
⡿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢿
⣏⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⡏⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⣹
⢹⢆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡰⡏
⠸⡀⠑⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡠⠊⢀⠇
⠀⢣⠀⠀⠉⠒⠤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⠤⠒⠉⠀⠀⡜⠀
⠀⠈⢢⠀⠀⠀⠀⠀⠉⠑⠒⠲⠤⢄⣀⣀⣀⡀⠀⠀⠀⡇⠀⠀⢀⣀⣀⣀⡠⠤⠔⠒⠊⠉⠀⠀⠀⠀⠀⡔⠁⠀
⠀⠀⠀⠱⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠉⠉⡏⠉⠉⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠎⠀⠀⠀
⠀⠀⠀⠀⠘⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡠⠃⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠑⢤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠤⠊⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠈⠒⢤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⣀⠤⠒⠁⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠓⠲⠤⢄⣀⣀⣀⡀⣇⣀⣀⣀⡠⠤⠔⠒⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
nil

Code Snippet 1: (the width of the outer unit circle depends on your Braille font)

The rest of the code was converted similarly. Uses of def were converted to let, and in-place mutations were replaced with the use of immutable data structures API.

ClojureScript came with its oddities. Clojure 1.11.0 introduced the clojure.math namespace that allows more portable code between implementations. However, the clojure.string namespace is still quite sparse. For instance, here’s the function that generates the Braille symbols:

(defn- braille [byte]
  (let [bytes (encode (+ 0x2800 byte))]
    #?(:clj
       (String. (byte-array bytes))
       :cljs
       ;; TODO: maybe there's a better way?
       (let [bytes (new js/Uint8Array (clj->js bytes))]
         (.decode (new js/TextDecoder "utf-8") (clj->js bytes))))))

It is pretty straightforward for Clojure, but in ClojureScript we have some shenanigans. I program in ClojureScript quite infrequently, so this code is, perhaps, completely terrible, so reach me out if you know a better way of doing this.

Closing thoughts

Well, Janet still feels weird as a language. I mean, it feels like its syntax, with regards to special characters, went through the Caesar cipher: # for comments, ; for splicing, @ for table mutability, ~ for quasiquote ` for long strings, | for short function notation. Its use of def is, well, unconventional, and code style, in general, is not lispy enough. I mean, Fennel isn’t a proper Lisp either, yet thanks to the runtime, it feels pretty natural, and its syntax doesn’t deviate too far from established conventions. It feels more mature than Fennel, but quirky at the same time.

In the future, though, I’d like to take a closer look at Janet’s virtual machine. Perhaps it can be used as a target for a language that I will personally like more.

Comment via email

GNOME Software

Wed, 06 Dec 2023 20:59:00 +0300

Have you tried using the GNOME Software? This thing:

Do you use it? Are you even using GNOME?

Oh, sorry, I think should point this out, it’s kinda important - I’m asking the developers of GNOME Software.

Because, apparently, they don’t.

OK, let’s start this post over.

GNOME Software

Wed, Dec 6, 2023

@rant GNOME tools ~8 minutes read

GNOME Software, or, just, Software, is an application that “allows you to find and install new apps and system extensions and remove existing installed apps”. I mean, that’s what description is given to it on the official project page. It is preinstalled on most distributions that use the GNOME desktop environment, except for, maybe Ubuntu, which has its own store for applications. But I’m not going to touch Ubuntu here, I don’t really want to touch Ubuntu at all, to be honest, although I have to use it five days a week.

At home, I use the Fedora Silverblue. I like Fedora, in my opinion, it is one of the finest out-of-the-box-oriented distributions today because of how pure it is. No fancy patches, or extensions, just pure GNOME as it is. In addition to that, the Silverblue variant is immutable, so in addition to providing pure GNOME, it barely provides anything else.

I’ve picked Silverblue because I like to keep my system clean, and immutability helps tremendously. Everything is either a Flatpak or lives inside the Podman container. To be frank, even before I used Silverblue I had the same kind of workflow on the regular Fedora distribution, and the same applies to Ubuntu that I use at work. If your system got messed up enough for you to care you can just delete the container, and recreate a new one, installing only the stuff you do actually require. But I’m getting off track here.

I feel that the fact that I’m using Fedora Silverblue is important for this post because this means that I’m using a distribution that I can’t make changes to. Well, I can, of course, but I don’t do that. Again, all my apps are either in Flatpaks or containers, so the system itself is barely different from the fresh installation. I had installed updates, but these come from Fedora, so I’m assuming they know what they’re doing.

So let’s get back to the topic - I install apps as Flatpaks quite often. I do so using GNOME Software!

And it is pain!

I have used GNOME since 2008 or 2009, can’t remember that exactly, so I’m a seasoned Linux user. I have used a lot of different distributions, and for the past five years, Fedora has been my daily driver because it is simple, reliable, and has everything I need. And what it doesn’t have, I can build from sources in the container with my own hands, so I don’t give a shit about stuff like AUR, Nix, etc. For simple stuff, like a music player, Flatpak is more than enough, and for stuff like Emacs, I’m running a container with all my development tools. And here comes GNOME Software.

For the past decade, ever since GNOME Shell was released in 2011 I have been using it. I had spent two years using KDE just to see if the grass was kreener on the other side but returned to GNOME eventually. And, for the past decade, I’ve been using GNOME Software.

I mean, what, do you expect me to open a terminal every time I need to install or update a package? Sure, back in the day I did that when I used Arch, and Ubuntu before it, but honestly, I don’t care if I need to input pm update in the shell or press an “Update” button in the GUI.

terminal is faster and more convenient for managing packages

Yeah, right. And how am I supposed to go and look for the screenshots of the app I’m about to install on the terminal? You could say it’s a silly reason, but during the post on Linux music players, I have done that a lot. Of course, when I know exactly what app I need, I can go to the shell, and type out the command in a fraction of the time that it takes for GNOME Software to fully load and become responsive. And that’s what brings me to the beginning of the post.

Dear GNOME Software developers, tell me, do you use your application or do you install everything from the terminal anyway?

I think that the developer should be, if not the main, but an active user of the project they’re developing. That applies both to open source and what you do at work - because if not, you don’t know your own software.

A theory is nothing without practice, and the same goes for software development. The code is a theory. You compile the code and verify that it is correct like any theory should be. The application of that code is practice.

And I feel that the GNOME Software developers don’t use the program they create. Here’s why.

As I said before, I use Fedora Silverblue, with everything installed as a Flatpak or inside a container, so the system is pretty much as if I just installed it. I rarely shut down my laptop, instead, I put it to sleep (not even into hibernation) most of the time. Now, imagine I want to go and install the Audacious music player. I open Software and start typing Audacious, and that’s what I see:

I stare at this for a few seconds, check that my internet connection is working, and realize that it happened again - GNOME Software just doesn’t do anything. I tried waiting for minutes, and the spinner didn’t go away. So I closed the app. I opened it again. And I still see the damn thing spinning because you know what? When you press the close button, the Software doesn’t actually close, it always runs in the background, for reasons.

So I have to go into the System Monitor and kill the damn thing. I reopen it again, type Audactions, after it finishes loading everything, and now I see the player:

And this happens to me every single time. On a system, that is, according to its promise is immutable, and hence this should probably happen even at the stock installation. And, you know what - it does! I have recently reinstalled Fedora on one of the laptops I have and left it working for a while, checking periodically if GNOME Software will behave like that. I didn’t have to wait for long, it happened the next day!

So this bug can be reproduced, with a bit of patience, on a clear installation! Yet, it haunts me for the last 5-6 years or so! Aarrgh! ARE YOU EVEN TESTING YOUR SOFTWARE AT ALL?

*breathes*

Sorry, I’ve got too ~~angry~~ dramatic even for my style of writing. I’m not the only one who experiences this particular bug, among other bugs:

A Reddit post describes this exact problem, and many others, specifically mentioning that they’re not new
A bug report was submitted to the official issue tracker, outlining the same problem. I like the comment there:

Without a log I’m afraid we can’t diagnose or fix the problem, so I’ll close this issue report for now.

Can’t diagnose? Seriously? Have you even tried though?

Seriously, as a software developer, I don’t understand this. I mean, sure, sometimes I don’t want to deal with bugs I personally don’t have, and it is open source, so I don’t really have to do that, someone else can and is welcome to. But to close the issue because there are no logs and it can’t be diagnosed, despite the fact that the issue exists? Especially, when it hits multiple people. I don’t get it. Don’t try to explain me, I won’t get it, because it’s hard for me to get it, so I’ll close the discussion for now.
Many search related requests are closed, yet the problem is still there.

It’s like they find a possible reason why this might happen, fix it, and close the issue without testing. I mean, unless you can reproduce the bug before the fix, how would you know that it is fixed?

I don’t know how the development process is organized in GNOME, especially for GNOME Software. However, I think that they don’t actually use their own thing, because it has so many obvious bugs that I can hit several during a single session. For instance, the Reddit post mentions this:

While installing, the window “reloads” a couple of times, as if it was refreshing.

Yup, happens to me every time.

Sometimes it feels like updates have been fetched, and the ones being displayed are the most recent, but updating only reveals those updates were not the newest ones and even updates need to be downloaded

Happens to me as well. To be more specific it happens like this:

You open the Software (that has been running in the background for long);
You see that the “Updates” tab has a notification;
You open the “Updates” tab and it shows a spinner;
Then the updates appear.

So I would assume, seeing the spinner, that it had fetched everything, but by pressing the “refresh” button, the spinner appears and goes for much longer, revealing that there are in fact more updates than there initially were. Literally happened to me while I was writing the previous line, as I’m writing and testing at the same time.

There’s another issue, that is closed on the tracker, but happens to me all the time, and is extremely easy to reproduce: while you install anything, the search stops working. The funny thing, it is closed as a duplicate of the issue with the name “Rework threading model”, which supposedly should fix all of that. However, how it is a duplicate? At best it is blocked by that ticket, but it doesn’t duplicate it in any way because it describes the actual issue. It should be closed only when the blocking issue is done, and the bug is tested to be gone. What a mess.

So, yeah, GNOME Software is full of bugs, as most of the software is today. If you’ve seen the “Preventing the Collapse of Civilization” talk by Jonathan Blow, you may remember there was a moment where he listed all of the bugs that happened to him in the past few days. Have a look, I’ve linked it with a timecode.

So, why am I complaining in the blog and not actually submitting an issue? Well, I don’t want to. GNOME has a long history of ignoring problems, saying that people use their software in the wrong ways, and so on. I just don’t want to go to their territory and be ignored too. Instead, I can write a quick rant in my personal space, where I’m always right, and no one can prove me wrong, ha ha ha! Obviously, I’m being sarcastic here, if you couldn’t tell.

I guess, someday these bugs will be fixed, it’s just a shame that they have existed for so many years, and it seems like the developers don’t care.

Anyway, that’s enough ranting for this year. Use what you’re making, and test your fixes, or at least ask for feedback once you think you fixed the bug. See ya!

Comment via email

Thoughts on Crafting Interpreters - Part 1

Sun, 03 Dec 2023 21:35:00 +0300

Today I would like to discuss the Crafting Interpreters book by Robert Nystrom.

It’s a book about designing an interpreter for a dynamic programming language called Lox. Well, not exactly. It’s split into two parts - in the first is about crafting a tree-walking interpreter, and the second is about writing a complete bytecode VM.

I’m also going to write two posts, one for each part of the book. This particular one is on the first part, which I have completed a long time ago. A year and a half, actually. So most of the things I’m going to talk about here are not fresh.

In the book, Robert chose Java as a language for the first half of the book. The reasoning behind it is practical and well thought out, but I simply don’t like Java. So instead, I chose Clojure as an implementation language, because I felt that it would be a good exercise, allowing me to sharpen my skills a bit more.

TL;DR for this post could be:

Clojure is a nice language, and fits this kind of task well enough, as the tree-walking interpreter is largely a data-manipulation task. Clojure’s protocols provide a great way of solving the expression problem with no boilerplate when compared to the Visitor pattern. Immutability helps tremendously, eliminating bugs related to state management, with little, yet noticeable cost in performance, for this particular task, at least.

Plus a few complaints about the book.

Now, if you’re interested in more, I welcome you to read the rest of the post. But first, let’s get this straight - I’ll try to follow the book’s structure here, but no promises. While the book is well-structured, it’s hard to reference specific parts of the code, as some pieces of the interpreter are changed frequently through the chapters.

I’m glad I’ve kept the repository around, as it contains all of the commit history, however, until the second part of the book I was loose on commit structure. E.g. my commits for the first half look like

...
a6f69c8 * use protocols to solve expression problem
7412fb2 * WIP AST
4068faf * WIP parser
7990eb4 * support escapes in strings
d9b3f02 * move tokenizer to its own namespace
3d8a602 * add column info
186095f * tokenizer

While for the second half of the book it is structured much better:

...
88dd209 * chapter 19: Strings
6116263 * chapter 18: Types of Values
742364c * chapter 17: Compiling Expressions
f1ceb50 * Chapter 16: Scanning on Demand
da0149c * chapter 15: A Virtual Machine
f2d8a18 * chapter 14: Chunks of Bytecode
...

So, forgive me if I mix things up in this post - the second part will be better structured. And, to be clear, this post isn’t really about the code, but about the book, although there’s plenty of code here. I will highlight interesting differences in Clojure and Java approaches but I would not consider this the main topic of this post.

Scanning

For the first chapter, we start with scanning. The chapter introduces various concepts, like error handling, but what I’d like to focus on here is token handling instead.

In the book, Robert uses an enumeration type, which is probably how it is done in the Java world:

package com.craftinginterpreters.lox;

enum TokenType {
  // Single-character tokens.
  LEFT_PAREN, RIGHT_PAREN, LEFT_BRACE, RIGHT_BRACE,
  COMMA, DOT, MINUS, PLUS, SEMICOLON, SLASH, STAR,

  // ...
}

Note, I don’t know Java - never learned it for real, and went straight to Clojure. So if you’re curious if it’s OK to go into Clojure without knowing Java first - it is. Although, I did have some Scheme background.

So unlike the Java solution, I went with Clojure’s way of enumerating things. Maps:

(def single-token-type
  {\( :left_paren
   \) :right_paren
   \{ :left_brace
   \} :right_brace
   \, :comma
   \. :dot
   \- :minus
   \+ :plus
   \; :semicolon
   \* :star
   \= :equal
   \< :less
   \> :greater
   \/ :slash
   \! :bang})

The same goes for all other token types. It makes sense, as we’re essentially using the Enum as a lookup table for character dispatch. Hence, instead of doing this:

char c = advance();
switch (c) {
    case '(': addToken(LEFT_PAREN); break;
    case ')': addToken(RIGHT_PAREN); break;
    case '{': addToken(LEFT_BRACE); break;
    case '}': addToken(RIGHT_BRACE); break;
    case ',': addToken(COMMA); break;
    // ...
}

We can do this:

;; ...
(case c
  (\( \) \{ \} \, \. \- \+ \; \*)
  (recur current line
         (conj tokens (make-token (single-token-type c) (str c) line))
         error?)
  (\= \! \< \>)
  ;; ...
  )

In other words, we share the body for all cases, and do a character lookup in the table to get the token type. This eliminates just a little bit of duplication.

Another thing to note, which is more important in my opinion, is that this parsing process is stateless. In the Java version, the addToken method modifies this class state. While handy, I don’t like this approach, it makes things harder to test. My version uses a conventional Clojure loop and simply adds tokens to the collection, which is then returned from the loop:

(defn tokenize [source]
  (let [at-end? (complement (partial > (count source)))]
    (loop [current 0     ;; current character
           line 1        ;; current line
           tokens []     ;; tokens
           error? false] ;; error indication
      (if (at-end? current)
        ;; when finished return a tuple containing tokens and error
        ;; indication
        [tokens error?]
        ;; otherwise, we take the `nth` character from the source, and
        ;; dispatch on it
        (let [c (nth source current)
              current (inc current)]
          (case c
            (\( \) \{ \} \, \. \- \+ \; \*)
            ;; Rebind the current char to the new `current` char;
            ;; Use the same line, as none of these introduce a line change;
            ;; Add new token to an immutable vector of `tokens`;
            ;; pass `error?` as is, as no errors were introduced.
            (recur current line
                   (conj tokens (make-token (single-token-type c) (str c) line))
                   error?)
            ;; ... rest of the clauses follow the same idea ...
            ))))))

So, tokenization is fully self-contained, and doesn’t require any state - it just accepts the string. I later added the column counting to this code, but it’s beside the point now.

Representing Code

This chapter mostly goes on how to read and define a BNF-like grammar. However, what’s really interesting is the second part of this chapter - generating the tree representation generation code. It is often referred to as metaprogramming, i.e. programs writing programs.

At first, I was puzzled by this - do we really need to write Java to generate some classes for us based on grammar? Well, I mean, there are libraries that can do that at runtime without an issue, like instaparse, but in this case, we’re generating stuff even before compilation. In other words, the book suggests generating source code.

Well, that’s just macros, I thought to myself, and after studying a bit what the intention is, I made the following macro:

(defmacro define-ast [& specs]
  (list* 'do
         (for [spec specs]
           (let [[name fields] (str/split spec #"\s+:\s+")
                 fields (->> (str/split fields #",?\s+")
                             (partition 2)
                             (map second)
                             (reduce (fn [fields name]
                                       (conj fields (symbol name)))
                                     []))]
             `(defrecord ~(symbol name) ~fields)))))

It expands the following grammar into a list of record definitions:

cljloc.ast> (macroexpand-1
             '(define-ast
                "Binary   : Expr left, Token operator, Expr right"
                "Grouping : Expr expression"
                "Literal  : Object value"
                "Unary    : Token operator, Expr right"))
(do
 (clojure.core/defrecord Binary [left operator right])
 (clojure.core/defrecord Grouping [expression])
 (clojure.core/defrecord Literal [value])
 (clojure.core/defrecord Unary [operator right]))

However I realized that it is a bit overkill to do it this way, so I ditched the macro completely. After all, writing these records requires much less work in Clojure than it is in Java, but we’ll get to that in a bit.

For comparison, here’s a definition from the book:

static class If extends Stmt {
    If(Expr condition, Stmt thenBranch, Stmt elseBranch) {
      this.condition = condition;
      this.thenBranch = thenBranch;
      this.elseBranch = elseBranch;
    }

    @Override
    <R> R accept(Visitor<R> visitor) {
      return visitor.visitIfStmt(this);
    }

    final Expr condition;
    final Stmt thenBranch;
    final Stmt elseBranch;
  }

And here’s the Clojure version:

(defrecord If [condition then else])

So, why does the book’s code appear to be more involved?

The Visitor pattern

There’s a pretty specific problem, often appearing in projects, like this - the expression problem. In the Java world, one of the solutions is the Visitor pattern, which the Book then explains thoroughly. I will not go into full details here, you’re free to read the book if you’re interested.

The script, that generates classes for statements also generates methods required for the Visitor pattern to work. We then can override these methods for each resulting type, and that’s basically it. In the book, it is explained by defining a pretty printer for AST, printing parsed data as S-expressions. Oh, sweet irony.

Here’s how it is done for the Binary expression:

package com.craftinginterpreters.lox;

class AstPrinter implements Expr.Visitor<String> {
  String print(Expr expr) {
    return expr.accept(this);
  }
  // ...
  private String parenthesize(String name, Expr... exprs) {
    StringBuilder builder = new StringBuilder();

    builder.append("(").append(name);
    for (Expr expr : exprs) {
      builder.append(" ");
      builder.append(expr.accept(this));
    }
    builder.append(")");

    return builder.toString();
  }

  @Override
  public String visitBinaryExpr(Expr.Binary expr) {
    return parenthesize(expr.operator.lexeme,
              expr.left, expr.right);
  }
  // ...
}

Here’s how it is done in Clojure:

;;; protocols.clj
(ns cljlox.protocols
  (:require
   [clojure.string :as str]))

(defprotocol IStringable
  (tostring [self]))

;;; ast.clj
(ns cljlox.ast
  (:require [cljlox.protocols :refer [IStringable tostring]])
  (:import (cljlox.tokenizer Token)))

(defrecord Binary [left, ^Token operator, right]
  IStringable
  (tostring [{:keys [operator left right]}]
    (format "(%s %s %s)" (tostring operator) (tostring left) (tostring right))))

And that’s it. But it doesn’t end here, because Clojure was designed in such a way that the expression problem is not a problem.

We can extend other types with our protocol, even after these types are defined. For, instance, here I extend the Object to handle numbers, and any other objects, as well as nil which is a special case for protocols in Clojure:

(extend-protocol IStringable
  Object
  (tostring [self]
    (cond (double? self) (-> self str (str/replace #"\.0$" ""))
          (nil? self) "nil"
          :else (str self)))
  nil
  (tostring [_] "nil"))

And there you have it - we don’t need the Visitor pattern in Clojure, and we’re free to extend existing methods with new classes at any point. The book demonstrates that the pattern works by defining overrides for all other data types, and the following code:

  public static void main(String[] args) {
    Expr expression = new Expr.Binary(
        new Expr.Unary(
            new Token(TokenType.MINUS, "-", null, 1),
            new Expr.Literal(123)),
        new Token(TokenType.STAR, "*", null, 1),
        new Expr.Grouping(
            new Expr.Literal(45.67)));

    System.out.println(new AstPrinter().print(expression));
  }

Which prints (* (- 123) (group 45.67)). We can do the same in the REPL:

cljlox.ast> (tostring
             (ast/->Binary
              (ast/->Unary
               (Token. :minus "-" nil [0 0])
               (Token. :literal "123" nil [0 0]))
              (Token. :star "*" nil [0 0])
              (ast/->Grouping
               (Token. :literal "45.67" nil [0 0]))))
"(* (- 123) (do 45.67))"

It is different a bit, as I’m passing a line number and a column as a tuple [0 0] but other than that it is the same. If you’re wondering, Token also implements the IStringable protocol:

(defrecord Token [type lexeme literal pos]
  Object
  (toString [_]
    (str lexeme))
  IStringable
  (tostring [_]
    (str lexeme)))

We’ll see more of the protocols stuff in a bit, so let’s move on.

Parsing Expressions

The parser described in the book is based on the recursive descent algorithm. I like it, it’s simple and effective and works reliably.

However, it is again done in the Java style, mutating private things in the Parser class. I don’t like it, it’s error-prone and hard to track what’s happening.

Instead, I made it functional and pure:

(defn parse
  "Parse a sequence of `Token`s into a sequence of expressions."
  [tokens]
  (try
    (loop [exprs []
           n 0]
      (if (< n (dec (count tokens)))
        (let [[expr n] (expression tokens n)]
          (recur (conj exprs expr) n))
        exprs))
    (catch ExceptionInfo e
      (let [{:keys [tokens n]} (ex-data e)
            token (get tokens n)
            [line col] (:pos token)]
        (log/errorf "[%s:%s] %s at '%s'" line col (ex-message e) (str token))))))

Each rule then simply returns the expression object, and the next n:

(defn- consume [tokens n type message]
  (if (#{type} (:type (get tokens n)))
    (inc n)
    (throw (ex-info message {:tokens tokens
                             :n n}))))

(defn- current [tokens n]
  (if-some [token (get tokens n)]
    token
    (throw (ex-info "Unfinished expression" {:tokens tokens
                                             :n (dec n)}))))

;; ...

(defn- equality [tokens n]
  (loop [[expr n] (comparison tokens n)]
    (if (and (not= expr :eof) (#{:bang_equal :equal_equal} (:type (current tokens n))))
      (let [n (inc n)
            operator (previous tokens n)
            [right n] (comparison tokens n)]
        (recur [(->Binary expr operator right) n]))
      [expr n])))

Yes, this involves passing n around, but if the function API is consistent, all is well. All functions in the parser accept a vector of tokens and the current n for a position in the array. No exceptions to that. Why would you want anything else?

And it’s so much easier to test too:

cljlox.parser> (equality [{:type :number, :lexeme "1", :literal 1.0, :pos [1 1]}
                          {:type :plus, :lexeme "+", :literal nil, :pos [1 3]}
                          {:type :number, :lexeme "1", :literal 1.0, :pos [1 5]} ;; starting from here
                          {:type :equal_equal, :lexeme "==", :literal nil, :pos [1 7]}
                          {:type :number, :lexeme "2", :literal 2.0, :pos [1 10]}
                          {:type :eof, :lexeme "EOF", :literal nil, :pos [1 11]}]
                         2)
[{:left {:value 1.0},
  :operator
  {:type :equal_equal, :lexeme "==", :literal nil, :pos [1 7]},
  :right {:value 2.0}}
 5]
cljlox.parser> (equality [{:type :number, :lexeme "1", :literal 1.0, :pos [1 1]} ;; starting from here
                          {:type :plus, :lexeme "+", :literal nil, :pos [1 3]}
                          {:type :number, :lexeme "1", :literal 1.0, :pos [1 5]}
                          {:type :equal_equal, :lexeme "==", :literal nil, :pos [1 7]}
                          {:type :number, :lexeme "2", :literal 2.0, :pos [1 10]}
                          {:type :eof, :lexeme "EOF", :literal nil, :pos [1 11]}]
                         0)
[{:left
  {:left {:value 1.0},
   :operator {:type :plus, :lexeme "+", :literal nil, :pos [1 3]},
   :right {:value 1.0}},
  :operator
  {:type :equal_equal, :lexeme "==", :literal nil, :pos [1 7]},
  :right {:value 2.0}}
 5]

Try doing that with a class that mutates current with every advance call.

Polymorphic dispatch

And I must note, that the resulting AST is printed as a sequence of plain maps, each element is actually a class:

cljlox.parser> (-> [{:type :number, :lexeme "1", :literal 1.0, :pos [1 1]}
                    {:type :plus, :lexeme "+", :literal nil, :pos [1 3]}
                    {:type :number, :lexeme "1", :literal 1.0, :pos [1 5]}
                    {:type :eof, :lexeme "EOF", :literal nil, :pos [1 6]}]
                   parse
                   first
                   class)
cljlox.ast.Binary

This is important because in Clojure we have two kinds of polymorphic dispatch. Earlier in this post I’ve mentioned protocols, which is a class-based dispatch. The other one is multimethod-based dispatch, which is a runtime data polymorphism.

Both allow for creating generic functions, just in a bit different flavor. For instance, here’s how multimethods are often described:

(defmulti interact (fn [a b] [(:type a) (:type b)]))

Here we have a generic function interact which expects two arguments, each containing a :type key, like:

(def ship
  {:type :ship,
   :name "Buran"})

(def rock
  {:type :asteroid,
   :mass {:value 420000 :unit :kg}})

Now we can define methods for these two to interact:

(defmethod interact [:ship :ship] [this other]
  (println (format "%s docks %s" (:name this) (:name other))))

(defmethod interact [:ship :asteroid] [this {{:keys [value unit]} :mass}]
  (println
   (format "%s destroys asteroid of mass %s %s"
           (:name this) value (name unit))))

(defmethod interact [:asteroid :ship] [{{:keys [value unit]} :mass} other]
  (println
   (format "asteroid of mass %s %s pierces the %s"
           value (name unit) (:name other))))

(defmethod interact [:asteroid :asteroid] [this other]
  (println
   (format "asteroid of mass %s %s collides with asteroid of mass %s %s"
           (-> this :mass :value) (-> this :mass :unit name)
           (-> other :mass :value) (-> other :mass :unit name))))

This means that the interact function receives two maps, takes a :type key from each, and forms a tuple [type-a type-b]. Then it looks as if it has a definition for said tuple, in our case we’ve defined a permutation of methods for :ship and :asteroid. So if we call interact with our objects, we would see this:

user> (interact ship rock)
Buran destroys asteroid of mass 420000 kg
nil
user> (interact rock ship)
asteroid of mass 420000 kg pierces the Buran
nil
user> (interact rock rock)
asteroid of mass 420000 kg collides with asteroid of mass 420000 kg
nil

It makes sense to use multimethods when dealing with data, and while our AST is data, I choose protocols, because we don’t really need to dispatch over a combination of values. Protocols are also more efficient, as they rely on JVM internal mechanisms for polymorphic dispatch. Additional reasoning was that at work we don’t really use Protocols, so I wanted to try them out in something involved.

A small downside is that it is a bit harder to represent plain data as AST nodes, as I have to use conversion functions, like ->Binary that takes a map, and converts it to a record. But it’s better to use Protocols here, so I don’t mind some extra preparation steps. Especially since it really only affects the tests, not the actual code.

Evaluating Expressions

And we’re back to Protocols vs Visitors.

The book, once again, uses the Visitor pattern to implement all necessary methods to evaluate our syntax tree as a program. I won’t go and repeat the comparison, instead, here’s how I’ve done it in CljLox:

;; cljlox.evaluator
(defprotocol IInterpretable
  (evaluate [self env locals]))

(extend-type Literal
  IInterpretable
  (evaluate [{value :value} _ _]
    value))

(extend-type Grouping
  IInterpretable
  (evaluate [{expr :expression} env locals]
    (evaluate expr env locals)))

;; ...

(extend-type Unary
  IInterpretable
  (evaluate [{:keys [operator right]} env locals]
    (let [right (evaluate right env locals)]
      (case (:type operator)
        :minus (do (check-number-op! operator right)
                   (- right))
        :bang (not (truth? right))
        (runtime-error "Unsupported unary operator" {:token operator})))))

;; ...

The key point here is that Protocols are open, it’s possible to extend them from other parts of the system. It’s doable in both ways too - you can extend an existing protocol with a new class, or you can give an existing class a new protocol. In this project, I use both extend-protocol and extend-type to provide all necessary method implementations.

Statements and State

Now we have to deal with the state.

(defonce *env (atom {}))

That’s it.

Control Flow

Implementing control flow is an interesting…

Just kidding, let’s talk about state.

Statements and State

So, state. Yes, I’m using an atom here to hold the state of the interpreter. Probably I could have used a volatile instead, but I chose atom for some reason. There are no threads in CljLox, at least not yet, so volatile would work just as fine. Later I switched to volatile, though.

I’ve changed/created the evaluate method for Var, Variable and Assign nodes of the Ast:

(extend-type Var
  IInterpretable
  (evaluate [{:keys [name initializer]}]
    (swap! *env
           assoc
           (:lexeme name)
           (when (some? initializer)
             (evaluate initializer)))
    nil))

(extend-type Variable
  IInterpretable
  (evaluate [{:keys [name]}]
    (let [val (get @*env (:lexeme name) ::not-found)]
      (if (= val ::not-found)
        (runtime-error (format "Undefined variable '%s'." (:lexeme name)) {:token name})
        val))))

(extend-type Assign
  IInterpretable
  (evaluate [{:keys [name value]}]
    (let [val (evaluate value)]
      (if (contains? @*env (:lexeme name))
        (swap! *env assoc (:lexeme name) val)
        (runtime-error (format "Undefined variable '%s'." (:lexeme name)) {:token name})))
    val))

This, however, only handles the global state, so I later had to change it to this:

(defn make-env
  ([] (make-env nil))
  ([parent]
   (atom {:enclosing parent
          :values {}})))

(defonce *global-env (make-env))

;; ...

(defn set-variable [env var value]
  (assoc-in env [:values var] value))

(extend-type Var
  IInterpretable
  (evaluate [{:keys [name initializer]} env]
    (swap! env
           set-variable
           (:lexeme name)
           (when (some? initializer)
             (evaluate initializer env)))
    nil))

(defn get-variable [env var]
  (let [env @env
        values (:values env)
        name (:lexeme var)]
    (if (contains? values name)
      (get values name)
      (if-some [enclosing (:enclosing env)]
        (recur enclosing var)
        (runtime-error (format "Undefined variable '%s'." name) {:token var})))))

(extend-type Variable
  IInterpretable
  (evaluate [{:keys [name]} env]
    (get-variable env name)))

(extend-type Assign
  IInterpretable
  (evaluate [{:keys [name value]} env]
    (let [val (evaluate value env)]
      (if (contains? @env (:lexeme name))
        (swap! env assoc (:lexeme name) val)
        (runtime-error (format "Undefined variable '%s'." (:lexeme name)) {:token name})))
    val))

(extend-type Block
  IInterpretable
  (evaluate [{:keys [statements]} env]
    (let [env' (make-env env)]
      (doseq [statement statements]
        (evaluate statement env')))))

That’s a lot of code to read, so I’ll outline the changes.

First, now the evaluate method accepts two arguments, instead of just one. Yes, we’re now passing the environment around. In a non-functional interpreter, we could just set the current environment of the Evaluator class, once we introduce new or leave a scope but here we can’t do that. Err, we can, I just chose not to, because again - it is much easier to reason about the code. The environment is just a map wrapped into an atom, after all.

Control Flow

Aside from adding a few new extensions to the protocol:

(extend-type If
  IInterpretable
  (evaluate [{:keys [condition then else]} env]
    (let [test (evaluate condition env)]
      (if (truth? test)
        (evaluate then env)
        (when else
          (evaluate else env))))))

(extend-type Logical
  IInterpretable
  (evaluate [{:keys [left operator right]} env]
    (let [left (evaluate left env)]
          (case (:type operator)
            :or (if (truth? left)
                  left
                  (evaluate right env))
            :and (if (not (truth? left))
                   left
                   (evaluate right env))))))

we also implement loops in this section. While I like the approach of implementing while in the host language, and for terms of our definition of while, honestly, I would prefer a tour about tail call elimination instead. It would probably make things more complicated, but I feel that it would bring more value to the table. Since this book teaches how to create a language, I think it is worth mentioning the tail call recursion approach at the very least.

When I was programming in C and had no idea an infinite recursion was even possible, learning that there are some languages that can do that was a huge revelation. How many algorithms could I have written in far less complicated ways, by using the continuation passing style? Yeah, a weird statement to make, but CSP can be easier to understand at times.

Functions

Now we’re getting into the meat of the interpreter. We can put together all our variables, scopes, and loops, and wrap them into something we can define and call later. Everything starts with a different protocol:

(defprotocol ICallable
  (call [self arguments token env]))

And later we can extend it with new types:

(extend-type LoxCallable
  ICallable
  (call [{:keys [arity function]} arguments token env]
    (if (= arity (count arguments))
      (apply function arguments)
      (runtime-error
       (format "Expected %s arguments but got %s." arity (count arguments))
       {:token token}))))

(extend-protocol ICallable
  Object
  (call [self & rest]
    (runtime-error "Can only call functions and classes."))
  nil
  (call [self & rest]
    (runtime-error "Can only call functions and classes.")))

Again, I really like this feature of Clojure - defining a new protocol and extending it with already existing types like Object or nil, is great. Same thing with already existing types, such as LoxCallable (used for native interpreter’s functions) - we can just give it a new protocol and implement it. No need for visitors, you just do:

(extend-type Function
  IInterpretable
  (evaluate [{:keys [name] :as self} env]
    (let [f (LoxFunction. self env)]
      (when name
        (swap! env assoc-in [:values (:lexeme name)] f))
      f)))

And then:

(defrecord LoxFunction [^Function declaration, closure]
  ICallable
  (call [{{:keys [params body]} :declaration} args _ _]
    (try
      (let [env (make-env closure)]
        (doseq [[arg val] (map vector params args)]
          (swap! env assoc-in [:values (:lexeme (:name arg))] val))
        (evaluate body env))
      (catch ExceptionInfo e
        (let [data (ex-data e)]
          (if (= :return (:type data))
            (:value data)
            (throw e))))))
  IStringable
  (tostring [self]
    (format "#<function: %s>" (->> self :declaration :name :lexeme))))

Every time I read things like visitFunctionStmt I question myself - what exactly visiting does do, why does it create an object, and so on? Well, I’m not a Java programmer for a reason, I guess, visitors, abstract fabrics, and other such things don’t tickle my fancy.

Resolving and Binding

I know, I’m being a bit shallow on details, but honestly, I just don’t remember well what was going on in my mind back then. The resolver, however, is another thing that I took the liberty to make immutable, so it’s quite different from the book. I have a giant commit where I basically change this in every protocol method:

 (defprotocol ICallable
-  (call [self arguments token env]))
+  (call [self arguments token env locals]))

Well, why would I do that? No reason, really. I just noticed that the only place I change locals is in the evaluation entry point and each new scope. However, none of these have to use the mutable state, as it is purely for convenience (and the style the book is written in), so I changed it.

(defn run
  ([source]
   (run source nil))
  ([source file]
   (let [fmt (if file (str file " %s") "%s")
         {:keys [errors tokens]} (tokenize source)]
     (if (seq errors)
        (with-out-err
          (doseq [error errors]
            (println (format fmt (str error)))))
-       (let [expressions (parse tokens)]
-         (reset! *locals {})
-         (doseq [expr expressions]
-           (lox-resolve expr []))
+       (let [expressions (parse tokens)
+             locals (reduce (fn [locals expr]
+                              (merge locals (second (lox-resolve expr [[] {}]))))
+                            {} expressions)]
          (reduce (fn [_ expr]
                    (when (seq expr)
-                     (interpret expr)))
+                     (interpret expr locals)))
                  nil expressions))))))

A similar change was needed in the resolve-local function, as now it would use an additional storage passed in as an argument:

-(defn- resolve-local [expr name scope-stack]
+(defn- resolve-local [expr name [scope-stack locals]]
   (when (seq scope-stack)
     (loop [i (dec (count scope-stack))]
       (if (contains? (get scope-stack i) (:lexeme name))
-        (register-expr-scope expr (- (count scope-stack) 1 i))
-        (when (> i 0)
-          (recur (dec i)))))))
+        [scope-stack (assoc locals expr (- (count scope-stack) 1 i))]
+        (if (> i 0)
+          (recur (dec i))
+          [scope-stack locals])))))

With the rest of the changes the resolver is fully immutable, but is it worth the effort? Well, kinda - it is much easier to debug, and I was able to walk the code, looking at when the locals appeared in the storage much easier. I could, theoretically, make the same change for the global scope storage, but that would be way too much work, and I was tired of this change already. That would help to deal with the annoyance that the scope storage refers to itself at some point, making the debugger unhappy and cycling infinitely during result inspection.

And yes, later I had to store locals in the volatile! storage to keep them in the REPL, but not because it is mandatory, I was just lazy to put it into the loop.

Classes

We’re entering the part of the book I don’t like, yay! As you know, I like Clojure, but not really because it is a Lisp. I like Clojure because it is a functional language without a type system burden.

If I were into functional programming and into type systems, I would be probably writing about Haskell or Idris, but I’m writing about Clojure, and more often Lua/Fennel. If I were into Lisp and LISP, I would probably be writing about Common Lisp or Scheme. But I’m into type-less, functional approach so Clojure is a perfect fit for me. Well, maybe Elixir as well, but they’re bringing a type system in, so I guess they’re off track on what makes the language fun. But I digress.

Out of all the features I dislike the most are classes and inheritance, and these are the final chapters of the first half of the book. I dislike classes and inheritance mainly because they are only complicated things in my past experience. I worked on a somewhat large code base in C++ before, and inheritance was a huge pain there.

Other languages that feature classes often over-complicate things as well - look at Java with its visitors, factories, and such. So I tend to dislike classes. To be fair, there are use cases for this paradigm, and there are fine implementations like CLOS or Smalltalk, but I would not try to base everything on classes.

Speaking of this book - why are we already at classes when we didn’t cover any base data structures? There are strings, yes, but where are arrays and records/hash-maps? What about tuples? Some could argue that strings can be used for arrays or their code could be repurposed, but not in our case, as we’re using the host platform’s strings which are immutable. Implementing strings as byte arrays is a thing, but it’s not what the book did, so there are simply no arrays in Lox.

Well, we can implement a linked list with classes, but there’s literally no way to implement an array without explicit access to the memory. I don’t really want to cover the changes in the code base, as there’s nothing you haven’t seen before in this post.

One thing I do want to cover though is the use of classes to implement namespaces. This is covered in the “Challenges” section after the actual chapter:

We have methods on instances, but there is no way to define “static” methods that can be called directly on the class object itself. Add support for them. Use a class keyword preceding the method to indicate a static method that hangs off the class object.
class Math {
  class square(n) {
    return n * n;
  }
}

print Math.square(3); // Prints "9".
You can solve this however you like, but the “metaclasses” used by Smalltalk and Ruby are a particularly elegant approach. Hint: Make LoxClass extend LoxInstance and go from there.

Saying I dislike this approach is to not say anything. Why would you do that? It’s like stating: “Our language can’t do namespaces but do not worry, we have static methods that provide exactly the same thing, you just do it the ugly way”. Well, it probably makes sense in some languages like Smalltalk, but not everything is Smalltalk.

Take a look at Lua for a second. It doesn’t have namespaces. It doesn’t have classes either. Yet, it does have a way of defining modules - it just uses hash tables for that. And it does the same thing with classes - you use a hash table for that. You don’t have to be Clojure to just use maps!

Then the book proceeds:

Most modern languages support “getters” and “setters”—members on a class that look like field reads and writes but that actually execute user-defined code. Extend Lox to support getter methods. These are declared without a parameter list. The body of the getter is executed when a property with that name is accessed.

Every time I see this, I ask myself “Why?!” - a pointless question.

Look at this code:

class Circle {
  init(radius) {
    this.radius = radius;
  }

  area {
    return 3.141592653 * this.radius * this.radius;
  }
}

var circle = Circle(4);
print circle.area; // Prints roughly "50.2655".

Accessing the area field we get the area of a circle with the radius of 4 units. Cool, right?

Now look at this:

class Circle {
  init(radius) {
    this.radius = radius;
  }

  area {
    while(true) {}
  }
}

var circle = Circle(4);
print circle.area; // good luck debugging this

Surely, looking at the circle.area part of the code you can’t see that there’s a call there. That’s why we have syntax, mind you! This “feature” to me looks like a deliberate shot to the foot - the expression power it gives you is equal to not hitting two keys on the keyboard yet you introduce ambiguity to your language.

It was at this point where I stopped even trying to do any of the so-called “Challenges”, but I will have to talk about them later, as I do have more things to say.

Inheritance

Most of the changes in this section are sorely related to how we access methods, fields, and so on, so again, I won’t go into actual code differences. Call me lazy.

I’m glad that the book went for a single inheritance approach though, as nothing gives me a stronger desire to punch someone than seeing a class that inherits from another 16, each of which also inherits from more than three. I have seen this in the C++ codebase I mentioned before. Looking up where the method is defined is hard enough, and this was a C++ written before things like language servers were a thing, so tags were the only way to navigate it in a somewhat intelligent way. Unfortunately, generating tags was slow, and I had to update them frequently, as I made the changes to the code. Often, the jump wasn’t possible because tags were outdated.

So, having a single inheritance is, perhaps, limiting as you can’t go nuts and say that Cat inherits from both Animal and Carnivore, which may be a bummer, I guess, but this is a textbook example, I have yet to see OOP code like that. Personally, I’ll take single inheritance over multiple inheritance any time. But I’d rather do no inheritance at all, to be honest, protocols are much more fun to use.

Final thoughts

This post is a total trainwreck. I should have published it in the middle of April 2022 but for some reason, I didn’t. To be fair, I still have a draft of the first version of this post, and it is horrible, so I guess it’s good that I didn’t publish it before. Such posts should be written during the time you spend with the book, not afterward, and especially not after almost two years.

So that’s why I’ll be re-writing the second half of the book in Zig and capturing my thoughts as I go through it! For the second time.

Yes, I already tried this approach, but given that I didn’t know Zig at all back then the resulting code is terrible. Can’t say I’m proud about the Clojure version either, but the Zig version is heaps and bounds worse in all areas. Hopefully, this time around I will be able to do a better job.

I’ve mentioned that I have more things to say about the “Challenges”, waiting for the reader at each chapter’s end. I guess, this will have to wait for the second post, which I’m sure won’t take years to appear. Ahem.

You see, most of the challenges, or should I say, exercises were not really out of the way in the first half of the book. Can’t say so for the second half though - I had major issues going through the book because I completed the challenges. This wasn’t an issue for the first book because most challenges are easy, but this can’t be said for the second half, as we’re no longer in the managed territory. Implementing your bytecode interpreter is much harder than using your host language for the heavy lifting.

Anyway, I hope you found this post interesting enough to read til here. If not, well, that’s fair, I didn’t find it interesting to read too, honestly. But I had to get it out of my system, and hopefully, the second part will be more refined.

Comment via email

Andrey Listopadov

Clojure on Fennel part one: Persistent Data Structures

Proper persistent data structures

Persistent Hash Map

Persistent Vector

Persistent Red-Black Tree

Persistent List

Persistent Queue

ClojureFnl

Slopes in AABB collision systems

AABB collisions

Implementing slopes with bump.lua

TIC-80

Creating a project and adding bump.lua

Creating sprites and using the map editor

Loading entities to the world

Working with bump.lua

Handling mouse events

Implementing slopes with AABB collisions

Calculating slope collision coordinates

Handling slope collisions

Conclusion

2025 Recap

Work stuff

Music

Gaming

New hobbies

Trips

Plans for 2026

MacOS Music Players

Strawberry

Audacious

Cog

Music

VLC

MPV

Museeks

Others

GNOME is better macOS than macOS

Good old days of Linux

A disclaimer

A new laptop

macOS

The Desktop

Virtual desktops

Fullscreen and maximize button

Mission Control

The Dock

The Finder

Files and folders

The media viewer

Hardware accessibility

The Microphone

The keyboard layout

The keyboard layout (physical)

No middle click on the touchpad

The camera cutout

The good parts

Replacing clojure-lsp with clj-kondo and Refactor-nREPL

Disabling Clojure-LSP

Bringing back refactoring

Symbol renaming

Problems with refactor-nrelp

Not a bashing on Clojure LSP

Implementing SSE with fnl-http and async.fnl

SSE stream

Reading the event stream and triggering events

Conclusion1

Effects of Slow and Fast systems

Fast systems

Interactive coding

Notebooks

LLM-assisted coding

Fast knowledge system

Slow systems

Slow hardware

Slow tools and workflows

Slow knowledge system

Effects of Slow and Fast systems

Implementing dynamic scope for Fennel and Lua

Implementing slopes with `bump.lua`

Creating a project and adding `bump.lua`

Working with `bump.lua`

Problems with `refactor-nrelp`

Conclusion¹

`interpolation.lua`

The `deps.fnl` file format

Runtime library manipulation with `deps.add-lib`, and `sync-deps`

`--tree`

`--fennel-ls`

`--profiles`

`deps.fnl`