Andrey Listopadov

Clojure on Fennel part one: Persistent Data Structures

Tue, 07 Apr 2026 02:47:00 +0300

Somewhere in 2019 I started a project that aimed to bring some of Clojure features to Lua runtime - fennel-cljlib. It was a library for Fennel that implemented a basic subset of clojure.core namespace functions and macros. My goal was simple - I enjoy working with Clojure, but I don’t use it for hobby projects, so I wanted Fennel to feel more Clojure-like, besides what it already provides for that.

This library grew over the years, I implemented lazy sequences, added immutability, made a testing library, inspired by clojure.test and kaocha, and even made a port of clojure.core.async. It was a passion project, I almost never used it to write actual software. One notable exception is fenneldoc - a tool for documentation generation for Fennel libraries. And I haven’t seen anyone else use it for a serious project.

The reason for that is simple - it was an experiment. Corners were cut, and Fennel, being a Clojure-inspired lisp is not associated with functional programming the same way Clojure is. As a matter of fact, I wouldn’t recommend using this library for anything serious… yet.

Recently, however, I started a new project: ClojureFnl. This is a Clojure-to-Fennel compiler that uses fennel-cljlib as a foundation. It’s still in early days of development, but I’ve been working on it for a few months in private until I found a suitable way to make things work in March. As of this moment, it is capable of compiling most of .cljc files I threw at it, but running the compiled code is a different matter. I mean, it works to some degree, but the support for standard library is far from done.

;; Welcome to ClojureFnl REPL
;; ClojureFnl v0.0.1
;; Fennel 1.6.1 on PUC Lua 5.5
user=> (defn prime? [n]
         (not (some zero? (map #(rem n %) (range 2 n)))))
#<function: 0x89ba7c550>
user=> (for [x (range 3 33 2)
             :when (prime? x)]
         x)
(3 5 7 11 13 17 19 23 29 31)
user=>

However, there was a problem.

My initial implementation of immutable data structures in the itable library had a serious flaw. The whole library was a simple hack based on the copy-on-write approach and a bunch of Lua metatables to enforce immutability. As a result, all operations were extremely slow. It was fine as an experiment, but if I wanted to go further with ClojureFnl, I had to replace it. The same problem plagued immutableredblacktree.lua, an implementation of a copy-on-write red-black tree I made for sorted maps. It did a full copy of the tree each time it was modified.

For associative tables it wasn’t that big of a deal - usually maps contain a small amount of keys, and itable only copied levels that needed to be changed. So, if you had a map with, say, ten keys, and each of those keys contained another map with ten keys, adding, removing or updating a key in the outer map meant only copying these ten keys - not the whole nested map. I could do that reliably, because inner maps were immutable too.

But for arrays the story is usually quite different. Arrays often store a lot of indices, and rarely are nested (or at least not as often as maps). And copying arrays on every change quickly becomes expensive. I’ve mitigated some of the performance problems by implementing my version of transients, however the beauty of Clojure’s data structures is that they’re quite fast even without this optimization.

Proper persistent data structures

Clojure uses Persistent HAMT as a base for its hash maps and sets, and a bit-partitioned trie for vectors. For sorted maps and sets, Clojure uses an immutable red-black tree implementation, but as far as I know it’s not doing a full copy of the tree, and it also has structural sharing properties.

I started looking into existing implementations of HAMT for Lua:

hamt.lua (based on mattbierner/hamt)
- seemed incomplete
ltrie
- no transients
- no hashset
- no ordered map (expectable, different algorithm)
- no compound vector/hash
Michael-Keith Bernard’s gist
- no custom hashing

I could use one of those, notably ltrie seemed the most appropriate one, but given that I’m working on a fennel library that I want later to embed into my Clojure compiler I needed a library implemented in Fennel.

So I made my own library: immutable.fnl. This library features HAMT-hash maps, hash-sets, and vectors, as well as a better implementation of a persistent red-black tree, and lazy linked lists.

Persistent Hash Map

I started the implementation with a Persistent HAMT with native Lua hashing. The data structure itself is a Hash Array Mapped Trie (HAMT) with 16-factor branching. Thus all operations are O(Log16 N), which is effectively O(1) for a practical amount of keys.

As far as I know, Clojure uses branching factor of 32, but for a Lua runtime this would mean that the popcount would be more expensive, and despite a shallower tree, each mutation would need to copy a larger sparse array. With branching factor of 16 a map with 50K entries is ~4 levels deep, which would be ~3 with 32 branching factor. So my logic was that it’ll be a compromise, especially since Lua is not JVM when it comes to performance.

Of course, it’s not as fast as a pure Lua table, which is to be expected. Lua tables are implemented in C, use efficient hashing, and dynamically re-allocated based on key count. So for my implementation most operations are a lot slower, but the total time for an operation is still usable.

Here are some benchmarks:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Regular operations are notably slower, when compared to Lua:

Operation	Lua table	Persistent HashMap	Ratio	per op
insert 50000 random keys	2.05 ms	164.80 ms	80.3x slower	3.3 us
lookup 50000 random keys	0.83 ms	92.51 ms	110.8x slower	1.9 us
delete all	0.78 ms	170.78 ms	219.8x slower	3.4 us
delete 10%	0.14 ms	19.50 ms	136.4x slower	3.9 us
iterate 50000 entries	1.74 ms	6.64 ms	3.8x slower	0.133 us

For transients the situation is a bit better, but not by much:

Operation	Lua table	Transient HashMap	Ratio	per op
insert 50000 random keys	2.05 ms	89.17 ms	43.5x slower	1.8 us
delete all	0.76 ms	104.31 ms	138.0x slower	2.1 us
delete 10%	0.16 ms	12.71 ms	82.0x slower	2.5 us

On LuaJIT numbers may seem worse, but per-operation cost is much lower, it’s just that native table operations are so much faster:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	Persistent HashMap	Ratio	per op
insert 50000 random keys	0.86 ms	49.05 ms	56.8x slower	0.981 us
lookup 50000 random keys	0.27 ms	14.21 ms	53.4x slower	0.284 us
delete all	0.13 ms	48.63 ms	374.1x slower	0.973 us
delete 10%	0.05 ms	6.49 ms	138.1x slower	1.3 us
iterate 50000 entries	0.07 ms	1.80 ms	27.7x slower	0.036 us

Operation	Lua table	Transient HashMap	Ratio	per op
insert 50000 random keys	0.76 ms	22.43 ms	29.6x slower	0.449 us
delete all	0.15 ms	34.16 ms	232.4x slower	0.683 us
delete 10%	0.04 ms	5.02 ms	132.1x slower	1.0 us

With a branching factor of 32 the situation gets worse on PUC Lua, but is slightly better on LuaJIT. So there’s still space for fine-tuning.

For hashing strings and objects I decided to use djb2 algorithm. I am almost as old as this hash function, so seemed like a good fit. JK. The main reason to use it was that it can be implemented even if we don’t have any bit-wise operators, and Lua doesn’t have them in all of the versions. It only uses +, *, and % arithmetic operators, so can be done on any Lua version. It’s prone to collisions, and I try to mitigate that by randomizing it when the library is loaded.

Still, collisions do happen, but HAMT core ensures that they will still resolve correctly by implementing a deep equality function for most objects.

However, when first working on this, I noticed this:

>> (local hash-map (require :io.gitlab.andreyorst.immutable.PersistentHashMap))
nil
>> (local {: hash} (require :io.gitlab.andreyorst.immutable.impl.hash))
nil
>> (hash (hash-map :foo 1 :bar 2))
161272824
>> (hash {:foo 1 :bar 2})
161272824
>> (hash-map (hash-map :foo 1 :bar 2) 1 {:foo 1 :bar 2} 2)
{{:foo 1 :bar 2} 2}

This is an interesting loophole. What object ended up in our hash map as a key - our persistent map or plain Lua table? Well, that depends on insertion order:

>> (each [_ k (pairs (hash-map (hash-map :foo 1 :bar 2) 1 {:foo 1 :bar 2} 2))]
     (print (getmetatable k)))
IPersistentHashMap: 0x824d9b570
nil
>> (each [_ k (pairs (hash-map {:foo 1 :bar 2} 2 (hash-map :foo 1 :bar 2) 1))]
     (print (getmetatable k)))
nil
nil

To reiterate, I’m creating a hash map, with a key set to another persistent hash map, and then insert a plain Lua table with the same content. The Lua table hashes to exactly the same hash, and goes into the same bucket, but there’s no collision, because objects are equal by value. But equality of mutable collections is very loosely defined - it may be equal right now, but the next time you look at it, it’s different. So a different hashing was needed for persistent collections, to avoid these kinds of collision. I ended up salting persistent collections with their prototype address in memory.

Other than that, the HAMT implementation is by the book, and the rest is the interface for interacting with maps.

Main operations:

new - construct a new map of key value pairs
assoc - associate a key with a value
dissoc - remove key from the map
conj - universal method for association, much like in Clojure
contains - check if key is in the map
count - map size, constant time
get - get a key value from a map
keys - get a lazy list of keys
vals - get a lazy list of values
transient - convert a map to a transient

Coercion/conversion:

from - create a map from another object
to-table - convert a map to a Lua table
iterator - get an iterator to use in Lua loops

Transient operations:

assoc! - mutable assoc
dissoc! - mutable dissoc
persistent - convert back to persistent variant, and mark transient as completed

This covers most of the needs in my fennel-cljlib library, as anything besides it I can implement myself, or just adapt existing implementations.

A Persistent Hash Set is also available as a thin wrapper around PersistentHashMap with a few method changes.

A note on PersistentArrayMap.

In Clojure there is a second kind of maps that are ordered, not sorted, called a Persistent Array Map. They are used by default when defining a map with eight keys or less, like {:foo 1 :bar 2}. The idea is simple - for such a small map, a linear search through all keys is faster than with a HAMT-based map.

However, in my testing on the Lua runtime, there’s no benefit in this kind of a data structure, apart from it being an ordered variant. Lookup is slower, because of a custom equality function, which does deep comparison.

Persistent Vector

Persistent Vectors came next, and while the trie structure is similar to hash maps, vectors use direct index-based navigation instead of hashing, with a branching factor of 32. Unlike maps, vector arrays in the HAMT are more densely packed, and therefore a higher branching factor is better for performance. So lookup, update, and pop are O(log32 N), append can be considered O(1) amortized.

Still, compared to plain Lua sequential tables the performance is not as good:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Operation	Lua table	Persistent Vector	Ratio	per op
insert 50000 elements	0.19 ms	21.07 ms	109.7x slower	0.421 us
lookup 50000 random indices	0.47 ms	14.05 ms	29.7x slower	0.281 us
update 50000 random indices	0.32 ms	70.04 ms	221.6x slower	1.4 us
pop all 50000 elements	0.25 ms	24.34 ms	96.2x slower	0.487 us
iterate 50000 elements	0.63 ms	10.16 ms	16.2x slower	0.203 us

Operation	Lua table	Transient Vector	Ratio	per op
insert 50000 elements	0.19 ms	7.81 ms	40.3x slower	0.156 us
update 50000 random indices	0.33 ms	20.76 ms	62.4x slower	0.415 us
pop all 50000 elements	0.25 ms	11.14 ms	44.4x slower	0.223 us

On LuaJIT:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	Persistent Vector	Ratio	per op
insert 50000 elements	0.10 ms	7.62 ms	74.0x slower	0.152 us
lookup 50000 random indices	0.06 ms	0.67 ms	11.8x slower	0.013 us
update 50000 random indices	0.04 ms	29.13 ms	710.4x slower	0.583 us
pop all 50000 elements	0.02 ms	8.62 ms	410.4x slower	0.172 us
iterate 50000 elements	0.02 ms	0.57 ms	28.7x slower	0.011 us

Operation	Lua table	Transient Vector	Ratio	per op
insert 50000 elements	0.05 ms	0.59 ms	11.6x slower	0.012 us
update 50000 random indices	0.04 ms	2.06 ms	51.6x slower	0.041 us
pop all 50000 elements	0.02 ms	0.84 ms	46.7x slower	0.017 us

I think this is an OK performance still. Vectors don’t use hashing, instead it is a direct index traversal via bit-shifting, so there’s no hashing, just index math.

Operations on vectors include:

new - constructor
conj - append to the tail
assoc - change a value at given index
count - element count (constant time)
get - get value at given index
pop - remove last
transient - convert to a transient
subvec - create a slice of the vector in constant time

Transient operations:

assoc! - mutable assoc
conj! - mutable conj
pop! - mutable pop
persistent - convert back to persistent and finalize

Interop:

from - creates a vector from any other collection
iterator - returns an iterator for use in Lua loops
to-table - converts to a sequential Lua table

One notable difference in both vector and hash-map is that it allows nil to be used as a value (and as a key, in case of the hash-map). Vectors don’t have the same problem that Lua sequential tables have, where length is not well-defined if the table has holes in it.

It’s a debate for another time, whether allowing nil as a value (and especially as a key) is a good decision to make, but Clojure already made it for me. So for this project I decided to support it.

Persistent Red-Black Tree

For sorted maps and sorted sets I chose Okasaki’s insertion and Germane & Might’s deletion algorithms. Most of the knowledge I got from this amazing blog post by Matt Might.

I believe the operations are O(Log N), as for any binary tree, but given that the deletion algorithm is tricky, I’m not exactly sure:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on PUC Lua 5.5

Operation	Lua table	PersistentTreeMap	Ratio	per op
insert 50000 random keys	2.10 ms	209.23 ms	99.8x slower	4.2 us
lookup 50000 random keys	0.88 ms	82.97 ms	94.2x slower	1.7 us
delete all	0.74 ms	173.76 ms	234.8x slower	3.5 us

On LuaJIT:

Median time over 7 rounds (1 warmup discarded), N = 50000 elements. GC stopped during measurement. Clock: os.clock (CPU). Runtime: Fennel 1.7.0-dev on LuaJIT 2.1.1774896198 macOS/arm64

Operation	Lua table	PersistentTreeMap	Ratio	per op
insert 50000 random keys	0.72 ms	101.08 ms	140.4x slower	2.0 us
lookup 50000 random keys	0.25 ms	12.67 ms	49.9x slower	0.253 us
delete all	0.14 ms	56.14 ms	403.9x slower	1.1 us

The API for sorted maps and sets is the same as to their hash counterparts with a small difference - no transients. Clojure doesn’t do them, and I’m not doing them too.

That’s all for benchmarks. I know that there are many problems with this kind of benchmarking, so take it with a grain of salt. Still, the results are far, far better than what I had with itable.

But there are two more data structures to talk about.

Persistent List

As I mentioned, I made a lazy persistent list implementation a while ago but it had its problems and I couldn’t integrate that library with the current one well enough.

The main problem was that this library uses a single shared metatable per data structure, and the old implementation of lazy lists didn’t. This difference makes it hard to check whether the object is a table, hash-map, list, vector, set, etc. So I reimplemented them.

The reason for old implementation to use different metatables was because I decided to try the approach described in Reversing the technical interview post by Kyle Kingsbury (Aphyr). I know this post is more of a fun joke, but it actually makes sense to define linked lists like that in Lua.

See, tables are mutable, and you can’t do much about it. Closures, on the other hand are much harder to mutate - you can still do it via the debug module, but it’s hard, and it’s not always present. So storing head and tail in function closures was a deliberate choice.

However, it meant that I needed to somehow attach metadata to the function, to make it act like a data structure, and you can’t just use setmetatable on a function. Again, you can do debug.setmetatable but all function objects share the same metadata table. So, while you can do fancy things like this:

>> (fn comp [f g] (fn [...] (f (g ...))))
#<function: 0x7bdb320a0>
>> (debug.setmetatable (fn []) {:__add comp})
#<function: 0x7bd17f040>
>> ((+ string.reverse string.upper) "foo")
"OOF"

You can also notice, that our + overload applied to functions in the string module.

So instead, we use a table, and wrap it with a metatable that has a __call metamethod, essentially making our table act like a function. This, in turn means, that we have to create two tables per list node - one to give to the user, the other to set our __call and use it as a meta-table.

Convoluted, I know. It’s all in the past now - current implementation is a simple {:head 42 :tail {...}} table. Not sure what is worse.

But that meant that I had to rework how lazy lists worked, because previously it was just a metatable swap. Now list stores a “thunk”, that when called replaces itself in the node with the :head and :tail keys. Unless it’s an empty list, of course - in that case we swap the metatable to an empty list one.

So Lists have three metatables now:

IPersistentList
IPersistentList$Empty
IPersistentList$Lazy

Instead of god knows how many in the old implementation.

The list interface is also better now. Previously it was hardcoded how to construct a list from a data structure. Current implementation also hardcodes it, but also allows to build a list in a lazy way from an iterator.

This is better, because now a custom data structure that has weird iteration schema (like maps and sets in this library), we still can convert it to a list. A general case is just:

(PersistentList.from-iterator #(pairs data) (fn [_ v] v))

Meaning that we pass a function that will produce the iterator, and a function to capture values from that iterator. Reminds me of clojure transducers in some way.

Persistent Queue

And the final data structure - a persistent queue. Fast append at the end, and also fast remove from the front.

It’s done by holding two collections - a linked list at the front, and a persistent vector for the rear. So removing from the list is O(1), and appending to the vector is also pretty much O(1).

Interesting things start to happen when we exhaust the list part - we need to move vector’s contents into the list. It is done by calling PersistentList.from on the rear. And building a list out of a persistent vector is an O(1) operation as well! Well, because nothing happens, we simply create an iterator, and build the list in a lazy way. But since indexing the vector is essentially ~O(1), we can say that we still retain this property.

Or at least that’s how I reasoned about this - I’m not that good with time-complexity stuff.

ClojureFnl

That concludes part one about ClojureFnl.

I know that this post was not about ClojureFnl at all, but I had to fix my underlying implementation first. Now, that I have better data structures to build from, I can get back working on the compiler itself. So the next post will hopefully be about the compiler itself.

Unless I get distracted again.

Comment via email

Lazy sequences and iterators

Sat, 09 Oct 2021 21:34:00 +0300

Today’s topic will be about lazy sequences and how these are different from iterators. I’ve wanted to make an article on this topic for some time, but unfortunately, there was no good way to show the differences using a single language (that I know), because usually, languages stick to one of those things. But since I’ve been into library writing lately, I wrote a lazy sequence library for the Fennel language, and because Fennel compiles to Lua, it makes Lua a great candidate for this article.

Lua has first-class support for iterators, as it is kind of a central paradigm very closely related to tables. And because iterators are so common in Lua, some libraries even treat them as sequences. However, iterators are not sequences, even though they can be used similarly. And proper sequences also can be created from tables much like iterators and would have similar use cases, without some problems that iterators have. So today we’ll look into Lua iterators, and lazy sequences, and will try to understand their differences.

Iterators in Lua

Let’s begin by understanding what iterators are, and what key features they provide.

In Lua, iterators are usually higher-order functions, produced by another function, based on its input. A standard way to get an iterator over a table is to call pairs function:

get_next, t, k = pairs({a = 1, b = 2, c = 3})

Lua supports multiple return values, so calling pairs returns us the iterator function as the first value, a table (our original table in this case), that we’re going to iterate as the second value, and the starting key, which in our case is nil.

Calling this get_next function, and passing it our table t, and the key k, we get back a pair of values, representing the key and a value associated with that key:

get_next(t, k) -- b	2

We then can use the returned key as an input to the next call to get_next and get the next key-value pair, repeating the process until get_next returns nil:

get_next(t, "b") -- c	3
get_next(t, "c") -- a	1
get_next(t, "a") -- nil

nil means our table is exhausted, and no more keys are present in it. The order may be different for you, as it depends on hashing.

This get_next is a stateless iterator - it doesn’t have any hidden state inside it, which means, that it can be reused. For example, we can pass a different table to the get_next function, and it will work on it without any problem:

get_next({f = 42, g = 27}, nil) -- g	27

This pattern is quite common, so Lua has such a function in its standard library, it is called next and is a basis for all table iterators created with pairs. In addition to pairs, Lua provides another iterator constructor called ipairs, which is used to iterate over sequential tables, and it works similarly to pairs except uses integers as input keys.

While stateless iterators are good, it’s not always efficient, and sometimes not even possible to create a stateless iterator. For example, opening a file in Lua and calling lines method on it creates an iterator over this file, which is a stateful iterator that can be exhausted, i.e. you can call it only so many times as there are lines in the file. Let’s create a file example.txt with the following contents, and create an iterator over it:

line 1
line 2
line 3

file = io.open("example.txt", "r")
get_next_line = file:lines()
get_next_line() -- line 1
get_next_line() -- line 2
get_next_line() -- line 3
get_next_line() -- nil

Note that even though we’re calling the get_next_line function without any arguments, like the file, or the line number, it always returns the next line of the file. This is an example of a stateful iterator, which we can only use once - even if we call the lines method on the file object again, the returned iterator will be exhausted:

new_get_next_line = file:lines()
new_get_next_line() -- nil

This happens because the state is held in the file handle itself, so we need to reopen the file to get a new iterator. Because of this you usually want to open the file, walk it, use or cache needed lines, and close the resource, ideally doing all of this in a protected call.

And lastly, let’s create our own stateful iterator, that keeps its state in the closure. For the purposes of this example, let’s create a range iterator, that can create various ranges based on initial input:

function range(lower, upper, step)
  local lower, upper, step = lower or 0, upper or math.huge, step or 1
  return function ()
    local tmp = lower
    lower = lower + step
    if tmp < upper then
      return tmp
    end
  end
end

To keep it simple I’ve left some corner cases out, but this is basically the idea for a range iterator. When we call range, local variables lower, upper, and step are created based on given arguments or their default values. Then an anonymous function is returned, which refers to these local variables as closures, and sets the lower variable to a new value each time it is called:

r = range(1, 4)
r() -- 1
r() -- 2
r() -- 3
r() -- nil

As you can see, when we exhaust our iterator it returns nil, similarly to iterators created with pairs or file iterators. Thanks to this, we can use it in Lua’s for statement:

for x in range(1, 10, 3) do
  print(x)
end
-- 1
-- 4
-- 7

So now, when we more or less know what iterators are, and how they’re defined, let’s discuss their key properties:

Iterators are lazy,
Stateless iterators accept some kind of state (usually an object and a key) as arguments to produce the next key-value pair, and hence can be reused,
Stateful iterators are like cursors, and can’t be reused unless created anew.

Laziness is a very good property of iterators - it allows us to create actual infinite streams of data, computed on demand. For example, we can open really huge, or even infinite files, and read those line by line without consuming more memory than is actually needed to hold a single line. Alternatively, as was shown with the range iterator above, we can create an infinite range, by not supplying an upper border for it, and it will use the math.huge value, which is inf in Lua. So if Lua had arbitrary precision math, the range would grow to infinity, but in the case of this function, it will just overflow again and again.

However, while laziness is an extremely handy and important part of iterators, their benefits pretty much end on it. Iterator is basically a one-shot thing - you create it, use it until it is exhausted, or you’ve accomplished what you wanted to do with them, and then you never use the same iterator again. This is due to the fact, that iterators were not meant to be shared - you can’t tell if an iterator is stateless or stateful, and if it is a stateful iterator, there’s no safe way to share it. Someone can exhaust it, and later users will not get any values. Stateless iterators are less dangerous to share, but the usefulness of this is also significantly lower, as you need to pass the context along with an iterator.

All of this can be summarized to one point - iterators are not a data structure. And while this is not a bad thing, the usefulness of the iterator is limited because of this. So what if we wanted to share an iterator, and also make sure, that it doesn’t have problems with stateful iterators, and can be used as a data structure? That’s where lazy sequences come in handy!

Lazy Sequences

What are sequences? Think of a lazy sequence as a singly-linked list, with the difference that the tail may not be an actual cell, but a special kind of lazily evaluated cell. In order to create a sequence, we need to implement a so-called cons cell:

function cons (head, tail)
  return function (ht)
    if ht then
      return head
    else
      return tail
    end
  end
end

Conses are an old term that comes from the LISP language and basically represents a pair. It’s a data structure that holds two elements - a value and a pointer to the next value or a cons cell. This is usually illustrated with these boxes:

These stacked boxes represent a single cons cell - a pair of boxes, each referring to a value. The first box refers to 1, and the second box refers to another cons cell, which in turn refers to 2, and finally to nothing, indicating the end of the list. This data structure suits our task quite well, but you may notice that our implementation isn’t exactly a data structure, but a function. So how do we work with it then?

We need two primitive operations. In LISP these are historically called car and cdr, but it’s not meaningful to use these names in the context of Lua, so let’s call them first and rest:

function first (cell)
  return cell(true)
end

function rest (cell)
  return cell(false)
end

Since our cons cell is a function of one argument with if statement in it, first will pass true to it, to get the first element, and rest will pass false to get the other element of a cons cell. Here’s a quick demonstration:

list = cons(1, cons(2, cons(3, nil))) -- 1 -> 2 -> 3 -> nil
first(list) -- 1
first(rest(rest(list))) -- 3

We’ve created a list variable, that holds a single-linked list with values 1, 2, 3. And the final value is nil, which is needed to indicate the end of the list since the last cons cell must refer to nothing. However, this list is not lazy just yet.

Let’s recap how iterators achieve laziness. Iterators produce the next value only when you call them, which makes them very similar to generators by nature. However, the key difference from lazy sequences is that all previous results are thrown away and not kept with the iterator.

So how we can achieve laziness in our sequences? We can try a naive approach of wrapping arguments into functions, and realizing them when a value is requested:

function lazy_cons (headf, tailf)
  return function (ht)
    if ht then
      return headf()
    else
      return tailf()
    end
  end
end

Now, when constructing a list, instead of passing the elements directly, we need to pass some functions that will return the list’s elements:

lazy_list = lazy_cons(
  function () print "realize head" return 1 end,
  function () print "realize tail" return 2 end
)

Because the interface is the same, we can use first and rest to get values:

first(lazy_list)
-- realize head
-- 1
rest(lazy_list)
-- realize tail
-- 2

You can see, that when we’re calling first on our lazy_list it realizes the value, and then returns it. This is very close to our goal, but we’re not there yet. The problem is, that if we call first on lazy_list again, it will call the realization function again and again without caching the result, which is not right:

first(lazy_list)
-- realize head
-- 1
first(lazy_list)
-- realize head
-- 1

To solve this issue, we’ll need to rewrite lazy_cons in a such way, that it will realize our cell only once, and cache the result somehow. This can be done by creating mutable closures, but I think there’s a better way, that will also allow us to do more interesting stuff later. And it is also kinda hard to differentiate if the closure is uninitialized or the realization set it to nil. So instead we can use a clever metatable trick to realize cons cell contents upon access:

function lazy_cons (headf, tailf)
  local cell = {}
  local function realize ()
    local head, tail = headf(), tailf()
    local function realized (_, ht)
      if ht then
        return head
      else
        return tail
      end
    end
    return setmetatable(cell, {__call = realized})
  end
  return setmetatable(cell, {__call = function (_, ht) return realize()(ht) end})
end

Basically, we create a table cell, to which we set a metamethod __call that works similarly to the function that is returned by cons, except not quite. Note the realize()(ht) call in this metamethod - it calls the realize function first, then invokes the resulting function. The realize function is defined above, and this function resets the __call metamethod to another function, that uses cached values. Now using first on a such cell will work as expected:

lazy_list = lazy_cons(
  function () print "realize head" return 1 end,
  function () print "realize tail" return 2 end
)

first(lazy_list)
-- realize head
-- realize tail
-- 1
first(lazy_list)
-- 1
rest(lazy_list)
-- 2

It realizes the cell and then caches the results in a closure, so when we call first on this cell later, we get the same values every time. Now we can create and manipulate lazy sequences with only these three basic functions! Here’s our list example from above, but now lazy:

lazy_list = lazy_cons(
  function () print "first" return 1 end,
  function ()
    return lazy_cons(
      function () print "second" return 2 end,
      function ()
        return lazy_cons (
          function () print "third" return 3 end,
          function () return nil end
        )
      end
    )
  end
)

And we can inspect it with the same first and rest functions:

first(lazy_list)
-- first
-- 1
first(rest(rest(lazy_list)))
-- second
-- third
-- 3
first(lazy_list)
-- 1

That’s pretty much it! We have a way of constructing lazy sequences via a very primitive lazy_cons function, and anonymous functions as arguments. The usefulness of this may not be obvious, so let’s see some more examples of how this can be utilized before we move on.

More examples

Let’s create a lazy sequence that works similarly to the range iterator we did earlier:

function range_seq(lower, upper, step)
  local lower, upper, step = lower or 0, upper or math.huge, step or 1
  return lazy_cons(
    function () return lower end,
    function ()
      local tmp = lower + step
      if tmp < upper then
        return range_seq(tmp, upper, step)
      else
        return cons(nil, nil)
      end
    end
  )
end

A bit more verbose, but notice that this is no longer a stateful thing! Instead, it is a recursive function, which returns a self-referencing cons cell, which calls itself with changed arguments.

A recursion? How’s it lazy then?

The thing is, that it’s not even a recursion! The next recursion step will only happen when a cell is realized, and it will simply produce another lazy cell, which will not go to the next recursion step immediately. And even if Lua had no tail call optimization it would still work just fine, as it acts more like a trampoline, and the call stack never actually grows. Here’s how we can see that our range_seq is working:

r = range_seq(1, 4)
first(rest(rest(r)))
-- 3
first(rest(rest(rest(r))))
-- nil

And for infinite ranges we can drop as many values before working with the range:

r = range_seq()
for i=1,1000000 do
  -- let's drop some elements
  r = rest(r)
end
first(r) -- 1000000
first(rest(r)) -- 1000001

It’s a quite common pattern when working with infinite sequences to drop some values before working with them. So if we want to create an infinite range, that starts from a given value, we can write a drop function, which will produce a new lazy sequence without first n elements in it.

Another thing to note is that it is a bit tedious to write nested rest calls every time, so let’s iterate over this range instead, with the power of recursion:

function loop (seq, f)
  local head, tail = first(seq), rest(seq)
  f(head)
  if tail then
    loop(tail, f)
  end
end

loop(range_seq(0, 10, 2), print)
-- 0
-- 2
-- 4
-- 6
-- 8

This loop function accepts a sequence and calls a supplied f function on each element for side effects. A very similar concept exists in most functional languages, and it is called mapping, so let’s create a lazy map function:

function map (seq, f)
  if first(seq) then
    return lazy_cons(
      function () return f(first(seq)) end,
      function () return map(rest(seq), f) end
    )
  else
    return cons(nil, nil)
  end
end

Now with this map function, we can not only apply a function to every element of the sequence, but we can also collect the results into another sequence. Let’s write a function that prints every element of the given sequence, and pass the result of the map call to it:

function print_seq (seq)
  io.stdout:write("[")
  local function print_seq (seq)
    if first(seq) then
      io.stdout:write(first(seq), ", ")
      print_seq(rest(seq))
    end
  end
  print_seq(seq)
  io.stdout:write("\b\b]\n")
end

function square (x) return x * x end

print_seq(map(range_seq(0, 10), square))
-- [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Now, to demonstrate that this is really a lazy map, let’s add a print call to square, and store the resulting sequence to a variable:

function square (x)
  print("square of "..x.." is "..x * x)
  return x * x
end

squares = map(range_seq(0, 10), square)

Nothing is printed until we realize some elements of our sequence:

first(squares)
-- prints: square of 0 is 0
first(rest(rest(squares)))
-- prints: square of 2 is 4
first(rest(rest(squares)))
-- prints nothing, as the result was already realized

Note that while this is still just a (kinda special) function, it behaves just like a real linked list, and thus represents a proper data structure. Although it’s a bit clumsy to use directly, as shown with the map example, we can build a library around this concept, that will abstract the construction of a lazy sequence away from the user. And that’s exactly what I did.

Lazy-seq library

It should be pointed out that the code above is a very rough example of a lazy sequence implementation. I think it’s fine for explaining the idea, but there are some edge cases that are not handled properly. For example, the map function we’ve written is not fully lazy, as it actually realizes its first element, and the range_seq will not properly work for negative ranges. And the lazy_cons is not really the most useful abstraction - instead of using it directly, a much more handy lazy_seq function should be created, that would accept another function, which in turn would return any kind of sequence. To address these shortcomings I’ve created a lazy-seq library, which has a lot of functions to create and manipulate lazy sequences.

Now, it’s a Fennel library, written in Fennel, and aimed toward it first. That is, mainly due to the fact that this library mimics Clojure’s vision of lazy sequences and ports a lot of functions from its core namespace. But the core principles are exactly the same as described above, and you still can use the library from Lua, just like we did earlier!

Except the semantics are expanded a bit - there are handy constructors for creating lazy sequences from tables or strings, supporting both sequential and associative tables. And, as you may remember from the previous section since our cons cells are secretly tables underneath, sequences defined by this library also implement various metamethods, like __len, __index, and so on, for more convenient usage.

Proper versions of map and range already exist in this library, along with other functions, which can be used as building blocks for creating all kinds of sequences. For example, here’s how one would create an infinite Fibonacci sequence using the lazy variant of the map function in Lua:

lazy = require "lazy-seq"
first, rest, map = lazy.first, lazy.rest, lazy.map
concat, drop, lazy_seq = lazy.concat, lazy.drop, lazy["lazy-seq"]
bc = require "bc"
function add (a, b) return a + b end
fib = concat(
  {bc.new(0), bc.new(1)},
  lazy_seq(function () return map(add, rest(fib), fib) end)
)
first(drop(400, fib))
-- 176023680645013966468226945392411250770384383304492191886725992896575345044216019675

In the example above I’m also using the lbc library which provides arbitrary precision math for Lua to illustrate that you can compute as many Fibonacci numbers as you want (try dropping the first 100000 elements for example) That’s a bit verbose still, but not as verbose as the manually created lazy linked list from previous sections. And note that the concat function, which is used to concatenate sequences, also works with tables, as shown in the example above. All sequence manipulating functions in the lazy-seq library work with tables, as tables are implicitly converted to sequences.

In Fennel, the same code is even more compact because we can use the lazy-cat macro instead of using concat and lazy-seq, and thanks to anonymous function shorthand:

(local {: first : rest : map : drop} (require "lazy-seq"))
(import-macros {: lazy-cat} "lazy-seq")
(local bc (require "bc"))
(global fib (lazy-cat [(bc.new 0) (bc.new 1)] (map #(+ $1 $2) (rest fib) fib)))
(first (drop 400 fib))
;; 176023680645013966468226945392411250770384383304492191886725992896575345044216019675

Note however that macros are a Fennel-only feature, not available in Lua, so where in Fennel you would write:

(import-macros {: lazy-cat : lazy-seq} "lazy-seq")

(lazy-cat [1 2 3] [4 5 6])
(lazy-seq (do (print "I'm lazy") [1 2 3]))

In Lua it would be:

lazy = require "lazy-seq"
concat, lazy_seq = lazy.concat, lazy["lazy-seq"]

concat(lazy_seq(function () return {1, 2, 3} end), lazy_seq(function () return {4, 5, 6} end))
lazy_seq(function () print "I'm lazy" {1, 2, 3} end)

This code is fully equivalent to fennel code, except that macros write all these anonymous functions for you.

But macros are not the only unique feature in Fennel, the other one is destructuring. I’ll not go into details here, but destructuring is kinda similar to pattern matching, and it allows specifying data structure shape and binding values of the data structure to variables defined by that shape. So similarly to how you would destructure a table, thanks to __len and __index metamethods, destructuring works on lazy sequences:

(local {: range} (require "lazy-seq"))
(let [[a b c & rest] (range 10)] (print a b c) rest)
;; 0 1 2
;; [3 4 5 6 7 8 9]

Just make sure you don’t use & destructuring with infinite sequences¹. And more than that, sequences define the __pairs metamethod, and thus support creating stateless iterators created with pairs!

fennel = require "fennel" -- for pretty printer
lazy = require "lazy-seq"
squares = lazy.map(function (x) return x * x end, {1, 2, 3, 4, 5})
for k,v in pairs(squares) do
  print(fennel.view(k), v)
end
-- @seq(4 9 16 25) 1
-- @seq(9 16 25)   4
-- @seq(16 25)     9
-- @seq(25)        16
-- @seq()          25

We’ve come a full circle.

I’ve required the fennel library, to provide the pretty-printed representation of a sequence for a better illustration. You can see that on each step of iteration we see the shorter and shorter sequence. Sequence automatically pretty-printed in the REPL like this, but that’s one of Fennel’s features, that I’ve previously written about, and implemented a very similar linked list there, so if you’re interested in that, you can find some more info there. And viewing such a sequence in this way really shows us that it is nothing else but a data structure!

Unlike tables, a stateless iterator over a sequence uses the tail of the sequence as the key to getting the next element, because the iterator is simply built on top of first and rest. It ends when it reaches the empty cons cell, which is at the end of each sequence. The only exceptions are infinite sequences, which don’t have empty cons at the end as, well, they’re infinite. So calling pairs and iterating over infinite sequences should have another guard for terminating iteration. Or, you can limit the size of the sequence with take, take-while, and other filtering functions.

There are also various functions like line-seq which accepts a file handle, so here’s what happens when we call it on a file:

(local {: line-seq : doall} (require "lazy-seq"))
(with-open [f (io.open "example.txt" :r)] (doall (line-seq f)))
;; @seq("line 1" "line 2" "line 3")

We have to realize the whole sequence before with-open automatically closes the file handle, but you can lazily work with this sequence while you’re inside of the with-open block without any issues. In contrary to the iterator, we can re-use this sequence of lines, as no results have been thrown away.

Finally, as I’ve mentioned, lazy sequences can be generated from tables with the seq function:

(local {: seq} (require "lazy-seq"))
(seq [1 2 3])
;; @seq(1 2 3)
(seq "abc")
;; @seq("a" "b" "c")

If we pass in an associative table we get a bit different representation:

(seq {:a 1 :b 2 :c 3})
;; @seq(["a" 1] ["c" 3] ["b" 2])

These sequences are all lazy because seq creates a wrapper around an iterator over the table. This means, that if you change the table before the sequence is fully realized, your resulting sequence will have updated elements much like an iterator would produce them. This is not recommended, and instead immutable tables a better be used to produce sequences, and in general. And since sequences themselves are immutable and persistent this is a win-win.

The lazy-seq library also provides a lot of other functions, like filtering, restructuring, modifying, and combining infinite sequences. It’ll take a lot of time to list all of them here, so I’d recommend you to read the documentation if you’re interested. But as an example of what you can do, here we create an infinite sequence of random numbers, filter out only those which are perfect squares, and group those by 2:

(local {: take : filter : repeatedly : partition} (require "lazy-seq"))
(local rands (repeatedly math.random 0 64))
(local perfect-squares (filter #(= 0 (% (math.sqrt $) 2)) rands))
(local rand-square-pairs (partition 2 perfect-squares))
(take 10 rand-square-pairs)
;; @seq(@seq(64 16)
;;      @seq(4 0)
;;      @seq(64 4)
;;      @seq(4 0)
;;      @seq(0 4)
;;      @seq(4 64)
;;      @seq(16 0)
;;      @seq(0 64)
;;      @seq(16 4)
;;      @seq(16 36))

Note that we did all operations on an infinite sequence, and then only used take to get the first 10 elements of it. This is a common pattern when working with infinite sequences - create something infinite, filter or modify it, and take the needed amount of elements.

Conclusion

That’s all I wanted to say about lazy sequences and their relationship with iterators. But of course, there’s much more to it, as there are a lot more functions for sequence manipulation I’ve left uncovered because it would simply be too long for an article. The main point I’ve wanted to share is that sequences are similar to iterators, but don’t have some downsides that iterators have. Someone can argue that these downsides are features, but I think sequences still have their uses.

So let’s recap what we have learned about sequences:

Sequences are a data structure, with very important properties of persistence and immutability.
Sequences are lazy, much like iterators, but they cache their values for later re-use.
Things that required stateful iterators, can be done with stateless sequences by using laziness and recursion.
Sequences, implemented by lazy-seq library, support common table operations, making it easy to integrate lazy computation into existing Lua code.
Operations on infinite sequences can be easily composed.

I hope you got a good understanding of lazy sequences, and I got you interested in at least trying them out! Despite being a Fennel library, lazy-seq can be used from Lua just fine, you only need to compile it to Lua first, which is not hard to do, given that Fennel itself is also just a library. But I suggest using this library with Fennel, as it adds a bit more facilities, like pretty-printing of your data structures, and provides macros for a more concise code.

: As of Fennel v1.0.0 the __fennelrest metamethod was added, and & destructuring works on infinite sequences. ↩︎

Comment via email