Andrey Listopadov

Clojure on Fennel part two: immutable.fnl optimizations

Wed, 15 Apr 2026 05:11:00 +0300

So the next post will hopefully be about the compiler itself.

Unless I get distracted again.

Sike!

While I did some work on the compiler, I’m not feeling ready to talk about it yet. I want the post about it to be, well, more in-depth, so for now, let’s go back to immutable data structures. I got some comments on their performance, which can be summarized to “bad”.

And I agree! The performance is not great, especially when compared to plain Lua tables. However, my benchmarking process was a bit flawed, and I’ve improved it since.

This post won’t be long, but I wanted to make it anyway, to illustrate what I’m dealing with. Again, this is not the final version, I’m still working on possible optimizations as I write this, but some things have improved already, though not by much.

There were some low-hanging fruits that I could tackle:

Vector had a local fragment function with pure arithmetic. I already have the same fragment function in my bitops module that uses native rshift+band. The modulo operation is expensive.
A similar case of using bit operations instead of modulo was in HashMap’s index function.
Increased the popcount table to 256 entries instead of 16. Halves the loop iterations for typical bitmaps.
Probably premature, but replaced most instances of faccumulate, fcollect, etc. with plain for loops. These loops are quite hot, and Fennel compiles them to have additional assignment at each iteration. Should be free, but actually had a small impact.
More pre-allocations when creating Lua tables. Using [nil nil nil nil] tells the Lua compiler to pre-allocate four slots when creating the table, meaning it won’t resize once we insert elements. Tables in Lua resize by a factor of 2 and copy everything on resize, which is not great for performance.
Vector iterator is now stateful. And also fast.
Internal fields for data structures used long, elaborate keys, which meant string comparison in certain cases of table lookup would be unnecessarily long. Again, changing them to shorter keys should not matter that much, but it did have some impact.
HashMap branching factor was increased to 32. After some tests, it proved to be slightly better, despite what I said in my previous post. More on that later.
Hashes are now cached in a weak table. Helps avoid re-hashing the same values at the expense of memory. Not sure how well it will scale.
Optimized array copying for sizes less than 4 by doing direct table construction. Avoids loop overhead.

With these changes in place (and some more I omitted), here are some of the benchmarks:

These images show per-commit changes to the performance. Really helped me track what was worth keeping, and what wasn’t.

Here’s the LuaJIT version of the benchmark:

Here’s another graph:

This is me testing how hash map branching factor affects operations. As can be seen, switching from 16 to 32 makes inserts slower on PUC Lua 5.5 but makes indexing faster. On LuaJIT both operations are faster.

Looking at the charts, the 7989c32 commit with that change (the second orange bar) — insertions are slower than the previous commit, and indexing is faster than the previous commit. Unfortunately, LuaJIT charts are noisy and I can’t confirm the speedup there. Something to do with instabilities in the LuaJIT tracer, I guess.

Note I know, the results for some operations are worse than before all optimizations that took place. This is still a work in progress. Just shows how important relative measuring is.

The results are a mixed bag. Most operations that improved did so by only 1 or 2 milliseconds overall, meaning the per-operation cost is still quite high, compared to Lua tables. Yes, some became faster, so I’m keeping these changes, but I can also see that for some operations, they actually got slower than they were before any optimization work. The reason for that may be that I fixed a few bugs along the way. Anyway, I tried to prioritize transient collection variants, as they benefit most from optimizations given how they’re used, and LuaJIT performance where possible.

Some bars are not meaningful for certain operations. E.g., I had changes that had nothing to do with Persistent Vector, but affected it anyway, so this benchmark is still a bit flawed. At least those differences are within the deviation range.

Anyhow, just wanted to show that optimization is hard, and not all changes yield the results you expect. In the case of this library, I don’t think there are more low-hanging fruits to optimize, or at least I don’t see any. For example, there are a lot of table allocations that are hard to get rid of in the current implementation. A deeper analysis is needed, probably by people with more Lua knowledge.

I’m still satisfied with some improvements I got, but we’ll see if more are on the way.

Update

After a lot more benchmarks, flamegraphs, and throwing stuff at the wall, I think I can’t do much more. I decided to update this post instead of writing another one (and really should have waited before publishing the original) because at that point, I thought there was nothing more to do. Now I think there really isn’t. So the results are in:

Figure 1: PUC Lua 5.5 (green - before, orange - after)

Figure 2: LuaJIT (green - before, orange - after)

As can be seen, the changes are more dramatic on Lua than on LuaJIT. Mostly because it’s hard to optimize for LuaJIT without sacrificing performance on Lua, so I re-prioritized Lua, since LuaJIT is already much faster.

The final benchmark tables are below.

PUC Lua 5.5

PersistentVector

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
build 50000	0.18 ms	18.88 ms	102x	0.378 us	119x	0.440 us
lookup random	0.48 ms	11.44 ms	24x	0.229 us	28x	0.266 us
assoc random	0.31 ms	42.72 ms	139x	0.854 us	222x	1.4 us
pop all	0.14 ms	14.56 ms	106x	0.291 us	150x	0.415 us
iterate	0.60 ms	2.74 ms	5x	0.055 us	17x	0.196 us

TransientVector

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
build 50000	0.19 ms	6.55 ms	35x	0.131 us	41x	0.154 us
assoc random	0.31 ms	17.64 ms	57x	0.353 us	67x	0.408 us
pop all	0.14 ms	5.74 ms	42x	0.115 us	51x	0.142 us

PersistentHashMap

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
insert 50000	2.01 ms	81.44 ms	41x	1.6 us	82x	3.2 us
lookup random	0.92 ms	34.03 ms	37x	0.681 us	81x	1.7 us
contains random	1.82 ms	33.08 ms	18x	0.662 us	49x	1.7 us
dissoc all	0.90 ms	76.97 ms	86x	1.5 us	187x	3.3 us
dissoc 10%	0.18 ms	9.81 ms	56x	2.0 us	112x	3.7 us
iterate	1.91 ms	7.22 ms	4x	0.144 us	4x	0.146 us
lookup structural keys	0.14 ms	2.86 ms	21x	0.572 us	146x	4.1 us
insert collision-heavy	0.13 ms	6.50 ms	51x	1.3 us	134x	3.5 us
mixed 80/20	1.92 ms	53.67 ms	28x	1.1 us	62x	2.3 us

TransientHashMap

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
insert 50000	2.02 ms	38.91 ms	19x	0.778 us	44x	1.7 us
lookup random	1.14 ms	35.56 ms	31x	0.711 us	86x	1.7 us
dissoc all	0.89 ms	44.52 ms	50x	0.890 us	112x	2.0 us
mixed 80/20	4.39 ms	44.87 ms	10x	0.897 us	22x	1.9 us

LuaJIT

PersistentVector

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
build 50000	0.11 ms	8.12 ms	71x	0.162 us	84x	0.177 us
lookup random	0.06 ms	0.67 ms	11x	0.013 us	11x	0.013 us
assoc random	0.04 ms	22.91 ms	559x	0.458 us	731x	0.585 us
pop all	0.02 ms	10.77 ms	718x	0.215 us	657x	0.210 us
iterate	0.02 ms	0.23 ms	12x	0.005 us	26x	0.011 us

TransientVector

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
build 50000	0.05 ms	0.79 ms	16x	0.016 us	12x	0.012 us
assoc random	0.04 ms	2.22 ms	55x	0.044 us	28x	0.022 us
pop all	0.02 ms	0.35 ms	23x	0.007 us	50x	0.016 us

PersistentHashMap

Operation	Lua table	Persistent	ratio	per op	ratio (old)	per op (old)
insert 50000	0.79 ms	32.32 ms	41x	0.646 us	50x	0.989 us
lookup random	0.34 ms	5.78 ms	17x	0.116 us	43x	0.248 us
contains random	0.30 ms	5.91 ms	20x	0.118 us	40x	0.257 us
dissoc all	0.23 ms	27.02 ms	119x	0.540 us	191x	0.894 us
dissoc 10%	0.05 ms	5.89 ms	118x	1.2 us	121x	1.4 us
iterate	0.07 ms	1.07 ms	16x	0.021 us	30x	0.040 us
lookup structural keys	0.02 ms	2.41 ms	121x	0.483 us	119x	0.498 us
insert collision-heavy	0.28 ms	5.20 ms	19x	1.0 us	36x	1.2 us
mixed 80/20	0.50 ms	46.59 ms	94x	0.932 us	40x	0.513 us

TransientHashMap

Operation	Lua table	Transient	ratio	per op	ratio (old)	per op (old)
insert 50000	0.91 ms	30.92 ms	34x	0.618 us	11x	0.242 us
lookup random	0.32 ms	34.90 ms	110x	0.698 us	44x	0.246 us
dissoc all	0.23 ms	36.35 ms	162x	0.727 us	57x	0.271 us
mixed 80/20	1.29 ms	40.71 ms	32x	0.814 us	11x	0.309 us

Conclusion

Yes, the benchmark shows that some operations slowed down, some sped up. Ideally, I’d like to have no slowdown, but that’s that. Still, I wouldn’t trust my LuaJIT benchmark, though - graphs show that for some weird reason unrelated changes affect performance in the modules not touched by those changes at all. So these changes are now merged into the main branch.

Main bottleneck now is table allocation and array copying - things I already spent optimizing a lot of time with mixed results. So I think I can’t do better than that. At least without major algorithmic restructuring.

With that, the next post should be about the compiler.

Comment via email

fnl-http - testing with fennel-test

Thu, 15 Aug 2024 23:01:00 +0300

fnl-http is my current passion project - I spend a lot of free time tinkering with it, and the last week was spent on testing and fixing bugs.

As you may know, I made a testing framework, called fennel-test, which has a dedicated test runner, and a set of macros for writing tests. I use it in all of my projects, and oftentimes, some project requires more features that are currently available in the framework, so I extend it as needed. I think it’s a great way of designing something - you use it, and enhance it as you encounter new needs.

I want programming to be exciting, so I spent a lot of time making my test report nice to look at. Here’s how the report looks by default:

Test run at Tue Aug  6 23:20:41 2024, seed: 1722975641242
(....)(......)(.)(.....)(.)(........)(...)(...)

The default reporter is called dots. Each test namespace is encoded with parentheses (), each successful test is encoded as a dot ., and failed tests are encoded as F. The runner randomizes the order of tests each time, so if we run it again, the result would be different:

Test run at Tue Aug  6 23:24:10 2024, seed: 1722975850235
(......)(........)(.)(...)(...)(....)(.....)(.)

However, this is all you get with a reporter like this. If a test fails, information about the error is printed after all tests have finished, so you know what test did fail.

There’s another inbuilt reporter, called namespaces:

Test run at Tue Aug  6 23:27:33 2024, seed: 1722976053226
test.http-parser-test: PASS
test.client-test: PASS
test.json-test: PASS
test.httpbin-test: PASS
test.body-test: PASS
test.headers-test: PASS
test.readers-test: PASS
test.url-test: PASS

As you can see, it prints each namespace that is currently being tested, followed by the namespace-wide status. If a test fails, you again see its error message, but only after all tests have finished, which isn’t ideal because tests in the fnl-http project take about two minutes to complete.

Fortunately enough, you can create reporters in the .fennel-test configuration file!

Reporters

A reporter is just a table with a set of functions. Each one is called when the respecting stage of testing is being run. The methods are:

ns-start - called when entering a new test namespace.

Used to print the information about the namespace, such as its name.
ns-report - called when exiting a test namespace.

Used to print the overall status of the namespace.
test-start - called when running a single test from a namespace.

Used to print the information about the test, such as the test’s name.
test-report - called after the test is finished.

Used to print the status of the test.
stats-report - called after running all tests in all namespaces.

Used to print overall statistics of tests - how many tests were executed, how many errors were encountered, were there any warnings, etc.

This was enough for me to implement other kind of reporters, like the one in the project’s readme. However, I wanted a bit more for this project - first, I wanted the report to be fun to look at, and second, I wanted some statistics. And third, actually, I noticed the need to skip tests in case something is not available on the machine.

After a bit of work, here’s what I’ve got:

With this reporter, if a specific test fails, I can immediately see it, without waiting for all tests to pass. In the end, you can see some statistics on the longest tests, and average test time of the namespace. For example, the test.httpbin-test namespace took 90 seconds to complete, but on average each test only takes 11 seconds. I’m not entirely sure why this metric would be necessary, but I’ve seen it in another test runner, so I implemented it in my reporter.

Now, here’s a problem - one of the tests takes a whopping 90 seconds. I don’t want to run this test all the time, especially, when I’m fixing a bug found by some other test. So I introduced a skip-test function as a first step.

Originally, this function had two modes - terminating and non-terminating, but I later removed the second mode. The reason for that is the with-open macro.

Here’s an example of one of the fixtures I use in the fnl-http project:

(use-fixtures
 :once
 (fn [t]
   (with-open [proc (io.popen (.. "podman run  -p " port ":80 kennethreitz/httpbin >/dev/null 2>&1 & echo $!"))]
     (let [pid (proc:read :*l)
           attempts 10]
       (if (wait-for-server attempts)
           (do (t)
               (kill pid))
           (do (kill pid)
               (skip-test (.. "coudln't connect to httpbin server after " attempts " attempts"))))))))

The with-open macro ensures that the resource is closed no matter what. Closing the process, however, doesn’t kill the program running in the background, so I had to get its PID with sketchy echo $! part and kill it later. Killing the process doesn’t release the file descriptor created by popen, so with-open is only responsible for releasing the file handle here.

A sudden bug

What with-open also does is catching, and re-throwing errors. It’s something akin to trying with resources in Java, and other languages that support this kind of abstraction. You can think about it as try/finally without the catch part.

Lua, however, doesn’t really have well-defined exceptions. Instead, functions usually throw string messages about the error and that’s it. Sometimes, however, it’s useful to throw a table instead. Tables are like objects, so you can embed some additional information in those, making it easier to match a particular error. It’s better to convert this table to a string if it ever reaches the user uncaught though.

What makes things harder is that in Lua it’s not possible to re-throw an error without losing its previous stack trace. While it is a common practice to use pcall to catch any errors and to re-throw them if you can’t do anything about the error, this is what causes losing the precious stack trace. More precisely, if the error message comes with the stack trace, and you would just re-throw it with error, you wouldn’t add your stack trace to the trace. Sounds convoluted, yeah?

So Lua gives you another option: xpcall. The Lua manual doesn’t have a thorough explanation of it though:

xpcall (f, msgh [, arg1, ···])

This function is similar to pcall, except that it sets a new message handler msgh.

It works the same as pcall, except, it accepts an additional argument - a function that would append a stack trace to the error. Usually it’s the debug.traceback function:

debug.traceback ([thread,] [message [, level]])

If message is present but is neither a string nor nil, this function returns message without further processing. Otherwise, it returns a string with a traceback of the call stack. The optional message string is appended at the beginning of the traceback. An optional level number tells at which level to start the traceback (default is 1, the function calling traceback).

Knowing all that, let’s look at how I’ve implemented the skip-test function:

(local Skip
  (setmetatable
   {}
   {:__tostring #:Skip
    :__fennelview #:Skip}))

(fn skip-test [reason]
  "Calling this function inside a test or a fixture will stop the test
early and mark it as skipped. The optional `reason` argument is a
message to display in the log if the reporter is configured to do so."
  (error [Skip reason]))

Right, skip-test throws an array with a special table object, defined as a singleton, so the test runner could distinguish between ordinary test errors and skipped tests. In theory, everything should have worked, but for some reason, some of the tests were skipped correctly, but others didn’t.

The culprit? It’s with-open, obviously, otherwise, why would I take such a tangent about it?

Internally, with-open uses the xpcall function to do its job. And theoretically, it should work just fine, however, it doesn’t use debug.traceback if Fennel’s own compiler.traceback is available.

Originally, I thought that’s just how it is, and implemented two modes in skip-test. After a bit of digging and consulting the manual, I found out that compiler.traceback mistakenly converted every message to a string. Fixing that made the skip-test function work in all cases.

Another sudden bug!

After the patch landed, and I updated Fennel to the newest commit, something else broke. Now, the JSON parsing test no longer succeeded, complaining that some number has no integer representation. The number in question is 23456789012000000000000000000000000000000000000000000000000000000000000000000 or, as it is written in the JSON test file that I use, 2.3456789012e+76.

Before the update, everything worked fine, so I started digging again. Turns out, recently, the way numbers are compiled by Fennel has changed:

@@ -534,8 +534,9 @@ (fn compile-sym
 ;; We do gsub transformation because some locales use , for
 ;; decimal separators, which will not be accepted by Lua.
 (fn serialize-number [n]
-  (pick-values 1 (-> (tostring n)
-                     (string.gsub "," "."))))
+  (if (= (math.floor n) n)
+      (string.format "%d" n)
+      (string.gsub (tostring n) "," ".")))

And now it is failing, because indeed, you can’t convert 2.3456789012e+76 to a string using %d. Except, you can if you’re using LuaJIT - it happily converts this number to "-9223372036854775808", which is even worse.

After a bit of searching, I found that the safest way to format integers as stings is by using "%.f format specifier. Strangely enough, the Lua manual doesn’t have any information on most number formatting patterns. A tiny bit of information was found on the lua-users.org page:

To make sure integer numbers are properly converted into string (without scientific notation), rather than using tostring(x), use format.string("%.0f",x)

I don’t really get why %.f would format a number as an integer given that %f formats floats. And, even though it works the same way in C, I have never seen this in my life. But OK, we could just replace %d with %.f, and be done with that, however, the number above is then would appear as "23456789012000000697746671432670411048021854502345623879537911769302235611136". I get it, it’s a big number, and Lua has limited precision for integers, but the main problem is that this number is much harder to read than its e-notation brother 2.3456789012e+76.

So I settled on finding a way to fix this bug so that both PUC Lua and LuaJIT would format this number in the same way. The result is a patch, that changes the compiler and pretty-printer parts of the language. The function that does that, looks horrifying:

(fn serialize-number [n]
  (let [val (if (= (math.floor n) n)
                (let [s1 (string.format "%.f" n)]
                  (if (= s1 (tostring n)) s1
                      (or (faccumulate [s nil i 0 99 :until s]
                            (let [s (string.format (.. "%." i "e") n)
                                  n* (tonumber s)]
                              (when (= n n*)
                                (let [exp (s:match "e%+?(%d+)$")]
                                  ;; Lua stops transforming numbers from
                                  ;; e-notation to integers at e+14
                                  (if (and exp (> (tonumber exp) 14))
                                      s
                                      s1)))
                              s1))))
                  (tostring n))]
        (pick-values 1 (string.gsub val "," "."))))

We start by checking if (tostring n) and (string.format "%.f" n) give the same string. If they are, we just return the string. If not, we try the scientific notation for each power from 0 to 99. I probably should’ve tested for more powers than that, but it’s OK for now.

We’re converting the number to a string using the e-notation and parse it back, comparing it with the original number. What this does, is that if the power is less than the original, we would lose precision, and comparison would fail. If we can’t find the exponent up to 99, we bail out and return the %.f representation. Otherwise, if the numbers match, we check if the exponent is greater than 14 as it is the exponent where PUC Lua starts using the e-notation. Otherwise, we again return the %.f representation.

Phew!

But I’m not done yet!

Let’s try to compile a really huge number:

>> 1e+99999
nil

What?

Let’s do it in an expression then, maybe that’s just the pretty printer freaking out:

>> (+ 1e+309 1)
attempt to perform arithmetic on a nil value (global 'inf')

~~What?~~ I, of course, jest, but the reason this happens is pretty peculiar.

You see - when we do (tostring "%.f" n) on a huge number like 1e+309 it returns the string "inf". The compiler then happily includes it in the output Lua code, which is then run by the Lua VM. In Lua, however, inf is not a reserved identifier or a number, so it treats inf like a variable name. The inf variable is unbound, hence the nil in the first example, and an error in the second.

So we need to account for that:

(fn serialize-number [n]
  (let [val (if (= (math.floor n) n)
                (let [s1 (string.format "%.f" n)]
                  (if (= s1 "inf") "(1/0)" ; portable inf
                      (= s1 "-inf") "(-1/0)"
                      (= s1 (tostring n)) s1 ; no precision loss
                      (or (faccumulate [s nil
                                        i 0 308 ; beyond 308 every number turns to inf
                                        :until s]
                            (let [s (string.format (.. "%." i "e") n)]
                              (when (= n (tonumber s))
                                (let [exp (s:match "e%+?(%d+)$")]
                                  ;; Lua keeps numbers in standard notation up to e+14
                                  (if (and exp (> (tonumber exp) 14))
                                      s
                                      s1)))))
                          s1)))
                (tostring n))]
    (pick-values 1 (string.gsub val "," "."))))

I also brought the exponent limit up to the last number representable in this notation without becoming inf.

Now we can go back to testing!

Counting skipped tests and assertions

The other thing I added to fennel-test was more counters. Counting skipped tests required some tinkering, like counting tests in the whole namespace if the tests were skipped from the :once fixture.

Counting assertions was even harder, because tests in fennel-test are implemented through macros, and assertions are macros too. I had to inject some runner state, conditionally, and modify it at runtime from each assertion. A bit shady but it works.

What is much more shady is the coverage reports for this project.

Coverage report

I have a firm belief that 100% is the only acceptable coverage. However, and this is a big however, it’s important that such coverage comes from public API tests only. Achieving 100% coverage this way means that your program doesn’t have any dead code and that your public API test at least touches all of our code at least once. It doesn’t speak about test quality, of course, and it doesn’t guarantee that the program is bug-free, but those other points alone make a huge difference.

Lua has one and only luacov that you can use to test your code. I’ve used it in some other Fennel projects in the past, but it comes with some nuances you have to deal with.

First, luacov itself is imprecise. In the post about 100% coverage I talk about it in more detail, but basically, I often see situations where the side effects happened, but the coverage report showed code responsible for the side effects as untouched. Other times, the coverage percentage is bigger than its actual value due to code formatting.

With the fnl-http project I hit a different problem. Two, actually.

The first one is related to the --correlate option of the Fennel compiler. I had an idea, that if I used --correlate I could substitute Lua sources used to produce the coverage with Fennel sources. And since the line numbers are matched by specifying the --correlate option, the coverage report would be so much easier to read.

Didn’t work out.

Unfortunately, --coverage is imprecise. Many parts of Fennel AST don’t contain any line information. Here’s an example:

(local {: a
        : b}
  (require :c))

{: a
 : b}

And here’s the corresponding Lua code:

local _local_1_ = require("c") local a = _local_1_["a"]
local b = _local_1_["b"]
-- empty line
-- empty line
return {a = a, b = b}
-- empty line

As you can see, the overall amount of lines is the same, but many lines that are not empty in the Fennel example are empty in Lua. It’s not a problem, as long as you’re measuring the coverage and viewing the report with Lua, but if we replace the Lua code with its Fennel counterpart, the report becomes invalid. I guess, when the report is constructed, luacov checks if the line is empty or not, and if it isn’t, and there were no hits registered, it counts it as a miss. I’m using luacov-console to view the report, maybe it’s a bug from there, I don’t know.

The only way to fix this is to write the code accounting for these nuances of --correlate:

(local {: a                ; local _local_1_ = require("c") local a = _local_1_["a"]
        : b} (require :c)) ; local b = _local_1_["b"]
                           ; <empty line>
{: a : b}                  ; return {a = a, b = b}

There are more examples like that in the project’s code.

The other problem is more peculiar. Luacov uses debug.sethook to build its report. And I use the async.fnl library in this project, which also uses debug.sethook to schedule tasks. So every time anything happens asynchronously, and trust me, there’s a lot of that in fnl-http, the Luacov report is no longer properly calculated.

So, Luacov has a lot of problems, but I understand why it is like that. You can’t really instrument the Lua code in some other way.

Fennel, on the other hand, isn’t like that. Being a language where everything is an expression, we can probably make some compiler plugin that injects some side-effecting code that writes the report for every expression in the code. It would be difficult to do with things like tables and destructuring, but it should be doable. Maybe I’ll try to tackle this problem in the future, who knows?

Anyway, that’s all from me for now! As usual, you can check out the project, which contains all of the stuff mentioned above regarding fennel-test and luacov. Thanks for reading!

Comment via email

100% is the only acceptable test coverage

Tue, 29 Mar 2022 18:09:00 +0300

…kinda.

I think that the only acceptable test coverage percentage is about 100%. And in this post, I’ll try to explain why I choose to believe it.

Why test coverage matters

I have a lot of projects of various sizes. In most cases, these projects are libraries, that I host publicly, and some tools I’ve written for myself only. What’s common about these projects is that no matter if it’s a public project or my personal one, I’m trying to test those as well as I can.

But this raises some important questions, one of which is how do we measure the quality of our tests? A rather obvious answer to that question is test coverage. And I can only say that this is the stupidest metric to measure test quality because coverage doesn’t actually guarantee that tests check all various corner cases data-wise.

Test quality is a bit different topic, and while code coverage is not what measures it, it is still important, and in my opinion, the reasonable percentage is 100% or something really close to it. I’ll explain why in a moment, but before that, I’ll address a question that probably has appeared in your head: “why go as high as 100%?”. And that’s a valid question.

Recently, I’ve talked with some developers who use different languages and was kinda surprised to hear that on average, the acceptable test coverage is ~60%. Some say that writing unit tests for functions that are inherently simple is meaningless, as those can be verified fully at the stage of writing the function itself. Others say that 60% is just enough to cover the public API and catch most of the bugs that the end-user may experience. Some even say that test coverage doesn’t matter at all, and only integration coverage actually does, but it’s hard to count, so nobody really cares. But even they agreed when I asked them if 60% would be enough for them.

I disagree strongly.

You may have heard of the 80/20 rule, also known as Pareto principle. The Wikipedia article I’ve linked has a section related to computer science, that states that:

In computer science, the Pareto principle can be applied to optimization efforts. For example, Microsoft noted that by fixing the top 20% of the most-reported bugs, 80% of the related errors and crashes in a given system would be eliminated. Lowell Arthur expressed that “20% of the code has 80% of the errors. Find them, fix them!”

And after a while of writing tests and measuring code coverage for my projects, I noticed something similar to that. The last 20% of code coverage held 80% of bugs of the code. And the interesting part was, that no matter how good my tests were, it didn’t matter, as they weren’t touching the parts of the code that actually had unnoticed bugs. So whenever I’ve stopped at code coverage of 80% saying to myself that this is good enough, I was lying to myself, as it clearly wasn’t.

What code coverage is about

So, now let me address some misconceptions about code coverage.

Foremost, test coverage is not about covering every function in your project with a unit test. This is pointless extra work that will be negated with the first tiny code refactoring. Because of that, all these tests will become obsolete, and new tests are likely won’t be ever written, because why bother, if they’ll become irrelevant with the next refactoring? And functions are really simple, why even test those?

Second, for those who love numbers, 60% coverage in a project with just 2k lines of code (LOC) means, that 800 lines were never even touched by the existing tests. In a project with 10k LOC, it’s already 4k lines. Not only you can’t reason about correctness, but you also can’t tell what’s actually used in your project and what’s not!

And that’s the point of code coverage - telling you what parts of your application/library are not reachable from the public API that you always must test. You don’t need to write unit tests for anything but the public API in almost all cases! Yes, it’s good to write tests that check complex functions, which do some data transformation for example, with a lot of different test inputs - I’m not against that at all. However, if you write a unit test for a private function, think of it for a moment, maybe you’re doing something wrong.

As of this moment, I’m mainly using two languages - Fennel and Clojure, and the part about testing private functions is what makes these languages a bit different for me when I’m reasoning about code and its public API. The difference is that in Fennel there’s no way of accessing a private function of a module, and hence there’s no way to test it directly. In Clojure, there is, and when I see, or even have to use (#'some-ns/some-private-fn)¹ in a test, it’s a huge red flag for me, that something is not right in the application code itself. Fennel (actually Lua) doesn’t have this problem, because if something is declared as local in a script file and not exported from it in a table, that’s that - you can’t have it.

So when writing tests in Fennel or Lua, your only option is to test exported functions only, which is a kind of your public API. And if you wrote tests, and see that all lines are covered, all expressions are visited, and you still can’t reach some private functions - well, that’s a clear indication of a dead code. Dead code is bad because it rots even if you don’t touch it. Even more - if you actually go and write unit tests for such dead code, you’ll make the situation even worse for yourself in the long run. After a certain period, these tests may be the reason you can’t move forward, because you’ll be afraid to change or even remove a code that appears to be unused - but you can’t be sure, because there are tests for this code.

Now, I know what you can say to this - modern IDEs can detect unused code by using static analysis. This is a great feature that helps to deal with this problem to some degree. First, some IDEs detect that code is used in tests, and that’s enough for them to consider code not to be dead, which isn’t actually true. It is an important feature, but not a silver bullet, and not all languages have it. Yes, in most cases it helps you see that some code is indeed unused, but that’s the only thing it does. Test coverage also shows that, plus provides additional correctness guarantees of your code because of tests that you’ve written.

Why 100% is not always possible

I hope you now see why I think why code coverage is very important. But percentage doesn’t exactly say to you what is left uncovered, so all code coverage tools provide some kind of report, usually in a color-coded form. Some coverage tools build interactive web pages that allow easy project navigation, and some just print results to stdout using ANSI color codes. Both are equally helpful because you see the results of your work.

However, unfortunately, it’s not as simple as that. If it had been, I guess the average coverage percentage expectage would be higher than 60%. So what can cause problems?

If you test an application, then in many cases you probably can’t test the main function, especially when it starts a server for example. This means, that the entry point to your application is untested, and this might cause problems in the future. As a workaround, it is possible to mock the server starting functions so main would return immediately. This, probably, won’t return any meaningful values for the test itself, but you’ll be able to see what parts of code are triggered by the initialization process and see if there are some dead parts. Alternatively, if all the main function does is initialization and service startup, these things can be tested separately, and main can be excluded from testing, but this will lower the coverage percentage, as expected.

Another problem is again solvable with mocks but seems to defeat the purpose of testing. It does not. I’m talking about interactions with external services. A common misconception is that by mocking the server, we create an ideal model that doesn’t help at unit testing. While this is somewhat true, it also depends on how you mock it. You can write such a mock that causes a timeout, or a mock that sends invalid or corrupted data to test error recovery. You can even write a small service inside your codebase that mimics the public API of a real one, which can help with testing. And with mock-based testing, your application code can mostly consist of private definitions, as you don’t need to export those for testing.

This is something in between integration and unit testing, and while it’s surely not as good as real integration testing, it is still something worth doing. Your public API probably won’t change as much as your code, and you’ll be more certain that nothing broke if you have such tests. This is also a very good way of seeing what parts of your code in the coverage report are touched sorely by interacting with external services, without manually patching in the data for transformation. Though I must say that writing mocks for every network problem imaginable is not an easy or even impossible task, it’s not always a solution.

Expression and line coverage

The last main problem is that coverage tools are not always accurate. This includes some quirks in the language runtime or tooling, that prevent it from tracking all lines or expressions. I often see this in Lua, when I run some tests, collect coverage reports and see some branches as not being hit. I then add a print call to these branches and see prints in the output, meaning that the code is indeed touched by the test. But the coverage tool doesn’t register it for some reason, and that particular call to print is shown as uncovered.

And the line-or-expression distinction in the coverage report is another problem with the opposite outcome. If the previous problem was about the impossibility of achieving 100% because some code just won’t register, this is about achieving 100% without hitting all code. The reason for that is that if you put multiple expressions on the same line, a coverage tool that doesn’t know about expressions may show a completely wrong coverage. Here’s an oversimplified example:

-- lib.lua
return {
  f = function (a, b)
    local res
    if a == 10 then res = a * b else res = a / b end
    return res
  end
}
-- lib_test.lua
local lib = require "lib"
local res = lib.f(10, 20)
assert(res == 200, "expected 200, got " .. res)

The if statement in the function f is written as a one-liner (for this particular example reasons). The test only checks one specific case, which can only trigger the true branch of the if. There are no tests for false branch, which may easily result in a division by zero if a is not 10, and b is 0. While this is not a problem in Lua, as it will just return inf, this may be a problem for your application, which will not know what to do with infinity.

This is an exaggerated example, only meant as an illustration of the problem. Running the test module with Luacov shows a 100% coverage:

$ lua -lluacov lib_test.lua && luacov-console && luacov-console -s
==============================================================================
Summary
==============================================================================

File         Hits Missed Coverage
---------------------------------
lib.lua      4    0      100.00%
lib_test.lua 4    0      100.00%
---------------------------------
Total        8    0      100.00%

Thankfully, I write my code in Fennel, and it usually generates a somewhat well-formatted Lua code, so coverage is more or less accurate. Doing the same with Clojure won’t have this problem at all, because Cloverage provides two separate statistics - line coverage and form coverage:

;; src/lib/core.clj
(ns lib.core)

(defn f [a b]
  (if (= a 10) (* a b) (/ a b)))

;; test/lib/core.clj
(ns lib.core-test
  (:require [clojure.test :refer [deftest is]]
            [lib.core :refer [f]]))

(deftest f-test
  (is (= 200 (f 10 20))))

As you can see, I’ve done the same thing as in Lua, but the coverage report suggests that something is not right:

$ lein cloverage
|-----------+---------+---------|
| Namespace | % Forms | % Lines |
|-----------+---------+---------|
|  lib.core |   75.00 |  100.00 |
|-----------+---------+---------|
| ALL FILES |   75.00 |  100.00 |
|-----------+---------+---------|

Again, the main point I wanted to show here is that not all coverage percentage reports are fully trustable, that’s why the visual report is important. Seeing a pretty long line with an if statement in Lua should tell you to be more careful with this code, as some parts of the code could have not been touched. Cloverage colors such line in orange, meaning that not all forms were covered, but doesn’t tell you which forms weren’t touched. Both cases can be fixed with the proper formatting.

Closing thoughts

Overall, I hope I was able to convince you that 60% code coverage is far from ideal and that 80% is still not enough. In my personal opinion, something starting from 97% is a way to go. Of course, it’s not always possible, but when it is, there’s no reason not to go for it. And I’ll repeat myself, having higher coverage percentage doesn’t mean your code is well tested, it’s merely well covered.

To summarize - test your public API and collect coverage of it specifically. Having 100% code coverage achieved through such testing means that:

There’s no dead code.
You have tests for everything in your application/library.
You have a very reasonable starting point for improving all tests with more cases.
- And because of that, there are more possibilities to find bugs in some far-to-reach places (80/20 rule).

I’ve already said that I’m against writing unit tests for private functions, but, well, see for yourself. I, personally, think that this means that the code was meant to be public, and making it private was a design mistake. But your view may vary.

While 100% coverage may not be always possible, carefully observe what kind of code was not covered. If that’s something because the coverage calculation tool just misbehaves, well, there’s little you can do with it. If that’s due to the inability to test something like a main function, or a very tricky part that communicates with another service - make sure it’s well tested through integration testing. Everything else, probably means you have some potentially incorrect design decisions that should be corrected.

Hope this was an interesting read, feel free to share your thoughts on the topic and tag me in any of the social networks if you disagree or have something to discuss on the topic!

This is how you access a private var in Clojure. ↩︎

Comment via email