Small languages with little tooling
When it comes to software I prefer things that are simple and small, even though I’m using Emacs. This is mainly the reason why my favorite languages are Clojure and Fennel. However, it doesn’t end on programming languages themselves, I like small tools in general. One such example would be Kakoune - a text editor, inspired by Vim, which focuses on multiple selections.
Kakoune doesn’t have a scripting language, and when I started using it, the collection of plugins was relatively small, and it didn’t have a plugin manager at all. With a limited amount of options, it’s easier to start using particular software, because you don’t have the problem of choice, especially when you know nothing about the things that you’re choosing from. You pick it up, and you’re good to go. This was a major problem for me to get into Emacs, as, as opposed to Kakoune, not only does it has a pretty huge core full of features, but it also has a ton of packages written by users over the years, and several package managers to install those.
Eventually, I figured things out, and now I’m a happy Emacs user, but its enormous pack of inbuilt features still bothers me, and Kakoune made me understand why I like small ecosystems. It’s because in a small ecosystem it’s easier to see what is needed by people, and you can participate in making things better not just for yourself but for everyone.
Don’t quote me out on this, but I think plug.kak was the first fully featured plugin and configuration manager for Kakoune, which I made mostly for myself, but to my surprise, it got to be pretty popular.
There even were plugins (written by other people) that suggested using plug.kak as a preferred way of installing them.
Before plug.kak people had to use the default mechanism of loading, which used an autoload directory to specify what Kakoune should load at startup, and creating a such directory in your config disabled system-wide autoload directory, making Kakoune featureless.
This was one of my motivations behind plug.kak, because I saw issues in Kakoune’s repository titled mentioned that the creation of the auotload dir broke their config in an unexpected way of not loading any of the inbuilt features.
My approach was to create a, hopefully, small system that knows where to look for external stuff and how to load and configure it.
And people seemed to like it.
I also wrote a bunch of other plugins for Kakoune, all of which are listed here, if you’re interested.
This helped me shape Kakoune into a thing that I liked to use, having features that I need, and also igniting my interest in programming in general, because the tools I built were used by other Kakoune users, and it was inspiring.
Later on alternatives to plug.kak appeared, and I’m glad because they were simpler and more efficient.
I no longer maintain plug.kak, as I no longer use Kakoune, but it still has a place in my heart.
Before Kakoune I made plugins for Vim, but given that the plugin ecosystem was already saturated, my plugins were made mostly for myself and I wasn’t hoping that anybody else would prefer them to alternatives. It was fun, still.
So my point is - I like to participate in small ecosystems, mostly because the points where an improvement is welcome are more visible. This leads us back to programming languages, and Fennel in particular.
The Fennel Programming Language
If you read my blog, you may have seen posts about Fennel before, and likely know what it is. If not, here’s a quick introduction:
Fennel is a small surface language implemented on top of the Lua language. It has a Clojure-inspired syntax, and Lisp macro system, allowing users to extend the language with new constructs pretty easily. Fennel compiles down to Lua, and doesn’t have its own semantics, besides mentioned macro system and being a bit more strict. And because Fennel is actually just a compiler, it includes almost no runtime features. Its only runtime components are REPL (Read Eval Print Loop), which is a common thing for Lisps, and the pretty-printer - a small library that prints data structures in a way that can be read back by Fennel compiler, or just examined by us, humans.
I’ve been using fennel for almost 2 years now, and I’ve been reflecting on what has changed for me personally. What’s improved, what things I made for myself, and so forth. And looking at my project workflow in Fennel, and seeing a lot of stuff happening, I get the impression of working in a mature language, with a mature ecosystem. So I decided to write about this topic, highlighting interesting changes, that happened in the past two years of my Fennel experience.
Pretty Printer
When I started using Fennel, the pretty-printer module was in a bit sad state. It worked, but its output was not formatted as one would expect from a language with Clojure-like syntax. Instead, it was formatted more like Lua code, and because of that, it was difficult to inspect deeply nested structures.
I have a post describing existing problems and my approach in detail, but in short, I had to rewrite the fennelview.fnl module from the ground up.
And as far as I can tell, the community liked the new pretty printer, so the effort was well worth it.
It needed some tweaks here and there afterward made by other people, but I think this is still my biggest contribution to Fennel, especially since we’re seeing results of the pretty printer every time we send something to the REPL.
Which made Fennel much more pleasant to work with (for me, at least).
The fennel-cljlib library
With the pretty printer in place, I started working on various libraries that were just interesting to make. The first one was fennel-cljlib, which was aimed at Clojure programmers that wanted to try out Fennel but didn’t want to give up some of the features they love in Clojure. The library was crappy and had a lot of problems, which I fixed over the years, but right now I would say that it is solid. Of course, there’s still a lot of room for improvement, but I’ve already utilized this library in some projects, and even seen its use in projects from other people!
One thing that this library made me good at is writing macros. Fennel has a similar syntax compared to Clojure, but it’s not the same, and some features are missing. For instance, in Clojure, there is such thing as multi-arity functions, which are extremely handy. Here’s how you might define such a function in Clojure:
(defn add
"Sum arbitrary amount of numbers."
([] 0)
([a] a)
([a b] (+ a b))
([a b c] (+ a b c))
([a b c d] (+ a b c d))
([a b c d & rest] (apply add (+ a b c d) rest)))
So if you call this function with zero arguments given, the ([] 0) arity is selected, and 0 is the return value.
If you call it with three arguments, the ([a b c] (+ a b c)) branch is selected, and the return is the sum of three arguments.
However, if you call it with five-or-more arguments, the last arity ([a b c d & rest] (apply add (+ a b c d) rest)) is selected, which will recursively call itself with less and fewer arguments, until it reaches another arity.
So the call stack is (add 1 2 3 4 5 6 7 8 9 10) to (add 10 5 6 7 8 9 10) followed by (add 28 8 9 10) and the result value is 55.
That’s three tail-recursive calls, to sum 10 numbers, and we do as much work to avoid unnecessary recursive calls.
Of course, math functions, such as this can be written in a more performant way without any recursion, this just illustrates the idea, that you can write functions in a pretty much declarative style: given this amount of arguments do this thing.
And with the cljlib library we can define such functions
But I’ve lied to you, that’s actually not Clojure up there, but Fennel code, defining one of the functions in cljlib.
And if you scroll through the linked code, you’ll see that pretty much every function utilizes this feature of the language.
Even more so, if you take a look at Clojure sources, you’ll notice that a lot of code was ported almost directly, which some changes needed for it to work on the Lua runtime.
But I’m going a bit off-topic here, this post is about the projects related to the language ecosystem, not just random projects of mine. However, this library motivated me to make a contribution to the ecosystem, because I felt that there’s a point for improvement in the current workflow. In the end, all I really want is a comfortable environment so I could enjoy programming a bit more, and maybe make it more enjoyable for other people as a side effect. And I’m glad to see other people using the next project I’m going to talk about.
Fenneldoc
Fenneldoc is the tooling I wanted for Fennel so badly that I had to make one myself.
This project is a rather simple documentation generation tool, which extracts documentation strings from functions’ metadata, and creates a set of Markdown formatted files.
I wanted this tool because, already mentioned, cljlib project had 60+ functions and 10+ macros, and I had to maintain documentation for all of that by hand.
And while I was working on this library and adding new functions, or changing existing ones, I had to keep the documentation in sync.
At some point, I was tired of noticing that I forgot to update some function’s documentation, or forgot to mention one entirely.
So I wrote fenneldoc and replaced, all of the manually written documentation with auto-generated one.
I know, a lot of people seem to agree on the topic that auto-generated documentation is bad, however, I don’t think it’s entirely true. There’s a popular post by Jacob Kaplan-Moss, which describes what to write in the documentation. It has a paragraph regarding auto-generated documentation:
Don’t.
Auto-generated documentation is almost worthless. At best it’s a slightly improved version of simply browsing through the source, but most of the time it’s easier just to read the source than to navigate the bullshit that these autodoc tools produce. About the only thing auto-generated documentation is good for is filling printed pages when contracts dictate delivery of a certain number of pages of documentation. I feel a particularly deep form of rage every time I click on a “documentation” link and see auto-generated documentation.
There’s no substitute for documentation written, organized, and edited by hand.
I’ll even go further and say that auto-generated documentation is worse than useless: it lets maintainers fool themselves into thinking they have documentation, thus putting off actually writing good reference by hand. If you don’t have documentation just admit to it. Maybe a volunteer will offer to write some! But don’t lie and give me that auto-documentation crap.
I’m not sure exactly what kind of documentation here was in mind, and I can agree if we’re talking about documentation that was generated from the code alone. I should also note that I have written a lot of documentation by hand before, and I would say that not all documentation is better when written like that. And documentation can be hard to write in general, it certainly was for me.
When working on a library, handwriting the documentation becomes very hard. Not only do you have to sync every documentation entry with every update in your code base, but you’ll also have to remember what’s in the documentation and what’s missing. And if you’re covering only the public API of your library it’s one thing, but when you also cover the private APIs for other developers to understand it’s a different kind of beast. Doing all of this by hand can leave you no time to work on features as a result.
The point is - it entirely depends on how documentation is generated, how it was written, and also for who you write it. Your handwritten documentation may as well be complete garbage, useless for the end user if you have written it in a bad style or with a great lack of detail. The same goes for auto-generated documentation - it can be good, if the tooling is good, and if programmers took their time in writing meaningful things in the documentation strings/comments.
I’m getting a bit ranty here.
So I think I found a compromise - I hand-write all documentation on a function level and generate the markdown files which enrich the documentation text with links between items automatically.
Paired with the fine-tuning of documentation order fenneldoc provides, and the ability to add a general notice on top, it is possible to create documentation that reads as a normal document, yet is still usable when you found yourself in the sources.
And it is never out of date, unless you forgot to update it, which also happens without auto-generating.
But there’s more. What I also wanted from Fenneldoc was documentation testing.
When I write documentation, I also include a lot of examples of how to use certain functions, or how they can be combined. But after I change these functions, some examples might stop working. And what’s good about having examples that don’t work when you try them?
So Fenneldoc runs all examples and ensures that they at least compile. You can include tests right into them, so they break when you’ve introduced an incompatible change and forgot to update the documentation. It also checks if you’ve mentioned all of the function’s arguments in the documentation, and warns if it couldn’t find a cross-link to an item.
For example, here’s another function from cljlib:
(defn into
"Returns a new coll consisting of `to` with all of the items of `from`
conjoined. A transducer `xform` may be supplied.
# Examples
Insert items of one collection into another collection:
```fennel
(assert-eq [1 2 3 :a :b :c] (into [1 2 3] \"abcd\"))
(assert-eq {:a 2 :b 3} (into {:a 1} {:a 2 :b 3}))
```
Transform a hash-map into a sequence of key-value pairs:
``` fennel
(assert-eq [[:a 1]] (into (vector) {:a 1}))
```
You can also construct a hash-map from a sequence of key-value pairs:
``` fennel
(assert-eq {:a 1 :b 2 :c 3}
(into (hash-map) [[:a 1] [:b 2] [:c 3]]))
```"
([] (vector))
([to] to)
([to from]
(match (getmetatable to)
{:cljlib/editable true}
(persistent! (reduce conj! (transient to) from))
_ (reduce conj to from)))
([to xform from]
(match (getmetatable to)
{:cljlib/editable true}
(persistent! (transduce xform conj! (transient to) from))
_ (transduce xform conj to from))))
You can see that it has a basic description of what it does, and a few examples of what happens when you use it.
Arguments to the function exist in the documentation, and they’re using a Markdown notation for monospaced text by being enclosed via the backticks, for instance, the `xform` argument.
Here the documentation is just a string of Markdown formatted text in general.
Below is the example section, which features code blocks denoted with the triple backticks, following a language name: ``` fennel ... ```.
In each example, there is a test, and you can see that all invocations are wrapped in assert-eq calls, which is an assertion macro from my fennel-test library.
In this particular example, I made a mistake and forgot to add the letter d into the result of the very first call to into.
When I try to export this documentation to Markdown, I get an error:
In file: 'init.fnl'
Error in docstring for: 'into'
In test:
``` fennel
(assert-eq [1 2 3 :a :b :c] (into [1 2 3] "abcd"))
(assert-eq {:a 2 :b 3} (into {:a 1} {:a 2 :b 3}))
```
Error:
assertion failed for expression:
(eq [1 2 3 "a" "b" "c"] (into [1 2 3] "abcd"))
Left: [1 2 3 "a" "b" "c"]
Right: [1 2 3 "a" "b" "c" "d"]
The error message comes from the assert-eq macro, which tests whether the left expression is equal to the right one.
Fenneldoc just makes it possible to see the test itself before it.
A simple test, but can catch some mistakes in the documentation!
I find this tool quite usable and a big part of my project workflow - every project I do has documentation generated with Fenneldoc, and I even read it from time to time, despite the fact that I was the one writing it.
Fennel test
Speaking of testing, I also created a fennel-test library because I didn’t like libraries aimed at testing Lua. My library provides its own test runner, and tests are structured similarly to how they are in Clojure. And if your project uses this library, it is possible to include it in the documentation testing pipeline as well, as I’ve shown before.
Now, including huge tests in the documentation is really bad - in my opinion it should hold only small examples that also act as tests. Also, I think that everything you write in the docstring should be displayed as is, so Fenneldoc doesn’t try to hide any stuff from the code blocks - the reader should see the whole picture. The Rust programming language took a different approach, even has hidden documentation tests, that are there only when you test the documentation, not when you export it. I don’t think this is the correct way, but the paragraph mentions that it has something to do with the internal limitations of their testing framework.
So you should write tests that are a lot more complicated than your documentation tests, but since it is reserved for the test suite it’s not a problem.
Here’s the test for the into function I’ve been talking about before:
(deftest test-into
(testing "into"
(assert-eq (core.into [] nil) [])
(assert-eq (core.into nil nil) nil)
(assert-eq (core.into nil [1 2 3]) (core.list 3 2 1))
(assert-eq (core.into [] []) [])
(assert-eq (core.into [1 2 3] []) [1 2 3])
(assert-eq (core.into [1 2 3] [4 5 6]) [1 2 3 4 5 6])
(assert-eq (core.into {} {}) {})
(assert-eq (core.into {:a 1} {}) {:a 1})
(assert-eq (core.into {:a 1} {:b 2}) {:a 1 :b 2})
(assert-eq (core.into [] {}) [])
(assert-eq (core.into {} []) [])
(assert-eq (. (getmetatable (core.into (core.vector) {})) :cljlib/type) :vector)
(assert-eq (. (getmetatable (core.into (core.hash-map) [])) :cljlib/type) :hash-map)
;; ... more various cases ...
))
Here, the deftest is a macro, that defines the test function which is then executed by the test runner.
The runner can shuffle tests, capture errors, and form the log afterward.
I like how it works, and it’s fine for my use cases.
There also are other facilities, like fixtures, which you can use to wrap each test, or all of the tests with additional code, but they only work when using the test runner.
OK, hopefully, this is all great and can help us write more understandable and maintainable programs in Fennel. What about support from text editors then?
Editor tooling
And text-editor support is another thing that often requires improvements when we’re dealing with young languages.
I’ve created Fennel syntax highlighting support for Kakoune, and I also currently maintain the fennel-mode package for Emacs.
I know that Visual Studio Code has some support for Fennel, and NeoVim has great support with interactive features, and even the ability to configure the editor in Fennel itself.
Similarly to NeoVim, there’s also a text editor, called Lite, which is written in Lua, and there are projects that make it extendable via Fennel, which is great!
Still, Emacs as of this moment has the best Fennel support (IMO), as it has good support for Lisps in general.
However, there are always places to improve tooling, and I improved Fennel’s support in Emacs by providing automatic completions, documentation popups, and a scratch buffer for a different approach for interactive sessions.
I have some more plans on improving the REPL support and dynamic highlighting of loaded macro forms.
In addition to that, I’ve made a small package to add support for Fennel to Org Mode, called ob-fennel. I have an extensive post that goes into detail about how it can be used, so if you’re interested you can read it.
There’s an interesting project on the horizon: fennel-ls - a language server for Fennel. A very welcome addition to the language ecosystem, and as far as I can tell after speaking with the author - a passionate project. And, as far as I can see, every mature language eventually gets a language server, so this is a good sign!
What’s next
There’s one point I want to replace in my current workflow with my Fennel project - coverage statistics.
Currently, I use the luacov project, which analyzes Lua code specifically.
This is not a huge problem for me - I can read Lua, and I learned to read Lua that Fennel generates, but it is still a point of inconvenience.
Ideally, I would like to have a tool similar to cloverage, which is a Clojure library for generating code coverage reports.
I like it, because not only it works with Clojure code, and not some Java decompiled byte code, but it also shows coverage in terms of forms, not just lines.
For Fennel, it is a huge deal, because like in many lisps, forms may sit on a single line.
Though it is not a problem because I opt out of the --correlate flag when I compile the project for coverage testing, so all one-liners are transformed into multiple lines, and I get pretty accurate coverage results.
Still inconvenient.
But all of the above is only a small fraction of what has happened with Fennel overall - it’s just things that I’ve experienced in my project workflow. There’s a lot of cool stuff happening even right now, and I welcome everybody to the upcoming FennelConf 2022.
From my point of view, Fennel grew a lot during the last two years. We had the first major release and extended the language with new macros, and there are a lot of projects in the works regarding making the compiler more general via a plugin system. I’m very excited about the future of Fennel and want to thank everybody in the community, which I’m glad I’m a part of.
And I believe this situation is not unique to Fennel. I know people who participate in other programming language-related communities, and I hear great things about them too. Being in a small community is also beneficial in that you get to know a lot of people, and when someone new comes with some crazy ideas it’s always great to see what they can create with the project you are also passionate about.
So keep hacking! I’m looking forward to seeing more projects that help make developing ecosystem better, and I do my best for the ones I participate in!