Andrey Listopadov

Archive Tags Categories Talks Notes About Projects RSS

deps.fnl - a new dependency manager for Fennel projects

@announcement tools lua fennel ~6 minutes read

I’d like to present a new project, aimed at one of the areas where the Fennel ecosystem can be improved - project dependency management.

Other programming languages have tools to pull in, build, and load external dependencies. Java has Maven, Clojure has Leiningen and other projects like this, Python has Pip and Poetry, and Lua has Luarocks. All these tools provide ways for pulling in dependencies concerning how the language is organized.

Since Fennel runs on Lua, we could use Luarocks, but, well, it’s not really good. It has a lot of design problems and doesn’t fit for Fennel for a number of reasons, like:

  • It is tailored for Lua dependencies only, so shipping Fennel libraries would require AOT compiling them to Lua;
  • Luarocks only knows how to build LUA_PATH and LUA_CPATH, but Fennel requires FENNEL_PATH and more importantly FENNEL_MACRO_PATH;
  • Shipping macros is hard or impossible.

A few notes on why I decided to make a dependency manager

I’ve made a lot of libraries for Fennel and encountered various problems with shipping these libraries to users.

Most of the Fennel libraries I write feature user-exposed macros, which means that the library can’t be shipped as a single file most of the time. I did find a workaround for that eventually, but it still poses some interesting challenges for library writers.

Another problem I encountered are libraries that need other libraries to work. Some of my libraries depend on other libraries I made, and thus I either need to include them in the source code directly for shipping or do some other trickery. And libraries depending on other libraries (transient dependencies) could potentially conflict with other dependencies in the project.

Finally, multifile libraries that can’t be compiled into a single file, since they have multiple modules that serve as entry points are hard to ship because users will have to properly set up PATH in their project.

So I made deps.fnl to solve these problems.

deps.fnl

This project consists of a single script, called deps, that works as a drop-in replacement for the fennel executable. This means that you can call deps with any arguments that you specify to Fennel, and it will work transparently. In other words:

deps --require-as-include --compile file.fnl > file.lua

But the main thing that deps was designed to do is to set up the environment for your project. When the deps.fnl file is present in the project, the deps script analyzes this file and pulls in all of the specified libraries, sets all required PATH variables, like LUA_PATH, LUA_CPATH, FENNEL_PATH, and FENNEL_MACRO_PATH, and launches fennel.

For example, here’s a deps.fnl for my fnl-http project:

{:deps
 [["https://gitlab.com/andreyorst/async.fnl"
   {:type "git" :sha "a83b13b397fdfab3ebf7f17a16d21a7ec2674ceb"}]
  ["https://gitlab.com/andreyorst/json.fnl"
   {:type "git" :sha "bfb8d7d03c26619768eefaa47782a3e95dbe5156"}]
  ["https://gitlab.com/andreyorst/reader.fnl"
   {:type "git" :sha "3ff2bc790c8b7922267af5712a64a14572a172cf"}]
  ["luasocket"
   {:type "rock" :version "3.0rc1-2"}]]
 :paths
 {:fennel ["src/?.fnl" "src/?/init.fnl"]}}

As can be seen, this project depends on four libraries: async.fnl, json.fnl, reader.fnl, and luasocket. Previously I had to vendor these dependencies in the repository, and instruct people how to install fnl-http and use it as a library.

Moreover, fnl-http is a multi-file library, so manually setting up PATH for it and all of its dependencies is not a pleasant task. But with deps you can use this library in your project by placing this deps.fnl file at your project root:

{:deps [["https://gitlab.com/andreyorst/fnl-http"
         {:type "git" :sha "ea885a8c767a627206126cc36f1befb1a2d71850"}]]
 :paths {:fennel ["?.fnl"]}}

Then just call deps --repl, and you’re good to go:

$ deps --repl
processing git repo: https://gitlab.com/andreyorst/fnl-http
processing git repo: https://gitlab.com/andreyorst/async.fnl
processing git repo: https://gitlab.com/andreyorst/json.fnl
processing git repo: https://gitlab.com/andreyorst/reader.fnl
processing rock: luasocket 3.0rc1-2
Welcome to Fennel 1.5.1-dev on PUC Lua 5.4!
Use ,help to see available commands.
>> (require :io.gitlab.andreyorst.fnl-http.client)
{:connect #<function: 0x5605682104e0>
 :delete #<function: 0x560568bc64c0>
 :get #<function: 0x5605685315c0>
 :head #<function: 0x5605689c9ca0>
 :options #<function: 0x5605682b0a20>
 :patch #<function: 0x5605687d8f40>
 :post #<function: 0x560568c40270>
 :put #<function: 0x5605681d9d80>
 :request #<function: 0x560568b37450>
 :trace #<function: 0x560568580320>}

In the output above after analyzing the project’s deps.fnl the deps script proceeds to download dependencies. The project’s only dependency is fnl-http, but after downloading it, the deps script check if there’s a deps.fnl in the root of the repository. Since there is, it proceeds to download all transient dependencies. Calling deps --repl the second time would not download anything, unless new dependencies were added to the deps.fnl file or hashes/version were changed.

After processing all of the dependencies the REPL is started as requested, and the library is ready to be used.

Dependency types

Dependencies can be of two types - git and rock.

Git dependencies are just a link to the repository and a commit hash to checkout. Until we have some kind of a registry and a way to package libraries for downloading this is the best we can do. But as seen above, this already makes using libraries a lot simpler.

Rock dependencies are processed with Luarocks. Yes, deps wraps not only the Fennel executable but Luarocks as well. Luarocks is fine enough to be used like this and provides enough command-line arguments for deps needs.

All of the dependencies are stored in the project’s root unless requested elsewhere. I decided not to go the .m2 route. One reason for that is the fact, that Luarocks can’t install two versions of the same library into a single “tree”, as they call it. Instead, it overrides any library already present in the tree. This is probably related to how the LUA_PATH is constructed by Luarocks.

Speaking of PATH=⁣s. The =deps.fnl file contains a mandatory field :paths, that contains a table. This table specifies the project’s path configuration. Each library may require a different path configuration, and when deps script processes a said deps.fnl file, it reads the path table and constructs the final PATH for your project.

You can analyze the path generated for the project with deps --path. It prints the path to standard out in a format similar to luarocks path.

Finally, some Git repositories may not use deps.fnl, but it is still possible to manage them with deps.fnl. For example, my library fennel-test doesn’t yet feature a deps.fnl file, but the deps script itself uses it for testing. To make this work, you can specify :paths field for each dependency:

{:deps [["https://gitlab.com/andreyorst/fennel-test"
         {:type :git
          :sha "01ea080dc8176512c5890462252c5bf214baf137"
          :paths {:fennel ["?.fnl"] :macro ["?.fnl"]}}]]
 :paths {}}

Thus, we can even pull in pure Lua dependencies, and specify the :paths table with something like {:lua ["?.lua"]}, and deps will set everything up.

You can read more information on the deps.fnl file format, and deps script usage in the project’s readme.

I need feedback

I have high hopes for this project, and I’m looking for feedback. If you’re working on some application that requires external dependencies, give deps.fnl a shot. If you’re a library maker like me, consider using deps.fnl so your users have an easier way of using your work. You can submit feedback either via email, or GitLab issues.

If the project gets enough traction and is to community’s satisfaction, Phil and I are discussing the possibility of including this into Fennel itself, so no installation would be required. Since deps is already a drop-in replacement for fennel, once deps is a part of the fennel executable, all commands from the above would work with just fennel itself.

Thanks for reading, and I hope this project will be useful to you!

Comment via email