Andrey Listopadov

Archive Tags Categories Talks Notes About Projects RSS

deps.fnl 0.2.2 released

@announcement tools lua fennel ~9 minutes read

I just released a new version of the deps.fnl project!

After getting some feedback on the 0.1.0 version and addressing most of it, I’m ready to present a new version with improved dependency and conflict handling.

Major changes

The deps.fnl file format

The :deps field was changed back to use maps, as it was during the prototyping stage:

{:deps {"https://gitlab.com/andreyorst/async.fnl"
        {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}
        "https://gitlab.com/andreyorst/json.fnl"
        {:type :git :sha "eebcb40750d6f41ed03fed04373f944dcf297383"}
        "https://gitlab.com/andreyorst/reader.fnl"
        {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}
        "luasocket"
        {:type :rock :version "3.1.0-1"}}
 :paths
 {:fennel ["src/?.fnl" "src/?/init.fnl"]}}

Before this change the deps.fnl file would have looked like this:

{:deps [["https://gitlab.com/andreyorst/async.fnl"
         {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}]
        ["https://gitlab.com/andreyorst/json.fnl"
         {:type :git :sha "eebcb40750d6f41ed03fed04373f944dcf297383"}]
        ["https://gitlab.com/andreyorst/reader.fnl"
         {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}]
        ["luasocket"
         {:type :rock :version "3.1.0-1"}]]
 :paths
 {:fennel ["src/?.fnl" "src/?/init.fnl"]}}

The main reason for a change to the use of vectors was the fact that all of the Luarocks dependencies were using the same tree. Because conflict resolution was made during dependency installation, and maps are unordered, and if any of the libraries had a Luarocks library as its transient dependency, Luarocks would silently override the dependency. This made conflict checking nondeterministic, and thus error-prone.

This is no longer an issue, and thus the format can be changed back to maps.

Conflict resolution

Previously, deps would use a deterministic order of dependencies and the --allow-conflicts flag to resolve conflicts. This meant, that if your project had two dependencies, that had the same transient dependency but of a different version, --allow-conflicts would honor dependencies that appeared earlier in the order. For example, if you had dependencies A, and B, both of which depended on C of versions v1 and v2 like this:

;; project deps.fnl
{:deps [["A" {:version "1"}]
        ["B" {:version "1"}]]}

;; A deps.fnl

{:deps [["C" {:version "1"}]]}

;; B deps.fnl

{:deps [["C" {:version "2"}]]}

The deps script would throw an error:

[A:1]->[C:1] conflicts with [B:1]->[C:2]

Previously, this could be resolved by using the --allow-conflicts flag, meaning that deps would see the conflict, but ignore it, and use the C library from the first one encountered in the deps.fnl. So for the following deps.fnl:

{:deps [["A" {:version "1"}]
        ["B" {:version "1"}]]}

The libraries used would be A version 1, transient C version 1 from A, and B version 1.

However, for the following deps.fnl (A and B appear in the reverse order):

{:deps [["B" {:version "1"}]
        ["A" {:version "1"}]]}

The libraries used would be A version 1, transient C version 2 from B, and B version 1.

This wasn’t clear enough, and could introduce unexpected results when editing deps.fnl.

For example, adding another dependency before A that (unknowingly) also depends on yet another version of C would override the version for the project.

At first, I was thinking about using :exlusions key to specify that we want to exclude certain transient dependencies, but it also introduced the need to do this at arbitrary depth:

{:deps [["A" {:version "1"}]
        ["B" {:version "1"
              :exclusions ["C"]}]]}

And we would still not know what version of C we’re using.

So instead, conflict resolution works by specifying the specific version of dependency C at the same level as A and B:

{:deps {"A" {:version "1"}
        "B" {:version "1"}
        "C" {:version "1"}}}

This way the project author is in full control of what versions will be used.

The --allow-conflicts flag is no more. By default, conflicts are checked, and the only way to resolve conflicts is either to update the library in question, or override it at the root level.

Conflict resolution is now module-based

Another new feature I’m pretty excited about is module-based conflict resolution. In the 0.1.0 version of deps it was not possible to detect conflicts when something is a fork.

What I mean by that is, say, I have a library: async.fnl and someone decides to fork it to change something or fix a bug. They add the following to their project deps.fnl:

{:deps {"https://gitlab.com/some-fork/async.fnl"
        {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}
        "https://gitlab.com/andreyorst/async.fnl"
        {:type :git :sha "98e8680b46a777f9ddcff9e761b782ec6bf077f5"}}}

The main question thus was - what async.fnl implementation would actually be loaded? We could answer this question if :deps was a list instead of a table because the ordering would rule out and (require :io.gitlab.andreyorst.async) would load the first library. However, with deps stored as a map, we would have gotten different implementations each time because key hashing is randomized.

The problem is - previously deps didn’t see this as a conflict. These two libraries are the same library but different projects.

Now, instead of comparing just library names (URLs in this case) and versions (SHAs for Git dependencies), deps also compares exported modules. So we can detect the conflict because both libraries export io.gitlab.andreyorst.async module. If the person who forked the library chooses to change the module name to io.gitlab.some-fork.async instead, there won’t be a conflict, as this is now a properly different library.

The example above is unlikely to happen as is, but if some library uses a fork as a transient dependency this might happen. This was pointed out to me by Ambrose Bonnaire-Sergeant off the mailing list, and we’ve come up with this solution.

Luarocks (transient) dependencies

Another problem with transient dependencies is the fact that Luaorcks dependency manifests are very lax about versions of the transient dependencies. Usually, you would see just library names, and sometimes version ranges. For example, here’s a fragment from the lapis library manifest describing dependencies:

dependencies = {
  "lua",

  "ansicolors",
  "argparse",
  "date",
  "etlua",
  "loadkit",
  "lpeg",
  "lua-cjson",
  "luaossl",
  "luasocket",
  "pgmoon",
}

The pgmoon dependency itself depends on lpeg. So Luarocks default strategy is to pull the latest available version of the library.

This makes conflict resolution harder, because deps.fnl requires specifying a concrete version of any dependency. Paired with the fact that Luarocks silently overrides already installed dependencies when you give it a specific version to install, and the fact that deps.fnl is now unordered, the only way to get deterministic versions from Luarocks was to ask users to specify all of the transient dependencies:

{:deps {"lapis"
        {:type :rock :version "1.16.0-1"
         :dependencies
         {"loadkit" {:type :rock :version "1.1.0-1"}
          "etlua" {:type :rock :version "1.3.0-1"}
          "luasocket" {:type :rock :version "3.1.0-1"}
          "lua-cjson" {:type :rock :version "2.1.0.10-1"}
          "argparse" {:type :rock :version "0.7.1-1"}
          "lpeg" {:type :rock :version "1.1.0-2"}
          "ansicolors" {:type :rock :version "1.0.2-3"}
          "date" {:type :rock :version "2.2.1-1"}
          "pgmoon" {:type :rock :version "1.16.0-1"
                    :dependencies {"lpeg" {:type :rock :version "1.1.0-2"}}}
          "luaossl" {:type :rock :version "20220711-0"}}}}}

The :dependencies key is akin to the dependencies map in the rock manifest.

The conflict resolution works exactly the same.

Runtime library manipulation with deps.add-lib, and sync-deps

Something that escaped my previous announcement, because it was developed a bit later is that it is now possible to add libraries while working on the application without restarting the REPL. When the REPL is started via the deps --repl command, a new deps module is available:

>> (require :deps)
{:_VERSION "0.2.0"
 :add-lib #<function: 0x560a243c2990>
 :add-libs #<function: 0x560a245b5820>
 :sync-deps #<function: 0x560a246d8d50>}

Thus, if you’re working on some project and in need of a new dependency, or want to compare a few and choose which one to use, you can quickly do so in the REPL:

>> (deps.add-libs {"bump" {:type :rock :version "3.1.7-1"}
                   "anim8" {:type :rock :version "v2.3.1-1"}})
["bump" "anim8"]
>> (deps.add-lib
    "penlight"
    {:type :rock :version "1.14.0-2"
     :dependencies {"luafilesystem" {:type :rock :version "1.8.0-1"}}})
["pl.input" "pl.path" "pl.array2d" "pl.compat" "pl.luabalanced" "pl.pretty"
 "pl.import_into" "pl.dir" "pl.func" "pl.Map" "pl.comprehension" "pl.app" "pl.seq"
 "pl.sip" "pl.file" "pl.stringx" "pl.class" "pl.Date" "pl.strict" "pl.template"
 "pl.types" "pl.permute" "pl.MultiMap" "pl.text" "pl.utils" "pl.OrderedMap" "pl.xml"
 "pl.url" "pl.tablex" "pl.test" "pl.lapp" "pl.List" "pl.Set" "pl.operator" "pl.data"
 "pl.lexer" "pl.config" "pl.init" "pl.stringio" "lfs"]

The returned value is a table of module names that are now available to require.

Alternatively, the deps.fnl file can be edited, and changes can be synced into the running REPL via the sync-deps call.

New flags

Several new flags were added.

--tree

Prints the dependency tree, outlining what dependencies are on your path, and what transient dependencies are brought by each dependency:

[["lapis" "1.16.0-1"
  [["argparse" "0.7.1-1"]
   ["luaossl" "20220711-0"]
   ["pgmoon" "1.16.0-1"
    [["lpeg" "1.1.0-2"]]]
   ["date" "2.2.1-1"]
   ["luasocket" "3.1.0-1"]
   ["loadkit" "1.1.0-1"]
   ["ansicolors" "1.0.2-3"]
   ["lpeg" "1.1.0-2"]
   ["etlua" "1.3.0-1"]
   ["lua-cjson" "2.1.0.10-1"]]]
 ["andreyorst/fennel-test"
  "9aae4dd52be1f5dd1ee97e61c639e6c5a2f9afee"]]

--fennel-ls

The deps script is now able to create or update an existing configuration for the fennel-ls project.

When calling deps --fennel-ls a flsproject.fnl file will be created if it doesn’t exist already. This file will contain all of the PATH-related data:

{:fennel-path "test/?.fnl;.deps/git/andreyorst/fennel-test/9aae4dd52be1f5dd1ee97e61c639e6c5a2f9afee/src/?/init.fnl"
 :lua-version "lua54"
 :macro-path ".deps/git/andreyorst/fennel-test/9aae4dd52be1f5dd1ee97e61c639e6c5a2f9afee/src/?/init.fnlm"}

--profiles

Another new feature - profiles! Before this flag existed profiles were kind of possible by using --merge and additional deps.fnl files:

;; deps.fnl
{:deps {"https://gitlab.com/andreyorst/reader.fnl"
        {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}}
 :paths
 {:fennel ["src/?.fnl"]}}

;; dev-deps.fnl
{:deps {"https://gitlab.com/andreyorst/fennel-test"
        {:type :git :sha "72a74394f89fdaf15abf11f33752a07e9e0bee91"}}
 :paths {:fennel ["tests/?.fnl"]}}

Having these two files, a final deps map can be obtained with deps --merge dev-deps.fnl:

;; output of `deps --merge dev-deps.fnl --show`
{:deps {"https://gitlab.com/andreyorst/fennel-test"
        {:sha "72a74394f89fdaf15abf11f33752a07e9e0bee91" :type "git"}
        "https://gitlab.com/andreyorst/reader.fnl"
        {:sha "252ea2474cb7399020e6922f700a5190373e6f98" :type "git"}}
 :paths {:clua {} :fennel ["src/?.fnl" "tests/?.fnl"] :lua {} :macro {}}}

This works, but semantically it is a bit unclear. To know that the development “profile” exists you need to actively look for it in the project files. Multiple such profiles also pile up in the project, which is not great.

Now, this can be done a bit more consistently with:

;; deps.fnl
{:deps
 {"https://gitlab.com/andreyorst/reader.fnl"
  {:type :git :sha "252ea2474cb7399020e6922f700a5190373e6f98"}}

 :paths
 {:fennel ["src/?.fnl"]}

 :profiles
 {:dev
  {:deps {"https://gitlab.com/andreyorst/fennel-test"
          {:type :git :sha "72a74394f89fdaf15abf11f33752a07e9e0bee91"}}
   :paths {:fennel ["tests/?.fnl"]}}}}

And the same final deps map can be obtained with deps --profiles dev:

;; output of `deps --profiles dev --show`
{:deps {"https://gitlab.com/andreyorst/fennel-test"
        {:sha "72a74394f89fdaf15abf11f33752a07e9e0bee91" :type "git"}
        "https://gitlab.com/andreyorst/reader.fnl"
        {:sha "252ea2474cb7399020e6922f700a5190373e6f98" :type "git"}}
 :paths {:clua {} :fennel ["src/?.fnl" "tests/?.fnl"] :lua {} :macro {}}}

The --merge flag is here to stay for compatibility. Maybe there are other use cases for it too, so it won’t be removed unless deemed unnecessary. Moreover, --profiles actually supports multiple comma-separated profile names, so it is much more compact compared to --merge:

$ deps --profiles dev,test,doc ...
$ deps --merge dev-deps.fnl --merge test-deps.fnl --merge doc-deps.fnl ...

Upcoming work

I’ll probably take a short break from working on deps for a while, or at least I won’t work on it as actively. Both to let it settle a bit, give it time to gather some feedback, and let me rest a bit too. I’m feeling good about what I did so far, and deps in my opinion is out of the alpha stage for sure. So I think it can be safely used in other projects, and I’m still open for feedback!

As for plans, I have something I want to experiment with.

I’ve started outlining an interface for custom backend implementations. Right now there are only two backends: Git and Luarocks, but if the community desires, I’ll extend deps in such a way that custom backends could be added via some sort of a plugin system. Thus other ways to download and add dependencies would be possible. For example, recently I learned that some Lua libraries exist only in SVN. There may be others that are only available through GNU Bazaar, the new hot Jujutsu VCS, or maybe some use Mercurial, I don’t know!

Additionally, I think it is safe to say, that using version control systems for libraries is not a great idea, so I might start working on a dependency packaging system for Fennel libraries that we could later upload to some package archive. Thinking of a name - Fennel Bulbs sounds fun!

Dependency building needs to be changed to be a bit more robust. Right now I think it should work fine, but probably only on *nix-based systems. Windows support for deps.fnl is there, but completely untested. So if any Fennel users happen to do their stuff on Windows - let me know if it works, please!

That’s all for now. Huge thanks to Phil Hagelberg, Ambrose Bonnaire-Sergeant, and Emma B. for the feedback and code reviews!

Comment via email