Condition system for Fennel language and Lua runtime
Not long ago I posted a small article on a Condition System in Clojure language. In that article, I was mostly trying to understand what a condition system is, and how it can enhance error handling in the code I write. Since that time, I’ve understood this system a lot better, by actually trying it in the Common Lisp language, the place where it came from, as far as I know. And you know what it means (because you’ve read the title) - it’s time to implement such a system for Fennel!
I like Fennel a lot, it’s a really cool language to tinker with - it’s small, slick, Lua runtime is actually really versatile, and can be bent to your needs without a lot of extra work. But the reason I’ve decided to implement this library for Fennel, and not for Lua itself, is because of a feature that is not available in Lua but is available in Fennel - lisp macros. The library is already available for everyone to use: fennel-conditions, and in this article, I’ll do a deep dive into the details behind the implementation.
Condition system?
First, let’s actually explain what the condition system is.
Condition system is a feature of the Common Lisp language, that can be very roughly described as resumable exceptions. Though, while this isn’t the only way you can use the condition system, it is the main reason I’ve decided to port this system into Fennel, as Lua error handling is not the best, and we can do better. You still can use a condition system for basic control flow to some extent, though not as versatile as in Common Lisp, I don’t think that Fennel really needs these features.
In Common Lisp, a condition is an object, which represents a certain event. You can bind handlers for particular events, and trigger these handlers by raising a condition. This may sound similar to the classic exception model found in languages like Java, but it has one major difference. Exceptions are a two-part system, while the condition system is three-part.
It’s really hard to come up with a small example to show what condition the system has to offer that exceptions don’t, and I don’t really know Common Lisp well enough to write it. So I encourage you to read Chapter 19 from the Practical Common Lisp book which covers this topic in detail before continuing reading. There’s also a good article on the wiki.c2.com. If you’ve read that before or just now, and didn’t understand what the condition system is, hopefully, you’ll get the idea in this post, but I can’t guarantee that.
If I were to describe the main difference between this system from classic exceptions, I would say that when you write a code that throws an exception, you have to deal with this exception somewhere else. And the place where you handle that exception is the place where the program would continue running after the handling is done.
In the condition system, the place where you’ve thrown such an exception can be the place where the program continues running if you use some special constructs, that allow you to return to that place.
An example
In the spirit of the Practical Common Lisp book example mentioned above, here’s a rather naive Clojure code that parses a log file, and uses exceptions to indicate that parsing failed:
(ns log-parser
(:require [clojure.java.io :as io])
(:import [clojure.lang ExceptionInfo]))
(defn well-formed?
"Checks if the line contains basic date-time, a service name, and a message
separated by colons."
[line]
(let [date-re #"([12]\d\d\d-[01]\d-[0-3]\d)"
time-re #"([01]\d:[0-5]\d:[0-5]\d\.\d+)"
service-re #"([^:]+)"
message-re #"(.*)"]
(re-matches (re-pattern (str date-re "T" time-re ":" service-re ":" message-re))
line)))
(defn parse-line
"Extract `date`, `time`, `service` name, and a `message` from a `line`."
[line]
(if-let [[_ date time service message] (well-formed? line)]
{:date date
:time time
:service service
:message message}
(throw (ex-info "malformed log entry" {:line line}))))
(defn parse
"Return a vector of parsed lines, each element being a map with the
following keys: `:date`, `:time`, `:service`, `:message`."
[lines]
(try
(mapv parse-line lines)
(catch ExceptionInfo e
(binding [*out* *err*]
(prn (format "%s:%s" (ex-message e) (:line (ex-data e))))))))
(defn parse-log [file]
(with-open [f (io/reader file)]
(parse (line-seq f))))
Note: Yes, there are more idiomatic ways of doing this task in Clojure, but for the purposes of demonstrating the difference between exceptions and conditions this will do.
So this code checks if the line has a valid format, and if not it will throw an exception.
And here’s a problem - if a single line is invalid, we will get no results at all.
We can avoid that by moving try into our iteration:
(defn parse
"Return a vector of parsed lines, each element being a map with the
following keys: `:date`, `:time`, `:service`, `:message`."
[lines]
(mapv #(try
(parse-line %)
(catch ExceptionInfo e
(binding [*out* *err*]
(prn (format "%s: %s" (ex-message e) (:line (ex-data e)))))))
lines))
This will correctly parse our log, but we will have nil in place of each malformed line.
This can once again be fixed by using keep instead of mapv, but what if we can’t afford to lose any lines?
Maybe we can fix the line and parse it one more time?
This again can be done by modifying our anonymous function, so it would catch the exception, try to fix the line, and then call parse-line again.
And if we can’t fix the line, we will log it as before, and don’t add it to the resulting vector:
(defn fix-line
"We don't care when this happened, just want to know what happened."
[line]
(when-let [[_ service message] (re-matches #".*:([^:]+):([^:]+)$" line)]
(str "1970-01-01T00:00:00.000:" service ":" message)))
(defn parse
"Return a vector of parsed lines, each element being a map with the
following keys: `:date`, `:time`, `:service`, `:message`."
[lines]
(into []
(keep #(try
(parse-line %)
(catch ExceptionInfo e
(if-let [fixed-line (fix-line (:line (ex-data e)))]
(parse-line fixed-line)
(binding [*out* *err*]
(prn (format "%s: %s" (ex-message e) (:line (ex-data e)))))))))
lines))
But then again, when we will use parse-line in some other part of the program we will need to do all this stuff over and over.
The main problem here is that we’re building concrete ways to recover from error as a part of the calling function that is meaningful only in the context of the parse-line and the responsibility for recovering should really be on it.
Condition system can help us write more generic code, that knows how it can recover from the error, but let the recovery happen on the calling side.
Consider the following Fennel code that uses fennel-conditions library:
(local {: error : make-condition : invoke-restart} (require :fennel-conditions))
(import-macros {: handler-bind : restart-case : define-condition} :fennel-conditions)
(define-condition MalformedLogEntry)
(fn well-formed? [line]
"Checks if the line starts with basic date-time followed by a colon and a word."
(let [date-pat "([12]%d%d%d%-[01]%d%-[0-3]%d)"
time-pat "([01]%d:[0-5]%d:[0-5]%d%.%d%d%d)"
service-pat "([^:]+)"
message-pat "(.*)"]
(string.match line (.. date-pat "T" time-pat ":" service-pat ":" message-pat))))
(fn parse-line [line]
"Extract `date`, `time`, `service` name, and a `message` from a `line`."
(restart-case
(match (well-formed? line)
(date time service message) {:date date
:time time
:service service
:message message}
_ (do (print line) (error (make-condition MalformedLogEntry {:line line}))))
(:use-value [x] x)
(:reparse-line [fixed-line] (parse-line fixed-line))
(:ignore [ignore-val] ignore-val)))
(fn fix-line [line]
"We don't care when this happens, just want to know what happened."
(match (string.match line ":([^:]+):([^:]+)$")
(service message) (.. "1970-01-01T00:00:00.000:" service ":" message)))
(fn parse [lines]
"Return a vector of parsed lines, each element being a map with the
following keys: `:date`, `:time`, `:service`, `:message`."
(handler-bind [MalformedLogEntry
(fn [e {: line}]
(match (fix-line line)
fixed-line (invoke-restart :reparse-line fixed-line)))
MalformedLogEntry
(fn [e {: line}]
(io.stderr:write (.. "malformed log entry: " line "\n"))
(invoke-restart :ignore nil))]
(icollect [_ line (ipairs lines)]
(parse-line line))))
(fn parse-log [file]
(with-open [f (io.open file :r)]
(when f
(parse (icollect [line (f:lines)] line)))))
For those unfamiliar with condition system special forms or Fennel itself, handler-bind is something similar to try with the difference that the only way to recover from the error is by calling invoke-restart 1.
with-open is the same thing as in Clojure or Common Lisp, and icollect is something similar to mapv in Clojure, but it’s a macro, that accepts body form instead of function.
So what’s different about this code from the code in Clojure’s example?
Let’s look at parse-line functions in Clojure and in Fennel:
;; Clojure
(defn parse-line
"Extract `date`, `time`, `service` name, and a `message` from a `line`."
[line]
(if-let [[_ date time service message] (well-formed? line)]
{:date date
:time time
:service service
:message message}
(throw (ex-info "malformed log entry" {:line line}))))
;; Fennel
(fn parse-line [line]
"Extract `date`, `time`, `service` name, and a `message` from a `line`."
(restart-case
(match (well-formed? line)
(date time service message) {:date date
:time time
:service service
:message message}
_ (error (make-condition MalformedLogEntry {:line line})))
(:use-value [x] x)
(:reparse-line [fixed-line] (parse-line fixed-line))
(:ignore [ignore-val] ignore-val)))
Apart from some syntactic differences, you can see that both functions are throwing an error - by using throw in Clojure, and error in Fennel.
However, while the Clojure version only throws the error, the Fennel version does some additional things.
What we’re interested in here is the restart-case macro, which is a special macro from the condition system, that allows us to return to its location when we want to handle the error.
So when parse-line throws an error in Clojure, JWM unwinds the stack, and we basically leave this function completely.
We then catch the exception in the parse-log function, which can’t do anything about it but handle it right there or let it fall through.
However, when parse-line throws an error in Fennel, we don’t actually unwind the stack at all.
Instead, we travel up the dynamic scope, in search of a handler bound to this error by the handler-bind macro in the parse function:
;; ---- 8< ----
(handler-bind [MalformedLogEntry
(fn [e {: line}]
(match (fix-line line)
fixed-line (invoke-restart :reparse-line fixed-line)))]
;; ---- 8< ----
)
This handler tries to fix the line, and calls (invoke-restart :reparse-line fixed-line).
invoke-restart is another special function in condition system - it once again travels through the second dynamic scope, and finds that there’s a restart named :reparse-line bound by restart-case in the parse-line function.
This restart is executed right there, despite the fact that we’ve “left” the lexical scope of that function to find a handler, and its value is used in the original parse-line call.
So the order in which all functions are called looks something like that:
In other words, when we wrote parse-line in Clojure, we only defined how to detect malformed lines and how to signal an error.
When we wrote parse-line in Fennel, we not only specified what to do if the line is malformed but also provided concrete ways to recover, that are meaningful in the context of this function.
When we caught the exception in Clojure, we had no choice but to handle it right there. In Fennel, when we caught the exception we could decide whether we can handle the error, and how we can handle it without leaving the original call.
I hope this illustration is clear enough to demonstrate the main difference between exception and condition. Again, yes, there are better ways of dealing with this particular error in Clojure, but what I wanted to illustrate is that there’s no way of achieving the same flexibility of the condition system with just exceptions. I’ve covered possible ways of implementing a rather basic condition system in pure Clojure in my previous post Condition System in Clojure language, so it’s not entirely impossible, but has its own shortcomings. With that in mind let’s talk about the implementation of this system.
The implementation
As I’ve mentioned previously, the dynamic scope is what allows us to travel through our code without transferring the control. However, Lua doesn’t have a dynamic scope builtin into the language. So how is this possible?
Well, as you might know, Lua has tables, and these tables are dynamic and mutable. So theoretically, we could create a table, that is available globally, properly encapsulate it, and use it as our dynamic scope. And that’s exactly what the fennel-conditions library does.
However, implementing dynamic scope via tables is not enough. First of all, we need to manage the dynamic scope in a sensible way. Second, we need some way of actually traveling through the lexical scope to the point where the dynamic scope was created, once we know where we need to move after all handling has been done. And finally, the scope should be thread-local, and even though Lua doesn’t have threads, it has coroutines, which can act as threads, so we should maintain dynamic scope with respect to this aspect of the language.
Let’s begin by listing things that will be implemented in the condition system:
- condition objects - conditions are objects, that can have inheritance, and can embed additional information inside to convey more info about a conditional situation when needed.
Condition objects are defined with
define-conditionmacro, and a particular instance is created from a base condition object withmake-conditionfunction. handler-case,handler-bind,restart-case- macros for creating new frame of the dynamic scope, in which the handlers, and restarts are bound to condition objects.
As well as:
cerror- a macro for raising a condition that can be ignored by invoking specialcontinuerestart.ignore-errors- a macro for suppressing all errors, similarly totrywith a catchall clause that does nothing.unwind-protect- a macro for catching all errors, similar totrywith just afinallyclause.
All these macros share the very common part about setting up and cleaning the dynamic scope, as well as some pcall trickery to make it all work, which makes it a good entry point to understand what’s happening under the hood.
The dynamic scope
Beware, this is going to get really messy. Dynamic scoping is a foreign concept for Lua, so we have to deal with it ourselves, and ensure that it is correct. But before we begin, I should illustrate the structure of the library:
fennel-conditions/
├─ impl/
│ ├─ condition-system.fnl
│ ├─ debugger.fnl
│ └─ utils.fnl
├─ init.fnl
└─ init-macros.fnl
The init.fnl, and init-macros.fnl at the top level are the main entry points for runtime functions, and macros.
These modules depend on the impl/condition-system.fnl module - the library’s private API.
Which in turn uses impl/debugger.fnl that holds implementation of the interactive debugger, and impl/utils.fnl which stores common functions, used by condition system and debugger.
The main things we’re interested in are init.fnl, init-macros.fnl, and impl/condition-system.fnl - the meat and bones of the library.
The dynamic scope is stored in the impl.condition-system module and this module is required every time we need to access the dynamic scope.
Lua has a system for caching required modules, so this is a rather cheap operation and this way we don’t need to expose dynamic scope in the global scope.
This is how the dynamic scope table is defined in impl.utils:
(local {: metadata : view} (require :fennel))
(local dynamic-scope
(metadata:set
{}
:fnl/docstring
"Dynamic scope for the condition system.
Dynamic scope is a maintained table where handlers and restarts are
stored thread-locally. Thread name is obtained with
`coroutine.running` call and each thread holds a table with the
following keys `:handlers`, `:restarts`, and `:current-context`.
Handlers and restarts itselves are tables."))
It’s nothing special really, just an ordinary table, but the documentation string mentions several interesting points.
First, that this is a maintained table, meaning that it should not be modified manually.
Second, that it stores scopes in thread-local contexts, meaning that each thread will have its own dynamic scope table.
Each such table in turn has a table for handlers, a separate table for restarts, and some current-context field, which we will cover a bit later.
So, to illustrate how the dynamic scope may look at some point of the program running:
{"thread: 0x564cab9a12a8"
{:handlers {:handler {#<condition MalformedLogEntry: 0x564caba2c6e0>
#<function: 0x564cabb01500>}
:handler-type "handler-bind"
:parent {:handler {#<condition MalformedLogEntry: 0x564caba2c6e0>
#<function: 0x564caba5ccf0>}
:handler-type "handler-bind"
:parent {}
:target {}}
:target {}}
:restarts {:parent {:parent {:parent {}
:restart {:args ["ignore-val"]
:name "ignore"
:restart #<function: 0x564cabb01580>}
:target {}}
:restart {:args ["fixed-line"]
:name "reparse-line"
:restart #<function: 0x564cabaa7a10>}
:target {}}
:restart {:args ["x"]
:name "use-value"
:restart #<function: 0x564caba5cd30>}
:target {}}}}
This is the state of the dynamic scope when we enter the restart-case macro and do all setups, in the fennel code from the example section.
There’s quite a lot happening here - lots of nested tables with similar internal structures, and even some empty tables.
First thing you can spot is that the outermost table has a key "thread: 0x564cab9a12a8" which holds the table with handlers and restarts.
This is the thread-local dynamic scope, representing the main thread.
Inside of it in the table under the handlers key, we see a table with a handler key holding a table with a condition, (which is actually just another table2), and a function:
{:handler {#<condition MalformedLogEntry: 0x564caba2c6e0>
#<function: 0x564cabb01500>}
:handler-type "handler-bind"
:parent {:handler {#<condition MalformedLogEntry: 0x564caba2c6e0>
#<function: 0x564caba5ccf0>}
:handler-type "handler-bind"
:parent {}
:target {}}
:target {}}
This table corresponds to all dynamic scopes set up with handler-bind macros.
But as you can see here, it has a parent key, that holds another very similar table, but as you might remember, there was only one handler-bind instance in the example code - why there are two tables?
Here’s a quick refresher of how handler-bind was used:
(handler-bind [MalformedLogEntry
(fn [e {: line}]
(match (fix-line line)
fixed-line (invoke-restart :reparse-line fixed-line)))
MalformedLogEntry
(fn [e {: line}]
(io.stderr:write (.. "malformed log entry: " line "\n"))
(invoke-restart :ignore nil))]
;; ---- 8< ----
)
Note that there are two separate entries for MalformedLogEntry condition.
Each such entry creates a separate frame in the dynamic scope, which results in two nested tables as shown above.
This is done this way because the handlers are bound sequentially, and their order in the handler bind determines the order of how they will be called.
We directly use this property, by writing the first handler in such a way, that if the line can’t be fixed with fix-line function, we are not calling invoke-restart and the condition falls through to the next handler.
This handler in turn calls :ignore restart, and successfully recovers from the error.
Now let’s look at the dynamic scope stored under :restarts:
{:parent {:parent {:parent {}
:restart {:args ["ignore-val"]
:name "ignore"
:restart #<function: 0x564cabb01580>}
:target {}}
:restart {:args ["fixed-line"]
:name "reparse-line"
:restart #<function: 0x564cabaa7a10>}
:target {}}
:restart {:args ["x"]
:name "use-value"
:restart #<function: 0x564caba5cd30>}
:target {}}
Very similarly, each level stores a :restart table, with an argument list, name, and the function itself.
Under the :parent key, the very similar table is stored, holding the next restart.
Since our code defined three restarts, we get a table of the depth of three.
One might wonder, why this is necessary, restarts are selected and not tried in order.
While this is true, nothing prevents you from creating two restarts with the same name:
(restart-case
(/ 1 nil)
(:use-value [x] x)
(:use-value [x] x))
This will create the following table:
{:parent {:parent {}
:restart {:args ["x"]
:name "use-value"
:restart #<function: 0x564cabb01380>}
:target {}}
:restart {:args ["x"]
:name "use-value"
:restart #<function: 0x564cabaa5b40>}
:target {}}
You then will be able to choose which restart to call via the interactive debugger interface. But more about that later.
Now that we have a rough understanding of how dynamic scopes are composed inside, let’s look at how those are really constructed.
Constructing dynamic scope for handlers
Let’s look at the handler-bind macro invocation once again:
(handler-bind [MalformedLogEntry
(fn [e {: line}]
(match (fix-line line)
fixed-line (invoke-restart :reparse-line fixed-line)))
MalformedLogEntry
(fn [e {: line}]
(io.stderr:write (.. "malformed log entry: " line "\n"))
(invoke-restart :ignore nil))]
;; ---- 8< ----
)
Much like let, it accepts a vector of bindings - a condition object, and a function that will be called.
Similarly to let it binds sequentially, but doesn’t override bindings, but creates a set of nested scope frames to store each binding.
Then it executes the body in the context of this dynamic scope.
Let’s look at the actual implementation and break it down:
(fn handler-bind [binding-vec ...]
(let [target (gensym :target)
scope (gensym :scope)
binding-len (length binding-vec)
setup '(do)]
(assert-compile (= (% binding-len 2) 0)
"expected even number of signal/handler bindings"
binding-vec)
(for [i binding-len 1 -2]
(let [condition-object (. binding-vec (- i 1))
handler (. binding-vec i)]
(assert-compile (or (sym? handler) (function-form? handler))
"handler must be a function"
handler)
(table.insert setup `(assert (not= nil ,condition-object)
"condition object must not be nil"))
(table.insert
setup
`(tset ,scope :handlers {:parent (. ,scope :handlers)
:target ,target
:handler-type :handler-bind
:handler {,condition-object
,handler}}))))
`(let [,target {}
{:pack pack#} (require ,utils)
{:pcall-handler-bind pcall-handler-bind#} (require ,condition-system)
,scope ,(current-scope)
orig-handlers# (. ,scope :handlers)]
,setup
(pcall-handler-bind#
#(pack# (do ,...)) ,scope ,target orig-handlers#))))
This is the entirety of handler-bind macro, minus the docstring and some comments, and while the actual body that gets spliced into the program’s source code is not very big, the setup looks somewhat complicated.
So let’s break it down.
(fn handler-bind [binding-vec ...]
(let [target (gensym :target)
scope (gensym :scope)
binding-len (length binding-vec)
setup '(do)]
(assert-compile (= (% binding-len 2) 0)
"expected even number of signal/handler bindings"
binding-vec)
;; ---- 8< ----
))
First, we create some unique names that we will use both at compile-time, and then at runtime.
You might remember the :target key in tables, that represented the dynamic scope.
This key is holding this target symbol, which we define at run time, but we need a way of putting it into the dynamic scope at compile-time, that’s why it’s defined here.
I’ll explain why it’s an important key soon.
Next up is the scope, this is an instance of dynamic scope we’re going to construct at compile time.
binding-len is just to avoid small pieces of code duplication, and the last one - setup is the code that will fill the missing bits of our dynamic scope at runtime.
After let, we’re doing a check, that the binding-vec, a vector that holds condition objects and their handlers, has an even amount of values.
We can’t properly construct the dynamic scope if a handler is missing, so it’s a compile-time error.
Now for the main part - the for loop, which creates a set of instructions that will construct the dynamic scope:
;; ---- 8< ----
(for [i binding-len 1 -2]
(let [condition-object (. binding-vec (- i 1))
handler (. binding-vec i)]
(assert-compile (or (sym? handler) (function-form? handler))
"handler must be a function"
handler)
(table.insert setup `(assert (not= nil ,condition-object)
"condition object must not be nil"))
(table.insert
setup
`(tset ,scope :handlers {:parent (. ,scope :handlers)
:target ,target
:handler-type :handler-bind
:handler {,condition-object
,handler}}))))
;; ---- 8< ----
It does several things - first, it checks that the handler is either a symbol or a function form, i.e. a list that will evaluate a function definition.
Next it inserts the runtime assertion, that verifies that the condition-object is not nil.
And finally, the tset call is inserted, that will populate current scope’s :handlers table with a new table, that refers to the original table via the :parent key.
You’ve might notice that the loop actually goes backward. This is done to preserve the correct order when constructing nested tables. As the dynamic scope is constructed as a set of nested tables, the order of construction has to be reversed, so the travel order gets preserved. But this code doesn’t actually construct the scope, it just describes how to construct it. The last portion of the macro is what gets spliced into the user’s code, so let’s look at it:
;; ---- 8< ----
`(let [,target {}
{:pack pack#} (require ,utils)
{:pcall-handler-bind pcall-handler-bind#} (require ,condition-system)
,scope ,(current-scope)
orig-handlers# (. ,scope :handlers)]
,setup
(pcall-handler-bind#
#(pack# (do ,...)) ,scope ,target orig-handlers#))
We’re finally setting some of those symbols, we’ve defined before!
The ,target {} thing, is a way to make sure we get a unique object address that we can later compare, to determine if we’ve reached the correct destination when “unwinding”.
Next, we import some helper functions, like pack# and pcall-handler-bind# - the first one is a portable table.pack implementation, and the last one is a runtime part of handler-bind.
After that, we’re calling something like current-scope.
It is a commonly used part of all macros, so I’ve put it into another macro.
In short, it just returns the current thread’s dynamic scope or creates it if it doesn’t exist yet.
And right after that, you can see that we’re caching original handlers right before ,setup.
This ,setup will splice in all that setup we’ve been constructing at compile, that will populate the current scope with new tables.
And then we’re calling pcall-handler-bind# function with all necessary info.
So this code from the example:
(handler-bind [MalformedLogEntry
(fn [e {: line}]
(match (fix-line line)
fixed-line (invoke-restart :reparse-line fixed-line)))
MalformedLogEntry
(fn [e {: line}]
(io.stderr:write (.. "malformed log entry: " line "\n"))
(invoke-restart :ignore nil))]
(icollect [line (f:lines)]
(parse-line line)))
uses handler-bind macro we’ve just examined.
Let’s look at what this code expands to:
(let [target_1_ {}
{:pack pack_10_auto} (require "fennel-conditions.impl.utils")
{:pcall-handler-bind pcall-handler-bind_11_auto} (require "fennel-conditions.impl.condition-system")
scope_2_ (let [cs_3_auto (require "fennel-conditions.impl.condition-system")
thread_4_auto (or (and coroutine
coroutine.running
(tostring (coroutine.running)))
"main")]
(match (. cs_3_auto "dynamic-scope" thread_4_auto)
scope_5_auto scope_5_auto
_ (let [scope_5_auto {:handlers {} :restarts {}}]
(tset cs_3_auto "dynamic-scope" thread_4_auto scope_5_auto)
scope_5_auto)))
orig-handlers_12_auto (. scope_2_ "handlers")]
(do (assert (not= nil MalformedLogEntry) "condition object must not be nil")
(tset scope_2_ "handlers"
{:handler {MalformedLogEntry
(fn [e {:line line}]
(io.stderr:write (.. "malformed log entry: " line "\n"))
(invoke-restart "ignore" nil))}
:handler-type "handler-bind"
:parent (. scope_2_ "handlers")
:target target_1_})
(assert (not= nil MalformedLogEntry) "condition object must not be nil")
(tset scope_2_ "handlers"
{:handler {MalformedLogEntry
(fn [e {:line line}]
(match (fix-line line)
fixed-line (invoke-restart "reparse-line" fixed-line)))}
:handler-type "handler-bind"
:parent (. scope_2_ "handlers")
:target target_1_}))
(pcall-handler-bind_11_auto
(hashfn (pack_10_auto (do (icollect [line (f:lines)]
(parse-line line)))))
scope_2_ target_1_ orig-handlers_12_auto))
Quite a lot of code, and there are even more things going on when we’re calling the pcall-handler-bind function!
But you can see that it does exactly what I’ve described above - requires some API functions, creates a thread-local scope, sets it up, and wraps our code with the pack function, for later execution.
Let’s now look at what pcall-handler-bind does.
Implementing control transfer via protected calls
The pcall-handler-bind function is defined in the library’s private API:
(fn pcall-handler-bind [f scope target orig-handlers]
"Call `f` in given `scope` and pass result up to `target`.
Restore dynamic scope handlers to `orig-handlers`."
(let [(ok res) (pcall f)]
(doto scope
(tset :current-context nil)
(tset :handlers orig-handlers))
(if ok (unpack res)
(match res
{:state :handled : target}
(raise res.type res.condition-object)
{:state :error :message msg}
(if (or scope.handlers.parent
scope.restarts.parent)
(error res)
(error msg 2))
_ (error res)))))
Looks pretty simple, but this is because we delegate most of the work to other functions.
It starts with calling f in a protected call and immediately removes current-context from the scope, and sets handlers to the original handlers, which we’ve stored back at the macro side.
Then, if the function exited successfully, we just return the results by unpacking those (for keeping nils in multiple value returns).
But when the function didn’t succeed, we expect a set of certain things as a return.
For those unfamiliar with match - it’s a pattern-matching macro from Fennel standard library.
It accepts an expression, and a pair of patterns and their respective bodies to run, if the result of an expression matches this pattern.
The first pattern here is {:state :handled :target target}, meaning that the function f returned a table with a key :state holding a :handled value, and a non-nil :target key that is equal to the value of target, passed as function argument.
The table returned in this case usually looks like this:
{:state :handled
:target {}
:type :error
:condition-object #<condition MalformedLogEntry: 0x564caba2c6e0>}
In this case, we raise the condition object stored in the result from calling f, with a correct type, meaning that the type is :error, we raise it as an error, or if it’s a :warning, as a warning and so on.
We’ll look at raise in a moment, but for now, all you need to know is that it will throw the condition object further up.
If the pattern contained :state :error we check if we have more handlers up the dynamic scope, and if we do, we throw the result as is, so it could be handled.
If not, that means that we’re already at a top level, so we convert this to Lua error with a message.
In all other cases this is an error that we want to throw up, as we can’t do anything about it because it is either not a conditional situation or target doesn’t match the expected one.
So the basic idea here is that we’re ensuring that the function will either succeed or fail as we expect, and if it fails in an unexpected way, we just rethrow the error. Lua doesn’t have a way to rethrow the stack traces, but it’s a small price to pay here.
Let’s look at raise now:
(fn raise [condition-type condition-object]
"Raises `condition-object' as a condition of `condition-type'.
`condition-object' must not be `nil'."
(assert (not= nil condition-object)
"condition must not be nil")
(let [thread (current-thread)
thread-scope (do (when (not (. dynamic-scope thread))
(tset dynamic-scope thread {:handlers {} :restarts {}}))
(. dynamic-scope thread))]
(when (= thread-scope.handlers.handler-type :handler-bind)
(match thread-scope.current-context
scope (let [target scope.target]
(var scope scope.parent)
(while (and scope (= target scope.target))
(set scope scope.parent))
(tset thread-scope :handlers scope))))
(match condition-type
:condition (raise-condition condition-object)
:warning (raise-warning condition-object)
:error (raise-error condition-object))))
This function does something interesting with current-context.
As you saw in pcall-handler-bind, we’ve set the current-context to nil, but raise actually check it.
Why?
This is because the condition can be raided inside the handler.
In such a case, we need to unwind the stack to the point where we were in the handler.
So each invocation of the handle function sets the current-context field, and this field is cleared when we exit the handler.
This stack unwinding needs to be done in order to avoid infinite recursion, which will result in a stack overflow error.
After that is done, we branch on the condition type and raise the condition accordingly.
For the purposes of this article let’s look at conditions of error type only, by looking at the raise-error function:
(fn raise-error [condition-object]
(match (raise-condition condition-object :error)
nil (if _G.condition-system-use-debugger?
(invoke-debugger condition-object)
(error (compose-error-message condition-object) 2))))
In case the condition is not handled it’s raised as a Lua error unless the debugger feature was enabled.
The condition itself is raised with the raise-condition function:
(fn raise-condition [condition-object type*]
(match (handle condition-object (or type* :condition))
(where (or {:state :handled &as res}
{:state :restarted &as res}))
(error res 2)
_ nil))
Here’s the handle innovation, mentioned above.
The result can be either a table with :handled or :restarted states, each of which tells the condition system what to do next.
Even though the condition is handled this result gets thrown up as an error because we actually need to tell the condition system that the condition was handled, and differentiate it from a successful return from the function.
So let’s look at how conditions are handled.
Handling conditions
The handle function looks like this:
(fn handle [condition-object type* ?scope]
"Handle the `condition-object' of `type*' and optional `?scope`.
Finds the `condition-object' handler in the dynamic scope. If found,
calls the handler, and returns a table with `:state` set to
`:handled`, and `:handler` bound to an anonymous function that calls
the restart."
(let [thread (current-thread)
thread-scope (. dynamic-scope thread)]
(match (find-handler
condition-object
type*
(or ?scope thread-scope.handlers))
{: handler : scope}
(do (tset thread-scope :current-context scope)
(match scope.handler-type
:handler-case {:state :handled
:handler #(handler condition-object (unpack (get-data condition-object)))
:target scope.target
:condition-object condition-object
:type type*}
:handler-bind (do (handler condition-object (unpack (get-data condition-object)))
(handle condition-object type* scope.parent))
_ {:state :error
:message (.. "wrong handler-type: " (view _))
:condition condition-object}))
_ {:state :error
:message (.. "no handler bound for condition: "
(get-name condition-object))
:condition condition-object})))
Let’s dive into it.
After getting the current thread and the scope, we call find-handler.
For now, it’s enough to know, that it takes a condition object, and walks the dynamic scope comparing this object to handlers.
Next up, if the handler were found, we set the current-context to the handler’s scope.
This will ensure that we know where we last were before actually executing a handler, so we could unwind the stack to this point in case an error happens inside the handler.
This was a very painful thing to figure out when I worked on the implementation.
After that, we branch on the handler type.
The handler-case macro allows handling without calling invoke-restart, so we just return a table with :state set to :handled, and a handler that actually does the handling.
The handler-bind is different - it requires handling with invoke-restart, which, as you might remember throws an error to transfer control flow.
So if the handler function exited normally, we go to the next handle invocation, which will propagate the condition object to another handler up the stack.
This ensures that we can decline to handle a condition.
If no handler is found the error is returned.
A similar pattern occurs for other condition types, with the difference being that some may not return an actual error when unhandled.
So how handlers are found?
The find-handler function, which accepts a condition object and a handler type, will walk the scope comparing the given condition object to bound handlers.
However, condition objects actually support inheritance, so this gets a bit more complicated.
We need to look for a handler in the current scope, and if no handler is found, we need to get the parent from the condition object and search its handler in the same scope.
If no handler and no parent handler are to be found, we go to the next scope frame.
Additionally, conditions may not be of the condition-object type, but be an arbitrary Lua value:
(handler-case (signal (math.random 5))
(0 [] :zero)
(1 [] :one)
(2 [] :two)
(Condition [] :else))
;; an unnecessarily complicated equivalent of
(match (math.random 5)
0 :zero
1 :one
2 :two
_ :else)
Notice, that while I’m handling conditions as numbers, the catch-all condition object is specified as Condition.
Let’s talk more about condition inheritance.
Condition inheritance
As was already mentioned, condition objects can have parent condition objects.
A parent can be explicitly set by using the :parent key when defining a condition:
(define-condition MathError)
(define-condition DivideByZero :parent MathError)
(define-condition NegSquareRoot :parent MathError)
(fn sqrt [x]
(if (< x 0)
(error (make-condition NegSquareRoot x))
(math.sqrt x)))
(fn div [a b]
(if (= b 0)
(error (make-condition DivideByZero))
(/ a b)))
(fn main []
(handler-case (sqrt -1)
(MathError [_ x] (io.stderr:write "Can't take square root of " x "\n")))
(handler-case (div 2 0)
(MathError [_ x] (io.stderr:write "Can't divide by zero\n"))))
Here’s an example of handling different errors through their common parent. You can create hierarchical errors, which then can be handled with more granularity, or as a whole class of errors.
But there are a bit more rules regarding inheritance.
All conditions raised with error function will by default inherit from the Error condition.
All conditions raised with warn will inherit from a Warning condition.
Both Warning and Error are inherited from the Condition condition.
This also applies to any arbitrary data you can throw with error, warn, or signal:
(fn raise-random-condition []
(match (math.random 2)
0 (error "an error")
1 (signal "a signal")
2 (warn "a warning")))
(handler-case
(raise-random-condition)
(Error [msg] (print "caught an error" msg))
(Warning [msg] (print "caught a warning" msg))
(Condition [msg] (print "caught a condition" msg)))
Here all conditions are just Lua strings, but we still have some hierarchy.
Note that the order of handlers here is very important - placing the (Condition [msg] (print "caught a condition" msg)) as the first handler will prevent Error and Warning handlers from ever running.
It is very similar to match in that regard - placing _ as the first pattern will make all other patterns unreachable.
Another thing to point out is that Lua errors also inherit from Error condition:
(handler-case (/ 1 nil)
(Error [] 0))
This is done for seamless integration with libraries that don’t use the condition system, or if your code did something wrong by mistake.
However, the integration is not as seamless as it could be. The point where it get’s not as good as it could be is when interfacing with the interactive debugger, provided by the library.
The debugger
Speaking of which, the debugger is not as featureful as in Common Lisp, but still provides a way to recover from an error, when the current scope allows for it. Consider this:
(restart-case (cs.error 10)
(:use-value [x] x))
Executing this code will raise a condition, which is not handled, so it gets propagated to a Lua error:
runtime error: condition 10 was raised
However, if we set _G.condition-system-use-debugger? to true, we’ll get this instead:
Debugger was invoked on unhandled condition: 10
restarts (invokable by number or by name):
1: [use-value] use-value
2: [throw ] Throw condition as a Lua error
debugger>>
Now every unhandled condition will invoke the debugger, which will provide you with available recovery options.
In the case of this example, there are two restarts bound - use-value, bound by the restart-case macro, and the other one is an automatically bound throw restart, that will exit the debugger.
By typing 1 or use-value in the prompt, we’ll enter the next prompt, that asks for a value to use for a selected restart:
debugger>> 1
Provide inputs for use-value (args: [x]) (^D to cancel)
debugger:use-value>> 5
5
Thus making the result of the expression to be 5, accordingly to the restart’s body.
This debugger has no runtime penalty unless the condition was not handled. So only When an unhandled condition pops to the top of the dynamic scope stack, the debugger begins its work.
Debugger again traverses the scope, gathering all available restarts, and forms this interactive prompt, with each restart in order of their definition:
(restart-case
(restart-case
(restart-case (/ 1 nil)
(:use-value [x] x)
(:use-value [x] "use-value (impostor)" (- x)))
(:do-something [] (+ 1 2 3)))
(:do-something-else [] (- 3 2 1)))
Note that the use-value restart was specified twice.
I’ve mentioned that this is possible, but it is meaningless unless we use a debugger.
Here’s what will be prompted to the user, if this code is executed:
Debugger was invoked on unhandled condition: attempt to perform arithmetic on a nil value"
restarts (invokable by number or by name):
1: [use-value ] use-value
2: use-value (impostor)
3: [do-something ] do-something
4: [do-something-else] do-something-else
5: [throw ] Throw condition as a Lua error
debugger>>
Note that the first restart is indeed named use-value, but the second one has no name, only its docstring is displayed.
This restart can be manually called by typing 2 in the prompt - it will ask for the argument similarly to use-value restart, but will negate it.
Other restarts are self-explanatory.
Ending thoughts
This should provide some information about this library’s internals.
Seeing how many pcall calls it does to transfer control flow around, and how it manages the dynamic stack, can make you think that it has not a very good performance.
And in fact, I would agree, the performance could be better.
But it’s not that bad.
For example, here’s a 10000000 invocations of Lua’s error function inside a pcall:
(macro time [expr times]
`(let [c# os.clock
s# (c#)
r# ,expr
e# (c#)]
(print (.. "Average time elapsed " (/ (* 1000 (- e# s#)) ,(or times 1)) " ms"))
r#))
(time (for [i 1 10000000]
(pcall error 10))
10000000)
;; Average time elapsed 0.0002025262 ms
And here’s an equivalent using condition system’s error and ignore-errors macro:
(local {: error} (require :condition-system))
(time (for [i 1 10000000]
(ignore-errors (error 10)))
10000000)
;; Average time elapsed 0.0051841138 ms
There’s one order of magnitude difference in performance here, meaning that ignore-errors is ~10 times slower than pure pcall.
But the benefits, that the condition system provides, out-weight this cost, in my opinion.
Also, a note on why I didn’t port the rest of Common Lisp’s condition system.
While this library is not a full port, I think it has all the necessary parts in the context of the Fennel language.
What I mean by that is, that since Fennel, for example, deliberately chose to avoid Lua’s return special, I don’t think a condition system should provide block and return, or go specials.
I don’t have strong reasoning behind it, but I feel that the error handling part in Lua is what’s needed to be improved, thus this library is only aimed at fixing this part of the runtime.