Andrey Listopadov

GRScheme Design Part 3 - Quasiquoting

Fri, 03 Apr 2020 15:37:00 +0300

At the end of the Part 2 I mentioned that quasiquoting was problematic in GRScheme, due to some of the internal representations and algorithms, but now it is not an issue (I hope).

I’ve figured out the correct way to maintain a list invariant, splice one tree into another tree, and so on. So this post will describe how quasiquote, unquote, and unquote-splicing procedures are done in GRScheme.

Also, I should mention, that I’ve changed my mind, and now results are printed without the leading quote:

;; before
> (car '(a b c))
'a
> (car '('a 'b 'c))
''a
;; now
> (car '(a b c))
a
> (car '('a 'b 'c))
'a

I was using this format because Racket language was the first thing I saw when I touched the world of Scheme and Lisp, so it was natural to me, to see that results are quoted. Although after some interactions with Clojure, Common Lisp, and GNU Guile, I’ve changed my mind. This doesn’t affect internal representation though, only printing. But if you don’t know about it, it probably will not matter at all.

Quasiquoting

First things I’ve did were quasiquote and unquote. Although those are directly related, since we can’t use unquote outside of quasiquote, let’s first look at how plain quasiquote behaves.

In Racket, quasiquote of any quote-able form yields quoted form:

Note: code examples below are representing Racket and GRScheme REPLs, so for better distinguishing purposes, each prompt will be marked with a language name, e.g. racket> or grscheme>.

racket> `a
'a
racket> `(a)
'(a)

This is also the case for GRScheme:

grscheme> `a
a
grscheme> `(a)
(a)

All nested quasiquote calls are preserved in their original form, e.g. not changed to quote:

racket> ``a
'`a
racket> `(`a)
'(`a)

And in GRScheme:

grscheme> ``a
`a
grscheme> `(`a)
(`a)

This means that the interpreter should be aware of what quasiquote it is expanding at the moment - is it nested or not? We’re not going to explicitly check for this by analyzing expressions up the tree, but instead, we’ll add a counter to our eval function:

pub fn eval(&mut self, expression: &NodePtr) -> Result<NodePtr, EvalError> {
    let mut stack = vec![expression.clone()];
    let mut res = None;
    let mut quasiquote_level = 0;

    while let Some(expr) = stack.last() {
        res = match Self::expression_type(&expr) {
            // …
        }
    }
}

So when our quasiquote_level is zero, we’re going to interpret its result as quoted form, and as quasiquote form otherwise. I know that this is probably not the best way to manage this, but believe me, the worst is yet to come. Quasiquoting in GRScheme is really ad hoc right now, but it works.

Now we need to actually interpret quasiquote forms. Let’s look at match arms that are responsible for that:

pub fn eval(&mut self, expression: &NodePtr) -> Result<NodePtr, EvalError> {
    let mut stack = vec![expression.clone()];
    let mut res = None;
    let mut quasiquote_level = 0;
    let mut unquote_level = 0;

    while let Some(expr) = stack.last() {
        res = match Self::expression_type(&expr) {
            // …
            Type::Quasiquote if quasiquote_level == 0 || quasiquote_level == unquote_level => {
                let args = Self::rest_expressions(&expr)?;
                let unquotes = Self::pre_quasiquote(&args)?;
                if !unquotes.is_empty() {
                    stack.extend_from_slice(&unquotes);
                }
                quasiquote_level += 1;
                continue;
            }
            Type::Quasiquote => {
                let (res, _) = Self::quote(&Self::rest_expressions(&expr)?)?;
                Tree::replace_tree(&expr, res);
                quasiquote_level -= 1;
                stack.pop()
            }
            // …
        }
    }
}

Notice that I’ve added unquote_level counter, and we use both to decide what match arm to take. If our quasiquote_level is 0 or our quasiquote_level is equal to unquote_level we’re taking first branch, that extends our tree traversal stack, increments quasiquote_level and goes to next iteration. Otherwise, we quote the result, which means that we’re out of current quasiquote context.

Let’s look at the first arm in more detail. Let’s imagine that we’re evaluating `(a b ,c d) expression here. First, we take out the vector of all arguments to quasiquote, which in this case is [a, b, (unquote c), d]. Next we pass it to pre_quasiquote procedure. As was described in Part 2 all pre_something procedures are used to decide what arguments should be evaluated before we go further. Mostly what we did before, was just filtering of arguments based on their type. But that’s not really the case for pre_quasiquote, because unquote has another property that must be maintained. But first, let’s what unquote does.

Let’s look at this expression:

racket> (define c 42)
racket> `(a b ,c d)
'(a b 42 d)

We can see that ,c was replaced with the value of c. That’s what unquote does in quasiquote. One can think of quasiquoting as of inversion of evaluation of list arguments:

racket> (list 'a 'b c 'd)
'(a b 42 c)

You can see that everything was quoted except c. In quasiquote form only c was unquoted. But there’s another property for unquote and it is quasiquote nesting level:

racket> `(a '(b ,c d))
'(a '(b 42 c))

Nothing changed, c still evaluates to 42, but if we change ' to `:

racket> `(a `(b ,c d))
'(a `(b ,c d))

Nothing happened to c. This is because there were two nested quasiquote calls and only one unquote one. If we add another unquote we’ll see this:

racket> `(a `(b ,,c d))
'(a `(b ,42 d))

,c was replaced by 42, but outer , is still there, thus ,42 is the result of ,,c. Nested quasiquotes must be escaped by the same amount of nested unquotes. This is also the case for unquote-splicing.

Racket documentation features this weird expression:

racket> `(1 ```,,@,,@(list (+ 1 2)) 4)
'(1 ```,,@,3 4)

That is quite complex, but it works in GRScheme in the same way:

grscheme> (define list (lambda x x))
grscheme> `(1 ```,,@,,@(list (+ 1 2)) 4)
(1 ```,,@,3 4)

With this information, we can now look at how pre_quasiquote is implemented.

Filtering arguments for `quasiquote`

The pre_quasiquote procedure is quite big, so we’ll split it into parts as was done in previous posts.

The first thing we do is check for argument count, and create two stacks, one for tree traversal, and another for unquote forms that are valid to evaluate:

fn pre_quasiquote(args: &[NodePtr]) -> Result<Vec<NodePtr>, EvalError> {
    Self::check_argument_count("quasiquote", ArgAmount::NotEqual(1), args)?;
    let mut stack = vec![args[0].clone()];
    let mut unquotes = vec![];

    // …

    Ok(unquotes)
}

In the end, we will return a vector of unquote forms, which may be empty, if no valid forms were found, either because of the fact that levels weren’t balanced, or because there simply were no unquote calls.

Next, we iterate over our stack, as in eval procedure, and check for expression type on each iteration:

while let Some(expr) = stack.pop() {
    match Self::expression_type(&expr) {
        // …
    }
}

Depending on what type we’ve got we going to, either put it to the stack, put it to quotes, or continue. The first type, which is the simplest case, that we need to check is Procedure:

Type::Procedure => stack.extend_from_slice(&expr.borrow().siblings),

If we see (something …) we expand our stack with all inside expressions, and continue.

Next arm checks for Quasiquote, Symbol, and List. We do one more check for arguments, because all these types take exactly one argument, and if everything is OK, we continue:

Type::Quasiquote | Type::Symbol | Type::List => {
    let args = Self::rest_expressions(&expr)?;
    Self::check_argument_count("quasiquote", ArgAmount::NotEqual(1), &args)?;
    stack.extend_from_slice(&args);
}

The second to last arm is for all kinds of unquote expressions, and that’s where everything happens. The last arm does nothing, as we don’t need to extend the stack with other types:

Type::Unquote | Type::UnquoteSplicing => {
    let args = Self::rest_expressions(&expr)?;
    Self::check_argument_count("unquote", ArgAmount::NotEqual(1), &args)?;
    let (q, u) = Self::quasiquote_unquote_levels(&expr);

    if q == 0 {
        return Err(EvalError::UnquoteNotInQquote);
    } else {
        if q == u {
            unquotes.push(expr.clone());
        }
        match Self::expression_type(&args[0]) {
            Type::Procedure | Type::Quasiquote | Type::List | Type::Symbol => {
                stack.extend_from_slice(&args[0].borrow().siblings)
            }
            Type::Unquote | Type::UnquoteSplicing if q > u => {
                stack.push(args[0].clone())
            }
            Type::Unquote | Type::UnquoteSplicing => {
                return Err(EvalError::UnquoteNotInQquote)
            }
            _ => (),
        }
    }
}
_ => (),

So what’s happening here is another check for arguments, and quasiquote_unquote_levels call. This function analyses the current expression, and all its parents to be either quasiquote or unquote and unqoute-splicing calls, and sets counters for the nest levels of those. This function returns a tuple, which we destructive binding to q and u variables, that stand for quasiquote and unquote.

If q is equal to 0 this means that we’ve got unquote outside of quasiquote, which is not allowed. Otherwise, if we got the same levels we push expression to unquotes stack. But this is not the end, we still have to analyze inner expressions, because unquote may contain other calls for unquote or quasiquote and so on, so our algorithm repeats itself one more time and pushes expressions to stack depending on their types and levels.

This function produces vector of valid unquote and unquote-splicing calls that we pass to our eval function and evaluate.

`Unquote` procedure

To perform unquoting, we have to take into account several things. But first let’s look at Unquote arms in eval procedure:

Type::Unquote if quasiquote_level > 0 => {
    let rest = Self::rest_expressions(&expr)?;
    let args_to_eval = Self::pre_unquote(&rest)?;
    if !args_to_eval.is_empty() {
        unquote_level += 1;
        stack.extend_from_slice(&args_to_eval);
        continue;
    }
    unquote_level -= 1;
    let (res, pop) = Self::unquote(&rest)?;
    Tree::replace_tree(&expr, res);
    if pop {
        stack.pop()
    } else {
        continue;
    }
}
Type::Unquote => return Err(EvalError::UnquoteNotInQquote),

Similarly to qasiquote arm, we check for quasiquote_level, but now if it is greater than 0. If it’s 0 it’s an error that we have to catch at the top level.

You can see that there’s another pre_ procedure here. That is because we’re going to evaluate unquote argument, and then actually unquote it. pre_unquote just checks for valid forms that we actually need to evaluate, similarly to other pre_ procedures, so I don’t think we need to look closely at it. We’ve seen a lot of those in Part 2, and this one is no different.

After everything is evaluated, we call Self::unquote:

fn unquote(args: &[NodePtr]) -> Result<(NodePtr, bool), EvalError> {
    Self::check_argument_count("unquote", ArgAmount::NotEqual(1), args)?;

    match Self::expression_type(&args[0]) {
        Type::List | Type::Symbol | Type::Quasiquote => {
            Ok((Self::rest_expressions(&args[0])?[0].clone(), true))
        }
        _ => Ok((args[0].clone(), false)),
    }
}

This procedure is extremely similar to quote, except it removes quote from List, Symbol, and Quasiquote forms, and returns all other forms as is. It also indicates if we should pop our tree traversal stack or not.

`Unquote-splicing` procedure

This procedure is kinda the same as unquote and actually uses unquote underneath, but the big thing is splicing part here.

Because GRScheme uses trees of vectors, which are not linked lists, I thought that maybe there should not be such a thing as splicing, but in the end, I’ve decided that it’ll be better with it.

Splicing works kinda like you would expect, except that it modifies the vector:

impl Tree<T> {
    // …
    pub fn splice_in_childs(root: &NodePtr<T>, pos: usize, tree: NodePtr<T>) -> &NodePtr<T> {
        let mut broot = root.borrow_mut();
        let pos = if pos > broot.siblings.len() {
            broot.siblings.len() - 1
        } else {
            pos
        };

        for child in tree.borrow().siblings.iter().rev() {
            broot.siblings.insert(pos, child.clone());
        }

        root
    }
    // …
}

Not very idiomatic to do such a thing with vectors, but, eh, I can live with that. This gives us the ability to splice in siblings of one tree into another one at a given place. If a place is more than the size of the current tree’s siblings, we just put those to the end. Now we can use it in UnquoteSplicing arm:

Type::UnquoteSplicing if quasiquote_level > 0 => {
    let parent = Tree::parent(&expr).unwrap();
    if !Self::splicing_context_valid(&expr) {
        return Err(EvalError::UnquoteSplicingInWrongContext)
    }
    let pos = Self::splicing_pos(&parent).unwrap();
    let rest = Self::rest_expressions(&expr)?;
    let args_to_eval = Self::pre_unquote(&rest)?;
    if !args_to_eval.is_empty() {
        unquote_level += 1;
        stack.extend_from_slice(&args_to_eval);
        continue;
    }
    unquote_level -= 1;
    let (res, pop) = Self::unquote(&rest)?;
    Tree::remove_child(&parent, pos);
    Tree::splice_in_childs(&parent, pos, res);
    if pop {
        stack.pop()
    } else {
        continue;
    }
}
Type::UnquoteSplicing => return Err(EvalError::UnquoteNotInQquote),

For the most part, it is the same as the unquote procedure, except that at the end we splice results in, and also we check that we’re inside the valid context. This check finds the quasiquote procedure and checks if it has an argument of Procedure:

`(,@'(1 2 3)) ;; valid
`,@'(1 2 3)   ;; invalid

As a little showcase of how everything works so far, let’s implement the thread first macro from Clojure ->:

grscheme> (define -> (lambda forms
            (define loop (lambda (x forms)
              (if (empty? forms)
                  x
                  (let (form (car forms))
                    (if (list? form)
                        (loop `(,(car form) ,x ,@(cdr form)) (cdr forms))
                        (loop `(,form ,x) (cdr forms)))))))
            (eval (loop (car forms) (cdr forms)))))
grscheme> ->
#procedure:->
grscheme> (-> "Thread first!\n"
              '(display))
Thread first!
grscheme> (-> ''(1 2 3)
              '(cdr)
              '(car)
              '(cons '(3 4 5)))
(2 3 4 5)

Of course, this is not a real macro, and everything happens in run time every time we call ->, but the algorithm is the same as in Clojure’s ->.

GRScheme complete?

Certainly not. When I started working on it, I knew literally nothing about Scheme, Lisp, and the basic ideas behind its implementations. I was using different Lisps to some extent, like Emacs Lisp for writing small functions for my Emacs config, and going through SICP with Guile, but up until I wrote my own evaluator I had no complete understanding of what I’m doing. Since then I’ve read some papers on Lisp design, got into Clojure, and progressed through SICP, so I know a little more on how things should work internally for the most part. And with this knowledge…

I understand that my implementation of the whole thing is bad, and I mean really bad :D

There are still no macros, internal data representation is really ugly, strings are useless, there’s no file IO, REPL is really basic, there’s no debugging, the stack doesn’t exist, and so on. I don’t think I should really turn GRScheme into complete language, at least right now.

Even if I wanted to create a real practical language, that could compete with other Schemes, Clojure, or Common Lisp, I would have to redesign everything from the bottom up, so probably it will be easier to write a completely new thing. And that will still mean that I’ll have to re-implement every feature that those languages provide, which is an insane amount of work.

Although I would say that I think Rust is a great choice as the implementation language because it helps produce memory-safe code, and probably will make the implementation of multi-threading less worrying, compared to C. Initially, I had planned on supporting execution in separate threads, but I’ve dropped this idea because the current tree traversal algorithm can’t be easily converted to use multiple threads.

This mostly concludes my experiment with the development of Scheme-like language. I hope it was at least an interesting read. Thanks!

Comment via email

GRScheme Design Part 2 - Evaluation

Sun, 22 Mar 2020 12:14:00 +0300

Good things happened since my previous post! I’ve finished rewriting the interpreter from recursive to iterative, which means that there will be no stack overflows when traversing deep trees. I’ve also simplified name binding and lookup systems added support for arbitrary-sized integers, fractions, and 64-bit floating point numbers, but most importantly figured out tail call elimination and closures. This post will describe all of these themes!

Right now I still need to figure out unquoting and quasiquoting (which will be mentioned at the end), as well as the macro system, but I think that it will not be that difficult since I’ve already done most part of the language.

I’ve already briefly described how interpretation works in GRScheme, but to refresh it, here’s a general mechanism.

Given text input, reader produces an internal data structure that represents the input:

Figure 1: Parsing input expression to GRData

After that, the data structure is passed to the evaluator:

Figure 2: Using GRData to provide result

In previous post, I’ve described what happens in the reader module, and this post is going to be about evaluator.

`Evaluator` structure

Since everything in Scheme is a list (in our case a vector, but this doesn’t matter right now), we have uniform rules for evaluation:

If something is a list:
- the first element is a procedure name,
- rest are procedure arguments.
Else if something is a non-primitive type:
- lookup in scope,
- evaluate
If something is a primitive type, evaluate it to self.

This means that things such as quoted lists, symbols, numeric constants, and strings are primitive types:

'(list of items)
'symbol
1, 3.14, 22/7
"string of text"

I’ve marked quoted lists and symbols as cursive because actually, it’s not primitive types in GRScheme, but calls to quote procedure:

(quote (list of items))
(quote symbol)

However I’ve said that primitive types are evaluated to themselves, and this is the case for quote calls - those produce quote calls, e.g. themselves. Well, most of the time. I’ll explain why a bit later.

This leads us to procedure application syntax. Everything that is wrapped with parentheses is a call to proceed with some arguments, where some may actually be none. For example, procedure car returns the first element of the list it was given:

> (car '(1 2 3)) ;; (car (quote (1 2 3)))
1

When we pass this expression to the reader it produces such a tree:

Figure 3: (car '(1 2 3)) expression as GRData

After that, evaluator calls eval function, on that expression, which iterates over the tree and checks the type of each leaf. The first leaf it finds, in this case, is the root of the tree, which is (. This means that it is a procedure call, so we check what procedure we should call. In this case, it is car. We check if this name exists in the current scope by using the lookup procedure, which in turn checks for each scope that is available for us at this moment - the scope of the caller, and the global scope. Since caller scope is empty ¹, we check global scope for name car and find out that it holds #procedure:car.

What is #procedure:car? It is a pattern - a special keyword that means something to the interpreter. Patterns are reserved (for now) and can’t be created, but can be assigned to names, which is the case for car. This pattern describes how we should treat this name, e.g. as a #procedure, which procedure it contains, car, and what to expect of its arguments - should we evaluate those, or not.

Since we need to evaluate arguments of car, our next step is the evaluation of the arguments. In our case, there’s one argument, and it is a primitive type one - a list '(1 2 3), so we evaluate the call to quote procedure. The result of calling a quote with a list is a list itself, so quote returns '(1 2 3). Now, when all arguments are evaluated, we can pass those to #procedure:car call.

car should take the first element of the list that we’ve passed to it, so in the case of car there should be only 1 parameter, and we need to check it. If there are more parameters than there should be, we will throw an error and abort the execution of the whole expression. In our case, there’s exactly one parameter of type list, which is what car expects. If the parameter has the wrong type car will signal an error and abort. If the list is empty, we will also signal an error. In our case, we successfully take the first element from the list and quote it.

So the result of (car '(1 2 3) is (quote 1), but because 1 is a primitive type and can’t be quoted, the result of (quote 1) expression is 1, which is what interpreter prints after executing our expression.

With this information, you should understand the general idea behind the evaluation process. Now let’s look at the code.

Function `eval`

Here’s the whole code of eval function at the moment of publication. In recursive form it was much shorter, however, shared the common problem of all recursion algorithms - stack overflow. Unfortunately, I could not figure out a way of making it tail-recursive to use trampolines, because we do some work after each iteration, so I’ve turned it to pure iterative form. If you’re interested in how this function looks with the recursion algorithm check here.

This function is quite big to list here in its full form so let’s tackle it piece by piece. It also works better for explanation purposes.

Let’s begin with the function signature:

pub fn eval(&mut self, expression: &NodePtr) -> Result<NodePtr, EvalError> {

This function takes two parameters - &mut self and expression: &NodePtr. NodePtr is a type that is defined in reader module as pub type NodePtr = tree::NodePtr<GRData>; and just a shorter way to manage tree leafs. The first parameter is self which means that this function is a method of an object it was called from. This object is Evaluator structure which is quite simple and has only one field:

pub struct Evaluator {
    global_scope: HashMap<String, NodePtr>,
}

Thus if we create new Evaluator and call eval method it will be able to see global_scope. We’ll get to that later.

The first thing we do in eval is stack creation, and variable to hold our end result:

let mut stack = vec![expression.clone()];
let mut res = None;

Now, when we have a place to store our result and a stack to iterate we can begin iteration itself:

while let Some(expr) = stack.last() {
    res = match Self::expression_type(&expr) {
        // …
    }
}
Ok(res.unwrap())

This is the core of our evaluator. Well, not exactly the core, but an important part of it. We pop the last element from the stack and analyze its type. Depending on that type we will perform different actions. There are 9 types right now:

#[derive(Debug, Clone, PartialEq, PartialOrd)]
enum Type {
    Integer,
    Rational,
    Float,
    Name,
    Str,
    Symbol,
    List,
    Unquote,
    Quasiquote,
    Procedure,
    Pattern,
}

Code Snippet 1: Types of expressions needed for evaluation

We have types for different numbers, including Integer for all integers, such as 1, 123, -123, 2352975723524562345, e.t.c., Rational for fractions: 1/2, 22/7, 235297594562345/2359524562347, and Float for 64 bit floating numbers. Next, we have Name, names are everything that is simply a sequence of letters and numbers without white space and reserved characters, e.g. car, quote, hello1234. And Str is for strings - arbitrary sequences of anything wrapped within pair of ".

The next ones are more interesting. We have Symbol. Symbols are quoted names and are primitive types. For example car is a name, and 'car is a symbol. List is another primitive type, which we have already seen before: '(1 2 3). A list can contain anything inside, including other lists.

And the big ones - Procedure and Pattern. These types are special because the first one determines what we will do with our list, and the second one is how we will do it. Rules are simple - everything that starts with ( is a Procedure, and everything that starts with # is a pattern. Now there’s one problem. We have lists and symbols, which are represented as '(a) and 'a but in fact are stored as (quote (a)) and (quote a). This means that these are procedure calls. So how does it work? Let’s look at the function that determines the type of the expression:

fn expression_type(s: &NodePtr) -> Type {
    let data = &s.borrow().data.data;
    match data {
        Data::String { data } => {
            if data.starts_with('#') {
                Type::Pattern
            } else if !s.borrow().siblings.is_empty() {
                match &s.borrow().siblings[0].borrow().data.data {
                    Data::String { data }
                    if data == "quote" && s.borrow().siblings.len() > 1 =>
                    {
                        match &s.borrow().siblings[1].borrow().data.data {
                            Data::String { data } if data == "(" => Type::List,
                            _ => Type::Symbol,
                        }
                    }
                    Data::String { data } if data == "unquote" => Type::Unquote,
                    Data::String { data } if data == "quasiquote" => Type::Quasiquote,
                    _ => Type::Procedure,
                }
            } else if data.starts_with('"') && data.ends_with('"') {
                Type::Str
            } else {
                Type::Name
            }
        }
        Data::Integer { .. } => Type::Integer,
        Data::Float { .. } => Type::Float,
        Data::Rational { .. } => Type::Rational,
    }
}

Code Snippet 2: Expression type analyzer

It’s quite simple, actually. We pass our tree to it and it analyses the top level. If our root has String data, there’s a possibility of it containing anything but numerical data. So we check if the data starts with #, and if that’s the case, we got Pattern. If not, we check if it has sub-nodes, which actually means that this is a procedure call. If the procedure we’re calling is quote and there are other sub-nodes around, we check if those have either ( or whatever, thus producing List and Symbol. If there are no extra sub-nodes, or the procedure is not quote that’s Procedure. Same think for Unquote and Quasiquote. If there are no sub-nodes at all, that’s either a Str, if it starts and ends with ", or a Name. In case there are numerical data, we take its type.

So how do we use this information? In our while let loop over the stack we get the expression type and match it against it. But we do not actually need to match against all of those:

res = match Self::expression_type(&expr) {
    Type::Procedure => { /* …  */ }
    Type::List | Type::Symbol => { /* …  */}
    Type::Quasiquote => { /* …  */ }
    Type::Unquote => { /* …  */ }
    Type::Name => { /* …  */ }
    _ => stack.pop(),
};

Code Snippet 3: Evaluation branching based on the type of current expression

We obviously need to evaluate Procedure, but because List and Symbol are a bit more fundamental (as building blocks) we evaluate those separately. Unquote and Quasiquote are implemented as hack, so I won’t really discuss it here ². Names are also evaluated in a special way, and the rest types are evaluated to themselves so we do not need to do anything with those, so we simply pop those from the stack. We do this at the end of each match arm, except for the first one. Let’s go from the bottom.

`Name` evaluation

That’s how we evaluate names. It’s simple but there’s a lot of what actually happens:

Type::Name => {
    Tree::replace_tree(&expr, self.lookup(&expr)?);
    stack.pop()
}

Code Snippet 4: Branch that evaluates Name expressions

First, we’re calling self.lookup procedure, that looks through all available scopes for the value of the name. There’s this ? operator, which you may have seen before in some code above. It is syntactic sugar for passing the correct value to the caller and the error value out of the caller. Because lookup returns Result type and it is a compound error type we need to pass an error to the handler which is not this function. We could write this call in a more explicit way:

match self.lookup(&expr) {
    Ok(result) => result,
    Err(error) => return Err(error),
}

Code Snippet 5: ? operator does exactly this

But it is tedious to do, so ? is great. Let’s not look at lookup for now, until we talk about scopes. What you need to know is that lookup will check every scope current expression car reach and the global scope. If value can not be found it will return Err(EvalError::UnboundIdentifier { name }) which will be passed to error handler. This error can’t be handled so we will see the message in stderr saying that name is not bound to anything.

If the value of a name was found we pass it to our next function Tree::replace_tree. The value of a Name is always some tree, so we need to replace our current expression with a new tree. Now we have finally touched the reduction process of the tree.

Our tree consists of reference-counted pointers to sub-trees. What this means is that if we have a pointer to the tree if we have a pointer and we’re not violating any of the borrowing rules we can change the value of this pointer. In the case of our eval procedure, we have the stack of such pointers, and when we take out each pointer we’re basically fine to do anything with it. So when we’ve found the value of the Name - new tree we can replace the old tree, holding that Name with our new tree. Let’s look on a simple example of (+ a 1) given that a holds 41:

Figure 4: (+ a 1) expression and global scope holding a

So we have this box labeled Global Scope. Scopes use HashMap to organize data and optimize search by hashing keys, but here we will use names directly. When the evaluator reaches a leaf, it calls lookup, which checks if the global scope has key a in the table, and gets out the copy of the tree that holds 42:

Figure 5: Process of looking up a and gathering the value by copying

Now that we got 42 we can replace a with it:

Figure 6: Replacing a with 41 and dropping pointer

We then drop our a from the expression, and replace it with 42, while dropping the pointer which originally held 42. The resulting expression is:

Figure 7: Expression with a evaluated to 41

This is a general idea behind tree manipulation. Tree::replace_tree keeps the original pointer, but drops all of the node’s sub-nodes changes its value to the value of another node, and pushes new sub-nodes from another node. Then the pointer to another node is dropped.

So for example if a would have held this list: '(1 2 3) we would get such expression:

Figure 8: Example of gathering a list from the global scope

Of course, it would not work, because you can’t add 1 to '(1 2 3), but this should illustrate the idea of what’s happening with the tree.

Now we can go to a bit more complex List and Symbol evaluation.

`List` and `Symbol` evaluation

On the last image we’ve seen list expression '(1 2 3) and how it’s represented in GRScheme Data:

Figure 9: '(1 2 3) list representation in GRData

This is done in such a way because we always have a full tree of our program at any time of evaluation. Therefore if we would apply the quote as the real Scheme does, we would get (1 2 3), store it in the tree like that, and when we will try to do something with that list, we would assume that it is a procedure. Which is not. So the quote is stored within the list permanently. The same goes for Symbols, those are stored exactly like lists, except the second subtree is just a Name, e.g. 'a:

Figure 10: 'a symbol in GRData

That’s how we evaluate Lists and Symbols:

Type::List | Type::Symbol => {
    let (res, change) = Self::quote(&Self::rest_expressions(&expr)?)?;
    if change {
        Tree::replace_tree(&expr, res);
    }
    stack.pop()
}

Code Snippet 6: Part of eval that responsible to List and Symbol evaluation

It is similar to the evaluation of the Name but a bit different. First, instead of calling lookup we call Self::quote over the result of Self::rest_expressions. What rest_expressions does, is:

takes the reference to expression,
checks whether it has sub-expressions,
- if there are sub expressions it checks if it has more than one of those,
- if There’s more than one, it creates a vector with pointers to those expressions, except the first one, and returns it,
- otherwise, it returns an empty vector,
if no subexpressions were found it signals an error.

So quote receives a reference to that vector of subexpressions. This is what quote does:

fn quote(args: &[NodePtr]) -> Result<(NodePtr, bool), EvalError> {
    Self::check_argument_count("quote", ArgAmount::NotEqual(1), args)?;

    let res = args[0].clone();
    match Self::expression_type(&res) {
        Type::Procedure | Type::Name | Type::Unquote | Type::Quasiquote => {
            let root = Tree::new(GRData::from_str("("));
            Tree::push_child(&root, GRData::from_str("#procedure:quote"));
            Tree::push_tree(&root, res);
            Ok((root, false))
        }
        _ => Ok((res, true)),
    }
}

Code Snippet 7: Code of the quote procedure

This function returns a compound type of Result<T, E>, where T is the success type, and E is the error type. In our case success type itself is compound, and it is a tuple that consists of a pointer to the subexpression and a Boolean. First, we check if there’s a correct amount of arguments passed, which in the case of quote is 1. Then we check its type. If it is Procedure, Name, Unquote, or Quasiquote we wrap those into quote and return with field argument set to false. If it is anything else, means that it cannot be quoted, so we return what was passed to quote with the second field set to true.

This second Boolean will indicate whether should we use replace_tree over our result. Because replacing a tree is somewhat expensive for long lists we should avoid it when possible. Why do we do extra work if we don’t use it, though? We’re using this function for quasiquoting without checking its second field. But later about that.

After we’ve replaced our expression with the result we pop it out of the stack. If it was the last expression we will use it as a result. If not it will be used by another expression up the tree. That pretty much concludes the evaluation process of these forms. Now we can look at the evaluation of Procedure.

Evaluation of a `Procedure`

This is the most complex part of the evaluator because it is the main part. I will split this into a lot of small parts so you can follow up and not get lost in all the stuff that happens because it spreads across the whole interpreter.

First things first, let’s remember how procedures are constructed. It’s a list, which has at least one item inside it. The first item is going to be the name of the procedure, and the rest, if any, will be parameters. Thus (+ 1 2) is a call to the procedure + with arguments of 1 and 2. That’s not always the case, though. The first argument can be another procedure call, e.g. ((get-plus) 1 2), where get-plus is a procedure that returns a + procedure, so we need to check for this type of thing. That’s what we do when we’ve reached Type::Procedure arm in our eval:

Type::Procedure => {
    let expr = expr.clone();
    let proc = Self::first_expression(&expr)?;
    let proc = match Self::expression_type(&proc) {
        Type::Procedure | Type::Name => {
            stack.push(proc);
            continue;
        }
        _ => proc,
    };
    // …
}

Code Snippet 8: Part of eval function responsible to Procedure evaluation

First, we create a copy of a pointer to the current expression due to some ownership rules. Then we use Self::first_expression on a reference to this expression to get the first sub-expression which will either be a procedure, name, or something else.

Next, we check its type, and if it is either a Procedure or Name we push it to the stack and go to the next iteration. This will evaluate Name and return what it holds to us, so we can use it, and in the case of Procedure, we will return to this arm and repeat the first steps.

What we are looking for is Pattern. And a very specific one. All functions are described through #procedure:name pattern. For example in this expression (+ 1 2), + actually holds a pattern #procedure:+, and + itself is a name, so we push it to the stack, evaluate it to #procedure:+ and return to our expression which now holds (#procedure:+ 1 1). Since it is not Procedure or Name we set our proc to it.

Patterns are used to describe what we should do with some particular thing, and also to describe what this particular thing should do with its arguments. So patterns is a special data type, that holds extra information about the environment it expects around it. Each pattern follows some kind of behavior pattern thus the name.

So once we’ve got our procedure pattern, we can get its arguments by calling Self::rest_expressions on our pointer to the expression:

let args = Self::rest_expressions(&expr)?;

This creates a vector of arguments and we’re not going to do much with it right now. Our next step is the name of the procedure, so we request it:

let proc_name = proc.borrow().data.data.to_string();

Now once we have it, we check if it is more than 11 chars long, and substitute the first 11 chars of it. If it’s not 11 chars long, we’ve probably got the wrong pattern, but we’ll catch that later:

if proc_name.len() > 11 {
    let name = &proc_name[11..];
    let user_defined = proc.borrow().data.user_defined_procedure;
    let args_to_eval: Vec<NodePtr> = match name {
        "if" if !user_defined => Self::pre_if(&args)?,
        "let" if !user_defined => Self::pre_let(&args)?,
        "define" if !user_defined => Self::pre_define(&args)?,
        "progn" if !user_defined => Self::pre_progn(&args)?,
        &_ if user_defined
            || !BUILTINS_NOEVAL.contains(&name)
            || Self::expression_type(&proc) == Type::List =>
        {
            Self::pre_eval(&args)
        }
        &_ => Vec::with_capacity(0),
    };
    if !args_to_eval.is_empty() {
        stack.extend_from_slice(&args_to_eval);
        continue;
    }
}

Code Snippet 9: Checking what arguments we have to evaluate before continuing

Now we have the name, in our case, it’s + again because we’ve stripped the #procedure: part of it away. We also get the information from our node if this procedure is user_defined or not, but about that later. Now the interesting part happens.

You can see that match with arms, such as "if", "let", "define", progn all calling these pre_ procedures. These are special forms. We check if those are user-defined because if those are, they are not special anymore and we’ll call those regular functions. But if they are not defined by the user, we call these pre_ functions and pass a reference to our array of arguments to those. Let’s look at what happens in pre_if:

fn pre_if(args: &[NodePtr]) -> Result<Vec<NodePtr>, EvalError> {
    Self::check_argument_count("if", ArgAmount::LessThan(2), args)?;

    match Self::expression_type(&args[0]) {
        Type::Name | Type::Procedure | Type::Quasiquote | Type::Unquote => {
            Ok(vec![args[0].clone()])
        }
        _ => Ok(vec![]),
    }
}

This function checks if our amount of arguments for if is valid, and then checks the type of the first one. If it is either Name, Procedure, Quasiquote, or Unquote it wraps it to the vector and returns, otherwise it returns an empty vector.

The result of that function will be put into args_to_eval variable, which we later check for emptiness, and if it’s not empty we push everything from it to the stack and go to the next iteration. So in the case of if we should only push the first argument, which is the condition, and evaluate it. But we will decide later what branch to take, based on this condition. So to illustrate this:

stack	current expression
EMPTY	`(if (eq 1 2) "what" "ok")`
`(if (eq 1 2) "what" "ok")`	`if`
`(if (eq 1 2) "what" "ok"), if`	NAME EVALUATION
`(#procedure:if (eq 1 2) "what" "ok")`	`(#procedure:if (eq 1 2) "what" "ok")`
`(#procedure:if (eq 1 2) "what" "ok"), (eq 1 2)`	CONTINUE
`(#procedure:if (eq 1 2) "what" "ok")`	`(eq 1 2)`
`(#procedure:if (eq 1 2) "what" "ok"), eq`	NAME EVALUATION
`(#procedure:if (#procedure:eq 1 2) "what" "ok")`	`(#procedure:eq 1 2)`
`(#procedure:if #f "what" "ok")`	CONTINUE

This is how our expression transforms to this point. These CONTINUE, and NAME EVALUATION are used to omit some things that would make this table somewhat longer. Now we have this expression, and we again call pre_if on its arguments, but the first argument no longer belongs to Name, Procedure, Quasiquote, or Unquote, so we return the empty vector, and thus go further.

stack	current expression
EMPTY	`(#procedure:if #f "what" "ok")`
`"ok"`	CONTINUE
EMPTY	END

When the stack is empty, means that we’ve traversed and reduced our tree fully, and the last popped item is the result, which in this case is "ok".

This happens with all pre_ procedures, those filter arguments are based on what special form we have as our procedure. E.g. for define we only need to evaluate the second one. For let we only need to evaluate each second one, because arguments are going in pairs. pre_progn is a bit different in a way that it reverses our arguments, because the order in progn is a thing that we must follow, and because we put expressions to the stack, we have to reverse those to keep the original order:

fn pre_progn(args: &[NodePtr]) -> Result<Vec<NodePtr>, EvalError> {
    Self::check_argument_count("progn", ArgAmount::LessThan(1), args)?;
    Ok(args
       .iter()
       .rev()
       .cloned()
       .filter(|arg| match Self::expression_type(arg) {
           Type::Name | Type::Procedure | Type::Quasiquote | Type::Unquote => true,
           _ => false,
       })
       .collect())
}

(progn a b c) without reverse would form this stack: [progn, a, b, c] which means that c will evaluate before a, but if c depends on a , we’ll get error. So we need to reverse arguments before we put those in the stack, thus getting this stack: [progn, c, b, a]. This way we will evaluate a, b, c, and then came back to progn, which will return it’s last expression c.

If we got neither of the special forms, and our procedure is not in the list of procedures that do not evaluate arguments at all, we call pre_eval which is identical to pre_progn but doesn’t reverse argument order.

So once we’ve dealt with our arguments it’s time to apply the procedure to those. For now, I will omit some things until we really need those, so in we call apply our proc to args:

Tree::replace_tree(&expr, self.apply(&proc, &args)?);
continue;

Code Snippet 10: Simplified procedure application call

`Apply` procedure to arguments

Now we’ve got to procedure application, and yet again, the first thing we do is check proc type:

fn apply(&mut self, proc: &NodePtr, args: &[NodePtr]) -> Result<NodePtr, EvalError> {
    match Self::expression_type(&proc) {
        // …
    }
}

As you can remember, I’ve said that if we’ve got the wrong pattern, then we will probably catch it. Here’s the place where this happens. And the first type that is correct for us is… List?

Type::List => Self::list_get(&proc, &args),

Wait, what?

Yes, lists are functions. Because the underlying data structure is a vector, and not a linked list, we actually can get elements from our list pretty fast! So if we call list as a function and provide an argument of a certain number, we’ll get an element located at that index, pretty much as if we were accessing an array in Rust with []:

> ('(1 2 3) 0)
1
> ('(a b c d) 1)
'b

Furthermore, if we provide two arguments, we can get sub list of a list, without using car, cdr, cons and recursion at all:

> (define names '("Adam" "Josh" "Peter" "Steven" "Genry"))
> (names 1 3)
'("Josh" "Peter" "Steven")

If we try to access an element beyond the boundaries of a list we’ll get an error:

> ('(1 2 3 4 5 6) 6) ; lists are indexed from 0
;; evaluation error: index 6 is out of bounds of a list of len 6

Because of that length is also written in terms of accessing internal data representation of a vector and checking its length.

Our next type is indeed a Pattern. I’ve mentioned that we’re looking for a #procedure: type pattern so our mathc arm has a check for it:

Type::Pattern
    if proc
    .borrow()
    .data
    .data
    .to_string()
    .starts_with("#procedure:") => { /* …  */ }
_ => Err(EvalError::WrongApply {
    expr: Self::tree_to_string(&proc),
}),

And if the pattern is not starting with a #procedure: we drop to the _ (any) pattern, which is an error.

So what’s happening in case we’ve got the correct pattern? Let’s see:

let name = proc.borrow().data.data.to_string()[11..].to_owned();
match name.as_ref() {
    "newline" => Self::newline(&args),
    "read" => Self::read(&args),
    "define" => self.define(&args),
    "lambda" => Self::lambda(&args),
    "if" => Self::if_proc(&args),
    "cond" => Self::cond(&args),
    "progn" => Self::progn(&args),
    "let" => Self::let_proc(&args),
    "car" => Self::car(&args),
    "cdr" => Self::cdr(&args),
    "cons" => Self::cons(&args),
    "display" => Self::display(&args),
    "eval" => Self::eval_proc(&args),
    "empty?" => Self::is_empty(&args),
    "length" => Self::length(&args),
    "+" | "-" | "*" | "/" | "%" | "remainder" => Self::math(&name, &args),
    "<" | ">" | "<=" | ">=" | "=" => Self::compare(&name, &args),
    _ => Err(EvalError::UnknownProc { name }),
}

Ah, another match, that matches the name of the procedure. These are built-in procedures, that are known to our interpreter and implemented internally. If we got progn it calls Self::progn and passes args to it. Then progn returns an expression. The same goes for every procedure here. Each of those returns a compound type of Result<NodePtr, EvalError>, which either signals about the error, or has a resulting expression. And if we got a procedure that is not on this list, we get an unknown procedure error. I will not go into details of each built-in procedure, but generally speaking, each of those takes an array of arguments, checks those to be the correct amount and type, and does something with those.

Now you’ve might think where are user-defined procedures handled? If this apply function supports only built-in procedures, what’s the point of it? That’s why I said earlier, at the end of the [Evaluation of the Procedure ](#code-snippet–simplified procedure application) that we will skip some code, regarding user-defined procedures, because user-defined procedures work kinda differently. And you may have noticed that the list of built-ins above contains that one procedure, that we can use for defining procedures - lambda. Let’s look at it!

Builtin `lambda` procedure

So what lambda does? How does it work? To understand this, we need to look at the syntax for the lambda:

(lambda (x)
  (* x x x))

Here’s a lambda that takes one argument x and multiplies it by itself by calling * procedure over three arguments of that x. This already gives us a lot of information. For example, not only does lambda create a procedure, but it also defines a binding for its argument, so it could then be used inside that lambda by other procedures, such as * in our case.

So how do we apply this procedure? We wrap with parentheses:

((lambda (x)
   (* x x x)) 2)

Here’s we apply our lambda to argument of 2, which binds x to be 2, so when * is evaluated we lookup each x, evaluate it to 2 and then apply * to the list of arguments [2, 2, 2] thus producing the result of 8.

Lambda also can take a variadic amount of arguments, and it is achieved by providing a single argument but without any parentheses:

(lambda x x)

This lambda will receive any amount of arguments and return those as a list:

> ((lambda x x) 1 2 'a "5")
'(1 2 a "5")

Let’s look at a bit more complex lambda expression:

(lambda (x)
  ((lambda (y) (+ x y)) 5))

This example creates a lambda that takes one argument x and in its body, it creates another lambda, that takes one argument y, and sums x and y in the body, and applies that lambda to the argument of 5, thus binding y to 5. So whenever we call original lambda and apply it to some number it will add 5 to that number. Not a very practical procedure, but this gives us the idea, that, perhaps, instead of evaluating our second lambda we could simply return it:

(lambda (x)
  (lambda (y) (+ x y)))

So, that’s a lambda that takes one argument x and returns a lambda that takes one argument y that sums up x and y. Wait… If we evaluate this procedure and apply it to some number like 5, we should receive (lambda (y) (+ x y)) as a result, and there’s no x in that definition. This means that lambda can grab bindings from the outside world and store those within itself!

This is called closure. And what it allows us here is to store outside world bindings within our lambda and reach those even if the original lambda no longer exists:

> (((lambda (x)
      (lambda (y)
        (+ x y))) 5) 6)
11
> (((lambda (x)
      (lambda (y)
        (+ x y))) 5) 10)
15

However, that’s not really good way to use such procedures. What we can do instead is bind the lambda to a name, thus creating a user-defined procedure:

(define gen-adder (lambda (x)
                    (lambda (y) (+ x y))))

This gives us gen-adder procedure (gen stands for generate). Once we call it and pass a parameter, it will return the procedure that adds this parameter to some other parameter:

> (gen-adder 5)
#procedure:anonymous
> ((gen-adder 5) 5)
10

This in turn means that we can bind this resulting procedure to the name:

> (define add2 (gen-adder 2))
> (add2 40)
42

OK. Now when we’ve hopefully worked this out let’s look at the code of the built-in lambda procedure. In parts, as usual:

fn lambda(args: &[NodePtr]) -> Result<NodePtr, EvalError> {
    Self::check_argument_count("lambda", ArgAmount::LessThan(2), args)?;
    // …
}

As I’ve already said, we check the number of arguments, and if there’s less than 2 that’s not a valid lambda call. E.g. (lambda (x)) is not a valid lambda definition.

let arg_list = args[0].clone();
let body = args[1..].to_vec();

Next, we take a pointer to the first argument of the lambda which will either hold the list of arguments to the function we will produce or one argument that will be represented as a list. And we take all pointers for body expressions that were passed to the lambda. This makes it possible to create procedures with any amount of expressions in the body: (lambda (x) (display x) (newline) (+ x 1)), this lambda takes one argument x and displays it, prints a newline then returns the result of x + 1.

Next, we check the type of our first argument. It can only be a Name or a Procedure³, thus x or (x). Everything else is not valid type:

match Self::expression_type(&arg_list) {
    Type::Procedure | Type::Name => (),
    _ => {
        return Err(EvalError::GeneralError {
            message: "lambda: wrong argument type".to_owned(),
        })
    }
}

If we got the correct argument we can create a new tree that will hold our lambda. We create a new node that will be our root, and it will be holding a Pattern of #procedure:anonymous created with GRData::from_proc_str function, and w immediately push our argument list to it as the first sub node:

let res = Tree::new(GRData::from_proc_str("#procedure:anonymous"));
Tree::push_tree(&res, arg_list);

Now the body. Because of the nature of the language, e.g. everything is stored in the tree, we actually need some root node for our body, and we also need to return only the result of the last expression, so progn looks like the perfect candidate:

let progn = Tree::new(GRData::from_str("("));
Tree::push_child(&progn, GRData::from_str("#procedure:progn"));

We will then put each sub-expression from the body to that progn, then push the progn itself to our #procedure:anonymous as its second argument, and return it as the result:

for child in body.iter() {
    Tree::push_tree(&progn, child.clone());
}

Tree::push_tree(&res, progn);

Ok(res)

So our tree for an anonymous function that takes one argument, prints it, prints a newline, and returns the square of it should look like so:

But wait! There’s more. I’ve talked about closures and bindings before, but where do actually those bindings go? Where do we store those? How lookup manage to find out these bindings? Let’s look at this missing part of our lambda function:

let mut current = Tree::parent(&body[0]);
while let Some(p) = current {
    if !p.borrow().data.scope.is_empty() {
        for (key, val) in p.borrow().data.scope.iter() {
            progn
                .borrow_mut()
                .data
                .scope
                .insert(key.clone(), val.clone());
        }
        break;
    }
    current = Tree::parent(&p);
}

So what this piece of code does is the creation of closure for our lambda. It goes to the expression parent, and if it has some bindings, it copies those into the current scope. Into the progn. Into it’s (.

So all bindings are being stored in parentheses. And whenever we go to the parent we immediately can see if it has bindings associated with it. This piece goes to the first parent, checks, and if there is something, then we’re interested in it, and only it. There’s no point to go all the way up.

This closure is created upon lambda creation, but that’s another closure that is being created upon lambda application, as well as some other things that happen in there, such as tail call elimination. So let’s look at how we apply our lambda functions

Application of `lambda` procedure

Let’s first look at the apply_lambda function signature:

fn apply_lambda(
    lambda: &NodePtr,
    args: &[NodePtr],
    tail_call: bool,
) -> Result<(bool, NodePtr), EvalError> {
    // …
}

Unlike apply, this function takes one extra parameter and returns an extra Boolean in the success variant.

To understand what tail_call does we have to go back to eval function, and see a proper invocation of apply and apply_lambda:

if proc.borrow().data.user_defined_procedure {
    if let Some(p) = Tree::parent(&expr) {
        if p.borrow().siblings.last().unwrap() == &expr
            && p.borrow()
            .siblings
            .first()
            .unwrap()
            .borrow()
            .data
            .data
            .to_string()
            == "#procedure:progn"
        {
            // possible tail call
            let (optimize, res) = Self::apply_lambda(&proc, &args, true)?;
            if optimize {
                // valid tail call
                Tree::replace_tree(&p, res);
                stack.pop();
            } else {
                Tree::replace_tree(&expr, res);
            }
            continue;
        }
    }
    Tree::replace_tree(&expr, Self::apply_lambda(&proc, &args, false)?.1);
} else {
    Tree::replace_tree(&expr, self.apply(&proc, &args)?);
}

Code Snippet 11: missing part of eval function that is responsible for user defined procedures

It’s big. Let’s dig into it.

First, we check if our procedure is user_defined_procedure, as it will allow us to early go for apply, and still be able to override, say + to user procedure with the same name, and call apply_lambda instead.

If it is user_defined_procedure we go in and check if it has a parent. If it is, this is possibly a tail call, but we also have to check that the last expression in our parent is exactly the expression we’re going to evaluate and that the first expression in the parent is progn because this indicates that it is really the last expression. If this is true, this is a possible tail call, but we can’t be sure yet. We call apply_lambda with third argument set to true because the check in apply_lambda will further tell us if this was a tail call or not.

So if apply_lambda returns true we optimize our call by replacing the parent with result of the lambda application and pop our tree traversal stack, then we continue. If it was not a tail call, we replace the current expression with the result and continue.

If there was a parent, but the expression was not last, or there was no progn as the first one we pass false as the third argument to avoid unnecessary checks for a tail call, and replace the current expression with the result.

So what exactly apply_lambda does? Let’s see:

let lambda_args = Self::first_expression(&lambda)?;
let lambda_body = Self::rest_expressions(&lambda)?[0].clone();
let mut can_optimize = tail_call;

This deconstructs the #procedure:anonymous pattern in the same way how we’ve constructed it in the lambda procedure. We also create can_optimize and initialize it with our tail_call value.

Next, we have to create a closure, and this is going to be messy. Here’s the code:

if tail_call {
    if let Some(p) = Tree::parent(&Tree::parent(&lambda).unwrap()) {
        let args: Vec<String> = lambda_args
            .borrow()
            .siblings
            .iter()
            .map(|a| a.borrow().data.data.to_string())
            .collect();
        let mut body = lambda_body.borrow_mut();
        for (key, val) in p.borrow().data.scope.iter() {
            if !args.is_empty() && !args.contains(key) {
                can_optimize = false;
            } else {
                body.data.scope.insert(key.clone(), val.clone());
            }
        }
    }
}

We only do this if this is a tail call because if it is not, we do not have to create any closures besides those that were created in the lambda procedure, because lookup will be able to find bindings in parents. However, we can’t simply create closure and always optimize our call. That’s how we check:

First, we check if our parent has a parent. Because at this point we only have sub expressions, we need this extra up the tree. Then, we create a vector of the string representation of our arguments and take mutable reference to the lambda body. Then, for each value in the scope of the parent’s parent, we do this: if the argument list is not empty, and the argument list contains the key from the parent’s scope we can’t optimize the tail call. Because we can lose the original value of that binding, which isn’t gonna work when we, say, pass continuations. If it is not, we put this binding to our lambda scope.

So if we can successfully put every binding from the parent expression to our current expression, means we no longer need the parent expression and can optimize it!

After that, we can deal with the usual lambda application stuff. First, we have to bind all argument names to respective values, but there’s a problem. Arguments can be in form of a list, or the list itself, so we have to deal with provided arguments differently:

let procedure_name = match &lambda.borrow().data.data {
    Data::String { data } if data.len() > 11 => data[11..].to_string(),
    _ => "anonymous".to_string(),
};

match Self::expression_type(&lambda_args) {
    // …
}

We store the name of our procedure⁴ so we could signal meaningful errors in case there’s a wrong argument count or other noise. Then we check the type of lambda_args. It either can be Procedure³ or Name. In the case of Procedure it is simple, because all we have to do is compare array sizes, and if those are the same, go through each array and bind data from one array to data from another, and put it into the scope:

Type::Procedure => {
    Self::check_argument_count(
        &procedure_name,
        ArgAmount::NotEqual(lambda_args.borrow().siblings.len()),
        &args,
    )?;
    let mut lambda_body = lambda_body.borrow_mut();
    for (key, val) in lambda_args.borrow().siblings.iter().zip(args) {
        lambda_body
            .data
            .scope
            .insert(key.borrow().data.to_string(), val.clone());
    }
}

Code Snippet 12: Binding each value to each argument of a procedure

In case it is Name we need to create a quoted list with all values from arguments that were passed to our function and bind it to the name:

Type::Name => {
    let quoted_args = Tree::new(GRData::from_str("("));
    Tree::push_child(&quoted_args, GRData::from_str("#procedure:quote"));
    let list = Tree::push_child(&quoted_args, GRData::from_str("("));
    for c in args.iter() {
        let res = match Self::expression_type(&c) {
            Type::List | Type::Symbol => {
                let res = Self::rest_expressions(&c)?;
                res[0].clone()
            }
            _ => c.clone(),
        };
        Tree::push_tree(&list, res);
    }
    lambda_body
        .borrow_mut()
        .data
        .scope
        .insert(lambda_args.borrow().data.to_string(), quoted_args);
}

Code Snippet 13: Constructing a list of argument values and binding it to procedure argument

E.g. if we call expression ((lambda x x) 1 2 3 4 5), this code above will create '(1 2 3 4 5) and bind it to x. Then our procedure will return this x.

In case it was not a Name or Procedure we throw error:

_ => {
    return Err(EvalError::GeneralError {
        message: "wrong binding type".to_owned(),
    })
}

This is pretty much it! After we’ve done all of this our body is ready! So we return it, as well as information about the possibility of tail call optimization:

Ok((can_optimize, lambda_body))

You remember that our body is just progn which contains all expressions, so our eval receives it and does those ones by one. So we simply convert all user-defined procedures to primitive procedures that are supported by an interpreter and evaluated.

Tail call elimination example

So above we’ve discussed how tail call elimination works in GRScheme. Let’s illustrate that process using trees for this simple procedure:

> (define f (lambda (x)
              (display x)
              (newline)
              (f (+ x 1))))
> (f 0)
0
1
2
3
…
9578583832728723847384
9578583832728723847385
9578583832728723847386
…

This procedure displays the number we give it, prints a newline, and calls itself with the number increased by 1. In most languages that have no tail call elimination the biggest number, we can see represents the size of the stack that we can possibly have. However in Scheme, there’s no such limitation, and we can evaluate this procedure in constant space, e.g. the amount of RAM will stay the same. Let’s see how this procedure is represented in our interpreter.

First, we evaluate lambda and produce such tree:

Then, define changes #procedure:anonymous to #procedure:f and binds f to the value of #procedure:f in global scope. Then we apply f to 0 we have this tree:

lookup procedure checks the value of f and replaces f with our #procedure:f:

Now we call apply_lambda where proc will hold #procedure:f and args will hold [0]. This binds x to 0 at the scope of the progn, and gives the evaluator this tree:

To clarify, the Scope here is not a tree node, but data within the ( node.

Now evaluator starts the process of evaluating this expression. It sees progn and evaluates all its arguments in order.

First, we evaluate display and its one and only argument x. We lookup it, and find the value in progn's parent (:

Figure 11: Application of (f 0)

Then display prints 0 on screen, and evaluates to #void:

Figure 12: Evaluation of (display x)

After that, we evaluate newline. It also prints \n on the screen, and evaluates to #void:

Figure 13: Evaluation of (newline)

After that we evaluate f, but first we need to evaluate its argument (+ x 1). We lookup x as we did for newline, and evaluate + with 0 and 1 to 1:

Figure 14: Evaluation of (+ x 1)

Now we are ready to call f. Let’s immediately apply it to see how it looks:

Figure 15: Unoptimized call result

And without tail call optimization each iteration will expand like this. However, notice that we’re evaluating the last expression, and the first one is progn. This means, that to this point we’ve evaluated everything in this progn and we can throw it away. So instead of expanding the last expression, we replace our old progn with a new one:

Figure 16: Optimized call result

We again apply all expressions in progn, then call (f 2) at the end and optimize the tail call. This process repeats every time and we always use the same amount of space.

This pretty much sums up how the interpreter works. All these trees can be viewed as a direct representation of what’s going on underneath.

Quasiquoting, Unquoting, and Unquote Splicing

Currently, GRScheme has poor support for these features. It supports quasiquoting but one thing is not done correctly, and this is due to internal data representation. I’m not sure where to fix this, because it can be fixed in two places.

To understand this problem let’s have two examples of quasiquoting and quoting in the list evaluated in Racket:

> (let ((name1 'a)
        (name2 'b))
    `(a ,name1 b ,name2))
'(a a b b)
> (let ((name1 'a)
        (name2 'b))
    (list 'a name1 'b name2))
'(a a b b)

If we evaluate the same expressions in GRScheme, we’ll get this:

> (let (name1 'a
        name2 'b)
    `(a ,name1 b ,name2))
'(a 'a b 'b)
> (let (name1 'a
        name2 'b)
    ((lambda x x) 'a name1 'b name2))
'(a a b b)

This happens because unquote evaluates name1, and name1 holds 'a. So it puts 'a to the list. However when we construct lists with (lambda x x) or with cons GRScheme imitates a regular Scheme and removes unnecessary quotes. I thought that this will work when I started developing GRScheme, but it seems that quasiquoting shatters my dreams here.

I can fix this in quasiquoting as a hack, but I think that instead, I should remove hacks related to list construction and all car-cdr related stuff. This way it will directly represent the internal data structure and will be much more reliable and consistent.

The other way I can fix this is to make it work correct, e.g. (quote (a)) will yield (a), but this will introduce some other additional checks in the tree, where we have to check if we should evaluate our expression or not. Maybe I’m missing something though, and this can be done without additional checks. Anyway, I’ll leave it for later. Unquote splicing is not supported at all.

Top level scope, in this case, is empty because we imitate the use of the REPL. When a file is being parsed its contents are put to progn procedure, and only the last expression result is returned. In the case of the REPL if you enter multiple expressions without line breaks it will also be put into the progn call, however, REPL removes it and evaluates expressions in global scope, thus keeping bindings. Actually, every parsed expression sequence is wrapped to progn. ↩︎
Quasiquote implemented in terms of top-down gathering all valid unquotes and evaluating those bottom-up. This is not really suitable technique for proper quasiquoting, because splicing is really hard to do this way. Also because we store symbols in quoted form unquoting a name that holds a symbol will put a symbol in the quoted form to our list. ↩︎
Procedure, in this case, means not a real procedure that we will apply, but a form of data that we expect. E.g. out of context, this code (x) is a procedure application, but in the context of a lambda, it is a list of arguments. Reusing the same patterns can be misleading, but I think it is better than adding new data structures, as some other languages do. E.g. in Fennel lambdas are created by using square brackets: ((lambda [x] (* x x)) 2) will produce 4, and ((lambda (x) (* x x)) 2) will produce an error: Compile error in 'fn' unknown:?: expected vector arg list [a b …]. This also applies to some other Lisp derivatives, like in Clojure: (fn [x] (* x x)). ↩︎ ↩︎
define can detect if we’re binding a value or a procedure. And if we’re binding anonymous procedure, define changes its name to the string representation of the first argument passed to define. E.g. if we do (define square (lambda (x) (* x x))), define will change #procedure:anonymous to #procedure:square. ↩︎

Comment via email

GRScheme Design Part 1 - Parsing

Tue, 25 Feb 2020 18:02:00 +0300

At the end of my previous post, I mentioned that I will write more about GRScheme - the language that I’m working on. As of this moment, I’m rewriting it to be more robust and to fix some design flaws that make it impossible to do such things as tail call elimination. This series of posts will describe various things I’ve done to build a working interpreter and build a language around it.

But first, let’s define what GRScheme is, and what I expect it to be when I finish development:

GRScheme is an interpreted functional language with scheme-like syntax, which uses graph reduction as a basis for evaluation.
GRScheme is not a practical programming language built for wide use and is only a self-educational project.

Note: There are far more comprehensive and reliable lisps out there, so if you are interested in one for some practical use, you have plenty to choose from. E.g. Arc, Clojure, Common Lisp, GNU Guile, Racket, e.t.c., and GRScheme are not part of this list. Please keep this in mind.

The basic process of program execution should be:

Here, Reader is a special unit that reads user input from the REPL or file contents character by character and produces GRScheme-data, a special data structure that we will be used for evaluation. The evaluator consumes this data structure and produces the effect by using graph reduction to provide the final result.

There’s also dashed block, that contains Compiler, and sub-blocks Macro-expander, and Optimizer. Currently, I already have a working Evaluator and Reader, and thinking about a possible way to produce an intermediate result that will speed up evaluation. There definitively will be macro-expander at some point, though it doesn’t require real compilation to be applied. But this is gonna wait till the next posts, as I have to figure this out first.

In this post, I would like to talk about the data structure I use to represent programs because it is an interesting topic in the language, that I’m using for writing language core which is the Rust Programming Language.

Let’s look at an example program in GRScheme:

(+ 1 (* 2 3 4) (/ 9 (+ 1 2)))

Code Snippet 1: arithmetic expression

This is a mathematical expression that can be written as this formula:

Which produces such tree:

However, this tree is not presented like this in GRScheme-data format.

First, this is not a binary tree, and even if it was, we still would need to implement some data structure in Rust that would represent this tree. So what’s the definition of such data structure would be?

Creating the `Tree` in Rust

If we will look at most tree definitions we will see that usually, we define some amount of sub-nodes, typically 2, for binary trees, but in our case, that would be a variable amount of siblings, and a field for data. For example in Python such Tree can be defined like this:

class Tree:
    def __init__(self, data = None, siblings = None):
        self.siblings = siblings
        self.data = data

Note: I’m not a Python programmer, and I don’t know Python, so this may be not the best way to achieve what I’m describing, but this works, and similar to most tree representations I’ve seen.

And we can set siblings to new Tree’s like this:

root = Tree(data = 0)
root.siblings = [Tree(data = 1),
                 Tree(data = 2)]

There are many ways to declare trees in Rust, each with its own weaknesses and strong points, such as arena tree, or boxed tree, but I’ve decided to go for more or less classic C-like reference counted tree, because arena trees suffer from an expensive remove of items in the tree, and I have not figured out how to manage parents with Box. So if we try to translate the Python tree directly to Rust, we will get this:

struct Tree<T> {
    data: T,
    siblings: Vec<Tree<T>>,
}

impl <T> Tree<T> {
    pub fn new(data: T) -> Tree<T> {
        Tree {
            data,
            siblings: vec![],
        }
    }
}

fn main() {
    let mut root = Tree::new(0);
    root.siblings = vec![
        Tree::new(1),
        Tree::new(2)
    ];
}

This is a generic data structure that can hold any type as T. We can create a Tree of u32, f64, String or any other data just by calling new with arguments of different types. For example, calling Tree::new(0) and Tree::new("0") will create trees of types Tree<i32>, and Tree<&str> respectively. Here we create the tree that holds 0 as its root value and has two sub-nodes that hold 1 and 2 values. So this is pretty much a tree already!

Though, this is not very usable, because we can traverse the tree only to the leaves, and can’t go back to the root or previous node. Let’s add the parent field to the tree. In Python it would be like this:

class Tree:
    def __init__(self, data = None, siblings = None, parent = None):
        self.siblings = siblings
        self.data = data
        self.parent = parent
    def __str__(self):
        return f'[data: {self.data}, parent: {self.parent}, siblings: {self.siblings}]'
    __repr__ = __str__

root = Tree(data = 0)
root.siblings = [Tree(data = 1, parent = root),
                 Tree(data = 2, parent = root)]

print(root)

Note: I’ve also defined a __str__(self) method that will be useful for printing the whole data structure in a recursive fashion. This is not required but helps investigate how things are working.

Running the code above will print this (formatted for more readability):

[ data: 0,
  parent: None,
  siblings: [ [ data: 1,
                parent: [ data: 0,
                          parent: None,
                          siblings: [...] ],
                siblings: None ],
              [ data: 2,
                parent: [ data: 0,
                          parent: None,
                          siblings: [...] ],
                siblings: None ] ] ]

We can see that 0 has no parent, two sub-nodes 1 and 2 each of which has a parent 0 that has [...] sub-nodes. This is a recursive data structure, but we only descend it for sake of printing, yet we’re able to ascend it from any sub-node to root as well.

So what should we add in rust in order to achieve the same result? One could use a reference, but this will fill our code with a lot of lifetimes. The other way is to use Box indirection, or reference counting. If we try to use plain Box we would fail to specify the correct parent:

// Deriving something in Rust tells the compiler to generate methods for
// our data structure.  In this case, we derive the `Clone` trait
// implementation.
#[derive(Clone)]
struct Tree<T> {
    data: T,
    parent: Option<Box<Tree<T>>>,
    siblings: Vec<Tree<T>>,
}

impl <T> Tree<T> {
    pub fn new(data: T, parent: Option<Box<Tree<T>>>) -> Tree<T> {
        Tree {
            data,
            parent,
            siblings: vec![],
        }
    }
}

fn main() {
    let mut root = Tree::new(0, None);
    root.siblings = vec![
        Tree::new(1, Some(Box::from(root.clone()))),
        Tree::new(2, Some(Box::from(root.clone())))
    ];
}

Since we have to specify a type for our parent, and we actually can have no parent (root case), I use Option type that can either hold a None or Some(T). Inside Option we have a Box, which is a pointer to the heap allocated data, and inside the Box there’s the Tree<T> itself.

The problem in this code is that we do clone of the root and create our Box pointer out of it. So if we go to the parent of the node which holds 1 we’ll arrive at the copy of the root node which has no sub-nodes. This problem can be solved if we use the reference counted pointer to the tree instead of the Box:

use std::rc::Rc;

#[derive(Clone)]
struct Tree<T> {
    data: T,
    parent: Option<Rc<Tree<T>>>,
    siblings: Vec<Rc<Tree<T>>>,
}

// -- snip --

fn main() {
    let mut root = Tree::new(0, None);
    *root.siblings = vec![
        Tree::new(1, Some(root.clone())),
        Tree::new(2, Some(root.clone())),
    ];
}

This looks like it, but there’s still a problem. If we try to run this code, we’ll get this error:

error[E0277]: the size for values of type `[std::rc::Rc<Tree<{integer}>>]` cannot be known at compilation time
  --> src/main.rs:22:5
   |
22 |     *root.siblings = &[
   |     ^^^^^^^^^^^^^^ doesn't have a size known at compile-time
   |
   = help: the trait `std::marker::Sized` is not implemented for `[std::rc::Rc<Tree<{integer}>>]`
   = note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait>
   = note: the left-hand-side of an assignment must have a statically known size

This is an interesting error, and documentation suggests a solution for this, but we can fix this by using a reference. Using a plain reference in the structure definition will force us to specify lifetime everywhere, and I’m not yet good enough with lifetimes, since it’s a pretty new concept for me. Instead, let’s use RefCell:

#[derive(Clone)]
struct Tree<T> {
    data: T,
    parent: Option<Rc<RefCell<Tree<T>>>>,
    siblings: Vec<Rc<RefCell<Tree<T>>>>,
}

Oh, this compiles! Let’s print our tree! To do so, we can derive the Debug trait implementation for our structure and print it with {:?} placeholder in the println! macro. It works pretty much the same as in Python code, but is generated automatically:

use std::cell::RefCell;
use std::rc::Rc;

#[derive(Clone, Debug)]
struct Tree<T> {
    data: T,
    parent: Option<Rc<RefCell<Tree<T>>>>,
    siblings: Vec<Rc<RefCell<Tree<T>>>>,
}

// -- snip --

fn main() {
    let root = Tree::new(0, None);
    root.borrow_mut().siblings = vec![
        Tree::new(1, Some(root.clone())),
        Tree::new(2, Some(root.clone())),
    ];
    println!("{:?}", root);
}

Let’s run it:

Tree { data: 0, parent: None, siblings: [RefCell { value: Tree { data: 1, parent: Some(RefCell { value: Tree { data: 0, parent: None, siblings: [RefCell {value: Tree { data: 1, parent: ...

!!! gazillion lines !!!

... Some(RefCell { value
thread 'main' has overflowed its stack
fatal runtime error: stack overflow
Aborted (core dumped)

Whoa… This produces a stack overflow on printing. Because compiler generated Debug trait implementation is recursive. But our tree only has two sub-nodes, how could recursion blow the stack? And why we only see data: 1 then data: 0, then data: 1 again. Where’s data: 2 node?

So the problem here is that our recursive code prints the data for the root node, then go into the next recursion step on the 1 node, which contains the reference to the root, so recursion goes to the root first, and loops infinitely inside a cycle of two nodes. These are called cyclic references and those are bad for many reasons. One of those is that cyclic references will never be freed because their reference counter will never reach zero. Luckily for us, Rust provides a solution!

Instead of using Rc for parent, let’s use Weak:

use std::cell::RefCell;
use std::rc::{Rc, Weak};

#[derive(Clone, Debug)]
struct Tree<T> {
    data: T,
    parent: Option<Weak<RefCell<Tree<T>>>>,
    siblings: Vec<Rc<RefCell<Tree<T>>>>,
}

impl<T> Tree<T> {
    pub fn new(data: T, parent: Option<Weak<RefCell<Tree<T>>>>) -> Rc<RefCell<Tree<T>>> {
        Rc::from(RefCell::from(Tree {
            data,
            parent,
            siblings: vec![],
        }))
    }
}

fn main() {
    let root = Tree::new(0, None);
    root.borrow_mut().siblings = vec![
        Tree::new(1, Some(Rc::downgrade(&root))),
        Tree::new(2, Some(Rc::downgrade(&root))),
    ];
    println!("{:?}", root);
}

Code Snippet 2: Tree in Rust using Rc and Weak

Now if we run this code we get the proper tree representation:

RefCell {
    value: Tree {
        data: 0,
        parent: None,
        siblings: [
            RefCell { value:
                Tree { data: 1,
                       parent: Some((Weak)),
                       siblings: [] } },
            RefCell { value:
                Tree { data: 2,
                       parent: Some((Weak)),
                       siblings: [] } }] } }

And it looks almost exactly how we declare it in the main function, and certainly looks like what we’ve got in the Python example! However, those types are hard to type every time. We can solve this by using type:

use std::cell::RefCell;
use std::rc::{Rc, Weak};

type NodePtr<T> = Rc<RefCell<Tree<T>>>;
type WeakNodePtr<T> = Weak<RefCell<Tree<T>>>;

#[derive(Clone, Debug)]
struct Tree<T> {
    data: T,
    parent: Option<WeakNodePtr<T>>,
    siblings: Vec<NodePtr<T>>,
}

impl<T> Tree<T> {
    pub fn new(data: T, parent: Option<WeakNodePtr<T>>) -> NodePtr<T> {
        Rc::from(RefCell::from(Tree {
            data,
            parent,
            siblings: vec![],
        }))
    }
}

And here’s a simple recursive traversal for our tree, that prints it as an S-expression:

impl<T> ToString for Tree<T>
where
    T: ToString,
{
    fn to_string(&self) -> String
    where
        T: ToString,
    {
        fn build_string<T>(node: &Rc<RefCell<Tree<T>>>, string: &mut String)
        where
            T: ToString,
        {
            if !node.borrow().siblings.is_empty() {
                string.push_str("(");
            }
            string.push_str(&format!("{} ", node.borrow().data.to_string()));
            for n in node.borrow().siblings.iter() {
                build_string(n, string);
            }
            *string = string.trim().to_owned();
            if !node.borrow().siblings.is_empty() {
                string.push_str(")");
            }
            string.push_str(" ");
        };

        let mut string = format!("({} ", self.data.to_string());

        for c in self.siblings.iter() {
            build_string(c, &mut string);
        }

        string = string.trim().to_owned();
        string.push_str(")");

        string
    }
}

Now let’s try it. Remember that formula I’ve shown before? Let’s convert it to the tree:

fn main() {
    let plus = Tree::new("+", None);
    plus.borrow_mut().siblings = vec![
        Tree::new("1", Some(Rc::downgrade(&plus))),
        Tree::new("*", Some(Rc::downgrade(&plus))),
        Tree::new("/", Some(Rc::downgrade(&plus))),
    ];

    let mul = plus.borrow().siblings[1].clone();
    mul.borrow_mut().siblings = vec![
        Tree::new("2", Some(Rc::downgrade(&mul))),
        Tree::new("3", Some(Rc::downgrade(&mul))),
        Tree::new("4", Some(Rc::downgrade(&mul))),
    ];

    let div = plus.borrow().siblings[2].clone();
    div.borrow_mut().siblings = vec![
        Tree::new("9", Some(Rc::downgrade(&div))),
        Tree::new("+", Some(Rc::downgrade(&div))),
    ];

    let plus1 = div.borrow().siblings[1].clone();
    plus1.borrow_mut().siblings = vec![
        Tree::new("1", Some(Rc::downgrade(&plus1))),
        Tree::new("2", Some(Rc::downgrade(&plus1))),
    ];

    println!("{}", plus.borrow().to_string());
}

If we run this code, we’ll see this:

(+ 1 (* 2 3 4) (/ 9 (+ 1 2)))

Which directly represents our example from the beginning. This gets us to the point where we can use this data structure to represent our tree. However, making trees in such a way is impractical and tedious. I’ve implemented some more methods for the tree, such as add_child, parent, adopt_tree, replace_tree, and so forth. You can check those in the GRScheme source code repository if you want.

But in order to automate this process so we could specify an expression and get its tree representation we have to write the parser, or so-called reader, which will read input character by character, analyze it and produce a tree as the result. Reader is a crucial part of our interpreter since it serves as a translation unit between the user and the interpreter itself.

Writing `reader`

Since our tree is generic we need to define a proper data structure to hold both the data from input and additional information we may need to build the tree correctly. For that purpose let’s create another structure in reader module:

struct GRData {
    data: String,
}

Not as big as one might expect, and we could use String directly in Tree<String> fashion, but it’ll get bigger, I promise. We’ll need some extra data later on, but for now, let’s limit ourselves to String holding field named data.

So, how will we parse text? First, we need to write a function that will do the text processing:

fn parse(expression: &str) -> Result<NodePtr<GRData>, ReadError> {}

There’s a lot going on already. First, we create a function named parse, that takes one parameter expression of type &str and returns the Result which can either be NodePtr<GRData> for our tree, or ReadError. Result is a special type for functions that might fail, so we can store some error information in the Err(T) field. If the function succeeds we store the result in Ok(T). So in our case we’ll be storing error messages as ReadError, and use our GRData for Ok value. This is what I like about Rust because with Result and Option there’s no need for exceptions. We can carry our error through functions that expect that error and know how to deal with it. Now let’s think about what we’ll be parsing.

The Scheme’s syntax is quite simple - mostly everything is enclosed with parentheses, but there are some special cases as well.

The first one is comments. Comments are line-wise and written using the ; symbol, so If we’re finding that symbol in our input we can skip parsing those. With some exceptions.

The other special case is double quote ". If we see it in our input it means we’re inside the string, and we should put everything as a single piece of information until we meet another ". This also includes comments, because those can be part of a string. However, even with quotes, there’s a special case.

Escaping quotes inside strings can be done with the \ symbol like this: "this \"string\" contains a nested string". So we should keep an eye on this symbol and what’s going after it.

Now on to regular things in Scheme’s syntax.

In my implementation, all parentheses will represent the same thing, e.g. the list boundaries. However parentheses must match each other, so if the list was started with the ( it must end with ), and if it was started with [, or { it must be ended with ], or } respectively.

Quoting of items is possible with ' and `, right before the list. Unquoting can be done with ,, and ,@ before the list as well. These characters can not be used inside words except for strings.

I’ve mentioned “word” in the previous paragraph. Word is a sequence of non-reserved non-white-space characters. In our case everything that is not a space, tab, newline, semicolon, quote, double-quote, `, @, ,, escape character, all kinds of parentheses, and #, is going to be a valid word. For the purpose of a more general mechanism, strings are words as well. These are valid words in terms of our parser:

a
word
dashed-word
arrowed->word
"strings are treated as words"
slashed/word
1
123
-=.+_*^&%$!~@

These are invalid words:

paren(inside
colon,inside
unquote,@splicing-indide
semicolon;inside
quotes'inside`

Doing such parsing is nothing really special in Rust. It’s a bunch of logic around iterating over characters. The interesting part is tree construction. If you’re interested in parse source, you can look at it here, though it can be implemented better.

So the basic idea is that we parse character stream into words or items, and decide what to do with those. For example, if we meet ( we should treat it as a new list, and if we meet a as a name. For that purpose, I’ve written a function that analyses words and produces tokens. And this enum contains all tokens that we will need:

enum Token {
    Value,
    Eval,
    Apply,
    Quote { kind: &'static str },
    Symbol,
    None,
}

Value is a token for anything that is essentially a value, e.g. "4", 42, but since we’re parsing raw character data, names are treated as Values too. Don’t worry, we will not need this information when we will do the tree evaluation, so it’s safe to mix these two.

Eval is used for (, {, and [, and basically means that we will start new sub-tree here. Apply is a matching counterpart, which will say that we must go up to a parent.

Next is Quote and it is a bit more interesting because it features this kind: &'static str thing. What this essentially means is that we will hold an additional static string with quote kind in this enum field so we could later distinguish quote from quasiquote and friends. What’s also interesting is that we store unwrapped information in the tree, so we convert 'a into (quote a), and ,a to (unquote a) and that’s where kind comes in handy.

Symbol is used to ascend from special quote variants, I’ll talk about it a bit later.

And the last is None and it is used as an initial value and never used in the parser anymore.

Now to the tokenizer itself:

fn tokenize(&mut self, word: &str) -> Result<Token, ReadError> {
    let last_token = &self.last_token;

    let token = match word {
        "(" | "[" | "{" => Token::Eval,
        ")" | "]" | "}" => match last_token {
            Token::Quote { .. } => {
                return Err(ReadError::UnexpectedExpressionEnd {
                    line: self.line_num,
                    column: self.column_num,
                })
            }
            _ => Token::Apply,
        },
        "'" | "," | ",@" | "`" => Token::Quote {
            kind: match word {
                "'" => "quote",
                "," => "unquote",
                "`" => "quasiquote",
                ",@" => "unquote-splicing",
                _ => {
                    return Err(ReadError::InvalidSyntax {
                        message: format!(
                            "unexpected quote type \"{}\". line: {}, column: {}",
                            word, self.line_num, self.column_num
                        ),
                    })
                }
            },
        },
        &_ => match last_token {
            Token::Quote { .. } => Token::Symbol,
            _ => Token::Value,
        },
    };

    self.last_token = token.clone();
    Ok(token)
}

It’s pretty simple. If we receive either of the open parentheses, it’s always Eval. If we receive any of the close ones, it’s always Apply, unless the last one was Quote, e.g. '), then that’s an error. If we receive any of the quotes then it’s Quote, the error there would never happen, but Rust complains that other patterns are not handled, so it is there. And everything else is either Value or Symbol if the last one was Quote. This may not seem enough, but we can actually build a valid program tree out of this information.

So how do we manage tree structure with our Tree<T> type? First, we need to declare some methods to add things to trees in various ways. The first and the most obvious is add_child:

impl<T> Tree<T>
where
    T: Clone,
{
    pub fn new(data: T) -> NodePtr<T> {
        Rc::from(RefCell::from(Tree {
            data,
            parent: None,
            siblings: vec![],
        }))
    }

    pub fn add_child(node: &NodePtr<T>, data: T) -> NodePtr<T> {
        let new_node = Rc::from(RefCell::from(Tree {
            data,
            parent: Some(Rc::downgrade(node)),
            siblings: vec![],
        }));
        node.borrow_mut().siblings.push(new_node.clone());
        new_node
    }
}

We can see that something is going on here. This function takes two arguments - node, for a node to which we will add a child node, and data, which holds the data for a new child. Inside we create new_node to hold reference counted pointer to our new Tree, that has Some parent, no siblings and our data. After that we mutually borrow original node and push our new_node onto its list of sub-nodes. And in the end, we return the pointer to our new_node out of our function. This allows us to build trees in this fashion:

fn main() {
    let root = Tree::new(0);
    Tree::add_child(&root, 1);
    Tree::add_child(&root, 2);

    println!("{:?}", root);
}

Which is essentially the same tree as in previous example but far more compact. If we run it produces the same output (again, formatted for readability):

RefCell {
    value: Tree {
        data: 0,
        parent: None,
        siblings: [
            RefCell {
                value: Tree {
                    data: 1,
                    parent: Some((Weak)),
                    siblings: [] } },
            RefCell {
                value: Tree {
                    data: 2,
                    parent: Some((Weak)),
                    siblings: [] } }] } }

This also allows us to select to which node we want to add a child:

fn main() {
    let root = Tree::new(0);
    let one = Tree::add_child(&root, 1);
    Tree::add_child(&one, 2);

    println!("{:?}", root);
}

Which yields:

RefCell {
    value: Tree {
        data: 0,
        parent: None,
        siblings: [
            RefCell {
                value: Tree {
                    data: 1,
                    parent: Some((Weak)),
                    siblings: [
                        RefCell {
                            value: Tree {
                            data: 2,
                            parent: Some((Weak)),
                            siblings: [] } }] } }] } }

This means that as long as we keep track of the current node, we can build any tree, and if we can go up when needed, we can build complex trees without recursion in an iterative process.

So how do we add items that we’ve parsed and tokenized to the tree? First, we need to create the tree, and for that purpose, I would like to use progn as it is called in Emacs Lisp, or begin as it is called in Scheme:

fn parse(&mut self, expression: &str) -> Result<NodePtr<GRData>, ReadError> {
    let mut comment = false;
    let mut inside_word = false;
    let mut inside_string = false;
    let mut unquote = false;
    let mut paren_stack = vec![];

    let mut item = String::new();

    // we create the root of the tree which holds `(progn
    let mut tree = Tree::new(GRData::from_str("("));
    Tree::add_child(&tree, GRData::from_str("progn"));

    // here we save the tree root for later use
    let root = tree.clone();

    self.line_num = 1;
    self.column_num = 0;

    for c in expression.chars() {
        // parsing characters, producing tokens, constructing the tree
    }

    Ok(root) // after we've parsed we return the pointer to the root
}

Here we can see, that here we are defining some variables to hold parser state, such as that we’re inside a comment or string, or that we’re currently inside a word, and creating item for holding our words, and paren_stack for checking the balance of different parentheses.

After that, we create our tree, that holds (, and immediately add a child that holds progn. Then we clone tree pointer to root and do the parsing in for loop. After that, we constructed our tree and return it to the caller.

Let’s look at the part of the for loop code:

if !comment && !inside_string {
    match c {
        '(' | '[' | '{' => {
            paren_stack.push(c);
            item.push(c);
            inside_word = false;
        }
        // …  see the source for full code if you're interested
        '"' => {
            inside_string = true;
            inside_word = true;
        }
        ';' => {
            comment = true;
            continue;
        }
        ' ' | '\t' | '\n' => inside_word = false,
        _ => inside_word = true,
    }
    if inside_word {
        item.push(c);
    } else if !item.is_empty() {
        tree = self.add_to_tree(&tree, &item)?;
        item.clear();
        unquote = false;
    }
} else if inside_string {
    item.push(c);
    match c {
        '\\' => escaped = true,
        '"' => {
            if !escaped {
                inside_string = false;
                tree = self.add_to_tree(&tree, &item)?;
                item.clear();
            }
            escaped = false;
        }
        '\n' => {
            escaped = false;
            self.line_num += 1;
            self.column_num = 0;
        }
        _ => escaped = false,
    }
} else if comment && c == '\n' {
    self.line_num += 1;
    self.column_num = 0;
    comment = false;
}

If we’re not in a comment and not inside a string, we do parsing as normal. We push characters inside item to form a word until inside_word becomes false, which happens when we meet some word delimiter, such as white-space. Once we’re no longer inside word and item is not empty, we do tree = self.add_to_tree(&tree, &item)?, clear item and go further. If we’re inside string we’re basically waiting for the closing double quote, unless it is escaped with \. And if we’re inside comment, we simply wait for a newline.

So what’s this self.add_to_tree(&tree, &item)? does? Let’s see:

fn add_to_tree(&mut self, node: &NodePtr<GRData>, item: &str) -> Result<NodePtr<GRData>, ReadError> {
    match self.tokenize(item)? {
        Token::Quote { kind } => {
            let eval = Tree::add_child(node, GRData::from("(", true));
            Tree::add_child(&eval, GRData::from_str(&kind));
            Ok(eval)
        }
        Token::Eval => Ok(Tree::add_child(node, GRData::from_str("("))),
        Token::Symbol => {
            Tree::add_child(node, GRData::from_str(item));
            self.parent(node)
        }
        Token::Apply => self.parent(node),
        _ => {
            Tree::add_child(node, GRData::from_str(item));
            Ok(node.clone())
        }
    }
}

First, we get a token for the current item, then, depending on the token, we add a sub-tree, or get the parent of a node. and return the pointer to the tree for later use. This means, that if the current token was Eval, we add ( to the tree, and return the pointer to it. If the current token was Apply we return the pointer to the parent of the node.

Quote and Symbol are bit interesting. You see, since we unwrap the tree to full form, e.g. 'name is being stored as (quote name), we need to define a way of going up without parenthesis.

The problem here is that when we see this input (quote name) we have all information needed to build a tree - ( marks the root, quote is the first child, name is second, and ) marks that we have to go up. In the case of 'name, we see ' which we expand to (quote, but how do we actually go up without )? We can’t easily wrap the whole expression without implementing a look-ahead parser. That’s when Quote, and Symbol tokens are becoming useful.

When we see ' we mark that our current token is quote, and generate the (quote-kind tree. So for ' it will be (quote. for ` - (quasiquote, and so forth. Next, when we meet =symbol, it means that we have some name, that must be wrapped with a parenthesis, and we should take one extra up.

That’s why I’ve mentioned that we’ll need more info in our GRData than simply a String for our value. GRData also encodes that current node should take extra up:

struct GRData {
    data: String,
    extra_up: bool,
}

And that GRData::from("(", true) method’s second true parameter is written to this extra_up field. You can see that when we finish our work with Symbol we call self.parent(node), which actually checks if we need to take that extra one up, which is stored in its parent:

fn parent(&mut self, node: &NodePtr<GRData>) -> Result<NodePtr<GRData>, ReadError> {
    let mut current = node.clone();
    while let Some(parent) = Tree::parent(&current) {
        if parent.borrow().data.extra_up {
            current = parent;
            current.borrow_mut().data.extra_up = false;
        } else {
            return Ok(parent);
        }
    }
    Err(ReadError::UnexpectedExpressionEnd {
        line: self.line_num,
        column: self.column_num,
    })
}

This way it works both for lists and symbols because the quote’s parenthesis always has this extra up. Because of this we do not actually need a look-ahead parser, and can correctly wrap all different cases of quote short-hands with appropriate parentheses. E.g. `(a ,b 'c) transforms into (quasiquote (a (unquote b) (quote c))), thus making shorthand for various quotes just a syntax sugar.

To sum this up, let’s see how the interpreter sees this expression (over-complicated for observation reasons):

(define list (lambda x `(,@x)))

Assuming that Current is a pointer to current Node, and Tree is holding (, and that we always operating on Current, when getting parent, or adding new nodes, we can view what’s happening as this table:

Input	Token	Action To Perform
`(`	`Apply`	add `Tree`, `Current` ← `Tree`
`define`	`Value`	add `define` to `Current`
`list`	`Value`	add `list` to `Current`
`(`	`Apply`	add `Tree`, `Current` ← `Tree`
`lambda`	`Value`	add `lambda` to `Current`
`x`	`Value`	add `x` to `Current`
`	`Quote`	add `Tree`, `Current` ← `Tree`, add `quasiquote`
`(`	`Apply`	add `Tree`, `Current` ← `Tree`
`,@`	`Quote`	add `Tree`, `Current` ← `Tree`, add `unquote-splicing`
`x`	`Value`	add `x` to `Current`
`)`	`Apply`	get `Parent`, `Current` ← `Parent`
`)`	`Apply`	get `Parent`, `Current` ← `Parent`
`)`	`Apply`	get `Parent`, `Current` ← `Parent`
END	END	END

You can notice that we do not have the final ) here, because the first ( was not the part of the tree. Yes, since the parse function wraps expressions with progn we could close it in the very end, but actually, we don’t have to. Because we do not store ) in a tree at all, those only serve as a marker that we should get the parent of the current expression, and only ( is part of the tree. To illustrate:

The expression above can be simplified down to (define list (lambda x x)), but I wanted to show how we deal with different quotes, while also having an expression that works as builtin list.

This conducts how reader is done in GRScheme, though I did not go into much depth and details, noting only important parts. For instance, our GRData will grow further when we will be discussing evaluation. All code can be found in GRScheme source repository.

Pairs in GRScheme

Although I want to point out some thoughts about cons-cells and this data structure, we’ve been talking about. Because the resulting underlying data structure of GRScheme is a tree (which hopefully will evolve into a graph), and not a binary tree there is only one possibility of having pairs, or cons-cells, which is emulation through syntax.

The first version of GRScheme supported this via the dot syntax, which is a usual syntax for pairs in Scheme: '(1 . 2). This was fully covered in previous post in detail, so here I would only say, that I’ve decided to drop pairs in favor of direct equality of code and underlying data structures because that’s what we love about Lisp (or at least I do).

This also means that there’s no real need for cons, because the underlying data structure is a tree of a vector of trees, and not a singly-linked list, which means that we can push trees to any side of the vector, and iterate over those without car and cdr. These procedures are still very usable, but their behavior can be altered to a certain extent to match the data structure a bit better.

For example cons will work as in Clojure, concatenating leaf with the tree, producing a new tree:

> (cons '(1) '(2 3))
'((1) 2 3)

However if cons is used to produce a pair, an error is signaled:

> (cons 1 2)
; error

In complement to cons an append tree may be added:

> (append '(1) '(2 3))
'(1 (2 3))
> (append '(1) 2)
'(1 2)
> (append 1 2)
; error

car and cdr will remain, but can be described through another procedures, such as get and slice:

> (car '(1 2 3 4))
1
> (cdr '(1 2 3 4))
'(2 3 4)
> (get 2 '(1 2 3 4))
3
> (slice 1 2 '(1 2 3 4))
'(2 3)

Where car may be a shorthand for (get 0 list) and cdr is (slice 1 -1 list) or something like that (I’m not sure about -1 part).

In the next post, I will write more about procedures, and their application to this data structure, which will touch on such topics, as scope, closures, lambdas, and iterative tree traversal.

Comment via email

On Scheme's Dots

Mon, 27 Jan 2020 13:22:00 +0300

This is the first public post that I wrote because I’ve finally thought that I’m clever enough to do so. Yet I must mention that I have never actually studied Programming Language Theory (PLT), so I’m not a specialist on the topic I gonna talk about here.

Year ago I decided that I should expand my knowledge in programming languages, and software development in general. I’ve already programmed for several years in C for bare metal, and in C++, as well as used some scripting languages, like Awk and Perl. But all these languages are mostly imperative styles, so I wanted something different.

I had a friend talk about Lisp, and he showed me this post by Paul Graham and I was inspired by the things that it describes. I’ve become interested in Lisp, so I got my digital copy of SICP and started reading it.

That was a year ago.

I’ve just finished chapter one.

Not that I’m stupid or something, but some exercises there are brutal. I remember that I set 3 days on one task where you have to complete two missing lines. It may not be that hard for others but certainly was very hard for me. So it took me a full year to go through the first chapter, but that only thing already changed how I look at the code today.

Additionally to SICP, I’ve studied Rust by reading TRPL. Because I wanted to learn both Rust and Scheme, I’ve decided that it will be a great exercise to write my own Scheme in Rust! Except, I’ve only finished the first chapter of SICP, and I don’t know how language works. And, actually, I don’t know anything about language design.

So how do you write a language in Rust, when you never studied PLT, and only programmed in C, barely know Rust, only toyed with different interpreted languages, finished the first chapter of SICP, and wrote some very basic scripts in Scheme during it?

Well, that’s a good question, I guess. You may think that I decided to read about Scheme and Lisp, something like this, but no, actually I’ve decided that I’ll just throw some code to the Scheme REPL and see how it behaves.

And that’s where the fun started.

Reverse-Engineering Scheme

I’m not a qualified reverse engineer, and this doesn’t actually have that much with real reverse engineering. I called it this way because the process of defining rules by behavior observation is somewhat related to reverse-engineering process, but does not go as deep as real reverse engineering does. I’ve taken two languages, one was GNU Guile, which implements R5RS standard, and the other was Racket, which is not strictly a Scheme, but a Programming-Language Programming Language, and started throwing different code in theirs REPLs. The observed behavior of such a concept as the dot is what motivated me to write this post.

The good thing about Scheme, and actually one of the main reasons why it is so powerful, is that it is very homoiconic, and code directly represents the underlying data structure, which is a list. Lisp itself stands for List Processor, and I never gave this enough thought until I started working on an implementation of a language. You take most of the information for granted (or at least I do) when it is about something you do not interact with too often.

So what is a list in Scheme? Well, anything that is enclosed inside parentheses. And we know that most everything is enclosed with parentheses in Lisp, so what’s the point of this information? Why everything should be a list anyway?

Because lists are easy to manipulate.

We need to know that Scheme has two main data structures: list and atom. Lists consist of atoms or other lists and can be destructed into individual atoms, and atoms can be combined into lists. For example, digits like 1, 2, 3, are atoms. The list of these digits is represented as '(1 2 3).

In Scheme, there are three primitive procedures that work with lists: cons, car, and cdr. Names may be not very representative, but that are historical names (and Lisp has a very long history) so just accept it for now. What do these procedures do? cons stands for construction, and it is used to construct lists. Here’s what you will see if you start Racket REPL and input this code:

> (cons 1 '(2 3))
'(1 2 3)

Note: I’ll be using Racket’s REPL format to represent things in this post because Racket’s default REPL is minimal and only consists of > character for prompt and output is typically on the next line, compared to Guile’s REPL which is more busy looking with all of its additional info. The Guile was taken as reference language though. But in terms of this text, Guile and Racket behave the same.

car retrieves the first element of the list and cdr returns all elements of the list after the first one:

> (car '(1 2 3))
1
> (cdr '(1 2 3))
'(2 3)

Combined with recursion and a functional style of programming we can traverse the list by taking out the first element, modifying it, and creating new lists, leaving the rest of the list for the next iteration. That’s may sound like a basic thing, but it is a really powerful concept because everything in Scheme is a list.

But I’ve actually lied to you. cons, car, and cdr doesn’t work with lists. Those work with pairs.

Like, what? What is a pair? Previously I’ve said everything is a list or atom, and now I say that there’s something other than that? Well, yes, pair is a pair of atoms, it is written as a list, but values are separated by the dot: (1 . 2). Pair is not a list, but a more fundamental thing - a cons cell.

> (cons 1 2)
'(1 . 2)

And lists actually consist of pairs.

If you have ever written a linked list in C or some language that has pointers, you know that any list consists of nodes. Node stores information and a pointer to the next node. If there’s no node after the node it stores a pointer to a special location. It may point to itself, or to void or somewhere else, so you could distinguish the end of the list and stop traversing it. How many nodes does a linked list have to have to be called a list? The answer is one, but without a pointer, to the next special node you can’t use it, so actually, a minimal list consists of two elements: the element with the data and the empty node or void. So that thing also can be called a pair.

Well, at least this can be thought that way. And this actually aligns with what happens in Scheme! Look:

> (cons 1 '())
'(1)

Because lists are essentially nested pairs, the last pair in the list has '() (empty list) as its second atom. We can nest cons to produce longer lists:

> (cons 1 (cons 2 (cons 3 '())))
'(1 2 3)

Why does this work? Because, given that cons produces a pair, if we evaluate the code ourselves, we will get this:

'(1 . (2 . (3 . ())))

Let’s explain why '(1 . (2 . (3 . ()))) evaluates to '(1 2 3). My observations lead me to think that dot has two rules. The rules are as follows:

Dot must be the second to last element in the list,
Dot can be removed if after the dot comes an open parenthesis, alongside with open parenthesis and matching closing parenthesis.

Because (cons 3 '()) produces a pair of dot separated values, and the first one is 3, and the other is '() we get the '(3 . ()) result (because of how quote works we remove ' from '() and get () in the resulting pair). Now let’s apply the dot’s second rule to this code. First let’s remove dot in the most nested example: '(3 . ()) becomes '(3), so now we have '(1 . (2 . (3))). We can repeat this to (2 . (3), producing (2 3). Repeating this the last dot produces the simple list: '(1 2 3).

Note: The rules above may not be an actual thing though, because the dot is just syntax on top of the internal data structure, and it’s not stored in the list, but we’ll talk about it later.

So why does car and cdr work on lists, if those are actually working on pairs? Because car removes the element before the first dot, and cdr returns everything after it!

> (car '(1 . (2 . (3 . ()))))
1 ;; 1 is before the `. (2 . (3 . ()))'
> (cdr '(1 . (2 . (3 . ()))))
'(2 . (3 . ())) ; or '(2 3) for short

So every time we see something like (a b c), the interpreter actually sees (a . (b . (c . ()))). That’s why car and cdr work both with pairs and lists. Because lists consist of pairs! And that’s really beautiful.

And then I saw pure beauty

Yet, before I fully understood what I saw, I thought that my beautiful observation is not correct at all. But first, we need to know how to define procedures (functions) in Scheme:

> (define (sum a b) (+ a b))

This constructs a procedure sum that accepts two arguments a and b and produces the sum of those as a result with the following body: (+ a b). We can call this procedure this:

> (sum 1 3)
4

Because I was figuring out the rules behind the dot, which I’ve stated before, I was wondering what would happen if I use a dot in this syntax, like this:

(define (foo . bar) bar)

Not only this did not produce any errors, but this also works in a strange way:

> (foo 22)
'(22)

And at first glance, this looks like nonsense (well at least to me, remember, I didn’t finish SICP yet) and I expected it to fail because dot seem to be in the wrong place but it worked just fine.

“What the hell?” - I’ve thought. I’ve tried to pass two arguments to the function and it still worked:

> (foo 1 2)
'(1 2)

At this point, I was thinking that everything I thought I understand is wrong, but after some long thinking process, I understood this and was shocked at how beautiful this thing is.

So what’s the problem here? I’ve shown you the syntax for defining sum procedure: (define (sum a b) (+ a b), but what’s actually going underneath is that anonymous procedure with arguments (a b) is being bound to the name sum like so:

(define sum (lambda (a b) (+a b))

We can write all our procedures this way, but it is tedious to do. So previous form with parentheses around sum a b is a shorthand for this form.

So what’s happening when we put the dot into the function definition? To understand this we have to understand why everything is a list, and why we need primitive operations on lists.

Note: The following part of this post mostly relies on what I’ve already done with my own implementation of the scheme-like language and how it works, but I think that the real scheme works somewhat similarly. I will write more about my language later in this post and in the following post that is yet to come.

To convert procedure from form (define (foo bar) bar) to (define foo (lambda (bar) bar)) we need to know two things: procedure’s name, to put after define and it’s arguments to put into lambda. So we car the list (foo bar) and receive foo. Now we need the argument list and we cdr the (foo bar), and get (bar). Now we can put foo after define and (bar) after lambda: (define foo (lambda (bar) bar).

What will happen if we cdr the list which has a dot?

> (cdr '(foo . bar))
'bar

This means that we getting bar without parentheses around it. So doing all previous steps for (define (foo . bar) bar) expression gives us (define foo (lambda bar bar)) where lambda has no parentheses around it’s argument. Let’s try to use such a form of lambda directly:

> ((lambda x x) 1)
'(1)
> ((lambda x x) 1 2)
'(1 2)

This is a procedure with variable argument count! Why? Let’s car and cdr this code!

First we evaluate (lambda x x), which provides #<procedure> in Racket:

(#procedure 1)

Now we can car what we will apply, and cdr it’s arguments:

(car (#procedure 1))
#procedure
(cdr (#procedure 1))
'(1)

So the arguments of the function are '(1). Let’s recap what our lambda does. It was defined as (lambda x x), which means that it takes 1 thing and returns it immediately. The thing it takes is this '(1), that’s why we receive '(1) if we call this lambda with 1.

Doing the same for ((lambda x x) 1 2) provides '(1 2) as applied argument.

Why this doesn’t work for lambdas which have their argument in the parentheses? My guess is that it is because we provide a list of a certain length, and since we’re passing a list, the items in one list get bound to the items in another, in the lambda body.

OK, let’s recap what we know about Scheme so far:

~~Everything is a list;~~
Scratch that, everything is a pair, and every list is a sequence of nested pairs, ending with an empty list;
pair is a cons cell, which uses a dot to separate elements;
car and cdr use dot to decide what to take out of the cons cell.

Applying the Concepts Above In the Interpreter.

When I started writing this post I had no idea that I still will be writing it after I’ll have a working interpreter prototype. It is located here, and it uses the concepts developed above to evaluate the code. It’s not very good though.

The whole language is implemented around these base functions:

parse - parses text into GRScheme data structure,
eval - which is recursively traverses the tree,
apply - which applies the procedure to the tree,
get_first_subexpression - which takes first subtree from current tree,
get_rest_subexpression - which takes the rest subtrees as a tree.

Other functions in evaluator source code provide builtin facilities like car, +, lambda, define etc.

The first thing I have to say is that GRScheme doesn’t have such a thing as a dot. It is emulated to behave the same way as in a real Scheme. I suppose that regular Scheme does not have a dot either but in another sense.

After parsing the code we receive a tree structure that has nodes with data, and the tree directly represents the original code. This allows me to think about the Scheme code in a very native way, because it is already a tree, and it actually simplifies parsing a lot. I store parentheses as tree items, atoms are also tree items, but typically only parentheses have child nodes. This allows later traversing algorithms to check if the current node is a procedure call by just checking if the current node is an open parenthesis.

The get_first_subexpression and get_rest_subexpression functions are not exactly car and cdr as it may seem, but a more general ones that operate directly on the program AST. But because the dot is emulated in my implementation those currently also check for the . in the tree. If it is presented, get_rest_subexpression returns the subtrees without it.

This allows us to destruct pairs and lists with these two functions, but we can’t use those directly. Instead car and cdr are implemented in terms of these functions, also applying quoting on the result, but that’s a topic for a different post.

I’ve mentioned that dot is emulated in my implementation of the interpreter. What this means is that I store the dot directly inside the tree as a node. Let’s have a look at how '(a . b) looks in my language’s data structure:

Figure 1: '(a . b) Pair representation in GRScheme

Note: because I write quoted lists in the short form, there is a possibility of misunderstanding why the tree in the figures doesn’t directly match the input, e.g. has more parentheses. For instance '(a . b) is actually (quote (a . b)) and it is represented in the tree this way.

We have a tree, with a root node holding ( value, which has two nodes: quote and open parenthesis. This second ( has three nodes, which are a, ., and b.

In a real Scheme, this isn’t stored this way. Because pair is really a pair in Scheme, the cons cell actually stores two pointers to the data like this:

Figure 2: '(a . b) Pair in Scheme

The lists are a bit different. In my implementation, the structure to which I parse the source code is a tree with any amount of child nodes per node, as seen in the pair example above That’s how '(1 2 3) list is stored in my implementation:

Figure 3: '(1 2 3) List in GRScheme

This is a tree, which is similar to the pair one, except instead of the dot we have 2. Let’s write list as a sequence of pairs '(1 . (2 . (3 . ()))) and see how it will be stored in my implementation:

Figure 4: Dotted list in GRScheme

We can see, that each subtree has exactly three nodes in it. However in a real scheme, it’s two nodes per tree, and the very same sequence of nested pairs is stored like this:

Figure 5: '(1 2 3) List in Scheme

Because any list can be written as a sequence of pairs, we can see that any list in a real scheme is actually a binary tree. Not in the case of my language though.

The parser that I’ve written for GRScheme knows how to deal with dots, thanks to the dot rules I’ve described above, and these two constructions are parsed to exactly the same form:

Figure 6: Both notations in GRScheme

And lists that don’t end with empty list, like '(1 2 . 3) and '(1 . (2 . 3)) are parsed like this:

Figure 7: List with pair at the end in GRScheme

Which, in a real Scheme, will look like this:

Figure 8: List with pair at the end in Scheme

So in the case of car in a real Scheme, we get 1, and in the case of cdr, we get the subtree '(2 . 3) when we apply it to this list and 3 when we apply it again to the previous result. This works the same way in my implementation, because we analyze the tree when we do cdr to see if there’s a dot in the second position, which may not be optimal, but works.

Dot in GRScheme

I’ve thought about dots during this post, and I think it is a very beautiful syntax solution to describe binary trees that do not end with a null pointer. However, this also means that dots are actually needed only for that case. Though we can emulate dots by storing those inside a tree directly, I don’t think that this is a good idea to do so.

Because the underlying structure in GRScheme is a graph, and not a binary tree, I think pairs are also meaningless, because they describe a particular tree case in Schemes, and in GRScheme we do not have this case. So one major difference from the real Scheme is that in GRScheme there are no cons cells, and some algorithms may depend on extracting the second element from the cons cell directly.

I’ve thought that it may be good to have a special procedure for that case, named along with car and cdr - Contents of Cons cell Representation. ccr should return the second element directly if it is the last element in the list, and return the rest of the list otherwise. This way it will work mostly the same as cdr but cdr always returns the list. I’m not sure how good this solution is, because this may be ambiguous when to use such a procedure, so there’s another procedure that may be used instead, which is cadr, which always takes the second element from the list.

I’m planning to write a second post about GRScheme diving more deeply into the details behind the language model I’ve developed. The current implementation is slightly wrong and has many problems, so I have to fix those first.

Comment via email

Andrey Listopadov

GRScheme Design Part 3 - Quasiquoting

Quasiquoting

Filtering arguments for quasiquote

Unquote procedure

Unquote-splicing procedure

GRScheme complete?

GRScheme Design Part 2 - Evaluation

Evaluator structure

Function eval

Name evaluation

List and Symbol evaluation

Evaluation of a Procedure

Apply procedure to arguments

Builtin lambda procedure

Application of lambda procedure

Tail call elimination example

Quasiquoting, Unquoting, and Unquote Splicing

GRScheme Design Part 1 - Parsing

Creating the Tree in Rust

Writing reader

Pairs in GRScheme

On Scheme's Dots

Reverse-Engineering Scheme

And then I saw pure beauty

Applying the Concepts Above In the Interpreter.

Dot in GRScheme

Filtering arguments for `quasiquote`

`Unquote` procedure

`Unquote-splicing` procedure

`Evaluator` structure

Function `eval`

`Name` evaluation

`List` and `Symbol` evaluation

Evaluation of a `Procedure`

`Apply` procedure to arguments

Builtin `lambda` procedure

Application of `lambda` procedure

Creating the `Tree` in Rust

Writing `reader`