A programming system

Not to be confused with a programming language.

In this post, I would like to cover what features I think a dynamic language environment should have. Or to rephrase that, what would the environment probably have if I were to design it.

I like small systems - when something is small and comprehensible, it’s easier to reason about its use. That’s partly the reason I like languages like Lua. And working with Lua actually motivated me to start thinking about writing my own bytecode interpreter. But more on that later.

When it comes to programming languages and their environments, the situation usually isn’t great. A lot of quirks exist, mostly due to the time period the system has been designed, or because the original scope of the system shifted and now the legacy design decisions show. For most people, these things are tolerable or already unnoticeable - some even go as far as to defend quirks of the system, as if it was a feature. Like weak typing in JavaScript, allowing one to add two empty arrays, and get an empty string as a result. But I digress.

I must say, there are many compelling systems in the programming field. By systems, I mostly mean the language runtime or a virtual machine/bytecode interpreter it hosts on, but in actuality it’s a bit more than that. Yes, each system almost always comes with some first-party language, think JVM and Java, BEAM and Erlang, often tightly coupled together, making it hard to separate the two, but not always. In fact, the systems I’ve mentioned are not tightly coupled at all!

So let’s dive in. I’ll briefly list systems I know enough to have some sort of opinion on. Don’t take me too seriously, though, as I’m not taught in compiler design, and only read a few books on the subject. If anything I say here is wrong just reach me out, and we’ll see if I can fix the text, or just remove it adding a note.

Scheme

If you read my blog regularly, you know I love lisps. Can’t say so for Scheme. Weird, right? That’s mostly related to the language itself, and this topic focuses on the other side of things.

Scheme is a small system. It is standardized, and the standard itself is not very big. This is both an upside and a downside - because the standard is easy to implement, there are IMO too many implementations. Each implementation is slightly different, and not always compatible, which makes it harder to choose one.

For example, I don’t want to commit to, say, Guile, and much later find that in retrospect Chez was a better choice. Or Racket, or Chicken, or Gambit, or MIT Scheme, or s7, or Cyclone, or Microscheme, or Loko Scheme, or Kawa, or whatever. Even I have a Scheme-like language, though it has nothing to do with the standard, and I would not consider it a real Scheme. Still, some code can be just copied to it, and it will hopefully work.

Though, even though I just said that the topic of the post isn’t about language itself, I feel like getting this out of my system. The other problem I have with Scheme as a language is that its default feature set is just weird. I mean, the language that processes lists doesn’t feature most list processing functions out of the box - it’s a part of SRFI-1. Yes, it’s as hard as doing (use-modules (srfi srfi-1)) but why a list library is in a SRFI is beyond me. But I digress, again.

So, yeah, not a huge fan of Scheme the language, but the Guile VM is pretty neat! It’s a compiler tower-based system, meaning that you can implement other languages for Guile virtual machine if you define your language to compile to the one supported by the compiler tower. For example, there are implementations for Lua and ECMAScript. And just recently there were news that the Guile got a WASM backend! Which is great, and as far as I understand, because of how Guile is structured, other languages implemented for higher levels of the tower will be able to compile to WASM as well. You can read more on the whole compilation process here.

Other important characteristics of Guile is that it is dynamic, like most other lisps, and that it supports continuations and delimited continuations, which are a great feature that can be used to implement all sorts of control flow operators. Additionally, Guile now has a register-based VM, which makes it generally faster than other bytecode interpreters because less bytecode instructions are required. But let’s move on to other lisp system.

Lisp

Same as Scheme, it’s a standardized language, albeit its standard is much bigger than Scheme’s. And in terms of system complexity the situation is similar - Lisp is bigger, yet not as complex as many other systems.

As for its implementations, there’s Common Lisp, which many lispers refer to as just Lisp, and they don’t like when other languages with parentheses around the calls are called lisps. And same as with Scheme, there are many implementations of Common Lisp - SBCL, CCL, ECL, ABCL, ClozureCL, and others. I’m a bit familiar with Common Lisp on SBCL, and toyed with the language enough to know some of its quirks and differences with other implementations. That is, I’m not an expert, far from it, so take my words with a grain of salt. The full Common Lisp system is hard to grasp, but it’s been ironed over time a lot, and it is comprehensible thanks to the standard.

What I like about Common Lisp systems is that they can work with Lisp images. Creating an image means that you tell the Lisp system to create an image of its current state, write it to the disk, and then later you can restore it, continuing like nothing happened.

I should have mentioned that I’m not going to go into every system listing all of its features - instead, I’m going to list ones that I think are important and make the difference.

Another such component of Common Lisp are conditions. I’m not sure if they’re implemented in the VM, but probably it is how it’s done, as they have to be performant. Conditions are like exceptions, except they’re three-part instead of two-part.

There’s another widely used Lisp system, perhaps even more than Common Lisp - Emacs. I guess it’s not a fair to compare a text editor to a Lisp runtime, but Emacs is not an editor, so we’re good. Some would argue that Emacs Lisp isn’t cut to be used as a general-purpose language, but there were large systems, entirely unrelated to Emacs, written in it, so it can be used in this way. Though many, probably, will think that you’re crazy for doing that.

Honestly, there aren’t many unique properties of Emacs Lisp I know of, in this regard it is pretty generic. But I still wanted to bring it up, as Emacs itself is a perfect example of a programming system. There’s a saying that Emacs is a great operating system, lacking only a decent editor, and while it’s a joke I don’t think it is true - the editor is pretty great.

Emacs is almost everything I could ask for when it comes to a system that I can program. I have a post describing my take on a possible GUI library for the Emacs system, but apart from that, Emacs is very customizable and programmable. I think the main reason for that is not the fact that it is a dynamic system, but that it has such feature as advice. Advises in Emacs allow anyone to redefine or wrap any function, making it possible to patch existing code without modifying it. I know that Common Lisp has a similar thing, but I think in Emacs it is much more common to do it than in other large Lisp applications.

JVM

Unpopular opinion - JVM is good. Like, really good. JVM is fast, portable, and scales well. And the library ecosystem is HUGE.

Contrary to popular belief because nearly all mainstream JVM-based languages are statically typed - the JVM itself is dynamic, which makes possible languages like Clojure to exist on it alongside with Java or Scala. And it’s a system that is known for its backward compatibility guarantees. I think that’s the main reason why Clojure compiler can generate bytecode, instead of translating Clojure to Java and then using JVM to compile it, like many languages targeting other platforms do nowadays.

And I’m not sure if there are many other systems that allow for that. Elixir seems to be compiled to BEAM bytecode, but I can’t actually find if it is really done this way. For example, nearly all languages that target the Lua VM just compile to Lua, not to the Lua bytecode - perhaps because Lua bytecode is not compatible between different releases.

I like JVM for its promise, but on the other hand it’s pretty huge. Slimmed down versions of JVM existed, but it’s not the same, as just being small and embeddable, which I also value greatly. Still, the capabilities JVM provide are great - it has a lot of state-of-the-art technology in it, making it viable as a runtime for high performance high throughput applications.

Maybe JVM isn’t everyone’s cup of tea, but I think it’s because Java isn’t - the hate just extends from the language to its runtime, which is unfortunate. But there are lots of languages that run on the JVM, and I think it’s a good thing and what any system should aim at if it wants to succeed.

BEAM

Erlang is another language I think is great, and the VM it runs on is incredible.

BEAM is a highly dynamic, yet fault-tolerant system with a lot of work dedicated for it to remain stable under high load of tasks. It was mostly developed for telecom, but it can be used for any scalable backend tasks in general today.

I would like to mention Prolog here too, although it’s a bit unfair, since Erlang pretty much rose from Prolog, but I’m just not familiar with Prolog enough. As a system, it is probably more generic than Erlang, and I like that it has graph querying and manipulating functions in it, as well as many other capabilities.

One of the main things I like about Erlang’s virtual machine is its ability to recover from errors. As described in the amazing Erlang The Movie II: The Sequel if your code crashes for one of the running threads, none other threads are affected by that. Then, it’s possible to fix the bug, recompile the code, and load it into the running system - fixing the failed thread. And all other threads will eventually pick the newer version. How cool is that?

Lua

Lua has a really neat and small bytecode VM, and one of the fastest JIT implementations across many similar languages. As far as I know, this is because of a combination of things. First, Lua uses a register-based VM, and LuaJIT implemented a tracing JIT for it, basically compiling many Lua register operations into real hardware register operations. Second, the instruction set of Lua is itself close to machine instructions, making it easier to implement JIT.

Lua VM itself is very nice and easy to understand - if you have moderate knowledge of C you can probably read and understand the source code of Lua in a weekend. It’s also highly dynamic, and really simple. There are some downsides to that - inbuilt data structures are not the greatest, and there are some wonky parts about multiple value return. Other than that, it’s very expressive, and you can bend it pretty much however you want.

One feature of the Lua VM that I absolutely love is its implementation of coroutines. Functions can be paused, inspected, and resumed at any time, which makes implementing asynchronous programming quite easy. I’ve written a few libraries for pure Lua async, and I think it is beautiful how coroutines provide an elegant solution to this problem.

Another great feature is that you can call C functions from Lua. FFI is something that many languages do, and I’m only talking about it in the context of Lua because that’s the only language I’ve used it firsthand.

Lastly, the important thing about Lua is that it can be used both as a standalone VM, and embedded into the application. Numerous applications embed Lua in different ways to allow scripting and extending the application. I think it’s a good characteristic of the VM, but maybe it isn’t a good fit for a programming system.

Smalltalk

Finally, Smalltalk. It’s one of the greatest systems, and much like Common Lisp, it is image based. Unlike Lisp though, you do load an image and work in it, instead of loading the source code.

Smalltalk is definitively on my list of languages to dive into, but as with Scheme, it is a bit hard to start because of all these different implementations. I’m looking at Glamorous Toolkit, but I’m also in fear that it might shatter my love for Emacs. Probably not, but who knows.

Anyway, Smalltalk is a very dynamic system, with a lot of introspection capabilities, which I like a lot. Much like Emacs, because you’re interacting with the system directly, a lot of things are possible, and I feel that more systems should allow for that.

Other systems

You may note that I’ve not listed many other systems, like CLR, OCaml, WASM or even Python. I don’t know anything about CLR other than “JVM but Microsoft”, haven’t really tried OCaml, and WASM is too new for me to get into it yet. And I despise Python as a language, so I just don’t wanna bring it to the discussion.

If you think I should look into any of these systems (except Python) for interesting features - let me know!

My ideal Programming System

I would try to introduce this term here - a Programming System. It’s not just a language, like, say Python or Lua, instead it is more like Emacs or Smalltalk.

So let’s take the best from all VMs and Systems and compose a list of things I want to see in my dream system. Note, the list is not about the language features, this will be covered separately in the future (maybe). I will mark each point with the list of systems that I think handle this well.

Now, in no particular order:

• Garbage Collection (JVM, Lisp, Smalltalk, BEAM)

  Duh.

  In all seriousness though, being able to write code without thinking about memory much is one of the reasons why VMs became popular, so I feel that garbage collection is a huge benefit. There are many approaches for garbage collection, and JVM continues to improve them even in situations where heaps are measured in terabytes.

  Manual memory management should be possible still, for implementing low level stuff, as there are clear benefits to that approach, and thus the system should be able to mix both approaches seamlessly. Maybe even go as far as allowing to say something like - I’ve acquired these resources myself, but due to the circumstances I don’t care about them anymore, you’re free to collect them in any manner.

• Dynamism all the way down (JVM, Lisp, Smalltalk)

  I see no reason for the system itself to require static typing. Look at JVM - Java does static typing. Kotlin does it too. Scala really does it. Yet JVM itself is dynamic and everyone is still happy.

  Static types should be only a language-level feature, not system-level. The system itself should be dynamic.

• Dynamic updating mechanism (BEAM, Lisp)

  It should be possible to load code into an already running system without requiring restarting or stopping the system. This includes extending the program with new features while it is running, adding dependencies as you work, re-loading code, and graceful error handling.

• Image storing and loading capabilities (Smalltalk, Lisp)

  The ability to dump your whole running system to disk and then resume it, maybe even on a different machine, is one of the greatest features a system can have. It allows for much better debugging, being able to re-re-run things as you need to, and experiment.

  This feature actually has more purposes than just being able to store state and resume from it. Being able to take a single thread and write it to the disk to resume it later means we can make snapshots of every worker in the system and restore them even if we get a power outage, for example. There are projects that do that, but without an explicit support from the platform it may not always be robust or could come with a lot of additional costs.

• Fast primitive data structures (JVM)

  The system should provide non-resizeable tuples (arrays), (preferably) immutable (and maybe interned) strings, and records. It should also provide a way to be able to destructure these data structures fast, like unpacking an array onto a stack, but we’ll talk about this later.

  I don’t think classes should be on that low level, though. Classes are just like records, except with some additional semantics.

• Lightweight threads/Coroutines/Continuations (BEAM, Lua, Scheme, recently JVM)

  In our modern world, asynchronous programming is a must for a system to be viable. Properly utilizing a single core, and allowing for horizontal scaling, is important for lots of scenarios. Be it a particle simulation, or a web server handling millions of requests, the system should have a robust mechanism for that.

  There are multiple ways of doing this, each with upsides and downsides, my personal favorite is stackful coroutines, but your opinion may vary. First-class continuations are another way of achieving it, and perhaps a bit more bare than coroutines.

  I should also include asynchronous IO here, but it is largely system-dependent.

• Multi-threading (JVM)

  Just as important is to be able to utilize multiple cores. Maybe even tip into GPU utilization.

• Access to vector instructions and multimedia instructions (SIMD)

  Another big performance gain, pretty much required today by many algorithms.

• JIT compilation (Lisp, JVM, LuaJIT)

  If possible, the system should compile everything to machine code, like Common Lisp implementations such as SBCL do. However, JIT compilation can also be a good compromise, as I imagine it is easier to just port the bytecode interpreter to some new architecture first, and add JIT later, rather than implement a compiler for it from the start. I like this property of Common Lisp, and it certainly makes things run faster, but even pure bytecode interpreters can achieve good performance if implemented correctly, and with JIT they often have comparable speed to native code.

• Backward compatibility (JVM)

  No bytecode operations should change their meaning, or be removed. If that’s not the case, implementing a compiler to bytecode (or other kind of IR) is a no-go for other languages.

• FFI (Lua, JVM)

  It’s hard to ignore the outside world, even if you’re making your own world for programs to live in. Talking to sockets, implementing efficient modules or simply extracting the language with new data structures is important, and isn’t often possible to do in the VM-native way. Many VMs feature FFI for calling C, and in this regard, I have nothing to say that maybe C is not the best choice for that, but we’re pretty much stuck with it. Maybe Zig FFI would be better? We’ll see.

• Optimized message passing (???)

  I don’t think there’s a system that can do it yet, or at least I haven’t heard of one. Similarly to Lisp’s images, I think it is an interesting idea to explore - using images for message passing. Maybe BEAM does it, I don’t know.

  When you need to send a message to another thread it is easy, you just share it via the memory, or just do a copy. Which of course produces all kinds of interesting problems with shared state. When you need to send a message to another process, however, it often involves serialization and deserialization, and it is largely a waste because ultimately serialization is only needed to send the data over the wire. Additionally, it creates barriers on what is ultimately possible. E.g. you can send functions over the wire, if your runtime supports dynamic evaluation of code, and you can somehow serialize functions, but you can’t really send a function that has a closure - because there’s no way to serialize it properly.

  So instead, what if we were to send the actual memory blob to the process, and it would map it directly onto its own memory? I’m not sure if this is possible, but if it is, it would allow to actually skip serialization and start sending objects around without a cost of a string parsing and evaluation. And it would also mean, that if someone to implement object memory layout in other language they would be able to send objects to such system directly as well. Sounds kind of like a binary protocol, I guess, but for memory.

This is a feature set of a VM I think I would pick when implementing one. I know there were a lot of text, so let’s wrap it up in a shorter list:

• Register-based VM with small instruction set, and stable bytecode.
  • Maybe use a compiler-tower approach.
• Support for different garbage collection strategies.
• Dynamic module and code loading support.
• Image (re)storing capabilities.
• Multi-threading capabilities and stackful coroutines.
• JIT/Compilation to native code.
• Access to vector instructions of the underlying machine with JIT.
• FFI.
• Fast primitive data structures to build upon.

I would like to list the last two in a separate list:

• Embeddability - not as high of a priority, but nice to have and helps keeping project small enough.
• Optimized message passing - needs lots of researching.

If we put this into a table, we can actually see what Systems that I’ve mentioned match my criteria:

Where:

• ❓ - I don’t know, or couldn’t find any info;
• ❌ - not supported (doesn’t always mean bad, though);
• ✅ - supported in one way or another.

Now, again, if I got anything of this wrong, which I probably did, reach me, and I’ll change that. Same if you know about any cells marked with ❓.

Additionally, if you think that I should have picked a different implementation of some of these systems, like Chez instead of Guile, or Squeak instead of Pharo, feel free to send me info on them, and I’ll include those to the table. Obviously, I can’t research all of them.

Ideal Programming System

I know, it is impossible to implement a perfect system. But if you look at the chart, some are actually extremely close to that, at least in my view. In particular, Guile, JVM and BEAM are great.

And despite the fact that Emacs has a lot more ❌ than others, it’s still a better programming system than many. Smalltalk is also a better programming system than many - because it was built to be a programmable system, much like Emacs. So is the chart flawed?

Well, yes and no. It’s just more complicated than that, as I hope I’ve shown before the chart when I was talking about benefits of each system. Some do some things better than others. Emacs isn’t a JVM, yet it managed an airport once.

There’s another problem, though. Software today is a hot topic, and many new languages are created every year. Most of these languages are just rehashing of existing ones, with some small changes, and almost no actual improvements. The same can’t be said for VMs, but many such languages feature their own VM, like for example Janet¹.

Occasionally, there are projects like Clojure, where the author with a great experience looked on programming, figured out what problems are currently unsolved, and tried to solve them. The result is one of the best languages I’ve personally used. Running on one of the best virtual machines. Sure, it can probably be done better, there are still a lot of hiccups here and there, but it is an improvement over what we had, especially for 2010s.

And while I’m a bit skeptical on this, but in the eyes of many, Elixir is another one such case. Taking a great technology of BEAM and implementing a friendlier version of Erlang is a huge step-up for many programmers who don’t want to work with Prolog-like syntax.

Then again, projects like Zig feel like a major improvement on what we have in C, so it’s not like I’m saying that people should stop working on new languages and platforms. I’m just saying that there should be enough of improvements on real problems.

Am I listing real problems, though? I would say yes but, really, I don’t know. For me, all capabilities I’ve listed would prove useful in my work - JVM’s lack of coroutines (I use older JVM) or image (re)storing is often a problem for me. SIMD is needed less often, but I really wouldn’t mind it, because it can be used to substantially speed up regular expression engines, or parsers. Register-based VM isn’t a hard requirement, as long as VM is fast, and JVM is, so it’s not big of a deal here. Though JVM is not exactly a programming system either. Then again, if I were to work in Lisp or Smalltalk, then I probably would have another set of complaints - so there’s probably no definitive answer to the question.

Would I work on such a programming system myself? Probably yes, but I’m far from it knowledge-wise. Again, I’ve read some books on the topic, but I need to learn much more. Perhaps I should take a compiler course too. It is a very hard area of programming, and probably people who are knowledgeable in this are offended by this article, so sorry for that!

Currently, I’m working through the second part of the Crafting Interpreters book in Zig. It is an interesting book, when I was researching the topic it felt like a good entry point to the field. Perhaps a dragon book should be next? Let me know if you have good literature recommendations on the matter, it’s always appreciated.

Anyhow, I hope this was entertaining. Thank you for reading!