Emacs GUI library

Lately, my Magit buffer broke once again because of something weird going on with major mode, and I couldn’t stash or commit hunks unless the point was at the beginning of the line. That once again reminded me that Emacs UI is not really a UI, all of it is mere text with a bunch of properties slapped on top. It’s easy to forget about it, and use such things like Treemacs, Customize, heck, even Org Mode, thinking that you’re interacting with UI elements (file nodes in Treemacs, text-boxes in Customize, Org Modes sub-trees and drawers). However, it is an illusion, though neatly crafted. It’s still text.

Well, what’s bad about it you might ask? We love to manipulate text, Emacs even makes it actually comfortable to do, with a lot of functions dedicated to it. And buffers are also a very fast way to manipulate even moderate amounts of text. But ask yourself, do you really want to do the same kind of thing in a different UI system, outside of Emacs? Maybe if it came with the same amount of useful functions? Well, maybe you’d want to, but I sure don’t.

I’ve made my own custom UI elements in Kakoune, making a file tree, and tag list plugins. That was before I actually started using Emacs as my main environment, and Kakoune text properties are much less sophisticated than ones of Emacs, but… It’s the same thing, really.

You programmatically spit some text into the buffer, then you parse it, find elements, walk on them, change them, and so on. Yes, text properties, markers, overlays, all of it makes it more robust in Emacs, but even though, things still break. Because Emacs still tries to interpret stiff in your buffer as text.

Because that’s what it does, it expects text! If you call forward-sexp on some Magit UI element it will try to do the right thing, even if there’s no right thing - there are no sexps in the Magit buffer. Well, maybe there are if you’re viewing diffs of your lisp code, and then yeah, the benefit of everything being text you can move on becomes obvious. Except when we’re talking about UI it’s not working more of the times than it is useful in an occasional context of that UI element.

I think you probably get where I’m going with this. And you probably don’t like it, but bear with me. And it’s an angry bear, so don’t threaten me more than he does, please.

What I’m trying to say is that Emacs, as a GUI application isn’t something that I’d be advertising to other people. You can make Emacs look beautiful, no doubt about that, Nano Emacs, DOOM, and others try very hard to make Emacs visually appealing. But it’s all too janky IMO. Nano Emacs looks great on the pictures, but resize a window and the illusion of a coherent interface breaks fast. So let’s look at different systems for inspiration:

Smalltalk

I’ve already mentioned Smalltalk in the past regarding comparison with Emacs in the GUI field. And I think it is a fair thing to do - both are dynamic systems that you can program with their respective language. Except, one is GUI oriented pretty much from the start, and the other one is Emacs.

Look at how Smalltalk looks when you fire up Pharo:

It’s a GUI system. Complete with windows, containing text editors, browsers, context-aware menus, and so on. Maybe it doesn’t look as pretty as your preferred GUI toolkit, but it’s not the main point.

Although, here’s how Smalltalk looks when you fire up GToolkit, which is written in Pharo:

There are much more pictures on the official web page.

The point of it is that you can build your own tools for your tasks in this toolkit, as shown in the pictures. They call it building an inspector, and GToolkit comes with a lot of inspectors for various kinds of data.

This is my favorite system so far, and it gets worse from here but I feel that we need to talk about other examples still.

Java Clojure

Why not? Clojure is another dynamic system, that you can change at runtime. While Java isn’t what comes to mind when speaking about beautiful and functional GUI, making GUI apps in Clojure is far more fun than it is in Java. Because again, you can update your UI elements on the fly.

There are some examples of apps where the UI is done in Clojure, one example is the game engine called Defold.

It looks like this:

As far as I understand, its Editor is made in Clojure, and UI is handled via the JavaFX toolkit.

Intellij Idea is many another example of a decent-looking GUI in Java, although your opinion may vary.

Still, JVM isn’t the greatest at UI, and somehow another language with a J in the name managed to do somewhat better.

Web

Lastly, the web is another way of building user interfaces, one that is quite popular today. Because the web works the same across a majority of operating systems, as browsers are ported to them with a lot of care. Browsers are also extremely capable of rendering all kinds of file formats, doing interactive stuff, and JavaScript frameworks are aimed at providing facilities for building UI that is highly responsive and interactive.

The web is however associated with slow, often unresponsive UI, a lot of wasted screen space, and unnecessary high CPU and memory usage. Mostly due to Electron packing lots of stuff. I think Electron wasn’t a bad idea, the execution was just poor. There are other projects that provide similar facilities like Ultralight, but I didn’t investigate further. I’m OK with the idea of building UI on web technologies, as long as we’re making a robust system that works across multiple devices and actually benefits from using the browser.

There’s also this project, which I find cool because it looks similar to GToolkit: natto.dev. Basically, you get an infinite canvas, on which you can place text editing widgets, video widgets, and other stuff. Keep this thing in mind, I find the idea of such UI interesting, and probably it will be explored more in the future.

But, let’s get back to Emacs, and I’ll try to explain why I was bragging about all these other systems.

Emacs

Emacs has a windowing system. Basically like a tiling window manager - you have a frame, you can divide it into windows, you can spawn other frames, you can divide them, and so on. But, these windows can only display basic text, among some graphical elements, like SVG graphics, or bitmaps. Needless to say, when compared to any of the systems I’ve talked about before the GUI capabilities of Emacs are quite limited.

Basically, Emacs is a glorified terminal, with ability to draw different font sizes and decent image support. Don’t make such a face, you know it’s not far from the truth. Again, this may be a bit harsh, but I believe we need to admit things in order to move forward.

Speaking of terminals, Emacs can actually run in a terminal! You can start it with emacs -nw and it will happily transform your terminal into Emacs. I mean, you no longer have a terminal, it just acts as a rendering engine for Emacs. So you basically get the same thing as GUI Emacs but worse, because now your terminal is responsible for rendering fonts, and you no longer have any image support. Well, there are terminals that support images, but come on.

And I think terminal support is one of the things that holds Emacs back. Think about it - why do you even need a terminal if you’re already using Emacs?

The common misconception is that you need a terminal version of Emacs to run it on headless environments, like servers, clouds, etc. You don’t. I work with remote servers constantly each day and I never open Emacs on the remote when I’m editing files there, I don’t even open a terminal for the SSH. Here are some basic workflows involving remote servers or just terminal tasks:

• Need to ssh somewhere to edit files or run some commands?
  • TRAMP got you covered.
    • You can even run async-shell-command in a TRAMP buffer and it will be run on the remote.
    • You even get to retain the entirety of your config without copying it onto the servers.
• Need to run some commands in general?
  • Again, You can call async-shell-command, it even will handle your input and password prompts in case there’s one;
  • Or you can run compile, which doesn’t handle prompts but still is much more interactive than the raw terminal output.
• Want to run some complex commands with editing convenience, or maybe even some TUIs?
  • You can run eshell, and go nuts on scripting in Emacs Lisp and calling sh scripts from the Emacs shell;
  • Or you can run a proper terminal emulator from within Emacs:
    • Vterm - a terminal based on libvterm C library;
    • EAT - a terminal emulator that can run in a region, in a buffer, and in Eshell, written in Emacs Lisp;
    • ansi-term, term - inbuilt terminal emulators in Emacs.

I don’t know how many times I got into a situation, where there’s a PDF documentation in one buffer, TRAMP in another, and I’m editing a remote file while referring to that documentation and staying within the comfort of Emacs. So no, with Emacs terminal itself isn’t needed in maybe 90% of situations, and terminal Emacs is just out of the question.

But hey, I too previously was a terminal enjoyer before Emacs, and it was too hard to even imagine how to even approach work without a terminal. As I’ve mentioned, I was using Kakoune, and prior to that, I was a Vim user. Now, with Emacs, I rarely open the terminal at all. There’s simply no need to, as there are more comfortable ways of interacting with the shell. And damn, even things like VSCode, which isn’t a terminal app at all, still manage to run as a server in a headless/remote/container context and provide you with all of its GUI features by acting as a client. So yeah, ditch the terminal version of Emacs, you’re ain’t gonna miss it.

Except, don’t. Use what you like, the solution I’m going to propose doesn’t require anything to be ditched at all. But before that, let’s talk XWidgets

XWidgets

When I was talking about the web, I’m sure some of you thought about XWidgets and Webkit support in Emacs. Like, here, I’m running the already mentioned natto.dev inside my Emacs right now:

So what gives? And at work, I occasionally use Portal to navigate through nested data structures while debugging Clojure code:

Again, what gives? This looks like we can have interactive GUIs in Emacs, much like in GToolkit and Pharo. However, there’s a catch. There’s always a catch, and in the case of Emacs, it’s that the XWidget UIs are foreign to Emacs.

Here, foreign means that inside that *xwidget-webkit: portal...* buffer nothing you expect to work will actually work. Remember how I mentioned that in Magit if you call forward-sexp it will try to do the right thing? Well, here it can’t do anything, because this isn’t a buffer really. It is an embedded widget that only looks like a buffer in the Emacs window, but it’s not.

There’s no point in calling forward-sexp because there’s literally no point in such a buffer. You can’t use Avy to jump to an element in such a buffer because there are no textual elements that Avy can look at.

Because of it, it is basically the same as if you’ve opened a separate web browser side by side with Emacs and used it to browse the web, or run Portal, and it would probably even work better. Unfortunately, we can’t embed something like Webkit into Emacs and hope that it will solve our UI problems. Or can we?

Emacs GUI Toolkit

So here’s my idea. I’m pretty sure it’s not new, but that’s not the point of this blog - it is merely a container for my thoughts, original or not.

What if, instead of embedding Webkit into Emacs, we turn this around? Hear me out.

When I was talking about opening a browser side by side with Emacs two seconds ago, I should have mentioned that window managers, or WMs for short, are quite good at placing windows. If you’re using some kind of a tiling manager, you know what I mean - you can leave window management to your window manager. Kakoune, for instance, does exactly that - instead of creating its own windowing system, like Vim or Emacs, it leaves it out for things like TMUX or i3 to solve. This, of course, has some problems too, but works well enough in general.

But I’m not saying that we should ditch windowing from Emacs and leave it to the WM, far from it. I would rather ditch frames from Emacs if you ask me, but that’s another topic.

Look at this as if Emacs was embedded into your window manager, which is basically how it works. You start your WM of choice¹, you launch Emacs, and as long as the Emacs window is focused, you live in Emacs. All of your shortcuts work, and all of your navigation habits are there.

So what if we expand on this idea? Instead of embedding Webkit, let’s embed Emacs in its current state as a UI widget.

The Widget Library

What I’m thinking is, let’s take, say SDL2 (or maybe something like ImGui), and create a library of widgets for Emacs. We’ll then integrate Emacs into it as the runtime system behind it, much like in the case of Smalltalk, JavaFX+Clojure, and Web+JS, and provide a set of functions to navigate through widgets in a meaningful way. The widgets would include basic stuff, like buttons, text boxes, buffers, drawers, menus… wait, buffers?

Yes, why ditch something that works well? Like when you focus Emacs in your WM and it just works, when the buffer widget is in focus it will just work too! We can keep the redisplay code, it will just need to learn how to render into such widgets, which I’m sure is possible, as there’s a very similar feature in Emacs already, called child-frames.

With enough widgets, one will be able to build GUI, much like in GToolkit, but with Emacs Lisp, and we’ll still have all the legacy textual interface working, e.g. things like Magit don’t have to throw everything out of the window and adapt - they can continue to work. We just need to be sure to provide enough extensibility, and a way to implement custom widgets, much like Emacs allows right now with its text properties, overlays, and such. Then new plugins for the GUI toolkit will arrive, making Emacs even more extensible than before.

Now, imagine, you open Emacs and it creates a window that is basically like a desktop of your operating system. You open a window, and it’s a real window, floating on that desktop. Don’t like it? Just maximize it, the window widget will work similarly to how windows are created in Emacs right now. Except it doesn’t have to be a window displaying a buffer and nothing else - it can display other widgets, some of which are buffers.

Frames become virtual desktops, which they really are, it’s just that they’re represented as WM windows because Emacs didn’t include ways of switching them virtually (it did in the terminal mode actually). You can still open a separate “frame” with a given virtual desktop, so again, everything would work as you’d expect.

I’ve made a quick prototype that implements this. Here’s a live demo:

And here’s Emacs running natto.dev and GToolkit pages in these child frames:

With this mode, I can create child frames, that act as small windows, and pan these around much like the natto.dev app I keep referring to. Note that when the window tries to split itself, instead it creates a new window near it, and tries to keep windows from overlapping by finding empty space nearby. Basically, this is almost like a different kind of window manager.

Though this thing doesn’t work as well as you’d imagine, if you want to try it, you can get it here.

The future of Emacs

…probably isn’t what I’m describing here. I don’t know. I’m just fantasizing, but the image of such a system I see behind my eyelids is great.

I depend on Emacs, it is my main tool for the job, this blog, my other hobbies, and other activities, so I want this system to evolve. This is just a route I think it could take. Again, maybe nothing of this makes any sense to you, but I encourage you to try out at least the GToolkit, I think it is the most interesting things I saw in the last years. Pharo can be a bit less exciting but it laid the foundation for GT, and I think that’s says something. If it wasn’t for Clojure, and maybe this blog which I do in Org-mode, and if I never used Emacs I would probably try GToolkit at some point, and maybe it would become my main tool, and I would probably look at Emacs as an inferior system. Well, at least when it comes to GUI capabilities. But I still love Emacs as it is, it fulfills my needs, and I can make it do what I want it to do, which is the main point.

This post was mostly a collection of random thoughts, hence the category, but I genuinely think that if a system like this can be created in Smalltalk or in the current state of the web, it is definitively possible in Emacs, there’s just not that much of an interest in it. Well, actually there is, as I’ve mentioned, people trying to make emacs more beautiful and appealing, but I think we should not forget about utility too, and thus if someone is to develop a GUI toolkit for Emacs they need to put a lot of thought into it:

• What widgets to include;
• How do you move from widget to widget in an efficient way;
• What is best done with a mouse, and what should not be done with it;
• What level of interactivity from Emacs Lisp we should get;
• Other questions like this.

So yeah, it will take a lot of planning, prototyping, maybe some competing systems to progress on this - let’s see where Emacs will go in the upcoming 30 years. Maybe another XEmacs will happen, or it will be forced to evolve by something like Project Mage or Nyxt, or even another system, that would embrace GToolkit’s modal-less interface and infinite canvas. The advent of virtual and/or augmented reality makes an interesting infinite-canvas type of interface candidate in my opinion, so maybe we’ll stop seeing flat screens in the future, who knows?

Anyway, let me know what you think. If you have any thoughts you like to share, you can find my contacts on the About page. Hope it was an interesting read!