Andrey Listopadov

Implementing SSE with fnl-http and async.fnl

Wed, 30 Jul 2025 01:05:00 +0300

Some time ago, I was working on an HTTP library for Fennel. As a proof of concept, I added a module that implements a simple web server and wanted to experiment with it. The server can serve files from a directory using this simple handler:

(fn handler [{: method : path &as request}]
  (case method
    :GET (let [file (io.open (path:gsub "^/" "") :r)]
           (or file {:status 404 :body "not found"}))
    _ {:status 405
       :body "Method not allowed"}))

This handler is simple - when the client sends a GET request with a given path, for example GET http://localhost:8080/some/file.txt, the handler simply removes the leading / and tries to open the given path as a file handle. If the file is present, we return it as is, and the server will handle the closure of this file once the client disconnects. Otherwise, we return a table with the 404 status code, indicating that the file wasn’t found. Other methods are not allowed, and we respond with a 405 error by matching on method with case.

Now we can require the server module and start a small file-serving server:

(local server
  (require :io.gitlab.andreyorst.fnl-http.server))

(local s
  (server.start handler {:port 8080}))

(s:wait)

The (s:wait) call simply locks the current thread because, without it, the process will just exit.

This is, of course, the simplest web server there is; it just accesses files in the current directory, returning them as application/octet-stream. The handler function can be more sophisticated, handling different methods, parsing arguments, etc. For our purposes today, we won’t need most of these features, but I thought it would be better to familiarize you with how the server is set up.

The topic of today’s post is Server-Sent Events, or SSE for short. It is a protocol for communicating between a server and multiple clients that uses a persistent connection and pushes messages from the server to the client. The client can then process these messages at their own pace and react accordingly.

We’ll need an asynchronous library to handle SSE for multiple clients, so let’s add it to the mix:

(import-macros {: go : go-loop}
  :io.gitlab.andreyorst.async)

(local {: <! : >! : timeout : alts! : pub : sub : chan : close! &as async}
  (require :io.gitlab.andreyorst.async))

(local json
  (require :io.gitlab.andreyorst.json))

I’ve also included a JSON library; both are already used by fnl-http. Now, let’s begin!

SSE stream

We’ll send events to our clients via a channel:

(local event-ch (chan))

This event channel will be our main way to send events to the clients, but we won’t read from it directly. Instead, we can create a “publication” or pub that will allow clients to subscribe to the messages:

(local event-pub
  (pub event-ch #:sse #(async.sliding-buffer 100)))

For the purposes of this post, there’s only one type of message in this event-pub thing, but in a more complex scenario, you can separate messages and subscribe to different topics altogether. Speaking of subscribing, here’s a helper function:

(fn subscribe-to-events [client-ch]
  (sub event-pub :sse client-ch))

Finally, we need a function to register an event:

(fn register-event [event]
  (>! event-ch event))

Now the fun part. Right now, our event system is ready to be accessed by clients, but the events themselves don’t make any sense, as they’re just some objects. We’ll define events as tables with the following structure:

{:event "optional event type"
 :id    "optional event id"
 :data  "event data"}

Since our register-event function simply writes to the event-ch, we need a way to intercept that. We can do so by writing middleware that accepts a channel with events as tables and returns a new channel that will have all of the events formatted:

(fn wrap-format-sse-event [ch]
  (let [out (chan)]
    (go-loop [e (<! ch)]
      (case e
        {: event : id : data}
        (doto out
          (>! (.. "event: " (tostring event) "\n"))
          (>! (.. "id: " (tostring id) "\n"))
          (>! (.. "data: " (tostring data) "\n")))
        {: event : data}
        (doto out
          (>! (.. "event: " (tostring event) "\n"))
          (>! (.. "data: " (tostring data) "\n")))
        {: data}
        (doto out
          (>! (.. "data: " (tostring data) "\n"))))
      (if e
          (recur (<! ch))
          (close! out)))
    out))

Now, the general mechanism is as follows:

When a client connects, we create a channel and subscribe to events.
We wrap this channel with our formatter and some other middleware if needed.
Finally, we return this channel as the response body.

Before we write an HTTP handler that does this, we need to consider one other thing: the connection. If the connection is not used for a long time, it can be closed automatically. Because of that, it is common to send meaningless heartbeat events every now and then. Let’s write another wrapper that will track if the client’s event stream doesn’t have any events for some time and return a heartbeat:

(fn wrap-heartbeat [ch timeout-ms]
  (let [out (chan)]
    (go-loop []
      (let [tout (timeout timeout-ms)]
        (match (alts! [tout ch])
          [_ tout]
          (do (>! out {:event "heartbeat" :data 0})
              (recur))
          [event ch]
          (do (>! out event)
              (recur))
          [nil ch] (close! out))))
    out))

Similarly to our previous wrapper, we accept a channel with events and return a new channel. The asynchronous go-loop, which I didn’t mention before, will run in a separate virtual thread, trying to read from both the given channel and a timeout channel. Whichever returns first will determine what event will be sent to the client. If a normal event comes through, we send it to the out channel as is and restart the timer. If no events come and our timer has been exhausted, we send a heartbeat and restart the timer again. If our events channel returns nil, it means that it was closed, so we also close the out channel.

Finally, we can implement our handler:

(fn handler [{: path : method : headers : content}]
  (case path
    (where "/events" (= :GET method))
    (let [events (chan)]
      (subscribe-to-events events)
      {:status 200
       :headers {:content-type "text/event-stream"}
       :body (-> events
                 (wrap-heartbeat 10_000)
                 wrap-format-sse-event)})
    (where "/notify" (= :POST method))
    (case (tonumber headers.Content-Length)
      len (let [event (json.decode (content:read len))]
            (register-event event)
            {:status 200
             :body "OK"})
      _ (error "malformed Content-Lenght"))
    _
    {:status 405
     :headers {:content-type "application/json"}
     :body (json.encode {:error "Method not allowed"})}))

This handler has two main entry points: /events and /notify. The /events entry is for subscribing to the events stream with a GET request. The /notify entry is for sending events from the client to the server (because right now we don’t want to make the server too complicated to generate events on its own).

Same as before, we can start a server:

(local s
  (server.start handler {:port 12345}))

(s:wait)

Reading the event stream and triggering events

With the server up and running, we can connect a pair of clients.

The first client will subscribe to the event stream by executing a GET request:

(local http
  (require :io.gitlab.andreyorst.fnl-http.client))

(local resp
  (http.get "http://localhost:12345/events" {:as :stream}))

If we look at the response, we’ll see the headers containing text/event-stream as the content type, as per the response from our server, and the body as a stream, as requested by the client:

{:body #<Reader: 0x564ae85ca7d0>
 :headers {:Connection "keep-alive"
           :Content-Type "text/event-stream"
           :Transfer-Encoding "chunked"}
 :http-client #<tcp-client: 0x564ae812fa10>
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 38
 :status 200
 :trace-redirects {}}

We can consume events with a helper function:

(fn consume-events [stream]
  (fn read-line [stream buf]
    (let [c (stream:read 1)]
      (when (not= "\n" c)
        (table.insert buf c)
        (read-line stream buf))))
  (while true
    (let [buf []]
      (read-line stream buf)
      (io.write "[" (os.date) "]\t" (table.concat buf) "\n"))))

(consume-events resp.body)

In the process output, we’ll be able to see heartbeat events appearing every ten seconds:

[Wed Jul 30 00:34:35 2025]	event: heartbeat
[Wed Jul 30 00:34:45 2025]	data: 0
[Wed Jul 30 00:34:45 2025]	event: heartbeat
[Wed Jul 30 00:34:55 2025]	data: 0

Next, we can send a POST request from another process, sending an event:

(local http
  (require :io.gitlab.andreyorst.fnl-http.client))
(local json
  (require :io.gitlab.andreyorst.json))

(http.post "http://localhost:12345/notify"
           {:body (json.encode {:event "message" :id "client-2" :data "Hi from client-2"})})

We immediately see the event in the first client’s output, followed by heartbeat events:

[Wed Jul 30 00:35:03 2025]	event: message
[Wed Jul 30 00:35:03 2025]	id: client-2
[Wed Jul 30 00:35:03 2025]	data: Hi from client-2
[Wed Jul 30 00:35:03 2025]	event: heartbeat
[Wed Jul 30 00:35:13 2025]	data: 0

As you can see, the time when the heartbeat event arrived is exactly ten seconds after the last non-heartbeat event. Of course, we can connect multiple clients subscribed to the event stream, and they will all receive events.

Conclusion¹

This should give you an idea of how to implement an SSE stream in your server using the fnl-http library. Of course, this implementation is only loosely based on the specification, so I wouldn’t suggest it as a general implementation guide, but I hope it provides a clear enough explanation of how things work.

The server module is still very much in a proof-of-concept stage, but as this article shows, it is working quite well. All in all, I believe the ability to return an asynchronous channel as a body from the server is a nice way to handle this kind of event system. The code can be written as usual, and no specific handling for returning asynchronous data via HTTP is required.

Thanks for reading, and I hope you’ll find fnl-http interesting enough to try!

As much as I don’t like “Conclusion” as the last heading in an article, I can’t think of a different one here. ↩︎

Comment via email

Extending JSON encoder with custom types

Mon, 04 Nov 2024 06:39:00 +0300

One thing I like about Lua is that it has a limited amount of built-in types and data structures. There are no objects, just tables. And if you need a custom behavior, a metatable can be attached to any table to extend it with custom methods that integrate with the rest of Lua runtime.

However, this is both a blessing and a curse. And unfortunately, Lua developers made a huge mistake by breaking this API¹ once already, in a minor release even.

As you may noticed, fnl-http has a lot of stuff. It has asynchronous Channels, and various Readers, which act like objects, but are still ordinary tables underneath. And this is true for a lot of Lua projects, tables are flexible enough to implement many cool things, so people do.

However, this poses an interesting problem - because objects are not a thing in Lua, and a set of default metamethods is quite limited, there’s no default way of extending objects with custom behavior. In Java, you would specify what interfaces are required for an object to have, and it often means that if something doesn’t implement them, a wrapper object is needed. In Clojure, we can get away with protocols and multimethods, but Lua has neither of those. In Lua, and subsequently Fennel, we have to be creative, because even though we can add stuff through metatables, not every value can have one.

While working on fnl-http, I added a basic JSON parser and encoder. The parser part is quite simple, as JSON has a specification and a fixed amount of data types to worry about. It’s not like EDN or XML, for example, which are both extensible data notations that can represent pretty much anything. In JSON you get only these data types:

Number
String
Array
Object
Boolean
Null

The only difference from Lua here is that null is nil and Lua doesn’t have arrays as a separate type. So implementing a parser was easy, and everything pretty much maps to Lua data structures one-to-one.

Implementing JSON encoder, however, was not as easy. For basic data types, it is easy enough, but once we touch custom objects created with metatables things get messy.

For example, let’s say we need a set data structure. A set is similar to a table, except the values are the same as the keys. But, we may want to distinguish them from ordinary tables, if, say, we’re making a library that has a set of functions for working with mathematical sets. Like intersection, difference, etc. Here’s a set constructor:

(fn Set [...]
  (let [data (faccumulate [res {}
                           i 1 (select :# ...)]
               (let [k (select i ...)]
                 (doto res (tset k k))))]
    (setmetatable
     {}
     {:__index data
      :__newindex (fn [_ k _] (tset data k k))})))

As you can see, we create an empty table and set its metatable to a bunch of functions that close over the local data. This technique is quite common in Lua and is called proxy tables. By using the empty table as a base, we ensure that any table access triggers the __index or __newindex metamethod, depending on the action.

However, there’s a problem. If we try to print such an object in the REPL, we don’t get any information about it, even though it has values:

>> (local s (Set :a :b :c))
nil
>> (. s :a)
"a"
>> (set (. s :d) 42)
nil
>> (. s :d)
"d"
>> s
{}

Additionally to that, we have no means of iteration over this data structure, as the default next function will just return nil:

>> (next s)
nil

We can fix this by adding a __pairs metamethod:

(fn Set [...]
  (let [data (faccumulate [res {}
                           i 1 (select :# ...)]
               (let [k (select i ...)]
                 (doto res (tset k k))))]
    (setmetatable
     {}
     {:__index data
      :__newindex (fn [_ k _] (tset data k k))
      :__pairs (fn [_] #(pick-values 1 (next data $2)))})))

Now, printing works, albeit in a bit strange way:

>> (Set :a :b :c)
{:a nil :b nil :c nil}

A few years ago, I remade the pretty-printer for Fennel, and support for custom objects was a huge part of its design. In short, we can fix our object textual representation by using the __fennelview metamethod.

Now, __fennelview is fine and all, but it is not exactly meant for extending existing objects. Of course, given a Lua library, that implements these proxy tables you can add support for Fennel, but it’s more of an ad-hoc solution. So it’s meant for library authors to make the Fennel REPL experience more user-friendly, and make sure that the printed representation can be read back by Fennel.

And it’s not always possible to add __fennelview to existing objects. For example, one common complaint in Lua is that arrays start from 1 and not 0. We can “fix” that by implementing our own Array type:

(fn Array [...]
  (let [vals [...]]
    (setmetatable
     []
     {:__index (fn [_ i]
                 (. vals (+ i 1)))
      :__newindex (fn [i val]
                    (tset vals (- i 1) val))
      :__len #(length vals)
      :__pairs (fn [_] #(next vals $2))})))

Right now, the object above prints like this:

>> (Array 1 2 3)
[1 2 3]

Let’s assume this came from the library, and we wish to print this differently, e.g. as (Array 1 2 3) instead of [1 2 3] to make sure that such arrays are read back as 0-based arrays, and not as ordinary Lua tables. We can do that, but it is tricky because we don’t have any means of accessing the data closure:

(fn wrap [arr]
  (tset (getmetatable arr)
      :__fennelview
    (fn pp-doc-example [t view options indent]
      (let [lines (icollect [i v (pairs t)]
                    (let [v (view v options (+ 7 indent))]
                      (if (= i 1) v
                          (.. "       " v))))]
        (doto lines
          (tset 1 (.. "(Array " (or (. lines 1) "")))
          (tset (length lines) (.. (. lines (length lines)) ")"))))))
  arr)

However, as you can see, we have to use wrap on every Array, because each instance comes with its own metatable, and setting it once wouldn’t work. And if we try to read back the printed representation we’ll get the old representation:

>> (wrap (Array 1 2 3))
(Array 1 2 3)
>> (Array 1 2 3)
[1 2 3]

So we can make another wrapper:

(local Array* Array)

(fn Array [...]
  (wrap (Array* ...)))

>> (Array 1 2 3)
(Array 1 2 3)

So you can see, how this is a bit problematic.

Custom JSON encoders

Now, this was a bit of a tangent, so let’s get back to the main topic - JSON encoding. Because I can’t ask everyone to do this kind of wrapping, I decided to implement a different way of extending the json.encode function. Thus, the library now features two more functions: register-encoder and unregister-encoder. The second one is for fixing mistakes, and won’t probably be used too much.

So, let’s see how we can add support for custom types. First, we need a function that will tell if the given object is an instance of something we want to encode:

(fn Array? [x]
  (and (= :table (type x))
       (not= nil (. x 0))
       Array))

A library may provide a custom predicate to check if the given object is of the same type, like chan? for asynchronous channels. However, it can’t be used as is, because to register an encoder, such a function needs to return a singleton, that can be compared with =. So we return Array from our example function.

Next, we can implement the encoder:

(fn encode-array [arr encode]
  (.. "["
      (-> (fcollect [i 0 (- (length arr) 1)]
            (encode (. arr i)))
          (table.concat ", "))
      "]"))

Finally, we can register it:

(json.register-encoder object object? encode-object)

The first argument is the object itself, and the second one is a function to obtain the object’s type. Internally, register-encoder calls the object? function on the first argument to provide a bit more flexibility, in case object? returns different values for different objects or based on the object’s properties. The third argument is a function that provides the encoding. It accepts a callback to encode nested values as a second argument.

With that, we can register the encoder and encode our array as JSON by calling (json (Array 1 2 3)):

>> (json.register-encoder (Array) Array? encode-array)
nil
>> (json (Array 1 2 3))
"[1, 2, 3]"

This should provide enough flexibility to support any proxy-based objects.

Custom decoders

I decided not to add custom decoder support to this library, yet. JSON doesn’t have any custom types, so it doesn’t make sense to parse its values to anything but default Lua types.

One could argue, that it would be nice to have a way to parse arrays to zero-based ones, given that we can convert zero-based arrays to JSON. While true, I think this only adds unnecessary complexity to the parser.

One reason, registering an encoder requires a singleton as type designator is to ensure that objects can’t be mixed up. If it was allowed to register encoders via string types, it would be possible to register two "Array" types, making ambiguous what encoder would be used or would require to change the name in case of conflict.

JSON has a fixed amount of types, so registering a decoder for, say, Array, would mean that all arrays would be parsed to this custom type in all of the project parts. Meaning, that if you use a different library, that also uses this JSON parser, and internally expects one-based arrays it will break.

I’m open to changing my mind on this, but for now, I feel that custom encoders are sufficient enough to make this library more usable. As a part of this change, I’m moving this library out from the fnl-http repo into its own library, making fnl-http depend on it instead, making it possible for other projects to use this parser more easily. You can find the code here.

Thanks for reading, and I hope this library will be useful!

Foontones

The deprecation of __ipairs metamethod in Lua 5.3 resulted in an inability to implement custom iteration over custom sequential data structures. For example, the zero-indexed array example would benefit from __ipairs because we could specify that the iteration starts from 0. With the default ipairs we would always miss the first value. It’s not hard to fix manually, by overriding ipairs with a custom function that checks for __ipairs metamethod:
```
       (local ipairs* ipairs)
       (fn _G.ipairs [tbl ...]
         (case (getmetatable tbl)
           {:__ipairs ip} (ip tbl ...)
           _ (ipairs* tbl ...)))
```
However, it’s not practical to intrude into Lua’s standard library in such a way. ↩︎

Comment via email

Streaming and receiving chunked multipart data with fnl-http

Sat, 02 Nov 2024 04:46:00 +0300

I’ve been absent for a while - you may have noticed that compared to the previous year, I posted a lot less this time. There are two closely related reasons for that. First, I felt burned out from programming. Second, I finally picked up a guitar after five or more years and started recording again. Thus, now I only do hobby programming when I feel passionate or challenged in an entertaining way. This works for me, but I have less material to write about.

One such passionate project for me this year is fnl-http. It’s an HTTP/1.1 client and server implementation for Fennel made using luasocket and async.fnl - another library of mine.

Last time I posted, I was mostly talking about testing. And I’m glad that I wrote all these tests, because the change I’m going to talk about this time broke a lot of them, and I was able to diagnose problems much quicker. But the main changes are related to the post I made before that: fnl-http Improvements. In that post, the most exciting things for me were chunked transfer encoding and support for multipart requests. I’ve implemented support for reading chunked responses (the client already could send chunked requests) and moved onto implementing multipart requests.

HTTP Server

For some time, the implementation was sufficient enough, so I started working on an POC asynchronous HTTP server. The server is pretty much bare-bones right now, though it features everything you need to implement a proper one. Here’s how it can be used:

(local server (require :io.gitlab.andreyorst.fnl-http.server))
(local json (require :io.gitlab.andreyorst.fnl-http.json))

(fn handler [{: headers &as request}]
  {:status 200
   :headers {:connection (or headers.Connection :keep-alive)
             :content-type :application/json}
   :body (json request)})

(with-open [s (server.start handler {:port 3000})]
  (s:wait))

It’s a rather simple echo server, that sends back the parsed request as JSON. Let’s try it:

>> (local http (require :io.gitlab.andreyorst.fnl-http.client))
nil
>> (http.get "localhost:3000" {:as :json})
{:body {:headers {:Host "localhost:3000"}
        :method "GET"
        :path "/"
        :protocol-version {:major 1 :minor 1 :name "HTTP"}}
 :headers {:Connection "keep-alive"
           :Content-Length "131"
           :Content-Type "application/json"}
 :http-client #<tcp-client: 0x561839afe390>
 :length 131
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 13
 :status 200
 :trace-redirects {}}

Now, of course, this won’t work as is, because not all Lua values can be encoded as JSON. Such examples are functions, and custom objects, like Readers. We can cheat, though, and encode it as a Fennel table and send it back as a string:

(local {: view} (require :fennel))

(fn handler [{: headers &as request}]
  {:status 200
   :headers {:connection (or headers.Connection :keep-alive)
             :content-type :application/json}
   :body (view request)})

Now we can try a POST request:

>> (http.post "localhost:3000" {:body "Hello HTTP"})
{:body "{:content #<Reader: 0x55eeb3016fc0>
 :headers {:Content-Length \"10\" :Host \"localhost:3000\"}
 :length 10
 :method \"POST\"
 :path \"/\"
 :protocol-version {:major 1 :minor 1 :name \"HTTP\"}}"
 :headers {:Connection "keep-alive"
           :Content-Length "183"
           :Content-Type "application/json"}
 :http-client #<tcp-client: 0x55eeb3059470>
 :length 183
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 20
 :status 200
 :trace-redirects {}}

This should give you an idea of how to implement a proper handler.

The reason the content is a Reader is simple - we don’t want to process any more data than we need when parsing the request. Once we obtain the HTTP-related data, such as headers, method, path, and version, we wrap the rest of the data in a Reader object, so the handler function can work on it later or even discard it. In this particular case, it’s as simple as doing (request.content:read (tonumber request.headers.Content-Length)) but it may vary depending on what was sent, and what path the client used. So I’m leaving this open. Using a Reader enables asynchronous and lazy processing of the request’s body.

Multipart requests

What’s interesting, however, is that there may be multiple bodies, if the request is a multipart one. We can send a multipart request like this:

(http.post "localhost:3000"
           {:multipart [{:name "foo" :content "some string"}
                        {:name "file" :filename "some-data.json"
                         :content (io.open "some-data.json")}]})

Ideally, the server should be able to first process the foo part, and then a file part. Additionally, a file can be really big, so we might want to avoid reading it fully into memory. For example, the JSON module of fnl-http can parse files without fully reading them into memory, as it is based around readers.

The question, however, is how do we give this to a handler without reading the request in full first? We don’t know how many parts there will be, and we might not know the sizes of each part. Let’s look at how the handler encodes this request:

>> (http.post "localhost:3000"
           {:multipart [{:name "foo" :content "some string"}
                        {:name "file" :filename "some-data.json"
                         :content (io.open "some-data.json")}]})
{:body "{:headers {:Content-Type \"multipart/form-data; boundary=------------aaec41d0-1c75-4037-a039-91cc2b31584c\"
           :Host \"localhost:3000\"}
 :method \"POST\"
 :parts #<function: 0x55eeb28e79c0>
 :path \"/\"
 :protocol-version {:major 1 :minor 1 :name \"HTTP\"}}"
 :headers {:Connection "keep-alive"
           :Content-Length "256"
           :Content-Type "application/json"}
 :http-client #<tcp-client: 0x55eeb31df9e0>
 :length 256
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 80
 :status 200
 :trace-redirects {}}

You can see, that there’s no content field anymore. Instead, there’s a parts field, and it is a function.

This function is a standard Lua iterator, that you can call until it returns nil or plug it in any of the iteration forms. Let’s change our handler to support both content and parts:

(fn handler [request]
  (case request
    {: content}
    (set request.content
      (if request.headers.Content-Length
          (content:read (tonumber request.headers.Content-Length))
          (= request.headers.Transfer-Encoding "chunked")
          (content:read :*a)))
    {: parts}
    (set request.parts
      (icollect [part parts]
        (doto part (tset :content (part.content:read :*a))))))
  {:status 200
   :body (json request)})

Let’s look at this a bit closer. The case has two branches - one for content and one for parts. The content branch is pretty straightforward - we replace request’s content field value with the data obtained from Reader. If there was a Content-Length header, we used it to determine how much to read. If not, we (rather crudely) check if the chunked Transfer-Encoding was used, and read all of the data from the Reader. We can do so, because chunked readers know when to stop, based on the final chunk header.

The parts branch is similar, except we have to go one level deeper, and it’s a bit simpler to work with the part’s contents. First, the main function of icollect is to produce a sequential table from the iterator values. We do so by replacing the content field of each part. The content field is again a Reader, but it’s a sized reader, so we can almost always read it in full. Even if content length was not specified, parts are separated by boundaries, so we always know when to stop.

Each part is a table of the following structure:

{:content #<Reader>
 :headers {...}
 :length N ;; optional field, if headers had a Content-Length specified
 :name "part name"
 :filename "name of the uploaded file"
 ;; alternatively :filename* can be present instead
 :type "part kind"}

As much information is given to make working with parts easier. And because parts are provided by an iterator, we don’t have to know in advance how many parts are coming.

However, there’s a caveat - we can’t obtain information about each part first, and process the contents later. Moving to the next part consumes the request body until the next boundary is given, so we can’t reverse the Reader back to its content. Thus, doing this won’t work:

(let [all-parts (icollect [part parts] part)]
  (each [_ part (ipairs all-parts)]
    (part.content:read :*a)))

You’ll have to either copy reader contents into another in-memory Reader, or process parts as they appear.

Now, we can try our request again:

>> (http.post "localhost:3000"
              {:multipart [{:name "foo" :content "some string"}
                           {:name "file" :filename "some-data.json"
                            :content (io.open "some-data.json")}]
               :as :json})
{:body
 {:headers
  {:Content-Type
   "multipart/form-data; boundary=------------a03996f2-0d1c-4aa4-97f5-ac90b06cb6ea"
   :Host "localhost:3000"}
  :method "POST"
  :parts [{:content "some string"
           :headers {:Content-Disposition "form-data; name=\"foo\""
                     :Content-Length "11"
                     :Content-Transfer-Encoding "8bit"
                     :Content-Type "text/plain; charset=UTF-8"}
           :length 11
           :name "foo"
           :type "form-data"}
          {:content "{\"value\": \"JSON data\"}\n"
           :filename "some-data.json"
           :headers {:Content-Disposition "form-data; name=\"file\"; filename=\"some-data.json\""
                     :Content-Transfer-Encoding "binary"
                     :Content-Type "application/octet-stream"
                     :Transfer-Encoding "chunked"}
           :name "file"
           :type "form-data"}]
  :path "/"
  :protocol-version {:major 1 :minor 1 :name "HTTP"}}
 :headers {:Connection "keep-alive" :Content-Length "808"}
 :http-client #<tcp-client: 0x55cf2d942450>
 :length 808
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 13
 :status 200
 :trace-redirects {}}

As you can see, the foo part had length of 12, and the file part had a chunked transfer encoding.

Before this post, there was no way of sending multipart with chunked encoding, so this is a pretty good change in my opinion. Definitively expands the possibilities. Still need to test this with some real HTTP servers, as I don’t see this feature used too often though.

Performance

Now, let’s talk performance. I used the wrk tool to measure the server’s RPS with a simple handler that does 50 millisecond delay on each request:

(local async (require :io.gitlab.andreyorst.async))

(fn handler [{: headers &as request}]
  (async.<! (async.timeout 50))
  {:status 200
   :headers {:connection (or headers.Connection "keep-alive")
             :content-length 11
             :content-type "text/plain"}
   :body "hello world"})

The handler function is always running in an asynchronous context, so we can use parking operations such as <! inside of it.

Running it with 12 threads and 400 connections for 30 seconds yields about 2k requests per second:

Running 30s test @ http://localhost:3000
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   146.42ms   94.13ms   2.00s    98.54%
    Req/Sec   205.07     90.83   343.00     62.36%
  69835 requests in 30.09s, 8.00MB read
  Socket errors: connect 0, read 0, write 0, timeout 151
Requests/sec:   2321.21
Transfer/sec:    272.37KB

Not bad for a single-threaded Lua runtime, I would say!

Now, of course, this kind of benchmark is more synthetic than real, so don’t expect it to work as well as it says here in all cases. With a more complex handler, that deals with routing, and data processing this number will go down. Still, I’m pretty satisfied with these results.

And CPU usage is quite low too. When idling a single CPU core is sitting at about 5%, and during the benchmark, it rises to 70%. Removing the asynchronous sleep, raises CPU usage to 100% (single core), peaking at 67MB of RAM and serving about 4k requests per second. So with an asynchronous sleep, our server process is doing some actual sleeping, leaving some resources to the OS!

And given how async.fnl is implemented, we can actually replace (async.<! (async.timeout 50)) with a blocking loop, like (for [i 1 1_000_000] nil) and still get around 400 RPS. That’s a busy loop with one million iterations per handler invocation on each request. So you shouldn’t worry too much when blocking the handler.

Is it a good result? You be the judge.

Comment via email

fnl-http - testing with fennel-test

Thu, 15 Aug 2024 23:01:00 +0300

fnl-http is my current passion project - I spend a lot of free time tinkering with it, and the last week was spent on testing and fixing bugs.

As you may know, I made a testing framework, called fennel-test, which has a dedicated test runner, and a set of macros for writing tests. I use it in all of my projects, and oftentimes, some project requires more features that are currently available in the framework, so I extend it as needed. I think it’s a great way of designing something - you use it, and enhance it as you encounter new needs.

I want programming to be exciting, so I spent a lot of time making my test report nice to look at. Here’s how the report looks by default:

Test run at Tue Aug  6 23:20:41 2024, seed: 1722975641242
(....)(......)(.)(.....)(.)(........)(...)(...)

The default reporter is called dots. Each test namespace is encoded with parentheses (), each successful test is encoded as a dot ., and failed tests are encoded as F. The runner randomizes the order of tests each time, so if we run it again, the result would be different:

Test run at Tue Aug  6 23:24:10 2024, seed: 1722975850235
(......)(........)(.)(...)(...)(....)(.....)(.)

However, this is all you get with a reporter like this. If a test fails, information about the error is printed after all tests have finished, so you know what test did fail.

There’s another inbuilt reporter, called namespaces:

Test run at Tue Aug  6 23:27:33 2024, seed: 1722976053226
test.http-parser-test: PASS
test.client-test: PASS
test.json-test: PASS
test.httpbin-test: PASS
test.body-test: PASS
test.headers-test: PASS
test.readers-test: PASS
test.url-test: PASS

As you can see, it prints each namespace that is currently being tested, followed by the namespace-wide status. If a test fails, you again see its error message, but only after all tests have finished, which isn’t ideal because tests in the fnl-http project take about two minutes to complete.

Fortunately enough, you can create reporters in the .fennel-test configuration file!

Reporters

A reporter is just a table with a set of functions. Each one is called when the respecting stage of testing is being run. The methods are:

ns-start - called when entering a new test namespace.

Used to print the information about the namespace, such as its name.
ns-report - called when exiting a test namespace.

Used to print the overall status of the namespace.
test-start - called when running a single test from a namespace.

Used to print the information about the test, such as the test’s name.
test-report - called after the test is finished.

Used to print the status of the test.
stats-report - called after running all tests in all namespaces.

Used to print overall statistics of tests - how many tests were executed, how many errors were encountered, were there any warnings, etc.

This was enough for me to implement other kind of reporters, like the one in the project’s readme. However, I wanted a bit more for this project - first, I wanted the report to be fun to look at, and second, I wanted some statistics. And third, actually, I noticed the need to skip tests in case something is not available on the machine.

After a bit of work, here’s what I’ve got:

With this reporter, if a specific test fails, I can immediately see it, without waiting for all tests to pass. In the end, you can see some statistics on the longest tests, and average test time of the namespace. For example, the test.httpbin-test namespace took 90 seconds to complete, but on average each test only takes 11 seconds. I’m not entirely sure why this metric would be necessary, but I’ve seen it in another test runner, so I implemented it in my reporter.

Now, here’s a problem - one of the tests takes a whopping 90 seconds. I don’t want to run this test all the time, especially, when I’m fixing a bug found by some other test. So I introduced a skip-test function as a first step.

Originally, this function had two modes - terminating and non-terminating, but I later removed the second mode. The reason for that is the with-open macro.

Here’s an example of one of the fixtures I use in the fnl-http project:

(use-fixtures
 :once
 (fn [t]
   (with-open [proc (io.popen (.. "podman run  -p " port ":80 kennethreitz/httpbin >/dev/null 2>&1 & echo $!"))]
     (let [pid (proc:read :*l)
           attempts 10]
       (if (wait-for-server attempts)
           (do (t)
               (kill pid))
           (do (kill pid)
               (skip-test (.. "coudln't connect to httpbin server after " attempts " attempts"))))))))

The with-open macro ensures that the resource is closed no matter what. Closing the process, however, doesn’t kill the program running in the background, so I had to get its PID with sketchy echo $! part and kill it later. Killing the process doesn’t release the file descriptor created by popen, so with-open is only responsible for releasing the file handle here.

A sudden bug

What with-open also does is catching, and re-throwing errors. It’s something akin to trying with resources in Java, and other languages that support this kind of abstraction. You can think about it as try/finally without the catch part.

Lua, however, doesn’t really have well-defined exceptions. Instead, functions usually throw string messages about the error and that’s it. Sometimes, however, it’s useful to throw a table instead. Tables are like objects, so you can embed some additional information in those, making it easier to match a particular error. It’s better to convert this table to a string if it ever reaches the user uncaught though.

What makes things harder is that in Lua it’s not possible to re-throw an error without losing its previous stack trace. While it is a common practice to use pcall to catch any errors and to re-throw them if you can’t do anything about the error, this is what causes losing the precious stack trace. More precisely, if the error message comes with the stack trace, and you would just re-throw it with error, you wouldn’t add your stack trace to the trace. Sounds convoluted, yeah?

So Lua gives you another option: xpcall. The Lua manual doesn’t have a thorough explanation of it though:

xpcall (f, msgh [, arg1, ···])

This function is similar to pcall, except that it sets a new message handler msgh.

It works the same as pcall, except, it accepts an additional argument - a function that would append a stack trace to the error. Usually it’s the debug.traceback function:

debug.traceback ([thread,] [message [, level]])

If message is present but is neither a string nor nil, this function returns message without further processing. Otherwise, it returns a string with a traceback of the call stack. The optional message string is appended at the beginning of the traceback. An optional level number tells at which level to start the traceback (default is 1, the function calling traceback).

Knowing all that, let’s look at how I’ve implemented the skip-test function:

(local Skip
  (setmetatable
   {}
   {:__tostring #:Skip
    :__fennelview #:Skip}))

(fn skip-test [reason]
  "Calling this function inside a test or a fixture will stop the test
early and mark it as skipped. The optional `reason` argument is a
message to display in the log if the reporter is configured to do so."
  (error [Skip reason]))

Right, skip-test throws an array with a special table object, defined as a singleton, so the test runner could distinguish between ordinary test errors and skipped tests. In theory, everything should have worked, but for some reason, some of the tests were skipped correctly, but others didn’t.

The culprit? It’s with-open, obviously, otherwise, why would I take such a tangent about it?

Internally, with-open uses the xpcall function to do its job. And theoretically, it should work just fine, however, it doesn’t use debug.traceback if Fennel’s own compiler.traceback is available.

Originally, I thought that’s just how it is, and implemented two modes in skip-test. After a bit of digging and consulting the manual, I found out that compiler.traceback mistakenly converted every message to a string. Fixing that made the skip-test function work in all cases.

Another sudden bug!

After the patch landed, and I updated Fennel to the newest commit, something else broke. Now, the JSON parsing test no longer succeeded, complaining that some number has no integer representation. The number in question is 23456789012000000000000000000000000000000000000000000000000000000000000000000 or, as it is written in the JSON test file that I use, 2.3456789012e+76.

Before the update, everything worked fine, so I started digging again. Turns out, recently, the way numbers are compiled by Fennel has changed:

@@ -534,8 +534,9 @@ (fn compile-sym
 ;; We do gsub transformation because some locales use , for
 ;; decimal separators, which will not be accepted by Lua.
 (fn serialize-number [n]
-  (pick-values 1 (-> (tostring n)
-                     (string.gsub "," "."))))
+  (if (= (math.floor n) n)
+      (string.format "%d" n)
+      (string.gsub (tostring n) "," ".")))

And now it is failing, because indeed, you can’t convert 2.3456789012e+76 to a string using %d. Except, you can if you’re using LuaJIT - it happily converts this number to "-9223372036854775808", which is even worse.

After a bit of searching, I found that the safest way to format integers as stings is by using "%.f format specifier. Strangely enough, the Lua manual doesn’t have any information on most number formatting patterns. A tiny bit of information was found on the lua-users.org page:

To make sure integer numbers are properly converted into string (without scientific notation), rather than using tostring(x), use format.string("%.0f",x)

I don’t really get why %.f would format a number as an integer given that %f formats floats. And, even though it works the same way in C, I have never seen this in my life. But OK, we could just replace %d with %.f, and be done with that, however, the number above is then would appear as "23456789012000000697746671432670411048021854502345623879537911769302235611136". I get it, it’s a big number, and Lua has limited precision for integers, but the main problem is that this number is much harder to read than its e-notation brother 2.3456789012e+76.

So I settled on finding a way to fix this bug so that both PUC Lua and LuaJIT would format this number in the same way. The result is a patch, that changes the compiler and pretty-printer parts of the language. The function that does that, looks horrifying:

(fn serialize-number [n]
  (let [val (if (= (math.floor n) n)
                (let [s1 (string.format "%.f" n)]
                  (if (= s1 (tostring n)) s1
                      (or (faccumulate [s nil i 0 99 :until s]
                            (let [s (string.format (.. "%." i "e") n)
                                  n* (tonumber s)]
                              (when (= n n*)
                                (let [exp (s:match "e%+?(%d+)$")]
                                  ;; Lua stops transforming numbers from
                                  ;; e-notation to integers at e+14
                                  (if (and exp (> (tonumber exp) 14))
                                      s
                                      s1)))
                              s1))))
                  (tostring n))]
        (pick-values 1 (string.gsub val "," "."))))

We start by checking if (tostring n) and (string.format "%.f" n) give the same string. If they are, we just return the string. If not, we try the scientific notation for each power from 0 to 99. I probably should’ve tested for more powers than that, but it’s OK for now.

We’re converting the number to a string using the e-notation and parse it back, comparing it with the original number. What this does, is that if the power is less than the original, we would lose precision, and comparison would fail. If we can’t find the exponent up to 99, we bail out and return the %.f representation. Otherwise, if the numbers match, we check if the exponent is greater than 14 as it is the exponent where PUC Lua starts using the e-notation. Otherwise, we again return the %.f representation.

Phew!

But I’m not done yet!

Let’s try to compile a really huge number:

>> 1e+99999
nil

What?

Let’s do it in an expression then, maybe that’s just the pretty printer freaking out:

>> (+ 1e+309 1)
attempt to perform arithmetic on a nil value (global 'inf')

~~What?~~ I, of course, jest, but the reason this happens is pretty peculiar.

You see - when we do (tostring "%.f" n) on a huge number like 1e+309 it returns the string "inf". The compiler then happily includes it in the output Lua code, which is then run by the Lua VM. In Lua, however, inf is not a reserved identifier or a number, so it treats inf like a variable name. The inf variable is unbound, hence the nil in the first example, and an error in the second.

So we need to account for that:

(fn serialize-number [n]
  (let [val (if (= (math.floor n) n)
                (let [s1 (string.format "%.f" n)]
                  (if (= s1 "inf") "(1/0)" ; portable inf
                      (= s1 "-inf") "(-1/0)"
                      (= s1 (tostring n)) s1 ; no precision loss
                      (or (faccumulate [s nil
                                        i 0 308 ; beyond 308 every number turns to inf
                                        :until s]
                            (let [s (string.format (.. "%." i "e") n)]
                              (when (= n (tonumber s))
                                (let [exp (s:match "e%+?(%d+)$")]
                                  ;; Lua keeps numbers in standard notation up to e+14
                                  (if (and exp (> (tonumber exp) 14))
                                      s
                                      s1)))))
                          s1)))
                (tostring n))]
    (pick-values 1 (string.gsub val "," "."))))

I also brought the exponent limit up to the last number representable in this notation without becoming inf.

Now we can go back to testing!

Counting skipped tests and assertions

The other thing I added to fennel-test was more counters. Counting skipped tests required some tinkering, like counting tests in the whole namespace if the tests were skipped from the :once fixture.

Counting assertions was even harder, because tests in fennel-test are implemented through macros, and assertions are macros too. I had to inject some runner state, conditionally, and modify it at runtime from each assertion. A bit shady but it works.

What is much more shady is the coverage reports for this project.

Coverage report

I have a firm belief that 100% is the only acceptable coverage. However, and this is a big however, it’s important that such coverage comes from public API tests only. Achieving 100% coverage this way means that your program doesn’t have any dead code and that your public API test at least touches all of our code at least once. It doesn’t speak about test quality, of course, and it doesn’t guarantee that the program is bug-free, but those other points alone make a huge difference.

Lua has one and only luacov that you can use to test your code. I’ve used it in some other Fennel projects in the past, but it comes with some nuances you have to deal with.

First, luacov itself is imprecise. In the post about 100% coverage I talk about it in more detail, but basically, I often see situations where the side effects happened, but the coverage report showed code responsible for the side effects as untouched. Other times, the coverage percentage is bigger than its actual value due to code formatting.

With the fnl-http project I hit a different problem. Two, actually.

The first one is related to the --correlate option of the Fennel compiler. I had an idea, that if I used --correlate I could substitute Lua sources used to produce the coverage with Fennel sources. And since the line numbers are matched by specifying the --correlate option, the coverage report would be so much easier to read.

Didn’t work out.

Unfortunately, --coverage is imprecise. Many parts of Fennel AST don’t contain any line information. Here’s an example:

(local {: a
        : b}
  (require :c))

{: a
 : b}

And here’s the corresponding Lua code:

local _local_1_ = require("c") local a = _local_1_["a"]
local b = _local_1_["b"]
-- empty line
-- empty line
return {a = a, b = b}
-- empty line

As you can see, the overall amount of lines is the same, but many lines that are not empty in the Fennel example are empty in Lua. It’s not a problem, as long as you’re measuring the coverage and viewing the report with Lua, but if we replace the Lua code with its Fennel counterpart, the report becomes invalid. I guess, when the report is constructed, luacov checks if the line is empty or not, and if it isn’t, and there were no hits registered, it counts it as a miss. I’m using luacov-console to view the report, maybe it’s a bug from there, I don’t know.

The only way to fix this is to write the code accounting for these nuances of --correlate:

(local {: a                ; local _local_1_ = require("c") local a = _local_1_["a"]
        : b} (require :c)) ; local b = _local_1_["b"]
                           ; <empty line>
{: a : b}                  ; return {a = a, b = b}

There are more examples like that in the project’s code.

The other problem is more peculiar. Luacov uses debug.sethook to build its report. And I use the async.fnl library in this project, which also uses debug.sethook to schedule tasks. So every time anything happens asynchronously, and trust me, there’s a lot of that in fnl-http, the Luacov report is no longer properly calculated.

So, Luacov has a lot of problems, but I understand why it is like that. You can’t really instrument the Lua code in some other way.

Fennel, on the other hand, isn’t like that. Being a language where everything is an expression, we can probably make some compiler plugin that injects some side-effecting code that writes the report for every expression in the code. It would be difficult to do with things like tables and destructuring, but it should be doable. Maybe I’ll try to tackle this problem in the future, who knows?

Anyway, that’s all from me for now! As usual, you can check out the project, which contains all of the stuff mentioned above regarding fennel-test and luacov. Thanks for reading!

Comment via email

fnl-http Improvements

Thu, 01 Aug 2024 22:57:00 +0300

In the last post two weeks ago I described the process of making an asynchronous HTTP/1.1 client from scratch (minus the socket part). At the end, I mentioned that there’s a lot more to implement:

Now, of course, that’s not all that needs to be implemented. In its current state, this client doesn’t support responses that do not specify the Content-Length or produce a response body in chunked encoding. HTTPS is out of the question for now too. Most of the code just takes a “happy path” without much thought given to errors or more complicated scenarios. But it’s fine, I can always make it better down the line.

HTTPS is still out of the question - to be honest, I don’t want to deal with binary protocols that much, and get myself involved with implementing SSL in Lua. I could, of course use a library, but I feel that Luasocket alone is already big enough dependency.

So here are improvements so far.

Chunked transfer-encoding

As previously mentioned, the client didn’t understand chunked responses, so I had to fix it with a chunked body reader:

(fn read-chunk-size [src]
  {:private true}
  ;; TODO: needs to process chunk extensions
  (case (src:read :*l)
    "" (read-chunk-size src)
    line
    (case (line:match "%s*([0-9a-fA-F]+)")
      size (tonumber (.. "0x" size))
      _ (error (format "line missing chunk size: %q" line)))))

(fn chunked-body-reader [src initial-chunk]
  "Reads body in chunks, buffering each fully, and requesting the next
chunk, once the buffer is empty."
  {:private true}
  (var chunk-size initial-chunk)
  (var buffer (or (src:read chunk-size) ""))
  (var more? true)
  (fn read-more []
    ;; TODO: needs to process entity headers after the last chunk.
    (when more?
      (set chunk-size (read-chunk-size src))
      (if (> chunk-size 0)
          (set buffer (.. buffer (or (src:read chunk-size) "")))
          (set more? false)))
    (values (> chunk-size 0) (string-reader buffer)))
  (make-reader
   src
   {:read-bytes (fn [_ pattern]
                  (let [rdr (string-reader buffer)]
                    (case pattern
                      (where n (= :number (type n)))
                      (let [buffer-content (rdr:read pattern)
                            len (if buffer-content (length buffer-content) 0)
                            read-more? (< len n)]
                        (set buffer (buffer:sub (+ len 1)))
                        (if read-more?
                            (let [(_ rdr) (read-more)]
                              (if buffer-content
                                  (.. buffer-content (or (rdr:read (- n len)) ""))
                                  (rdr:read (- n len))))
                            buffer-content))
                      (where (or :*l :l))
                      (let [buffer-content (rdr:read :*l)
                            (_ read-more?) (not (buffer:find "\n"))]
                        (when buffer-content
                          (set buffer (buffer:sub (+ (length buffer-content) 2))))
                        (if read-more?
                            (let [rdr (read-more)]
                              (if buffer-content
                                  (.. buffer-content (or (rdr:read :*l) ""))
                                  (rdr:read :*l)))
                            buffer-content))
                      (where (or :*a :a))
                      (let [buffer-content (rdr:read :*a)]
                        (set buffer "")
                        (while (read-more) nil)
                        (let [rdr (string-reader buffer)]
                          (set buffer "")
                          (case (rdr:read :*a)
                            nil (when buffer-content
                                  buffer-content)
                            data (.. (or buffer-content "") data))))
                      _ (error (tostring pattern)))))
    :read-line (fn [src]
                 (let [rdr (string-reader buffer)
                       buffer-content (rdr:read :*l)
                       read-more? (not (buffer:find "\n"))]
                   (when buffer-content
                     (set buffer (buffer:sub (+ (length buffer-content) 2))))
                   (if read-more?
                       (if buffer-content
                           (.. buffer-content (or (src:read :*l) ""))
                           (src:read :*l))
                       buffer-content)))
    :close (fn [src] (src:close))
    :peek (fn [_ bytes]
            (assert (= :number (type bytes)) "expected number of bytes to peek")
            (let [rdr (string-reader buffer)
                  content (or (rdr:read bytes) "")
                  len (length content)]
              (if (= bytes len)
                  content
                  (let [(_ rdr) (read-more)]
                    (let [data (rdr:read (- bytes len))]
                      (set buffer (.. buffer (or data "")))
                      buffer)))))}))

It’s a mess, but it works. Please, refer to the previous post for the explanation of what a reader is.

The main idea here is that, when the chunked transfer encoding is detected, we create a reader with chunked-body-reader instead of a regular body-reader, and set up an internal string buffer. We then process all reading operations in terms of this buffer, until it depletes, or when we request more data than left in the buffer. After that, we download the next chunk.

We still get a stream-like behavior out of this, although it has a somewhat big issue - if the server responds with a chunk of enormous size, we can fill up all available memory. I’ve thought of reworking this function to fill the buffer with smaller chunks, but it complicates the code even further, so I decided not to. For now, at least.

And this now works both ways! We can send a chunked body to the server if we’ve didn’t specify the Content-Length header, and the body is not a string. This happens automatically for files, other readers, and channels. Yes, with this library, you can stream data to a server from a channel:

(let [ch (chan)]
  (go #(with-open [body (io.open "some-file" :r)]
         (each [line (body:lines)]
           (<! (timeout 300))
           (>! ch (.. line "\n")))
         (close! ch)))
  (http.post "http://some-server.org" {:body ch}))

This will stream the file line by line, sending each line every 300 milliseconds. The server will receive each line as a separate chunk.

I won’t go into much detail on how this is implemented, as I would need to go through a lot of code.

Multipart requests

After I finished with chunked requests, I turned my eyes on multipart/form-data. It’s an interesting part of HTTP, and after a closer look at it, I realized that email is just a multipart request with weird headers. I’ve been editing emails in Emacs for quite some time but never noticed that the head of the mail is exactly like the head of an HTTP request. What a revelation!

That aside, implementing multipart came with a challenge. I don’t know if it’s just me, but I couldn’t find anywhere if the total Content-Length header is required for a multipart request or not. Logically speaking, since every part is separated by a boundary, a server should be able to read a request’s body until the final boundary. So theoretically, it doesn’t have to know the total content length of the request.

However, I’ve been testing my client with the httpbin.org service and, for some reason, it doesn’t parse the multipart request unless the Content-Length header is present. And it’s not just the sum of part lengths, it’s also their headers, empty lines that separate the header part from content, and so forth.

So I had to implement a set of functions to format multipart parts, and count their lengths:

(fn format-multipart-part [{: name : filename : filename*
                            : content :length content-length
                            : headers
                            : mime-type} boundary]
  "Format a single multipart entry.
The part starts with the `boundary`, followed by headers, created from
`name`, optional `filename` or `filename*` for files, `mime-type`, and
`content-length` which is either calculated from `content` or provided
explicitly.

Default headers include `content-disposition`, `content-length`,
`content-type`, and `content-transfer-encoding`. Provide `headers` for
additional or to change the default ones."
  {:private true}
  (let [content (wrap-body content)]
    (format
     "--%s\r\n%s\r\n"
     boundary
     (headers->string
      (collect [k v (pairs (or headers {}))
                :into {:content-disposition (format "form-data; name=%q%s%s" name
                                                    (if filename
                                                        (format "; filename=%q" filename)
                                                        "")
                                                    (if filename*
                                                        (format "; filename*=%s" (urlencode-string filename*))
                                                        ""))
                       :content-length (if (= :string (type content))
                                           (length content)
                                           (or content-length (content:length)))
                       :content-type (or mime-type (guess-content-type content))
                       :content-transfer-encoding (guess-transfer-encoding content)}]
        k v)))))

This function just formats a single multipart header, i.e. it doesn’t include the content. The content is streamed later, so we don’t need it here.

But with this function, we can now calculate the total length of our request:

(fn multipart-content-length [multipart boundary]
  "Calculate the total length of `multipart` body.
Needs to know the `boundary`."
  (+ (accumulate [total 0
                  _ {:length content-length
                     : name
                     : content
                     &as part}
                  (ipairs multipart)]
       (let [content (wrap-body content)]
         (+ total
            (length (format-multipart-part part boundary))
            (if (= :string (type content)) (+ (length content) 2)
                (reader? content)
                (+ 2 (or content-length
                         (content:length)
                         (error (format "can't determine length for multipart content %q" name) 2)))
                (not= nil content-length)
                (+ content-length 2)
                (error (format "missing length field on non-string multipart content %q" name) 2)))))
     (length (format "--%s--\r\n" boundary))))

This function doesn’t construct the whole body either - it only calculates the length based on headers, and the length of content fields that it can reasonably calculate. There’s a small bug in the format-multipart-part function though. We can override any header, including content-length, meaning we can break the request, if we send a string, and provide a different content-length for it either by mistake or deliberately. This can be fixed, of course, by either not allowing to change content length for strings, or by wrapping string bodies in a string reader. I’ll have to think about what method is better here.

That aside, we can now stream the multipart request:

(fn stream-multipart [dst multipart boundary]
  "Write `multipart` entries to `dst` separated with the `boundary`."
  (each [_ {: name : filename
            : content :length content-length
            : mime-type
            &as part}
         (ipairs multipart)]
    (assert (not= nil content) "Multipart content cannot be nil")
    (assert name "Multipart body must contain at least content and name")
    (let [content (wrap-body content)]
      (->> (if (= :string (type content)) content "")
           (.. (format-multipart-part part boundary))
           (dst:write))
      (when (not= :string (type content))
        (stream-body dst content {:content-length (or content-length (content:length))})))
    (dst:write "\r\n"))
  (dst:write (format "--%s--\r\n" boundary)))

Again, I’m not going to dive too deep here. If you’re interested in the code, you can find and read it here.

Here’s an example of a multipart request body:

--a25537b6-51d6-491f-b18b-a33d197ae4c6\r
Content-Disposition: form-data; name="json-test"; filename="test.json"\r
Content-Length: 1438\r
Content-Transfer-Encoding: binary\r
Content-Type: application/octet-stream\r
\r
[
    "JSON Test Pattern pass1",
    {"object with 1 member":["array with 1 element"]},
    {},
    [],
    -42,
    true,
    false,
    null,
    {
        "integer": 1234567890,
        "real": -9876.543210,
        "e": 0.123456789e-12,
        "E": 1.234567890E+34,
        "":  23456789012E66,
        "zero": 0,
        "one": 1,
        "space": " ",
        "quote": "\"",
        "backslash": "\\",
        "controls": "\b\f\n\r\t",
        "slash": "/ & /",
        "alpha": "abcdefghijklmnopqrstuvwyz",
        "ALPHA": "ABCDEFGHIJKLMNOPQRSTUVWYZ",
        "digit": "0123456789",
        "0123456789": "digit",
        "special": "`1~!@#$%^&*()_+-={':[,]}|;.</>?",
        "hex": "\u0123\u4567\u89AB\uCDEF\uabcd\uef4A",
        "true": true,
        "false": false,
        "null": null,
        "array":[  ],
        "object":{  },
        "address": "50 St. James Street",
        "url": "http://www.JSON.org/",
        "comment": "// /* <!-- --",
        "# -- --> */": " ",
        " s p a c e d " :[1,2 , 3

,

4 , 5	   ,  		        6           ,7        ],"compact":[1,2,3,4,5,6,7],
        "jsontext": "{\"object with 1 member\":[\"array with 1 element\"]}",
        "quotes": "&#34; \u0022 %22 0x22 034 &#x22;",
        "/\\\"\uCAFE\uBABE\uAB98\uFCDE\ubcda\uef4A\b\f\n\r\t`1~!@#$%^&*()_+-=[]{}|;:',./<>?"
: "A key can be any string"
    },
    0.5 ,98.6
,
99.44
,

1066,
1e1,
0.1e1,
1e-1,
1e00,2e+00,2e-00
,"rosebud"]
\r
--a25537b6-51d6-491f-b18b-a33d197ae4c6\r
Content-Disposition: form-data; name="description"\r
Content-Length: 27\r
Content-Transfer-Encoding: 8bit\r
Content-Type: text/plain; charset=UTF-8\r
\r
a JSON file to test parsers\r
--a25537b6-51d6-491f-b18b-a33d197ae4c6--\r

And here’s this request issued with the client library:

>> (http.post "http://httpbin.org/post"
              {:multipart [{:name "json-test"
                            :content (io.open "test/data/valid.json")
                            :filename "test.json"}
                           {:name "description" :content "a JSON file to test parsers"}]
               :as :json})
{:body {:args {}
        :data ""
        :files {:json-test "..."} ;; contents were excluded to save some vertical space
        :form {:description "a JSON file to test parsers"}
        :headers {:Content-Length "1946"
                  :Content-Type "multipart/form-data; boundary=------------5af9b616-6e9d-4c19-8aa5-cc0c5f52ca11"
                  :Host "httpbin.org"
                  :X-Amzn-Trace-Id "Root=1-66aad04d-5155e53a4023b5726bdd134e"}
        :url "http://httpbin.org/post"}
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "2118"
           :Content-Type "application/json"
           :Date "Thu, 01 Aug 2024 00:01:18 GMT"
           :Server "gunicorn/19.9.0"}
 :http-client #<SocketChannel: 0x559596d0e200>
 :length 2118
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 388
 :status 200}

As you can see, the JSON got into the files field, and the description into the form field.

Redirects and query-params

Honestly, implementing redirection was a pain in the butt. But I’m glad that I did!

First, I now finally understand how redirection works. Second, I had to do a major code refactoring for it, which made some parts more approachable, and robust. That was a lot of changes, though.

Alongside redirects, I improved the URL parser and implemented a query builder, that can merge multiple queries into one. Here’s an example, again with httpbin.org:

>> (http.get "http://httpbin.org/redirect-to?status_code=302"
             {:query-params {:url "http://httpbin.org/get"}
              :as :json})
{:body {:args {}
        :headers {:Host "httpbin.org"
                  :X-Amzn-Trace-Id "Root=1-66aad21a-4c52d9316b88f5d862a4b535"}
        :url "http://httpbin.org/get"}
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "199"
           :Content-Type "application/json"
           :Date "Thu, 01 Aug 2024 00:08:58 GMT"
           :Server "gunicorn/19.9.0"}
 :http-client #<SocketChannel: 0x559596734da0>
 :length 199
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :request-time 394
 :status 200}

As can be seen, the query params in the URL string and the query-params field in the options table were merged. During the merge, all values are URL-encoded, so you can pass in URLs that are not entirely correct, and it should work.

Of course, we can change how we deal with redirects, limiting the total amounts of redirects:

>> (http.get "http://httpbin.org/absolute-redirect/4"
             {:max-redirects 3
              :as :json})
too many redirecs

Or, we can choose not to follow redirects altogether:

>> (http.get "http://httpbin.org/absolute-redirect/4"
             {:follow-redirects? false})
{:body "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: <a href=\"http://httpbin.org/absolute-redirect/3\">http://httpbin.org/absolute-redirect/3</a>.  If not click the link."
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "283"
           :Content-Type "text/html; charset=utf-8"
           :Date "Thu, 01 Aug 2024 00:24:48 GMT"
           :Location "http://httpbin.org/absolute-redirect/3"
           :Server "gunicorn/19.9.0"}
 :http-client #<SocketChannel: 0x5589fcb69530>
 :length 283
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "FOUND"
 :request-time 228
 :status 302}

Changes in the asynchronous API

I’ve decided to make it easier for users of this library to work with the client’s asynchronous API, and you should no longer be required to use async.fnl directly. Instead, every request function now accepts callbacks for processing responses, and exceptions:

(http.get "https://lua-users.org"
          {:async? true}
          (fn on-response [resp] (print resp.body))
          (fn on-raise [err] (print err.status)))

This means, that the only need for async.fnl is if you need to wait for certain requests to complete. It also means, that the http.METHOD functions will no longer return a promise channel, as I promised in the previous post.

In addition to that, it is now possible to read the response body, returned as a stream on the main thread, even if the request was asynchronous. This is possible because I’ve cheated:

(fn <!? [port]
  "Takes a value from `port`.  Will return `nil` if closed.  Will block,
if nothing is available and used on the main thread.  Will park if
used in the `(go ...)` block."
  (if (main-thread?)
      (<!! port)
      (<! port)))

(fn >!? [port val]
  "Puts a `val` into `port`.  `nil` values are not allowed.  Will block,
if no buffer space or consumer are available and used on the main
thread.  Will park if used inside a `(go ...)` block.  Returns `true`
unless `port` is already closed."
  (if (main-thread?)
      (>!! port val)
      (>! port val)))

Now try to do something like that in Clojure! You can read why this works in my post on async.fnl, or watch my talk at Fennelconf 2023 where I explain this with some more examples.

Performance check

Since the client got bigger and more complicated, let’s check the performance! In the previous post, the client couldn’t accept a chunked response, but now it can, meaning we can compare it with Luasocket’s HTTP client:

>> (local lhttp (require :socket.http))
nil
>> (= (lhttp.request "http://lua-users.org/wiki/FiltersSourcesAndSinks")
      (. (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks") :body))
true
>>

Let’s measure the time again:

>> (time (do (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks") nil))
Elapsed 501.31988525391 ms
nil
>> (time (do (lhttp.request "http://lua-users.org/wiki/FiltersSourcesAndSinks") nil))
Elapsed 630.47289848328 ms
nil

It’s about the same. However, we can be slightly faster, if the user doesn’t need to process the body immediately:

>> (time (do (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks" {:as :stream}) nil))
Elapsed 317.51012802124 ms
nil

Of course, the time is saved by not reading the body of the response, which shows that reading the body alone is about 200ms. But Luasocket’s HTTP client can’t do that at all - it always processes the body, even if the generic form is used and a sink is supplied. That’s an advantage, I would say!

And, thanks to the asynchronous processing of requests, we can actually issue multiple requests at the same time:

>> (let [n-requests 10
         ch (chan)]
     (print ";;; luasocket")
     (time
      (let [start (gettime)]
        (for [i 1 n-requests]
          ;; issuing a synchronous request
          (lhttp.request "http://lua-users.org/wiki/FiltersSourcesAndSinks")
          (io.write i " response processed in " (- (gettime) start)
                    " ms since the first request\n"))))
     (print ";;; fnl-http")
     (time
      (let [start (gettime)]
        (for [i 1 n-requests]
          ;; issuing multiple requests. The on-success handler
          ;; will put the request number into the channel `ch`
          ;; on completion.
          (http.get "http://lua-users.org/wiki/FiltersSourcesAndSinks"
                    {:async? true} #(>! ch [i (- (gettime) start)]) #nil))
        (for [i 1 n-requests]
          ;; avaiting each response
          (let [[i ms] (<!! ch)]
            (io.write i " response processed in " ms
                      " ms since the first request\n"))))))
;;; luasocket
1 response processed in 0.55836296081543 ms since the first request
2 response processed in 1.0487260818481 ms since the first request
3 response processed in 1.4974920749664 ms since the first request
4 response processed in 1.9714229106903 ms since the first request
5 response processed in 2.4879760742188 ms since the first request
6 response processed in 3.010272026062 ms since the first request
7 response processed in 3.6146268844604 ms since the first request
8 response processed in 4.1402359008789 ms since the first request
9 response processed in 4.6472299098969 ms since the first request
10 response processed in 5.1433019638062 ms since the first request
Elapsed 5143.6059474945 ms
;;; fnl-http
5 response processed in 1.3388519287109 ms since the first request
7 response processed in 1.4401869773865 ms since the first request
9 response processed in 1.5485229492188 ms since the first request
4 response processed in 1.6689200401306 ms since the first request
8 response processed in 1.7789130210876 ms since the first request
6 response processed in 1.9086010456085 ms since the first request
1 response processed in 2.0185270309448 ms since the first request
10 response processed in 2.1092948913574 ms since the first request
2 response processed in 2.2185280323029 ms since the first request
3 response processed in 2.3184778690338 ms since the first request
Elapsed 2318.8180923462 ms

Still ‘bout twice as fast!

And, as you can see, the order of request completion in the asynchronous case is different, since multiple requests were processed in parallel. Well, it’s not really parallel execution, since the Lua runtime only has one thread, and all response reading is synchronous, but the moment we’ve written the request into the socket, the operating system, and networking hardware could act independently. Thus, the response was sent, and the server on the other side received a series of requests, and probably processed them in parallel, resulting in a faster overall response. While we wait for the data to come back to our socket, other tasks can be executed, such as making another request or processing a part of the response. Of course, when we process the response, we do it asynchronously as well, creating opportunities for other asynchronous threads to collaborate.

Neat stuff!

Even further development, I guess?

When I started this project, I just wanted to use async.fnl for something more real and see if it works. After working on it for about a month, and reading the source code of the clj-http client, this project gradually became something like a re-implementation of clj-http in Fennel. So I changed the project name to fnl-http. It’s not a one-to-one port like async.fnl where I studied the source code - here all code is written from scratch without looking at other clients, but still, I tried to make it faithful.

There are still a lot of parts of HTTP/1.1 that I haven’t processed, and I still have some refactoring ideas and TODOs, so stay tuned!

Comment via email

Asynchronous HTTP client for Fennel

Mon, 15 Jul 2024 20:08:00 +0300

A while ago, I made a library for asynchronous programming in Fennel. It’s based on Clojure’s core.async vision of asynchronous programming using only channels. As an experiment, I’ve added a TCP support layer in that library, allowing one to create a TCP channel, and use it in the same way as a regular channel. My original plan was to implement a small asynchronous HTTP client using that, and I’ve tried it a bunch of times in the last year.

A bit later, I felt unusually burned out from programming as a hobby. In April, I made a post about hobbies, saying that I’d like to switch my hobbies from computers for some time. Truth be told, I’ve been in this stance since February, and only now do I once again feel some interest in programming as a hobby.

So today, I’ll walk you through why I decided to write an HTTP client almost from scratch, and why Luasocket’s HTTP client didn’t work out.

Luasocket’s HTTP client

So, there’s an HTTP client in the luasocket library. And while it does support streaming the body through the LTN12 module, the responses are always synchronous. Which means, that if you have a big enough body, that you’d like to process on demand, you only get the convenience of a stream that you’ve provided as a sink. I’d like the body to be returned as some kind of stream that you can process later, reading data from the client, not from some kind of buffer.

The first thing I tried was patching http.open function at runtime. Then we can open the connection and change the close method on the socket, to work with our intentions. Here’s a piece of code that does this:

(let [{: open : request &as http} (require :socket.http)]
  (fn http.open [...] ;; replacing `open` with our version
    (let [h (open ...) ;; using the original `open` inside
          close (. (getmetatable h) :__index :close)]
      (set close-handler #(close h))
      (fn h.close [self]
        (if (not got-body)
            (close-handler self)
            1))
      h))
  (let [(ok? code resp) (request opts)] ;; "thankfully" `http.request`
                                        ;; resolves `http.open`
                                        ;; dynamically
    (set http.open open) ;; restoring the original `http.open`
    (if ok?
        {:status code
         :headers resp
         :body out} ;; the body is now a channel
        (error code))))

There are a lot of missing details, for example, the out is an ordinary channel, however, there’s a sink that passed via the opts table that pumps everything to it on demand. Once the sink is drained the connection is closed.

All in all, this is a very stateful and shaky approach, so I didn’t want to rely on that. Furthermore, the sink is being pumped in chunks that can’t be controlled very well, making it harder to work with the data in a meaningful way. So I knew I should do something else.

I decided to write my own HTTP client based on Luasocket sockets, and streams. Also, I wanted a bit more familiar calling interface, because Luasocket’s HTTP client is not pleasant to use in my opinion. So let’s start with the HTTP parser!

Reader

Actually, before parsing HTTP, we need to be able to read data from a stream. Sockets are kinda like streams already, but we’re not there yet, so let’s create a string reader:

>> (string-reader "foo\nbar\nbaz")
#<Reader: 0x55f6ca3475b0>

The string-reader function accepts a string, and returns a Reader object that provides the following methods:

read - returns a specified amount of bytes, determined either by the number of bytes or by a supported read pattern.
lines - returns a function that when called will return the next logical line.
peek - returns a specified amount of bytes, without moving the caret.
close - closes the Reader.

Here’s the implementation of string-reader:

(fn string-reader [string]
  (var (i closed) (values 1 false))
  (let [len (length string)
        try-read-line (fn [s pattern]
                        (case (s:find pattern i)
                          (start end s)
                          (do (set i (+ end 1)) s)))
        read-line (fn [s]
                    (when (<= i len)
                      (or (try-read-line s "(.-)\r?\n")
                          (try-read-line s "(.-)\r?$"))))]
    (make-reader
     string
     {:close (fn [_]
               (when (not closed)
                 (set i (+ len 1))
                 (set closed true)
                 closed))
      :read-bytes (fn [s pattern]
                    (when (<= i len)
                      (case pattern
                        (where (or :*l :l))
                        (read-line s)
                        (where (or :*a :a))
                        (s:sub i)
                        (where bytes (= :number (type bytes)))
                        (let [res (s:sub i (+ i bytes -1))]
                          (set i (+ i bytes))
                          res))))
      :read-line read-line
      :peek (fn [s pattern]
              (when (<= i len)
                (case pattern
                  (where bytes (= :number (type bytes)))
                  (let [res (s:sub i (+ i bytes -1))]
                    res)
                  _ (error "expected number of bytes to peek"))))})))

It’s a bit big, but there’s a reason for that - strings in Lua are immutable, and we have to maintain the state when reading. What makes this more complicated is the peek method, but we’ll get to that later.

The make-reader function is a generic Reader constructor, that accepts a table of optional methods:

(fn ok? [ok? ...] (when ok? ...))

(fn make-reader [source {: read-bytes : read-line : close : peek}]
  (let [close (if close
                  (fn [_ ...]
                    (ok? (pcall close source ...)))
                  #nil)]
    (-> {: close
         :read (if read-bytes
                   (fn [_ pattern ...]
                     (ok? (pcall read-bytes source pattern ...))) #nil)
         :lines (if read-line
                    (fn []
                      (fn [_ ...]
                        (ok? (pcall read-line source ...))))
                    (fn [] #nil))
         :peek (if peek
                   (fn [_ pattern ...]
                     (ok? (pcall peek source pattern ...)))
                   #nil)}
        (setmetatable
         {:__close close
          :__name "Reader"
          :__fennelview #(.. "#<" (: (tostring $) :gsub "table:" "Reader:") ">")}))))

As you can see, it just wraps everything with pcall and passes the specified source to the methods. Perhaps, there’s a more clear way of doing this, but it’s fine for now.

Here’s a file-reader, for example:

(fn file-reader [file]
  (let [file (case (type file)
               :string (io.open file :r)
               _ file)]
      (make-reader file
                   {:close #(: $ :close)
                    :read-bytes (fn [f pattern] (f:read pattern))
                    :read-line (file:lines)
                    :peek (fn [f pattern]
                            (assert (= :number (type pattern)) "expected number of bytes to peek")
                            (let [res (f:read pattern)]
                              (f:seek :cur (- pattern))
                              res))})))

It accepts either a file handle or a string that is treated as a path to open the file. It’s much shorter than the string-reader since files can be seek’d and the file object is already stateful.

In the end, we can do this:

>> (local rdr (reader.string-reader "foo\nbar\nbaz\nqux"))
nil
>> (rdr:read 2)
"fo"
>> (rdr:read :*l)
"o"
>> (rdr:read :*l)
"bar"
>> (rdr:read :*a)
"baz
qux"

We’ll be implementing a socket Reader soon, but before that, let’s start implementing a set of functions to generate HTTP messages.

HTTP generation

We start with the build-http-request function:

(local HTTP-VERSION "HTTP/1.1")

(fn build-http-request [method request-target ?headers ?content]
  "Formaths the HTTP request string as per the HTTP/1.1 spec."
  (string.format
   "%s %s %s\r\n%s\r\n%s"
   (string.upper method)
   request-target
   HTTP-VERSION
   (or (headers->string ?headers) "")
   (or ?content "")))

As can be seen, it’s just a string.format call, which we fill in with specific parts. The method and request-target are mandatory arguments, as nothing will work without them. ?headers and ?content on the other hand are optional. Let’s implement the headers->string function now:

(fn header->string [header value]
  "Converts `header` and `value` arguments into a valid HTTP header
string."
  (.. (utils.capitalize-header header) ": " (tostring value) "\r\n"))

(fn headers->string [headers]
  "Converts a `headers` table into a multiline string of HTTP headers."
  (when (and headers (next headers))
    (-> (icollect [header value (pairs headers)]
          (header->string header value))
        table.concat)))

We simply check whether headers is not nil, and if it has any keys. If so, we iterate over the headers table, and convert each pair to a string, then concatenating the resulting table:

>> (headers->string {:connection "close" :content-type "application/json"})
"Content-Type: application/json\r
Connection: close\r
"

The utils.capitalize-header function accepts a string in any case, and transforms it to an HTTP case, meaning each word is capitalized and separated with a dash:

>> (utils.capitalize-header "foo_bar")
"Foo-Bar"
>> (utils.capitalize-header "FooBar")
"Foo-Bar"
>> (utils.capitalize-header "FOO BAR")
"Foo-Bar"

I’ll let you think about implementation, as I’m not particularly fond of my own :)

With that, we can generate a request:

>> (build-http-request
    :get "/"
    {:host "localhost:1234"
     :connection "close"
     :content-type "text/plain"}
    "foobar")
"GET / HTTP/1.1\r
Content-Type: text/plain\r
Host: localhost:1234\r
Connection: close\r
\r
foobar"

While we’re at it, let’s make the build-http-response function too:

(fn build-http-response [status reason ?headers ?content]
  "Formats the HTTP response string as per the HTTP/1.1 spec."
  (string.format
   "%s %s %s\r\n%s\r\n%s"
   HTTP-VERSION
   (tostring status)
   reason
   (or (headers->string ?headers) "")
   (or ?content "")))

Same as before, string.format, filling missing parts, no new functions used:

>> (build-http-response
    200 "OK"
    {:connection "close"
     :content-type "application/json"}
    "{\"foo\": \"bar\"}")
"HTTP/1.1 200 OK\r
Content-Type: application/json\r
Connection: close\r
\r
{\"foo\": \"bar\"}"

Now, we can parse these using our Reader objects.

HTTP Parsing

Let’s start with parse-http-response, as we’re more likely to parse responses, rather than requests, as this library is mainly a client. We’ll write parse-http-request as well, but just for fun.

(fn parse-http-response [src read-fn]
  (let [status (read-response-status-line src read-fn)
        headers (read-headers src read-fn)]
    (doto status
      (tset :headers headers)
      (tset :body src))))

Obviously, this isn’t all we really need, but it’s a good start. First, let’s parse the status line, i.e. the first line of the response: HTTP/1.1 200 OK

(fn parse-response-status-line [status]
  ((fn loop [reader fields res]
     (case fields
       [field & fields]
       (let [part (reader)]
         (loop reader fields
               (case field
                 :protocol-version
                 (let [(name major minor) (part:match "([^/]+)/(%d).(%d)")]
                   (doto res
                     (tset field {: name :major (tonumber major) :minor (tonumber minor)})))
                 _ (doto res
                     (tset field (utils.as-data part))))
               ))
       _
       (let [reason (-> "%s/%s.%s +%s +"
                        (string.format
                         res.protocol-version.name
                         res.protocol-version.major
                         res.protocol-version.minor
                         res.status)
                        (status:gsub ""))]
         (doto res
           (tset :reason-phrase reason)))))
   (status:gmatch "([^ ]+)")
   [:protocol-version :status]
   {}))

(fn read-response-status-line [src read-fn]
  (parse-response-status-line (read-fn src :*l)))

You may be wondering, why these functions accept read-fn. Well, this is something we need to do because we’ll later pass in a socket-channel as a stream.

With these functions, we can read and parse the status line:

(read-response-status-line
 (reader.string-reader "HTTP/1.1 200 OK\r\n")
 (fn [src pattern] (src:read pattern)))
{:protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

You may also wonder, why the loop doesn’t just continue to read the reason-phrase, instead relying on status:gsub. Well, reason can include spaces:

>> (read-response-status-line
    (reader.string-reader "HTTP/1.1 404 NOT FOUND\r\n")
    (fn [src pattern] (src:read pattern)))
{:protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "NOT FOUND"
 :status 404}

Onto headers:

(fn parse-header [line]
  (case (line:match " *([^:]+) *: *(.*)")
    (header value) (values header value)))

(fn read-headers [src read-fn ?headers]
  (let [headers (or ?headers {})
        line (read-fn src :*l)]
    (case line
      (where (or "\r" ""))
      headers
      _ (read-headers
         src
         read-fn
         (case (parse-header (or line ""))
           (header value)
           (doto headers (tset header value)))))))

This one is tail-recursive, accumulating parsed headers in the ?headers table:

>> (read-headers
    (reader.string-reader "Content-Type: application/json\r\nConnection: close\r\n\n\r")
    (fn [src pattern] (src:read pattern)))
{:Connection "close" :Content-Type "application/json"}

Finally, for the body, we just return the reader as it is, as we’ve already read from it all the necessary data, and the rest are the contents of the request. Together, we get a fully parsed response:

>> (parse-http-response
    (reader.string-reader "HTTP/1.1 200 OK\r
Content-Type: application/json\r
Connection: close\r
\r
{\"foo\": \"bar\"}")
    (fn [src pattern] (src:read pattern))
    )
{:body #<Reader: 0x55a54832b810>
 :headers {:Connection "close" :Content-Type "application/json"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

We can ensure that the parsed response can be encoded back:

>> (let [req (build-http-response 200 "OK" {:connection :close} "foobar")
         rdr (reader.string-reader req)
         {: body : headers : reason-phrase : status}
         (parse-http-response rdr (fn [src pattern] (src:read pattern)) {:as :raw})]
     (= req (build-http-response status reason-phrase headers body)))
true

Now, let’s parse the request for completeness:

(fn parse-request-status-line [status]
  ((fn loop [reader fields res]
     (case fields
       [field & fields]
       (let [part (reader)]
         (loop reader fields
               (doto res
                 (tset field (utils.as-data part)))))
       _ res))
   (status:gmatch "([^ ]+)")
   [:method :path :http-version]
   {}))

(fn read-request-status-line [src read-fn]
  (parse-request-status-line (read-fn src :*l)))

(fn parse-http-request [src read-fn]
  (let [status (read-request-status-line src read-fn)
        headers (read-headers src read-fn)]
    (doto status
      (tset :headers headers)
      (tset :content (read-fn src :*a)))))

It’s similar, but we need to parse the status line a bit differently. Same as before, let’s ensure that we can do both:

>> (let [req (build-http-request :get "/" {:connection :close} "foobar")
       rdr (reader.string-reader req)
       {: headers : method : path : content}
       (parse-http-request rdr (fn [src pattern] (src:read pattern)) rdr)]
   (= req (build-http-request method path headers content)))
true

Now, we can start implementing the client itself!

Async.fnl

For communication, we’ll use already existing TCP channels, that enclose socket objects. Although, we’ll need a small modification to the async.fnl library, as currently it always reads data in chunks of fixed size.

Here’s the function that returns a socket channel:

(fn set-chunk-size [self pattern-or-size]
  (set self.chunk-size pattern-or-size))

(fn socket-channel [client xform err-handler]
  (let [recv (chan 1024 xform err-handler)
        resp (chan 1024 xform err-handler)
        ready (chan)
        close (fn [self] (recv:close!) (resp:close!) (set self.closed true))
        c (-> {:puts recv.puts
               :takes resp.takes
               :put! (fn [_  val handler enqueue?]
                       (recv:put! val handler enqueue?))
               :take! (fn [_ handler enqueue?]
                        (offer! ready :ready)
                        (resp:take! handler enqueue?))
               :close! close
               :close close
               :chunk-size 1024
               :set-chunk-size set-chunk-size}
              (setmetatable
               {:__index Channel
                :__name "SocketChannel"
                :__fennelview
                #(.. "#<" (: (tostring $) :gsub "table:" "SocketChannel:") ">")}))]
    (go-loop [data (<! recv) i 0]
      (when (not= nil data)
        (case (socket.select nil [client] 0)
          (_ [s])
          (case (s:send data i)
            (nil :timeout j)
            (do (<! (timeout 10)) (recur data j))
            (nil :closed)
            (do (s:close) (close! c))
            _ (recur (<! recv) 0))
          _ (do (<! (timeout 10))
                (recur data i)))))
    (go-loop [wait? true]
      (when wait?
        (<! ready))
      (case (client:receive c.chunk-size)
        data
        (do (>! resp data) (recur true))
        (nil :closed "")
        (do (client:close) (close! c))
        (nil :closed data)
        (do (client:close) (>! resp data) (close! c))
        (nil :timeout "")
        (do (<! (timeout 10)) (recur false))
        (nil :timeout data)
        (do (>! resp data) (<! (timeout 10)) (recur true))))
    c))

It’s a lot of code, but in reality, it isn’t that complicated. The socket-channel function is responsible for three things:

Creating a channel, that has separate queues for puts and takes, such that when we put something to a channel, we won’t be able to read it back immediately.
Creating an asynchronous loop that waits for data from the socket, and puts it into the receive queue
Creating another asynchronous loop, that waits data on the channel, and puts it into the socket.

These loops account for timeouts, as we’re going to set the socket timeout to 0, to ensure non-blocking behavior. Both asynchronous loops employ their own timeout handling.

The resulting object is a channel, but with some custom methods to account for its duplex nature. And, there are also other properties, such as chunk-size and set-chunk-size which we’re going to use with our streams.

Notice, how the second go-loop has a wait? argument. Because the channel is buffered, it will happily read everything until the buffer is full, which is not ideal, since we may want to change the chunk-size later. So we need to delay the actual read, which we do in the take! method using offer!.

Finally, the tcp.chan function acts as a constructor for TCP channels:

(fn tcp.chan [{: host : port} xform err-handler]
  "Creates a channel that connects to a socket via `host` and `port`.
Optionally accepts a transducer `xform`, and an error handler.
`err-handler` must be a fn of one argument - if an exception occurs
during transformation it will be called with the thrown value as an
argument, and any non-nil return value will be placed in the channel.
The read pattern f a socket can be controlled with the
`set-chunk-size` method."
  (assert socket "tcp module requires luasocket")
  (let [host (or host :localhost)]
    (match-try (socket.connect host port)
      client (client:settimeout 0)
      _ (socket-channel client xform err-handler)
      (catch (nil err) (error err)))))

This is all we need on the async.fnl side of things. No more patching Luasocket’s HTTP client.

Now we can start implementing our own client.

HTTP client

We already have most of the things we need, but we need to change the parse-http-response a bit. Remember, our function returns the body as the Reader, but our source is actually a channel. We can account for that by passing in an appropriate read-fn function but our users shouldn’t deal with that. Instead, the body should provide proper Reader methods, so let’s create a body-reader:

(fn body-reader [src read-fn]
  (var buffer "")
  (make-reader
   src
   {:read-bytes (fn [src pattern]
                  (let [rdr (reader.string-reader buffer)
                        buffer-content (rdr:read pattern)]
                    (case pattern
                      (where n (= :number (type n)))
                      (let [len (if buffer-content (length buffer-content) 0)
                            read-more? (< len n)]
                        (set buffer (string.sub buffer (+ len 1)))
                        (if read-more?
                            (if buffer-content
                                (.. buffer-content (or (read-fn src (- n len)) ""))
                                (read-fn src (- n len)))
                            buffer-content))
                      (where (or :*l :l))
                      (let [read-more? (not (buffer:find "\n"))]
                        (when buffer-content
                          (set buffer (string.sub buffer (+ (length buffer-content) 2))))
                        (if read-more?
                            (if buffer-content
                                (.. buffer-content (or (read-fn src pattern) ""))
                                (read-fn src pattern))
                            buffer-content))
                      (where (or :*a :a))
                      (do (set buffer "")
                          (case (read-fn src pattern)
                            nil (when buffer-content
                                  buffer-content)
                            data (.. (or buffer-content "") data)))
                      _ (error (tostring pattern)))))
    :read-line (fn [src]
                 (let [rdr (reader.string-reader buffer)
                       buffer-content (rdr:read :*l)
                       read-more? (not (buffer:find "\n"))]
                   (when buffer-content
                     (set buffer (string.sub buffer (+ (length buffer-content) 2))))
                   (if read-more?
                       (if buffer-content
                           (.. buffer-content (or (read-fn src :*l) ""))
                           (read-fn src :*l))
                       buffer-content)))
    :close (fn [src] (src:close))
    :peek (fn [src bytes]
            (assert (= :number (type bytes)) "expected number of bytes to peek")
            (let [rdr (reader.string-reader buffer)
                  content (or (rdr:read bytes) "")
                  len (length content)]
              (if (= bytes len)
                  content
                  (let [data (read-fn src (- bytes len))]
                    (set buffer (.. buffer (or data "")))
                    buffer))))}))

Now, that’s a lot of code again. The reason for that is the peek method - same as for strings, we need to implement a way to peek at data without modifying the reader, or at least hide the modification from the user. This is essentially what this function does - it creates a string buffer that we can act upon, accounting for possible data in it before actually reading from the channel itself.

With that function, we can update parse-http-response:

(fn parse-http-response [src read-fn {: as}]
  (let [status (read-response-status-line src read-fn)
        headers (read-headers src read-fn)
        stream (body-reader src read-fn)]
    (doto status
      (tset :headers headers)
      (tset :body
            (case as
              :raw (stream:read (or headers.Content-Length :*a))
              :stream stream
              _ (error (string.format "unsupported coersion method '%s'" as)))))))

As you can see, I’ve added another parameter with options. Right now, there’s only one option as that will choose how to return the body. Specifying {:as :raw} to this function will return the contents of the reader. And {:as :stream} will return the stream itself.

Now, we can write the request function:

(local http {})

(fn http.request [method url ?opts]
  "Makes a `method` request to the `url`, returns the parsed response,
containing a stream data of the response. The `method` is a string,
describing the HTTP method per the HTTP/1.1 spec. The `opts` is a
table containing the following keys:

- `:async?` - a boolean, whether the request should be asynchronous.
  The result is an instance of a `promise-chan`, and the body must
  be read inside of a `go` block.
- `:headers` - a table with the HTTP headers for the request
- `:body` - an optional string body.
- `:as` - how to coerce the output.

Several options available for the `as` key:

- `:stream` - the body will be a stream object with a `read` method.
- `:raw` - the body will be a string.
  This is the default value for `as`.

When supplying a non-string body, headers should contain a
\"content-length\" key. For a string body, if the \"content-length\"
header is missing it is automatically determined by calling the
`length` function, ohterwise no attempts at detecting content-length
are made."
  (let [{: host : port &as parsed} (http-parser.parse-url url)
        opts (collect [k v (pairs (or ?opts {}))
                       :into {:as :raw
                              :async? false}]
               k v)
        headers (collect [k v (pairs (or opts.headers {}))
                          :into {:host
                                 (.. host (if port (.. ":" port) ""))
                                 :content-length
                                 (case opts.body
                                   (where body (= :string (type body)))
                                   (length body))}]
                  k v)
        path (utils.format-path parsed)
        req (build-http-request method path headers opts.body)
        chan (tcp.chan parsed)]
    (if opts.async?
        (let [res (promise-chan)]
          (go (>! chan req)
              (>! res (http-parser.parse-http-response
                       chan
                       (make-read-fn <!)
                       opts)))
          res)
        (do (>!! chan req)
            (http-parser.parse-http-response
             chan
             (make-read-fn <!!)
             opts)))))

We’re still missing some important parts, but we’ll get to those in a bit. This function is long, but it’s not that complicated.

First, we parse the given url into its components, mainly the host and port. Then, we prepare the default opts table to include as key and async? key if none were passed.

Next, we prepare headers. The HTTP spec requires the Host header, so we put it in based on the parsed URL. The path part of the request is then formatted from the rest data from the parsed URL.

Finally, we build-http-request, using the method, path, headers and body if supplied. And create the tcp.chan using the parsed object that contains host and port needed for the constructor.

Then, if opts.async? is true, we create a promise-chan, which is like Clojure’s promise, which is only computed once, and then returns the same result over and over. We start the go thread, in which we asynchronously put the request into the socket channel, and then we asynchronously parse the response supplying our read-fn with the <! function to take from the channel asynchronously. And we return res - the promise channel I mentioned earlier.

If the opts.async? wasn’t true, we do all the same thing, but in a blocking way, using >!! and <!! to work with the channel.

Let’s look at parse-url:

(fn parse-authority [authority]
  (let [userinfo (authority:match "([^@]+)@")
        port (authority:match ":(%d+)")
        host (if userinfo
                 (authority:match (.. "@([^:]+)" (if port ":" "")))
                 (authority:match (.. "([^:]+)" (if port ":" ""))))]
    {: userinfo : port : host}))

(fn parse-url [url]
  (let [scheme (url:match "^([^:]+)://")
        {: host : port : userinfo}
        (parse-authority
         (if scheme
             (url:match "//([^/]+)/")
             (url:match "^([^/]+)/")))
        scheme (or scheme "http")
        port (or port (case scheme :https 443 :http 80))
        path (url:match "//[^/]+/([^?#]+)")
        query (url:match "%?([^#]+)#?")
        fragment (url:match "#([^?]+)%??")]
    {: scheme : host : port : userinfo : path : query : fragment}))

Nothing special, just some patterns to match components of the URL:

>> (parse-url "https://lua-users.org/?some-query-param=some-value")
{:host "lua-users.org"
 :port 443
 :query "some-query-param=some-value"
 :scheme "https"}

And the format-path function constructs the path back:

(fn format-path [{: path : query : fragment}]
  (.. "/" (or path "") (if query (.. "?" query) "") (if fragment (.. "?" fragment) "")))

Here’s what it does on the previously parsed URL:

>> (format-path {:host "lua-users.org"
 :port 443
 :query "some-query-param=some-value"
 :scheme "https"})
"/?some-query-param=some-value"

Finally, the mysterious make-read-fn function creates a function that will work with our socket-channel:

(fn make-read-fn [receive]
  "Returns a function that receives data from a socket by a given
`pattern`.  The `pattern` can be either `\"*l\"`, `\"*a\"`, or a
number of bytes to read."
  (fn [src pattern]
    (src:set-chunk-size pattern)
    (receive src)))

Since our channel does reading on demand, we can’t pass the chunk-size/pattern in, so we use the set-chunk-size method instead. Paired with delayed read, I’ve explained earlier, this works really well.

With all of that, we’re ready to do some requests!

>> (http.request :get "http://lua-users.org/" {:headers {:connection "close"} :as :raw})
{:body "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">

<html>

<head>
<title>lua-users.org</title>
<meta http-equiv=\"content-type\" content=\"text/html; charset=ISO-8859-1\">
<LINK TYPE=\"text/css\" REL=\"stylesheet\" HREF=\"/styles/main.css\">
</head>

<body>

<img src=\"/images/lua-users-org.png\" alt=\"lua-users.org logo\">

<br clear=all>
<p>
<b>lua-users.org</b> is an internet site for and by users of the programming language Lua.
</p>
<p>
To find out what Lua is and for documentation, downloads, mailing list info, etc. please visit the <a href=\"http://www.lua.org/\">official Lua web site</a>.  The <b>lua-users.org</b> web is not affiliated with Tecgraf.
</p>
<p>
Currently available resources:
</p>
<ul>
    <li> <a href=\"/wiki/\">lua-users wiki</a> - collaborative web site
    <li> <a href=\"/lists/lua-l/\">lua-l archive</a> - searchable web archive of
        the official Lua mailing list
    <li> <a href=\"MiniCharter.html\">lua-users.org mini charter</a>
    <li> <a href=\"Acknowledgements.html\">acknowledgements</a>
</ul>

</body>

</html>
"
 :headers {:Accept-Ranges "bytes"
           :Connection "close"
           :Content-Length "1055"
           :Content-Type "text/html"
           :Date "Mon, 15 Jul 2024 12:03:00 GMT"
           :ETag "\"61acad05-41f\""
           :Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
           :Server "nginx/1.20.1"
           :Strict-Transport-Security "max-age=0;"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

Or, we can use :as :stream, and process the stream later:

(local resp (http.request :get "http://lua-users.org/" {:headers {:connection "close"} :as :stream}))
nil
>> resp
{:body #<Reader: 0x55d7090707c0>
 :headers {:Accept-Ranges "bytes"
           :Connection "close"
           :Content-Length "1055"
           :Content-Type "text/html"
           :Date "Mon, 15 Jul 2024 12:03:47 GMT"
           :ETag "\"61acad05-41f\""
           :Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
           :Server "nginx/1.20.1"
           :Strict-Transport-Security "max-age=0;"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}
>> (resp.body:read :*l)
"<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">"

However, writing (http.request :get ...) all the time, is a bit tedious. Let’s instead define functions for each HTTP method. And, to do so, we’ll use macros, because I’m too lazy, and I don’t want to write documentation for each function:

(macro define-http-method [method]
  "Defines an HTTP method for the given `method`."
  `(fn ,(sym (.. :http. (tostring method)))
     [url# opts#]
     {:fnl/arglist [url opts]
      :fnl/docstring ,(.. "Makes a `" (string.upper (tostring method))
                          "` request to the `url`, returns the parsed response,
containing a stream data of the response. The `method` is a string,
describing the HTTP method per the HTTP/1.1 spec. The `opts` is a
table containing the following keys:

- `:async?` - a boolean, whether the request should be asynchronous.
  The result is an instance of a `promise-chan`, and the body must
  be read inside of a `go` block.
- `:headers` - a table with the HTTP headers for the request
- `:body` - an optional string body.
- `:as` - how to coerce the output.

Several options available for the `as` key:

- `:stream` - the body will be a stream object with a `read` method.
- `:raw` - the body will be a string.
  This is the default value for `as`.

When supplying a non-string body, headers should contain a
\"content-length\" key. For a string body, if the \"content-length\"
header is missing it is automatically determined by calling the
`length` function, ohterwise no attempts at detecting content-length
are made.")}
     (http.request ,(tostring method) url# opts#)))

(define-http-method get)
(define-http-method post)
(define-http-method put)
(define-http-method patch)
(define-http-method options)
(define-http-method trace)
(define-http-method head)
(define-http-method delete)
(define-http-method connect)

Now, we can call any HTTP method by just writing (http.method-name ...):

>> (http.get "http://lua-users.org/" {:headers {:connection "close"} :as :stream})
{:body #<Reader: 0x55d7095dbb20>
 :headers {:Accept-Ranges "bytes"
           :Connection "close"
           :Content-Length "1055"
           :Content-Type "text/html"
           :Date "Mon, 15 Jul 2024 12:08:28 GMT"
           :ETag "\"61acad05-41f\""
           :Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
           :Server "nginx/1.20.1"
           :Strict-Transport-Security "max-age=0;"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

And we can even see the documentation for each method in our editor:

Now, that body-reader, with a custom buffer, seemed a bit out of the way, so let’s use it for good.

JSON parsing

Let’s extend our parse-http-response function with another as method:

;; ---- 8< ----
      (tset :body
            (case as
              :raw (stream:read (or parsed-headers.Content-Length :*a))
              :json (json.parse stream)
              :stream stream
              _ (error (string.format "unsupported coersion method '%s'" as))))
;; ---- 8< ----

Now, we can implement a simple JSON parser:

(fn parse [rdr]
  "Accepts a reader object `rdr`, that supports `peek`, and `read`
methods.  Parses the contents to a Lua table."
  ((fn loop []
     (case (rdr:peek 1)
       "{" (parse-obj rdr loop)
       "[" (parse-arr rdr loop)
       "\"" (parse-string rdr)
       (where "t" (= "true" (rdr:peek 4))) (do (rdr:read 4) true)
       (where "f" (= "false" (rdr:peek 5))) (do (rdr:read 5) false)
       (where "n" (= "null" (rdr:peek 4))) (do (rdr:read 4) nil)
       (where c (c:match "[ \t\n]")) (loop (skip-space rdr))
       (where n (n:match "[-0-9]")) (parse-num rdr)
       nil (error "JSON parse error: end of stream")
       _ (error (string.format "JSON parse error: unexpected token ('%s' (code %d))" _ (_:byte)))))))

First, let’s look at parse-string:

(fn parse-string [rdr]
  (rdr:read 1)
  ((fn loop [chars escaped?]
     (case (rdr:read 1)
       "\\" (loop chars (not escaped?))
       "\"" (if escaped?
                (loop (.. chars "\"") false)
                chars)
       _ (loop (.. chars _) false)
       ))
   "" false))

This function handles strings, and escaped patterns withing set strings, like nested strings:

>> (parse (reader.string-reader "\"foo \\\"bar\\\"\""))
"foo \"bar\""

Now, let’s look at the parse-arr

(fn parse-arr [rdr parse]
  (rdr:read 1)
  ((fn loop [arr]
     (skip-space rdr)
     (case (rdr:peek 1)
       "]" (do (rdr:read 1) arr)
       _ (let [val (parse)]
           (tset arr (+ 1 (length arr)) val)
           (skip-space rdr)
           (case (rdr:peek 1)
             "," (do (rdr:read 1) (loop arr))
             "]" (do (rdr:read 1) arr)
             _ (error (.. "JSON parse error: expected ',' or ']' after the value: " (json val)))))))
   []))

This function accepts the parse callback to parse nested values, so if we try to parse an array of strings, we’ll call parse on each value, and then expect either a colon or the closing bracket to mark the end of the array.

>> (parse (reader.string-reader "[\"a\", \"b\"]"))
["a" "b"]

The parse-obj works for objects in a similar way:

(fn parse-obj [rdr parse]
  (rdr:read 1)
  ((fn loop [obj]
     (skip-space rdr)
     (case (rdr:peek 1)
       "}" (do (rdr:read 1) obj)
       _ (let [key (parse)]
           (skip-space rdr)
           (case (rdr:peek 1)
             ":" (let [_ (rdr:read 1)
                       value (parse)]
                   (tset obj key value)
                   (skip-space rdr)
                   (case (rdr:peek 1)
                     "," (do (rdr:read 1) (loop obj))
                     "}" (do (rdr:read 1) obj)
                     _ (error (.. "JSON parse error: expected ',' or '}' after the value: " (json value)))))
             _ (error (.. "JSON parse error: expected colon after the key: " (json key)))))))
   {}))

Except, it waits for the : after each key, and then the value:

>> (parse (reader.string-reader "{\"arr\": [\"a\", \"b\"]}"))
{:arr ["a" "b"]}

Finally, there’s the parse-num and code to parse Booleans, and null:

(fn parse-num [rdr]
  ((fn loop [numbers]
     (case (rdr:peek 1)
       (where n (n:match "[-0-9.eE+]"))
       (do (rdr:read 1) (loop (.. numbers n)))
       _ (tonumber numbers)))
   (rdr:read 1)))

With that, we can call http.get passing it {:as :json} as opts:

>> (http.get "http://httpbin.org/get" {:as :json})
{:body {:args {}
        :headers {:Host "httpbin.org"
                  :X-Amzn-Trace-Id "Root=1-6695183d-27c59aff535ae9f11df8c98d"}
        :origin "185.169.167.87"
        :url "http://httpbin.org/get"}
 :headers {:Access-Control-Allow-Credentials "true"
           :Access-Control-Allow-Origin "*"
           :Connection "keep-alive"
           :Content-Length "199"
           :Content-Type "application/json"
           :Date "Mon, 15 Jul 2024 12:38:21 GMT"
           :Server "gunicorn/19.9.0"}
 :protocol-version {:major 1 :minor 1 :name "HTTP"}
 :reason-phrase "OK"
 :status 200}

And our body was successfully parsed.

Asynchronous HTTP calls

Now, this client is asynchronous, meaning we can download lots of resources simultaneously. I’m not saying in parallel, because there are no threads in Lua, but still, we can process several requests at once using our asynchronous library.

Let’s define a simple time macro first:

(macro time [expr]
  `(let [clock# socket.gettime
         start# (clock#)
         res# ,expr
         end# (clock#)]
     (print (.. "Elapsed " (* (- end# start#) 1000) " ms"))
     res#))

With it, we can measure the execution time of a body. Let’s make three asynchronous requests, that upon completion will count the lengths of the body for each, and return their sum:

>> (time
    (let [headers {:connection "close"}
          a (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
          b (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
          c (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
          lengths (chan 3)]
      (go (let [resp (<! a)]
            (>! lengths (length resp.body))))
      (go (let [resp (<! b)]
            (>! lengths (length resp.body))))
      (go (let [resp (<! c)]
            (>! lengths (length resp.body))))
      (+ (<!! lengths) (<!! lengths) (<!! lengths))))
Elapsed 530.24077415466 ms
597

In comparison, three synchronous requests, take almost twice as much time:

>> (time
    (let [headers {:connection "close"}
          a (http.get "http://httpbin.org/get" {:headers headers :as :raw})
          b (http.get "http://httpbin.org/get" {:headers headers :as :raw})
          c (http.get "http://httpbin.org/get" {:headers headers :as :raw})]
      (+ (length a.body) (length b.body) (length c.body))))
Elapsed 1047.5831031799 ms
597

For the reference, here’s Luasocket’s HTTP client:

>> (local lhttp (require :socket.http))
nil
>> (time
    (let [a (lhttp.request "http://httpbin.org/get")
          b (lhttp.request "http://httpbin.org/get")
          c (lhttp.request "http://httpbin.org/get")]
      (+ (length a) (length b) (length c))))
Elapsed 1919.6438789368 ms
717

(Note, the resulting length is different, because Luasocket specifies its user-agent, and httpbin includes it in the answer.)

Further development

Now, of course, that’s not all that needs to be implemented. In its current state, this client doesn’t support responses that do not specify the Content-Length or produce a response body in chunked encoding. HTTPS is out of the question for now too. Most of the code just takes a “happy path” without much thought given to errors or more complicated scenarios. But it’s fine, I can always make it better down the line.

But all in all, it was a fun experiment, and I’m feeling better, as I finally have some use for async.fnl. You can find the source code for this project on my GitLab profile. And by the way, even though it is written in Fennel and uses some Fennel libraries, like async.fnl, this library works with ordinary Lua:

Lua 5.4.4  Copyright (C) 1994-2022 Lua.org, PUC-Rio
> http = require "http"
> http.get("http://httpbin.org/get", {headers = {connection = "close"}}).body
{
  "args": {},
  "headers": {
    "Host": "httpbin.org",
    "X-Amzn-Trace-Id": "Root=1-6695424f-01e0510f2ab49b9f46bd51dc"
  },
  "origin": "185.165.163.78",
  "url": "http://httpbin.org/get"
}
> http.get("http://httpbin.org/get", {headers = {connection = "close"}, as = "stream"}).body
Reader: 0x55a6c0eff710

So if you like what you see, but you don’t use Fennel, you can still try out this library. It just needs to be compiled first.

Anyway, that’s all from me, so… thanks for reading!

Comment via email

Andrey Listopadov

Implementing SSE with fnl-http and async.fnl

SSE stream

Reading the event stream and triggering events

Conclusion1

Extending JSON encoder with custom types

Custom JSON encoders

Custom decoders

Foontones

Streaming and receiving chunked multipart data with fnl-http

HTTP Server

Multipart requests

Performance

fnl-http - testing with fennel-test

Reporters

A sudden bug

Another sudden bug!

Counting skipped tests and assertions

Coverage report

fnl-http Improvements

Chunked transfer-encoding

Multipart requests

Redirects and query-params

Changes in the asynchronous API

Performance check

Even further development, I guess?

Asynchronous HTTP client for Fennel

Luasocket’s HTTP client

Reader

HTTP generation

HTTP Parsing

Async.fnl

HTTP client

JSON parsing

Asynchronous HTTP calls

Further development

Conclusion¹