Asynchronous HTTP client for Fennel
A while ago, I made a library for asynchronous programming in Fennel.
It’s based on Clojure’s core.async vision of asynchronous programming using only channels.
As an experiment, I’ve added a TCP support layer in that library, allowing one to create a TCP channel, and use it in the same way as a regular channel.
My original plan was to implement a small asynchronous HTTP client using that, and I’ve tried it a bunch of times in the last year.
A bit later, I felt unusually burned out from programming as a hobby. In April, I made a post about hobbies, saying that I’d like to switch my hobbies from computers for some time. Truth be told, I’ve been in this stance since February, and only now do I once again feel some interest in programming as a hobby.
So today, I’ll walk you through why I decided to write an HTTP client almost from scratch, and why Luasocket’s HTTP client didn’t work out.
Luasocket’s HTTP client
So, there’s an HTTP client in the luasocket library.
And while it does support streaming the body through the LTN12 module, the responses are always synchronous.
Which means, that if you have a big enough body, that you’d like to process on demand, you only get the convenience of a stream that you’ve provided as a sink.
I’d like the body to be returned as some kind of stream that you can process later, reading data from the client, not from some kind of buffer.
The first thing I tried was patching http.open function at runtime.
Then we can open the connection and change the close method on the socket, to work with our intentions.
Here’s a piece of code that does this:
(let [{: open : request &as http} (require :socket.http)]
(fn http.open [...] ;; replacing `open` with our version
(let [h (open ...) ;; using the original `open` inside
close (. (getmetatable h) :__index :close)]
(set close-handler #(close h))
(fn h.close [self]
(if (not got-body)
(close-handler self)
1))
h))
(let [(ok? code resp) (request opts)] ;; "thankfully" `http.request`
;; resolves `http.open`
;; dynamically
(set http.open open) ;; restoring the original `http.open`
(if ok?
{:status code
:headers resp
:body out} ;; the body is now a channel
(error code))))
There are a lot of missing details, for example, the out is an ordinary channel, however, there’s a sink that passed via the opts table that pumps everything to it on demand.
Once the sink is drained the connection is closed.
All in all, this is a very stateful and shaky approach, so I didn’t want to rely on that.
Furthermore, the sink is being pumped in chunks that can’t be controlled very well, making it harder to work with the data in a meaningful way.
So I knew I should do something else.
I decided to write my own HTTP client based on Luasocket sockets, and streams. Also, I wanted a bit more familiar calling interface, because Luasocket’s HTTP client is not pleasant to use in my opinion. So let’s start with the HTTP parser!
Reader
Actually, before parsing HTTP, we need to be able to read data from a stream. Sockets are kinda like streams already, but we’re not there yet, so let’s create a string reader:
>> (string-reader "foo\nbar\nbaz")
#<Reader: 0x55f6ca3475b0>
The string-reader function accepts a string, and returns a Reader object that provides the following methods:
read- returns a specified amount of bytes, determined either by the number of bytes or by a supported read pattern.lines- returns a function that when called will return the next logical line.peek- returns a specified amount of bytes, without moving the caret.close- closes the Reader.
Here’s the implementation of string-reader:
(fn string-reader [string]
(var (i closed) (values 1 false))
(let [len (length string)
try-read-line (fn [s pattern]
(case (s:find pattern i)
(start end s)
(do (set i (+ end 1)) s)))
read-line (fn [s]
(when (<= i len)
(or (try-read-line s "(.-)\r?\n")
(try-read-line s "(.-)\r?$"))))]
(make-reader
string
{:close (fn [_]
(when (not closed)
(set i (+ len 1))
(set closed true)
closed))
:read-bytes (fn [s pattern]
(when (<= i len)
(case pattern
(where (or :*l :l))
(read-line s)
(where (or :*a :a))
(s:sub i)
(where bytes (= :number (type bytes)))
(let [res (s:sub i (+ i bytes -1))]
(set i (+ i bytes))
res))))
:read-line read-line
:peek (fn [s pattern]
(when (<= i len)
(case pattern
(where bytes (= :number (type bytes)))
(let [res (s:sub i (+ i bytes -1))]
res)
_ (error "expected number of bytes to peek"))))})))
It’s a bit big, but there’s a reason for that - strings in Lua are immutable, and we have to maintain the state when reading.
What makes this more complicated is the peek method, but we’ll get to that later.
The make-reader function is a generic Reader constructor, that accepts a table of optional methods:
(fn ok? [ok? ...] (when ok? ...))
(fn make-reader [source {: read-bytes : read-line : close : peek}]
(let [close (if close
(fn [_ ...]
(ok? (pcall close source ...)))
#nil)]
(-> {: close
:read (if read-bytes
(fn [_ pattern ...]
(ok? (pcall read-bytes source pattern ...))) #nil)
:lines (if read-line
(fn []
(fn [_ ...]
(ok? (pcall read-line source ...))))
(fn [] #nil))
:peek (if peek
(fn [_ pattern ...]
(ok? (pcall peek source pattern ...)))
#nil)}
(setmetatable
{:__close close
:__name "Reader"
:__fennelview #(.. "#<" (: (tostring $) :gsub "table:" "Reader:") ">")}))))
As you can see, it just wraps everything with pcall and passes the specified source to the methods.
Perhaps, there’s a more clear way of doing this, but it’s fine for now.
Here’s a file-reader, for example:
(fn file-reader [file]
(let [file (case (type file)
:string (io.open file :r)
_ file)]
(make-reader file
{:close #(: $ :close)
:read-bytes (fn [f pattern] (f:read pattern))
:read-line (file:lines)
:peek (fn [f pattern]
(assert (= :number (type pattern)) "expected number of bytes to peek")
(let [res (f:read pattern)]
(f:seek :cur (- pattern))
res))})))
It accepts either a file handle or a string that is treated as a path to open the file.
It’s much shorter than the string-reader since files can be seek’d and the file object is already stateful.
In the end, we can do this:
>> (local rdr (reader.string-reader "foo\nbar\nbaz\nqux"))
nil
>> (rdr:read 2)
"fo"
>> (rdr:read :*l)
"o"
>> (rdr:read :*l)
"bar"
>> (rdr:read :*a)
"baz
qux"
We’ll be implementing a socket Reader soon, but before that, let’s start implementing a set of functions to generate HTTP messages.
HTTP generation
We start with the build-http-request function:
(local HTTP-VERSION "HTTP/1.1")
(fn build-http-request [method request-target ?headers ?content]
"Formaths the HTTP request string as per the HTTP/1.1 spec."
(string.format
"%s %s %s\r\n%s\r\n%s"
(string.upper method)
request-target
HTTP-VERSION
(or (headers->string ?headers) "")
(or ?content "")))
As can be seen, it’s just a string.format call, which we fill in with specific parts.
The method and request-target are mandatory arguments, as nothing will work without them.
?headers and ?content on the other hand are optional.
Let’s implement the headers->string function now:
(fn header->string [header value]
"Converts `header` and `value` arguments into a valid HTTP header
string."
(.. (utils.capitalize-header header) ": " (tostring value) "\r\n"))
(fn headers->string [headers]
"Converts a `headers` table into a multiline string of HTTP headers."
(when (and headers (next headers))
(-> (icollect [header value (pairs headers)]
(header->string header value))
table.concat)))
We simply check whether headers is not nil, and if it has any keys.
If so, we iterate over the headers table, and convert each pair to a string, then concatenating the resulting table:
>> (headers->string {:connection "close" :content-type "application/json"})
"Content-Type: application/json\r
Connection: close\r
"
The utils.capitalize-header function accepts a string in any case, and transforms it to an HTTP case, meaning each word is capitalized and separated with a dash:
>> (utils.capitalize-header "foo_bar")
"Foo-Bar"
>> (utils.capitalize-header "FooBar")
"Foo-Bar"
>> (utils.capitalize-header "FOO BAR")
"Foo-Bar"
I’ll let you think about implementation, as I’m not particularly fond of my own :)
With that, we can generate a request:
>> (build-http-request
:get "/"
{:host "localhost:1234"
:connection "close"
:content-type "text/plain"}
"foobar")
"GET / HTTP/1.1\r
Content-Type: text/plain\r
Host: localhost:1234\r
Connection: close\r
\r
foobar"
While we’re at it, let’s make the build-http-response function too:
(fn build-http-response [status reason ?headers ?content]
"Formats the HTTP response string as per the HTTP/1.1 spec."
(string.format
"%s %s %s\r\n%s\r\n%s"
HTTP-VERSION
(tostring status)
reason
(or (headers->string ?headers) "")
(or ?content "")))
Same as before, string.format, filling missing parts, no new functions used:
>> (build-http-response
200 "OK"
{:connection "close"
:content-type "application/json"}
"{\"foo\": \"bar\"}")
"HTTP/1.1 200 OK\r
Content-Type: application/json\r
Connection: close\r
\r
{\"foo\": \"bar\"}"
Now, we can parse these using our Reader objects.
HTTP Parsing
Let’s start with parse-http-response, as we’re more likely to parse responses, rather than requests, as this library is mainly a client.
We’ll write parse-http-request as well, but just for fun.
(fn parse-http-response [src read-fn]
(let [status (read-response-status-line src read-fn)
headers (read-headers src read-fn)]
(doto status
(tset :headers headers)
(tset :body src))))
Obviously, this isn’t all we really need, but it’s a good start.
First, let’s parse the status line, i.e. the first line of the response: HTTP/1.1 200 OK
(fn parse-response-status-line [status]
((fn loop [reader fields res]
(case fields
[field & fields]
(let [part (reader)]
(loop reader fields
(case field
:protocol-version
(let [(name major minor) (part:match "([^/]+)/(%d).(%d)")]
(doto res
(tset field {: name :major (tonumber major) :minor (tonumber minor)})))
_ (doto res
(tset field (utils.as-data part))))
))
_
(let [reason (-> "%s/%s.%s +%s +"
(string.format
res.protocol-version.name
res.protocol-version.major
res.protocol-version.minor
res.status)
(status:gsub ""))]
(doto res
(tset :reason-phrase reason)))))
(status:gmatch "([^ ]+)")
[:protocol-version :status]
{}))
(fn read-response-status-line [src read-fn]
(parse-response-status-line (read-fn src :*l)))
You may be wondering, why these functions accept read-fn.
Well, this is something we need to do because we’ll later pass in a socket-channel as a stream.
With these functions, we can read and parse the status line:
(read-response-status-line
(reader.string-reader "HTTP/1.1 200 OK\r\n")
(fn [src pattern] (src:read pattern)))
{:protocol-version {:major 1 :minor 1 :name "HTTP"}
:reason-phrase "OK"
:status 200}
You may also wonder, why the loop doesn’t just continue to read the reason-phrase, instead relying on status:gsub.
Well, reason can include spaces:
>> (read-response-status-line
(reader.string-reader "HTTP/1.1 404 NOT FOUND\r\n")
(fn [src pattern] (src:read pattern)))
{:protocol-version {:major 1 :minor 1 :name "HTTP"}
:reason-phrase "NOT FOUND"
:status 404}
Onto headers:
(fn parse-header [line]
(case (line:match " *([^:]+) *: *(.*)")
(header value) (values header value)))
(fn read-headers [src read-fn ?headers]
(let [headers (or ?headers {})
line (read-fn src :*l)]
(case line
(where (or "\r" ""))
headers
_ (read-headers
src
read-fn
(case (parse-header (or line ""))
(header value)
(doto headers (tset header value)))))))
This one is tail-recursive, accumulating parsed headers in the ?headers table:
>> (read-headers
(reader.string-reader "Content-Type: application/json\r\nConnection: close\r\n\n\r")
(fn [src pattern] (src:read pattern)))
{:Connection "close" :Content-Type "application/json"}
Finally, for the body, we just return the reader as it is, as we’ve already read from it all the necessary data, and the rest are the contents of the request. Together, we get a fully parsed response:
>> (parse-http-response
(reader.string-reader "HTTP/1.1 200 OK\r
Content-Type: application/json\r
Connection: close\r
\r
{\"foo\": \"bar\"}")
(fn [src pattern] (src:read pattern))
)
{:body #<Reader: 0x55a54832b810>
:headers {:Connection "close" :Content-Type "application/json"}
:protocol-version {:major 1 :minor 1 :name "HTTP"}
:reason-phrase "OK"
:status 200}
We can ensure that the parsed response can be encoded back:
>> (let [req (build-http-response 200 "OK" {:connection :close} "foobar")
rdr (reader.string-reader req)
{: body : headers : reason-phrase : status}
(parse-http-response rdr (fn [src pattern] (src:read pattern)) {:as :raw})]
(= req (build-http-response status reason-phrase headers body)))
true
Now, let’s parse the request for completeness:
(fn parse-request-status-line [status]
((fn loop [reader fields res]
(case fields
[field & fields]
(let [part (reader)]
(loop reader fields
(doto res
(tset field (utils.as-data part)))))
_ res))
(status:gmatch "([^ ]+)")
[:method :path :http-version]
{}))
(fn read-request-status-line [src read-fn]
(parse-request-status-line (read-fn src :*l)))
(fn parse-http-request [src read-fn]
(let [status (read-request-status-line src read-fn)
headers (read-headers src read-fn)]
(doto status
(tset :headers headers)
(tset :content (read-fn src :*a)))))
It’s similar, but we need to parse the status line a bit differently. Same as before, let’s ensure that we can do both:
>> (let [req (build-http-request :get "/" {:connection :close} "foobar")
rdr (reader.string-reader req)
{: headers : method : path : content}
(parse-http-request rdr (fn [src pattern] (src:read pattern)) rdr)]
(= req (build-http-request method path headers content)))
true
Now, we can start implementing the client itself!
Async.fnl
For communication, we’ll use already existing TCP channels, that enclose socket objects.
Although, we’ll need a small modification to the async.fnl library, as currently it always reads data in chunks of fixed size.
Here’s the function that returns a socket channel:
(fn set-chunk-size [self pattern-or-size]
(set self.chunk-size pattern-or-size))
(fn socket-channel [client xform err-handler]
(let [recv (chan 1024 xform err-handler)
resp (chan 1024 xform err-handler)
ready (chan)
close (fn [self] (recv:close!) (resp:close!) (set self.closed true))
c (-> {:puts recv.puts
:takes resp.takes
:put! (fn [_ val handler enqueue?]
(recv:put! val handler enqueue?))
:take! (fn [_ handler enqueue?]
(offer! ready :ready)
(resp:take! handler enqueue?))
:close! close
:close close
:chunk-size 1024
:set-chunk-size set-chunk-size}
(setmetatable
{:__index Channel
:__name "SocketChannel"
:__fennelview
#(.. "#<" (: (tostring $) :gsub "table:" "SocketChannel:") ">")}))]
(go-loop [data (<! recv) i 0]
(when (not= nil data)
(case (socket.select nil [client] 0)
(_ [s])
(case (s:send data i)
(nil :timeout j)
(do (<! (timeout 10)) (recur data j))
(nil :closed)
(do (s:close) (close! c))
_ (recur (<! recv) 0))
_ (do (<! (timeout 10))
(recur data i)))))
(go-loop [wait? true]
(when wait?
(<! ready))
(case (client:receive c.chunk-size)
data
(do (>! resp data) (recur true))
(nil :closed "")
(do (client:close) (close! c))
(nil :closed data)
(do (client:close) (>! resp data) (close! c))
(nil :timeout "")
(do (<! (timeout 10)) (recur false))
(nil :timeout data)
(do (>! resp data) (<! (timeout 10)) (recur true))))
c))
It’s a lot of code, but in reality, it isn’t that complicated.
The socket-channel function is responsible for three things:
- Creating a channel, that has separate queues for puts and takes, such that when we put something to a channel, we won’t be able to read it back immediately.
- Creating an asynchronous loop that waits for data from the socket, and puts it into the receive queue
- Creating another asynchronous loop, that waits data on the channel, and puts it into the socket.
These loops account for timeouts, as we’re going to set the socket timeout to 0, to ensure non-blocking behavior.
Both asynchronous loops employ their own timeout handling.
The resulting object is a channel, but with some custom methods to account for its duplex nature.
And, there are also other properties, such as chunk-size and set-chunk-size which we’re going to use with our streams.
Notice, how the second go-loop has a wait? argument.
Because the channel is buffered, it will happily read everything until the buffer is full, which is not ideal, since we may want to change the chunk-size later.
So we need to delay the actual read, which we do in the take! method using offer!.
Finally, the tcp.chan function acts as a constructor for TCP channels:
(fn tcp.chan [{: host : port} xform err-handler]
"Creates a channel that connects to a socket via `host` and `port`.
Optionally accepts a transducer `xform`, and an error handler.
`err-handler` must be a fn of one argument - if an exception occurs
during transformation it will be called with the thrown value as an
argument, and any non-nil return value will be placed in the channel.
The read pattern f a socket can be controlled with the
`set-chunk-size` method."
(assert socket "tcp module requires luasocket")
(let [host (or host :localhost)]
(match-try (socket.connect host port)
client (client:settimeout 0)
_ (socket-channel client xform err-handler)
(catch (nil err) (error err)))))
This is all we need on the async.fnl side of things.
No more patching Luasocket’s HTTP client.
Now we can start implementing our own client.
HTTP client
We already have most of the things we need, but we need to change the parse-http-response a bit.
Remember, our function returns the body as the Reader, but our source is actually a channel.
We can account for that by passing in an appropriate read-fn function but our users shouldn’t deal with that.
Instead, the body should provide proper Reader methods, so let’s create a body-reader:
(fn body-reader [src read-fn]
(var buffer "")
(make-reader
src
{:read-bytes (fn [src pattern]
(let [rdr (reader.string-reader buffer)
buffer-content (rdr:read pattern)]
(case pattern
(where n (= :number (type n)))
(let [len (if buffer-content (length buffer-content) 0)
read-more? (< len n)]
(set buffer (string.sub buffer (+ len 1)))
(if read-more?
(if buffer-content
(.. buffer-content (or (read-fn src (- n len)) ""))
(read-fn src (- n len)))
buffer-content))
(where (or :*l :l))
(let [read-more? (not (buffer:find "\n"))]
(when buffer-content
(set buffer (string.sub buffer (+ (length buffer-content) 2))))
(if read-more?
(if buffer-content
(.. buffer-content (or (read-fn src pattern) ""))
(read-fn src pattern))
buffer-content))
(where (or :*a :a))
(do (set buffer "")
(case (read-fn src pattern)
nil (when buffer-content
buffer-content)
data (.. (or buffer-content "") data)))
_ (error (tostring pattern)))))
:read-line (fn [src]
(let [rdr (reader.string-reader buffer)
buffer-content (rdr:read :*l)
read-more? (not (buffer:find "\n"))]
(when buffer-content
(set buffer (string.sub buffer (+ (length buffer-content) 2))))
(if read-more?
(if buffer-content
(.. buffer-content (or (read-fn src :*l) ""))
(read-fn src :*l))
buffer-content)))
:close (fn [src] (src:close))
:peek (fn [src bytes]
(assert (= :number (type bytes)) "expected number of bytes to peek")
(let [rdr (reader.string-reader buffer)
content (or (rdr:read bytes) "")
len (length content)]
(if (= bytes len)
content
(let [data (read-fn src (- bytes len))]
(set buffer (.. buffer (or data "")))
buffer))))}))
Now, that’s a lot of code again.
The reason for that is the peek method - same as for strings, we need to implement a way to peek at data without modifying the reader, or at least hide the modification from the user.
This is essentially what this function does - it creates a string buffer that we can act upon, accounting for possible data in it before actually reading from the channel itself.
With that function, we can update parse-http-response:
(fn parse-http-response [src read-fn {: as}]
(let [status (read-response-status-line src read-fn)
headers (read-headers src read-fn)
stream (body-reader src read-fn)]
(doto status
(tset :headers headers)
(tset :body
(case as
:raw (stream:read (or headers.Content-Length :*a))
:stream stream
_ (error (string.format "unsupported coersion method '%s'" as)))))))
As you can see, I’ve added another parameter with options.
Right now, there’s only one option as that will choose how to return the body.
Specifying {:as :raw} to this function will return the contents of the reader.
And {:as :stream} will return the stream itself.
Now, we can write the request function:
(local http {})
(fn http.request [method url ?opts]
"Makes a `method` request to the `url`, returns the parsed response,
containing a stream data of the response. The `method` is a string,
describing the HTTP method per the HTTP/1.1 spec. The `opts` is a
table containing the following keys:
- `:async?` - a boolean, whether the request should be asynchronous.
The result is an instance of a `promise-chan`, and the body must
be read inside of a `go` block.
- `:headers` - a table with the HTTP headers for the request
- `:body` - an optional string body.
- `:as` - how to coerce the output.
Several options available for the `as` key:
- `:stream` - the body will be a stream object with a `read` method.
- `:raw` - the body will be a string.
This is the default value for `as`.
When supplying a non-string body, headers should contain a
\"content-length\" key. For a string body, if the \"content-length\"
header is missing it is automatically determined by calling the
`length` function, ohterwise no attempts at detecting content-length
are made."
(let [{: host : port &as parsed} (http-parser.parse-url url)
opts (collect [k v (pairs (or ?opts {}))
:into {:as :raw
:async? false}]
k v)
headers (collect [k v (pairs (or opts.headers {}))
:into {:host
(.. host (if port (.. ":" port) ""))
:content-length
(case opts.body
(where body (= :string (type body)))
(length body))}]
k v)
path (utils.format-path parsed)
req (build-http-request method path headers opts.body)
chan (tcp.chan parsed)]
(if opts.async?
(let [res (promise-chan)]
(go (>! chan req)
(>! res (http-parser.parse-http-response
chan
(make-read-fn <!)
opts)))
res)
(do (>!! chan req)
(http-parser.parse-http-response
chan
(make-read-fn <!!)
opts)))))
We’re still missing some important parts, but we’ll get to those in a bit. This function is long, but it’s not that complicated.
First, we parse the given url into its components, mainly the host and port.
Then, we prepare the default opts table to include as key and async? key if none were passed.
Next, we prepare headers.
The HTTP spec requires the Host header, so we put it in based on the parsed URL.
The path part of the request is then formatted from the rest data from the parsed URL.
Finally, we build-http-request, using the method, path, headers and body if supplied.
And create the tcp.chan using the parsed object that contains host and port needed for the constructor.
Then, if opts.async? is true, we create a promise-chan, which is like Clojure’s promise, which is only computed once, and then returns the same result over and over.
We start the go thread, in which we asynchronously put the request into the socket channel, and then we asynchronously parse the response supplying our read-fn with the <! function to take from the channel asynchronously.
And we return res - the promise channel I mentioned earlier.
If the opts.async? wasn’t true, we do all the same thing, but in a blocking way, using >!! and <!! to work with the channel.
Let’s look at parse-url:
(fn parse-authority [authority]
(let [userinfo (authority:match "([^@]+)@")
port (authority:match ":(%d+)")
host (if userinfo
(authority:match (.. "@([^:]+)" (if port ":" "")))
(authority:match (.. "([^:]+)" (if port ":" ""))))]
{: userinfo : port : host}))
(fn parse-url [url]
(let [scheme (url:match "^([^:]+)://")
{: host : port : userinfo}
(parse-authority
(if scheme
(url:match "//([^/]+)/")
(url:match "^([^/]+)/")))
scheme (or scheme "http")
port (or port (case scheme :https 443 :http 80))
path (url:match "//[^/]+/([^?#]+)")
query (url:match "%?([^#]+)#?")
fragment (url:match "#([^?]+)%??")]
{: scheme : host : port : userinfo : path : query : fragment}))
Nothing special, just some patterns to match components of the URL:
>> (parse-url "https://lua-users.org/?some-query-param=some-value")
{:host "lua-users.org"
:port 443
:query "some-query-param=some-value"
:scheme "https"}
And the format-path function constructs the path back:
(fn format-path [{: path : query : fragment}]
(.. "/" (or path "") (if query (.. "?" query) "") (if fragment (.. "?" fragment) "")))
Here’s what it does on the previously parsed URL:
>> (format-path {:host "lua-users.org"
:port 443
:query "some-query-param=some-value"
:scheme "https"})
"/?some-query-param=some-value"
Finally, the mysterious make-read-fn function creates a function that will work with our socket-channel:
(fn make-read-fn [receive]
"Returns a function that receives data from a socket by a given
`pattern`. The `pattern` can be either `\"*l\"`, `\"*a\"`, or a
number of bytes to read."
(fn [src pattern]
(src:set-chunk-size pattern)
(receive src)))
Since our channel does reading on demand, we can’t pass the chunk-size/pattern in, so we use the set-chunk-size method instead.
Paired with delayed read, I’ve explained earlier, this works really well.
With all of that, we’re ready to do some requests!
>> (http.request :get "http://lua-users.org/" {:headers {:connection "close"} :as :raw})
{:body "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">
<html>
<head>
<title>lua-users.org</title>
<meta http-equiv=\"content-type\" content=\"text/html; charset=ISO-8859-1\">
<LINK TYPE=\"text/css\" REL=\"stylesheet\" HREF=\"/styles/main.css\">
</head>
<body>
<img src=\"/images/lua-users-org.png\" alt=\"lua-users.org logo\">
<br clear=all>
<p>
<b>lua-users.org</b> is an internet site for and by users of the programming language Lua.
</p>
<p>
To find out what Lua is and for documentation, downloads, mailing list info, etc. please visit the <a href=\"http://www.lua.org/\">official Lua web site</a>. The <b>lua-users.org</b> web is not affiliated with Tecgraf.
</p>
<p>
Currently available resources:
</p>
<ul>
<li> <a href=\"/wiki/\">lua-users wiki</a> - collaborative web site
<li> <a href=\"/lists/lua-l/\">lua-l archive</a> - searchable web archive of
the official Lua mailing list
<li> <a href=\"MiniCharter.html\">lua-users.org mini charter</a>
<li> <a href=\"Acknowledgements.html\">acknowledgements</a>
</ul>
</body>
</html>
"
:headers {:Accept-Ranges "bytes"
:Connection "close"
:Content-Length "1055"
:Content-Type "text/html"
:Date "Mon, 15 Jul 2024 12:03:00 GMT"
:ETag "\"61acad05-41f\""
:Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
:Server "nginx/1.20.1"
:Strict-Transport-Security "max-age=0;"}
:protocol-version {:major 1 :minor 1 :name "HTTP"}
:reason-phrase "OK"
:status 200}
Or, we can use :as :stream, and process the stream later:
(local resp (http.request :get "http://lua-users.org/" {:headers {:connection "close"} :as :stream}))
nil
>> resp
{:body #<Reader: 0x55d7090707c0>
:headers {:Accept-Ranges "bytes"
:Connection "close"
:Content-Length "1055"
:Content-Type "text/html"
:Date "Mon, 15 Jul 2024 12:03:47 GMT"
:ETag "\"61acad05-41f\""
:Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
:Server "nginx/1.20.1"
:Strict-Transport-Security "max-age=0;"}
:protocol-version {:major 1 :minor 1 :name "HTTP"}
:reason-phrase "OK"
:status 200}
>> (resp.body:read :*l)
"<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">"
However, writing (http.request :get ...) all the time, is a bit tedious.
Let’s instead define functions for each HTTP method.
And, to do so, we’ll use macros, because I’m too lazy, and I don’t want to write documentation for each function:
(macro define-http-method [method]
"Defines an HTTP method for the given `method`."
`(fn ,(sym (.. :http. (tostring method)))
[url# opts#]
{:fnl/arglist [url opts]
:fnl/docstring ,(.. "Makes a `" (string.upper (tostring method))
"` request to the `url`, returns the parsed response,
containing a stream data of the response. The `method` is a string,
describing the HTTP method per the HTTP/1.1 spec. The `opts` is a
table containing the following keys:
- `:async?` - a boolean, whether the request should be asynchronous.
The result is an instance of a `promise-chan`, and the body must
be read inside of a `go` block.
- `:headers` - a table with the HTTP headers for the request
- `:body` - an optional string body.
- `:as` - how to coerce the output.
Several options available for the `as` key:
- `:stream` - the body will be a stream object with a `read` method.
- `:raw` - the body will be a string.
This is the default value for `as`.
When supplying a non-string body, headers should contain a
\"content-length\" key. For a string body, if the \"content-length\"
header is missing it is automatically determined by calling the
`length` function, ohterwise no attempts at detecting content-length
are made.")}
(http.request ,(tostring method) url# opts#)))
(define-http-method get)
(define-http-method post)
(define-http-method put)
(define-http-method patch)
(define-http-method options)
(define-http-method trace)
(define-http-method head)
(define-http-method delete)
(define-http-method connect)
Now, we can call any HTTP method by just writing (http.method-name ...):
>> (http.get "http://lua-users.org/" {:headers {:connection "close"} :as :stream})
{:body #<Reader: 0x55d7095dbb20>
:headers {:Accept-Ranges "bytes"
:Connection "close"
:Content-Length "1055"
:Content-Type "text/html"
:Date "Mon, 15 Jul 2024 12:08:28 GMT"
:ETag "\"61acad05-41f\""
:Last-Modified "Sun, 05 Dec 2021 12:13:57 GMT"
:Server "nginx/1.20.1"
:Strict-Transport-Security "max-age=0;"}
:protocol-version {:major 1 :minor 1 :name "HTTP"}
:reason-phrase "OK"
:status 200}
And we can even see the documentation for each method in our editor:
Now, that body-reader, with a custom buffer, seemed a bit out of the way, so let’s use it for good.
JSON parsing
Let’s extend our parse-http-response function with another as method:
;; ---- 8< ----
(tset :body
(case as
:raw (stream:read (or parsed-headers.Content-Length :*a))
:json (json.parse stream)
:stream stream
_ (error (string.format "unsupported coersion method '%s'" as))))
;; ---- 8< ----
Now, we can implement a simple JSON parser:
(fn parse [rdr]
"Accepts a reader object `rdr`, that supports `peek`, and `read`
methods. Parses the contents to a Lua table."
((fn loop []
(case (rdr:peek 1)
"{" (parse-obj rdr loop)
"[" (parse-arr rdr loop)
"\"" (parse-string rdr)
(where "t" (= "true" (rdr:peek 4))) (do (rdr:read 4) true)
(where "f" (= "false" (rdr:peek 5))) (do (rdr:read 5) false)
(where "n" (= "null" (rdr:peek 4))) (do (rdr:read 4) nil)
(where c (c:match "[ \t\n]")) (loop (skip-space rdr))
(where n (n:match "[-0-9]")) (parse-num rdr)
nil (error "JSON parse error: end of stream")
_ (error (string.format "JSON parse error: unexpected token ('%s' (code %d))" _ (_:byte)))))))
First, let’s look at parse-string:
(fn parse-string [rdr]
(rdr:read 1)
((fn loop [chars escaped?]
(case (rdr:read 1)
"\\" (loop chars (not escaped?))
"\"" (if escaped?
(loop (.. chars "\"") false)
chars)
_ (loop (.. chars _) false)
))
"" false))
This function handles strings, and escaped patterns withing set strings, like nested strings:
>> (parse (reader.string-reader "\"foo \\\"bar\\\"\""))
"foo \"bar\""
Now, let’s look at the parse-arr
(fn parse-arr [rdr parse]
(rdr:read 1)
((fn loop [arr]
(skip-space rdr)
(case (rdr:peek 1)
"]" (do (rdr:read 1) arr)
_ (let [val (parse)]
(tset arr (+ 1 (length arr)) val)
(skip-space rdr)
(case (rdr:peek 1)
"," (do (rdr:read 1) (loop arr))
"]" (do (rdr:read 1) arr)
_ (error (.. "JSON parse error: expected ',' or ']' after the value: " (json val)))))))
[]))
This function accepts the parse callback to parse nested values, so if we try to parse an array of strings, we’ll call parse on each value, and then expect either a colon or the closing bracket to mark the end of the array.
>> (parse (reader.string-reader "[\"a\", \"b\"]"))
["a" "b"]
The parse-obj works for objects in a similar way:
(fn parse-obj [rdr parse]
(rdr:read 1)
((fn loop [obj]
(skip-space rdr)
(case (rdr:peek 1)
"}" (do (rdr:read 1) obj)
_ (let [key (parse)]
(skip-space rdr)
(case (rdr:peek 1)
":" (let [_ (rdr:read 1)
value (parse)]
(tset obj key value)
(skip-space rdr)
(case (rdr:peek 1)
"," (do (rdr:read 1) (loop obj))
"}" (do (rdr:read 1) obj)
_ (error (.. "JSON parse error: expected ',' or '}' after the value: " (json value)))))
_ (error (.. "JSON parse error: expected colon after the key: " (json key)))))))
{}))
Except, it waits for the : after each key, and then the value:
>> (parse (reader.string-reader "{\"arr\": [\"a\", \"b\"]}"))
{:arr ["a" "b"]}
Finally, there’s the parse-num and code to parse Booleans, and null:
(fn parse-num [rdr]
((fn loop [numbers]
(case (rdr:peek 1)
(where n (n:match "[-0-9.eE+]"))
(do (rdr:read 1) (loop (.. numbers n)))
_ (tonumber numbers)))
(rdr:read 1)))
With that, we can call http.get passing it {:as :json} as opts:
>> (http.get "http://httpbin.org/get" {:as :json})
{:body {:args {}
:headers {:Host "httpbin.org"
:X-Amzn-Trace-Id "Root=1-6695183d-27c59aff535ae9f11df8c98d"}
:origin "185.169.167.87"
:url "http://httpbin.org/get"}
:headers {:Access-Control-Allow-Credentials "true"
:Access-Control-Allow-Origin "*"
:Connection "keep-alive"
:Content-Length "199"
:Content-Type "application/json"
:Date "Mon, 15 Jul 2024 12:38:21 GMT"
:Server "gunicorn/19.9.0"}
:protocol-version {:major 1 :minor 1 :name "HTTP"}
:reason-phrase "OK"
:status 200}
And our body was successfully parsed.
Asynchronous HTTP calls
Now, this client is asynchronous, meaning we can download lots of resources simultaneously. I’m not saying in parallel, because there are no threads in Lua, but still, we can process several requests at once using our asynchronous library.
Let’s define a simple time macro first:
(macro time [expr]
`(let [clock# socket.gettime
start# (clock#)
res# ,expr
end# (clock#)]
(print (.. "Elapsed " (* (- end# start#) 1000) " ms"))
res#))
With it, we can measure the execution time of a body. Let’s make three asynchronous requests, that upon completion will count the lengths of the body for each, and return their sum:
>> (time
(let [headers {:connection "close"}
a (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
b (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
c (http.get "http://httpbin.org/get" {:headers headers :as :raw :async? true})
lengths (chan 3)]
(go (let [resp (<! a)]
(>! lengths (length resp.body))))
(go (let [resp (<! b)]
(>! lengths (length resp.body))))
(go (let [resp (<! c)]
(>! lengths (length resp.body))))
(+ (<!! lengths) (<!! lengths) (<!! lengths))))
Elapsed 530.24077415466 ms
597
In comparison, three synchronous requests, take almost twice as much time:
>> (time
(let [headers {:connection "close"}
a (http.get "http://httpbin.org/get" {:headers headers :as :raw})
b (http.get "http://httpbin.org/get" {:headers headers :as :raw})
c (http.get "http://httpbin.org/get" {:headers headers :as :raw})]
(+ (length a.body) (length b.body) (length c.body))))
Elapsed 1047.5831031799 ms
597
For the reference, here’s Luasocket’s HTTP client:
>> (local lhttp (require :socket.http))
nil
>> (time
(let [a (lhttp.request "http://httpbin.org/get")
b (lhttp.request "http://httpbin.org/get")
c (lhttp.request "http://httpbin.org/get")]
(+ (length a) (length b) (length c))))
Elapsed 1919.6438789368 ms
717
(Note, the resulting length is different, because Luasocket specifies its user-agent, and httpbin includes it in the answer.)
Further development
Now, of course, that’s not all that needs to be implemented. In its current state, this client doesn’t support responses that do not specify the Content-Length or produce a response body in chunked encoding. HTTPS is out of the question for now too. Most of the code just takes a “happy path” without much thought given to errors or more complicated scenarios. But it’s fine, I can always make it better down the line.
But all in all, it was a fun experiment, and I’m feeling better, as I finally have some use for async.fnl.
You can find the source code for this project on my GitLab profile.
And by the way, even though it is written in Fennel and uses some Fennel libraries, like async.fnl, this library works with ordinary Lua:
Lua 5.4.4 Copyright (C) 1994-2022 Lua.org, PUC-Rio
> http = require "http"
> http.get("http://httpbin.org/get", {headers = {connection = "close"}}).body
{
"args": {},
"headers": {
"Host": "httpbin.org",
"X-Amzn-Trace-Id": "Root=1-6695424f-01e0510f2ab49b9f46bd51dc"
},
"origin": "185.165.163.78",
"url": "http://httpbin.org/get"
}
> http.get("http://httpbin.org/get", {headers = {connection = "close"}, as = "stream"}).body
Reader: 0x55a6c0eff710
So if you like what you see, but you don’t use Fennel, you can still try out this library. It just needs to be compiled first.
Anyway, that’s all from me, so… thanks for reading!