How to Manage Third-Party Packages of OpenResty Through LuaRocks and OPM?

API7.ai

September 17, 2022

OpenResty (NGINX + Lua)

In the previous post, we got a general overview of some official OpenResty projects. However, if we use OpenResty for the production environment, these libraries that come with the OpenResty package are insufficient. For example, there is no lua-resty library to send HTTP requests, and there is no way to interact with Kafka.

So what should we do?

Because OpenResty is not a fork of NGINX, nor is it a repackaging of NGINX with some standard libraries, but instead just using NGINX as the underlying network library.

When using NGINX, you're not thinking about how to initiate custom HTTP requests and how to interact with Kafka. And in the OpenResty world, thanks to cosocket, developers can quickly write lua-resty-http and lua-resty-kafka to handle these needs, just like you can with development languages like Python, PHP, etc.

Another piece of advice for you: you should not use any Lua world libraries to solve the above problems but instead use cosocket's lua-resty-* library. Libraries from the Lua world are likely to introduce blocking, taking an otherwise high-performance service and dropping it by several orders of magnitude straight away. This is a common mistake for OpenResty beginners and is not easily noticeable.

So how do we find these non-blocking lua-resty-* libraries? Next, let me introduce you to the following ways.

OPM

OPM (OpenResty Package Manager) is the package manager that comes with OpenResty and can be used after you have installed OpenResty. We can try to find the library that sends http requests by using the opm search http command.

The query may be slow the first time, usually taking a few seconds. opm.openresty.org will do a query from the PostgreSQL database and cache the results for a while.

openresty/lua-resty-upload                            Streaming reader and parser for HTTP file uploading based on ngx_lua cosocket

When you see this result, you might wonder: what does this lua-resty-upload package have to do with sending http?

It turns out that when OPM searched, it searched for both the package name and the package profile using the latter keyword. This is why the search above lasts a few seconds. It does a full-text search of the string inside PostgreSQL.

But, anyway, this return is not friendly. Let's modify the keyword and search again for it.

$ opm search lua-resty-http

ledgetech/lua-resty-http                          Lua HTTP client cosocket driver for OpenResty/ngx_lua
pintsized/lua-resty-http                          Lua HTTP client cosocket driver for OpenResty/ngx_lua
agentzh/lua-resty-http                            Lua HTTP client cosocket driver for OpenResty/ngx_lua

In the OpenResty world, if you implement a package using cosocket, you must use the lua-resty prefix, which is an unwritten rule.

Looking back at the search results, the OPM uses the contributor's GitHub repository address as the package name, i.e. GitHub ID / repo name. It returns three lua-resty-http third-party libraries. Which one should we choose?

Before choosing the package, let's look at its star count and last update time: only a dozen stars; the latest update was in 2016. This is an abandoned pit. Looking deeper, pintsized/lua-resty-http and ledgetech/lua-resty-http point to the same repository. So it doesn't matter which one you choose, it's the same.

Also, OPM's website is relatively simple and does not provide the number of package downloads or the dependencies for this package. It would help if you spent more time figuring out which lua-resty libraries are the right ones to use when that should be the maintainer's job.

LuaRocks

LuaRocks is another package manager for the OpenResty world, born before OPM. Unlike OPM, which contains only OpenResty-related packages, LuaRocks also has libraries from the Lua world. For example, LuaSQL-MySQL in LuaRocks is a package that connects to MySQL in the Lua world and cannot be used in OpenResty.

Still using the HTTP library as an example, let's try to use LuaRocks to find.

$ luarocks search http

As you can see, it also returns a bunch of packages.

We might as well change the keyword again.

$ luarocks search lua-resty-http

This time only one package was returned. We can go to the LuaRocks website and look at this package's details. Here is a screenshot of the website page.

LuaRocks Package Details

The screenshot above contains the author, License, GitHub URL, number of downloads, feature summary, version history, dependencies, etc. Unlike OPM, LuaRocks does not use GitHub user information directly but requires developers to register with LuaRocks separately.

The open-source API gateway project, Apache APISIX, uses LuaRocks for package management. Let's briefly examine how APISIX's package management configuration is written.

The latest version of APISIX is 2.15, and you can find the apisix-2.15.0-0.rockspec file under the Apache APISIX project.

package = "apisix"
version = "2.15.0-0"
supported_platforms = {"linux", "macosx"}

source = {
    url = "git://github.com/apache/apisix",
    branch = "2.15.0",
}

description = {
    summary = "Apache APISIX is a cloud-native microservices API gateway, delivering the ultimate performance, security, open source and scalable platform for all your APIs and microservices.",
    homepage = "https://github.com/apache/apisix",
    license = "Apache License 2.0",
}

dependencies = {
    "lua-resty-ctxdump = 0.1-0",
    "lua-resty-dns-client = 6.0.2",
    "lua-resty-template = 2.0",
    "lua-resty-etcd = 1.8.2",
    "api7-lua-resty-http = 0.2.0",
    "lua-resty-balancer = 0.04",
    "lua-resty-ngxvar = 0.5.2",
    "lua-resty-jit-uuid = 0.0.7",
    "lua-resty-healthcheck-api7 = 2.2.0",
    "api7-lua-resty-jwt = 0.2.4",
    "lua-resty-hmac-ffi = 0.05",
    "lua-resty-cookie = 0.1.0",
    "lua-resty-session = 3.10",
    "opentracing-openresty = 0.1",
    "lua-resty-radixtree = 2.8.2",
    "lua-protobuf = 0.3.4",
    "lua-resty-openidc = 1.7.5",
    "luafilesystem = 1.7.0-2",
    "api7-lua-tinyyaml = 0.4.2",
    "nginx-lua-prometheus = 0.20220527",
    "jsonschema = 0.9.8",
    "lua-resty-ipmatcher = 0.6.1",
    "lua-resty-kafka = 0.20-0",
    "lua-resty-logger-socket = 2.0.1-0",
    "skywalking-nginx-lua = 0.6.0",
    "base64 = 1.5-2",
    "binaryheap = 0.4",
    "api7-dkjson = 0.1.1",
    "resty-redis-cluster = 1.02-4",
    "lua-resty-expr = 1.3.1",
    "graphql = 0.0.2",
    "argparse = 0.7.1-1",
    "luasocket = 3.0rc1-2",
    "luasec = 0.9-1",
    "lua-resty-consul = 0.3-2",
    "penlight = 1.9.2-1",
    "ext-plugin-proto = 0.5.0",
    "casbin = 1.41.1",
    "api7-snowflake = 2.0-1",
    "inspect == 3.1.1",
    "lualdap = 1.2.6-1",
    "lua-resty-rocketmq = 0.3.0-0",
    "opentelemetry-lua = 0.1-3",
    "net-url = 0.9-1",
    "xml2lua = 1.5-2",
    "nanoid = 0.1-1",
    "lua-resty-mediador = 0.1.2-1"
}

build = {
    type = "make",
    build_variables = {
        CFLAGS="$(CFLAGS)",
        LIBFLAG="$(LIBFLAG)",
        LUA_LIBDIR="$(LUA_LIBDIR)",
        LUA_BINDIR="$(LUA_BINDIR)",
        LUA_INCDIR="$(LUA_INCDIR)",
        LUA="$(LUA)",
        OPENSSL_INCDIR="$(OPENSSL_INCDIR)",
        OPENSSL_LIBDIR="$(OPENSSL_LIBDIR)",
    },
    install_variables = {
        ENV_INST_PREFIX="$(PREFIX)",
        ENV_INST_BINDIR="$(BINDIR)",
        ENV_INST_LIBDIR="$(LIBDIR)",
        ENV_INST_LUADIR="$(LUADIR)",
        ENV_INST_CONFDIR="$(CONFDIR)",
    },
}

As you can see from the file, the dependencies are mixed with lua-resty-* libraries and libraries from the pure Lua world, which can only be partially installed using OPM. After writing the configuration, use the upload command of LuaRocks to upload this configuration, and the user can use LuaRocks to download and install APISIX.

LuaRocks supports this. You can specify the path and name of the C source code in the .rockspec file, and LuaRocks will compile it locally for you. OPM does not support this feature yet.

awesome-resty

The awesome-resty project maintains almost all of the packages available for OpenResty, organized into categories. This is the best place to go when you are unsure if there is a suitable third-party package.

Let's take the HTTP library as an example. In awesome-resty it naturally belongs to the networking category.

lua-resty-http by @pintsized — Lua HTTP client cosocket driver for OpenResty / ngx_lua
lua-resty-http by @liseen — Lua http client driver for the ngx_lua based on the cosocket API
lua-resty-http by @DorianGray — Lua HTTP client driver for ngx_lua based on the cosocket API
lua-resty-http-simple — Simple Lua HTTP client driver for ngx_lua
lua-resty-httpipe — Lua HTTP client cosocket driver for OpenResty / ngx_lua
lua-resty-httpclient — Nonblocking Lua HTTP Client library for aLiLua & ngx_lua
lua-httpcli-resty — Lua HTTP client module for OpenResty
lua-resty-requests — Yet Another HTTP Library for OpenResty

We see that there are eight third-party libraries for lua-resty-http. Comparing the previous results, we only found 2 using OPM and only 1 in LuaRocks. However, if it is hard to choose, use the first one, which is the same as the one in LuaRocks.

For engineers willing to experiment, I recommend the last library: lua-resty-requests, a more human-friendly HTTP access library with the same interface style as Python's famous Requests. If you're a Python lover like me, you'll love lua-resty-requests, which are authored by active tokers in the OpenResty community so that you can use them confidently.

Summary

For an open source project to thrive, it needs not only hardcore technology, good documentation, and complete testing but also an ecosystem that brings more developers and companies together. As the Apache Way said: community over code.

There is no systematic learning material for OpenResty, no official code guide, and many optimization points have been written in the open-source project. Still, most developers know what they are talking about but don't know why. That's the purpose of this Ebook. After learning it, I hope you can write a more efficient OpenResty code and participate more efficiently in OpenResty-related open source projects, for example, Apache APISIX.

What do you think about the OpenResty ecosystem? Feel free to leave a comment, and let's talk about it. Also, welcome to share this article.