Subprojects Behind OpenResty
API7.ai
September 12, 2022
How to extract Lua code from nginx.conf and keep it readable and maintainable? The solution is quite simple. Let's see how to implement this by OpenResty.
First, create a directory called lua. Then, we will put all .lua files in it.
$ mkdir lua
$ cat lua/hello.lua
ngx.say("hello, world")
Second, use content_by_lua_block to replace content_by_lua_file in the nginx.conf file.
pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
server {
listen 8080;
location / {
content_by_lua_file lua/hello.lua;
}
}
}
Third, restart OpenResty's services, and it's done!
$ sudo kill -HUP `cat logs/nginx.pid`
Using content_by_lua_file, we could update the Lua file directly instead of updating nginx.conf. But there have some questions:
- We use a relative path in the
content_by_lua_file lua/hello.luasection. How does OpenResty find the actual Lua file? - After modifying Lua codes, we need to restart OpenResty to make it work, does there any way to debug efficiently?
- How to add the directory which contains Lua files to OpenResty's lookup path?
I encourage you to think about these questions, which can all be answered in the official documentation. That's why I always stress the importance of documentation.
For the first question, if the relative path is given, then when OpenResty starts, it will prefix the -p PATH in the command line parameters of OpenResty startup and concatenate the relative path into an absolute path. In this way, OpenResty can find the Lua file smoothly.
The second problem is that Lua code is loaded on the first request and cached by default. So every time you change the Lua source file, you have to reload OpenResty to make it work. You can avoid reloading by turning off lua_code_cache in nginx.conf. However, it is essential to note that this method can only be used temporarily for development and debugging. So if you are deploying on production, you must enable the cache. Otherwise, it will have a significant performance impact.
For the last question, OpenResty provides a lua_package_path directive to set the lookup path for Lua modules. For example, we can set lua_package_path to $prefix/lua/? .lua;;:
$prefixis the-p PATHin the startup parameter./lua/?.luaindicates all files in the Lua directory with the.luasuffix.- The last two semicolons represent the built-in code search path.
Directory structure after installation
After understanding the first hello world program, let's go to the bottom and see what the directory structure of OpenResty looks like after it is installed and what files are contained in it.
We first pass the -V option to see where OpenResty is. For the following result, I omitted many module compilation parameters, which we will add later:
$ openresty -V
nginx version: openresty/1.13.6.2
built by clang 10.0.0 (clang-1000.10.44.4)
built with OpenSSL 1.1.0h 27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/Cellar/openresty/1.13.6.2/nginx ...
I use brew to install OpenResty on my Mac. The path is /usr/local/Cellar/openresty/1.13.6.2/nginx, which may differ from your environment. This path contains the bin, luajit, lualib, nginx, pod and other directories. It is essential to understand the meaning of these folders so that we can learn OpenResty better. So let's take a look at them one by one.
First is the important bin directory.
$ ll /usr/local/Cellar/openresty/1.13.6.2/bin
total 320
-r-xr-xr-x 1 ming admin 19K 3 27 12:54 md2pod.pl
-r-xr-xr-x 1 ming admin 15K 3 27 12:54 nginx-xml2pod
lrwxr-xr-x 1 ming admin 19B 3 27 12:54 openresty -> ../nginx/sbin/nginx
-r-xr-xr-x 1 ming admin 62K 3 27 12:54 opm
-r-xr-xr-x 1 ming admin 29K 3 27 12:54 resty
-r-xr-xr-x 1 ming admin 15K 3 27 12:54 restydoc
-r-xr-xr-x 1 ming admin 8.3K 3 27 12:54 restydoc-index
This contains both the OpenResty CLI resty we mentioned in the previous section and the core executable, openresty, which is actually a soft link to nginx. As for the other tools in the directory, there is no doubt that, like resty, they are all Perl scripts.
Among them, opm is a package management tool that allows us to manage all kinds of third-party packages, which will be covered in a section later; and restydoc, an old friend from the first section, is a documentation viewer provided by OpenResty, which allows us to view OpenResty and NGINX documentation.
$ restydoc -s ngx.say
$ restydoc -s proxy_pass
The two examples above query the OpenResty API and NGINX commands, respectively. restydoc is a tool that is very helpful for server-side engineers to focus on development.
After browsing the bin directory, let's move on to the pod directory.
First, let's emphasize that pod here has nothing to do with the concept of a pod in Kubernetes. Instead, the pod is a markup language used in Perl to write documentation for Perl modules. And this directory contains documentation for OpenResty, NGINX, lua-resty-*, and LuaJIT, which are all tied together with the restydoc mentioned earlier.
Next are the familiar NGINX and luajit directories. These two are easy to understand. They mainly store NGINX and LuaJIT executable files and dependencies and are the cornerstones of OpenResty. Many people say OpenResty is based on Lua, but that's inaccurate. As we can see above, OpenResty is actually based on LuaJIT.
In fact, in the early stage, OpenResty came with both Lua and LuaJIT, and we could decide whether to use Lua or LuaJIT by compiling options. Still, Lua is being deprecated, and only the higher-performance LuaJIT is supported.
Finally, let's look at the lualib directory. It contains the Lua libraries used in OpenResty, mainly divided into two directories: ngx and resty.
- The
ngxdirectory holds the Lua codes from the official lua-resty-core project, which is based on the FFI re-implementation of the OpenResty API. I will explain in a particular chapter why we need to re-implement it. - The resty directory contains Lua code from various
lua-resty-*projects, which we will touch on next.
Following the convention of this course, at this point I will give the source of the source of these directories. This is also one of the joys of open source projects. If you like to break the casserole and ask to the end, you will always find more exciting things.
Here is the OpenResty packaging script for CentOS, which contains all the directories mentioned above.
%files
%defattr(-,root,root,-)
/etc/init.d/%{name}
/usr/bin/%{name}
%{orprefix}/bin/openresty
%{orprefix}/site/lualib/
%{orprefix}/luajit/*
%{orprefix}/lualib/*
%{orprefix}/nginx/html/*
%{orprefix}/nginx/logs/
%{orprefix}/nginx/sbin/*
%{orprefix}/nginx/tapset/*
%config(noreplace) %{orprefix}/nginx/conf/*
%{orprefix}/COPYRIGHT
OpenResty project overview
When it comes to OpenResty, we will think of lua-nginx-module. Yes, this NGINX C module is indeed the core of OpenResty, but it is not equivalent to OpenResty. Many engineers call OpenResty as ngx + lua, which is also used in books shared and published by many technical conferences. This is not rigorous and is not advocated by the OpenResty community.
Let me talk about why and what other related projects besides lua-nginx-module are in OpenResty.
Open the project homepage of OpenResty on GitHub, and you can see that OpenResty contains 68 public projects, which are roughly divided into the following seven categories. Let me briefly introduce them separately so you can have a preliminary impression and learn it quickly.
NGINX C Modules
OpenResty's project naming is standardized, and those named *-nginx-module is the NGINX C modules.
There are more than 20 C modules in OpenResty, and we can find them by using openresty -V, which we used at the beginning of this section.
$ openresty -V
nginx version: openresty/1.13.6.2
built by clang 10.0.0 (clang-1000.10.44.4)
built with OpenSSL 1.1.0h 27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/Cellar/openresty/1.13.6.2/nginx --with-cc-opt='-O2 -I/usr/local/include -I/usr/local/opt/pcre/include -I/usr/local/opt/openresty-openssl/include' --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.13 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.5 --with-ld-opt='-Wl,-rpath,/usr/local/Cellar/openresty/1.13.6.2/luajit/lib -L/usr/local/lib -L/usr/local/opt/pcre/lib -L/usr/local/opt/openresty-openssl/lib' --pid-path=/usr/local/var/run/openresty.pid --lock-path=/usr/local/var/run/openresty.lock --conf-path=/usr/local/etc/openresty/nginx.conf --http-log-path=/usr/local/var/log/nginx/access.log --error-log-path=/usr/local/var/log/nginx/error.log --with-pcre-jit --with-ipv6 --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-dtrace-probes --with-stream --with-stream_ssl_module --with-http_ssl_module
Here --add-module= is followed by OpenResty's C module. The core ones are lua-nginx-module and stream-lua-nginx-module, the former to handle Layer 7 traffic and the latter to drive Layer 4 traffic.
Some of these C modules require special attention and are not recommended, although they are compiled into OpenResty by default. For example, redis2-nginx-module, redis-nginx-module, and memc-nginx-module are used to interoperate with Redis and Memcached. These C libraries were recommended by OpenResty in the early stage, but have been replaced by lua-resty-redis and lua-resty-memcached after the cosocket feature was added, and are maintained inactive.
OpenResty will not develop more NGINX C libraries later but will focus on the cosocket-based Lua library, which is the future.
lua-resty-* libraries
The official OpenResty repository contains 18 lua-resty-* libraries, including Redis, MySQL, Memcached, WebSocket, DNS, traffic control, string processing, in-process caching, and other standard libraries. In addition to the official ones that come with them, there are more third-party libraries. They are essential, and we will devote more time to these libraries in the next section.
Self-maintained LuaJIT branch
OpenResty maintains its LuaJIT branch in addition to its own OpenSSL patch. In 2015, LuaJIT's author Mike Pall announced his retirement to find a new LuaJIT maintainer, but Mike did not find a suitable maintainer. He is now mainly doing bugfix maintenance, and the development of new features has been suspended, so OpenResty maintains its LuaJIT branch.
Compared to Lua, LuaJIT adds a lot of essential and unique functions, but not many engineers know about them, so they are semi-hidden skills that I will introduce later.
Testing Framework
OpenResty's testing framework is test-nginx, also developed in Perl, and as you can see from the name, it is specifically designed to test NGINX-related projects. All of OpenResty's official test cases for C modules and the lua-resty library are driven by test-nginx. This is a much more powerful and independent system, unlike the common assertion-based frameworks.
Some OpenResty contributors haven't figured out this testing framework either and sometimes submit PRs that contain complex C and Lua code but are still often intimidated by the idea of writing corresponding test cases. So if you've looked at some of the test cases in the /t directory of the OpenResty project and are still confused, don't doubt yourself yet. Most people are the same.
In addition to test-nginx, the mockeagain project simulates a slow network, allowing programs to read and write one byte at a time. This is a handy tool for web servers.
Debug toolchain
The OpenResty project has spent a lot of effort on how to debug code scientifically and dynamically.
The two OpenResty projects, openresty-systemtap-toolkit, and stapxx, are based on systemtap, a dynamic debugging and tracing tool. The most significant advantage of using a systemtap is that it enables in vivo analysis while being entirely non-intrusive for the target application.
For example, a systemtap is like going to the hospital and getting a CT scan, painless and non-perceptive. Even better, the systemtap can generate visual flame graphs for performance analysis, which I will describe later, so here's a flame graph to give you a sense of what's going on.
Package
The packaging scripts of OpenResty in different distribution operating systems (such as CentOS, Ubuntu, macOS, etc.) are hand-written for more satisfactory controllability. When we introduced the directory structure after installation, we already covered these packaging-related projects: openresty-packaging and home-brew. If you are interested in this, you can learn it yourself, and I will not repeat it here.
Engineering Tools
In addition to these larger projects, OpenResty has several engineering tools that are mostly hidden.
For example, openresty-devel-utils is the toolset for developing OpenResty and NGINX. Of course, they are also developed in Perl, and most of the tools are undocumented. But for OpenResty developers, these tools are handy. I'll start by picking a few and briefly introducing them.
- lj-releng is a simple and effective LuaJIT code inspection tool, similar to
luacheck, which can find potential problems with global variables. - reindex, which means rebuilding the index, is a tool for formatting
test-nginxtest cases, rearranging test case numbers, and removing extra whitespace.reindexis one of the tools that OpenResty developers use every day. - opsboy is used to do automated deployments. it is used to deploy and drive the regression tests that OpenResty does on AWS EC2 clusters before each release. For detailed information, you can refer to the official documentation.
opsboyis a DSL implemented in Perl. The OpenResty authors like to create different DSLs to solve problems.
Summary
Today, we mainly learned the directory structure of OpenResty after installation and some sub-projects behind it. After learning today's content, I hope you can learn more about OpenResty's projects. OpenResty has gone far beyond the NGINX load balancing and reverse proxy scope and has realized its own ecology. Next time we will talk about this in detail.