Roadblock in Contributing Code: `test::nginx`

Testing is an essential part of software development. The concept of Test Driven Development (TDD) has become so popular that almost every software company has a QA (Quality assurance) team to take care of the testing work.

Testing is the cornerstone of OpenResty's quality and great reputation, but it is also the most neglected part of OpenResty's open-source projects. Many developers use the lua-nginx-module every day and occasionally run a flame graph, but how many people will run the test cases? Even many OpenResty-based open-source projects are without test cases. But an open-source project without test cases and continuous integration is not trustworthy.

However, unlike commercial companies, there are no dedicated software testing engineers in most open-source projects, so how do they ensure the quality of their code? The answer is simple: "test automation" and "continuous integration", with the key points being automative and continuous, both of which OpenResty has achieved to the greatest extent.

OpenResty has 70 open source projects, and their unit testing, integration testing, performance testing, mock testing, fuzz testing, and other workloads are challenging to solve purely manually by the community contributors. Therefore, OpenResty invested more in automation testing at the beginning. This may seem to slow down the project in the short term, but it can be said that the investment in this area is very cost-effective in the long run. So when I talk to other engineers about OpenResty's logic and toolset for testing, they are amazed.

Let's talk about OpenResty's testing philosophy.

Concept

test::nginx is the core of the OpenResty testing architecture, which is used by OpenResty itself and the surrounding lua-resty libraries to organize and write test sets. It is a testing framework with a very high threshold. The reason is that, unlike common testing frameworks, test::nginx is not based on assertions and does not use the Lua language, which requires developers to learn and use test::nginx from scratch and reverse their inherent knowledge of testing frameworks.

I know several OpenResty contributors who can submit C and Lua code to OpenResty but feel it is difficult to write test cases using test::nginx. They either didn't know how to write them or how to fix them when encountering test failures. Therefore, I call test::nginx a roadblock in contributing code.

test::nginx combines Perl, data-driven, and DSL (Domain-specific language). For the same test case set, by controlling the parameters and environment variables, you can achieve different effects such as random execution, multiple repetitions, memory leak detection, stress testing, etc.

Installation and examples

Before we use test::nginx, let's learn how to install it.

As for software installation in the OpenResty system, only the official CI installation method is the most timely and effective; other ways of installation always encounter various problems. That's why I recommend you take official methods as a reference, where you can find the installation and use of test::nginx as well. There are four steps.

First, install Perl's package manager cpanminus.
Then, install test::nginx via cpanm.

sudo cpanm --notest Test::Nginx IPC::Run > build.log 2>&1 || (cat build.log && exit 1)

Next, clone the latest source code.

git clone https://github.com/openresty/test-nginx.git

Finally, load the test-nginx library via Perl's prove command and run the set of test cases in the /t directory.

prove -Itest-nginx/lib -r t

After the installation, let's look at the simplest test case in test::nginx. The following code is adapted from the official documentation, and I have removed all the customized control parameters.

use Test::Nginx::Socket 'no_plan';


run_tests();

__DATA__

=== TEST 1: set Server
--- config
    location /foo {
        echo hi;
        more_set_headers 'Server: Foo';
    }
--- request
    GET /foo
--- response_headers
Server: Foo
--- response_body
hi

Although test::nginx is written in Perl and works as one of the modules, can you see anything in Perl or any other language from the above test? That's right. It's because `test::nginx is the author's own Perl implementation of DSL, abstracted specifically for testing NGINX and OpenResty.

So, when we first see this kind of test, we most likely don't understand. But don't worry; let's analyze the above test case.

First of all, use Test::Nginx::Socket;, which is the way of Perl references libraries, just like the require in Lua. This also reminds us that test::nginx is a Perl program.

The second line, run_tests(); is a Perl function in test::nginx, the entry function for the testing framework. If you want to call any other Perl functions in test::nginx, they must be placed before run_tests to be valid.

The __DATA__ in the third line is a flag indicating that everything below it is test data, and Perl functions should be completed before this flag.

The next === TEST 1: set Server, the title of the test case, indicates the purpose of this test, and it has a tool that automatically arranges the numbering inside.

--- config is the NGINX configuration field. In the above case, we used NGINX commands, not Lua, and if you want to add Lua code, you will do so here with a directive like content_by_lua.

--- request is used to simulate a terminal to send a request, followed by GET /foo, which specifies the method and URI of the request.

--- response_headers, which is used to detect response headers. The following Server: Foo indicates the header and value that must appear in the response headers. If not, the test will fail.

The last one --- response_body, is used to detect the corresponding body. The following hi is the string that must appear in the response body; if it does not, the test will fail.

Well, here, the simplest test case is finished analyzing. So, understanding the test case is a prerequisite for completing OpenResty-related development work.

Write your test cases

Next, it's time to get into the hands-on testing. Remember how we tested Memcached server in the last article? That's right; we used resty to send the request manually, which is represented by the following code.

resty -e 'local memcached = require "resty.memcached"
    local memc, err = memcached:new()

    memc:set_timeout(1000) -- 1 sec
    local ok, err = memc:connect("127.0.0.1", 11212)
    local ok, err = memc:set("dog", 32)
    if not ok then
        ngx.say("failed to set dog: ", err)
        return
    end

    local res, flags, err = memc:get("dog")
    ngx.say("dog: ", res)'

But isn't sending it manually not smart enough? No worries. We can try to turn manual tests into automated ones after learning test::nginx. For example:

use Test::Nginx::Socket::Lua::Stream;

run_tests();

__DATA__
  
=== TEST 1: basic get and set
--- config
        location /test {
            content_by_lua_block {
                local memcached = require "resty.memcached"
                local memc, err = memcached:new()
                if not memc then
                    ngx.say("failed to instantiate memc: ", err)
                    return
                end

                memc:set_timeout(1000) -- 1 sec
                local ok, err = memc:connect("127.0.0.1", 11212)

                local ok, err = memc:set("dog", 32)
                if not ok then
                    ngx.say("failed to set dog: ", err)
                    return
                end

                local res, flags, err = memc:get("dog")
                ngx.say("dog: ", res)
            }
        }

--- stream_config
    lua_shared_dict memcached 100m;

--- stream_server_config
    listen 11212;
    content_by_lua_block {
        local m = require("memcached-server")
        m.go()
    }

--- request
GET /test
--- response_body
dog: 32
--- no_error_log
[error]

In this test case, I have added --- stream_config, --- stream_server_config, --- no_error_log as configuration items, but they are essentially the same, i.e.

The data and testing of the tests are stripped down to make readability and extensibility better by abstracting the configuration.

This is where test::nginx is fundamentally different from other testing frameworks. This DSL is a double-edged sword in that it makes the test logic clear and easily extensible. However, it raises the learning cost, requiring you to relearn new syntax and configuration before you can start writing test cases.

Summary

The test::nginx is powerful, but many times it may not always be suitable for your scenario. Why break a butterfly on the wheel? In OpenResty, you also have the option to use the assertion-style testing framework busted. The busted combined with resty becomes a command line tool, and can also meet many testing needs.

Finally, I'll leave you with a question. Can you run this test for Memcached locally? If you can add a new test case, that would be great.