Little-Known Usage of `test::nginx`

In the previous two articles, you have mastered most of the usage of test::nginx, and I believe you can understand most of the test case sets in the OpenResty project. This is more than enough for learning OpenResty and its surrounding libraries.

But if you are interested in becoming an OpenResty code contributor, or if you are using test::nginx to write test cases in your projects, then you need to learn some more advanced and complex usage.

Today's article will probably be the most "unpopular" part of the series because it's something no one has ever shared before. Take lua-nginx-module, the core module in OpenResty, as an example, which has over 70 contributors worldwide, but not every contributor has written a test case. So if you read today's article, your understanding of test::nginx will enter the Top 100 worldwide.

Debugging in test

First, let's look at some of the simplest and most commonly used sections that developers use in normal debugging. Here, we will introduce the usage scenarios of these debugging-related sections one by one.

ONLY

Very often, we add a new test case to the original set of test cases. If the test file contains a lot of test cases, it is time-consuming to run through it, especially when you need to modify the test cases repeatedly.

So, is there any way to run only one of the test cases you specify? This can easily be done with the ONLY section.

=== TEST 1: sanity
=== TEST 2: get
--- ONLY

The above pseudocode shows how to use this section. By putting --- ONLY in the last line of the test case that needs to be run alone, then when you use prove to run the test case file, all other test cases will be ignored, and only this one test will be run.

However, this is only appropriate when you are doing debugging. So, when the prove command finds the ONLY section, it will also prompt you not to forget to remove it when you commit your code.

SKIP

The requirement corresponding to executing only one test case is to ignore a particular test case. The SKIP section, which is typically used to test functionality that has not yet been implemented:

=== TEST 1: sanity
=== TEST 2: get
--- SKIP

As you can see from this pseudocode, its usage is similar to ONLY. Because we are test-driven development, we need to write test cases first; and when we are coding the implementation collectively, we may need to delay the implementation of a feature due to the difficulty or priority of implementation. Then, you can skip the corresponding test case set first, and then remove the SKIP section when the implementation is complete.

LAST

Another common section is LAST, which is also simple to use, as the test cases before it will be executed and the ones after it will be ignored.

=== TEST 1: sanity
=== TEST 2: get
--- LAST
=== TEST 3: set

You may wonder, I can understand the importance of ONLY and SKIP, but what is the use of LAST? In fact, sometimes your test cases have dependencies, and you need to execute the first few test cases before the subsequent tests make sense. So, in this case, LAST is very useful when you continue debugging.

plan

Of all the test::nginx functions, plan is one of the most maddening and difficult to understand. It is derived from Perl's Test::Plan module, the documentation of which is not in test::nginx, and finding an explanation of it is not easy. Therefore, I'll introduce it in the early part. I've seen several OpenResty code contributors who have fallen down this hole and couldn't even climb out.

Here is an example of a similar configuration you can see at the beginning of every file in the official OpenResty test set:

plan tests => repeat_each() * (3 * blocks());

The meaning of plan here is how many tests should be done according to the plan in the whole test file. If the result of the final run does not match the plan, the test will fail.

For this example, if the value of repeat_each is 2 and there are 10 test cases, then the value of plan should be 2 x 3 x 10 = 60. The only thing you may feel confused about is the meaning of the number 3, which looks like a magic number!

Don't worry, let's continue to look at the example, you will be able to figure it out in a moment. First, can you figure out what the correct value of plan is in the following test case?

=== TEST 1: sanity
--- config
    location /t {
        content_by_lua_block {
            ngx.say("hello")
        }
    }
--- request
GET /t
--- response_body
hello

I believe everyone would conclude that plan = 1, since the test only checks the response_body.

But that's not the case! The correct answer is that plan = 2. Why? Because test::nginx has an implied check, i.e., --- error_code: 200, which detects if the HTTP response code is 200 by default.

So, the magic number 3 above really means that each test is explicitly checked twice, for example for body and error log, and implicitly for response code.

Since this is so error-prone, I recommend you turn off plan by using the following method.

use Test::Nginx::Socket 'no_plan';

If you can't turn it off, for example, if you encounter an inaccurate plan in the official OpenResty test set, it is recommended that you don't delve into the cause, but simply add or subtract numbers to the plan's expressions.

plan tests => repeat_each() * (3 * blocks()) + 2;

This is also the official method that will be used.

Preprocessor

We know that there may be some public settings between different test cases of the same test file. If the settings are repeated in each test case, it will make the code redundant and troublesome to modify later.

At this point, you can use the add_block_preprocessor directive to add a piece of Perl code, such as the following:

add_block_preprocessor(sub {
    my $block = shift;

    if (!defined $block->config) {
        $block->set_value("config", <<'_END_');
    location = /t {
        echo $arg_a;
    }
    _END_
    }
});

This preprocessor adds a config section to all test cases, and the content is location /t, so that in your later test cases, you can omit the config and access it directly.

=== TEST 1:
--- request
    GET /t?a=3
--- response_body
3

=== TEST 2:
--- request
    GET /t?a=blah
--- response_body
blah

Custom Functions

In addition to adding Perl code to the preprocessor, you can also arbitrarily add Perl functions, or custom functions as we call them, before the run_tests function.

Here is an example that adds a function that reads a file and combines it with the eval directive to implement a POST file:

sub read_file {
    my $infile = shift;
    open my $in, $infile
        or die "cannot open $infile for reading: $!";
    my $content = do { local $/; <$in> };
    close $in;
    $content;
}

our $CONTENT = read_file("t/test.jpg");

run_tests;

__DATA__

=== TEST 1: sanity
--- request eval
"POST /\n$::CONTENT"

Shuffle

In addition to the above, test::nginx has a little-known pitfall: it executes test cases in random order by default, instead of following the order and numbering of the test cases.

It was initially intended to test for more problems. After all, after each test case is run, the NGINX process is closed, and a new NGINX process is started to execute it, so the results should not be related to the order.

For underlying-level projects, this is true. However, for application-level projects, persistent storage such as databases exists externally. A haphazard execution can lead to wrong results. Since it is random each time, it may or may not report an error, and the error may be different each time. This obviously causes confusion for developers, including me, as I have stumbled here many times.

So, my advice is: please turn off this feature. You can turn it off with the following two lines of code:

no_shuffle();
run_tests;

In particular, the no_shuffle is used to disable randomization and allows the tests to run strictly in the order of the test cases.

reindex

OpenResty's test case set has strict formatting requirements. Each test case needs to be separated by three newlines, and the test case numbering has to be strictly self-growing.

Fortunately, we have an automatic tool, reindex, to do this tedious stuff, which is hidden in the openresty-devel-utils project. As there is no documentation about it, only a few people know about it.

If you are interested, you can try to mess up the test case numbering, or add or remove the number of line breaks, and then use this tool to sort it out and see if you can restore it.

Summary

This is the end of the introduction to test::nginx. Of course, there are more functions, we have only talked about the core and most important ones. "Give a man a fish and you feed him for a day; teach him how to fish and you feed him for a lifetime." I have taught you the basic methods and precautions for learning testing, then you can dig into the official test case set for better understanding.

Finally, please think about the questions below. Are there tests in your project development? And what framework do you use to test? Welcome you to share this article with more people to exchange and learn together.