Documentation and Test Cases: Powerful Tools for Solving OpenResty Development Problems

After learning the principles and a few essential concepts of OpenResty, we're finally going to start learning the API.

From my personal experience, learning OpenResty API is relatively easy, so it doesn't take many articles to introduce. You may wonder: isn't the API the most common and essential part? Why not spend a lot of time on it? There are two primary considerations.

First, OpenResty provides very detailed documentation. Compared with many other programming languages or platforms, OpenResty provides not only the API parameters and return value definitions but also complete and runnable code examples, clearly showing you how the API handles various boundary conditions.

Following the API definition with example code and caveats is a consistent style of OpenResty documentation. Therefore, after reading the API description, you can immediately run the sample code in your environment and modify the parameters and the documentation to verify them and deepen your understanding.

Second, OpenResty provides comprehensive test cases. As I mentioned, the OpenResty documentation shows code examples of APIs. However, due to space constraints, the document does not present error reporting and processing in various abnormal situations and the method of using multiple APIs.

But don't worry. You can find most of these contents in the test case set.

For OpenResty developers, the best API learning materials are the official documentation and test cases, which are professional and friendly to readers.

Give a man a fish, and you feed him for a day; teach a man to fish and feed him for a lifetime. Let's use a real example to experience how to exert the power of the documentation and test case set in OpenResty development.

Take the `get` API of shdict as an example

Based on NGINX shared memory area, the shared dict (shared dictionary) is a Lua dictionary object, which can access data across multiple workers and store data such as rate limiting, cache, etc. There are more than 20 APIs related to shared dict-the most commonly used and crucial API in OpenResty.

Let's take the most straightforward get operation as an example; you can click on the documentation link for comparison. The following minimized code example is adapted from the official documentation.

http {
      lua_shared_dict dogs 10m;
      server {
          location /demo {
              content_by_lua_block {
                  local dogs = ngx.shared.dogs
                  dogs:set("Jim", 8)
                  local v = dogs:get("Jim")
                  ngx.say(v)
              }
          }
      }
  }

As a quick note, before we can use shared dict in Lua code, we need to add a block of memory in nginx.conf with the lua_shared_dict directive, which is named "dogs" and has a size of 10M. After modifying nginx.conf, you need to restart the process and access it with a browser or curl command to see the results.

Doesn't this seem a bit tedious? Let's modify it more straightforwardly. As you can see, using the resty CLI in this way has the same effect as embedding the code in nginx.conf.

$ resty --shdict 'dogs 10m' -e 'local dogs = ngx.shared.dogs
 dogs:set("Jim", 8)
 local v = dogs:get("Jim")
 ngx.say(v)
 '

You now know how nginx.conf and Lua code work together, and you have successfully run the set and get methods of the shared dict. Generally, most developers stop there. There are a few things worth noting here.

which stages can't use the shared memory-related APIs?
We see in the sample code get function has only one return value. Then when will there be more than one return value?
What is the type of input to the get function? Is there a length limit?

Don't underestimate these questions; they can help us to understand OpenResty better, and I'll take you through them individually.

Let's look at the first question. The answer is straightforward; the documentation has a dedicated context (i.e. context section) that lists the environments in which the API can be used.

context: set_by_lua*, rewrite_by_lua*, access_by_lua*, content_by_lua*, header_filter_by_lua*, body_filter_by_lua*, log_by_lua*, ngx.timer.*, balancer_by_lua*, ssl_certificate_by_lua*, ssl_session_fetch_by_lua*, ssl_session_store_by_lua*

As you can see, the init and init_worker phases are not included, which means that the get API that shared memory cannot be used in these two phases. Please note that each shared memory API can be used in different phases. For example, the set API can be used in the init phase.

Always, read the documentation when using it. Of course, OpenResty's documentation sometimes contains errors and omissions, so you need to verify them with actual tests.

Next, let's modify the test set to make sure that the init phase can run the get API of the shared dict

How can we find the test case set related to shared memory? OpenResty's test cases are all placed in the /t directory and named regularly, i.e., self-incremented-number-function-name.t. Search for shdict, and you will find 043-shdict.t, the shared memory test case set, which contains close to 100 test cases, including tests for various normal and abnormal circumstances.

Let's try to modify the first test case.

You can replace the content phase with an init phase and remove the extraneous code to see if the get interface works. You don't have to understand how the test case is written, organized, and run at this stage. You only need to know that it is testing the get interface.

 === TEST 1: string key, int value
     --- http_config
         lua_shared_dict dogs 1m;
     --- config
         location = /test {
             init_by_lua '
                 local dogs = ngx.shared.dogs
                 local val = dogs:get("foo")
                 ngx.say(val)
             ';
         }
     --- request
     GET /test
     --- response_body
     32
     --- no_error_log
     [error]
     --- ONLY

You should have noticed that at the end of the test case, I added the --ONLY flag, which means to ignore all other test cases, and only run this one, thus improving the running speed. Later in the test section, I will specifically explain the various tags.

After the modification, we can run the test case with the prove command.

prove t/043-shdict.t

Then, you will get an error that corroborates the phase limits described in the documentation.

nginx: [emerg] "init_by_lua" directive is not allowed here

Question 2: When does the `get` function have multiple return values?

Let's look at the second question, which can be summarized from the official documentation. The documentation begins with the syntax description of this interface.

value, flags = ngx.shared.DICT:get(key)

Under normal circumstances.

The first parameter value returns the value corresponding to the key in the dictionary; however, when the key does not exist or expires, the value value is nil.
The second parameter, flags, is a little more complicated; if the set interface sets flags, it returns them. Otherwise, it does not.

If the API call goes wrong, value returns nil, and flags return a specific error message.

From the information summarized in the documentation, we can see that local v = dogs:get("Jim") is written with only one receiving parameter. Such writing is incomplete because it only covers the typical usage scenario without receiving a second parameter or conducting exception handling. We could modify it into the following.

local data, err = dogs:get("Jim")
if data == nil and err then
    ngx.say("get not ok: ", err)
    return
end

As with the first question, we can search the test case set to confirm our understanding of the documentation.

  === TEST 65: get nil key
     --- http_config
         lua_shared_dict dogs 1m;
     --- config
         location = /test {
             content_by_lua '
                 local dogs = ngx.shared.dogs
                 local ok, err = dogs:get(nil)
                 if not ok then
                     ngx.say("not ok: ", err)
                     return
                 end
                 ngx.say("ok")
             ';
         }
     --- request
     GET /test
     --- response_body
     not ok: nil key
     --- no_error_log
     [error]

In this test case, the get interface has nil input, and the returned err message is nil key. This verifies that our analysis of the documentation is correct and provides a partial answer to the third question. At least, the input to get cannot be nil.

Question 3: What is the type of input to the `get` function?

As for the third question, what kind of input parameters to get can it be? Let's check the documentation first, but unfortunately, you will find that the documentation does not specify what the legal types of keys are. What should we do?

Don't worry. At least we know that the key can be a string type and cannot be nil. Do you remember the data types in Lua? In addition to strings and nil, there are numbers, arrays, boolean types, and functions. The latter two are unnecessary as keys, so we only need to verify the first two: numbers and arrays. We can start by searching the test file for cases where numbers are used as key.

=== TEST 4: number keys, string values

With this test case, you can see that numbers can also be used as keys, and internally they will be converted to strings. What about arrays? Unfortunately, the test case doesn't cover it, so we need to try it ourselves.

$ resty --shdict 'dogs 10m' -e 'local dogs = ngx.shared.dogs
 dogs:get({})
 '

Not surprisingly, the following error was reported.

ERROR: (command line -e):2: bad argument #1 to 'get' (string expected, got table)

In summary, we can conclude that the key types accepted by the get API are strings and numbers.

So is there a limit length of the incoming key? There is a corresponding test case here.

=== TEST 67: get a too-long key
     --- http_config
         lua_shared_dict dogs 1m;
     --- config
         location = /test {
             content_by_lua '
                 local dogs = ngx.shared.dogs
                 local ok, err = dogs:get(string.rep("a", 65536))
                 if not ok then
                     ngx.say("not ok: ", err)
                     return
                 end
                 ngx.say("ok")
             ';
         }
     --- request
     GET /test
     --- response_body
     not ok: key too long
     --- no_error_log
     [error]

When the string length is 65536, you will be prompted that the key is too long. You can try to change the length to 65535, although only 1 byte less, but no more errors. This means that the maximum length of the key is exactly 65535.

Summary

Finally, I would like to remind you that in the OpenResty API, any return value with an error message must have a variable to receive and do error handling, otherwise make a mistake. For example, if the wrong connection is put into the connection pool, or if the API call fails to continue the logic behind it, it makes people complain incessantly.

So, if you encounter a problem when you write OpenResty code, what is your usual way to solve it? Is it documentation, mailing lists, or other channels?

Welcome to share this article with your colleagues and friends so we can communicate and improve.

Documentation and Test Cases: Powerful Tools for Solving OpenResty Development Problems

Take the get API of shdict as an example

Question 1: Which stages cannot use shared memory-related APIs?

Question 2: When does the get function have multiple return values?

Question 3: What is the type of input to the get function?

Summary

Take the `get` API of shdict as an example

Question 2: When does the `get` function have multiple return values?

Question 3: What is the type of input to the `get` function?