What Is the Difference Between LuaJIT And Standard Lua?

Let's learn about LuaJIT, another cornerstone of OpenResty, and I'll leave the central part of today's post to some essential and lesser-known aspects of Lua and LuaJIT.

You can learn more about the basics of Lua through search engines or Lua books, and I recommend the book Programming in Lua by Lua author.

Of course, the threshold for writing the correct LuaJIT code in OpenResty is not high. Still, it is not easy to write efficient LuaJIT code, and I will cover the key elements here in detail in the OpenResty performance optimization section later.

Let's look at where LuaJIT fits into the overall OpenResty architecture.

As mentioned earlier, OpenResty Worker processes are obtained by forking the Master process. The LuaJIT virtual machine in the Master process is also forked. All Worker processes within the same Worker share this LuaJIT virtual machine, and Lua code execution is done in this virtual machine.

This is the basics of how OpenResty works, which we'll discuss in more detail in subsequent articles. Today we'll start by straightening out the relationship between Lua and LuaJIT.

Relationship between Standard Lua and LuaJIT

Let's get the essential things out of the way first.

Standard Lua and LuaJIT are two different things. LuaJIT is only compatible with Lua 5.1 syntax.

The latest version of standard Lua is now 5.4.4, and the latest version of LuaJIT is 2.1.0-beta3. In older versions of OpenResty from a few years ago, you could choose to use either the standard Lua VM or the LuaJIT VM as the execution environment when compiling, but now support for standard Lua has been removed, and only LuaJIT is supported.

The LuaJIT syntax is compatible with Lua 5.1, with optional support for Lua 5.2 and 5.3. So we should first learn the Lua 5.1 syntax and build on that to learn the LuaJIT features. In the previous article, I took you to the basic syntax of Lua. Today I will only mention some unique features of Lua.

It is worth noting that OpenResty does not directly use the official LuaJIT version 2.1.0-beta3, but instead extends it with its fork: openresty-luajit2.

These unique APIs were added during the actual development of OpenResty for performance reasons. So, the LuaJIT we mention later refers to the LuaJIT branch maintained by OpenResty itself.

Why LuaJIT?

After all this talk about the relationship between LuaJIT and Lua, you may wonder why you don't use Lua directly, but use LuaJIT. In fact, the main reason is the performance advantage of LuaJIT.

Lua code is not interpreted directly but compiled into Byte Code by the Lua compiler and then executed by the Lua virtual machine.

The LuaJIT runtime environment, in addition to an assembly implementation of the Lua interpreter, has a JIT compiler that can generate machine code directly. At first, LuaJIT starts like standard Lua, with Lua code compiled to byte code, which is interpreted and executed by LuaJIT's interpreter.

The difference is that the LuaJIT interpreter records some runtime statistics while executing the bytecode, such as the actual number of times each Lua function call entry is run and the actual number of times each Lua loop is executed. When these counts exceed a random threshold, the corresponding Lua function entry or Lua loop is considered hot enough to trigger the JIT compiler to start working.

The JIT compiler tries to compile the corresponding Lua code path, starting from the hot function's entry or the hot loop's location. The compilation process converts the LuaJIT bytecode into LuaJIT's own defined IR(Intermediate Representation) and then generates machine code for the target architecture.

So, the so-called LuaJIT performance optimization is essentially about making as much Lua code available as possible for machine code generation by the JIT compiler, rather than falling back to the interpreted execution mode of the Lua interpreter. Once you understand this, you can understand the nature of OpenResty performance optimization that you will learn later.

Lua's special features

As described in the previous article, the Lua language is relatively simple. For engineers with a background in other development languages, it's easy to see the logic of the code once you notice some unique aspects of Lua. Next, let's look at some of the more unusual aspects of the Lua language.

1. Index starts from 1

Lua is the only programming language I know of that starts with an index of 1. This, while better understood by non-programmer backgrounds, is prone to program bugs. Here is an example.

$ resty -e 't={100}; ngx.say(t[0])'

You might expect the program to print out 100 or to report an error saying that the index 0 does not exist. But surprisingly, nothing is printed, and no errors are reported. So let's add the type command and see what the output is.

$ resty -e 't={100};ngx.say(type(t[0]))'
nil

It turns out to be nil value. In fact, in OpenResty, the determination and handling of nil values is also a confusing point, so we will talk more about it later when we talk about OpenResty.

2. use .. to concatenate strings

Unlike most languages that use +, Lua uses two dot marks to concatenate strings.

$ resty -e "ngx.say('hello' .. ', world')"
hello, world

In actual project development, we generally use multiple development languages, and Lua's unusual design will always make developers think when the string splicing sticks a bit is also confusing.

3. The table is the only data structure

Unlike Python, a language rich in built-in data structures, Lua has only one data structure, the table, which can include arrays and hash tables.

local color = {first = "red", "blue", third = "green", "yellow"}
print(color["first"])                 --> output: red
print(color[1])                         --> output: blue
print(color["third"])                --> output: green
print(color[2])                         --> output: yellow
print(color[3])                         --> output: nil

If you don't explicitly assign a value as a key-value pair, the table defaults to a number as an index, starting with 1. So color[1] is blue.

Also, getting the correct length in the table is difficult, so let's look at these examples.

local t1 = { 1, 2, 3 }
print("Test1 " .. table.getn(t1))

local t2 = { 1, a = 2, 3 }
print("Test2 " .. table.getn(t2))

local t3 = { 1, nil }
print("Test3 " .. table.getn(t3))

local t4 = { 1, nil, 2 }
print("Test4 " .. table.getn(t4))

Result:

Test1 3
Test2 2
Test3 1
Test4

As you can see, except for the first test case that returns a length of 3, the subsequent tests are all outside our expectations. In fact, to get the table length in Lua, it is important to note that the correct value is returned only if the table is a sequence.

So what is a sequence? First of all, a sequence is a subset of an array. That is, the elements of a table are accessible with a positive integer index and there are no key-value pairs. In the above code, all the tables are arrays except for t2.

Second, the sequence does not contain a hole, i.e., nil. Combining these two points, the above table, t1, is a sequence, while t3 and t4 are arrays but not sequences.

Up to this point, you may still have a question, why the length of t4 will be 1? This is because when nil is encountered, the logic to get the length does not continue to run but returns directly.

I don't know if you understand it completely. This part is really quite complicated. So is there any way to get the table length we want? Naturally, there is. OpenResty extends this, and I'll talk about it later in the chapter dedicated to the table, so let's leave suspense here.

4. All variables are global by default

I'd like to emphasize that unless you're fairly certain, you should always declare new variables as local variables.

local s = 'hello'

This is because, in Lua, variables are global by default and are placed in a table named _G. Variables that are not local are looked up in the global table, which is an expensive operation. Misspelling variable names can lead to bugs that are difficult to identify and fix.

So, in OpenResty, I strongly recommend that you always declare variables using local, even when you require a module.

-- Recommended
local xxx = require('xxx')

-- Avoid
require('xxx')

LuaJIT

With these four special features of Lua in mind, let's move on to LuaJIT.

LuaJIT, in addition to being Lua 5.1 compliant and supporting JIT, is tightly integrated with the FFI (Foreign Function Interface), allowing you to call external C functions and use C data structures directly in your Lua code. Here is the simplest example.

local ffi = require("ffi")
ffi.cdef[[
int printf(const char *fmt, ...);
]]
ffi.C.printf("Hello %s!", "world")

In just a few lines of code, you can call C's printf function directly from Lua and print out Hello world! You can use the resty command to run it and see if it works.

Similarly, we can use FFI to call the C functions of NGINX and OpenSSL to do much more. The FFI approach performs better than the traditional Lua/C API approach, which is why the lua-resty-core project exists. In the next section, we will talk about FFI and lua-resty-core.

In addition, for performance reasons, LuaJIT extends the table functions: table.new and table.clear, two essential performance optimization functions frequently used in OpenResty's lua-resty library. However, few developers are familiar with them, as the documentation is intense and there is no sample code. We'll save them for the performance optimization section.

Summary

Let's review today's content.

OpenResty chooses LuaJIT rather than standard Lua for performance reasons and maintains its LuaJIT branch. LuaJIT is based on the Lua 5.1 syntax and is selectively compatible with some Lua 5.2 and Lua 5.3 syntax to form its system. As for the Lua syntax you need to master, it has its distinctive features in index, string splicing, data structures, and variables, which you should pay special attention to when writing code.

Have you encountered any pitfalls when learning Lua and LuaJIT? Feel free to share your opinions with us, and I've written a post to share the pitfalls I've encountered. You are also welcome to share this post with your colleagues and friends to learn and progress together.