How Apache APISIX Enables Wasm Functionality
Xinxin Zhu
April 7, 2023
To enable high-performance applications to run in web browsers, Apache APISIX supports Wasm at the gateway level, a binary instruction format that enables efficient and secure execution of code on the web. Consequently, developers can use high-level programming languages such as C/C++/Go/Rust and adhere to the proxy-wasm specification to create Wasm plugins. The proxy-wasm specification ensures compatibility and interoperability with other systems. Let's get into this article and learn more details.
What is Wasm
WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine.
Prior to the emergence of Wasm, only Javascript could be executed in web browsers. However, with the introduction of Wasm, high-level languages like C/C++/Golang can now be run in web browsers. Major browsers such as Chrome, Firefox, and Safari now all support Wasm. Moreover, due to the advancements made by the WASI (the WebAssembly System Interface) project, server-side environments can also support the execution of Wasm instructions.
At present, Apache APISIX supports Wasm at the gateway level. Developers can use high-level programming languages such as C/C++/Go/Rust and adhere to the proxy-wasm specification to create Wasm plugins.
Why Does APISIX Support Wasm Plugins?
Compared to native Lua plugins, Wasm plugins offer several advantages:
-
Scalability: By supporting Wasm, APISIX can utilize the SDK provided by proxy-wasm to develop plugins in high-level programming languages such as C++, Golang, and Rust. These languages often have richer ecosystems, allowing developers to implement more feature-rich plugins.
-
Security: Since the interaction between APISIX and Wasm relies on the Application Binary Interface (ABI) provided by proxy-wasm, this access is more secure. Wasm plugins can only make specific modifications to requests, ensuring that they cannot perform malicious actions. Furthermore, since Wasm plugins run in a separate VM, even if the plugin crashes, it will not affect the main APISIX process.
How Does APISIX Support Wasm?
Now that we understand what Wasm is, let's take a top-down look at how APISIX supports Wasm plugins.
APISIX Wasm Plugins
APISIX allows developers to create plugins using popular high-level programming languages such as C/C++, Go, and Rust. These plugins can be built using the corresponding SDK and following the proxy-wasm specification.
proxy-wasm is a specification for ABIs between L4/L7 proxies, introduced by Envoy. This specification defines ABIs including memory management, L4 proxy, and L7 proxy extensions. For example, in HTTP(L7), the proxy-wasm specification defines ABIs such as proxy_on_http_request_headers, proxy_on_http_request_body, proxy_on_http_request_trailers, and proxy_on_http_response_headers, allowing modules to retrieve and modify request content at various stages.
For example, we will use Golang and proxy-wasm-go-sdk to develop this plugin:
The proxy-wasm-go-sdk is an SDK for the proxy-wasm specification that helps developers to create proxy-wasm plugins more easily using Golang. However, it's important to note that due to some issues with native Golang's support for WASI, this SDK is implemented based on TinyGo. For more information, you can click here to view the details.
This plugin's main function is to modify the HTTP response status code and response body of an HTTP modification request, as referenced from the APISIX link.
...
func (ctx *pluginContext) OnPluginStart(pluginConfigurationSize int) types.OnPluginStartStatus {
data, err := proxywasm.GetPluginConfiguration()
if err != nil {
proxywasm.LogErrorf("error reading plugin configuration: %v", err)
return types.OnPluginStartStatusFailed
}
var p fastjson.Parser
v, err := p.ParseBytes(data)
if err != nil {
proxywasm.LogErrorf("error decoding plugin configuration: %v", err)
return types.OnPluginStartStatusFailed
}
ctx.Body = v.GetStringBytes("body")
ctx.HttpStatus = uint32(v.GetUint("http_status"))
if v.Exists("percentage") {
ctx.Percentage = v.GetInt("percentage")
} else {
ctx.Percentage = 100
}
// schema check
if ctx.HttpStatus < 200 {
proxywasm.LogError("bad http_status")
return types.OnPluginStartStatusFailed
}
if ctx.Percentage < 0 || ctx.Percentage > 100 {
proxywasm.LogError("bad percentage")
return types.OnPluginStartStatusFailed
}
return types.OnPluginStartStatusOK
}
func (ctx *httpLifecycle) OnHttpRequestHeaders(numHeaders int, endOfStream bool) types.Action {
plugin := ctx.parent
if !sampleHit(plugin.Percentage) {
return types.ActionContinue
}
err := proxywasm.SendHttpResponse(plugin.HttpStatus, nil, plugin.Body, -1)
if err != nil {
proxywasm.LogErrorf("failed to send local response: %v", err)
return types.ActionContinue
}
return types.ActionPause
}
...
Afterwards, we use TinyGo to compile the above Golang code and generate a .wasm file.
tinygo build -o wasm_fault_injection.go.wasm -scheduler=none -target=wasi ./main.go
After the compilation is completed, we obtain the fault_injection.go.wasm file.
If you're interested in the contents of the wasm file, you can use the wasm-tool to view the specific contents of the wasm file.
wasm-tools dump hello.go.wasm
Configure wasm_fault_injection.go.wasm in the config.yaml file of APISIX and name the plugin as wasm_fault_injection.
apisix:
...
wasm:
plugins:
- name: wasm_fault_injection
priority: 7997
file: wasm_fault_injection.go.wasm
Afterwards, we start APISIX and create a route that references the Wasm plugin:
curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '{
"uri":"/*",
"upstream":{
"type":"roundrobin",
"timeout":{
"connect":1,
"read":1,
"send":1
},
"nodes":{
"httpbin.org:80":1
}
},
"plugins":{
"wasm_fault_injection":{
"conf":"{\"http_status\":200, \"body\":\"Hello WebAssembly!\n\"}"
}
},
"name":"wasm_fault_injection"
}'
After conducting an access test, we found that the response body has been modified to "Hello WebAssembly", indicating that the Wasm plugin is now effective.
curl 127.0.0.1:9080/get -v
* Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
> GET /get HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 09 Feb 2023 07:46:50 GMT
< Content-Type: text/plain; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Server: APISIX/3.1.0
<
Hello WebAssembly!
Wasm-nginx-module
After learning how Apache APISIX uses Wasm plugins, let's dive deeper into "why can we access and modify request contents in Wasm plugins?"
Since APISIX uses OpenResty as the underlying framework, to be able to access and modify request contents in Wasm plugins, we need to interact with APIs provided by OpenResty or NGINX. This is exactly what wasm-nginx-module does.
wasm-nginx-module is an NGINX module that supports Wasm and was developed by API7. This module attempts to implement proxy-wasm-abi based on NGINX and encapsulates Lua API, allowing us to complete proxy-wasm-abi calls at the Lua level. For more information, please refer to wasm-nginx-module.
For example, when APISIX runs to the access phase, it calls the Lua method on_http_request_headers provided by wasm-nginx-module.
-- apisix/wasm.lua
...
local ok, err = wasm.on_http_request_headers(plugin_ctx)
if not ok then
core.log.error(name, ": failed to run wasm plugin: ", err)
return 503
end
end
...
Later in this method, the ngx_http_wasm_on_http method provided by wasm-nginx-module will be called.
ngx_int_t
ngx_http_wasm_on_http(ngx_http_wasm_plugin_ctx_t *hwp_ctx, ngx_http_request_t *r,
ngx_http_wasm_phase_t type, const u_char *body, size_t size,
int end_of_body)
{
...
ctx = ngx_http_wasm_get_module_ctx(r);
if (type == HTTP_REQUEST_HEADERS) {
cb_name = &proxy_on_request_headers;
} else if (type == HTTP_REQUEST_BODY) {
cb_name = &proxy_on_request_body;
} else if (type == HTTP_RESPONSE_HEADERS) {
cb_name = &proxy_on_response_headers;
} else {
cb_name = &proxy_on_response_body;
}
if (type == HTTP_REQUEST_HEADERS || type == HTTP_RESPONSE_HEADERS) {
if (hwp_ctx->hw_plugin->abi_version == PROXY_WASM_ABI_VER_010) {
rc = ngx_wasm_vm->call(hwp_ctx->hw_plugin->plugin,
cb_name,
true, NGX_WASM_PARAM_I32_I32, http_ctx->id, 0);
} else {
rc = ngx_wasm_vm->call(hwp_ctx->hw_plugin->plugin,
cb_name,
true, NGX_WASM_PARAM_I32_I32_I32, http_ctx->id,
0, 1);
}
} else {
rc = ngx_wasm_vm->call(hwp_ctx->hw_plugin->plugin,
cb_name,
true, NGX_WASM_PARAM_I32_I32_I32, http_ctx->id,
size, end_of_body);
}
...
}
In wasm-nginx-module, we set the cb_name based on different stages, such as HTTP_REQUEST_HEADERS corresponds to proxy_on_request_headers, and then call the method in the VM through ngx_wasm_vm->call, which is the OnHttpRequestHeaders method of the wasm plugin mentioned earlier in this article.
With this, the entire calling chain of APISIX calling wasm plugin and running Golang is completed. The calling chain is as follows:
Wasm VM
Wasm VM is a virtual machine that is used to execute Wasm code. wasm-nginx-module implements two types of virtual machines, "wasmtime" and "wasmedge", for executing Wasm code. In APISIX, "wasmtime" is the default for running Wasm code.
Wasmtime is a fast and secure runtime for WebAssembly and WASI, open-sourced by Bytecode Alliance. It can run WebAssembly code outside the web environment and can be used as a command-line tool or embedded as a WebAssembly runtime engine in other programs as a library. Wasmedge is a lightweight, high-performance, and scalable WebAssembly (Wasm) virtual machine optimized for edge computing. It can be used for cloud-native, edge, and decentralized applications.
First, in the Wasm VM, we use the load method to load the .wasm file into memory. After that, we can use the call method of the VM to invoke these functions. The VM is based on the WASI interface implementation, which enables Wasm code to not only run on the browser-side but also support running on the server-side.
Summary
We have gained an understanding of what Wasm is and how APISIX supports Wasm plugins. By offering support for Wasm plugins, APISIX not only enhances its capabilities in supporting multiple languages, such as C++, Rust, Golang, and AssemblyScript for plugin development but also benefits from the extensive ecosystem and use cases of WebAssembly, which is expanding beyond the browser to cloud-native environments.
As a result, APISIX can leverage Wasm to deliver more advanced features on the API gateway side, enabling it to address a wider range of usage scenarios.