360: Apache APISIX in practice in the basic operation and maintenance platform project


Update At 12/11/2020

Today, I share an article about Apache APISIX, which describes the implementation practice of Apache APISIX gateway in 360 basic operation and maintenance platform from the developer's perspective.

PS: rich front-line technology, diversified forms of expression, as in the "360 cloud computing", point attention Oh!

API Gateway Selection

In October 2019, our team planned to revamp the gateway layer of 360 Basic Operations and Maintenance Platform. At that time, we mainly investigated several active gateways in the community, such as Kong, Orange, Apache APISIX, and finally chose Apache APISIX mainly because the storage selection etcd of Apache APISIX was more in line with our usage scenario.


Online operation

The number of APIs we have added to the gateway is close to 900, with an average daily PV of 10 million, and from the monitoring system, the gateway and our microservices are running well.

  • Average Daily PV


  • Gateway POD Monitoring


  • Microservice load monitoring diagram


Basic operation and maintenance platform architecture diagram

The following diagram is the final architecture of our O&M platform project, the gateway service we deployed on the company's container cloud, and the etcd service we deployed a set of clusters on 3 virtual machines.


Containerized Development and Deployment

Next I'll describe how we built the gateway service using Apache APISIX. First, I'll show you the code structure of our gateway project


When I showed Wang Yuansheng (one of the Apache APISIX PMCs) the code structure of our project, he was surprised and asked me how he didn't see the core code of Apache APISIX.

This is actually a path that we have explored when installing Apache APISIX using containers. The biggest benefit it gives us is that our business code is completely separate from the core Apache APISIX code, making it easy to upgrade Apache APISIX and to iterate on our business code.

I'll give you a step-by-step demonstration of how we build one of these environments. (This assumes that you all know about docker container technology)

  • Start the openresty container

    Our team repackaged a new image based on the official openresty image with some Apache APISIX dependencies such as luarocks installed by default, details can be found at https://hub.docker.com/r/hulklab/openresty/dockerfile

    1> docker run -itd -p 9080:9080 -p9443:9443 --name myapisix hulklab/openresty:0.0.1
  • Enter the container

    1> docker exec -it myapisix bash
  • Install apisix version 0.9

    1> luarocks install apisix 0.9-0
    2# At the end of the installation you will see the following line of output, from which we can see that the apisix installation directory is "/usr/local/apisix":
    3apisix 0.9-0 is now installed in /usr/local (license: Apache License 2.0)
  • Go to the Apache APISIX installation directory

    1> cd /usr/local/apisix
    2# When you go to the apisix installation directory, you will find only two directories inside `conf` `logs`
    3> ls -l
    4drwxr-xr-x 3 root   root 4096 Dec 18 11:52 confdrwxr-xr-x 2 root   root 4096 Dec 18 11:37 logs
  • Start Apache APISIX (running Openresty)

    1> apisix start
    2> ps aux|grep openresty
    3root         1  0.0  0.0  11316  4464 pts/0    Ss+  11:24   0:00 nginx: master process /usr/bin/openresty -g daemon off;root      5040  0.0  0.0  87436  2588 ?        Ss   11:37   0:00 nginx: master process openresty -p /usr/local/apisix -c /usr/local/apisix/conf/nginx.conf

Note: If apisix start fails, because Apache APISIX relies on etcd, you need to start etcd, please refer to the official documentation of etcd [1] for how to start etcd, after starting etcd, you need to modify "/usr/local/apisix/conf/config.yaml", e.g.

1#config.yaml:69etcd:  host: ""   # etcd address
  • View nginx.conf

apisix runs successfully, but there is no code in the installation directory "/usr/local/apisix", so where is the apisix core code and dependencies?

From the started openresty process, we can see that there is an extra nginx.conf under the apisix/conf directory. This nginx.conf configuration file is initialized when the apisix start command is executed. Let's check the lua in nginx.conf Package reference path.

1> cat /usr/local/apisix/conf/nginx.conf|grep lua_package_path
2lua_package_path  "$prefix/deps/share/lua/5.1/?.lua;/usr/local/apisix/lua/?.lua;;/usr/local/apisix/deps/share/lua/5.1/apisix/lua/?.lua;/usr/local/apisix/deps/share/lua/5.1/?.lua;/usr/share/lua/5.1/apisix/lua/?.lua;/usr/local/share/lua/5.1/apisix/lua/?.lua;/root/.luarocks/share/lua/5.1/?.lua;/root/.luarocks/share/lua/5.1/?/init.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;./?.lua;/usr/local/lib/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua;";

Looking at the above lua_package_paths one by one, we find two useful pieces of information.

  1. Path to Apache APISIX core code: "/usr/local/share/lua/5.1/apisix/lua/" .
  2. The lua files under the Apache APISIX installation path /usr/local/apisix/lua have the highest priority for loading.

So we made an attempt, imitating the plugin path of Apache APISIX, creating lua/apisix/plugins/my-plugin.lua in the /usr/local/apisix directory, and adding the plugin to the configuration file config.yaml, and found that It took effect.

  • Dockerfile

Post the Dockerfile of our project for your reference. In the end, our project has only two directories, conf and lua. conf stores our own config.yaml and nginx.conf configuration files, and lua stores our custom plug-ins and class libraries.

1FROM hulklab/openresty:0.0.1
3RUN luarocks install apisix 0.9-0; \
4    luarocks install lua-resty-cookie; \
5    luarocks install lua-resty-kafka; \
6    luarocks install lua-resty-url
8WORKDIR /usr/local/apisix
10RUN rm -rf conf/*; \
11    mkdir -p lua; \
12    mkdir -p logs/archive; \
13    install -d -m 777 /tmp/apisix_cores/
15COPY conf conf
16COPY lua lua
17COPY logrotate /etc/logrotate.d
19EXPOSE 9080 9443
21ENTRYPOINT ["openresty", "-p", "/usr/local/apisix", "-c", "/usr/local/apisix/conf/nginx.conf", "-g", "daemon off;"]

Plug-in development

1、Project plug-in introduction

As you can see in the code structure diagram above, there are two directories in the apisix directory of our project, libs and plugins. In libs we put some commonly used libraries, and plugins store our custom business plugins. We All businesses are developed using plug-in mechanisms. The picture below is the plug-in currently used in our project.


To explain a little, there are two entry domains for our project, one for openApi access, which uses Basic Auth for the authentication plugin, and one for web browser access, which uses web auth (cookie authentication) for the authentication plugin.

Corresponding to OpenResty's request processing flow, our plugin focuses on the access and log phases.

ip-restrictionaccess_by_luaip flow limiting, using apisix native plugin
basic-authaccess_by_luaRequesting user authentication for openApi, self-developed plugin
web-authaccess_by_luaRequesting user authentication for webApi, self-developed plugin
limit-rateaccess_by_luaUser level and user+request parameter level flow restriction for requests, self-developed plugins
proxy-rewriteaccess_by_lua,balancer_by_luaForwarding of requests, setting interface-level timeouts, self-developed plug-ins
loglog_by_luaRecord the request log to kafka, and then read it to es through logstash, self-developed plug-in
alarmlog_by_luaAlarm according to the response statusCode, self-developed plug-in

2、Sample plug-in development

The next section describes how the Apache APISIX plugin was developed, using the basic-auth plugin as an example.

  • Defining Plugin Objects

    1  local plugin_name = "odin-basic-auth"
    3  local schema = {
    4  	type = "object",
    5  	properties = {
    6  		enable = { type = "boolean", default = true, enum = { true, false } },
    7  	},
    8  }
    10  local _M = {
    11  	version = 0.1,
    12  	priority = 2802,
    13  	name = plugin_name,
    14  	schema = schema,
    15  }

The odin-basic-auth plug-in has only one parameter enable. The enable parameter indicates whether to use this plug-in. This is because the Apache APISIX plug-in can be bound to the service or route. If the plug-in is bound to the service, the route is not The plug-in is closed, so a parameter is needed to finely control the plug-in that a route does not use service binding. It is recommended that official plug-ins are equipped with this parameter.

  • Implementing a method for detecting plug-in parameters

    1  function _M.check_schema(conf)
    2  	local ok, err = core.schema.check(schema, conf)
    4  	if not ok then
    5  		return false, err
    6  	end
    8  	return true
    9  end

The check_schema method is basically the same for every plugin.

  • Methods for implementing the corresponding phases of the plug-in

    1  function _M.access(conf, ctx)
    3  	-- 0. Check the configuration file to see if enable is enabled
    4  	if not conf.enable then
    5  		return
    6  	end
    8  	-- 1. Get the username and password in basic_auth
    9  	local headers = ngx.req.get_headers()
    10  	if not headers.Authorization then
    11  		return 401, { message = "authorization is required" }
    12  	end
    14  	local username, password, err = extract_auth_header(headers.Authorization)
    15  	if err then
    16  		return 401, { message = err }
    17  	end
    19  	-- 2. Check etcd to get the record corresponding to the username
    20  	local res = authorizations_etcd:get(username)
    21  	if res == nil then
    22  		return 401, { message = "failed to find authorization from etcd" }
    23  	end
    25  	-- 3. If not, report authentication failure
    26  	if not res.value or not res.value.id then
    27  		return 401, { message = "user is not found" }
    28  	end
    30  	local value = res.value
    32  	-- 4. If so, determine if the user password is correct
    33  	if value.password ~= password then
    34  		return 401, { message = "password is error" }
    35  	end
    36  end

3、Etcd cache objects

In the second step of the above example, we use authorizations_etcd:get(username) to get the actual password of the current requesting user and the list of authorized routes, using the etcd cache object of Apache APISIX.

The principle of etcd caching objects is to use etcd's watch function to cache data from etcd to memory objects, which are directly read from the memory during business use to avoid network io consumption. etcd's watch function also guarantees the real-time data. This feature of Apache APISIX is simply amazing.

The following describes how to use.

  • Define an etcd environment object variable

    1  local authorizations_etcd
    3  -- Define the scheme of values stored in the etcd object
    4  local appkey_scheme = {
    5  	type = "object",
    6  	properties = {
    7  		username = {
    8  			description = "username",
    9  			type = "string",
    10  		},
    11  		password = {
    12  			type = "string",
    13  		}
    14  	},
    15  }
  • Instantiated in the init phase of the plugin

    1  function _M.init()
    3  	authorizations_etcd, err = core.config.new("/authorizations", {
    4  		automatic = true,
    5  		item_schema = appkey_scheme
    6  	})
    8  	if not authorizations_etcd then
    9  		error("failed to create etcd instance for fetching authorizations: " .. err)
    10  		return
    11  	end
    13  end

The init method of the plug-in occurs in the init_worker_by_lua phase of OpenResty, in other words, each worker is initialized only once. If the automatic parameter is set to true, Apache APISIX will enable the watch function. The business layer only needs to instantiate the etcd cache object, and Apache APISIX does the rest.

  • Using etcd cache objects in plugins

    1   local res = authorizations_etcd:get(username)

4、Use of Plugin API

The essence of the etcd cache object above still requires fetching data from etcd, so how is the user-related data used in this plugin added to etcd? This brings us to another screaming feature of the plugin: the API feature.

  • Define API

    1  function _M.API()
    2  	return {
    3  		{
    4  			methods = { "POST", "PUT" },
    5  			uri = "/apisix/plugin/basic-auth/set",
    6  			handler = set_auth,
    7  		}
    8  	}
    9  end
  • Implement the handler for the API

    1  local function set_auth()
    2  	local username = req.get_str("username")
    3  	local password = req.get_str("password")
    5  	local key = "/authorizations/" .. username
    7  	-- 此处存入到 etcd
    8  	local res, err = core.etcd.set(key, { username = username, password = password})
    9  	if not res then
    10  		core.response.exit(500, err)
    11  	end
    13  	core.response.exit(res.status, res.body)
    14  end
  • Calling the interface

    1  > curl -i -X PUT '' -d username=zhangsan -d password=hao123 -d user_id=3 -d action_ids=,1,2,3,

Problems encountered after going online

crontab clear day-end

As our gateway was deployed in the container, after running for a while, the log file exceeded the default quota of 50G, we later installed cron and logrotate in the image by default, and then turned on cron in the container entrypoint to solve this problem.


Finally, a special thanks to the contributors of Apache APISIX, posting the Apache APISIX website[2] and the Apache APISIX Github address[3].


[1] etcd Documents: https://doczhcn.gitbook.io/etcd/index

[2] Apache APISIX website: https://apisix.apache.org/

[3] Apache APISIX Github: https://github.com/apache/apisix

The above is the content of this sharing ~

If you have any suggestions, you can also leave them in our comment section for your reference and learning.