Application of API Gateway Apache APISIX in 360

December 11, 2020

Case Study

Today, I share an article about Apache APISIX, which describes the implementation practice of Apache APISIX gateway in 360 basic operation and maintenance platform from the developer's perspective.

PS: rich front-line technology, diversified forms of expression, as in the "360 cloud computing", point attention Oh!

API Gateway Selection

In October 2019, our team planned to revamp the gateway layer of 360 Basic Operations and Maintenance Platform. At that time, we mainly investigated several active gateways in the community, such as Kong, Orange, Apache APISIX, and finally chose Apache APISIX mainly because the storage selection etcd of Apache APISIX was more in line with our usage scenario.


Online operation

The number of APIs we have added to the gateway is close to 900, with an average daily PV of 10 million, and from the monitoring system, the gateway and our microservices are running well.

  • Average Daily PV


  • Gateway POD Monitoring


  • Microservice load monitoring diagram


Basic operation and maintenance platform architecture diagram

The following diagram is the final architecture of our O&M platform project, the gateway service we deployed on the company's container cloud, and the etcd service we deployed a set of clusters on 3 virtual machines.


Containerized Development and Deployment

Next I'll describe how we built the gateway service using Apache APISIX. First, I'll show you the code structure of our gateway project


When I showed Wang Yuansheng (one of the Apache APISIX PMCs) the code structure of our project, he was surprised and asked me how he didn't see the core code of Apache APISIX.

This is actually a path that we have explored when installing Apache APISIX using containers. The biggest benefit it gives us is that our business code is completely separate from the core Apache APISIX code, making it easy to upgrade Apache APISIX and to iterate on our business code.

I'll give you a step-by-step demonstration of how we build one of these environments. (This assumes that you all know about docker container technology)

  • Start the openresty container

    Our team repackaged a new image based on the official openresty image with some Apache APISIX dependencies such as luarocks installed by default, details can be found at

    docker run -itd -p 9080:9080 -p9443:9443 --name myapisix hulklab/openresty:0.0.1

  • Enter the container

    docker exec -it myapisix bash

  • Install apisix version 0.9

    luarocks install apisix 0.9-0

    At the end of the installation you will see the following line of output, from which we can see that the apisix installation directory is "/usr/local/apisix":

    apisix 0.9-0 is now installed in /usr/local (license: Apache License 2.0)

  • Go to the Apache APISIX installation directory

    cd /usr/local/apisix

    When you go to the apisix installation directory, you will find only two directories inside conf logs

    ls -l drwxr-xr-x 3 root root 4096 Dec 18 11:52 confdrwxr-xr-x 2 root root 4096 Dec 18 11:37 logs

  • Start Apache APISIX (running Openresty)

    apisix start ps aux|grep openresty root 1 0.0 0.0 11316 4464 pts/0 Ss+ 11:24 0:00 nginx: master process /usr/bin/openresty -g daemon off;root 5040 0.0 0.0 87436 2588 ? Ss 11:37 0:00 nginx: master process openresty -p /usr/local/apisix -c /usr/local/apisix/conf/nginx.conf

Note: If apisix start fails, because Apache APISIX relies on etcd, you need to start etcd, please refer to the official documentation of etcd [1] for how to start etcd, after starting etcd, you need to modify "/usr/local/apisix/conf/config.yaml", e.g.

# config.yaml:69etcd:  host: ""   # etcd address
  • View nginx.conf

apisix runs successfully, but there is no code in the installation directory "/usr/local/apisix", so where is the apisix core code and dependencies?

From the started openresty process, we can see that there is an extra nginx.conf under the apisix/conf directory. This nginx.conf configuration file is initialized when the apisix start command is executed. Let's check the lua in nginx.conf Package reference path.

> cat /usr/local/apisix/conf/nginx.conf|grep lua_package_path
lua_package_path  "$prefix/deps/share/lua/5.1/?.lua;/usr/local/apisix/lua/?.lua;;/usr/local/apisix/deps/share/lua/5.1/apisix/lua/?.lua;/usr/local/apisix/deps/share/lua/5.1/?.lua;/usr/share/lua/5.1/apisix/lua/?.lua;/usr/local/share/lua/5.1/apisix/lua/?.lua;/root/.luarocks/share/lua/5.1/?.lua;/root/.luarocks/share/lua/5.1/?/init.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;./?.lua;/usr/local/lib/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua;";

Looking at the above lua_package_paths one by one, we find two useful pieces of information.

  1. Path to Apache APISIX core code: "/usr/local/share/lua/5.1/apisix/lua/".
  2. The lua files under the Apache APISIX installation path /usr/local/apisix/lua have the highest priority for loading.

So we made an attempt, imitating the plugin path of Apache APISIX, creating lua/apisix/plugins/my-plugin.lua in the /usr/local/apisix directory, and adding the plugin to the configuration file config.yaml, and found that It took effect.

  • Dockerfile

Post the Dockerfile of our project for your reference. In the end, our project has only two directories, conf and lua. conf stores our own config.yaml and nginx.conf configuration files, and lua stores our custom plug-ins and class libraries.

FROM hulklab/openresty:0.0.1

RUN luarocks install apisix 0.9-0; \
    luarocks install lua-resty-cookie; \
    luarocks install lua-resty-kafka; \
    luarocks install lua-resty-url

WORKDIR /usr/local/apisix

RUN rm -rf conf/*; \
    mkdir -p lua; \
    mkdir -p logs/archive; \
    install -d -m 777 /tmp/apisix_cores/

COPY conf conf
COPY lua lua
COPY logrotate /etc/logrotate.d

EXPOSE 9080 9443

ENTRYPOINT ["openresty", "-p", "/usr/local/apisix", "-c", "/usr/local/apisix/conf/nginx.conf", "-g", "daemon off;"]

Plug-in development

  1. Project plug-in introduction

As you can see in the code structure diagram above, there are two directories in the apisix directory of our project, libs and plugins. In libs we put some commonly used libraries, and plugins store our custom business plugins. We All businesses are developed using plug-in mechanisms. The picture below is the plug-in currently used in our project.


To explain a little, there are two entry domains for our project, one for openApi access, which uses Basic Auth for the authentication plugin, and one for web browser access, which uses web auth (cookie authentication) for the authentication plugin.

Corresponding to OpenResty's request processing flow, our plugin focuses on the access and log phases.

ip-restrictionaccess_by_luaip flow limiting, using apisix native plugin
basic-authaccess_by_luaRequesting user authentication for openApi, self-developed plugin
web-authaccess_by_luaRequesting user authentication for webApi, self-developed plugin
limit-rateaccess_by_luaUser level and user+request parameter level flow restriction for requests, self-developed plugins
proxy-rewriteaccess_by_lua,balancer_by_luaForwarding of requests, setting interface-level timeouts, self-developed plug-ins
loglog_by_luaRecord the request log to kafka, and then read it to es through logstash, self-developed plug-in
alarmlog_by_luaAlarm according to the response statusCode, self-developed plug-in
  1. Sample plug-in development

The next section describes how the Apache APISIX plugin was developed, using the basic-auth plugin as an example.

  • Defining Plugin Objects

    local plugin_name = "odin-basic-auth"
    local schema = {
        type = "object",
        properties = {
            enable = { type = "boolean", default = true, enum = { true, false } },
    local _M = {
        version = 0.1,
        priority = 2802,
        name = plugin_name,
        schema = schema,

The odin-basic-auth plug-in has only one parameter enable. The enable parameter indicates whether to use this plug-in. This is because the Apache APISIX plug-in can be bound to the service or route. If the plug-in is bound to the service, the route is not The plug-in is closed, so a parameter is needed to finely control the plug-in that a route does not use service binding. It is recommended that official plug-ins are equipped with this parameter.

  • Implementing a method for detecting plug-in parameters

    function _M.check_schema(conf)
        local ok, err = core.schema.check(schema, conf)
        if not ok then
            return false, err
        return true

The check_schema method is basically the same for every plugin.

  • Methods for implementing the corresponding phases of the plug-in

    function _M.access(conf, ctx)
        -- 0. Check the configuration file to see if enable is enabled
        if not conf.enable then
        -- 1. Get the username and password in basic_auth
        local headers = ngx.req.get_headers()
        if not headers.Authorization then
            return 401, { message = "authorization is required" }
        local username, password, err = extract_auth_header(headers.Authorization)
        if err then
            return 401, { message = err }
        -- 2. Check etcd to get the record corresponding to the username
        local res = authorizations_etcd:get(username)
        if res == nil then
            return 401, { message = "failed to find authorization from etcd" }
        -- 3. If not, report authentication failure
        if not res.value or not then
            return 401, { message = "user is not found" }
        local value = res.value
        -- 4. If so, determine if the user password is correct
        if value.password ~= password then
            return 401, { message = "password is error" }
  1. Etcd cache objects

In the second step of the above example, we use authorizations_etcd:get(username) to get the actual password of the current requesting user and the list of authorized routes, using the etcd cache object of Apache APISIX.

The principle of etcd caching objects is to use etcd's watch function to cache data from etcd to memory objects, which are directly read from the memory during business use to avoid network io consumption. etcd's watch function also guarantees the real-time data. This feature of Apache APISIX is simply amazing.

The following describes how to use.

  • Define an etcd environment object variable

    local authorizations_etcd
    -- Define the scheme of values stored in the etcd object
    local appkey_scheme = {
        type = "object",
        properties = {
            username = {
                description = "username",
                type = "string",
            password = {
                type = "string",
  • Instantiated in the init phase of the plugin

    function _M.init()
        authorizations_etcd, err ="/authorizations", {
            automatic = true,
            item_schema = appkey_scheme
        if not authorizations_etcd then
            error("failed to create etcd instance for fetching authorizations: " .. err)

The init method of the plug-in occurs in the init_worker_by_lua phase of OpenResty, in other words, each worker is initialized only once. If the automatic parameter is set to true, Apache APISIX will enable the watch function. The business layer only needs to instantiate the etcd cache object, and Apache APISIX does the rest.

  • Using etcd cache objects in plugins

     local res = authorizations_etcd:get(username)
  1. Use of Plugin API

The essence of the etcd cache object above still requires fetching data from etcd, so how is the user-related data used in this plugin added to etcd? This brings us to another screaming feature of the plugin: the API feature.

  • Define API

    function _M.API()
        return {
                methods = { "POST", "PUT" },
                uri = "/apisix/plugin/basic-auth/set",
                handler = set_auth,
  • Implement the handler for the API

    local function set_auth()
        local username = req.get_str("username")
        local password = req.get_str("password")
        local key = "/authorizations/" .. username
        -- This is deposited to etcd
        local res, err = core.etcd.set(key, { username = username, password = password})
        if not res then
            core.response.exit(500, err)
        core.response.exit(res.status, res.body)
  • Calling the interface

    > curl -i -X PUT '' -d username=zhangsan -d password=hao123 -d user_id=3 -d action_ids=,1,2,3,

Problems encountered after going online

crontab clear day-end

As our gateway was deployed in the container, after running for a while, the log file exceeded the default quota of 50G, we later installed cron and logrotate in the image by default, and then turned on cron in the container entrypoint to solve this problem.


Finally, a special thanks to the contributors of Apache APISIX, posting the Apache APISIX website[2] and the Apache APISIX Github address[3].


  1. etcd Documents:

  2. Apache APISIX website:

  3. Apache APISIX Github:

The above is the content of this sharing ~

If you have any suggestions, you can also leave them in our comment section for your reference and learning. api gatewayapi gatewayapi gateway

APISIX BasicsAPI SecurityPlugin