Application of API Gateway Apache APISIX in 360

Today, I share an article about Apache APISIX, which describes the implementation practice of Apache APISIX gateway in 360 basic operation and maintenance platform from the developer's perspective.

PS: rich front-line technology, diversified forms of expression, as in the "360 cloud computing", point attention Oh!

API Gateway Selection

In October 2019, our team planned to revamp the gateway layer of 360 Basic Operations and Maintenance Platform. At that time, we mainly investigated several active gateways in the community, such as Kong, Orange, Apache APISIX, and finally chose Apache APISIX mainly because the storage selection etcd of Apache APISIX was more in line with our usage scenario.

Online operation

The number of APIs we have added to the gateway is close to 900, with an average daily PV of 10 million, and from the monitoring system, the gateway and our microservices are running well.

• Average Daily PV

• Gateway POD Monitoring

• Microservice load monitoring diagram

Basic operation and maintenance platform architecture diagram

The following diagram is the final architecture of our O&M platform project, the gateway service we deployed on the company's container cloud, and the etcd service we deployed a set of clusters on 3 virtual machines.

Containerized Development and Deployment

Next I'll describe how we built the gateway service using Apache APISIX. First, I'll show you the code structure of our gateway project

When I showed Wang Yuansheng (one of the Apache APISIX PMCs) the code structure of our project, he was surprised and asked me how he didn't see the core code of Apache APISIX.

This is actually a path that we have explored when installing Apache APISIX using containers. The biggest benefit it gives us is that our business code is completely separate from the core Apache APISIX code, making it easy to upgrade Apache APISIX and to iterate on our business code.

I'll give you a step-by-step demonstration of how we build one of these environments. (This assumes that you all know about docker container technology)

• Start the openresty container

  Our team repackaged a new image based on the official openresty image with some Apache APISIX dependencies such as luarocks installed by default, details can be found at https://hub.docker.com/r/hulklab/openresty/dockerfile

  docker run -itd -p 9080:9080 -p9443:9443 --name myapisix hulklab/openresty:0.0.1

• Enter the container

  docker exec -it myapisix bash

• Install apisix version 0.9

  luarocks install apisix 0.9-0

  At the end of the installation you will see the following line of output, from which we can see that the apisix installation directory is "/usr/local/apisix":

  apisix 0.9-0 is now installed in /usr/local (license: Apache License 2.0)

• Go to the Apache APISIX installation directory

  cd /usr/local/apisix

  When you go to the apisix installation directory, you will find only two directories inside conf logs

  ls -l drwxr-xr-x 3 root root 4096 Dec 18 11:52 confdrwxr-xr-x 2 root root 4096 Dec 18 11:37 logs

• Start Apache APISIX (running Openresty)

  apisix start ps aux|grep openresty root 1 0.0 0.0 11316 4464 pts/0 Ss+ 11:24 0:00 nginx: master process /usr/bin/openresty -g daemon off;root 5040 0.0 0.0 87436 2588 ? Ss 11:37 0:00 nginx: master process openresty -p /usr/local/apisix -c /usr/local/apisix/conf/nginx.conf

Note: If apisix start fails, because Apache APISIX relies on etcd, you need to start etcd, please refer to the official documentation of etcd [1] for how to start etcd, after starting etcd, you need to modify "/usr/local/apisix/conf/config.yaml", e.g.

# config.yaml:69etcd:  host: "http://172.17.0.1:2379"   # etcd address

• View nginx.conf

apisix runs successfully, but there is no code in the installation directory "/usr/local/apisix", so where is the apisix core code and dependencies?

From the started openresty process, we can see that there is an extra nginx.conf under the apisix/conf directory. This nginx.conf configuration file is initialized when the apisix start command is executed. Let's check the lua in nginx.conf Package reference path.

> cat /usr/local/apisix/conf/nginx.conf|grep lua_package_path
lua_package_path  "$prefix/deps/share/lua/5.1/?.lua;/usr/local/apisix/lua/?.lua;;/usr/local/apisix/deps/share/lua/5.1/apisix/lua/?.lua;/usr/local/apisix/deps/share/lua/5.1/?.lua;/usr/share/lua/5.1/apisix/lua/?.lua;/usr/local/share/lua/5.1/apisix/lua/?.lua;/root/.luarocks/share/lua/5.1/?.lua;/root/.luarocks/share/lua/5.1/?/init.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;./?.lua;/usr/local/lib/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua;";

Looking at the above lua_package_paths one by one, we find two useful pieces of information.

1. Path to Apache APISIX core code: "/usr/local/share/lua/5.1/apisix/lua/".
2. The lua files under the Apache APISIX installation path /usr/local/apisix/lua have the highest priority for loading.

So we made an attempt, imitating the plugin path of Apache APISIX, creating lua/apisix/plugins/my-plugin.lua in the /usr/local/apisix directory, and adding the plugin to the configuration file config.yaml, and found that It took effect.

• Dockerfile

Post the Dockerfile of our project for your reference. In the end, our project has only two directories, conf and lua. conf stores our own config.yaml and nginx.conf configuration files, and lua stores our custom plug-ins and class libraries.

FROM hulklab/openresty:0.0.1

RUN luarocks install apisix 0.9-0; \
    luarocks install lua-resty-cookie; \
    luarocks install lua-resty-kafka; \
    luarocks install lua-resty-url

WORKDIR /usr/local/apisix

RUN rm -rf conf/*; \
    mkdir -p lua; \
    mkdir -p logs/archive; \
    install -d -m 777 /tmp/apisix_cores/

COPY conf conf
COPY lua lua
COPY logrotate /etc/logrotate.d

EXPOSE 9080 9443

ENTRYPOINT ["openresty", "-p", "/usr/local/apisix", "-c", "/usr/local/apisix/conf/nginx.conf", "-g", "daemon off;"]

Plug-in development

1. Project plug-in introduction

As you can see in the code structure diagram above, there are two directories in the apisix directory of our project, libs and plugins. In libs we put some commonly used libraries, and plugins store our custom business plugins. We All businesses are developed using plug-in mechanisms. The picture below is the plug-in currently used in our project.

To explain a little, there are two entry domains for our project, one for openApi access, which uses Basic Auth for the authentication plugin, and one for web browser access, which uses web auth (cookie authentication) for the authentication plugin.

Corresponding to OpenResty's request processing flow, our plugin focuses on the access and log phases.

2. Sample plug-in development

The next section describes how the Apache APISIX plugin was developed, using the basic-auth plugin as an example.

• Defining Plugin Objects

  local plugin_name = "odin-basic-auth"
  
  local schema = {
      type = "object",
      properties = {
          enable = { type = "boolean", default = true, enum = { true, false } },
      },
  }
  
  local _M = {
      version = 0.1,
      priority = 2802,
      name = plugin_name,
      schema = schema,
  }
  

The odin-basic-auth plug-in has only one parameter enable. The enable parameter indicates whether to use this plug-in. This is because the Apache APISIX plug-in can be bound to the service or route. If the plug-in is bound to the service, the route is not The plug-in is closed, so a parameter is needed to finely control the plug-in that a route does not use service binding. It is recommended that official plug-ins are equipped with this parameter.

• Implementing a method for detecting plug-in parameters

  function _M.check_schema(conf)
      local ok, err = core.schema.check(schema, conf)
  
      if not ok then
          return false, err
      end
  
      return true
  end
  

The check_schema method is basically the same for every plugin.

• Methods for implementing the corresponding phases of the plug-in

  function _M.access(conf, ctx)
  
      -- 0. Check the configuration file to see if enable is enabled
      if not conf.enable then
          return
      end
  
      -- 1. Get the username and password in basic_auth
      local headers = ngx.req.get_headers()
      if not headers.Authorization then
          return 401, { message = "authorization is required" }
      end
  
      local username, password, err = extract_auth_header(headers.Authorization)
      if err then
          return 401, { message = err }
      end
  
      -- 2. Check etcd to get the record corresponding to the username
      local res = authorizations_etcd:get(username)
      if res == nil then
          return 401, { message = "failed to find authorization from etcd" }
      end
  
      -- 3. If not, report authentication failure
      if not res.value or not res.value.id then
          return 401, { message = "user is not found" }
      end
  
      local value = res.value
  
      -- 4. If so, determine if the user password is correct
      if value.password ~= password then
          return 401, { message = "password is error" }
      end
  end
  

3. Etcd cache objects

In the second step of the above example, we use authorizations_etcd:get(username) to get the actual password of the current requesting user and the list of authorized routes, using the etcd cache object of Apache APISIX.

The principle of etcd caching objects is to use etcd's watch function to cache data from etcd to memory objects, which are directly read from the memory during business use to avoid network io consumption. etcd's watch function also guarantees the real-time data. This feature of Apache APISIX is simply amazing.

The following describes how to use.

• Define an etcd environment object variable

  local authorizations_etcd
  
  -- Define the scheme of values stored in the etcd object
  local appkey_scheme = {
      type = "object",
      properties = {
          username = {
              description = "username",
              type = "string",
          },
          password = {
              type = "string",
          }
      },
  }
  

• Instantiated in the init phase of the plugin

  function _M.init()
  
      authorizations_etcd, err = core.config.new("/authorizations", {
          automatic = true,
          item_schema = appkey_scheme
      })
  
      if not authorizations_etcd then
          error("failed to create etcd instance for fetching authorizations: " .. err)
          return
      end
  
  end
  

The init method of the plug-in occurs in the init_worker_by_lua phase of OpenResty, in other words, each worker is initialized only once. If the automatic parameter is set to true, Apache APISIX will enable the watch function. The business layer only needs to instantiate the etcd cache object, and Apache APISIX does the rest.

• Using etcd cache objects in plugins

   local res = authorizations_etcd:get(username)
  

4. Use of Plugin API

The essence of the etcd cache object above still requires fetching data from etcd, so how is the user-related data used in this plugin added to etcd? This brings us to another screaming feature of the plugin: the API feature.

• Define API

  function _M.API()
      return {
          {
              methods = { "POST", "PUT" },
              uri = "/apisix/plugin/basic-auth/set",
              handler = set_auth,
          }
      }
  end
  

• Implement the handler for the API

  local function set_auth()
      local username = req.get_str("username")
      local password = req.get_str("password")
  
      local key = "/authorizations/" .. username
  
      -- This is deposited to etcd
      local res, err = core.etcd.set(key, { username = username, password = password})
      if not res then
          core.response.exit(500, err)
      end
  
      core.response.exit(res.status, res.body)
  end
  

• Calling the interface

  > curl -i -X PUT 'http://127.0.0.1:9080/apisix/plugin/basic-auth/set' -d username=zhangsan -d password=hao123 -d user_id=3 -d action_ids=,1,2,3,
  

Problems encountered after going online

crontab clear day-end

As our gateway was deployed in the container, after running for a while, the log file exceeded the default quota of 50G, we later installed cron and logrotate in the image by default, and then turned on cron in the container entrypoint to solve this problem.

Thanks

Finally, a special thanks to the contributors of Apache APISIX, posting the Apache APISIX website[2] and the Apache APISIX Github address[3].

References

1. etcd Documents: https://doczhcn.gitbook.io/etcd/index

2. Apache APISIX website: https://apisix.apache.org/

3. Apache APISIX Github: https://github.com/apache/apisix

The above is the content of this sharing ~

If you have any suggestions, you can also leave them in our comment section for your reference and learning. api gatewayapi gatewayapi gateway