Integrating Splunk HTTP Event Collector With Apache APISIX

The complexity of systems is increasing as technology iterates and enterprise architecture evolves. Logs can support and be compatible with different analysis engines to reduce the cost for users in the selection, operation and maintenance process. Log-based analysis and observation plays a very important role as the cornerstone to ensure system stability.

Apache APISIX is not only an API Gateway with exceptional performance, but also has supported most of the mainstream open source and commercial logging solutions through the communication with community users on data and logging operation and maintenance, including: HTTP Logger, TCP Logger, Kafka Logger, UDP Logger, RocketMQ Logger, SkyWalking Logger, Aliyun Cloud Logging(SLS), Google Cloud Logging, etc.

We now have a new addition to the Apache APISIX Logger support matrix: Splunk HEC Logging.

This article explains how to configure and use the Splunk HEC service in Apache APISIX.

About Splunk HTTP Event Collector

Splunk is a full-text search engine for machine data that can be used to collect, index, search, and analyze data from a variety of applications. According to DB Engines' search engine ranking, Splunk is currently in second place and is a widely used full-text search software. Splunk, like ElasticSearch, is a quasi-real-time data stream that provides uninterrupted search results.

Splunk HTTP Event Collector (HEC)is an HTTP event collector provided by Splunk that provides the ability to send data and application events to Splunk using the HTTP(S) protocol.

About splunk-hec-logging lugin

The splunk-hec-logging is used to forward Apache APISIX request logs to Splunk for analysis and storage. When enabled, Apache APISIX will take the request context information during the Log phase, serialize it into Splunk Event Data Format and submit it to the batch queue. The data in the queue is committed to Splunk HEC when the maximum processing capacity of the batch queue per batch is triggered, or when the maximum time to refresh the buffer is reached.

How to use the splunk-hec-logging plugin

Splunk Configuration

Deploy Splunk Enterprise

Please refer to Splunk's installation guide for deployment. This article will demonstrate deployment via Docker.

Docker command parameters are as follows.

docker run -p 18088:8088 -p 18000:8000 \    # 8088 is the HEC port, 8000 is the management backend port
  -e "SPLUNK_PASSWORD=your-password" \      # Admin Login Password
  -e "SPLUNK_START_ARGS=--accept-license" \ # Accept the license terms (Splunk will provide an Enterprise Trial License by default)
  -e "SPLUNK_HEC_TOKEN=your-hec-token" \    # Set the default HEC token, this will create a default HEC after configuration
  -itd --rm --name splunk-example splunk/splunk:latest

The command parameters are explained in the Docker Splunk Documentation.

Configure Splunk HEC

The default HEC is already configured and created in Docker, so we won't go into the process of creating it here. For details on the manual creation process, please refer to the documentation: Set up and use HTTP Event Collector in Splunk Web.

Access the mapped port of Docker through the browser. Since you need to map the 8000 port of the management backend to the 18000 port of the host, you can access it from the browser by "loopback address plus port" on the host during operation. For example: http://127.0.0.1:18000, the default username for login is admin, and the password is the SPLUNK_PASSWORD value set in the environment variable in the above example.

As shown in the figure below, it means the login is successful.

Splunk UI

Click on "Settings > Data Inputs" at the top right of the screen to check if the default HEC is set successfully.

Default HEC

We can already see the number of HECs in the Inputs column of the HTTP Event Collector, indicating successful setup.

the number of HECs

At this point, you can click HTTP Event Collector to enter the HEC details list to view the Token information of HECs.

HECs Token information

Token Values is the value of SPLUNK_HEC_TOKEN configured in the Docker environment variable above.

Apache APISIX Configuration

Enable the splunk-hec-logging plugin

Run the following command to enable the splunk-hec-logging plugin.

curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "plugins":{
        "splunk-hec-logging":{
            "endpoint":{
                // HEC endpoint
                "uri":"http://127.0.0.1:18088/services/collector",
                // HEC Token
                "token":"BD274822-96AA-4DA6-90EC-18940FB2414C"
            },
            // // Maximum time (in seconds) to refresh the batch queue buffer
            "inactive_timeout":2,
             // Maximum number of log entries per batch queue
            "batch_max_size":10
        }
    },
    "upstream":{
        "type":"roundrobin",
        "nodes":{
            "127.0.0.1:1980":1
        }
    },
    "uri":"/splunk.do"
}'

The plug-in parameters are described in the following table.

Name	Required	Default Value	Description
endpoint	Yes	N/A	Splunk HEC Endpoint Configuration Information
endpoint.uri	Yes	N/A	Splunk HEC Event Collection API
endpoint.token	Yes	N/A	Splunk HEC Identity Token
endpoint.channel	No	N/A	Splunk HEC send channel identification, refer to: About HTTP Event Collector Indexer Acknowledgment
endpoint.timeout	No	10	Splunk HEC data submission timeout in seconds.
ssl_verify	No	TRUE	Enable SSL authentication, refer to: OpenResty Documentation.
max_retry_count	No	0	Maximum number of retries before removal from the processing pipeline.
retry_delay	No	1	Number of seconds that process execution should be delayed if execution fails.
buffer_duration	No	60	The maximum duration (in seconds) of the oldest entry in the batch must be processed first.
inactive_timeout	No	5	Maximum time to refresh the buffer in seconds.
batch_max_size	No	1000	Maximum number of entries per batch queue.

Send the request

Run the following command to send a request to Splunk.

$ curl -i http://127.0.0.1:9080/splink.do
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Date: Fri, 10 Dec 2021 09:57:52 GMT
Server: APISIX/2.11.0

Hello, Splunk HEC Logging

Verify the log

Splunk console

Type source="apache-apisix-splunk-hec-logging" in the search box to query the sent request logs.

query the logs

Disable the splunk-hec-logging plugin

Remove the splunk-hec-logging configuration to disable the plugin.

curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri":"/logging.do",
    "upstream":{
        "type":"roundrobin",
        "nodes":{
            "127.0.0.1:1980":1
        }
    },
    "plugins":{
    }
}'

Summary

Apache APISIX is also currently working on additional plugins to support integration with more services, so if you're interested, feel free to start a discussion thread in our GitHub Discussion or communicate via the mailing list.