--- title: "Cloud Monitoring With Datadog in Apache APISIX - API7.ai" description: "As the complexity of IT products and consumer-facing application development increases, monitoring ..." url: "https://api7.ai/blog/apache-apisix-datadog-integration" --- [Blog](https://api7.ai/blog) [Products](https://api7.ai/category/products)[Technology](https://api7.ai/category/technology)[Ecosystem](https://api7.ai/category/ecosystem)[Case Study](https://api7.ai/category/usercase)[Team & Community](https://api7.ai/category/community) Search ⌘ + K [Blog](https://api7.ai/blog) Ecosystem # Cloud Monitoring With Datadog in Apache APISIX API7.ai November 12, 2021 [Ecosystem](https://api7.ai/category/ecosystem) ![cover picture](https://static.api7.ai/2022/10/03/633a432861dab.png?imageMogr2/format/webp) As the complexity of IT products and consumer-facing application development increases, monitoring becomes an integral part of any application delivery. Additionally, to meet the endless demand of rapid upgrade cycles while ensuring stability, streamlined performance and keeping a perfect balance between service level indicators (SLI) with Service-level objectives (SLO) and Service-level agreement (SLA) - effective monitoring is immensely important. As a Cloud API Management Product, [Apache APISIX](http://apisix.apache.org/) decouples observability concerns from the application, which gives the developers an advantage of building applications focusing just on the business logic while Apache APISIX will take care of observability for the platform of their choice. Apache APISIX recently released a new plugin: [APISIX-Datadog](http://apisix.apache.org/docs/apisix/next/plugins/datadog), to provide a deeper integration with Datadog. This article introduces the APISIX-Datadog Plugin and its capabilities. ## How APISIX-Datadog plugin works ![APISIX-Datadog plugin Architecture](https://static.api7.ai/2022/10/03/633a432844c7a.png?imageMogr2/format/webp) The APISIX-Datadog plugin pushes its custom metrics to the DogStatsD server, comes bundled with Datadog agent over the UDP connection. DogStatsD basically is an implementation of StatsD protocol. It collects the custom metrics for Apache APISIX agent, aggregates it into a single data point and sends it to the configured Datadog server. To learn more about DogStatsD, please visit [DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?tab=hostagent). When APISIX-Datadog is activated, Apache APISIX agent exports the following metrics to DogStatsD server for every request response cycle: | Metric Name | StatsD Type | Description | | ---------------- | ----------- | ----------------------------------------------------------------------------------------------------------- | | Request Counter | Counter | Number of requests received. | | Request Latency | Histogram | Time taken to process the request (in milliseconds). | | Upstream latency | Histogram | Time taken since proxying the request to the upstream server till a response is received (in milliseconds). | | APISIX Latency | Histogram | Time taken by APISIX agent to process the request (in milliseconds). | | Ingress Size | Timer | Request body size in bytes. | | Egress Size | Timer | Response body size in bytes. | The metrics will be sent to the DogStatsD agent with the following tags. If there is no suitable value for any particular tag, the tag will simply be omitted. | Metric Name | Description | | ---------------- | ------------------------------------------------------------------------------------------------------- | | route\_name | Name specified in the route schema definition. If not present, it will fall back to the route id value. | | service\_id | If a route has been created with the abstraction of service, the particular service id will be used. | | consumer | If the route has a linked consumer, the consumer Username will be added as a tag. | | balancer\_ip | IP of the Upstream balancer that has processed the current request. | | response\_status | HTTP response status code. | | scheme | Scheme that has been used to make requests, such as HTTP, gRPC, gRPCs etc. | The plugin maintains a buffer with a timer. When the timer expires, APISIX-Datadog plugin flashes the buffered metrics as a batch to the locally run dogstatsd server. This approach is less resource-hungry (though it might be insignificant as UDP sockets are very lightweight) by reusing the same UDP socket and doesn't overload the network all the time as the timer can be configured. ## Steps to Run Datadog Agent 1. If you are already using Datadog inside your infrastructure, you must have a datadog agent installed in your systems. It may either be a docker container, pod or binary for a respective package manager. In this case, you are good to go. Just make sure port 8125/udp is allowed through the firewall (if any) i.e more specifically, the Apache APISIX agent can reach port 8125 of the datadog agent. You may skip this subsection. > To learn more about how to install a full-fledged datadog agent, visit [here](https://docs.datadoghq.com/agent/). 2. If you are new to Datadog 1. First create an account by visiting [www.datadoghq.com](http://www.datadoghq.com/). 2. Generate an API Key. ![Generate an API Key](https://static.api7.ai/2022/10/03/633a43285b28e.png?imageMogr2/format/webp) 3. APISIX-Datadog plugin requires only the dogstatsd component of `datadog/agent` as the plugin asynchronously send metrics to the dogstatsd server following the statsd protocol over standard UDP socket. That's why APISIX recommends using the standalone `datadog/dogstatsd` image instead of using the full agent. It's extremely lightweight (only \~11 MB in size) compared to \~2.8GB of `datadog/agent` image. To run it as a container: ``` # pull the latest image docker pull datadog/dogstatsd:latest # run a detached container docker run -d --name dogstatsd-agent -e DD_API_KEY= -p 8125:8125/udp datadog/dogstatsd ``` If you are using Kubernetes in your production environment, you can deploy `dogstatsd` as a `Daemonset` or as a `Multi-Container Pod` alongside Apache APISIX agent. ## How to Use Datadog with Apache APISIX ### Activate the APISIX-Datadog plugin The following is an example on how to activate the datadog plugin for a specific route. We are assuming your `dogstatsd` agent is already up an running. ``` # enable plugin for a specific route curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d ' { "plugins": { "datadog": {} }, "upstream": { "type": "roundrobin", "nodes": { "127.0.0.1:1980": 1 } }, "uri": "/hello" }' ``` Now any requests to endpoint uri `/hello` will generate the above metrics and push it to local DogStatsD server of the datadog agent. ### Custom Configuration In default configuration, the plugin expects the dogstatsd service to be available at `127.0.0.1:8125`. If you wish to update the config, please update the plugin metadata: #### Metadata Schema | Name | Type | Required | Default | Description | | -------------- | ------- | -------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------- | | hosts | string | optional | "127.0.0.1" | The DogStatsD server host address | | port | integer | optional | 8125 | The DogStatsD server host port | | namespace | string | optional | "apisix" | Prefix for all the custom metrics sent by APISIX agent. Useful for finding entities for metric graph. e.g. (apisix.request.counter) | | constant\_tags | array | optional | \["source"] | Static tags embedded into generated metrics. Useful for grouping metric over certain signals. | To know more about how to effectively write tags, please visit [here](https://docs.datadoghq.com/getting_started/tagging/#defining-tags) Make a request to /apisix/admin/plugin\_metadata endpoint with the updated metadata as following: ``` curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/datadog -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d ' { "host": "127.0.0.1", "port": 8125, "constant_tags": [ "source:apisix", "service:custom" ], "namespace": "apisix" }' ``` #### Plugin Schema Similarly, there are few attributes that can be tweaked while enabling the plugin. | Name | Type | Required | Default | Valid | Description | | ----------------- | ------- | -------- | ------- | -------- | ---------------------------------------------------------------------------------------- | | batch\_max\_size | integer | optional | 5000 | \[1,...] | Max buffer size of each batch | | inactive\_timeout | integer | optional | 5 | \[1,...] | Maximum age in seconds when the buffer will be flushed if inactive | | buffer\_duration | integer | optional | 60 | \[1,...] | Maximum age in seconds of the oldest entry in a batch before the batch must be processed | | max\_retry\_count | integer | optional | 1 | \[1,...] | Maximum number of retries if one entry fails to reach dogstatsd server | As all the fields are optional and if no attributes are set, the datadog plugin gets instantiated with the default values. To update any attribute, just update the required route, service or consumer with the updated attribute value. For example, the code below modifies the maximum buffer size of each batch: ``` '{ ... "plugins": { "datadog": { "batch_max_size": 10 } } ... }' ``` ### Deactivate the APISIX-Datadog plugin Now, to deactivate the plugin, simply remove the corresponding json configuration in the plugin configuration to disable the `datadog`. APISIX plugins are hot-reloaded, therefore no need to restart APISIX. ``` # disable plugin for a route curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d ' { "uri": "/hello", "plugins": {}, "upstream": { "type": "roundrobin", "nodes": { "127.0.0.1:1980": 1 } } }' ``` Tags: [APISIX Basics](https://api7.ai/tags/apisix-basics) [Datadog](https://api7.ai/tags/datadog) [Plugin](https://api7.ai/tags/plugin) Share article link Share Related Posts [![Forward-Auth, Another Choice for Authentication Function](https://static.api7.ai/2022/11/18/63772e00affaf.png)](https://api7.ai/blog/forward-auth-another-choice-for-authentication-function) ## [Forward-Auth, Another Choice for Authentication Function](https://api7.ai/blog/forward-auth-another-choice-for-authentication-function) [![HashiCorp Vault Secure Storage Backend in Apache APISIX Ecosystem](https://static.api7.ai/apisix-webp/2022/blog/0818/plugins/vault.webp)](https://api7.ai/blog/hashicorp-vault-secure-storage-backend-in-apache-apisix-ecosystem) ## [HashiCorp Vault Secure Storage Backend in Apache APISIX Ecosystem](https://api7.ai/blog/hashicorp-vault-secure-storage-backend-in-apache-apisix-ecosystem) Table of Contents - [How APISIX-Datadog plugin works](#how-apisix-datadog-plugin-works) - [Steps to Run Datadog Agent](#steps-to-run-datadog-agent) - [How to Use Datadog with Apache APISIX](#how-to-use-datadog-with-apache-apisix) - [Activate the APISIX-Datadog plugin](#activate-the-apisix-datadog-plugin) - [Custom Configuration](#custom-configuration) - [Deactivate the APISIX-Datadog plugin](#deactivate-the-apisix-datadog-plugin) ## Related Posts [![Why Do You Need Apache APISIX When Using Amazon Lambda](https://static.api7.ai/uploads/2024/12/25/PGShc0sm_apisix-aws-lambda.png)](https://api7.ai/blog/why-do-you-need-apache-apisix-when-using-amazon-lambda) Ecosystem December 9, 2022 #### [Why Do You Need Apache APISIX When Using Amazon Lambda](https://api7.ai/blog/why-do-you-need-apache-apisix-when-using-amazon-lambda) Keeping pace with cloud-natice trends, APISIX supports serverless computing by providing serverless, azure-functions, openwhisk, and openfunction plugins. [APISIX Basics](https://api7.ai/tags/apisix-basics) [Amazon Lambda](https://api7.ai/tags/amazon-lambda) [Plugin](https://api7.ai/tags/plugin) [![How to Build an Apache APISIX Plugin From 0 to 1?](https://static.api7.ai/2022/11/18/63772ca27585f.png)](https://api7.ai/blog/how-to-build-an-apache-apisix-plugin-from-0-to-1) Ecosystem February 16, 2022 #### [How to Build an Apache APISIX Plugin From 0 to 1?](https://api7.ai/blog/how-to-build-an-apache-apisix-plugin-from-0-to-1) Over the past few months, community users have added many plugins to ... [APISIX Basics](https://api7.ai/tags/apisix-basics) [Plugin](https://api7.ai/tags/plugin) [![Send Apache APISIX Logs to Splunk with HEC](https://static.api7.ai/2022/11/18/6377314b77522.png)](https://api7.ai/blog/integrating-splunk-http-event-collector-with-apache-apisix) Ecosystem February 10, 2022 #### [Send Apache APISIX Logs to Splunk with HEC](https://api7.ai/blog/integrating-splunk-http-event-collector-with-apache-apisix) Configure the Apache APISIX splunk-hec-logging plugin, send gateway access logs to Splunk HTTP Event Collector, and verify delivery safely. [APISIX Basics](https://api7.ai/tags/apisix-basics) [Plugin](https://api7.ai/tags/plugin)