Smooth Canary Release Using APISIX Ingress Controller with Flagger
Hengliang Tan
January 19, 2023
Author: Hengliang Tan, Engineering Manager at XPENG
In the process of project development, service updates are often a challenge. To provide the best user experience, we need to avoid the risk of service unavailability as much as possible. Thus, continuous delivery was born, accepted as an enterprise software practice, and a natural evolution of well-established continuous integration principles. However, continuous deployment is still very rare due to the complexity of management and the fear that deployment failures will affect system availability. Canary release is probably the most classic scenario in the continuous delivery system. Based on this, we can quickly discover unhealthy and problematic services and roll back to the previous version effortlessly.
Canary Release
Canary release is also known as grayscale release. Generally speaking, the new version of the application is released and deployed as a "canary" to test the performance. The old version remains for normal operations at the same stage. During the upgrade, some users will be directed to use the new version, while other users will continue to use the old version. On the premise of ensuring the overall system's stability, it enables early detection of bugs and timely adjustment.
The canary release does not directly release the update. It slowly guides a certain percentage of traffic to a small number of users. If there are no errors detected, it will be promoted to all users, and the old version will be phased out. This method reduces the risk of introducing new functions into the production environment.
This article will introduce how to achieve smooth canary release through Apache APISIX Ingress and Flagger, improve release efficiency, and reduce release risks.
About Apache APISIX Ingress
Apache APISIX Ingress is realized by the Kubernetes Ingress Controller that uses Apache APISIX as the data plane proxy. It provides hundreds of functions, such as load balancing, dynamic upstream, canary release, fine-grained routing, rate-limiting, service degradation, service circuit breaker, authentication, and observability. It has been adopted by domestic and foreign companies and organizations, including Zoom, Tencent Cloud, Jiakaobaodian, Horizon Robotics, European Copernicus Reference System, etc.
About Flagger
Flagger is a CNCF (Cloud Native Computing Foundation) project and part of the Flux family of GitOps tools. Recently, the CNCF also announced the official graduation of Flux, which is a good indicator of the success and promising future of cloud-native technology. As a progressive delivery tool, Flagger automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring analytics metrics and running conformance tests.
After continuous efforts of the Apache APISIX and Flux communities, Flagger recently released v1.27.0, which supports automated canary releases using Apache APISIX Ingress and Flagger.
Let's experience this smooth canary release process together.
Environment
Requires a v1.19 or newer Kubernetes cluster, which you can install via kind.
Install Components
Use Helm V3 to install Apache APISIX and Apache APISIX Ingress Controller
helm repo add apisix https://charts.apiseven.com kubectl create ns apisix helm upgrade -i apisix apisix/apisix --version=0.11.3 \ --namespace apisix \ --set apisix.podAnnotations."prometheus\.io/scrape"=true \ --set apisix.podAnnotations."prometheus\.io/port"=9091 \ --set apisix.podAnnotations."prometheus\.io/path"=/apisix/prometheus/metrics \ --set pluginAttrs.prometheus.export_addr.ip=0.0.0.0 \ --set pluginAttrs.prometheus.export_addr.port=9091 \ --set pluginAttrs.prometheus.export_uri=/apisix/prometheus/metrics \ --set pluginAttrs.prometheus.metric_prefix=apisix_ \ --set ingress-controller.enabled=true \ --set ingress-controller.config.apisix.serviceNamespace=apisix
Install the Flagger and Prometheus components in the apisix namespace.
helm repo add flagger https://flagger.app helm upgrade -i flagger flagger/flagger \ --namespace apisix \ --set prometheus.install=true \ --set meshProvider=apisix
Note: if you need to customize Prometheus or Prometheus Operator, you can search related articles for modification.
Application Initialization
Flagger can be applied to Kubernetes deployment and other workloads and can also be combined with HPA. It will create a series of objects: Kubernetes deployments, ClusterIP services, and ApisixRoute. These objects can expose applications to outside clusters to provide services and are used for the analysis of the canary release process.
Create a new test namespace:
kubectl create ns test
Create a new deployment and HPA. Here we extract the official code sample from Flagger.
kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main
Deploy Flagger's load testing service to generate traffic during canary release for analysis.
helm upgrade -i flagger-loadtester flagger/loadtester \ --namespace=test
Create the ApisixRoute of Apache APISIX, and then Flagger will reference the created resource and generate the ApisixRoute of Apache APISIX Ingress in the canary version. (Replace app.example.com in the below example with your actual domain name)
apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: name: podinfo namespace: test spec: http: - backends: - serviceName: podinfo servicePort: 80 match: hosts: - app.example.com methods: - GET paths: - /* name: method plugins: - name: prometheus enable: true config: disable: false prefer_name: true
Save it as podinfo-apisixroute.yaml and submit it to the cluster:
kubectl apply -f ./podinfo-apisixroute.yaml
Create a Flagger custom resource Canary. (Replace app.example.com in the example with your actual domain name)
apiVersion: flagger.app/v1beta1 kind: Canary metadata: name: podinfo namespace: test spec: provider: apisix targetRef: apiVersion: apps/v1 kind: Deployment name: podinfo # Refer apisix route routeRef: apiVersion: apisix.apache.org/v2 kind: ApisixRoute name: podinfo progressDeadlineSeconds: 60 service: port: 80 targetPort: 9898 analysis: interval: 10s # maximum number of failures for roll back threshold: 10 # maximum percentage of traffic to the canary version # (0-100) maxWeight: 50 # the step size of the canary analysis # (0-100) stepWeight: 10 # use Prometheus to check the traffic information of APISIX metrics: - name: request-success-rate # the minimum success rate (none 5xx responses) # (0-100) thresholdRange: min: 99 interval: 1m - name: request-duration # P99 is the largest request delay(ms) thresholdRange: max: 500 interval: 30s webhooks: # automated traffic for canary analysis, modified based on the actual scenario - name: load-test url: http://flagger-loadtester.test/ timeout: 5s type: rollout metadata: cmd: |- hey -z 1m -q 10 -c 2 -h2 -host app.example.com http://apisix-gateway.apisix/api/info
Save it as podinfo-canary.yaml and submit it to the cluster:
kubectl apply -f ./podinfo-canary.yaml
Flagger will automatically generate related resources:
# Submitted deployment.apps/podinfo horizontalpodautoscaler.autoscaling/podinfo apisixroute/podinfo canary.flagger.app/podinfo
# Auto-generated deployment.apps/podinfo-primary horizontalpodautoscaler.autoscaling/podinfo-primary service/podinfo service/podinfo-canary service/podinfo-primary apisixroute/podinfo-podinfo-canary
At this point, you can access the application through the domain name app.example.com (Replace app.example.com in the example with your actual domain name), and you will see the current version of the application.
Automation of Canary Release
Flagger implements a control loop that gradually shifts traffic to canary nodes while measuring key performance metrics such as HTTP request success rate, average request duration, and pod health. According to the analysis of relevant indicators, release or stop the canary deployment and publish the analysis results to relevant platforms such as Slack, MS Teams or Prometheus Alert Manager, etc.
Trigger a canary release by updating the container image version
kubectl -n test set image deployment/podinfo \ podinfod=stefanprodan/podinfo:6.0.1
Flagger detects that there is a new version of the deployment and will start a trial run of the canary analysis release.
kubectl -n test describe canary/podinfo Status: Canary Weight: 0 Conditions: Message: Canary analysis completed successfully, promotion finished. Reason: Succeeded Status: True Type: Promoted Failed Checks: 1 Iterations: 0 Phase: Succeeded Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Synced 2m59s flagger podinfo-primary.test not ready: waiting for rollout to finish: observed deployment generation less than desired generation Warning Synced 2m50s flagger podinfo-primary.test not ready: waiting for rollout to finish: 0 of 1 (readyThreshold 100%) updated replicas are available Normal Synced 2m40s (x3 over 2m59s) flagger all the metrics providers are available! Normal Synced 2m39s flagger Initialization done! podinfo.test Normal Synced 2m20s flagger New revision detected! Scaling up podinfo.test Warning Synced 2m (x2 over 2m10s) flagger canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 1 (readyThreshold 100%) updated replicas are available Normal Synced 110s flagger Starting canary analysis for podinfo.test Normal Synced 109s flagger Advance podinfo.test canary weight 10 Warning Synced 100s flagger Halt advancement no values found for apisix metric request-success-rate probably podinfo.test is not receiving traffic: running query failed: no values found Normal Synced 90s flagger Advance podinfo.test canary weight 20 Normal Synced 80s flagger Advance podinfo.test canary weight 30 Normal Synced 69s flagger Advance podinfo.test canary weight 40 Normal Synced 59s flagger Advance podinfo.test canary weight 50 Warning Synced 30s (x2 over 40s) flagger podinfo-primary.test not ready: waiting for rollout to finish: 1 old replicas are pending termination Normal Synced 9s (x3 over 50s) flagger (combined from similar events): Promotion completed! Scaling down podinfo.test
During the canary release process, you will receive different responses when you access the application through the domain name app.example.com (Replace app.example.com with your actual domain name).
By viewing the ApisixRoute resource podinfo-podinfo-canary of Apache APISIX created automatically by Flagger, you will find that the weights of service podinfo-primary and service podinfo-canary change along with the publishing process.
spec: http: - backends: - serviceName: podinfo-primary servicePort: 80 # Auto-adjusted by Flagger weight: 80 - serviceName: podinfo-canary servicePort: 80 # Auto-adjusted by Flagger weight: 20
You will see the latest stable version when the final release is complete.
Note: Flagger will re-run the canary analysis if you change the deployment again during the canary release.
You can observe all canary releases with this command:
watch kubectl get canaries --all-namespaces NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME test podinfo-2 Progressing 10 2022-11-23T05:00:54Z test podinfo Succeeded 0 2022-11-23T06:00:54Z
Rollback
During canary release analysis, you can test Flagger to suspend the canary release and rollback to the old version by generating an HTTP 500 Bad Request.
Trigger another canary release:
kubectl -n test set image deployment/podinfo \ podinfod=stefanprodan/podinfo:6.0.2
Enter load tester container
kubectl -n test exec -it deploy/flagger-loadtester bash
Generate HTTP 500 error:
hey -z 1m -c 5 -q 5 -host app.example.com http://apisix-gateway.apisix/status/500
Simulate server delay:
watch -n 1 curl -H \"host: app.example.com\" http://apisix-gateway.apisix/delay/1
When the number of detected failures reaches the threshold of canary analysis, the traffic is automatically routed back to the master node, the canary node is scaled down to zero, and the canary release process is marked as failed.
kubectl -n apisix logs deploy/flagger -f | jq .msg "New revision detected! Scaling up podinfo.test" "canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 1 (readyThreshold 100%) updated replicas are available" "Starting canary analysis for podinfo.test" "Advance podinfo.test canary weight 10" "Halt podinfo.test advancement success rate 0.00% < 99%" "Halt podinfo.test advancement success rate 26.76% < 99%" "Halt podinfo.test advancement success rate 34.19% < 99%" "Halt podinfo.test advancement success rate 37.32% < 99%" "Halt podinfo.test advancement success rate 39.04% < 99%" "Halt podinfo.test advancement success rate 40.13% < 99%" "Halt podinfo.test advancement success rate 48.28% < 99%" "Halt podinfo.test advancement success rate 50.35% < 99%" "Halt podinfo.test advancement success rate 56.92% < 99%" "Halt podinfo.test advancement success rate 67.70% < 99%" "Rolling back podinfo.test failed checks threshold reached 10" "Canary failed! Scaling down podinfo.test"
Customize Metrics for Canary Analysis
Canary analysis can be extended by querying Prometheus metrics. We customize based on actual business scenarios. Create a metric template and submit it to the cluster.
apiVersion: flagger.app/v1beta1 kind: MetricTemplate metadata: name: not-found-percentage namespace: test spec: provider: type: prometheus address: http://flagger-prometheus.apisix:9090 query: | sum( rate( apisix_http_status{ route=~"{{ namespace }}_{{ route }}-{{ target }}-canary_.+", code!~"4.." }[{{ interval }}] ) ) / sum( rate( apisix_http_status{ route=~"{{ namespace }}_{{ route }}-{{ target }}-canary_.+" }[{{ interval }}] ) ) * 100 # Modify the analysis in the canary release and add the indicator template created above. analysis: metrics: - name: "404s percentage" templateRef: name: not-found-percentage thresholdRange: max: 5 interval: 1m
The configuration above validates the canary by checking if the QPS (Queries per second) of HTTP 404 requests is higher than 5% of the total traffic. The canary rollout fails if the HTTP 404 requests exceed the 5% threshold.
Summary
The above process can be extended with more custom metric checks, Webhook, manual approvals and Slack or MS Teams notifications.
A very smooth canary release is achieved through the integration of Apache APISIX and Flagger, which improves release efficiency and reduces release risks. In the future, the two communities will cooperate more closely to realize more publishing capabilities such as Blue/Green Mirroring and A/B Testing.