Using APISIX Ingress Controller With AWS ACM

Xin Rong

December 30, 2022

Ecosystem

Why do we need AWS ACM?

Problems caused by manual management

A certificate contains information such as the holder, issuer, and expiration date, while the expired certificate cannot be used and needs to be renewed before it expires. Therefore, the certificate information must be recorded, for example, manually recorded to a spreadsheet, to ensure that certificates can be revoked and renewed in time.

However, as there are more and more certificates, the manual management method cannot manage certificate correctly and efficiently. The spreadsheet can't send an automatic notification, nor will it automate certificate renewal when it expires, bringing unnecessary risks.

In order to avoid the risk of certificate interruption and keep automatic updates according to new information and regulations, automated certificate management comes up, reducing the certificate expiration risks and releasing developers' burdens.

ACM manages certificates automatically

AWS Certificate Manager (ACM) could easily provision, manage, deploy, and renew SSL/TLS certificates. You can directly issue certificates through ACM to protect your AWS websites and applications or import third-party certificates into the ACM management system.

With ACM, you no longer need to go through the extensive manual processes associated with using and managing SSL/TLS certificates like previously. You can also export ACM certificates signed by AWS Private CA to use in any location within your internal PKI. In addition, it is integrated with AWS Elastic Load Balancing (ELB).

acm

What issues does ACM Private CA resolve?

Self-signed certificates are used to deploy internal applications without any private CA within an organization. When these applications try to access each other, they may refuse each other's access because their certificates are not trusted. Blindly trusting unknown sources of applications could bring security risks. In this case, we need a hosted Private CA service to create and manage a CA hierarchy to ensure that all applications within the organization are trusted.

ACM Private CA is a highly available hosted service that creates and maintains an internal PKI for your organization, eliminating the cost of continuous maintenance. Private keys are stored in AWS-hosted Hardware Security Modules (HSM) validated by FIPS 140-2, which offers a more secure certificate-issuing organizations solution compared to the default CA in Kubernetes.

pca

How to use ACM with Kubernetes?

There are two different configurations to terminate TLS certificates in Kubernetes:

Configurations

  • Terminating TLS certificates in Ingress: When there is a need for encryption between applications, we need to terminate TLS in the Ingress controller. Each application can manage its certificate through Ingress without any interference from each other. This method is easier to configure and manage and ensures that communication between applications is trusted.
  • Terminating TLS certificates in NLB: Terminating TLS certificates at the NLB level is the most common use case for using publicly trusted certificates. This method is easy to configure and bind an ACM publicly trusted certificate to the NLB. Access to applications internal cluster still uses HTTP without any additional encryption and decryption operations.

In the following examples, you could configure APISIX Ingress directly on Amazon EKS. Using these two methods, we will demonstrate how APISIX Ingress works with ACM (and ACM Private CA).

Prerequisites

Before we start, we should meet the following requirements:

  • AWS An AWS account and the AWS command line interface (AWS CLI)
  • You must have permission to use the Amazon EKS IAM role and service-related roles AWS CloudFormation, Amazon Virtual Private Cloud (Amazon VPC) and related resources. Please check this article in the IAM user manual "Operations, resources, and condition keys for Amazon Elastic Container Service for Kubernetes" and use service related roles. In addition, this IAM security principal needs to have the AWSCertificateManagerPrivateCAFullAccess IAM managed policy attached.
  • Install and configure kubectl and eksctl tools.

Amazon EKS cluster

Amazon EKS is a hosted service which allows you to run Kubernetes on AWS and easily deploy and manage your Kubernetes clusters. This article will use eksctl command tool to manage clusters.

  • use eksctl default configuration to create clusters (Please ignore this part if you already have EKS cluster)
eksctl create cluster

Install APISIX Ingress

  1. Install APISIX Ingress in the EKS cluster, and set it to LoadBalancer type.
helm repo add apisix https://charts.apiseven.com
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install apisix apisix/apisix \
  --set gateway.type=LoadBalancer \
  --set gateway.tls.enabled=true \
  --set ingress-controller.enabled=true \
  --namespace ingress-apisix \
  --create-namespace

Note:

Please make sure your cluster can add persistent volumes; please see Amazon EKS Storage for more details. If you just want to try out this tutorial, configure --set etcd.persistence.enabled=false during installation to declare that persistent volumes will not be used.

  1. Execute the following commands to check the status, and make sure all pods are in Running status.
$ kubectl get pods -n ingress-apisix
NAME                                         READY   STATUS    RESTARTS   AGE
apisix-78bfc58588-qspmm                      1/1     Running   0          103s
apisix-etcd-0                                1/1     Running   0          103s
apisix-etcd-1                                1/1     Running   0          103s
apisix-etcd-2                                1/1     Running   0          103s
apisix-ingress-controller-6ff56cd4b4-rktr9   1/1     Running   0          103s
  1. Check NLB status; please pay attention to the PORT(S) and EXTERNAL-IP.
$ kubectl get svc apisix-gateway -n ingress-apisix
NAME             TYPE           CLUSTER-IP      EXTERNAL-IP                                                                    PORT(S)                      AGE
apisix-gateway   LoadBalancer   10.100.178.65   a6cffe9f6fc5c47b9929cb758610fc5a-2074689558.ap-northeast-1.elb.amazonaws.com   80:30851/TCP,443:32735/TCP   103s

Terminate TLS certificates in NLB

Prepare ACM certificate

  1. Open ACM console, and apply for a public ACM certificate for your custom domain or import a custom certificate.

acm

LoadBalancer configuration certificate

  1. Open EC2 console, and select your load balancers -> listener -> edit

nlb-1

  1. Set NLB protocol to HTTPS, port to 443, instance protocol to HTTP and port to 30851

nlb-2

  1. Attach the TLS certificate from ACM to the NLB.

nlb-3

Associate the custom domain with the load balancer name

You can use your DNS provider's console to direct your application's DNS record to the NLB's URL via CNAME. For example, Route53 and configure the CNAME records that direct to your NLB.

httpbin.example-test.org CNAME a6cffe9f6fc5c47b9929cb758610fc5a-2074689558.ap-northeast-1.elb.amazonaws.com

Access application's domain

curl https://httpbin.example-test.org

Terminate TLS certificate in Ingress

Before we start this example, ensure that the configuration of the AWS NLB is restored, as shown in the following image.

nls-0

Install cert-manager

Cert-manager is a Kubernetes add-on that can automatically manage and issue TLS certificates from various sources. You use a common way to install the cert-manager.

Create ACM Private CA

  1. Open ACM PCA console, choose "Create a private CA", and install.

pca-1

  1. After CA has been successfully created, and the status is active. The PCA's ARN will be frequently used in the later sections.

pca-2

Set up EKS node permissions for ACM Private CA

By default, there is no issuance permission. To issue a certificate from ACM Private CA, you must add an IAM policy to the EKS NodeInstanceRole, including the iamserviceaccount service role.

  1. Create pca-iam-policy.json file, and you need to replace ${PCA_ARN} with your own PCA_ARN.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "awspcaissuer",
            "Action": [
                "acm-pca:DescribeCertificateAuthority",
                "acm-pca:GetCertificate",
                "acm-pca:IssueCertificate"
            ],
            "Effect": "Allow",
            "Resource": "${PCA_ARN}"
        }
    ]
}
  1. Create IAM and iamserviceaccount based on pca-iam-policy.json, and need to replace ${account_id} with your AWS ID.
aws iam create-policy \
    --policy-name AWSPCAIssuerIAMPolicy \
    --policy-document file://pca-iam-policy.json
# create namespace
kubectl create namespace aws-pca-issuer
eksctl create iamserviceaccount \
    --cluster=${cluster_name} \
    --namespace=aws-pca-issuer \
    --name=aws-pca-issuer \
    --attach-policy-arn=arn:aws:iam::${account_id}:policy/AWSPCAIssuerIAMPolicy \
    --override-existing-serviceaccounts \
    --approve

Install aws-privateca-issuer

AWS PrivateCA Issuer serves as a cert-manager add-on (External Issuers to sign certificate requests.

  1. Use Helm to install
helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
helm repo update
helm install aws-pca-issuer awspca/aws-privateca-issuer \
    -n aws-pca-issuer \
    --set serviceAccount.create=false \
    --set serviceAccount.name=aws-pca-issuer
  1. Check the status
$ kubectl get pods -n aws-pca-issuer
NAME                                                   READY   STATUS    RESTARTS   AGE
aws-pca-issuer-aws-privateca-issuer-5cdd4b4687-z52n7   1/1     Running   0          20s

Create issuer and apply for certificate

  1. Replace ${PCA_ARN} in the issuer-cert.yaml file with your configuration and execute the following command:
  • kubectl apply -f issuer-cert.yaml
# issuer-cert.yaml
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
  name: demo-test-root-ca
spec:
  arn: ${PCA_ARN}
---

kind: Certificate
apiVersion: cert-manager.io/v1
metadata:
  name: nlb-lab-tls-cert
spec:
  commonName: httpbin.example-test.org # need to replace with your custom domain name
  dnsNames:
    - httpbin.example-test.org # need to replace with your custom domain name
  duration: 2160h0m0s
  issuerRef:
    group: awspca.cert-manager.io
    kind: AWSPCAClusterIssuer
    name: demo-test-root-ca
  renewBefore: 360h0m0s
  secretName: nlb-tls-app-secret
  usages:
    - server auth
    - client auth
  privateKey:
    algorithm: "RSA"
    size: 2048
  1. Run the following command to verify that the certificate has been issued and the secret has been generated.
$ kubectl get cert
NAME                READY   SECRET                AGE
nlb-lab-tls-cert    True    nlb-tls-app-secret    10s
$ kubectl get secret
NAME                  TYPE                                  DATA   AGE
nlb-tls-app-secret    kubernetes.io/tls                     3      8s

Publicize and protect httpbin application

Make sure APISIX Ingress has been successfully installed. Please check this section

  1. Deploy httpbin application
kubectl run httpbin --image kennethreitz/httpbin --port 80
kubectl expose pod httpbin --port 80
  1. Create Ingress to publicize and protect httpbin application
kubectl apply -f ingress-httpbin.yaml

```yaml
# ingress-httpbin.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin-demo-apisix
spec:
  ingressClassName: apisix
  tls:
  - hosts:
    - httpbin.example-test.org
    secretName: nlb-tls-app-secret
  rules:
  - host: httpbin.example-test.org
    http:
      paths:
      - backend:
          service:
            name: httpbin
            port:
              number: 80
        path: /
        pathType: Prefix
  1. Access application domain name

Make sure your custom domain name and load balancer's name have been associated. Please check this section

$ curl https://httpbin.example-test.org/headers --cacert acm-pca/cacert.pem
{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.example-test.org",
    "User-Agent": "curl/7.74.0",
    "X-Forwarded-Host": "httpbin.example-test.org"
  }
}

Conclusion

This article demonstrates the use of APISIX Ingress with AWS ACM and ACM Private CA components through practical examples and introduces two configurations for terminating TLS certificates in Kubernetes: Terminating TLS certificates in NLB and Ingress. The ACM + NLB configuration is more suitable for publicly trusted certificates. If there is a requirement for encryption between applications, ACM Private CA could provide a more secure hosted management service.

We hope those practice articles could help readers configure and manage TLS traffic in their AWS EKS clusters more effectively. For more information about API gateway, please visit blogs or Contact Us.

Tags:
APISIX Ingress ControllerAWSKubernetes