Canary Deployment with Istio on EKS
Canary deployment is a way of deploying the application in a phased manner. In this pattern, we deploy a new version of the application alongside the production version, then rollout the change to a small subset of servers. Once new version of the application is tested by the real users, then rollout the change out to the rest of the servers.
Canary deployments can be complex and involve testing in production and manual verification.
To demonstrate Canary deployments, we will setup an EKS cluster, install Istio, deploy sample application and setup canary release of new version of application.
you can follow the steps below or use the script here.
Setup EKS Cluster
Setup an EKS cluster of version
us-east-1 region with a managed node group
default-pool of machine type
Download and install the latest version of
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp sudo mv /tmp/eksctl /usr/local/bin
for Mac and other operating systems, follow the steps here.
- Create EKS cluster
eksctl create cluster --name canary-cluster \ --version 1.21 \ --region us-east-1 \ --nodegroup-name default-pool \ --node-type t3a.medium \ --nodes 3 \ --nodes-min 0 \ --nodes-max 4 \ --managed
Once the Control plane is ready, you should see the output like this
2021-08-13 23:59:52 [ℹ] waiting for the control plane availability... 2021-08-13 23:59:52 [✔] saved kubeconfig as "/Users/shardulsrivastava/.kube/config" 2021-08-13 23:59:52 [ℹ] no tasks 2021-08-13 23:59:52 [✔] all EKS cluster resources for "canary-cluster" have been created 2021-08-14 00:01:57 [ℹ] kubectl command should work with "/Users/shardulsrivastava/.kube/config", try 'kubectl get nodes' 2021-08-14 00:01:57 [✔] EKS cluster "canary-cluster" in "us-east-1" region is ready
Note: You would need a minimum of these permissions to run the eksctl commands above.
Download and install
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
Check the pods running in cluster:
kubectl get pods -A
At this point, your cluster would have only the
Istio provides a convenient binary
istioctl to set up and interact with Istio components.
istioctland if you’re running these commands from an ec2 instance, add the installation directory to system path:
curl -L https://istio.io/downloadIstio | sh - export PATH="$PATH:/home/ec2-user/istio-1.11.0/bin"
Istio has multiple configuration profiles, these profiles provide customization of the Istio control plane and of the sidecars for the Istio data plane.
defaultprofile is recommended for production deployments.
istioctl install --set profile=default -y
alternatively, you can also take the dump of the manifests and apply using kubectl:
istioctl manifest generate --set profile=default > generated-manifest.yaml kubectl apply -f generated-manifest.yaml
Once installed, you should see the output like this for a successful installation.
Note: If you get the below error, you should check the instance type you are using as there is a limit to the number of pods that can be scheduled on the node based on the node’s instance type. See here for the list of instance types and the max pods.
Istio injects a envoy proxy side-car container to the pods to intercept traffic from pods, this behavior is enabled using label
istio-injection=enabledon the namespace level and
sidecar.istio.io/inject=trueon the pod level.
Add this label to the
defaultnamespace to instruct Istio to automatically inject Envoy sidecar proxies.
kubectl label namespace default istio-injection=enabled
Read more about Istio side-car injection here.
Grafanato monitor and visualize mesh
kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/prometheus.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/grafana.yaml kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/kiali.yaml
Note: If you get an error like this
unable to recognize "https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/kiali.yaml": no matches for kind "MonitoringDashboard" in version "monitoring.kiali.io/v1alpha1", then re-run the command:
kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/kiali.yaml
Setup sample application
To demonstrate how canary deployment works, let’s set up version
v1 of an Nginx-based sample application and set up a service to expose it at port
v1vesion of sample application :
apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx-v1 spec: replicas: 1 selector: matchLabels: app: nginx version: v1 template: metadata: labels: app: nginx version: v1 spec: containers: - image: quay.io/shardul/nginx:v1 name: nginx-v1 imagePullPolicy: Always ports: - name: http containerPort: 80 livenessProbe: httpGet: path: /health port: 80 readinessProbe: httpGet: path: /health port: 80
Expose the application as a service:
apiVersion: v1 kind: Service metadata: labels: app: nginx name: nginx spec: ports: - name: http port: 80 selector: app: nginx
Setup a test pod to test the connectivity to the nginx service :
kubectl run -it test-connection --image=radial/busyboxplus:curl -- sh [ [email protected]:/ ]$ while true; do curl nginx && echo ; done You're at the root of nginx server v1 You're at the root of nginx server v1 You're at the root of nginx server v1 You're at the root of nginx server v1
Canary Deployment with Istio
With Istio, traffic routing and replica deployment are totally independent of each other. Istio routing rules provide fine-grained control over how to route traffic based on
source labels and control the distribution of traffic.
Deploy another version
v2of the same application that uses image
apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx-v2 spec: replicas: 1 selector: matchLabels: app: nginx version: v2 template: metadata: labels: app: nginx version: v2 spec: containers: - image: quay.io/shardul/nginx:v2 name: nginx-v2 imagePullPolicy: Always ports: - name: http containerPort: 80 livenessProbe: httpGet: path: /health port: 80 readinessProbe: httpGet: path: /health port: 80
v2version is deployed, when we hit the service again, we would get the below output :
kubectl exec -it test-connection -- sh [ [email protected]:/ ]$ while true; do curl nginx && echo ; done You're at the root of nginx server v2 You're at the root of nginx server v2 You're at the root of nginx server v1 You're at the root of nginx server v1 You're at the root of nginx server v2 You're at the root of nginx server v1 You're at the root of nginx server v1 You're at the root of nginx server v2
Since there are two deployments with different versions exposed from the same service, whenever you hit the service, it will hit the different versions of the application in a round-robin manner and hence this output.
Istio Gateway acts as a load balancer receiving incoming and outgoing HTTP/TCP connections and it’s bound to the
istio-ingressgatewayresource created during installation as a
default-gatewayfor all the incoming traffic on port
apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: default-gateway namespace: istio-system spec: selector: istio: ingressgateway servers: - hosts: - '*' port: name: http number: 80 protocol: HTTP
Destination Rule allows you to define subsets of an application based on a set of labels. For example, we have deployed two subsets
v2of an application, and they are identified by the label
Setup a DestinationRule
nginx-dest-ruleto define two subsets
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: nginx-dest-rule spec: host: nginx subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2
Virtual Service acts just like an Ingress resource and matches traffic and directs it to a service based on the HTTP routing rules.
It can operate on internal as well as external service and can match traffic based on HTTP host, path (with full regular expression support), method, headers, ports, query parameters.
Setup a VirtualService
nginxthat receives the incoming traffic on
default-gatewayand routes 90% of the traffic to the subset
v1and 10% to subset
apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: nginx-virtual-svc spec: hosts: - nginx gateways: - istio-system/default-gateway http: - route: - destination: host: nginx subset: v1 weight: 90 - destination: host: nginx subset: v2 weight: 10
90% of the traffic will be routed to version
v2 of the
nginx service. Access the nginx service via
kubectl exec -it test-connection -- sh while true; do curl -H "Host: nginx" istio-ingressgateway.istio-system && echo ; done
We can visualize the traffic flow using the Kiali dashboard:
istioctl dashboard kiali
In production, after testing the
canary version to ensure that it’s working fine, we can update the VirtualService to route 100% of the traffic to this version and rollout the newer version for all the users.