Istio Canary Deploys
Performing a canary release, rather it be for A/B testing or controlled rollout, is a common practice within software deployments. Istio enables canary routing, through a plethora of HTTP Match Requests, but for my particular use case, I wanted to route to the canary release based on a particular request header. Internally, this enables us to dark deploy, run a/b tests, and enable developers to test features before cutting our production traffic over.
- Istio installed namespace
For this POC, I built a sample app which simply returns “test-a” if it's using the “a” image tag, or “test-b” if using the “b” tag. When a user makes an HTTP request call, if it has the header
version: canary, it will route to the canary release, otherwise, it will route to the production release. All source code is linked below.
With istio installed namespace, we need to deploy two deployments of our application, which is fronted by a service. For simplicity, I’ve redacted the config to only the essential pieces:
# Production Release apiVersion: apps/v1 kind: Deployment metadata: labels: app: app-canary-testing name: app-production spec: replicas: 1 selector: matchLabels: app: app-canary-testing template: metadata: labels: app: app-canary-testing version: production spec: containers: - image: danquack/canary-testing:a name: canary-testing ports: - containerPort: 9000 name: http protocol: TCP
# Canary Release apiVersion: apps/v1 kind: Deployment metadata: labels: app: app-canary-testing name: app-canary spec: replicas: 1 selector: matchLabels: app: app-canary-testing template: metadata: labels: app: app-canary-testing version: canary spec: containers: - image: danquack/canary-testing:b name: canary-testing ports: - containerPort: 9000 name: http protocol: TCP
A couple of the key differences between these two deployments is the image tag (prod: a, canary: b) and the pod version label, which we will utilize later.
With the deployment defined, next is setting up the service:
apiVersion: v1 kind: Service metadata: labels: app: app-canary-testing name: app-canary-testing spec: ports: - name: http port: 80 protocol: TCP targetPort: http selector: app: app-canary-testing
To configure routing we are going to configure three resources, which will define ingress, subsets (groups driven from labels), and routing rules.
Configure a Istio Gateway to route traffic for our hostname app.local
apiVersion: networking.istio.io/v1beta1 kind: Gateway metadata: name: canary-testing spec: selector: istio: ingressgateway servers: - hosts: - app.local port: name: http number: 80 protocol: HTTP
Destination rules are designed to define traffic policies for load balancing or group individual versions of services. With subsets, we can define individual groups driven from the label we set on our pod. This config states for our service hostname, group the pods based on the label
apiVersion: networking.istio.io/v1beta1 kind: DestinationRule metadata: name: canary-testing spec: host: app-canary-testing subsets: - labels: version: canary name: canary - labels: version: production name: production
Now that all the core pieces are in place, we can configure the joining definition, which will allow us to route the traffic in the desired manner. When the traffic comes through the ingress gateway, istio will check our virtual service, to determine where to send that packet. In this setup, because the desired response from a requestor who sends an HTTP request with the header
version: canary is to route to the canary pod, we will configure the matching header, and then route to the service with the matching subset created from the above Destination Rule.
apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: name: canary-testing spec: gateways: - canary-testing hosts: - app.local http: - match: - headers: version: exact: canary name: canary-route route: - destination: host: app-canary-testing subset: canary - name: Production Route route: - destination: host: app-canary-testing subset: production
To verify our deploy is working, first, grab the ingress host:
$ kubectl -n istio-system get service istio-ingressgateway
Then well try a sample curl call to our production endpoint:
$ curl http://$INGRESS_HOST -H "Host: app.local" test-a
Then append the http header,
version: canary, which will verify your canary release
$ curl http://$INGRESS_HOST -H "Host: app.local" -H "version: canary" test-b
All source code included can be found on Github: