Istio Canary Deploys

Performing a canary release, rather it be for A/B testing or controlled rollout, is a common practice within software deployments. Istio enables canary routing, through a plethora of HTTP Match Requests, but for my particular use case, I wanted to route to the canary release based on a particular request header. Internally, this enables us to dark deploy, run a/b tests, and enable developers to test features before cutting our production traffic over.

Prerequisites:

  • Istio installed namespace

Architecture:

Alt Text

For this POC, I built a sample app which simply returns “test-a” if it's using the “a” image tag, or “test-b” if using the “b” tag. When a user makes an HTTP request call, if it has the header version: canary, it will route to the canary release, otherwise, it will route to the production release. All source code is linked below.

Setup

With istio installed namespace, we need to deploy two deployments of our application, which is fronted by a service. For simplicity, I’ve redacted the config to only the essential pieces:

# Production Release
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-canary-testing
  name: app-production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-canary-testing
  template:
    metadata:
      labels:
        app: app-canary-testing
        version: production
    spec:
      containers:
      - image: danquack/canary-testing:a
        name: canary-testing
        ports:
        - containerPort: 9000
          name: http
          protocol: TCP

# Canary Release
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-canary-testing
  name: app-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-canary-testing
  template:
    metadata:
      labels:
        app: app-canary-testing
        version: canary
    spec:
      containers:
      - image: danquack/canary-testing:b
        name: canary-testing
        ports:
        - containerPort: 9000
          name: http
          protocol: TCP

A couple of the key differences between these two deployments is the image tag (prod: a, canary: b) and the pod version label, which we will utilize later.

With the deployment defined, next is setting up the service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: app-canary-testing
  name: app-canary-testing
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: app-canary-testing

Istio Configuration

To configure routing we are going to configure three resources, which will define ingress, subsets (groups driven from labels), and routing rules.

Ingress

Configure a Istio Gateway to route traffic for our hostname app.local

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: canary-testing
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - app.local
    port:
      name: http
      number: 80
      protocol: HTTP

Destination Rule

Destination rules are designed to define traffic policies for load balancing or group individual versions of services. With subsets, we can define individual groups driven from the label we set on our pod. This config states for our service hostname, group the pods based on the label version:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: canary-testing
spec:
  host: app-canary-testing
  subsets:
  - labels:
      version: canary
    name: canary
  - labels:
      version: production
    name: production

Virtual Service

Now that all the core pieces are in place, we can configure the joining definition, which will allow us to route the traffic in the desired manner. When the traffic comes through the ingress gateway, istio will check our virtual service, to determine where to send that packet. In this setup, because the desired response from a requestor who sends an HTTP request with the header version: canary is to route to the canary pod, we will configure the matching header, and then route to the service with the matching subset created from the above Destination Rule.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: canary-testing
spec:
  gateways:
  - canary-testing
  hosts:
  - app.local
  http:
  - match:
    - headers:
        version:
          exact: canary
    name: canary-route
    route:
    - destination:
        host: app-canary-testing
        subset: canary
  - name: Production Route
    route:
    - destination:
        host: app-canary-testing
        subset: production

Testing

To verify our deploy is working, first, grab the ingress host:

$ kubectl -n istio-system get service istio-ingressgateway

Then well try a sample curl call to our production endpoint:

$ curl http://$INGRESS_HOST -H "Host: app.local"
test-a

Then append the http header, version: canary, which will verify your canary release

$ curl http://$INGRESS_HOST -H "Host: app.local" -H "version: canary"
test-b

All source code included can be found on Github:

NFS-API