Shadowing – VirtualServices in Practice

Shadowing or Mirroring is used when we want to test a change in production but not affect end-users, so we mirror the requests into a second instance that has the change and evaluate it. To phrase it simpler it’s when one of your colleagues picks the most critical issue and makes a Big ball of mud Pull Request, that nobody can really review.

To test out this feature lets create a second instance of SA-Logic that is buggy by executing the command below:

$ kubectl apply -f resource-manifests/kube/shadowing/sa-logic-service.buggy.yaml

Execute the following command and verify that all instances are labeled with the respective versions and additionally with app=sa-logic:

$ kubectl get pods -l app=sa-logic --show-labels
NAME                              READY   LABELS
sa-logic-568498cb4d-2sjwj         2/2     app=sa-logic,version=v1
sa-logic-568498cb4d-p4f8c         2/2     app=sa-logic,version=v1
sa-logic-buggy-76dff55847-2fl66   2/2     app=sa-logic,version=v2
sa-logic-buggy-76dff55847-kx8zz   2/2     app=sa-logic,version=v2

Because the service sa-logic targets pods labeled with app=sa-logic, any incoming requests will be load-balanced between all instances, as shown in figure 1.

But we want requests to be routed to the instances with version v1 and mirrored to the instances with version v2, as shown in figure 2.

This is achieved using a VirtualService in combination with a DestinationRule, where the destination rule specifies the subsets and VirtualService routes to the desired subset.

Specifying Subsets with Destination Rules

We define the subsets with the following configuration:

1. Host defines that this rule applies only when routing has occurred towards sa-logic service
2. Subset name used when routing to instances of a subset.
3. Label defines the key-value pairs that need to match for an instance to be part of the subset.

Apply the configuration executing the command below:

$ kubectl apply -f resource-manifests/istio/shadowing/sa-logic-subsets-destinationrule.yaml
destinationrule.networking.istio.io/sa-logic created

With the subsets defined we can move on and configure a VirtualService to apply to requests towards sa-logic where the requests are:

1. Routed to the subset named v1 and,
2. Mirrored to the subset named v2.

And this is achieved with the manifest below:

As everything is self-explanatory let’s just see it in action:

$ kubectl apply -f resource-manifests/istio/shadowing/sa-logic-subsets-shadowing-vs.yaml
virtualservice.networking.istio.io/sa-logic created

Add some load by executing the following command:

$ while true; do curl -v http://$EXTERNAL_IP/sentiment \ 
    -H "Content-type: application/json" \ 
    -d '{"sentence": "I love yogobella"}'; \
    sleep .8; done

Check the results in Grafana, where we can see that the buggy version is failing about 60% of the requests, but none of the failures affected the end-users as they were responded by the currently active service.

In this section we saw for the first time a VirtualService that was applied to the envoys of our services, to be clearer, when the sa-web-app makes a request towards sa-logic this goes through sa-web-apps envoy, which via the VirtualService is configured to route to the subset v1 and mirror to the subset v2 of the sa-logic service.

I can see you thinking “Darn man Virtual Services are amazing!”, and they are going to get even better!