Dark Launcher and Automatic Tests

Before StartYou should have NO virtualservice nor destinationrule (in tutorial namespace) kubectl get virtualservice kubectl get destinationruleif so run:
  1. ./scripts/clean.sh

Preparation

We are assuming that you have recommendation v1 and v2 deployed on tutorial namespace.

The first thing you need to do is to apply Istio resources to redirect all traffic to recommendation v1.

From the main istio-tutorial directory,

  1. istioctl create -f istiofiles/destination-rule-recommendation-v1-v2.yml -n tutorial
  2. istioctl create -f istiofiles/virtual-service-recommendation-v1.yml -n tutorial
  3. curl customer-tutorial.$(minishift ip).nip.io
  4. customer => preference => recommendation v1 from '2819441432-qsp25': 3

Now you see that your production traffic always goes to recommendation v1.

Explanation

Dark Launching is a process where software is selectively or stealthily released to your users to validate that this new version behaves correctly and into expected parameters.

There are many techniques under Dark Launch technique, you’ve seen one in this tutorial ROOT:5advanced-routerules.adoc#mirroringtraffic, in this case, we are going to see how you can make that your tests (even BDD tests) can be used to test your new version without affecting your current users.

You might need to use this technique if, for example, you don’t want to start stressing your new service with public traffic at first (like happens in Mirroring Traffic technique at first.

To write these tests you can use Arquillian Cube.

arquillian cube

Arquillian Cube allows you to control Docker, Kubernetes and OpenShift containers in your tests with ease.As a side effect, it also supports Istio, so you can write tests that apply some Istio rules to configured cluster, runs the test, and finally restores the state of Istio.

For this use case what you want is to execute an automatic test against recommendation v2 meanwhile public traffic still goes to recommendation v1.For this concrete example and for sake of simplicity, we set that if user-agent header contains DarkLaunch string then traffic is redirected to v2.

Recommendation project comes with a test that it is ready to be executed, but in the next section, you’ll read how we have arrived at the given solution.

Code

The first things you need to do is adding Arquillian Cube dependency in recommendation project.

recommendation/java/vertx/pom.xml

  1. <dependency>
  2. <groupId>org.arquillian.cube</groupId>
  3. <artifactId>arquillian-cube-openshift-starter</artifactId> (1)
  4. <version>${cube.version}</version> (2)
  5. <scope>test</scope>
  6. </dependency>
  7. <dependency>
  8. <groupId>org.arquillian.cube</groupId>
  9. <artifactId>arquillian-cube-istio-kubernetes</artifactId> (3)
  10. <version>${cube.version}</version>
  11. <scope>test</scope>
  12. </dependency>
1Dependency for OpenShift. There is also the same for Kubernetes.
2Version 1.18.2 or beyond.
3Dependency for Istio integration.

Then you need to configure the namespace of the project.For this purpose, you need to create an arquillian.xml file.

recommendation/java/vertx/src/test/resources/arquillian.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <arquillian xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  3. xmlns="http://jboss.org/schema/arquillian"
  4. xsi:schemaLocation="http://jboss.org/schema/arquillian http://jboss.org/schema/arquillian/arquillian_1_0.xsd">
  5. <extension qualifier="openshift"> (1)
  6. <property name="namespace.use.existing">tutorial</property> (2)
  7. <property name="env.init.enabled">false</property> (3)
  8. </extension>
  9. </arquillian>
1OpenShift cluster.
2Use existing namespace named tutorial.
3Do not initialize anything, application is already running.

INFO: Arquillian Cube can also create a namespace and deploy any Kubernetes/OpenShift resource automatically. But since the application has been already deployed in Deploy Microservices, this step is skipped.

And finally, you need to create an automatic test that runs against v2.

In this kind of tests you only want to validate happy path, just a quality gate that allows you to validate that everything is in place.

recommendation/java/vertx/src/test/java/com/…​/recommendation/DarkLaunchIT.java

  1. @RunWith(Arquillian.class) (1)
  2. @IstioResource("classpath:dark-launch-redirect-traffic-to-new-version.yml") (2)
  3. @RestoreIstioResource("classpath:virtual-service-recommendation-v1.yml") (3)
  4. public class DarkLaunchIT {
  5. @RouteURL("customer")
  6. @AwaitRoute
  7. private URL url; (4)
  8. @ArquillianResource
  9. IstioAssistant istioAssistant; (5)
  10. @Before
  11. public void waitUntilIstioResourcesArePopulated() { (6)
  12. istioAssistant.await(createRequestForRecommendationV2(), response -> {
  13. try {
  14. return response.body().string().contains("v2");
  15. } catch (IOException e) {
  16. return false;
  17. }
  18. });
  19. }
  20. @Test
  21. public void should_return_accessing_v2_message() throws IOException { (7)
  22. // Given
  23. final Request request = createRequestForRecommendationV2(); (8)
  24. // When
  25. final String content = makeRequest(request);
  26. // Then
  27. assertThat(content)
  28. .startsWith("customer => preference => recommendation v2 from"); (9)
  29. }
  30. private String makeRequest(Request request) throws IOException {
  31. final HttpLoggingInterceptor interceptor = new HttpLoggingInterceptor();
  32. interceptor.setLevel(HttpLoggingInterceptor.Level.HEADERS);
  33. OkHttpClient client = new OkHttpClient.Builder()
  34. .addNetworkInterceptor(interceptor)
  35. .build();
  36. try(Response response = client.newCall(request).execute()) {
  37. return response.body().string();
  38. }
  39. }
  40. private Request createRequestForRecommendationV2() {
  41. return new Request.Builder()
  42. .url(url.toString())
  43. .addHeader("User-Agent", "Recommendation-v2-DarkLaunch-Test")
  44. .build();
  45. }
  46. }
1Arquillian JUnit runner.
2Resource to apply before executing the test.
3Resource to apply after executing the test.
4Public URL of application.
5IstioAssistant instance to interact with Istio ecosystem.
6Waits until Istio resources are populated acorss the cluster.
7Test to validate that recommendation v2 works as expected.
8Creates a request setting user-agent as DarkLaunch.
9Validates the response.
You’ve seen a simple example here, but you can use other strategies to get the same effect:-Instead of using user-agent you can create a custom header, or just use the request IP range.-@IstioResource and @RestoreIstioResource supports expressions like ${prop} resolving the property from environment variables or system properties.-Use failsafe plugin to run this kind of tests so they are not run all the time.-This example works with OpenShift route, but you can use @PortForwarding annotation in the test to do a port forwarding to given service.-It is so important to add a guard to wait until Istio resources are populated across the cluster. In case of minishift/kube it is almost instantly but in case of real clusters, it might take some tenths of seconds.

Test project can be found here.

Clean Up

  1. istioctl delete -f istiofiles/virtual-service-recommendation-v1.yml -n tutorial
  2. istioctl delete -f istiofiles/destination-rule-recommendation-v1-v2.yml -n tutorial

or you can run:

  1. ./scripts/clean.sh