Writing a New Stage

This doc covers the back-end changes you need to make to Orca to implement a new stage type.

Overview

To create a new stage, you need to make backend changes to orca to implement the logic of the stage, and the front-end changes to deck to implement the UI. Depending on what the stage does, you may need to implement new cloud provider-specific logic into clouddriver as well.

This doc currently only covers the backend changes made to orca.

Backend (orca)

For the backend, you need to define:

  • A stage class
  • One or more task classes associated with the stage

Stage class

A stage class must implement the com.netflix.spinnaker.orca.pipeline.StageDefinitionBuilder interface.

For providing additional functionality, it can also implement other interfaces:

  • A CancellableStage can be cancelled.
  • A RestartableStage can be restarted.
  • A CloudProviderAware stage exposes information about the cloud provider.
  • An AuthenticatedStage can perform custom authentication.

Here’s an example based on a stage that is used internally at Netflix to integrate with the Chaos Automation Platform (ChAP).

The stage is composed of two tasks:

  • beginChap starts a new ChAP run
  • monitorChap waits for the ChAP run to finish
  1. package com.netflix.spinnaker.orca.pipeline;
  2. import com.google.common.collect.ImmutableMap;
  3. import com.netflix.spinnaker.orca.CancellableStage;
  4. import com.netflix.spinnaker.orca.chap.ChapService;
  5. import com.netflix.spinnaker.orca.chap.Run;
  6. import com.netflix.spinnaker.orca.chap.tasks.BeginChapTask;
  7. import com.netflix.spinnaker.orca.chap.tasks.MonitorChapTask;
  8. import com.netflix.spinnaker.orca.pipeline.model.Execution;
  9. import com.netflix.spinnaker.orca.pipeline.model.Stage;
  10. import org.springframework.beans.factory.annotation.Autowired;
  11. import org.springframework.stereotype.Component;
  12. @Component
  13. public class ChapStage implements StageDefinitionBuilder, CancellableStage {
  14. @Autowired
  15. public ChapService chapService;
  16. @Override
  17. public <T extends Execution<T>> void taskGraph(Stage<T> stage, TaskNode.Builder builder {
  18. builder
  19. .withTask("beginChap", BeginChapTask.class)
  20. .withTask("monitorChap", MonitorChapTask.class);
  21. }
  22. @Override
  23. public CancellableStage.Result cancel(Stage stage) {
  24. Run run = (Run) stage.getContext().get("run");
  25. if (run != null) {
  26. Run latestDetails = chapService.cancelChap(run.id.toString(), "");
  27. return new CancellableStage.Result(stage, ImmutableMap.of("run", latestDetails));
  28. }
  29. return null;
  30. }
  31. }

Task classes

A task class must implement a com.netflix.spinnaker.orca.Task , or an interface that extends it, such as:

  • A RetryableTask can be retried if it fails.
  • A PreconditionTask defines preconditions that the task will enforce.

To communicate that a task failed, throw a RuntimeException.

In our example, the ChapStage consists of two tasks:

  1. BeginChapTask
  2. MonitorChapTask

BeginChapTask

The BeginChapTask:

  1. retrieves the testCaseId specified by the user during the task configuration stage
  2. calls the ChAP service via REST API call (see ChapService.startChap)
  3. returns a DefaultTaskResult, passing it the response from the ChAP REST API call
  1. package com.netflix.spinnaker.orca.chap.tasks;
  2. import com.netflix.spinnaker.orca.*;
  3. import com.netflix.spinnaker.orca.chap.ChapService;
  4. import com.netflix.spinnaker.orca.chap.Run;
  5. import com.netflix.spinnaker.orca.pipeline.model.Stage;
  6. import org.springframework.beans.factory.annotation.Autowired;
  7. import org.springframework.stereotype.Component;
  8. import java.util.HashMap;
  9. import java.util.Map;
  10. import java.util.concurrent.TimeUnit;
  11. @Component
  12. public class BeginChapTask implements RetryableTask {
  13. @Override
  14. public TaskResult execute(Stage stage) {
  15. Map<String, Object> ctx = stage.getContext();
  16. Object testCaseId = ctx.get("testCaseId");
  17. if(testCaseId == null || !(testCaseId instanceof String)) {
  18. throw new RuntimeException("Cannot begin ChAP experiment without a testCaseId.");
  19. }
  20. Map<String, Object> params = new HashMap<>();
  21. params.put("testCaseId", testCaseId);
  22. Run chapRun = chapService.startChap(params);
  23. Map<String, Object> map = new HashMap<>();
  24. map.put("run", chapRun);
  25. return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
  26. }
  27. public ChapService getChapService() {
  28. return chapService;
  29. }
  30. public void setChapService(ChapService chapService) {
  31. this.chapService = chapService;
  32. }
  33. @Autowired
  34. private ChapService chapService;
  35. @Override
  36. public long getBackoffPeriod() {
  37. return TimeUnit.SECONDS.toMillis(5);
  38. }
  39. @Override
  40. public long getTimeout() {
  41. return TimeUnit.MINUTES.toMillis(1);
  42. }
  43. }

MonitorChapTask

The MonitorChapTask polls the ChAP service for the status of the ChAP run.

  1. package com.netflix.spinnaker.orca.chap.tasks;
  2. import com.fasterxml.jackson.databind.ObjectMapper;
  3. import com.netflix.spinnaker.orca.DefaultTaskResult;
  4. import com.netflix.spinnaker.orca.ExecutionStatus;
  5. import com.netflix.spinnaker.orca.RetryableTask;
  6. import com.netflix.spinnaker.orca.TaskResult;
  7. import com.netflix.spinnaker.orca.chap.ChapService;
  8. import com.netflix.spinnaker.orca.chap.Run;
  9. import com.netflix.spinnaker.orca.pipeline.model.Stage;
  10. import org.springframework.beans.factory.annotation.Autowired;
  11. import org.springframework.stereotype.Component;
  12. import java.util.HashMap;
  13. import java.util.Map;
  14. import java.util.concurrent.TimeUnit;
  15. @Component
  16. public class MonitorChapTask implements RetryableTask {
  17. @Autowired
  18. private ObjectMapper objectMapper;
  19. @Autowired
  20. public ChapService chapService;
  21. @Override
  22. public TaskResult execute(Stage stage) {
  23. Map<String, Object> ctx = stage.getContext();
  24. Run run = objectMapper.convertValue(ctx.get("run"), Run.class);
  25. if (run == null) {
  26. throw new RuntimeException("Cannot monitor Chap task without a valid Run in the context.");
  27. }
  28. Run latestDetails = chapService.getChap(run.id.toString());
  29. Map<String, Object> map = new HashMap<>();
  30. map.put("run", latestDetails);
  31. if(latestDetails.outcome == Run.Outcome.PASSED){
  32. return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
  33. }
  34. ExecutionStatus status;
  35. switch (latestDetails.state) {
  36. case COMPLETED:
  37. //workflow is complete, but the outcome didnt pass, consider this a failure.
  38. case FAILED:
  39. throw new RuntimeException("ChAP experiment failed.");
  40. case CANCELLED:
  41. status = ExecutionStatus.CANCELED;
  42. break;
  43. default:
  44. status = ExecutionStatus.RUNNING;
  45. break;
  46. }
  47. return new DefaultTaskResult(status, map);
  48. }
  49. public ChapService getChapService() {
  50. return chapService;
  51. }
  52. public void setChapService(ChapService chapService) {
  53. this.chapService = chapService;
  54. }
  55. @Override
  56. public long getBackoffPeriod() {
  57. return TimeUnit.MINUTES.toMillis(1);
  58. }
  59. @Override
  60. public long getTimeout() {
  61. return TimeUnit.DAYS.toMillis(1);
  62. }
  63. public ObjectMapper getObjectMapper() {
  64. return objectMapper;
  65. }
  66. public void setObjectMapper(ObjectMapper objectMapper) {
  67. this.objectMapper = objectMapper;
  68. }
  69. }

Other classes used

The details of the com.netflix.spinnaker.orca.chap.Run class and com.netflix.spinnaker.orca.chap.ChapService interface aren’t directly relevant to learning how to write a Spinnaker stage, but for completeness, here’s what those look like for this case:

Run

The Run class is a Java object that is deserialized from JSON.

  1. package com.netflix.spinnaker.orca.chap;
  2. import com.fasterxml.jackson.annotation.JsonAnyGetter;
  3. import com.fasterxml.jackson.annotation.JsonAnySetter;
  4. import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
  5. import java.util.HashMap;
  6. import java.util.Map;
  7. import java.util.UUID;
  8. @JsonIgnoreProperties(ignoreUnknown = true)
  9. public class Run {
  10. public UUID id;
  11. // Other properties not shown
  12. // ...
  13. // Support arbitrary properties without needing to define them explicitly
  14. public Map<String, Object> properties = new HashMap<>();
  15. @JsonAnySetter
  16. public void set(String fieldName, Object value) {
  17. this.properties.put(fieldName, value);
  18. }
  19. @JsonAnyGetter
  20. public Object get(String fieldName) {
  21. return this.properties.get(fieldName);
  22. }
  23. }

ChapService

The ChapService defines a REST client API for talking to the ChAP service. It uses the Retrofit library.

  1. package com.netflix.spinnaker.orca.chap;
  2. import retrofit.http.Body;
  3. import retrofit.http.GET;
  4. import retrofit.http.POST;
  5. import retrofit.http.Path;
  6. import java.util.Map;
  7. public interface ChapService {
  8. @POST("/v1/runs")
  9. Run startChap(@Body Map params);
  10. @GET("/v1/runs/{id}")
  11. Run getChap(@Path("id") String id);
  12. @POST("/v1/runs/{id}/stop")
  13. Run cancelChap(@Path("id") String id, @Body String body);
  14. }

ChapConfig

To implement the ChapService interface, we do not define a class that extends the interface. Instead, we define a class named ChapConfig with a Spring @Configuration annotation. Note that this implementation uses the chap.baseUrl configuration value that is defined in a separate Spinnaker configuration file.

  1. package com.netflix.spinnaker.config;
  2. import com.fasterxml.jackson.databind.ObjectMapper;
  3. import com.netflix.spinnaker.orca.chap.ChapService;
  4. import com.netflix.spinnaker.retrofit.Slf4jRetrofitLogger;
  5. import org.springframework.beans.factory.annotation.Autowired;
  6. import org.springframework.beans.factory.annotation.Value;
  7. import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
  8. import org.springframework.context.annotation.Bean;
  9. import org.springframework.context.annotation.ComponentScan;
  10. import org.springframework.context.annotation.Configuration;
  11. import retrofit.Endpoint;
  12. import retrofit.RestAdapter;
  13. import retrofit.client.Client;
  14. import retrofit.converter.JacksonConverter;
  15. import static retrofit.Endpoints.newFixedEndpoint;
  16. @Configuration
  17. @ComponentScan({
  18. "com.netflix.spinnaker.orca.chap.pipeline",
  19. "com.netflix.spinnaker.orca.chap.tasks"
  20. })
  21. @ConditionalOnProperty(value = "chap.baseUrl")
  22. public class ChapConfig {
  23. @Bean
  24. Endpoint chapEndpoint(@Value("${chap.baseUrl}") String chapBaseUrl) {
  25. return newFixedEndpoint(chapBaseUrl);
  26. }
  27. @Bean
  28. ChapService chapService(Endpoint chapEndpoint,
  29. Client retrofitClient,
  30. RestAdapter.LogLevel retrofitLogLevel,
  31. ObjectMapper objectMapper) {
  32. return new RestAdapter.Builder()
  33. .setEndpoint(chapEndpoint)
  34. .setClient(retrofitClient)
  35. .setLogLevel(retrofitLogLevel)
  36. .setLog(new Slf4jRetrofitLogger(ChapService.class))
  37. .setConverter(new JacksonConverter(objectMapper))
  38. .build()
  39. .create(ChapService.class);
  40. }
  41. }

Last modified June 24, 2021: Redesign Progress (#83) (8231bcf)