Add New Chaos Experiment Type

This document describes how to add a new chaos experiment type.

The following walks you through an example of HelloWorldChaos, a new chaos experiment type that prints Hello World! to the log. The steps include:

  • Step 1: Define the schema type
  • Step 2: Register the CRD
  • Step 3: Register the event handler for this chaos object
  • Step 4: Build the Docker image
  • Step 5: Run the chaos experiment

Step 1: Define the schema type

  1. Add helloworldchaos_types.go in the API directory api/v1alpha1 with the following content:

    1. package v1alpha1
    2. import (
    3. metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    4. )
    5. // +kubebuilder:object:root=true
    6. // +chaos-mesh:experiment
    7. // +chaos-mesh:oneshot=true
    8. // HelloWorldChaos is the Schema for the helloworldchaos API
    9. type HelloWorldChaos struct {
    10. metav1.TypeMeta `json:",inline"`
    11. metav1.ObjectMeta `json:"metadata,omitempty"`
    12. Spec HelloWorldChaosSpec `json:"spec"`
    13. Status HelloWorldChaosStatus `json:"status,omitempty"`
    14. }
    15. // HelloWorldChaosSpec is the content of the specification for a HelloWorldChaos
    16. type HelloWorldChaosSpec struct {
    17. // ContainerSelector specifies target
    18. ContainerSelector `json:",inline"`
    19. // Duration represents the duration of the chaos action
    20. // +optional
    21. Duration *string `json:"duration,omitempty"`
    22. }
    23. // HelloWorldChaosStatus represents the status of a HelloWorldChaos
    24. type HelloWorldChaosStatus struct {
    25. ChaosStatus `json:",inline"`
    26. }
    27. // GetSelectorSpecs is a getter for selectors
    28. func (obj *HelloWorldChaos) GetSelectorSpecs() map[string]interface{} {
    29. return map[string]interface{}{
    30. ".": &obj.Spec.ContainerSelector,
    31. }
    32. }

    This file defines the schema type of HelloWorldChaos, which can be described in a YAML file:

    1. apiVersion: chaos-mesh. rg/v1alpha1
    2. kind: HelloWorldChaos
    3. metadata:
    4. name: <resource name>
    5. name: <namespace>
    6. spec:
    7. duration: <duration>
    8. status:
    9. experiment: <experimental status>
    10. ...
  2. Run make generate in the root directory of Chaos Mesh, which generates a boilerplate for Chaos Mesh to compile.

Step 2: Register the CRD

You need to register the CRD (Custom Resource Definition) of HelloWorldChaos to interact it with Kubernetes API.

  1. Run make yaml in the root directory. The generated YAML file is at config/crd/bases/chaos-mesh.org_helloworldchaos.yaml.

  2. To combine this YAML file into manifests/crd.yaml, add a new line in config/crd/kustomization.yaml:

    1. resources:
    2. - bases/chaos-mesh.org_podchaos.yaml
    3. - bases/chaos-mesh.org_networkchaos.yaml
    4. - bases/chaos-mesh.org_iochaos.yaml
    5. - bases/chaos-mesh.org_helloworldchaos.yaml # This is the new line
  3. Run make yaml again. You can see the definition of HelloWorldChaos in manifests/crd.yaml. To confirm, you can use the git diff command.

Step 3: Register the event handler for this chaos object

  1. Create a new file controllers/chaosimpl/helloworldchaos/types.go with the following content:

    1. package helloworldchaos
    2. import (
    3. "context"
    4. "github.com/go-logr/logr"
    5. "go.uber.org/fx"
    6. "sigs.k8s.io/controller-runtime/pkg/client"
    7. "github.com/chaos-mesh/chaos-mesh/api/v1alpha1"
    8. "github.com/chaos-mesh/chaos-mesh/controllers/chaosimpl/utils"
    9. "github.com/chaos-mesh/chaos-mesh/controllers/common"
    10. "github.com/chaos-mesh/chaos-mesh/controllers/utils/chaosdaemon"
    11. )
    12. type Impl struct {
    13. client.Client
    14. Log logr.Logger
    15. decoder *utils.ContainerRecordDecoder
    16. }
    17. // This corresponds to the Apply phase of HelloWorldChaos. The execution of HelloWorldChaos will be triggered.
    18. func (impl *Impl) Apply(ctx context.Context, index int, records []*v1alpha1.Record, obj v1alpha1.InnerObject) (v1alpha1.Phase, error) {
    19. impl.Log.Info("Hello world!")
    20. return v1alpha1.Injected, nil
    21. }
    22. // This corresponds to the Recover phase of HelloWorldChaos. The reconciler will be triggered to recover the chaos action.
    23. func (impl *Impl) Recover(ctx context.Context, index int, records []*v1alpha1.Record, obj v1alpha1.InnerObject) (v1alpha1.Phase, error) {
    24. impl.Log.Info("Goodbye world!")
    25. return v1alpha1.NotInjected, nil
    26. }
    27. func NewImpl(c client.Client, log logr.Logger, decoder *utils.ContainerRecordDecoder) *common.ChaosImplPair {
    28. return &common.ChaosImplPair{
    29. Name: "helloworldchaos",
    30. Object: &v1alpha1.HelloWorldChaos{},
    31. Impl: &Impl{
    32. Client: c,
    33. Log: log.WithName("helloworldchaos"),
    34. decoder: decoder,
    35. },
    36. ObjectList: &v1alpha1.HelloWorldChaosList{},
    37. }
    38. }
    39. var Module = fx.Provide(
    40. fx.Annotated{
    41. Group: "impl",
    42. Target: NewImpl,
    43. },
    44. )
  2. Chaos Mesh uses the fx library for dependency injection. To register HelloWorldChaos in the Controller Manager, add a line in controllers/chaosimpl/fx.go:

    1. ...
    2. gcpchaos.Module,
    3. stresschaos.Module,
    4. jvmchaos.Module,
    5. timechaos.Module,
    6. helloworldchaos.Module //Add a new line. Make sure you have imported HelloWorldChaos to this file first.

    In controllers/types/types.go, add the following content:

    1. ...
    2. fx.Annotated{
    3. Group: "objs",
    4. Target: Object{
    5. Name: "timechaos",
    6. Object: &v1alpha1.TimeChaos{},
    7. },
    8. },
    9. fx.Annotated{
    10. Group: "objs",
    11. Target: Object{
    12. Name: "gcpchaos",
    13. Object: &v1alpha1.GCPChaos{},
    14. },
    15. },
    16. fx.Annotated{
    17. Group: "objs",
    18. Target: Object{
    19. Name: "helloworldchaos",
    20. Object: &v1alpha1.HelloWorldChaos{},
    21. },
    22. },

Step 4: Build the Docker image

  1. Build the Docker image:

    1. make
  2. Push the image to your local Docker Registry:

    1. make docker-push
  3. If you deploy Kubernetes clusters using kind, then you need to load images into kind:

    1. kind load docker-image localhost:5000/chaos-mesh/chaos-mesh:latest
    2. kind load docker-image localhost:5000/chaos-mesh/chaos-daemon:latest
    3. kind load docker-image localhost:5000/chaos-mesh/chaos-dashboard:latest

Step 5: Run the chaos experiment

In this step, you need to deploy Chaos Mesh with your latest changes to test HelloWorldChaos.

Before you pull any image for Chaos Mesh (using helm install or helm upgrade), modify the helm template helm/chaos-mesh/values.yaml to replace the default image with what you just pushed to your local Docker registry.

The templates in Chaos Mesh use chaos-mesh/chaos-mesh:latest as the default Registry. You need to set the path with the environment variable of DOCKER_REGISTRY (The default value is localhost:5000), as shown below:

  1. controllerManager:
  2. image: localhost:5000/chaos-mesh/chaos-mesh:latest
  3. ...
  4. chaosDaemon:
  5. image: localhost:5000/chaos-mesh/chaos-daemon:latest
  6. ...
  7. dashboard:
  8. image: localhost:5000/chaos-mesh/chaos-dashboard:latest
  9. ...

After you update the template, try running HelloWorldChaos.

  1. Register the CRD in your cluster:

    1. kubectl create -f manifests/crd.yaml

    You can see HelloWorldChaos is created from the output:

    1. customresourcedefinition.apiextensions.k8s.io/helloworldchaos.chaos-mesh.org created

    Now you can get the CRD of HelloWorldChaos using the command below:

    1. kubectl get crd helloworldchaos.chaos-mesh.org
  2. Deploy Chaos Mesh:

    To verify the deployment is successful, you can check all Pods in the chaos-testing namespace:

    1. kubectl get pods --namespace chaos-testing -l app.kubernetes.io/instance=chaos-mesh

    Add New Chaos Experiment Type - 图1Arguments --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock are used to run NeteworkChaos on kind. :::

  3. Deploy the Pod for testing:

    1. kubectl apply -f https://raw.githubusercontent.com/chaos-mesh/apps/master/ping/busybox-statefulset.yaml

    Make sure the Pod for testing works properly.

  4. Create a chaos.yaml file in any location with the following content:

    1. apiVersion: chaos-mesh.org/v1alpha1
    2. kind: HelloWorldChaos
    3. metadata:
    4. name: hello-world
    5. namespace: chaos-testing
    6. spec:
    7. selector:
    8. namespaces:
    9. - busybox
    10. mode: one
    11. duration: 1h
  5. Run the chaos experiment:

    1. kubectl apply -f /path/to/chaos.yaml
    1. kubectl get HelloWorldChaos -n chaos-testing

    Now you can see Hello World! in the logs of chaos-controller-manager:

    1. kubectl logs chaos-controller-manager-{pod-post-fix} -n chaos-testing

    Example output:

    1. 2021-06-24T06:42:26.858Z INFO records apply chaos {"id": "chaos-testing/chaos-daemon-vsmc5"}
    2. 2021-06-24T06:42:26.858Z INFO helloworldchaos Hello World!

    :::note {pod-post-fix} is a random string generated by Kubernetes. You can check it by executing kubectl get pod -n chaos-testing. :::

What’s Next

If you encounter any problems during the process, create an issue in the Chaos Mesh repository.

If you want to dive deep into developing new chaos experiment types, see Extend Chaos Daemon interface.