This document describes how to use Chaos Mesh to inject faults into Kubernetes Pod to simulate Pod or container faults. Chaos Dashboard and YAML files are provided to create PodChaos experiments.

PodChaos introduction

PodChaos is a fault type in Chaos Mesh. By creating a PodChaos experiment, you can simulate fault scenarios of the specified Pods or containers. Currently, PodChaos supports the following fault types:

  • Pod Failure: injects fault into a specified Pod to make the Pod unavailable for a period of time.
  • Pod Kill: kills a specified Pod.To ensure that the Pod can be successfully restarted, you need to configure ReplicaSet or similar mechanisms.
  • Container Kill: kills the specified container in the target Pod.

Usage restrictions

Currently, Chaos Mesh only supports fault injection to certain types of Pod, such as Deployment, StatefulSet, and DaemonSet. Chaos Mesh does not support injecting faults into an independent Pod. An independent Pod means a Pod that is not bound to ReplicaSet or Deployment Pod.

Notes

Before creating PodChaos experiments, ensure the following:

  • There is no Control Manager of Chaos Mesh running on the target Pod.
  • If the fault type is Pod Kill, replicaSet or a similar mechanism is configured to ensure that Pod can restart automatically.

Create Experiments Using Chaos Dashboard

:::note

Before create experiments using Chaos Dashboard, ensure the following:

  • Chaos Dashboard is installed.
  • If Chaos Dashboard is already installed, you can run kubectl port-forward to access Dashboard: bash kubectl port-forward -n chaos-testing svc/chaos-dashboard 2333:2333. Then you can enter http://localhost:2333 to access Chaos Dashboard.

:::

  1. Open Chaos Dashboard, and click NEW EXPERIMENT on the page to create a new experiment.

Create a New Experiment

  1. In the Choose a Target area, choose POD FAULT and select a specific behavior, such as POD FAILURE.

  2. Fill out the experiment information, and specify the experiment scope and the scheduled experiment duration.

  3. Submit the experiment information.

Create experiments using YAML configuration files

pod-failure example

  1. Write the experiment configuration to the pod-failure.yaml file:
  1. apiVersion: chaos-mesh.org/v1alpha1
  2. kind: PodChaos
  3. metadata:
  4. name: pod-failure-example
  5. namespace: chaos-testing
  6. spec:
  7. action: pod-failure
  8. mode: one
  9. duration: '30s'
  10. selector:
  11. labelSelectors:
  12. 'app.kubernetes.io/component': 'tikv'

Based on this example, Chaos Mesh injects pod-failure into the specified Pod and makes the Pod unavailable for 30 seconds.

  1. After the configuration file is prepared, use kubectl to create an experiment:
  1. kubectl apply -f ./pod-failure.yaml

pod-kill example

  1. Write the experiment configuration to the pod-kill.yaml file:
  1. apiVersion: chaos-mesh.org/v1alpha1
  2. kind: PodChaos
  3. metadata:
  4. name: pod-kill-example
  5. namespace: chaos-testing
  6. spec:
  7. action: pod-kill
  8. mode: one
  9. selector:
  10. namespaces:
  11. - tidb-cluster-demo
  12. labelSelectors:
  13. 'app.kubernetes.io/component': 'tikv'

Based on this example, Chaos Mesh injects pod-kill into the specified Pod and kills the Pod once.

  1. After the configuration file is prepared, use kubectl to create an experiment:
  1. kubectl apply -f ./pod-kill.yaml

container-kill example

  1. Write the experiment configuration to the container-kill.yaml file:
  1. apiVersion: chaos-mesh.org/v1alpha1
  2. kind: PodChaos
  3. metadata:
  4. name: container-kill-example
  5. namespace: chaos-testing
  6. spec:
  7. action: container-kill
  8. mode: one
  9. containerNames: ['prometheus']
  10. selector:
  11. labelSelectors:
  12. 'app.kubernetes.io/component': 'monitor'

Based on this example, Chaos Mesh injects container-kill into the specified container and kills the container once.

  1. After the configuration file is prepared, use kubectl to create an experiment:
  1. kubectl apply -f ./container-kill.yaml

Field description

The following table describes the fields in the YAML configuration file.

Parameter Type Description Default value Required Example
action string Specifies the fault type to inject. The supported types include pod-failure, pod-kill, and container-kill. None Yes pod-kill
mode string Specifies the mode of the experiment. The mode options include one (selecting a random Pod), all (selecting all eligible Pods), fixed (selecting a specified number of eligible Pods), fixed-percent (selecting a specified percentage of Pods from the eligible Pods), and random-max-percent (selecting the maximum percentage of Pods from the eligible Pods). None Yes 1
value string Provides parameters for the mode configuration, depending on mode.For example, when mode is set to fixed-percent, value specifies the percentage of Pods. None No 2
selector struct Specifies the target Pod. For details, refer to Define the experiment scope. None Yes
containerNames []string When you configure action to container-killed, this configuration is mandatory to specify the target container name for injecting faults. None No [‘prometheus’]
gracePeriod int64 When you configure action to pod-kill, this configuration is mandatory to specify the duration before deleting Pod. 0 No 0
duration string Specifies the duration of the experiment. None Yes 30s