Simulate Network Faults

This document introduces how to use Chaosd to simulate network faults. The simulations can be completed by modifying network routing and traffic flow control using iptables, ipsets, tc, etc.

Notes

Make sure the NET_SCH_NETEM module is installed in the Linux kernel. If you are using CentOS, you can install the module through the kernel-modules-extra package. Most other Linux distributions have installed it already by default.

Create network fault experiments using command-line mode

This section introduces how to create network fault experiments using command-line mode.

Before creating an experiment, you can run the following command to check the types of network faults supported by Chaosd:

  1. chaosd attack network --help

The output is as follows:

  1. Network attack related commands
  2. Usage:
  3. chaosd attack network [command]
  4. Available Commands:
  5. corrupt corrupt network packet
  6. delay delay network
  7. duplicate duplicate network packet
  8. loss loss network packet
  9. Flags:
  10. -h, --help help for network
  11. Global Flags:
  12. --log-level string the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'
  13. Use "chaosd attack network [command] --help" for more information about a command.

Currently, you can simulate four experimental scenarios using Chaosd: network corruption, network latency, network duplication, and network loss.

Network corruption

You can run the following command to see the configuration of simulated network corruption using Chaosd:

  1. chaosd attack network corrupt --help

The output is as follows:

  1. corrupt network packet
  2. Usage:
  3. chaosd attack network corrupt [flags]
  4. Flags:
  5. -c, --correlation string correlation is percentage (10 is 10%) (default "0")
  6. -d, --device string the network interface to impact
  7. -e, --egress-port string only impact egress traffic to these destination ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  8. -h, --help help for corrupt
  9. -H, --hostname string only impact traffic to these hostnames
  10. -i, --ip string only impact egress traffic to these IP addresses
  11. --percent string percentage of packets to corrupt (10 is 10%) (default "1")
  12. -p, --protocol string only impact traffic using this IP protocol, supported: tcp, udp, icmp, all
  13. -s, --source-port string only impact egress traffic from these source ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  14. Global Flags:
  15. --log-level string the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'

The related configuration items are described as follows:

Configuration itemAbbreviationDescriptionValue
correlationcThe correlation between the percentage of current corrupt occurrence and the previous occurrence.Int. It is a percentage ranging from 0 to 100 (10 is 10%) (“0” by default ).
devicedName of the impacted network interface card.String, such as “eth0”. The value is required.
egress-porteThe egress traffic that only impacts specific destination ports. It can only be configured when the protocol is TCP or UDP.String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.
hostnameHThe host name impacted by traffic.String, such as “chaos-mesh.org”.
ipiThe IP address impacted by egress traffic.String, such as “123.123.123.123”.
protocolpThe IP protocol impacted by traffic.String. Supported protocols: tcp, udp, icmp, all (all network protocols).
source-portsThe egress traffic which only impact specific source ports. It can only be configured when the protocol is tcp or udp.String. Use a ‘,’ to delimit the specific port or to indicate the range of the ports, such as “80,8001:8010”.

Network latency

You can run the following command to see the configuration of simulated network latency using Chaosd:

  1. chaosd attack network delay --help

The output is as follows:

  1. delay network
  2. Usage:
  3. chaosd attack network delay [flags]
  4. Flags:
  5. -c, --correlation string correlation is percentage (10 is 10%) (default "0")
  6. -d, --device string the network interface to impact
  7. -e, --egress-port string only impact egress traffic to these destination ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  8. -h, --help help for delay
  9. -H, --hostname string only impact traffic to these hostnames
  10. -i, --ip string only impact egress traffic to these IP addresses
  11. -j, --jitter string jitter time, time units: ns, us (or µs), ms, s, m, h.
  12. -l, --latency string delay egress time, time units: ns, us (or µs), ms, s, m, h.
  13. -p, --protocol string only impact traffic using this IP protocol, supported: tcp, udp, icmp, all
  14. -s, --source-port string only impact egress traffic from these source ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  15. Global Flags:
  16. --log-level string the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'

The related configuration items are described as follows:

Configuration itemAbbreviationDescriptionValue
correlationcThe correlation between the current latency and the previous one.Int. It is a percentage ranging from 0 to 100 (10 is 10%) (“0” by default).
devicedName of the impacted network interface card.String, such as “eth0”. The value is required.
egress-porteThe egress traffic which only impact specific destination ports. It can only be configured when the protocol is TCP or UDP.String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.
hostnameHThe host name impacted by traffic.String, such as “chaos-mesh.org”.
ipiThe IP address impacted by egress traffic.String, such as “123.123.123.123”.
jitterjRange of the length of network delay time.String. The time units can be: ns, us (µs), ms, s, m, h, such as “1ms”.
latencylLength of network delay time.String. The time units can be: ns, us (μs), ms, s, m, h, such as “1ms”.
protocolpThe IP protocol impacted by traffic.String. It supports the following protocol types: tcp, udp, icmp, all (all network protocols).
source-portsThe egress traffic that only impacts specified source ports. It can only be configured when the protocol is TCP or UDP.String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.

Network duplication

You can run the following command to see the configuration of simulated network duplication using Chaosd:

  1. chaosd attack network duplicate --help

The output is as follows:

  1. duplicate network packet
  2. Usage:
  3. chaosd attack network duplicate [flags]
  4. Flags:
  5. -c, --correlation string correlation is percentage (10 is 10%) (default "0")
  6. -d, --device string the network interface to impact
  7. -e, --egress-port string only impact egress traffic to these destination ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  8. -h, --help help for duplicate
  9. -H, --hostname string only impact traffic to these hostnames
  10. -i, --ip string only impact egress traffic to these IP addresses
  11. --percent string percentage of packets to duplicate (10 is 10%) (default "1")
  12. -p, --protocol string only impact traffic using this IP protocol, supported: tcp, udp, icmp, all
  13. -s, --source-port string only impact egress traffic from these source ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  14. Global Flags:
  15. --log-level string the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'

The related configuration items are described as follows:

Configuration itemAbbreviationDescriptionValue
correlationcThe correlation between the percentage of current duplication occurrence and the previous one.Int. It is a percentage which range is 0 to 100 (10 is 10%) (default “0”).
devicedName of the impacted network interface card.String, such as “eth0”. The value is required.
egress-porteThe egress traffic that only impacts specified destination ports. It can only be configured when the protocol is TCP or UDP.String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.
hostnameHThe host name impacted by traffic.String, such as “chaos-mesh.org”.
ipiThe IP address impacted by egress traffic.String, such as “123.123.123.123”.
percentnoneRatio of network packet duplicate.Int. It is a percentage which range is 0 to 100 (10 is 10%) (default “1”).
protocolpThe IP protocol impacted by traffic.String. It supports the following protocol types: tcp, udp, icmp, all (all network protocols).
source-portsThe egress traffic which only impact specific source ports. It can only be configured when the protocol is tcp or udp.String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.

Network loss

You can run the following command to see the configuration of simulated network loss using Chaosd:

  1. chaosd attack network loss --help

The output is as follows:

  1. loss network packet
  2. Usage:
  3. chaosd attack network loss [flags]
  4. Flags:
  5. -c, --correlation string correlation is percentage (10 is 10%) (default "0")
  6. -d, --device string the network interface to impact
  7. -e, --egress-port string only impact egress traffic to these destination ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  8. -h, --help help for loss
  9. -H, --hostname string only impact traffic to these hostnames
  10. -i, --ip string only impact egress traffic to these IP addresses
  11. --percent string percentage of packets to drop (10 is 10%) (default "1")
  12. -p, --protocol string only impact traffic using this IP protocol, supported: tcp, udp, icmp, all
  13. -s, --source-port string only impact egress traffic from these source ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  14. Global Flags:
  15. --log-level string the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'

The related configuration items are described as follows:

Configuration itemAbbreviationDescriptionValue
correlationcThe correlation between the percentage of the current network loss and the previous one.Int. It is a percentage which range is 0 to 100 (10 is 10%) (default “0”).
devicedName of the impacted network interface card.String, such as “eth0”. The value is required.
egress-porteThe egress traffic that only impacts specified destination ports. It can only be configured when the protocol is TCP or UDP.String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.
hostnameHThe host name impacted by traffic.String, such as “chaos-mesh.org”.
ipiThe IP address impacted by egress traffic.String, such as “123.123.123.123”.
percentnoneRatio of network packet loss.Int. It is a percentage which range is 0 to 100 (10 is 10%) (default “1”).
protocolpOnly impact traffic using this IP protocol.String. It supports the following protocol types: tcp, udp, icmp, all (all network protocols).
source-portsThe egress traffic which only impact specific source ports. It can only be configured when the protocol is tcp or udp.String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.

Examples

Simulate network corruption:

  1. chaosd attack network corrupt -d eth0 -i 172.16.4.4 --percent 50

The output is as follows:

  1. Attack network successfully, uid: 4eab1e62-8d60-45cb-ac85-3c17b8ac4825

Simulate network latency:

  1. chaosd attack network delay -d eth0 -i 172.16.4.4 -l 10ms

The output is as follows:

  1. Attack network successfully, uid: 4b23a0b5-e193-4b27-90a7-3e04235f32ab

Simulate network duplication:

  1. chaosd attack network duplicate -d eth0 -i 172.16.4.4 --percent 50

The output is as follows:

  1. Attack network successfully, uid: 7bcb74ee-9101-4ae4-82f0-e44c8a7f113c

Simulate network loss:

  1. chaosd attack network loss -d eth0 -i 172.16.4.4 --percent 50

The output is as follows:

  1. Attack network successfully, uid: 1e818adf-3942-4de4-949b-c8499f120265

When running the experiments, remember to save the uid of the experiments. To end a network fault simulation, use recover:

  1. chaosd recover 1e818adf-3942-4de4-949b-c8499f120265

The output is as follows:

  1. Recover 1e818adf-3942-4de4-949b-c8499f120265 successfully

Create network fault experiments using service mode

(To be added)