Simulate JVM Application Faults

Chaosd simulates the faults of JVM application through Byteman. The supported fault types are as follows:

  • Throw custom exceptions
  • Trigger garbage collection
  • Increase method latency
  • Modify return values of a method
  • Trigger faults by setting Byteman configuration files
  • Increase JVM pressure

This document describes how to use Chaosd to create the above fault types of JVM experiments.

Create experiments using the command-line mode

This section introduces how to create the experiments of JVM application faults using the command-line mode.

Before creating the experiment, you can run the following command line to see the types of JVM application faults supported by Chaosd:

  1. chaosd attack jvm -h

The result is as follows:

  1. JVM attack related commands
  2. Usage:
  3. chaosd attack jvm [command]
  4. Available Commands:
  5. exception throw specified exception for specified method
  6. gc trigger GC for JVM
  7. latency inject latency to specified method
  8. return return specified value for specified method
  9. rule-file inject fault with configured byteman rule file
  10. stress inject stress to JVM
  11. Flags:
  12. -h, --help help for jvm
  13. --pid int the pid of Java process which needs to attach
  14. --port int the port of agent server (default 9288)
  15. Global Flags:
  16. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  17. --uid string the experiment ID
  18. Use "chaosd attack jvm [command] --help" for more information about a command.

Throw custom exceptions using the command-line mode

Commands for throwing custom exceptions

To see the usage and configuration items of the command that throws custom exceptions, run the following command:

  1. chaosd attack jvm exception --help

The result is as follows:

  1. throw specified exception for specified method
  2. Usage:
  3. chaosd attack jvm exception [options] [flags]
  4. Flags:
  5. -c, --class string Java class name
  6. --exception string the exception which needs to throw for action 'exception'
  7. -h, --help help for exception
  8. -m, --method string the method name in Java class
  9. Global Flags:
  10. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  11. --pid int the pid of Java process which needs to attach
  12. --port int the port of agent server (default 9288)
  13. --uid string the experiment ID

Configuration description for throwing custom exceptions

Configuration itemAbbreviationDescriptionValue
classcThe name of the Java classstring type, required
exceptionNoneThe thrown custom exceptionstring type, required
methodmThe name of the methodstring type, required to be configured
pidNoneThe Java process ID where the fault is to be injectedint type, required
portNoneThe port number attached to the Java process agent. The fault is injected into the Java process through this port number.int type. The default value is 9288.
uidNoneThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for throwing custom exceptions

  1. chaosd attack jvm exception -c Main -m sayhello --exception 'java.io.IOException("BOOM")' --pid 30045

The result is as follows:

  1. [2021/08/05 02:39:39.106 +00:00] [INFO] [jvm.go:208] ["byteman rule"] [rule="\nRULE Main-sayhello-exception-q6nd0\nCLASS Main\nMETHOD sayhello\nAT ENTRY\nIF true\nDO \n\tthrow new java.io.IOException(\"BOOM\");\nENDRULE\n"] [file=/tmp/rule.btm296930759]
  2. Attack jvm successfully, uid: 26a45ae2-d395-46f5-a126-2b2c6c85ae9d

Trigger garbage collection using the command-line mode

Commands for triggering garbage collection

To see the usage and configuration items of the command that triggers garbage collection, run the following command:

  1. chaosd attack jvm gc --help
  1. trigger GC for JVM
  2. Usage:
  3. chaosd attack jvm gc [flags]
  4. Flags:
  5. -h, --help help for gc
  6. Global Flags:
  7. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  8. --pid int the pid of Java process which needs to attach
  9. --port int the port of agent server (default 9288)
  10. --uid string the experiment ID

Configuration description for triggering garbage collection

Configuration itemAbbreviationDescriptionValue
pidNoneThe Java process ID where the fault is to be injectedint type, required
portNoneThe port number attached to the Java process agent. The fault is injected into the Java process through this port number.int type. The default value is 9288.
uidNoneThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for triggering garbage collection

  1. chaosd attack jvm gc --pid 89345

The result is as follows:

  1. [2021/08/05 02:49:47.850 +00:00] [INFO] [jvm.go:208] ["byteman rule"] [rule="\nRULE --gc-u0mlf\nGC\nENDRULE\n"] [file=/tmp/rule.btm012481052]
  2. Attack jvm successfully, uid: f360e70a-5359-49b6-8526-d7e0a3c6f696

Triggering garbage collection is a one-time operation, and the experiment does not require recovery.

Increase method latency using the command-line mode

Commands for increasing method latency

To see the usage and configuration items of the command that increases method latency, run the following command:

  1. chaosd attack jvm latency --help

The result is as follows:

  1. inject latency to specified method
  2. Usage:
  3. chaosd attack jvm latency [options] [flags]
  4. Flags:
  5. -c, --class string Java class name
  6. -h, --help help for latency
  7. --latency int the latency duration, unit ms
  8. -m, --method string the method name in Java class
  9. Global Flags:
  10. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  11. --pid int the pid of Java process which needs to attach
  12. --port int the port of agent server (default 9288)
  13. --uid string the experiment ID

Configuration description for increasing method latency

Configuration itemAbbreviationDescriptionValue
classcThe name of the Java classstring type, required
latencyNoneThe duration of increasing method latencyint type, required. The unit is milisecond.
methodmThe name of the methodstring type, required
pidNoneThe Java process ID where the fault is to be injectedint type, required
portNoneThe port number attached to the Java process agent. The fault is injected into the Java process through this port number.int type. The default value is 9288.
uidNoneThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for increasing method latency

  1. chaosd attack jvm latency --class Main --method sayhello --latency 5000 --pid 100840

The result is as follows:

  1. [2021/08/05 03:08:50.716 +00:00] [INFO] [jvm.go:208] ["byteman rule"] [rule="\nRULE Main-sayhello-latency-hlib2\nCLASS Main\nMETHOD sayhello\nAT ENTRY\nIF true\nDO \n\tThread.sleep(5000);\nENDRULE\n"] [file=/tmp/rule.btm359997255]
  2. [2021/08/05 03:08:51.155 +00:00] [INFO] [jvm.go:94] ["submit rules"] [output="install rule Main-sayhello-latency-hlib2\n\n"]
  3. Attack jvm successfully, uid: bbe00c57-ac9d-4113-bf0c-2a6f184be261

Modify return values of a method using the command-line mode

Commands for modifying return values of a method

To see the usage and configuration items of the command that modifies return values of a method, run the following command:

  1. chaosd attack jvm return --help
  1. return specified value for specified method
  2. Usage:
  3. chaosd attack jvm return [options] [flags]
  4. Flags:
  5. -c, --class string Java class name
  6. -h, --help help for return
  7. -m, --method string the method name in Java class
  8. --value string the return value for action 'return'. Only supports number and string types.
  9. Global Flags:
  10. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  11. --pid int the pid of Java process which needs to attach
  12. --port int the port of agent server (default 9288)
  13. --uid string the experiment ID

Configuration description for modifying return values of a method

Configuration itemAbbreviationDescriptionValue
classcThe name of the Java classstring type, required to be configured
methodmThe name of the methodstring type, required to be configured
valueNoneSpecifies the return value of the methodstring type, required to be configured. Currently, the item can be numeric and string types. If the item (return value) is string, double quotes are required, like “chaos”.
pidNoneThe Java process ID where the fault is needed to be injectedint type, required to be configured
portNoneThe port number attached to the Java process agent. The faults is injected into the Java process through this port number.int type. The default value is 9288.
uidNoneThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for simulating the scenario of modifying return values of a method

  1. chaosd attack jvm return --class Main --method getnum --value 999 --pid 112694

The result is as follows:

  1. [2021/08/05 03:35:10.603 +00:00] [INFO] [jvm.go:208] ["byteman rule"] [rule="\nRULE Main-getnum-return-i6gb7\nCLASS Main\nMETHOD getnum\nAT ENTRY\nIF true\nDO \n\treturn 999;\nENDRULE\n"] [file=/tmp/rule.btm051982059]
  2. [2021/08/05 03:35:10.820 +00:00] [INFO] [jvm.go:94] ["submit rules"] [output="install rule Main-getnum-return-i6gb7\n\n"]
  3. Attack jvm successfully, uid: e2f204f6-4bed-4d92-aade-2b4a47b02e5d

Trigger faults by setting Byteman configuration files using the command-line mode

You can set the fault rules in the Byteman rule configuration file, and then inject the faults by specifying the path of the configuration file using Chaosd. Regarding the Byteman rule configuration, refer to byteman-rule-language.

Commands for triggering faults by setting Byteman configuration files

To see the usage and configuration items of the command that triggers faults by setting Byteman configuration files, run the following command:

  1. chaosd attack jvm rule-file --help

The result is as follows:

  1. inject fault with configured byteman rule file
  2. Usage:
  3. chaosd attack jvm rule-file [options] [flags]
  4. Flags:
  5. -h, --help help for rule-file
  6. -p, --path string the path of configured byteman rule file
  7. Global Flags:
  8. --log-level string the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'
  9. --pid int the pid of Java process which needs to attach
  10. --port int the port of agent server (default 9288)
  11. --uid string the experiment ID

Configuration description for triggering faults by setting Byteman configuration files

Configuration itemAbbreviationDescriptionValue
pathNoneSpecifies the path of the Byteman configuration filestring type, required
pidNoneThe Java process ID where the fault is to be injectedint type, required
portNoneThe port number attached to the Java process agent. The fault is injected into the Java process through this port number.int type. The default value is 9288.
uidNoneThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for triggering faults by setting Byteman configuration files

First, based on the specific Java program and referring to the Byteman rule language, write a rule configuration file. For example:

  1. RULE modify return value
  2. CLASS Main
  3. METHOD getnum
  4. AT ENTRY
  5. IF true
  6. DO
  7. return 9999
  8. ENDRULE

Then, save the configuration file to the return.btm file. After that, run the following command to inject faults.

  1. chaosd attack jvm rule-file -p ./return.btm --pid 112694

The result is as follows:

  1. [2021/08/05 03:45:40.757 +00:00] [INFO] [jvm.go:152] ["rule file data:RULE modify return value\nCLASS Main\nMETHOD getnum\nAT ENTRY\nIF true\nDO\n return 9999\nENDRULE\n"]
  2. [2021/08/05 03:45:41.011 +00:00] [INFO] [jvm.go:94] ["submit rules"] [output="install rule modify return value\n\n"]
  3. Attack jvm successfully, uid: 5ca2e06d-a7c6-421d-bb67-0c9908bac17a

Increase JVM stress using the command-line mode

Commands for increasing JVM stress

To see the usage and configuration items of the command that increases JVM stress, run the following command:

  1. chaosd attack jvm stress --help

The result is as follows:

  1. inject stress to JVM
  2. Usage:
  3. chaosd attack jvm stress [options] [flags]
  4. Flags:
  5. --cpu-count int the CPU core number
  6. -h, --help help for stress
  7. --mem-type int the memory type to be allocated. The value can be 'stack' or 'heap'.
  8. Global Flags:
  9. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  10. --pid int the pid of Java process which needs to attach
  11. --port int the port of agent server (default 9288)
  12. --uid string the experiment ID

Configuration description for increasing JVM stress

Configuration itemAbbreviationDescriptionValue
cpu-countNoneThe number of CPU cores used for increasing JVM stressint type. You must configure one item between cpu-count and mem-type.
mem-typeNoneThe type of OOMstring type. Currently, both ‘stack’ and ‘heap’ OOM types are supported. You must configure one item between cpu-count and mem-type.
pidNoneThe Java process ID where the fault is to be injectedint type, required
portNoneThe port number attached to the Java process agent. The fault is injected into the Java process through this port number.int type. The default value is 9288.
uidNoneThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for increasing JVM stress

  1. chaosd attack jvm stress --cpu-count 2 --pid 123546

The result is as follows:

  1. [2021/08/05 03:59:51.256 +00:00] [INFO] [jvm.go:208] ["byteman rule"] [rule="\nRULE --stress-jfeiu\nSTRESS CPU\nCPUCOUNT 2\nENDRULE\n"] [file=/tmp/rule.btm773062009]
  2. [2021/08/05 03:59:51.613 +00:00] [INFO] [jvm.go:94] ["submit rules"] [output="install rule --stress-jfeiu\n\n"]
  3. Attack jvm successfully, uid: b9b997b5-0a0d-4f1f-9081-d52a32318b84

Create experiments using the service mode

You can follow the instructions below to create experiments using the service mode.

  1. Execute Chaosd in service mode:

    1. chaosd server --port 31767
  2. Send HTTP POST request to the /api/attack/{uid} path of Chaosd service.

    1. curl -X POST 172.16.112.130:31767/api/attack/jvm -H "Content-Type:application/json" -d '{fault-configuration}'

    For the fault-configuration part in the above command, you need to configure it according to the fault types. For the corresponding parameters, refer to the parameters and examples of each fault type in the following sections.

note

When running an experiment, remember to save the UID information of the experiment. When you want to end the experiment corresponding to the UID, you need to send an HTTP DELETE request to the /api/attack/{uid} path of Chaosd service.

Throw custom exceptions using the service mode

Parameters for throwing custom exceptions

ParameterDescriptionValue
actionThe action of the experimentSet to “exception”
classThe name of the Java classstring type, required
exceptionThe thrown custom exceptionstring type, required
methodThe name of the methodstring type, required
pidThe Java process ID where the fault is to be injectedint type, required
portThe port number attached to the Java process agent. The faults is injected into the Java process through this port number.int type. The default value is 9288.
uidThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for throwing custom exceptions using the service mode

  1. curl -X POST 172.16.112.130:31767/api/attack/jvm -H "Content-Type:application/json" -d '{"action":"exception","class":"Main","method":"sayhello","exception":"java.io.IOException(\"BOOM\")","pid":1828622}'

The result is as follows:

  1. {"status":200,"message":"attack successfully","uid":"c3c519bf-819a-4a7b-97fb-e3d0814481fa"}

Trigger garbage collection using service mode

Parameters for triggering garbage collection

ParameterDescriptionValue
actionThe action of the experimentSet to “gc”
pidThe Java process ID where the fault is to be injectedint type, required
portThe port number attached to the Java process agent. The fault is injected into the Java process through this port number.int type. The default value is 9288.
uidThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for triggering garbage collection using the service mode

  1. curl -X POST 172.16.112.130:31767/api/attack/jvm -H "Content-Type:application/json" -d '{"action":"gc","pid":1828622}'

The result is as follows:

  1. {"status":200,"message":"attack successfully","uid":"c3c519bf-819a-4a7b-97fb-e3d0814481fa"}

Triggering garbage collection is a one-time operation. The experiment does not require recovery.

Increase method latency using service mode

Parameters for increasing method latency

ParameterDescriptionValue
actionThe action of the experimentSet to “latency”
classThe name of the Java classstring type, required
latencyThe duration of increasing method latencyint type, required. The unit is milisecond.
methodThe name of the methodstring type, required
pidThe Java process ID where the fault is to be injectedint type, required
portThe Java process ID where the fault is needed to be injectedint type, required
uidThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for increasing method latency using the service mode

  1. curl -X POST 172.16.112.130:31767/api/attack/jvm -H "Content-Type:application/json" -d '{"action":"latency","class":"Main","method":"sayhello","latency":5000,"pid":1828622}'

The result is as follows:

  1. {"status":200,"message":"attack successfully","uid":"a551206c-960d-4ac5-9056-518e512d4d0d"}

Modify return values of a method using service mode

Parameters for modifying return values of a method

ParameterDescriptionValue
actionThe action of the experimentSet to “return”
classThe name of the Java classstring type, required
methodThe name of the methodstring type, required
valueSpecifies the return value of the methodstring type, required. Currently, the item can be numeric and string types. If the item (return value) is string, double quotes are required, like “chaos”.
pidThe Java process ID where the fault is to be injectedint type, required
portThe port number attached to the Java process agent. The fault is injected into the Java process through this port number.int type. The default value is 9288.
uidThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for modifying return values of a method using the service mode

  1. curl -X POST 172.16.112.130:31767/api/attack/jvm -H "Content-Type:application/json" -d '{"action":"return","class":"Main","method":"getnum","value":"999","pid":1828622}'

The result is as follows:

  1. {"status":200,"message":"attack successfully","uid":"a551206c-960d-4ac5-9056-518e512d4d0d"}

Trigger faults by setting Byteman configuration files using service mode

You can set the fault rules according to the Byteman rule configuration. Regarding to the Byteman rule configuration, refer to byteman-rule-language.

Parameters for triggering faults by setting Byteman configuration files

ParameterDescriptionValue
actionThe action of the experimentSet to “rule-data”
rule-dataSpecifies the Byteman configuration datastring type, required
pidThe Java process ID where the fault is to be injectedint type, required
portThe port number attached to the Java process agent. The fault is injected into the Java process through this port number.int type. The default value is 9288.
uidThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for triggering faults by setting Byteman configuration files using the service mode

First, based on the specific Java program and referring to the Byteman rule language, write a rule configuration file. For example:

  1. RULE modify return value
  2. CLASS Main
  3. METHOD getnum
  4. AT ENTRY
  5. IF true
  6. DO
  7. return 9999
  8. ENDRULE

Then, escape the line breaks in the configuration file to the newline character “\n”, and use the escaped text as the value of “rule-data”. Run the following command:

  1. curl -X POST 127.0.0.1:31767/api/attack/jvm -H "Content-Type:application/json" -d '{"action":"rule-data","pid":30045,"rule-data":"\nRULE modify return value\nCLASS Main\nMETHOD getnum\nAT ENTRY\nIF true\nDO return 9999\nENDRULE\n"}'

The result is as follows:

  1. {"status":200,"message":"attack successfully","uid":"a551206c-960d-4ac5-9056-518e512d4d0d"}

Increase JVM stress using the service mode

Parameters for increasing JVM stress

ParameterDescriptionValue
actionThe action of the experimentSet to “stress”
cpu-countThe number of CPU cores used for increasing CPU stressint type. You must configure one item between cpu-count and mem-type.
mem-typeThe type of OOMstring type. Currently, both ‘stack’ and ‘heap’ OOM types are supported. You must configure one item between cpu-count and mem-type.
pidNoneThe Java process ID where the fault is to be injectedint type, required
portNoneThe port number attached to the Java process agent. The fault is injected into the Java process through this port number.int type. The default value is 9288.
uidNoneThe experiment IDstring type. This item is not required to be configured, because Chaosd randomly creates one.

Example for increasing JVM stress using the service mode

  1. curl -X POST 172.16.112.130:31767/api/attack/jvm -H "Content-Type:application/json" -d '{"action":"stress","cpu-count":1,"pid":1828622}'

The result is as follows:

  1. {"status":200,"message":"attack successfully","uid":"a551206c-960d-4ac5-9056-518e512d4d0d"}