Jepsen Testing

Jepsen is a tool, written by Kyle Kingsbury, designed to test the partition tolerance of distributed systems. It creates network partitions while fuzzing the system with random operations. The results are analyzed to see if the system violates any of the consistency properties it claims to have.

As part of our Consul testing, we ran a Jepsen test to determine if any consistency issues could be uncovered. In our testing, Consul gracefully recovered from partitions without introducing any consistency issues.

Running the tests

At the moment, testing with Jepsen is rather complex as it requires setting up multiple virtual machines, SSH keys, DNS configuration, and a working Clojure environment. We hope to contribute our Consul testing code upstream and to provide a Vagrant environment for Jepsen testing soon.

Output

Below is the output captured from Jepsen. We ran Jepsen multiple times, and it passed each time. This output is only representative of a single run and has been edited for length. Please reach out on Consul’s Discuss if you would like to reproduce the Jepsen results.

  1. $ lein test :only jepsen.system.consul-test
  2. lein test jepsen.system.consul-test
  3. INFO jepsen.os.debian - :n5 setting up debian
  4. INFO jepsen.os.debian - :n3 setting up debian
  5. INFO jepsen.os.debian - :n4 setting up debian
  6. INFO jepsen.os.debian - :n1 setting up debian
  7. INFO jepsen.os.debian - :n2 setting up debian
  8. INFO jepsen.os.debian - :n4 debian set up
  9. INFO jepsen.os.debian - :n5 debian set up
  10. INFO jepsen.os.debian - :n3 debian set up
  11. INFO jepsen.os.debian - :n1 debian set up
  12. INFO jepsen.os.debian - :n2 debian set up
  13. INFO jepsen.system.consul - :n1 consul nuked
  14. INFO jepsen.system.consul - :n4 consul nuked
  15. INFO jepsen.system.consul - :n5 consul nuked
  16. INFO jepsen.system.consul - :n3 consul nuked
  17. INFO jepsen.system.consul - :n2 consul nuked
  18. INFO jepsen.system.consul - Running nodes: {:n1 false, :n2 false, :n3 false, :n4 false, :n5 false}
  19. INFO jepsen.system.consul - :n2 consul nuked
  20. INFO jepsen.system.consul - :n3 consul nuked
  21. INFO jepsen.system.consul - :n4 consul nuked
  22. INFO jepsen.system.consul - :n5 consul nuked
  23. INFO jepsen.system.consul - :n1 consul nuked
  24. INFO jepsen.system.consul - :n1 starting consul
  25. INFO jepsen.system.consul - :n2 starting consul
  26. INFO jepsen.system.consul - :n4 starting consul
  27. INFO jepsen.system.consul - :n5 starting consul
  28. INFO jepsen.system.consul - :n3 starting consul
  29. INFO jepsen.system.consul - :n3 consul ready
  30. INFO jepsen.system.consul - :n2 consul ready
  31. INFO jepsen.system.consul - Running nodes: {:n1 true, :n2 true, :n3 true, :n4 true, :n5 true}
  32. INFO jepsen.system.consul - :n5 consul ready
  33. INFO jepsen.system.consul - :n1 consul ready
  34. INFO jepsen.system.consul - :n4 consul ready
  35. INFO jepsen.core - Worker 0 starting
  36. INFO jepsen.core - Worker 2 starting
  37. INFO jepsen.core - Worker 1 starting
  38. INFO jepsen.core - Worker 3 starting
  39. INFO jepsen.core - Worker 4 starting
  40. INFO jepsen.util - 2 :invoke :read nil
  41. INFO jepsen.util - 3 :invoke :cas [4 4]
  42. INFO jepsen.util - 0 :invoke :write 4
  43. INFO jepsen.util - 1 :invoke :write 1
  44. INFO jepsen.util - 4 :invoke :cas [4 0]
  45. INFO jepsen.util - 2 :ok :read nil
  46. INFO jepsen.util - 4 :fail :cas [4 0]
  47. (Log Truncated...)
  48. INFO jepsen.util - 4 :invoke :cas [3 3]
  49. INFO jepsen.util - 4 :fail :cas [3 3]
  50. INFO jepsen.util - :nemesis :info :stop nil
  51. INFO jepsen.util - :nemesis :info :stop "fully connected"
  52. INFO jepsen.util - 0 :fail :read nil
  53. INFO jepsen.util - 1 :fail :write 0
  54. INFO jepsen.util - :nemesis :info :stop nil
  55. INFO jepsen.util - :nemesis :info :stop "fully connected"
  56. INFO jepsen.core - nemesis done
  57. INFO jepsen.core - Worker 3 done
  58. INFO jepsen.util - 1 :invoke :read nil
  59. INFO jepsen.core - Worker 2 done
  60. INFO jepsen.core - Worker 4 done
  61. INFO jepsen.core - Worker 0 done
  62. INFO jepsen.util - 1 :ok :read 3
  63. INFO jepsen.core - Worker 1 done
  64. INFO jepsen.core - Run complete, writing
  65. INFO jepsen.core - Analyzing
  66. (Log Truncated...)
  67. INFO jepsen.core - Analysis complete
  68. INFO jepsen.system.consul - :n3 consul nuked
  69. INFO jepsen.system.consul - :n2 consul nuked
  70. INFO jepsen.system.consul - :n4 consul nuked
  71. INFO jepsen.system.consul - :n1 consul nuked
  72. INFO jepsen.system.consul - :n5 consul nuked
  73. 1964 element history linearizable. :D
  74. Ran 1 tests containing 1 assertions.
  75. 0 failures, 0 errors.
  1. $ lein test :only jepsen.system.consul-test
  2. lein test jepsen.system.consul-test
  3. INFO jepsen.os.debian - :n5 setting up debian
  4. INFO jepsen.os.debian - :n3 setting up debian
  5. INFO jepsen.os.debian - :n4 setting up debian
  6. INFO jepsen.os.debian - :n1 setting up debian
  7. INFO jepsen.os.debian - :n2 setting up debian
  8. INFO jepsen.os.debian - :n4 debian set up
  9. INFO jepsen.os.debian - :n5 debian set up
  10. INFO jepsen.os.debian - :n3 debian set up
  11. INFO jepsen.os.debian - :n1 debian set up
  12. INFO jepsen.os.debian - :n2 debian set up
  13. INFO jepsen.system.consul - :n1 consul nuked
  14. INFO jepsen.system.consul - :n4 consul nuked
  15. INFO jepsen.system.consul - :n5 consul nuked
  16. INFO jepsen.system.consul - :n3 consul nuked
  17. INFO jepsen.system.consul - :n2 consul nuked
  18. INFO jepsen.system.consul - Running nodes: {:n1 false, :n2 false, :n3 false, :n4 false, :n5 false}
  19. INFO jepsen.system.consul - :n2 consul nuked
  20. INFO jepsen.system.consul - :n3 consul nuked
  21. INFO jepsen.system.consul - :n4 consul nuked
  22. INFO jepsen.system.consul - :n5 consul nuked
  23. INFO jepsen.system.consul - :n1 consul nuked
  24. INFO jepsen.system.consul - :n1 starting consul
  25. INFO jepsen.system.consul - :n2 starting consul
  26. INFO jepsen.system.consul - :n4 starting consul
  27. INFO jepsen.system.consul - :n5 starting consul
  28. INFO jepsen.system.consul - :n3 starting consul
  29. INFO jepsen.system.consul - :n3 consul ready
  30. INFO jepsen.system.consul - :n2 consul ready
  31. INFO jepsen.system.consul - Running nodes: {:n1 true, :n2 true, :n3 true, :n4 true, :n5 true}
  32. INFO jepsen.system.consul - :n5 consul ready
  33. INFO jepsen.system.consul - :n1 consul ready
  34. INFO jepsen.system.consul - :n4 consul ready
  35. INFO jepsen.core - Worker 0 starting
  36. INFO jepsen.core - Worker 2 starting
  37. INFO jepsen.core - Worker 1 starting
  38. INFO jepsen.core - Worker 3 starting
  39. INFO jepsen.core - Worker 4 starting
  40. INFO jepsen.util - 2 :invoke :read nil
  41. INFO jepsen.util - 3 :invoke :cas [4 4]
  42. INFO jepsen.util - 0 :invoke :write 4
  43. INFO jepsen.util - 1 :invoke :write 1
  44. INFO jepsen.util - 4 :invoke :cas [4 0]
  45. INFO jepsen.util - 2 :ok :read nil
  46. INFO jepsen.util - 4 :fail :cas [4 0]
  47. (Log Truncated...)
  48. INFO jepsen.util - 4 :invoke :cas [3 3]
  49. INFO jepsen.util - 4 :fail :cas [3 3]
  50. INFO jepsen.util - :nemesis :info :stop nil
  51. INFO jepsen.util - :nemesis :info :stop "fully connected"
  52. INFO jepsen.util - 0 :fail :read nil
  53. INFO jepsen.util - 1 :fail :write 0
  54. INFO jepsen.util - :nemesis :info :stop nil
  55. INFO jepsen.util - :nemesis :info :stop "fully connected"
  56. INFO jepsen.core - nemesis done
  57. INFO jepsen.core - Worker 3 done
  58. INFO jepsen.util - 1 :invoke :read nil
  59. INFO jepsen.core - Worker 2 done
  60. INFO jepsen.core - Worker 4 done
  61. INFO jepsen.core - Worker 0 done
  62. INFO jepsen.util - 1 :ok :read 3
  63. INFO jepsen.core - Worker 1 done
  64. INFO jepsen.core - Run complete, writing
  65. INFO jepsen.core - Analyzing
  66. (Log Truncated...)
  67. INFO jepsen.core - Analysis complete
  68. INFO jepsen.system.consul - :n3 consul nuked
  69. INFO jepsen.system.consul - :n2 consul nuked
  70. INFO jepsen.system.consul - :n4 consul nuked
  71. INFO jepsen.system.consul - :n1 consul nuked
  72. INFO jepsen.system.consul - :n5 consul nuked
  73. 1964 element history linearizable. :D
  74. Ran 1 tests containing 1 assertions.
  75. 0 failures, 0 errors.