crimson

Crimson is the code name of crimson-osd, which is the next generation ceph-osd.It targets fast networking devices, fast storage devices by leveraging state ofthe art technologies like DPDK and SPDK, for better performance. And it willkeep the support of HDDs and low-end SSDs via BlueStore. Crismon will try tobe backward compatible with classic OSD.

Running Crimson

As you might expect, crimson is not featurewise on par with its predecessor yet.

daemonize

Unlike ceph-osd, crimson-osd does daemonize itself even if thedaemonize option is enabled. Because, to read this option, crimson-osdneeds to ready its config sharded service, but this sharded service livesin the seastar reactor. If we fork a child process and exit the parent afterstarting the Seastar engine, that will leave us with a single thread which isthe replica of the thread calls fork(). This would unnecessarily complicatethe code, if we would have tackled this problem in crimson.

Since a lot of GNU/Linux distros are using systemd nowadays, which is able todaemonize the application, there is no need to daemonize by ourselves. Forthose who are using sysvinit, they can use start-stop-daemon for daemonizingcrimson-osd. If this is not acceptable, we can whip up a helper utilityto do the trick.

logging

Currently, crimson-osd uses the logging utility offered by Seastar. seesrc/common/dout.h for the mapping between different logging levels tothe severity levels in Seastar. For instance, the messages sent to derrwill be printed using logger::error(), and the messages with debug levelover 20 will be printed using logger::trace().

cephseastar
< 0error
0warn
[1, 5)info
[5, 20]debug
> 20trace

Please note, crimson-osddoes not send the logging message to specified log_file. It writesthe logging messages to stdout and/or syslog. Again, this behavior can bechanged using —log-to-stdout and —log-to-syslog command lineoptions. By default, log-to-stdout is enabled, and the latter disabled.

vstart.sh

To facilitate the development of crimson, following options would be handy whenusing vstart.sh,

  • —crimson

  • start crimson-osd instead of ceph-osd

  • —nodaemon

  • do not daemonize the service

  • —redirect-output

  • redirect the stdout and stderr of service to out/$type.$num.stdout.

  • —osd-args

  • pass extra command line options to crimson-osd or ceph-osd. It’s quiteuseful for passing Seastar options to crimson-osd.

So, a typical command to start a single-crimson-node cluster is:

  1. MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n -x --without-dashboard --memstore \
  2.   --crimson --nodaemon --redirect-output \
  3.   --osd-args "--memory 4G --smp 1 --cpuset 0"

Where we assign 4 GiB memory, a single thread running on core-0 to crimson-osd.Please refer crimson-osd —help-seastar for more Seastar specific commandline options.

You could stop the vstart cluster using:

  1. ../src/stop.sh --crimson

CBT Based Testing

We can use cbt for performing perf tests:

  1. $ git checkout master
  2. $ make crimson-osd
  3. $ ../src/script/run-cbt.sh --cbt ~/dev/cbt -a /tmp/baseline ../src/test/crimson/cbt/radosbench_4K_read.yaml
  4. $ git checkout yet-another-pr
  5. $ make crimson-osd
  6. $ ../src/script/run-cbt.sh --cbt ~/dev/cbt -a /tmp/yap ../src/test/crimson/cbt/radosbench_4K_read.yaml
  7. $ ~/dev/cbt/compare.py -b /tmp/baseline -a /tmp/yap -v
  8. 19:48:23 - INFO     - cbt      - prefill/gen8/0: bandwidth: (or (greater) (near 0.05)):: 0.183165/0.186155  => accepted
  9. 19:48:23 - INFO     - cbt      - prefill/gen8/0: iops_avg: (or (greater) (near 0.05)):: 46.0/47.0  => accepted
  10. 19:48:23 - WARNING  - cbt      - prefill/gen8/0: iops_stddev: (or (less) (near 0.05)):: 10.4403/6.65833  => rejected
  11. 19:48:23 - INFO     - cbt      - prefill/gen8/0: latency_avg: (or (less) (near 0.05)):: 0.340868/0.333712  => accepted
  12. 19:48:23 - INFO     - cbt      - prefill/gen8/1: bandwidth: (or (greater) (near 0.05)):: 0.190447/0.177619  => accepted
  13. 19:48:23 - INFO     - cbt      - prefill/gen8/1: iops_avg: (or (greater) (near 0.05)):: 48.0/45.0  => accepted
  14. 19:48:23 - INFO     - cbt      - prefill/gen8/1: iops_stddev: (or (less) (near 0.05)):: 6.1101/9.81495  => accepted
  15. 19:48:23 - INFO     - cbt      - prefill/gen8/1: latency_avg: (or (less) (near 0.05)):: 0.325163/0.350251  => accepted
  16. 19:48:23 - INFO     - cbt      - seq/gen8/0: bandwidth: (or (greater) (near 0.05)):: 1.24654/1.22336  => accepted
  17. 19:48:23 - INFO     - cbt      - seq/gen8/0: iops_avg: (or (greater) (near 0.05)):: 319.0/313.0  => accepted
  18. 19:48:23 - INFO     - cbt      - seq/gen8/0: iops_stddev: (or (less) (near 0.05)):: 0.0/0.0  => accepted
  19. 19:48:23 - INFO     - cbt      - seq/gen8/0: latency_avg: (or (less) (near 0.05)):: 0.0497733/0.0509029  => accepted
  20. 19:48:23 - INFO     - cbt      - seq/gen8/1: bandwidth: (or (greater) (near 0.05)):: 1.22717/1.11372  => accepted
  21. 19:48:23 - INFO     - cbt      - seq/gen8/1: iops_avg: (or (greater) (near 0.05)):: 314.0/285.0  => accepted
  22. 19:48:23 - INFO     - cbt      - seq/gen8/1: iops_stddev: (or (less) (near 0.05)):: 0.0/0.0  => accepted
  23. 19:48:23 - INFO     - cbt      - seq/gen8/1: latency_avg: (or (less) (near 0.05)):: 0.0508262/0.0557337  => accepted
  24. 19:48:23 - WARNING  - cbt      - 1 tests failed out of 16

Where we compile and run the same test against two branches. One is master, another is yet-another-pr branch.And then we compare the test results. Along with every test case, a set of rules is defined to check if we haveperformance regressions when comparing two set of test results. If a possible regression is found, the rule andcorresponding test results are highlighted.

Debugging Crimson

debugging tips

When a seastar application crashes, it leaves us a serial of addresses, like:

  1. Segmentation fault.
  2. Backtrace:
  3. 0x00000000108254aa
  4. 0x00000000107f74b9
  5. 0x00000000105366cc
  6. 0x000000001053682c
  7. 0x00000000105d2c2e
  8. 0x0000000010629b96
  9. 0x0000000010629c31
  10. 0x00002a02ebd8272f
  11. 0x00000000105d93ee
  12. 0x00000000103eff59
  13. 0x000000000d9c1d0a
  14. /lib/x86_64-linux-gnu/libc.so.6+0x000000000002409a
  15. 0x000000000d833ac9
  16. Segmentation fault

seastar-addr2line offered by Seastar can be used to decipher theseaddresses. After running the script, it will be waiting for input from stdin,so we need to copy and paste the above addresses, then send the EOF by inputtingcontrol-D in the terminal:

  1. $ ../src/seastar/scripts/seastar-addr2line -e bin/crimson-osd
  2.  
  3. 0x00000000108254aa
  4. 0x00000000107f74b9
  5. 0x00000000105366cc
  6. 0x000000001053682c
  7. 0x00000000105d2c2e
  8. 0x0000000010629b96
  9. 0x0000000010629c31
  10. 0x00002a02ebd8272f
  11. 0x00000000105d93ee
  12. 0x00000000103eff59
  13. 0x000000000d9c1d0a
  14. 0x00000000108254aa
  15. [Backtrace #0]
  16. seastar::backtrace_buffer::append_backtrace() at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1136
  17. seastar::print_with_backtrace(seastar::backtrace_buffer&) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1157
  18. seastar::print_with_backtrace(char const*) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1164
  19. seastar::sigsegv_action() at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5119
  20. seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5105
  21. seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5101
  22. ?? ??:0
  23. seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5418
  24. seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /home/kefu/dev/ceph/build/../src/seastar/src/core/app-template.cc:173 (discriminator 5)
  25. main at /home/kefu/dev/ceph/build/../src/crimson/osd/main.cc:131 (discriminator 1)