Apache Doris Be development and debugging

initial preparation work

This tutorial was conducted under Ubuntu 20.04

  1. Download the doris source code

    URL：apache/incubator-doris: Apache Doris (Incubating) (github.com)BE development and debugging environment under Linux - 图1 (opens new window)

  2. Install GCC 8.3.1+, Oracle JDK 1.8+, Python 2.7+, confirm that the gcc, java, python commands point to the correct version, and set the JAVA_HOME environment variable

  3. Install other dependent packages

  1. sudo apt install build-essential openjdk-8-jdk maven cmake byacc flex automake libtool-bin bison binutils-dev libiberty-dev zip unzip libncurses5-dev curl git ninja-build python brotli
  2. sudo add-apt-repository ppa:ubuntu-toolchain-r/ppa
  3. sudo apt update
  4. sudo apt install gcc-10 g++-10 
  5. sudo apt-get install autoconf automake libtool autopoint
  1. install : libssl-dev
  1. sudo apt install -y libssl-dev

Compile

The following steps are carried out in the /home/workspace directory

  1. dowload source
  1. git clone https://github.com/apache/incubator-doris.git
  1. Compile third-party dependency packages
  1.  cd /home/workspace/incubator-doris/thirdparty
  2.  ./build-thirdparty.sh
  1. Compile doris product code
  1. cd /home/workspace/incubator-doris
  2. ./build.sh

Note: This compilation has the following instructions:

  1. ./build.sh  #Compile be and fe at the same time
  2. ./build.sh  --be #Only compile be
  3. ./build.sh  --fe #Only compilefe
  4. ./build.sh  --fe --be --clean#Delete and compile be fe at the same time
  5. ./build.sh  --fe  --clean#Delete and compile fe
  6. ./build.sh  --be  --clean#Delete and compile be
  7. ./build.sh  --be --fe  --clean#Delete and compile be fe at the same time

If nothing happens, the compilation should be successful, and the final deployment file will be output to the /home/zhangfeng/incubator-doris/output/ directory. If you still encounter other problems, you can refer to the doris installation document http://doris.apache.org.

Deployment and debugging

  1. Authorize be compilation result files
  1. chmod  /home/workspace/incubator-doris/output/be/lib/palo_be

Note: /home/workspace/incubator-doris/output/be/lib/palo_be is the executable file of be.

  1. Create a data storage directory

By viewing /home/workspace/incubator-doris/output/be/conf/be.conf

  1. # INFO, WARNING, ERROR, FATAL
  2. sys_log_level = INFO
  3. be_port = 9060
  4. be_rpc_port = 9070
  5. webserver_port = 8040
  6. heartbeat_service_port = 9050
  7. brpc_port = 8060
  9. # Note that there should at most one ip match this list.
  10. # If no ip match this rule, will choose one randomly.
  11. # use CIDR format, e.g. 10.10.10.0/
  12. # Default value is empty.
  13. priority_networks = 192.168.59.0/24 # data root path, seperate by ';'
  14. storage_root_path = /soft/be/storage 
  15. # sys_log_dir = ${PALO_HOME}/log
  16. # sys_log_roll_mode = SIZE-MB-
  17. # sys_log_roll_num =
  18. # sys_log_verbose_modules =
  19. # log_buffer_level = -
  20. # palo_cgroups

Need to create this folder, this is where the be data is stored

  1. mkdir -p /soft/be/storage

  1. Open vscode, and open the directory where the be source code is located. In this case, open the directory as /home/workspace/incubator-doris/，For details on how to vscode, refer to the online tutorial

  2. Install the vscode ms c++ debugging plug-in, the plug-in identified by the red box in the figure below

BE development and debugging environment under Linux - 图2

  1. Create a launch.json file, the content of the file is as follows:
  1. {
  2.     "version": "0.2.0",
  3.     "configurations": [
  4.         {
  5.             "name": "(gdb) Launch",
  6.             "type": "cppdbg",
  7.             "request": "launch",
  8.             "program": "/home/workspace/incubator-doris/output/be/lib/palo_be",
  9.             "args": [],
  10.             "stopAtEntry": false,
  11.             "cwd": "/home/workspace/incubator-doris/",
  12.             "environment": [{"name":"PALO_HOME","value":"/home/workspace/incubator-doris/output/be/"},
  13.                             {"name":"UDF_RUNTIME_DIR","value":"/home/workspace/incubator-doris/output/be/lib/udf-runtime"},
  14.                             {"name":"LOG_DIR","value":"/home/workspace/incubator-doris/output/be/log"},
  15.                             {"name":"PID_DIR","value":"/home/workspace/incubator-doris/output/be/bin"}
  16.                            ],
  17.             "externalConsole": true,
  18.             "MIMode": "gdb",
  19.             "setupCommands": [
  20.                 {
  21.                     "description": "Enable pretty-printing for gdb",
  22.                     "text": "-enable-pretty-printing",
  23.                     "ignoreFailures": true
  24.                 }
  25.             ]
  26.         }
  27.     ]
  28. }

Among them, environment defines several environment variables DORIS_HOME UDF_RUNTIME_DIR LOG_DIR PID_DIR, which are the environment variables needed when palo_be is running. If it is not set, the startup will fail

Note: If you want attach (additional process) debugging, the configuration code is as follows:

  1. {
  2.     "version": "0.2.0",
  3.     "configurations": [
  4.           {
  5.           "name": "(gdb) Launch",
  6.           "type": "cppdbg",
  7.           "request": "attach",
  8.           "program": "/home/workspace/incubator-doris/output/lib/palo_be",
  9.           "processId":,
  10.           "MIMode": "gdb",
  11.           "internalConsoleOptions":"openOnSessionStart",
  12.           "setupCommands": [
  13.                 {
  14.                     "description": "Enable pretty-printing for gdb",
  15.                     "text": "-enable-pretty-printing",
  16.                     "ignoreFailures": true
  17.                 }
  18.             ]
  19.         }
  20.     ]
  21. }

In the configuration “request”: “attach”, “processId”: PID, these two configurations are the key points: set the debug mode of gdb to attach and attach the processId of the process, otherwise it will fail. To find the process id, you can enter the following command in the command line:

  1. ps -ef | grep palo*

As shown in the figure:

BE development and debugging environment under Linux - 图3

Among them, 15200 is the process id of the currently running be.

An example of a complete lainch.json is as follows:

  1.  {
  2.     "version": "0.2.0",
  3.     "configurations": [
  4.         {
  5.             "name": "(gdb) Attach",
  6.             "type": "cppdbg",
  7.             "request": "attach",
  8.             "program": "/home/workspace/incubator-doris/output/be/lib/palo_be",
  9.             "processId": 17016,
  10.             "MIMode": "gdb",
  11.             "setupCommands": [
  12.                 {
  13.                     "description": "Enable pretty-printing for gdb",
  14.                     "text": "-enable-pretty-printing",
  15.                     "ignoreFailures": true
  16.                 }
  17.             ]
  18.         },
  19.         {
  20.             "name": "(gdb) Launch",
  21.             "type": "cppdbg",
  22.             "request": "launch",
  23.             "program": "/home/workspace/incubator-doris/output/be/lib/palo_be",
  24.             "args": [],
  25.             "stopAtEntry": false,
  26.             "cwd": "/home/workspace/incubator-doris/output/be",
  27.             "environment": [
  28.                 {
  29.                     "name": "DORIS_HOME",
  30.                     "value": "/home/workspace/incubator-doris/output/be"
  31.                 },
  32.                 {
  33.                     "name": "UDF_RUNTIME_DIR",
  34.                     "value": "/home/workspace/incubator-doris/output/be/lib/udf-runtime"
  35.                 },
  36.                 {
  37.                     "name": "LOG_DIR",
  38.                     "value": "/home/workspace/incubator-doris/output/be/log"
  39.                 },
  40.                 {
  41.                     "name": "PID_DIR",
  42.                     "value": "/home/workspace/incubator-doris/output/be/bin"
  43.                 }
  44.             ],
  45.             "externalConsole": false,
  46.             "MIMode": "gdb",
  47.             "setupCommands": [
  48.                 {
  49.                     "description": "Enable pretty-printing for gdb",
  50.                     "text": "-enable-pretty-printing",
  51.                     "ignoreFailures": true
  52.                 }
  53.             ]
  54.         }
  55.     ]
  56. }

  1. Click to debug

    You can start your debugging journey with the rest,

BE development and debugging environment under Linux - 图4