System configuration

Do the following configuration steps on each of the nodes in the cluster.

ntp

If your instance does not have public Internet access, make sure the following packages have been installed (all can be retrieved from the yum repo epel, make sure to use the latest epel release repository):

  • epel-release
  • ntp

Here’s the command to install these packages.

  1. $ sudo yum install -y epel-release ntp

ulimits

In Linux, ulimit is used to limit and control the usage of system resources (threads, files, and network connections) on a per-process or per-user basis.

Checking ulimits

Run the following command to check the ulimit settings.

  1. $ ulimit -a

The following settings are recommended when running YugabyteDB.

  1. core file size          (blocks, -c) unlimited
  2. data seg size           (kbytes, -d) unlimited
  3. scheduling priority             (-e) 0
  4. file size               (blocks, -f) unlimited
  5. pending signals                 (-i) 119934
  6. max locked memory       (kbytes, -l) 64
  7. max memory size         (kbytes, -m) unlimited
  8. open files                      (-n) 1048576
  9. pipe size            (512 bytes, -p) 8
  10. POSIX message queues     (bytes, -q) 819200
  11. real-time priority              (-r) 0
  12. stack size              (kbytes, -s) 8192
  13. cpu time               (seconds, -t) unlimited
  14. max user processes              (-u) 12000
  15. virtual memory          (kbytes, -v) unlimited
  16. file locks                      (-x) unlimited

Setting system-wide ulimits

You can change values by substituting the -n option for any possible value in the output of ulimit -a. Issue a command in the following form to change a ulimit setting.

  1. $ ulimit -n <value>
  1. -f (file size): unlimited
  2. -t (cpu time): unlimited
  3. -v (virtual memory): unlimited [1]
  4. -l (locked-in-memory size): unlimited
  5. -n (open files): 64000
  6. -m (memory size): unlimited [1] [2]
  7. -u (processes/threads): 64000

Note

  • After changing a ulimit setting, the YB-Master and YB-TServer servers must be restarted in order for the new settings to take effect. Check the /proc/<process pid> file to see the current settings.
  • Changes made using ulimit may revert following a system restart depending on the system configuration.

Most of these settings can also be applied permanently by adding the following in /etc/security/limits.conf.

  1. *                -       core            unlimited
  2. *                -       data            unlimited
  3. *                -       fsize           unlimited
  4. *                -       sigpending      119934
  5. *                -       memlock         64
  6. *                -       rss             unlimited
  7. *                -       nofile          1048576
  8. *                -       msgqueue        819200
  9. *                -       stack           8192
  10. *                -       cpu             unlimited
  11. *                -       nproc           12000
  12. *                -       locks           unlimited

On CentOS, /etc/security/limits.d/20-nproc.conf must also be configured

  1. *          soft    nproc     12000

NoteAfter changing a ulimit setting in /etc/security/limits.conf, you will need to log out and back in. To update system processes, you may need to restart.

Note

If you’re using a desktop-distribution, such as ubuntu-desktop, the settings above may not suffice.The OS needs additional steps to change ulimit for gui login. In the case of ubuntu-desktop:

In /etc/systemd/user.conf and /etc/systemd/system.conf, add at the end of file DefaultLimitNOFILE=64000.

Something similar may be needed for other distributions.