TiDB Kubernetes Control User Guide

TiDB Kubernetes Control (tkctl) is a command line utility that is used for TiDB Operator to maintain and diagnose the TiDB cluster in Kubernetes.

tkctl - 图1Note

PingCAP is no longer maintaining tkctl from v1.1.x, some of the following functions may not be usable, please use the equivalent kubectl commands directly.

Installation

To install tkctl, you can download the pre-built binary or build tkctl from source.

Download the latest pre-built binary

After unzipping the downloaded file, you can add the tkctl executable file to your PATH to finish the installation.

Build from source

Requirement: Go >= the 1.11 version or later

  1. git clone --depth=1 https://github.com/pingcap/tidb-operator.git && \
  2. GOOS=<YOUR_GOOS> make cli &&\
  3. mv tkctl /usr/local/bin/tkctl

Shell auto-completion

You can configure the shell auto-completion for tkctl to simplify its usage.

To configure the auto-completion for BASH, you need to first install the bash-completion package, and configure with either of the two methods below:

  • Configure auto-completion in the current shell:

    1. source <(tkctl completion bash)
  • Add auto-completion permanently to your bash shell:

    1. echo "if hash tkctl 2>/dev/null; then source <(tkctl completion bash); fi" >> ~/.bashrc

To configure the auto-completion for ZSH, you can choose from either of the two methods below:

  • Configure auto-completion in the current shell:

    1. source <(tkctl completion zsh)
  • Add auto-completion permanently to your zsh shell:

    1. echo "if hash tkctl 2>/dev/null; then source <(tkctl completion zsh); fi" >> ~/.zshrc

Kubernetes configuration

tkctl reuses the kubeconfig file (the default location is ~/.kube/config) to connect with the Kubernetes cluster. You can verify whether kubeconfig is correctly configured by using the following command:

  1. tkctl version

If the above command correctly outputs the version of TiDB Operator on the server side, then kubeconfig is correctly configured.

Commands

tkctl version

This command is used to show the version of the local tkctl and tidb-operator installed in the target cluster.

For example:

  1. tkctl version
  1. Client Version: v1.0.0-beta.1-p2-93-g6598b4d3e75705-dirty
  2. TiDB Controller Manager Version: pingcap/tidb-operator:latest
  3. TiDB Scheduler Version: pingcap/tidb-operator:latest

tkctl list

This command is used to list all installed TiDB clusters.

FlagAbbreviationDescription
—all-namespaces-AWhether to search all Kubernetes namespaces
—output-oThe output format; you can choose from [default,json,yaml], and the default format is default

For example:

  1. tkctl list -A
  1. NAMESPACE NAME PD TIKV TIDB AGE
  2. foo demo-cluster 3/3 3/3 2/2 11m
  3. bar demo-cluster 3/3 3/3 1/2 11m

tkctl use

This command is used to specify the TiDB cluster that the current tkctl command operates on. After you specify a TiDB cluster by using this command, all commands that operates on a cluster will automatically select this cluster so the --tidbcluster option can be omitted.

For example:

  1. tkctl use --namespace=foo demo-cluster
  1. Tidb cluster switched to foo/demo-cluster

tkctl info

This command is used to display information about the TiDB cluster.

FlagAbbreviationDescription
—tidb-cluster-tSpecify a TiDB cluster; default to the TiDB cluster that is being used

For example:

  1. tkctl info
  1. Name: demo-cluster
  2. Namespace: foo
  3. CreationTimestamp: 2019-04-17 17:33:41 +0800 CST
  4. Overview:
  5. Phase Ready Desired CPU Memory Storage Version
  6. ----- ----- ------- --- ------ ------- -------
  7. PD: Normal 3 3 200m 1Gi 1Gi pingcap/pd:v3.0.0-rc.1
  8. TiKV: Normal 3 3 1000m 2Gi 10Gi pingcap/tikv:v3.0.0-rc.1
  9. TiDB Upgrade 1 2 500m 1Gi pingcap/tidb:v3.0.0-rc.1
  10. Endpoints(NodePort):
  11. - 172.16.4.158:31441
  12. - 172.16.4.155:31441

tkctl get [component]

This is a group of commands that are used to get the details of TiDB cluster components.

You can query the following components: pd, tikv, tidb, volume and all (to query all components).

FlagAbbreviationDescription
—tidb-cluster-tSpecify a TiDB cluster; default to the TiDB cluster that is being used
—output-oThe output format; you can choose from [default,json,yaml], and the default format is default

For example:

  1. tkctl get tikv
  1. NAME READY STATUS MEMORY CPU RESTARTS AGE NODE
  2. demo-cluster-tikv-0 2/2 Running 2098Mi/4196Mi 2/2 0 3m19s 172.16.4.155
  3. demo-cluster-tikv-1 2/2 Running 2098Mi/4196Mi 2/2 0 4m8s 172.16.4.160
  4. demo-cluster-tikv-2 2/2 Running 2098Mi/4196Mi 2/2 0 4m45s 172.16.4.157
  1. tkctl get volume
  1. VOLUME CLAIM STATUS CAPACITY NODE LOCAL
  2. local-pv-d5dad2cf tikv-demo-cluster-tikv-0 Bound 1476Gi 172.16.4.155 /mnt/disks/local-pv56
  3. local-pv-5ade8580 tikv-demo-cluster-tikv-1 Bound 1476Gi 172.16.4.160 /mnt/disks/local-pv33
  4. local-pv-ed2ffe50 tikv-demo-cluster-tikv-2 Bound 1476Gi 172.16.4.157 /mnt/disks/local-pv13
  5. local-pv-74ee0364 pd-demo-cluster-pd-0 Bound 1476Gi 172.16.4.155 /mnt/disks/local-pv46
  6. local-pv-842034e6 pd-demo-cluster-pd-1 Bound 1476Gi 172.16.4.158 /mnt/disks/local-pv74
  7. local-pv-e54c122a pd-demo-cluster-pd-2 Bound 1476Gi 172.16.4.156 /mnt/disks/local-pv72

tkctl debug [pod_name]

This command is used to diagnose the Pods in a TiDB cluster. It launches a debug launcher Pod which then starts a debug container using the specified docker image on the same host of the target Pod. The container has necessary troubleshooting tools and shares the same namespaces with the container in the target Pod, so you can diagnose the target container by using various tools in the debug container.

FlagAbbreviationDescription
—imageSpecify the docker image used by the debug container; default to pingcap/tidb-debug:lastest
—launcher-imageSpecify the docker image for the debug launcher pod which is responsible for launching the debug container; default to pingcap/debug-launcher:latest
—container-cSelect the container to be diagnosed; default to the first container of the target Pod
—docker-socketSpecify the docker socket on the target node; default to /var/run/docker.sock
—privilegedWhether to enable the privileged mode for the debug container

tkctl - 图2Note

The default image of the debug container contains various troubleshooting tools, so the image size is relatively large. If you only need pd-ctl and tidb-ctl, you can specify using the tidb-control image by using the --image=pingcap/tidb-control:latest command line option.

For example:

  1. tkctl debug demo-cluster-tikv-0
  1. ps -ef

Using tools like GDB and perf in the debug container requires special operations because of the difference in root filesystems of the target container and the debug container.

GDB

When you use GDB to debug the process in the target container, make sure you set the program option to the binary in the target container. Additionally, if you use images other than tidb-debug as the debug container or if the pid of the target process is not 1, you have to configure the location of dynamic libraries via the set sysroot command as follows:

  1. tkctl debug demo-cluster-tikv-0
  1. gdb /proc/${pid:-1}/root/tikv-server 1

The .gdbinit pre-configured in the tidb-debug image will set sysroot to /proc/1/root/ automatically. For this reason, you can omit this following step if you are using the tidb-debug image and the pid of the target process is 1.

  1. (gdb) set sysroot /proc/${pid}/root/

Start debugging:

  1. (gdb) thread apply all bt
  1. (gdb) info threads

Perf and flame graphs

To use the perf command and the run_flamegraph.sh script properly, you must copy the program from the target container to the same location in the debug container:

  1. tkctl debug demo-cluster-tikv-0
  1. cp /proc/1/root/tikv-server /
  1. ./run_flamegraph.sh 1

This script automatically uploads the generated flame graph (SVG format) to transfer.sh, and you can visit the link outputted by the script to download the flame graph.

tkctl ctop

The complete form of the command is tkctl ctop [pod_name | node/node_name ].

This command is used to view the real-time monitoring stats of the target Pod or node in the cluster. Compared with kubectl top, tkctl ctop also provides network and disk stats, which are important for diagnosing problems in the TiDB cluster.

FlagAbbreviationDescription
—imageSpecify the docker image of ctop; default to quay.io/vektorlab/ctop:0.7.2
—docker-socketSpecify the docker socket that ctop uses; default to /var/run/docker.sock

For example:

  1. tkctl ctop node/172.16.4.155
  1. tkctl ctop demo-cluster-tikv-0

tkctl help [command]

This command is used to print help messages of the sub commands.

For example:

  1. tkctl help debug

tkctl options

This command is used to view the global flags of tkctl.

For example:

  1. tkctl options
  1. The following options can be passed to any command:
  2. --alsologtostderr=false: log to standard error as well as files
  3. --as='': Username to impersonate for the operation
  4. --as-group=[]: Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
  5. --cache-dir='/Users/alei/.kube/http-cache': Default HTTP cache directory
  6. --certificate-authority='': Path to a cert file for the certificate authority
  7. --client-certificate='': Path to a client certificate file for TLS
  8. --client-key='': Path to a client key file for TLS
  9. --cluster='': The name of the kubeconfig cluster to use
  10. --context='': The name of the kubeconfig context to use
  11. --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will
  12. make your HTTPS connections insecure
  13. --kubeconfig='': Path to the kubeconfig file to use for CLI requests.
  14. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace
  15. --log_dir='': If non-empty, write log files in this directory
  16. --logtostderr=true: log to standard error instead of files
  17. -n, --namespace='': If present, the namespace scope for this CLI request
  18. --request-timeout='0': The length of time to wait before giving up on a single server request. Non-zero values
  19. should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.
  20. -s, --server='': The address and port of the Kubernetes API server
  21. --stderrthreshold=2: logs at or above this threshold go to stderr
  22. -t, --tidbcluster='': Tidb cluster name
  23. --token='': Bearer token for authentication to the API server
  24. --user='': The name of the kubeconfig user to use
  25. -v, --v=0: log level for V logs
  26. --vmodule=: comma-separated list of pattern=N settings for file-filtered logging

These options are mainly used to connect with the Kubernetes cluster and two commonly used options among them are as follows:

  • --context: specify the target Kubernetes cluster
  • --namespace: specify the Kubernetes namespace