VTTablet YAML configuration

Tabletserver YAML parameters

<< VTTablet with RBR mode

Background

This issue is an expansion of #5791, and covers the details of how we’ll use yaml to implement a hierarchy of parameters for initializing and customizing TabletServer components within a process.

Using the yaml approach, we intend to solve the following problems:

  • Multiple tablet servers need to exist within a process.
  • Badly named command line parameters: The new yaml specifications will have simpler and easier to remember parameter names.
  • Sections for improved readability.
  • Unreasonable default values: The new specs will introduce a simplified approach that will save the user from tweaking too many variables without knowing their consequences.
  • Repetition: If multiple TabletServers have to share the same parameter values, they should not be repeated.
  • Backward compatibility: The system must be backward compatible.

Proposed Approach

We will introduce two new command-line options that will be specified like this:

  1. -defaults=defaults.yaml -tablets=tablets.yaml

The defaults option allows the user to specify the default settings for all tablets. The “tablets” option allows the user to specify tablet specific options.

We’ll provide predefined “defaults” files that people can use as starting points. We’ll start with something simple like small.yaml, medium.yaml, and large.yaml.

Protobuf

We explored the use of protobufs for converting to and from yaml. However, the protobuf JSON implementation, which was required to correctly parse enums, was unable to preserve the original values for sub-objects. This was a showstopper for making defaults work.

The existing tabletenv.TabletConfig data structure will be converted into a struct with the new names and appropriate JSON tags.

Detailed specification

The following section lists all the properties that can be specified in a yaml file along with the existing defaults.

During implementation, we’ll need to decide between creating a universal protobuf that represents this structure vs converting the original one into something more usable.

Convention used: same as the one followed by kubernetes, and we’ll be making use of https://github.com/kubernetes-sigs/yaml for the implementation.

Note that certain properties (like tabletID) are only applicable to tablet-specific files.

  1. tabletID: zone-1234
  3. init:
  4.   dbName:            # init_db_name_override
  5.   keyspace:          # init_keyspace
  6.   shard:             # init_shard
  7.   tabletType:        # init_tablet_type
  8.   timeoutSeconds: 60 # init_timeout
  10. db:
  11.   socket:     # db_socket
  12.   host:       # db_host
  13.   port: 0     # db_port
  14.   charSet:    # db_charset
  15.   flags: 0    # db_flags
  16.   flavor:     # db_flavor
  17.   sslCa:      # db_ssl_ca
  18.   sslCaPath:  # db_ssl_ca_path
  19.   sslCert:    # db_ssl_cert
  20.   sslKey:     # db_ssl_key
  21.   serverName: # db_server_name
  22.   connectTimeoutMilliseconds: 0 # db_connect_timeout_ms
  23.   app:
  24.     user: vt_app      # db_app_user
  25.     password:         # db_app_password
  26.     useSsl: true      # db_app_use_ssl
  27.     preferSocket: true
  28.   dba:
  29.     user: vt_dba      # db_dba_user
  30.     password:         # db_dba_password
  31.     useSsl: true      # db_dba_use_ssl
  32.     preferSocket: true
  33.   filtered:
  34.     user: vt_filtered # db_filtered_user
  35.     password:         # db_filtered_password
  36.     useSsl: true      # db_filtered_use_ssl
  37.     preferSocket: true
  38.   repl:
  39.     user: vt_repl     # db_repl_user
  40.     password:         # db_repl_password
  41.     useSsl: true      # db_repl_use_ssl
  42.     preferSocket: true
  43.   appdebug:
  44.     user: vt_appdebug # db_appdebug_user
  45.     password:         # db_appdebug_password
  46.     useSsl: true      # db_appdebug_use_ssl
  47.     preferSocket: true
  48.   allprivs:
  49.     user: vt_allprivs # db_allprivs_user
  50.     password:         # db_allprivs_password
  51.     useSsl: true      # db_allprivs_use_ssl
  52.     preferSocket: true
  54. oltpReadPool:
  55.   size: 16                 # queryserver-config-pool-size
  56.   timeoutSeconds: 0        # queryserver-config-query-pool-timeout
  57.   idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout
  58.   prefillParallelism: 0    # queryserver-config-pool-prefill-parallelism
  59.   maxWaiters: 50000        # queryserver-config-query-pool-waiter-cap
  61. olapReadPool:
  62.   size: 200                # queryserver-config-stream-pool-size
  63.   timeoutSeconds: 0        # queryserver-config-query-pool-timeout
  64.   idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout
  65.   prefillParallelism: 0    # queryserver-config-stream-pool-prefill-parallelism
  66.   maxWaiters: 0
  68. txPool:
  69.   size: 20                 # queryserver-config-transaction-cap
  70.   timeoutSeconds: 1        # queryserver-config-txpool-timeout
  71.   idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout
  72.   prefillParallelism: 0    # queryserver-config-transaction-prefill-parallelism
  73.   maxWaiters: 50000        # queryserver-config-txpool-waiter-cap
  75. oltp:
  76.   queryTimeoutSeconds: 30 # queryserver-config-query-timeout
  77.   txTimeoutSeconds: 30    # queryserver-config-transaction-timeout
  78.   maxRows: 10000          # queryserver-config-max-result-size
  79.   warnRows: 0             # queryserver-config-warn-result-size
  81. hotRowProtection:
  82.   mode: disable|dryRun|enable # enable_hot_row_protection, enable_hot_row_protection_dry_run
  83.   # Default value is same as txPool.size.
  84.   maxQueueSize: 20            # hot_row_protection_max_queue_size
  85.   maxGlobalQueueSize: 1000    # hot_row_protection_max_global_queue_size
  86.   maxConcurrency: 5           # hot_row_protection_concurrent_transactions
  88. consolidator: enable|disable|notOnMaster # enable-consolidator, enable-consolidator-replicas
  89. heartbeatIntervalMilliseconds: 0         # heartbeat_enable, heartbeat_interval
  90. shutdownGracePeriodSeconds: 0            # transaction_shutdown_grace_period
  91. passthroughDML: false                    # queryserver-config-passthrough-dmls
  92. streamBufferSize: 32768                  # queryserver-config-stream-buffer-size
  93. queryCacheSize: 5000                     # queryserver-config-query-cache-size
  94. schemaReloadIntervalSeconds: 1800        # queryserver-config-schema-reload-time
  95. watchReplication: false                  # watch_replication_stream
  96. terseErrors: false                       # queryserver-config-terse-errors
  97. messagePostponeParallelism: 4            # queryserver-config-message-postpone-cap
  98. cacheResultFields: true                  # enable-query-plan-field-caching
  101. # The following flags are currently not supported.
  102. # enforce_strict_trans_tables
  103. # queryserver-config-strict-table-acl
  104. # queryserver-config-enable-table-acl-dry-run
  105. # queryserver-config-acl-exempt-acl
  106. # enable-tx-throttler
  107. # tx-throttler-config
  108. # tx-throttler-healthcheck-cells
  109. # enable_transaction_limit
  110. # enable_transaction_limit_dry_run
  111. # transaction_limit_per_user
  112. # transaction_limit_by_username
  113. # transaction_limit_by_principal
  114. # transaction_limit_by_component
  115. # transaction_limit_by_subcomponent

There are also other global parameters. VTTablet has a total of 405 flags. We may have to later move some of them into these yamls.

Implementation

We’ll use the unified tabletenv.TabletConfig data structure to load defaults as well as tablet-specific values. The following changes will be made:

  • TabletConfig will be changed to match the above specifications.
  • The existing flags will be changed to update the values in a global TabletConfig.
  • In case of type mismatch (like time.Duration vs a “Seconds” variable), an extra step will be performed after parsing to convert the flag variables into TabletConfig members.
  • The code will be changed to honor the new TabletConfig members.
  • After the flag.Parse step, a copy of the global Config will be used as input to load the defaults file. This means that any command line flags that are not overridden in the yaml files will be preserved. This behavior is needed to support backward compatibility in case we decide to move more flags into the yaml.
  • For each tablet entry, we create a copy of the result TabletConfig and read the tablet yaml into those, which will set the tablet-specific values. This will then be used to instantiate a TabletServer.

The exact format of the tablets.yaml file is not fully finalized. Our options are to allow a list of files where each is for a single tablet, or, to require only one file containing a dictionary of tablet-specific overrides.

<< VTTablet with RBR mode