Metrics

Metrics related to backup and restore functionality

Backup and restore operations export several metrics using the expvars interface. These are available at the /debug/vars endpoint of Vtbackup’s and VTTablet’s http status pages. More details can be found here.

Backup metrics

Metrics related to backup operations are available in both Vtbackup and VTTablet.

BackupBytes, BackupCount, BackupDurationNanoseconds

Depending on the Backup Engine and Backup Storage in-use, a backup may be a complex pipeline of operations, including but not limited to:

  • Reading files from disk.
  • Compressing files.
  • Uploading compress files to cloud object storage.

These operations are counted and timed, and the number of bytes consumed or produced by each stage of the pipeline are counted as well.

backup_duration_seconds

Deprecated

backup_duration_seconds times the duration of a backup. This metric is deprecated and will be removed in a future release. Use BackupDurationNanoseconds instead.

Restore metrics

Metrics related to restore operations are available in both Vtbackup and VTTablet.

RestoreBytes, RestoreCount, RestoreDurationNanoseconds

Depending on the Backup Engine and Backup Storage in-use, a restore may be a complex pipeline of operations, including but not limited to:

  • Downloading compressed files from cloud object storage.
  • Decompressing files.
  • Writing decompressed files to disk.

These operations are counted and timed, and the number of bytes consumed or produced by each stage of the pipeline are counted as well.

restore_duration_seconds

Deprecated

restore_duration_seconds times the duration of a restore. This metric is deprecated and will be removed in a future release. Use RestoreDurationNanoseconds instead.

Vtbackup metrics

Vtbackup exports some metrics which are not available elsewhere.

DurationByPhaseSeconds

Vtbackup fetches the last backup, restores it to an empty mysql installation, replicates recent changes into that installation, and then takes a backup of that installation.

DurationByPhaseSeconds exports timings for these individual phases.

Example

A snippet of vtbackup metrics after running it against the local example after creating the initial cluster

(Processed with jq for readability.)

  1. {
  2.   "BackupBytes": {
  3.     "BackupEngine.Builtin.Source:Read": 4777,
  4.     "BackupEngine.Builtin.Compressor:Write": 4616,
  5.     "BackupEngine.Builtin.Destination:Write": 162,
  6.     "BackupStorage.File.File:Write": 163
  7.   },
  8.   "BackupCount": {
  9.     "-.-.Backup": 1,
  10.     "BackupEngine.Builtin.Source:Open": 161,
  11.     "BackupEngine.Builtin.Source:Close": 322,
  12.     "BackupEngine.Builtin.Compressor:Close": 161,
  13.     "BackupEngine.Builtin.Destination:Open": 161,
  14.     "BackupEngine.Builtin.Destination:Close": 322
  15.   },
  16.   "BackupDurationNanoseconds": {
  17.     "-.-.Backup": 4188508542,
  18.     "BackupEngine.Builtin.Source:Open": 10649832,
  19.     "BackupEngine.Builtin.Source:Read": 55901067,
  20.     "BackupEngine.Builtin.Source:Close": 960826,
  21.     "BackupEngine.Builtin.Compressor:Write": 278358826,
  22.     "BackupEngine.Builtin.Compressor:Close": 79358372,
  23.     "BackupEngine.Builtin.Destination:Open": 16456627,
  24.     "BackupEngine.Builtin.Destination:Write": 11021043,
  25.     "BackupEngine.Builtin.Destination:Close": 17144630,
  26.     "BackupStorage.File.File:Write": 10743169
  27.   },
  28.   "DurationByPhaseSeconds": {
  29.     "InitMySQLd": 2,
  30.     "RestoreLastBackup": 6,
  31.     "CatchUpReplication": 1,
  32.     "TakeNewBackup": 4
  33.   },
  34.   "RestoreBytes": {
  35.     "BackupEngine.Builtin.Source:Read": 1095,
  36.     "BackupEngine.Builtin.Decompressor:Read": 950,
  37.     "BackupEngine.Builtin.Destination:Write": 209,
  38.     "BackupStorage.File.File:Read": 1113
  39.   },
  40.   "RestoreCount": {
  41.     "-.-.Restore": 1,
  42.     "BackupEngine.Builtin.Source:Open": 161,
  43.     "BackupEngine.Builtin.Source:Close": 322,
  44.     "BackupEngine.Builtin.Decompressor:Close": 161,
  45.     "BackupEngine.Builtin.Destination:Open": 161,
  46.     "BackupEngine.Builtin.Destination:Close": 322
  47.   },
  48.   "RestoreDurationNanoseconds": {
  49.     "-.-.Restore": 6204765541,
  50.     "BackupEngine.Builtin.Source:Open": 10542539,
  51.     "BackupEngine.Builtin.Source:Read": 104658370,
  52.     "BackupEngine.Builtin.Source:Close": 773038,
  53.     "BackupEngine.Builtin.Decompressor:Read": 165692120,
  54.     "BackupEngine.Builtin.Decompressor:Close": 51040,
  55.     "BackupEngine.Builtin.Destination:Open": 22715122,
  56.     "BackupEngine.Builtin.Destination:Write": 41679581,
  57.     "BackupEngine.Builtin.Destination:Close": 26954624,
  58.     "BackupStorage.File.File:Read": 102416075
  59.   },
  60.   "backup_duration_seconds": 4,
  61.   "restore_duration_seconds": 6
  62. }

Some notes to help understand these metrics:

  • BackupBytes["BackupStorage.File.File:Write"] measures how many bytes were read from disk by the file Backup Storage implementation during the backup phase.
  • DurationByPhaseSeconds["CatchUpReplication"] measures how long it took to catch-up replication after the restore phase.
  • DurationByPhaseSeconds["RestoreLastBackup"] measures to the duration of the restore phase.
  • RestoreDurationNanoseconds["-.-.Restore"] also measures to the duration of the restore phase.