Metrics

Metrics related to backup and restore functionality

Backup and restore operations export several metrics using the expvars interface. These are available at the /debug/vars endpoint of Vtbackup’s and VTTablet’s http status pages. More details can be found here.

Backup metrics

Metrics related to backup operations are available in both Vtbackup and VTTablet.

BackupBytes, BackupCount, BackupDurationNanoseconds

Depending on the Backup Engine and Backup Storage in-use, a backup may be a complex pipeline of operations, including but not limited to:

  • Reading files from disk.
  • Compressing files.
  • Uploading compress files to cloud object storage.

These operations are counted and timed, and the number of bytes consumed or produced by each stage of the pipeline are counted as well.

Restore metrics

Metrics related to restore operations are available in both Vtbackup and VTTablet.

RestoreBytes, RestoreCount, RestoreDurationNanoseconds

Depending on the Backup Engine and Backup Storage in-use, a restore may be a complex pipeline of operations, including but not limited to:

  • Downloading compressed files from cloud object storage.
  • Decompressing files.
  • Writing decompressed files to disk.

These operations are counted and timed, and the number of bytes consumed or produced by each stage of the pipeline are counted as well.

RestoredBackupTime, RestorePosition

RestoredBackupTime captures the timestamp associated with the backup from which the current process was restored. RestorePosition captures the GTID position associated with that backup.

Vtbackup metrics

Vtbackup exports some metrics which are not available elsewhere.

DurationByPhaseSeconds

Deprecated. See next section.

Phase

Phase a binary-valued gauge that reports the currently active phase.

PhaseStatus

Vtbackup emits status metrics for its phases of execution. PhaseStatus reports a 1 (active) or a 0 (inactive) for each of the following phases and statuses:

  • CatchUpReplication phase has statuses Stalled and Stopped.
    • Stalled is set to 1 when replication stops advancing.
    • Stopped is set to 1 when replication stops before vtbackup catches up with the primary.

Example

A snippet of vtbackup metrics after running it against the local example after creating the initial cluster

(Processed with jq for readability.)

  1. {
  2.   "BackupBytes": {
  3.     "BackupEngine.Builtin.Source:Read": 4777,
  4.     "BackupEngine.Builtin.Compressor:Write": 4616,
  5.     "BackupEngine.Builtin.Destination:Write": 162,
  6.     "BackupStorage.File.File:Write": 163
  7.   },
  8.   "BackupCount": {
  9.     "-.-.Backup": 1,
  10.     "BackupEngine.Builtin.Source:Open": 161,
  11.     "BackupEngine.Builtin.Source:Close": 322,
  12.     "BackupEngine.Builtin.Compressor:Close": 161,
  13.     "BackupEngine.Builtin.Destination:Open": 161,
  14.     "BackupEngine.Builtin.Destination:Close": 322
  15.   },
  16.   "BackupDurationNanoseconds": {
  17.     "-.-.Backup": 4188508542,
  18.     "BackupEngine.Builtin.Source:Open": 10649832,
  19.     "BackupEngine.Builtin.Source:Read": 55901067,
  20.     "BackupEngine.Builtin.Source:Close": 960826,
  21.     "BackupEngine.Builtin.Compressor:Write": 278358826,
  22.     "BackupEngine.Builtin.Compressor:Close": 79358372,
  23.     "BackupEngine.Builtin.Destination:Open": 16456627,
  24.     "BackupEngine.Builtin.Destination:Write": 11021043,
  25.     "BackupEngine.Builtin.Destination:Close": 17144630,
  26.     "BackupStorage.File.File:Write": 10743169
  27.   },
  28.   "DurationByPhaseSeconds": {
  29.     "InitMySQLd": 2,
  30.     "RestoreLastBackup": 6,
  31.     "CatchUpReplication": 1,
  32.     "TakeNewBackup": 4
  33.   },
  34.   "Phase": {
  35.     "InitialBackup": 0,
  36.     "RestoreLastBackup": 0,
  37.     "CatchupReplication": 0,
  38.     "TakeNewBackup": 0
  39.   },
  40.   "RestoreBytes": {
  41.     "BackupEngine.Builtin.Source:Read": 1095,
  42.     "BackupEngine.Builtin.Decompressor:Read": 950,
  43.     "BackupEngine.Builtin.Destination:Write": 209,
  44.     "BackupStorage.File.File:Read": 1113
  45.   },
  46.   "RestoreCount": {
  47.     "-.-.Restore": 1,
  48.     "BackupEngine.Builtin.Source:Open": 161,
  49.     "BackupEngine.Builtin.Source:Close": 322,
  50.     "BackupEngine.Builtin.Decompressor:Close": 161,
  51.     "BackupEngine.Builtin.Destination:Open": 161,
  52.     "BackupEngine.Builtin.Destination:Close": 322
  53.   },
  54.   "RestoreDurationNanoseconds": {
  55.     "-.-.Restore": 6204765541,
  56.     "BackupEngine.Builtin.Source:Open": 10542539,
  57.     "BackupEngine.Builtin.Source:Read": 104658370,
  58.     "BackupEngine.Builtin.Source:Close": 773038,
  59.     "BackupEngine.Builtin.Decompressor:Read": 165692120,
  60.     "BackupEngine.Builtin.Decompressor:Close": 51040,
  61.     "BackupEngine.Builtin.Destination:Open": 22715122,
  62.     "BackupEngine.Builtin.Destination:Write": 41679581,
  63.     "BackupEngine.Builtin.Destination:Close": 26954624,
  64.     "BackupStorage.File.File:Read": 102416075
  65.   },
  66.   "RestorePosition": "MySQL56/f00e54ca-0fbf-11ee-ad84-eddb850690bf:1-61",
  67.   "RestoredBackupTime": "2023-06-21T00:39:00Z",
  68. }

Some notes to help understand these metrics:

  • BackupBytes["BackupStorage.File.File:Write"] measures how many bytes were read from disk by the file Backup Storage implementation during the backup phase.
  • Phase["CatchupReplication"] reports whether the catch-up replication phase is active (1) or not (0).
  • Phase["RestoreLastBackup"] reports whether the restore last backup phase is active (1) or not (0).
  • RestoreDurationNanoseconds["-.-.Restore"] also measures to the duration of the restore phase.