Upgrading the MDS Cluster

Currently the MDS cluster does not have built-in versioning or file systemflags to support seamless upgrades of the MDSs without potentially causingassertions or other faults due to incompatible messages or other functionaldifferences. For this reason, it’s necessary during any cluster upgrade toreduce the number of active MDS for a file system to one first so that twoactive MDS do not communicate with different versions. Further, it’s alsonecessary to take standbys offline as any new CompatSet flags will propagatevia the MDSMap to all MDS and cause older MDS to suicide.

The proper sequence for upgrading the MDS cluster is:

  • Reduce the number of ranks to 1:
  1. ceph fs set <fs_name> max_mds 1
  • Wait for cluster to stop non-zero ranks where only rank 0 is active and the rest are standbys.
  1. ceph status # wait for MDS to finish stopping
  • Take all standbys offline, e.g. using systemctl:
  1. systemctl stop ceph-mds.target
  • Confirm only one MDS is online and is rank 0 for your FS:
  1. ceph status
  • Upgrade the single active MDS, e.g. using systemctl:
  1. # use package manager to update cluster
  2. systemctl restart ceph-mds.target
  • Upgrade/start the standby daemons.
  1. # use package manager to update cluster
  2. systemctl restart ceph-mds.target
  • Restore the previous max_mds for your cluster:
  1. ceph fs set <fs_name> max_mds <old_max_mds>