Host Management

Users can view and manage Harvester nodes from the host page. The first node always defaults to be a management node of the cluster. When there are three or more nodes, the two other nodes that first joined are automatically promoted to management nodes to form a HA cluster.

Host Management - 图1note

Because Harvester is built on top of Kubernetes and uses etcd as its database, the maximum node fault toleration is one when there are three management nodes.

host.png

Node Maintenance

For admin users, you can click Enable Maintenance Mode to evict all VMs from a node automatically. It will leverage the VM live migration feature to migrate all VMs to other nodes automatically. Note that at least two active nodes are required to use this feature.

node-maintenance.png

Cordoning a Node

Cordoning a node marks it as unschedulable. This feature is useful for performing short tasks on the node during small maintenance windows, like reboots, upgrades, or decommissions. When you’re done, power back on and make the node schedulable again by uncordoning it.

cordon-node.png

Deleting a Node

Deleting a node is done in two phases:

  1. Delete the node from Harvester

    • Go to the Hosts page
    • On the node you want to modify, click ⋮ > Delete
  2. Uninstall RKE2 from the node

    • Login to the node as root
    • Run rke2-uninstall.sh to delete the whole RKE2 service.

Host Management - 图5caution

You will lose all data of the control plane node after deleing the RKE2 service.

Host Management - 图6note

There’s a known issue about node hard delete. Once resolved, the last step can be skipped.

delete.png

Multi-disk Management

Add Additional Disks

Users can view and add multiple disks as additional data volumes from the edit host page.

  1. Go to the Hosts page.
  2. On the node you want to modify, click ⋮ > Edit Config.

Edit Config

  1. Select the Storage tab and click Add Disk.

Add Disks

Host Management - 图10caution

As of Harvester v1.0.2, we no longer support adding partitions as additional disks. If you want to add it as an additional disk, be sure to delete all partitions first (e.g., using fdisk).

  1. Select an additional raw block device to add as an additional data volume.
    • The Force Formatted option is required if the block device has never been force-formatted.

Force Format

  1. Last, you can click ⋮ > Edit Config again to check the newly added disk. Meanwhile, you can also add the “Host/Disk” tag (details are described in the next section).

Check Result

Host Management - 图13note

In order for Harvester to identify the disks, each disk needs to have a unique WWN. Otherwise, Harvester will refuse to add the disk. If your disk does not have a WWN, you can format it with the EXT4 filesystem to help Harvester recognize the disk.

Host Management - 图14note

If you are testing Harvester in a QEMU environment, you’ll need to use QEMU v6.0 or later. Previous versions of QEMU will always generate the same WWN for NVMe disks emulation. This will cause Harvester to not add the additional disks, as explained above.

Storage Tags

The storage tag feature enables only certain nodes or disks to be used for storing Longhorn volume data. For example, performance-sensitive data can use only the high-performance disks which can be tagged as fast, ssd or nvme, or only the high-performance nodes tagged as baremetal.

This feature supports both disks and nodes.

Setup

The tags can be set up through the Harvester UI on the host page:

  1. Click Hosts -> Edit Config -> Storage
  2. Click Add Host/Disk Tags to start typing and hit enter to add new tags.
  3. Click Save to update tags.
  4. On the StorageClasses page, create a new storage class and select those defined tags on the Node Selector and Disk Selector fields.

All the existing scheduled volumes on the node or disk won’t be affected by the new tags.

Host Management - 图15note

When multiple tags are specified for a volume, the disk and the nodes (that the disk belongs to) must have all the specified tags to become usable.

Remove disks

Before removing a disk, you must first evict Longhorn replicas on the disk.

Host Management - 图16note

The replica data would be rebuilt to another disk automatically to keep the high availability.

Identify the disk to remove (Harvester dashboard)

  1. Go to the Hosts page.
  2. On the node containing the disk, select the node name and go to the Storage tab.
  3. Find the disk you want to remove. Let’s assume we want to remove /dev/sdb, and the disk’s mount point is /var/lib/harvester/extra-disks/1b805b97eb5aa724e6be30cbdb373d04.

Find disk to remove

Evict replicas (Longhorn dashboard)

  1. Please follow this session to enable the embedded Longhorn dashboard.
  2. Visit the Longhorn dashboard and go to the Node page.
  3. Expand the node containing the disk. Confirm the mount point /var/lib/harvester/extra-disks/1b805b97eb5aa724e6be30cbdb373d04 is in the disks list.

Check the removing disk

  1. Select Edit node and disks.

Edit node and disks

  1. Scroll to the disk you want to remove.
  • Set Scheduling to Disable.
  • Set Eviction Requested to True. Select Save. Do not select the delete icon.

Evict disk

  1. The disk will be disabled. Please wait until the disk replica count becomes 0 to proceed with removing the disk.

Wait replicas

Remove the disk (Harvester dashboard)

  1. Go to the Hosts page.
  2. On the node containing the disk, select ⋮ > Edit Config.
  3. Go to the Storage tab and select x to remove the disk.

Remove disk

  1. Select Save to remove the disk.

Ksmtuned Mode

Available as of v1.1.0

Ksmtuned is a KSM automation tool deployed as a DaemonSet to run Ksmtuned on each node. It will start or stop the KSM by watching the available memory percentage ratio (i.e. Threshold Coefficient). By default, you need to manually enable Ksmtuned on each node UI. You will be able to see the KSM statistics from the node UI after 1-2 minutes.(check KSM for more details).

Quick Run

  1. Go to the Hosts page.
  2. On the node you want to modify, click ⋮ > Edit Config.
  3. Select the Ksmtuned tab and select Run in Run Strategy.
  4. (Optional) You can modify Threshold Coefficient as needed.

Edit Ksmtuned

  1. Click Save to update.
  2. Wait for about 1-2 minutes and you can check its Statistics by clicking Your Node > Ksmtuned tab.

View Ksmtuned Statistics

Parameters

Run Strategy:

  • Stop: Stop Ksmtuned and KSM. VMs can still use shared memory pages.
  • Run: Run Ksmtuned.
  • Prune: Stop Ksmtuned and prune KSM memory pages.

Threshold Coefficient: configures the available memory percentage ratio. If the available memory is less than the threshold, KSM will be started; otherwise, KSM will be stopped.

Merge Across Nodes: specifies if pages from different NUMA nodes can be merged.

Mode:

  • Standard: The default mode. The control node ksmd uses about 20% of a single CPU. It uses the following parameters:
  1. Boost: 0
  2. Decay: 0
  3. Maximum Pages: 100
  4. Minimum Pages: 100
  5. Sleep Time: 20
  • High-performance: Node ksmd uses 20% to 100% of a single CPU and has higher scanning and merging efficiency. It uses the following parameters:
  1. Boost: 200
  2. Decay: 50
  3. Maximum Pages: 10000
  4. Minimum Pages: 100
  5. Sleep Time: 20
  • Customized: You can customize the configuration to reach the performance that you want.

Ksmtuned uses the following parameters to control KSM efficiency:

ParametersDescription
BoostThe number of scanned pages is incremented each time if the available memory is less than the Threshold Coefficient.
DecayThe number of scanned pages is decremented each time if the available memory is greater than the Threshold Coefficient.
Maximum PagesMaximum number of pages per scan.
Minimum PagesThe minimum number of pages per scan, also the configuration for the first run.
Sleep Time (ms)The interval between two scans, which is calculated with the formula (Sleep Time 16 1024* 1024 / Total Memory). Minimum: 10ms.

For example, assume you have a 512GiB memory node that uses the following parameters:

  1. Boost: 300
  2. Decay: 100
  3. Maximum Pages: 5000
  4. Minimum Pages: 1000
  5. Sleep Time: 50

When Ksmtuned starts, initialize pages_to_scan in KSM to 1000 (Minimum Pages) and set sleep_millisecs to 10 (50 * 16 * 1024 * 1024 / 536870912 KiB < 10).

KSM starts when the available memory falls below the Threshold Coefficient. If it detects that it is running, pages_to_scan increments by 300 (Boost) every minute until it reaches 5000 (Maximum Pages).

KSM will stop when the available memory is above the Threshold Coefficient. If it detects that it is stopped, pages_to_scan decrements by 100 (Decay) every minute until it reaches 1000 (Minimum Pages).