Manage HugePages

Configure and manage huge pages as a schedulable resource in a cluster.

FEATURE STATE: Kubernetes v1.20 [stable]

Kubernetes supports the allocation and consumption of pre-allocated huge pages by applications in a Pod. This page describes how users can consume huge pages.

Before you begin

  1. Kubernetes nodes must pre-allocate huge pages in order for the node to report its huge page capacity. A node can pre-allocate huge pages for multiple sizes.

The nodes will automatically discover and report all huge page resources as schedulable resources.

API

Huge pages can be consumed via container level resource requirements using the resource name hugepages-<size>, where <size> is the most compact binary notation using integer values supported on a particular node. For example, if a node supports 2048KiB and 1048576KiB page sizes, it will expose a schedulable resources hugepages-2Mi and hugepages-1Gi. Unlike CPU or memory, huge pages do not support overcommit. Note that when requesting hugepage resources, either memory or CPU resources must be requested as well.

A pod may consume multiple huge page sizes in a single pod spec. In this case it must use medium: HugePages-<hugepagesize> notation for all volume mounts.

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: huge-pages-example
  5. spec:
  6.   containers:
  7.   - name: example
  8.     image: fedora:latest
  9.     command:
  10.     - sleep
  11.     - inf
  12.     volumeMounts:
  13.     - mountPath: /hugepages-2Mi
  14.       name: hugepage-2mi
  15.     - mountPath: /hugepages-1Gi
  16.       name: hugepage-1gi
  17.     resources:
  18.       limits:
  19.         hugepages-2Mi: 100Mi
  20.         hugepages-1Gi: 2Gi
  21.         memory: 100Mi
  22.       requests:
  23.         memory: 100Mi
  24.   volumes:
  25.   - name: hugepage-2mi
  26.     emptyDir:
  27.       medium: HugePages-2Mi
  28.   - name: hugepage-1gi
  29.     emptyDir:
  30.       medium: HugePages-1Gi

A pod may use medium: HugePages only if it requests huge pages of one size.

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: huge-pages-example
  5. spec:
  6.   containers:
  7.   - name: example
  8.     image: fedora:latest
  9.     command:
  10.     - sleep
  11.     - inf
  12.     volumeMounts:
  13.     - mountPath: /hugepages
  14.       name: hugepage
  15.     resources:
  16.       limits:
  17.         hugepages-2Mi: 100Mi
  18.         memory: 100Mi
  19.       requests:
  20.         memory: 100Mi
  21.   volumes:
  22.   - name: hugepage
  23.     emptyDir:
  24.       medium: HugePages
  • Huge page requests must equal the limits. This is the default if limits are specified, but requests are not.
  • Huge pages are isolated at a container scope, so each container has own limit on their cgroup sandbox as requested in a container spec.
  • EmptyDir volumes backed by huge pages may not consume more huge page memory than the pod request.
  • Applications that consume huge pages via shmget() with SHM_HUGETLB must run with a supplemental group that matches proc/sys/vm/hugetlb_shm_group.
  • Huge page usage in a namespace is controllable via ResourceQuota similar to other compute resources like cpu or memory using the hugepages-<size> token.
  • Support of multiple sizes huge pages is feature gated. It can be disabled with the HugePageStorageMediumSize feature gate on the kubelet and kube-apiserver (--feature-gates=HugePageStorageMediumSize=true).