Setting Quotas

Overview

A resource quota, defined by a **ResourceQuota** object, provides constraints that limit aggregate resource consumption per project. It can limit the quantity of objects that can be created in a project by type, as well as the total amount of compute resources and storage that may be consumed by resources in that project.

Resources Managed by Quota

The following describes the set of compute resources and object types that may be managed by a quota.

You can configure an object count quota for these standard namespaced resource types using the count/<resource>.<group> syntax while creating a quota.

$ oc create quota <name> --hard=count/<resource>.<group>=<quota> (1)

Setting Resource Quota for Extended Resources

Overcommitment of resources is not allowed for extended resources, so you must specify requests and limits for the same extended resource in a quota. Currently, only quota items with the prefix requests. are allowed for extended resources. The following is an example scenario of how to set resource quota for the GPU resource nvidia.com/gpu.

Procedure

1. Determine how many GPUs are available on a node in your cluster. For example:

   # oc describe node ip-172-31-27-209.us-west-2.compute.internal | egrep 'Capacity|Allocatable|gpu'
                       openshift.com/gpu-accelerator=true
   Capacity:
    nvidia.com/gpu:  2
   Allocatable:
    nvidia.com/gpu:  2
     nvidia.com/gpu  0           0

   In this example, 2 GPUs are available.

2. Set a quota in the namespace nvidia. In this example, the quota is 1:

   # cat gpu-quota.yaml
   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: gpu-quota
     namespace: nvidia
   spec:
     hard:
       requests.nvidia.com/gpu: 1

3. Create the quota:

   # oc create -f gpu-quota.yaml
   resourcequota/gpu-quota created

4. Verify that the namespace has the correct quota set:

   # oc describe quota gpu-quota -n nvidia
   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  0     1

5. Run a pod that asks for a single GPU:

   # oc create pod gpu-pod.yaml

   apiVersion: v1
   kind: Pod
   metadata:
     generateName: gpu-pod-
     namespace: nvidia
   spec:
     restartPolicy: OnFailure
     containers:
     - name: rhel7-gpu-pod
       image: rhel7
       env:
         - name: NVIDIA_VISIBLE_DEVICES
           value: all
         - name: NVIDIA_DRIVER_CAPABILITIES
           value: "compute,utility"
         - name: NVIDIA_REQUIRE_CUDA
           value: "cuda>=5.0"
       command: ["sleep"]
       args: ["infinity"]
       resources:
         limits:
           nvidia.com/gpu: 1

6. Verify that the pod is running:

   # oc get pods
   NAME              READY     STATUS      RESTARTS   AGE
   gpu-pod-s46h7     1/1       Running     0          1m

7. Verify that the quota Used counter is correct:

   # oc describe quota gpu-quota -n nvidia
   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  1     1

8. Attempt to create a second GPU pod in the nvidia namespace. This is technically available on the node because it has 2 GPUs:

   # oc create -f gpu-pod.yaml
   Error from server (Forbidden): error when creating "gpu-pod.yaml": pods "gpu-pod-f7z2w" is forbidden: exceeded quota: gpu-quota, requested: requests.nvidia.com/gpu=1, used: requests.nvidia.com/gpu=1, limited: requests.nvidia.com/gpu=1

   This Forbidden error message is expected because you have a quota of 1 GPU and this pod tried to allocate a second GPU, which exceeds its quota.

Quota Scopes

Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches the intersection of enumerated scopes.

Adding a scope to a quota restricts the set of resources to which that quota can apply. Specifying a resource outside of the allowed set results in a validation error.

A BestEffort scope restricts a quota to limiting the following resources:

• **pods**

A Terminating, NotTerminating, and NotBestEffort scope restricts a quota to tracking the following resources:

• **pods**

• **memory**

• **requests.memory**

• **limits.memory**

• **cpu**

• **requests.cpu**

• **limits.cpu**

• **ephemeral-storage**

• **requests.ephemeral-storage**

• **limits.ephemeral-storage**

Quota Enforcement

After a resource quota for a project is first created, the project restricts the ability to create any new resources that may violate a quota constraint until it has calculated updated usage statistics.

After a quota is created and usage statistics are updated, the project accepts the creation of new content. When you create or modify resources, your quota usage is incremented immediately upon the request to create or modify the resource.

When you delete a resource, your quota use is decremented during the next full recalculation of quota statistics for the project. A configurable amount of time determines how long it takes to reduce quota usage statistics to their current observed system value.

If project modifications exceed a quota usage limit, the server denies the action, and an appropriate error message is returned to the user explaining the quota constraint violated, and what their currently observed usage stats are in the system.

Requests Versus Limits

When allocating compute resources, each container may specify a request and a limit value each for CPU, memory, and ephemeral storage. Quotas can restrict any of these values.

If the quota has a value specified for **requests.cpu** or **requests.memory**, then it requires that every incoming container make an explicit request for those resources. If the quota has a value specified for **limits.cpu** or **limits.memory**, then it requires that every incoming container specify an explicit limit for those resources.

Sample Resource Quota Definitions

core-object-counts.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: core-object-counts
spec:
  hard:
    configmaps: "10" (1)
    persistentvolumeclaims: "4" (2)
    replicationcontrollers: "20" (3)
    secrets: "10" (4)
    services: "10" (5)

openshift-object-counts.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: openshift-object-counts
spec:
  hard:
    openshift.io/imagestreams: "10" (1)

compute-resources.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    pods: "4" (1)
    requests.cpu: "1" (2)
    requests.memory: 1Gi (3)
    requests.ephemeral-storage: 2Gi (4)
    limits.cpu: "2" (5)
    limits.memory: 2Gi (6)
    limits.ephemeral-storage: 4Gi (7)

besteffort.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: besteffort
spec:
  hard:
    pods: "1" (1)
  scopes:
  - BestEffort (2)

compute-resources-long-running.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources-long-running
spec:
  hard:
    pods: "4" (1)
    limits.cpu: "4" (2)
    limits.memory: "2Gi" (3)
    limits.ephemeral-storage: "4Gi" (4)
  scopes:
  - NotTerminating (5)

compute-resources-time-bound.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources-time-bound
spec:
  hard:
    pods: "2" (1)
    limits.cpu: "1" (2)
    limits.memory: "1Gi" (3)
    limits.ephemeral-storage: "1Gi" (4)
  scopes:
  - Terminating (5)

storage-consumption.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-consumption
spec:
  hard:
    persistentvolumeclaims: "10" (1)
    requests.storage: "50Gi" (2)
    gold.storageclass.storage.k8s.io/requests.storage: "10Gi" (3)
    silver.storageclass.storage.k8s.io/requests.storage: "20Gi" (4)
    silver.storageclass.storage.k8s.io/persistentvolumeclaims: "5" (5)
    bronze.storageclass.storage.k8s.io/requests.storage: "0" (6)
    bronze.storageclass.storage.k8s.io/persistentvolumeclaims: "0" (7)

Creating a Quota

To create a quota, first define the quota in a file, such as the examples in Sample Resource Quota Definitions. Then, create using that file to apply it to a project:

$ oc create -f <resource_quota_definition> [-n <project_name>]

For example:

$ oc create -f core-object-counts.yaml -n demoproject

Creating Object Count Quotas

You can create an object count quota for all OKD standard namespaced resource types, such as BuildConfig, and DeploymentConfig. An object quota count places a defined quota on all standard namespaced resource types.

When using a resource quota, an object is charged against the quota if it exists in server storage. These types of quotas are useful to protect against exhaustion of storage resources.

To configure an object count quota for a resource, run the following command:

$ oc create quota <name> --hard=count/<resource>.<group>=<quota>,count/<resource>.<group>=<quota>

For example:

$ oc create quota test --hard=count/deployments.extensions=2,count/replicasets.extensions=4,count/pods=3,count/secrets=4
resourcequota "test" created
$ oc describe quota test
Name:                         test
Namespace:                    quota
Resource                      Used  Hard
--------                      ----  ----
count/deployments.extensions  0     2
count/pods                    0     3
count/replicasets.extensions  0     4
count/secrets                 0     4

This example limits the listed resources to the hard limit in each project in the cluster.

Viewing a Quota

You can view usage statistics related to any hard limits defined in a project’s quota by navigating in the web console to the project’s Quota page.

You can also use the CLI to view quota details:

1. First, get the list of quotas defined in the project. For example, for a project called demoproject:

   $ oc get quota -n demoproject
   NAME                AGE
   besteffort          11m
   compute-resources   2m
   core-object-counts  29m

2. Then, describe the quota you are interested in, for example the core-object-counts quota:

   $ oc describe quota core-object-counts -n demoproject
   Name:            core-object-counts
   Namespace:        demoproject
   Resource        Used    Hard
   --------        ----    ----
   configmaps        3    10
   persistentvolumeclaims    0    4
   replicationcontrollers    3    20
   secrets            9    10
   services        2    10

Configuring Quota Synchronization Period

When a set of resources are deleted, the synchronization time frame of resources is determined by the **resource-quota-sync-period** setting in the /etc/origin/master/master-config.yaml file.

Before quota usage is restored, a user may encounter problems when attempting to reuse the resources. You can change the **resource-quota-sync-period** setting to have the set of resources regenerate at the desired amount of time (in seconds) and for the resources to be available again:

kubernetesMasterConfig:
  apiLevels:
  - v1beta3
  - v1
  apiServerArguments: null
  controllerArguments:
    resource-quota-sync-period:
      - "10s"

After making any changes, restart the master services to apply them.

# master-restart api
# master-restart controllers

Adjusting the regeneration time can be helpful for creating resources and determining resource usage when automation is used.

Accounting for Quota in Deployment Configurations

If a quota has been defined for your project, see Deployment Resources for considerations on any deployment configurations.

Require Explicit Quota to Consume a Resource

If a resource is not managed by quota, a user has no restriction on the amount of resource that can be consumed. For example, if there is no quota on storage related to the gold storage class, the amount of gold storage a project can create is unbounded.

For high-cost compute or storage resources, administrators may want to require an explicit quota be granted in order to consume a resource. For example, if a project was not explicitly given quota for storage related to the gold storage class, users of that project would not be able to create any storage of that type.

In order to require explicit quota to consume a particular resource, the following stanza should be added to the master-config.yaml.

admissionConfig:
  pluginConfig:
    ResourceQuota:
      configuration:
        apiVersion: resourcequota.admission.k8s.io/v1alpha1
        kind: Configuration
        limitedResources:
        - resource: persistentvolumeclaims (1)
        matchContains:
        - gold.storageclass.storage.k8s.io/requests.storage (2)

In the above example, the quota system will intercept every operation that creates or updates a PersistentVolumeClaim. It checks what resources understood by quota would be consumed, and if there is no covering quota for those resources in the project, the request is denied. In this example, if a user creates a PersistentVolumeClaim that uses storage associated with the gold storage class, and there is no matching quota in the project, the request is denied.

Known Issues

• Invalid objects can cause quota resources for a project to become exhausted. Quota is incremented in admission prior to validation of the resource. As a result, quota can be incremented even if the pod is not ultimately persisted. This will be resolved in a future release. (BZ1485375)