Quotas

CephFS allows quotas to be set on any directory in the system. Thequota can restrict the number of bytes or the number of _files_stored beneath that point in the directory hierarchy.

Limitations

  • Quotas are cooperative and non-adversarial. CephFS quotas rely onthe cooperation of the client who is mounting the file system tostop writers when a limit is reached. A modified or adversarialclient cannot be prevented from writing as much data as it needs.Quotas should not be relied on to prevent filling the system inenvironments where the clients are fully untrusted.

  • Quotas are imprecise. Processes that are writing to the filesystem will be stopped a short time after the quota limit isreached. They will inevitably be allowed to write some amount ofdata over the configured limit. How far over the quota they areable to go depends primarily on the amount of time, not the amountof data. Generally speaking writers will be stopped within 10s ofseconds of crossing the configured limit.

  • _Quotas are implemented in the kernel client 4.17 and higher._Quotas are supported by the userspace client (libcephfs, ceph-fuse).Linux kernel clients >= 4.17 support CephFS quotas but only onmimic+ clusters. Kernel clients (even recent versions) will failto handle quotas on older clusters, even if they may be able to setthe quotas extended attributes.

  • Quotas must be configured carefully when used with path-basedmount restrictions. The client needs to have access to thedirectory inode on which quotas are configured in order to enforcethem. If the client has restricted access to a specific path(e.g., /home/user) based on the MDS capability, and a quota isconfigured on an ancestor directory they do not have access to(e.g., /home), the client will not enforce it. When usingpath-based access restrictions be sure to configure the quota onthe directory the client is restricted too (e.g., /home/user)or something nested beneath it.

  • Snapshot file data which has since been deleted or changed does not counttowards the quota. See also: http://tracker.ceph.com/issues/24284

Configuration

Like most other things in CephFS, quotas are configured using virtualextended attributes:

  • ceph.quota.max_files – file limit

  • ceph.quota.max_bytes – byte limit

If the attributes appear on a directory inode that means a quota isconfigured there. If they are not present then no quota is set onthat directory (although one may still be configured on a parent directory).

To set a quota:

  1. setfattr -n ceph.quota.max_bytes -v 100000000 /some/dir     # 100 MB
  2. setfattr -n ceph.quota.max_files -v 10000 /some/dir         # 10,000 files

To view quota settings:

  1. getfattr -n ceph.quota.max_bytes /some/dir
  2. getfattr -n ceph.quota.max_files /some/dir

Note that if the value of the extended attribute is 0 that meansthe quota is not set.

To remove a quota:

  1. setfattr -n ceph.quota.max_bytes -v 0 /some/dir
  2. setfattr -n ceph.quota.max_files -v 0 /some/dir