Configuration options

Using a configuration file

k0s can be installed without a config file. In that case the default configuration will be used. You can, though, create and run your own non-default configuration (used by the k0s controller nodes).

k0s supports providing only partial configurations. In case of partial configuration is provided, k0s will use the defaults for any missing values.

  1. Generate a yaml config file that uses the default settings.

    1. mkdir -p /etc/k0s
    2. k0s config create > /etc/k0s/k0s.yaml
  2. Modify the new yaml config file according to your needs, refer to Configuration file reference below. You can remove the default values if wanted as k0s supports partial configs too.

  3. Install k0s with your new config file.

    1. sudo k0s install controller -c /etc/k0s/k0s.yaml
  4. If you need to modify your existing configuration later on, you can change your config file also when k0s is running, but remember to restart k0s to apply your configuration changes.

    1. sudo k0s stop
    2. sudo k0s start

Configuring k0s via k0sctl

k0sctl can deploy your configuration options at cluster creation time. Your options should be placed in the spec.k0s.config section of the k0sctl’s configuration file. See the section on how to install k0s via k0sctl and the k0sctl README for more information.

Configuration file reference

CAUTION: As many of the available options affect items deep in the stack, you should fully understand the correlation between the configuration file components and your specific environment before making any changes.

A YAML config file follows, with defaults as generated by the k0s config create command:

  1. apiVersion: k0s.k0sproject.io/v1beta1
  2. kind: ClusterConfig
  3. metadata:
  4. name: k0s
  5. spec:
  6. api:
  7. address: 192.168.68.104
  8. externalAddress: my-lb-address.example.com
  9. k0sApiPort: 9443
  10. port: 6443
  11. sans:
  12. - 192.168.68.104
  13. tunneledNetworkingMode: false
  14. controllerManager: {}
  15. extensions:
  16. helm:
  17. concurrencyLevel: 5
  18. charts: null
  19. repositories: null
  20. storage:
  21. create_default_storage_class: false
  22. type: external_storage
  23. installConfig:
  24. users:
  25. etcdUser: etcd
  26. kineUser: kube-apiserver
  27. konnectivityUser: konnectivity-server
  28. kubeAPIserverUser: kube-apiserver
  29. kubeSchedulerUser: kube-scheduler
  30. konnectivity:
  31. adminPort: 8133
  32. agentPort: 8132
  33. network:
  34. calico: null
  35. clusterDomain: cluster.local
  36. dualStack: {}
  37. kubeProxy:
  38. metricsBindAddress: 0.0.0.0:10249
  39. mode: iptables
  40. kuberouter:
  41. autoMTU: true
  42. hairpin: Enabled
  43. ipMasq: false
  44. metricsPort: 8080
  45. mtu: 0
  46. peerRouterASNs: ""
  47. peerRouterIPs: ""
  48. nodeLocalLoadBalancing:
  49. enabled: false
  50. envoyProxy:
  51. apiServerBindPort: 7443
  52. image:
  53. image: docker.io/envoyproxy/envoy-distroless
  54. version: v1.24.1
  55. konnectivityServerBindPort: 7132
  56. type: EnvoyProxy
  57. podCIDR: 10.244.0.0/16
  58. provider: kuberouter
  59. serviceCIDR: 10.96.0.0/12
  60. scheduler: {}
  61. storage:
  62. etcd:
  63. externalCluster: null
  64. peerAddress: 192.168.68.104
  65. type: etcd
  66. telemetry:
  67. enabled: true
  68. featureGates:
  69. - name: feature_XXX
  70. enabled: true
  71. components: ["kubelet", "kube-api", "kube-scheduler"]
  72. - name: feature_YYY
  73. enabled: true
  74. -
  75. name: feature_ZZZ
  76. enabled: false

spec Key Detail

spec.api

ElementDescription
externalAddressThe loadbalancer address (for k0s controllers running behind a loadbalancer). Configures all cluster components to connect to this address and also configures this address for use when joining new nodes to the cluster.
addressLocal address on which to bind an API. Also serves as one of the addresses pushed on the k0s create service certificate on the API. Defaults to first non-local address found on the node.
sansList of additional addresses to push to API servers serving the certificate.
extraArgsMap of key-values (strings) for any extra arguments to pass down to Kubernetes api-server process.
port¹Custom port for kube-api server to listen on (default: 6443)
k0sApiPort¹Custom port for k0s-api server to listen on (default: 9443)
tunneledNetworkingModeWhether to tunnel Kubernetes access from worker nodes via local port forwarding. (default: false)

¹ If port and k0sApiPort are used with the externalAddress element, the loadbalancer serving at externalAddress must listen on the same ports.

spec.storage

ElementDescription
typeType of the data store (valid values:etcd or kine). Note: Type etcd will cause k0s to create and manage an elastic etcd cluster within the controller nodes.
etcd.peerAddressNode address used for etcd cluster peering.
etcd.extraArgsMap of key-values (strings) for any extra arguments to pass down to etcd process.
kine.dataSourcekine datasource URL.

spec.network

ElementDescription
providerNetwork provider (valid values: calico, kuberouter, or custom). For custom, you can push any network provider (default: kuberouter). Be aware that it is your responsibility to configure all of the CNI-related setups, including the CNI provider itself and all necessary host levels setups (for example, CNI binaries). Note: Once you initialize the cluster with a network provider the only way to change providers is through a full cluster redeployment.
podCIDRPod network CIDR to use in the cluster.
serviceCIDRNetwork CIDR to use for cluster VIP services.
clusterDomainCluster Domain to be passed to the kubelet and the coredns configuration.

spec.network.calico

ElementDescription
modevxlan (default), ipip or bird
overlayOverlay mode: Always (default), CrossSubnet or Never (requires mode=vxlan to disable calico overlay-network).
vxlanPortThe UDP port for VXLAN (default: 4789).
vxlanVNIThe virtual network ID for VXLAN (default: 4096).
mtuMTU for overlay network (default: 0, which causes Calico to detect optimal MTU during bootstrap).
wireguardEnable wireguard-based encryption (default: false). Your host system must be wireguard ready (refer to the Calico documentation for details).
flexVolumeDriverPathThe host path for Calicos flex-volume-driver(default: /usr/libexec/k0s/kubelet-plugins/volume/exec/nodeagent~uds). Change this path only if the default path is unwriteable (refer to Project Calico Issue #2712 for details). Ideally, you will pair this option with a custom volumePluginDir in the profile you use for your worker nodes.
ipAutodetectionMethodUse to force Calico to pick up the interface for pod network inter-node routing (default: “”, meaning not set, so that Calico will instead use its defaults). For more information, refer to the Calico documentation.
envVarsMap of key-values (strings) for any calico-node environment variable.

spec.network.calico.envVars

Environment variable’s value must be string, e.g.:

  1. spec:
  2. network:
  3. provider: calico
  4. calico:
  5. envVars:
  6. TEST_BOOL_VAR: "true"
  7. TEST_INT_VAR: "42"
  8. TEST_STRING_VAR: test

K0s runs Calico with some predefined vars, which can be overwritten by setting new value in spec.network.calico.envVars:

  1. CALICO_IPV4POOL_CIDR: "{{ spec.network.podCIDR }}"
  2. CALICO_DISABLE_FILE_LOGGING: "true"
  3. FELIX_DEFAULTENDPOINTTOHOSTACTION: "ACCEPT"
  4. FELIX_LOGSEVERITYSCREEN: "info"
  5. FELIX_HEALTHENABLED: "true"
  6. FELIX_PROMETHEUSMETRICSENABLED: "true"
  7. FELIX_FEATUREDETECTOVERRIDE: "ChecksumOffloadBroken=true"

FELIX_FEATUREDETECTOVERRIDE: ChecksumOffloadBroken=true disables VXLAN offloading because of projectcalico/calico#4727.

In SingleStack mode there are additional vars:

  1. FELIX_IPV6SUPPORT: "false"

In DualStack mode there are additional vars:

  1. CALICO_IPV6POOL_NAT_OUTGOING: "true"
  2. FELIX_IPV6SUPPORT: "true"
  3. IP6: "autodetect"
  4. CALICO_IPV6POOL_CIDR: "{{ spec.network.dualStack.IPv6podCIDR }}"

spec.network.kuberouter

ElementDescription
autoMTUAutodetection of used MTU (default: true).
mtuOverride MTU setting, if autoMTU must be set to false).
metricsPortKube-router metrics server port. Set to 0 to disable metrics (default: 8080).
peerRouterIPsComma-separated list of global peer addresses.
peerRouterASNsComma-separated list of global peer ASNs.
hairpinHairpin mode, supported modes Enabled: enabled cluster wide, Allowed: must be allowed per service using annotations, Disabled: doesn’t work at all (default: Enabled)
hairpinModeDeprecated Use hairpin instead. If both hairpin and hairpinMode are defined, this is ignored. If only hairpinMode is configured explicitly activates hairpinMode (https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#hairpin-mode).
ipMasqIP masquerade for traffic originating from the pod network, and destined outside of it (default: false)

Note: Kube-router allows many networking aspects to be configured per node, service, and pod (for more information, refer to the Kube-router user guide).

spec.network.kubeProxy

ElementDescription
disabledDisable kube-proxy altogether (default: false).
modeKube proxy operating mode, supported modes iptables, ipvs, userspace (default: iptables)
iptablesKube proxy iptables settings
ipvsKube proxy ipvs settings

Default kube-proxy iptables settings:

  1. iptables:
  2. masqueradeAll: false
  3. masqueradeBit: null
  4. minSyncPeriod: 0s
  5. syncPeriod: 0s

Default kube-proxy ipvs settings:

  1. ipvs:
  2. excludeCIDRs: null
  3. minSyncPeriod: 0s
  4. scheduler: ""
  5. strictARP: false
  6. syncPeriod: 0s
  7. tcpFinTimeout: 0s
  8. tcpTimeout: 0s
  9. udpTimeout: 0s

spec.network.nodeLocalLoadBalancing

Configuration options related to k0s’s node-local load balancing feature.

Note: This feature is experimental! Expect instabilities and/or breaking changes.

ElementDescription
enabledIndicates if node-local load balancing should be used to access Kubernetes API servers from worker nodes. Default: false.
typeThe type of the node-local load balancer to deploy on worker nodes. Default: EnvoyProxy. (This is the only option for now.)
envoyProxyConfiguration options related to the “EnvoyProxy” type of load balancing.
spec.network.nodeLocalLoadBalancing.envoyProxy

Configuration options required for using Envoy as the backing implementation for node-local load balancing.

Note: This type of load balancing is not supported on ARMv7 workers.

ElementDescription
imageThe OCI image that’s being used for the Envoy Pod.
imagePullPolicyThe pull policy being used used for the Envoy Pod. Defaults to spec.images.default_pull_policy if omitted.
apiServerBindPortPort number on which to bind the Envoy load balancer for the Kubernetes API server to on a worker’s loopback interface. Default: 7443.
konnectivityServerBindPortPort number on which to bind the Envoy load balancer for the konnectivity server to on a worker’s loopback interface. Default: 7132.

spec.controllerManager

ElementDescription
extraArgsMap of key-values (strings) for any extra arguments you want to pass down to the Kubernetes controller manager process.

spec.scheduler

ElementDescription
extraArgsMap of key-values (strings) for any extra arguments you want to pass down to Kubernetes scheduler process.

spec.workerProfiles

Worker profiles are used to manage worker-specific configuration in a centralized manner. A ConfigMap is generated for each worker profile. Based on the --profile argument given to the k0s worker, the configuration in the corresponding ConfigMap is is picked up during startup.

The worker profiles are defined as an array. Each element has following properties:

PropertyDescription
nameString; name to use as profile selector for the worker process
valuesObject; Kubelet configuration overrides, see below for details

spec.workerProfiles[].values (Kubelet configuration overrides)

The Kubelet configuration overrides of a profile override the defaults defined by k0s.

Note that there are several fields that cannot be overridden:

  • clusterDNS
  • clusterDomain
  • apiVersion
  • kind
  • staticPodURL

spec.featureGates

Available components are:

  • kube-apiserver
  • kube-controller-manager
  • kubelet
  • kube-scheduler
  • kube-proxy

If components are omitted, propagates to all kube components.

Modifies extraArgs.

Example

  1. spec:
  2. featureGates:
  3. - name: feature-gate-0
  4. enabled: true
  5. components: ["kube-apiserver", "kube-controller-manager", "kubelet", "kube-scheduler"]
  6. - name: feature-gate-1
  7. enabled: true
  8. - name: feature-gate-2
  9. enabled: false

Kubelet feature gates example

The below is an example of a k0s config with feature gates enabled:

  1. spec:
  2. featureGates:
  3. - name: DevicePlugins
  4. enabled: true
  5. components: ["kubelet"]
  6. - name: Accelerators
  7. enabled: true
  8. components: ["kubelet"]
  9. - name: AllowExtTrafficLocalEndpoints
  10. enabled: false

Configuration examples

Custom volumePluginDir
  1. spec:
  2. workerProfiles:
  3. - name: custom-pluginDir
  4. values:
  5. volumePluginDir: /var/libexec/k0s/kubelet-plugins/volume/exec
Eviction Policy
  1. spec:
  2. workerProfiles:
  3. - name: custom-eviction
  4. values:
  5. evictionHard:
  6. memory.available: "500Mi"
  7. nodefs.available: "1Gi"
  8. imagefs.available: "100Gi"
  9. evictionMinimumReclaim:
  10. memory.available: "0Mi"
  11. nodefs.available: "500Mi"
  12. imagefs.available: "2Gi"
Unsafe Sysctls
  1. spec:
  2. workerProfiles:
  3. - name: custom-eviction
  4. values:
  5. allowedUnsafeSysctls:
  6. - fs.inotify.max_user_instances

spec.images

Nodes under the images key all have the same basic structure:

  1. spec:
  2. images:
  3. coredns:
  4. image: quay.io/coredns/coredns
  5. version: v1.7.0

If you want the list of default images and their versions to be included, use k0s config create --include-images.

Available keys

  • spec.images.konnectivity
  • spec.images.metricsserver
  • spec.images.kubeproxy
  • spec.images.coredns
  • spec.images.calico.cni
  • spec.images.calico.flexvolume
  • spec.images.calico.node
  • spec.images.calico.kubecontrollers
  • spec.images.kuberouter.cni
  • spec.images.kuberouter.cniInstaller
  • spec.images.repository¹

¹ If spec.images.repository is set and not empty, every image will be pulled from images.repository

If spec.images.default_pull_policy is set and not empty, it will be used as a pull policy for each bundled image.

Image example

  1. images:
  2. repository: "my.own.repo"
  3. konnectivity:
  4. image: calico/kube-controllers
  5. version: v3.16.2
  6. metricsserver:
  7. image: registry.k8s.io/metrics-server/metrics-server
  8. version: v0.6.4

In the runtime the image names are calculated as my.own.repo/calico/kube-controllers:v3.16.2 and my.own.repo/metrics-server/metrics-server:v0.6.4. This only affects the the imgages pull location, and thus omitting an image specification here will not disable component deployment.

spec.extensions.helm

spec.extensions.helm is the config file key in which you configure the list of Helm repositories and charts to deploy during cluster bootstrap (for more information, refer to Helm Charts).

spec.extensions.storage

spec.extensions.storage controls bundled storage provider. The default value external makes no storage deployed.

To enable embedded host-local storage provider use the following configuration:

  1. spec:
  2. extensions:
  3. storage:
  4. type: openebs_local_storage

spec.konnectivity

The spec.konnectivity key is the config file key in which you configure Konnectivity-related settings.

  • agentPort agent port to listen on (default 8132)
  • adminPort admin port to listen on (default 8133)

spec.telemetry

To improve the end-user experience k0s is configured by defaul to collect telemetry data from clusters and send it to the k0s development team. To disable the telemetry function, change the enabled setting to false.

The telemetry interval is ten minutes.

  1. spec:
  2. telemetry:
  3. enabled: true

Disabling controller components

k0s allows to completely disable some of the system components. This allows users to build a minimal Kubernetes control plane and use what ever components they need to fulfill their need for the control plane. Disabling the system components happens through a command line flag for the controller process:

  1. --disable-components strings disable components (valid items: api-config,autopilot,control-api,coredns,csr-approver,endpoint-reconciler,helm,konnectivity-server,kube-controller-manager,kube-proxy,kube-scheduler,metrics-server,network-provider,node-role,system-rbac,worker-config)

Note: As of k0s 1.26, the kubelet-config component has been replaced by the worker-config component. k0s will issue a warning when the old component name is being used. It is scheduled for removal in k0s 1.27. Please update to the new component name.

If you use k0sctl, just add the flag when installing the cluster for the first controller at spec.hosts.installFlags in the config file like e.g.:

  1. spec:
  2. hosts:
  3. - role: controller
  4. installFlags:
  5. - --disable-components=metrics-server

As seen from the component list, the only always-on component is the Kubernetes api-server, without that k0s serves no purpose.