Configurable Parameters in Kubespray

Generic Ansible variables

You can view facts gathered by Ansible automaticallyhere.

Some variables of note include:

  • ansible_user: user to connect to via SSH
  • ansible_default_ipv4.address: IP address Ansible automatically chooses.Generated based on the output from the command ip -4 route get 8.8.8.8

Common vars that are used in Kubespray

  • calico_version - Specify version of Calico to use
  • calico_cni_version - Specify version of Calico CNI plugin to use
  • docker_version - Specify version of Docker to used (should be quotedstring). Must match one of the keys defined for docker_versioned_pkgin roles/container-engine/docker/vars/*.yml.
  • etcd_version - Specify version of ETCD to use
  • ipip - Enables Calico ipip encapsulation by default
  • kube_network_plugin - Sets k8s network plugin (default Calico)
  • kube_proxy_mode - Changes k8s proxy mode to iptables mode
  • kube_version - Specify a given Kubernetes hyperkube version
  • searchdomains - Array of DNS domains to search when looking up hostnames
  • nameservers - Array of nameservers to use for DNS lookup
  • preinstall_selinux_state - Set selinux state, permitted values are permissive and disabled.

Addressing variables

  • ip - IP to use for binding services (host var)
  • access_ip - IP for other hosts to use to connect to. Often required whendeploying from a cloud, such as OpenStack or GCE and you have separatepublic/floating and private IPs.
  • ansible_default_ipv4.address - Not Kubespray-specific, but it is used if ipand access_ip are undefined
  • loadbalancer_apiserver - If defined, all hosts will connect to thisaddress instead of localhost for kube-masters and kube-master[0] forkube-nodes. See more details in theHA guide.
  • loadbalancer_apiserver_localhost - makes all hosts to connect tothe apiserver internally load balanced endpoint. Mutual exclusive to theloadbalancer_apiserver. See more details in theHA guide.

Cluster variables

Kubernetes needs some parameters in order to get deployed. These are thefollowing default cluster parameters:

  • cluster_name - Name of cluster (default is cluster.local)
  • dns_domain - Name of cluster DNS domain (default is cluster.local)
  • kube_network_plugin - Plugin to use for container networking
  • kube_service_addresses - Subnet for cluster IPs (default is10.233.0.0/18). Must not overlap with kube_pods_subnet
  • kube_pods_subnet - Subnet for Pod IPs (default is 10.233.64.0/18). Must notoverlap with kube_service_addresses.
  • kube_network_node_prefix - Subnet allocated per-node for pod IPs. Remaininbits in kube_pods_subnet dictates how many kube-nodes can be in cluster.
  • skydns_server - Cluster IP for DNS (default is 10.233.0.3)
  • skydns_server_secondary - Secondary Cluster IP for CoreDNS used with coredns_dual deployment (default is 10.233.0.4)
  • cloud_provider - Enable extra Kubelet option if operating inside GCE orOpenStack (default is unset)
  • kube_hostpath_dynamic_provisioner - Required for use of PetSets type inKubernetes
  • kube_feature_gates - A list of key=value pairs that describe feature gates foralpha/experimental Kubernetes features. (defaults is [])
  • authorization_modes - A list of authorization modethat the cluster should be configured for. Defaults to ['Node', 'RBAC'](Node and RBAC authorizers).Note: Node and RBAC are enabled by default. Previously deployed clusters can beconverted to RBAC mode. However, your apps which rely on Kubernetes API willrequire a service account and cluster role bindings. You can override thissetting by setting authorization_modes to [].

Note, if cloud providers have any use of the 10.233.0.0/16, like instances’private addresses, make sure to pick another values for kube_service_addressesand kube_pods_subnet, for example from the 172.18.0.0/16.

DNS variables

By default, hosts are set up with 8.8.8.8 as an upstream DNS server and allother settings from your existing /etc/resolv.conf are lost. Set the followingvariables to match your requirements.

  • upstream_dns_servers - Array of upstream DNS servers configured on host inaddition to Kubespray deployed DNS
  • nameservers - Array of DNS servers configured for use by hosts
  • searchdomains - Array of up to 4 search domains

For more information, see DNSStack.

Other service variables

  • docker_options - Commonly used to set--insecure-registry=myregistry.mydomain:5000
  • http_proxy/https_proxy/no_proxy - Proxy variables for deploying behind aproxy. Note that no_proxy defaults to all internal cluster IPs and hostnamesthat correspond to each node.
  • kubelet_deployment_type - Controls which platform to deploy kubelet on.Available options are host and docker. docker modeis unlikely to work on newer releases. Starting with Kubernetes v1.7series, this now defaults to host. Before v1.7, the default was Docker.This is because of cgroup issues.
  • kubelet_load_modules - For some things, kubelet needs to load kernel modules. For example,dynamic kernel services are needed for mounting persistent volumes into containers. These may not beloaded by preinstall kubernetes processes. For example, ceph and rbd backed volumes. Set this variable totrue to let kubelet load kernel modules.
  • kubelet_cgroup_driver - Allows manual override of thecgroup-driver option for Kubelet. By default autodetection is usedto match Docker configuration.
  • kubelet_rotate_certificates - Auto rotate the kubelet client certificates by requesting new certificatesfrom the kube-apiserver when the certificate expiration approaches.
  • node_labels - Labels applied to nodes via kubelet —node-labels parameter.For example, labels can be set in the inventory as variables or more widely in group_vars.node_labels must be defined as a dict:
    1. node_labels:
    2. label1_name: label1_value
    3. label2_name: label2_value
  • node_taints - Taints applied to nodes via kubelet —register-with-taints parameter.For example, taints can be set in the inventory as variables or more widely in group_vars.node_taints has to be defined as a list of strings in format key=value:effect, e.g.:
    1. node_taints:
    2. - "node.example.com/external=true:NoSchedule"
  • podsecuritypolicy_enabled - When set to true, enables the PodSecurityPolicy admission controller and defines two policies privileged (applying to all resources in kube-system namespace and kubelet) and restricted (applying all other namespaces).Addons deployed in kube-system namespaces are handled.

  • kubernetes_audit - When set to true, enables Auditing.The auditing parameters can be tuned via the following variables (which default values are shown below):

    • audit_log_path: /var/log/audit/kube-apiserver-audit.log
    • audit_log_maxage: 30
    • audit_log_maxbackups: 1
    • audit_log_maxsize: 100
    • audit_policy_file: “{{ kube_config_dir }}/audit-policy/apiserver-audit-policy.yaml”

    By default, the audit_policy_file contains default rules that can be overridden with the audit_policy_custom_rules variable.

Custom flags for Kube Components

For all kube components, custom flags can be passed in. This allows for edge cases where users need changes to the default deployment that may not be applicable to all deployments. This can be done by providing a list of flags. The kubelet_node_custom_flags apply kubelet settings only to nodes and not masters. Example:

  1. kubelet_custom_flags:
  2.   - "--eviction-hard=memory.available<100Mi"
  3.   - "--eviction-soft-grace-period=memory.available=30s"
  4.   - "--eviction-soft=memory.available<300Mi"

The possible vars are:

  • apiserver_custom_flags
  • controller_mgr_custom_flags
  • scheduler_custom_flags
  • kubelet_custom_flags
  • kubelet_node_custom_flags

User accounts

By default, a user with admin rights is created, named kube.The password can be viewed after deployment by looking at the file{{ credentials_dir }}/kube_user.creds (credentials_dir is set to {{ inventory_dir }}/credentials by default). This contains a randomly generatedpassword. If you wish to set your own password, just precreate/modify thisfile yourself or change kube_api_pwd var.