Guide for Running Windows Containers in Kubernetes

This page provides a walkthrough for some steps you can follow to run Windows containers using Kubernetes. The page also highlights some Windows specific functionality within Kubernetes.

It is important to note that creating and deploying services and workloads on Kubernetes behaves in much the same way for Linux and Windows containers. The kubectl commands to interface with the cluster are identical. The examples in this page are provided to jumpstart your experience with Windows containers.

Objectives

Configure an example deployment to run Windows containers on a Windows node.

Before you begin

You should already have access to a Kubernetes cluster that includes a worker node running Windows Server.

Getting Started: Deploying a Windows workload

The example YAML file below deploys a simple webserver application running inside a Windows container.

Create a manifest named win-webserver.yaml with the contents below:

  1. ---
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: win-webserver
  6.   labels:
  7.     app: win-webserver
  8. spec:
  9.   ports:
  10.     # the port that this service should serve on
  11.     - port: 80
  12.       targetPort: 80
  13.   selector:
  14.     app: win-webserver
  15.   type: NodePort
  16. ---
  17. apiVersion: apps/v1
  18. kind: Deployment
  19. metadata:
  20.   labels:
  21.     app: win-webserver
  22.   name: win-webserver
  23. spec:
  24.   replicas: 2
  25.   selector:
  26.     matchLabels:
  27.       app: win-webserver
  28.   template:
  29.     metadata:
  30.       labels:
  31.         app: win-webserver
  32.       name: win-webserver
  33.     spec:
  34.      containers:
  35.       - name: windowswebserver
  36.         image: mcr.microsoft.com/windows/servercore:ltsc2019
  37.         command:
  38.         - powershell.exe
  39.         - -command
  40.         - "<#code used from https://gist.github.com/19WAS85/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ;  ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus)  } ; "
  41.      nodeSelector:
  42.       kubernetes.io/os: windows

Note: Port mapping is also supported, but for simplicity this example exposes port 80 of the container directly to the Service.

  1. Check that all nodes are healthy:

    1. kubectl get nodes

  2. Deploy the service and watch for pod updates:

    1. kubectl apply -f win-webserver.yaml
    2. kubectl get pods -o wide -w

    When the service is deployed correctly both Pods are marked as Ready. To exit the watch command, press Ctrl+C.

  3. Check that the deployment succeeded. To verify:

    • Several pods listed from the Linux control plane node, use kubectl get pods
    • Node-to-pod communication across the network, curl port 80 of your pod IPs from the Linux control plane node to check for a web server response
    • Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node) using kubectl exec
    • Service-to-pod communication, curl the virtual service IP (seen under kubectl get services) from the Linux control plane node and from individual pods
    • Service discovery, curl the service name with the Kubernetes default DNS suffix
    • Inbound connectivity, curl the NodePort from the Linux control plane node or machines outside of the cluster
    • Outbound connectivity, curl external IPs from inside the pod using kubectl exec

Note: Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack. Only Windows pods are able to access service IPs.

Observability

Capturing logs from workloads

Logs are an important element of observability; they enable users to gain insights into the operational aspect of workloads and are a key ingredient to troubleshooting issues. Because Windows containers and workloads inside Windows containers behave differently from Linux containers, users had a hard time collecting logs, limiting operational visibility. Windows workloads for example are usually configured to log to ETW (Event Tracing for Windows) or push entries to the application event log. LogMonitor, an open source tool by Microsoft, is the recommended way to monitor configured log sources inside a Windows container. LogMonitor supports monitoring event logs, ETW providers, and custom application logs, piping them to STDOUT for consumption by kubectl logs <pod>.

Follow the instructions in the LogMonitor GitHub page to copy its binaries and configuration files to all your containers and add the necessary entrypoints for LogMonitor to push your logs to STDOUT.

Configuring container user

Using configurable Container usernames

Windows containers can be configured to run their entrypoints and processes with different usernames than the image defaults. Learn more about it here.

Managing Workload Identity with Group Managed Service Accounts

Windows container workloads can be configured to use Group Managed Service Accounts (GMSA). Group Managed Service Accounts are a specific type of Active Directory account that provide automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers. Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA. Learn more about configuring and using GMSA for Windows containers here.

Taints and tolerations

Users need to use some combination of taint and node selectors in order to schedule Linux and Windows workloads to their respective OS-specific nodes. The recommended approach is outlined below, with one of its main goals being that this approach should not break compatibility for existing Linux workloads.

You can (and should) set .spec.os.name for each Pod, to indicate the operating system that the containers in that Pod are designed for. For Pods that run Linux containers, set .spec.os.name to linux. For Pods that run Windows containers, set .spec.os.name to windows.

Note: If you are running a version of Kubernetes older than 1.24, you may need to enable the IdentifyPodOS feature gate to be able to set a value for .spec.pod.os.

The scheduler does not use the value of .spec.os.name when assigning Pods to nodes. You should use normal Kubernetes mechanisms for assigning pods to nodes to ensure that the control plane for your cluster places pods onto nodes that are running the appropriate operating system.

The .spec.os.name value has no effect on the scheduling of the Windows pods, so taints and tolerations (or node selectors) are still required to ensure that the Windows pods land onto appropriate Windows nodes.

Ensuring OS-specific workloads land on the appropriate container host

Users can ensure Windows containers can be scheduled on the appropriate host using taints and tolerations. All Kubernetes nodes running Kubernetes 1.29 have the following default labels:

  • kubernetes.io/os = [windows|linux]
  • kubernetes.io/arch = [amd64|arm64|…]

If a Pod specification does not specify a nodeSelector such as "kubernetes.io/os": windows, it is possible the Pod can be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice for Kubernetes 1.29 is to use a nodeSelector.

However, in many cases users have a pre-existing large number of deployments for Linux containers, as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with operators. In those situations, you may be hesitant to make the configuration change to add nodeSelector fields to all Pods and Pod templates. The alternative is to use taints. Because the kubelet can set taints during registration, it could easily be modified to automatically add a taint when running on Windows only.

For example: --register-with-taints='os=windows:NoSchedule'

By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). In order for a Windows Pod to be scheduled on a Windows node, it would need both the nodeSelector and the appropriate matching toleration to choose Windows.

  1. nodeSelector:
  2.     kubernetes.io/os: windows
  3.     node.kubernetes.io/windows-build: '10.0.17763'
  4. tolerations:
  5.     - key: "os"
  6.       operator: "Equal"
  7.       value: "windows"
  8.       effect: "NoSchedule"

Handling multiple Windows versions in the same cluster

The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows Server versions in the same cluster, then you should set additional node labels and nodeSelector fields.

Kubernetes automatically adds a label, node.kubernetes.io/windows-build to simplify this.

This label reflects the Windows major, minor, and build number that need to match for compatibility. Here are values used for each Windows Server version:

Product NameVersion
Windows Server 201910.0.17763
Windows Server 202210.0.20348

Simplifying with RuntimeClass

RuntimeClass can be used to simplify the process of using taints and tolerations. A cluster administrator can create a RuntimeClass object which is used to encapsulate these taints and tolerations.

  1. Save this file to runtimeClasses.yml. It includes the appropriate nodeSelector for the Windows OS, architecture, and version.

    1. ---
    2. apiVersion: node.k8s.io/v1
    3. kind: RuntimeClass
    4. metadata:
    5.   name: windows-2019
    6. handler: example-container-runtime-handler
    7. scheduling:
    8.   nodeSelector:
    9.     kubernetes.io/os: 'windows'
    10.     kubernetes.io/arch: 'amd64'
    11.     node.kubernetes.io/windows-build: '10.0.17763'
    12.   tolerations:
    13.   - effect: NoSchedule
    14.     key: os
    15.     operator: Equal
    16.     value: "windows"

  2. Run kubectl create -f runtimeClasses.yml using as a cluster administrator

  3. Add runtimeClassName: windows-2019 as appropriate to Pod specs

    For example:

    1. ---
    2. apiVersion: apps/v1
    3. kind: Deployment
    4. metadata:
    5.   name: iis-2019
    6.   labels:
    7.     app: iis-2019
    8. spec:
    9.   replicas: 1
    10.   template:
    11.     metadata:
    12.       name: iis-2019
    13.       labels:
    14.         app: iis-2019
    15.     spec:
    16.       runtimeClassName: windows-2019
    17.       containers:
    18.       - name: iis
    19.         image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
    20.         resources:
    21.           limits:
    22.             cpu: 1
    23.             memory: 800Mi
    24.           requests:
    25.             cpu: .1
    26.             memory: 300Mi
    27.         ports:
    28.           - containerPort: 80
    29.  selector:
    30.     matchLabels:
    31.       app: iis-2019
    32. ---
    33. apiVersion: v1
    34. kind: Service
    35. metadata:
    36.   name: iis
    37. spec:
    38.   type: LoadBalancer
    39.   ports:
    40.   - protocol: TCP
    41.     port: 80
    42.   selector:
    43.     app: iis-2019