OpenYurt 安装前置条件

1.背景说明

OpenYurt为适应边端环境，需要用户对K8S做一些调整，如：CoreDNS，KubeProxy等。

2. CoreDNS调整

一般场景下，CoreDNS是以Deployment形式部署，在边端场景下，域名解析请求无法跨NodePool，所以CoreDNS需要以Daemonset或者YurtAppDaemon形式部署，以实现将hostname解析为tunnelserver地址。

2.1 CoreDNS 配置修改

修改kube-system namespace下的ConfigMap coredns，增加如下内容：

        hosts /etc/edge/tunnel-nodes { # 增加hosts插件
            reload 300ms
            fallthrough
        }

修改后效果如下：

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        log . {
          class denial success
        }
        health {
           lameduck 5s
        }
        ready
        hosts /etc/edge/tunnel-nodes { # 增加hosts插件
            reload 300ms
            fallthrough
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system

2.2 CoreDNS 支持服务拓扑

增加annotation，利用openyurt的机制实现边缘服务选择。

# 利用openyurt实现endpoint过滤
kubectl annotate svc kube-dns -n kube-system openyurt.io/topologyKeys='openyurt.io/nodepool'

修改后效果：

apiVersion: v1
kind: Service
metadata:
  annotations:
    openyurt.io/topologyKeys: openyurt.io/nodepool
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  creationTimestamp: "2022-02-14T10:13:37Z"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: KubeDNS
  name: kube-dns
  namespace: kube-system
  resourceVersion: "65474309"
  selfLink: /api/v1/namespaces/kube-system/services/kube-dns
  uid: ee23195f-44c3-4c70-99e2-aff4d5cf0ae1
spec:
  clusterIP: 10.254.0.10
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  - name: metrics
    port: 9153
    protocol: TCP
    targetPort: 9153
  selector:
    k8s-app: kube-dns
  sessionAffinity: None
  type: ClusterIP

2.2 CoreDNS DaemonSet部署

如果CoreDNS原本使用DaemonSet部署，可以手工进行如下调整：

1）可以调整CoreDNS的镜像为自己的版本；

2）需要挂载Volume ConfigMap yurt-tunnel-nodes；

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
        image: registry.aliyuncs.com/google_containers/coredns:1.7.0
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        volumeMounts:
        - mountPath: /etc/coredns
          name: config-volume
          readOnly: true
        - mountPath: /etc/edge
          name: hosts
          readOnly: true
      dnsPolicy: Default
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccount: coredns
      serviceAccountName: coredns
      tolerations:
      - operator: Exists
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: Corefile
            path: Corefile
          name: coredns
        name: config-volume
      - configMap:
          defaultMode: 420
          name: yurt-tunnel-nodes
        name: hosts

2.4 减少CoreDNS Deployment 副本数

如果k8s不是用Deployment部署，可以不进行操作。

kubectl scale --replicas=0 deployment/coredns -n kube-system

3. KubeProxy调整

kubeadm部署的k8s集群会为KubeProxy生成kubeconfig配置，在不配置Service Topology 和 Topology Aware Hints 情况下，KubeProxy使用这个kubeconfig拿到的endpoints是全量的。

云边端场景下，边缘节点间很有可能无法互通，因此需要endpoints基于nodepool进行拓扑。直接将kube-proxy的kubeconfig配置删除，将apiserver请求经过yurthub即可解决服务拓扑问题。

KubeProxy支持流量拓扑

kubectl edit cm -n kube-system kube-proxy

注释掉config.conf文件下的clientConnection.kubeconfig，修改完后效果如下：

apiVersion: v1
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    bindAddress: 0.0.0.0
    bindAddressHardFail: false
    clientConnection:
      acceptContentTypes: ""
      burst: 0
      contentType: ""
      #kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 0
    clusterCIDR: 100.64.0.0/10
    configSyncPeriod: 0s
// 省略

重启KubeProxy Pod

为使上述配置生效，需要重启kubeproxy的pod，线上环境谨慎操作。

kubectl delete pod -n kube-system -l k8s-app=kube-proxy

KubeProxy功能验证

可以通过KubeProxy的日志进行验证是否修改成功，为防止日志过多，生产环境谨慎使用。

kubectl edit ds -n kube-system kube-proxy

在command后面追加参数--v=6，修改后效果：

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "3"
  creationTimestamp: "2022-05-10T06:27:27Z"
  generation: 3
  labels:
    k8s-app: kube-proxy
  name: kube-proxy
  namespace: kube-system
  resourceVersion: "5377081"
  uid: 0f8eccdd-d26f-48f0-8401-8d762a630dc8
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-proxy
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kube-proxy
    spec:
      containers:
      - command:
        - /usr/local/bin/kube-proxy
        - --config=/var/lib/kube-proxy/config.conf
        - --hostname-override=$(NODE_NAME)
        - --v=6

检查KubeProxy的Pod输出日志，如果apiserver地址是：169.254.2.1:10268代表修改成功。日志输出样例：

I0521 02:57:01.986790       1 round_trippers.go:454] GET https://169.254.2.1:10268/api/v1/nodes/jd-sh-qianyi-test-02 200 OK in 12 milliseconds
I0521 02:57:02.021682       1 round_trippers.go:454] POST https://169.254.2.1:10268/api/v1/namespaces/default/events 201 Created in 4 milliseconds