Kubernetes 集群安装

本文主要介绍如何使用 helm 或 kubectl 将 clickvisual 部署到 Kubernetes 集群。

1. 部署要求

  • Kubernetes >= 1.17
  • Helm >= 3.0.0

2. 部署 fluent-bit（参考）

可以直接参考 fluent-bit 官方网站进行部署 https://docs.fluentbit.io/，只需要保证，写入 kafka 的数据包含以下两个字段即可。

  • time
  • log

如果采用 DaemonSet 方式部署，可以使用如下的 DaemonSet yaml。注意需要挂载 configMap。fluentbit-daemonset.yaml 如下：

  1. apiVersion: apps/v1
  2. kind: DaemonSet
  3. metadata:
  4.   name: fluent-bit
  5.   namespace: kube-system
  6.   labels:
  7.     k8s-app: fluent-bit-logging
  8.     version: v1
  9.     kubernetes.io/cluster-service: "true"
  10. spec:
  11.   updateStrategy:
  12.     type: RollingUpdate
  13.   selector:
  14.     matchLabels:
  15.       k8s-app: fluent-bit-logging
  16.   template:
  17.     metadata:
  18.       labels:
  19.         k8s-app: fluent-bit-logging
  20.         version: v1
  21.         kubernetes.io/cluster-service: "true"
  22.     spec:
  23.       containers:
  24.       - name: fluent-bit  
  25.         image: bitnami/fluent-bit:1.8.12      
  26.         imagePullPolicy: Always
  27.         env:
  28.         - name: CLUSTER_NAME
  29.           value: ${CLUSTER_NAME}
  30.         - name: KAFKA_BROKERS
  31.           value: ${KAFKA_BROKERS}
  32.         - name: NODE_IP
  33.           valueFrom:
  34.             fieldRef:
  35.               apiVersion: v1
  36.               fieldPath: status.hostIP
  37.         resources:
  38.           requests:
  39.             cpu: 5m
  40.             memory: 32Mi
  41.           limits:
  42.             cpu: 500m
  43.             memory: 512Mi
  44.         volumeMounts:
  45.         - name: varlog
  46.           mountPath: /var/log
  47.         - name: varlibdockercontainers
  48.           mountPath: /var/lib/docker/containers
  49.           readOnly: true
  50.         - name: fluent-bit-config
  51.           mountPath: /fluent-bit/etc/
  52.       volumes:
  53.       - name: varlog
  54.         hostPath:
  55.           path: /var/log
  56.       - name: varlibdockercontainers
  57.         hostPath:
  58.           path: /var/lib/docker/containers
  59.       - name: fluent-bit-config
  60.         configMap:
  61.           name: fluent-bit-config

挂载的 fluentbit-configmap.yaml 配置可以参考如下：

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: fluent-bit-config
  5.   namespace: kube-system
  6.   labels:
  7.     k8s-app: fluent-bit
  8. data:
  9.   # Configuration files: server, input, filters and output
  10.   # ======================================================
  11.   fluent-bit.conf: |
  12.     [SERVICE]
  13.         Flush         1
  14.         Log_Level     info
  15.         Daemon        off
  16.         Parsers_File  parsers.conf
  17.         HTTP_Server   On
  18.         HTTP_Listen   0.0.0.0
  19.         HTTP_Port     2020
  21.     @INCLUDE input-kubernetes.conf
  22.     @INCLUDE filter-kubernetes.conf
  23.     @INCLUDE filter-modify.conf
  24.     @INCLUDE output-kafka.conf
  26.     # Deamonet中有配置ENV时禁用
  27.     #@Set CLUSTER_NAME=shimodev
  28.     #@Set KAFKA_BROKERS=127.0.0.1:9092
  30.   input-kubernetes.conf: |
  31.     [INPUT]
  32.         Name              tail
  33.         # Tag 标识数据源，用于后续处理流程Filter,output时选择数据
  34.         Tag               ingress.*
  35.         Path              /var/log/containers/nginx-ingress-controller*.log
  36.         Parser            docker
  37.         DB                /var/log/flb_ingress.db
  38.         Mem_Buf_Limit     15MB
  39.         Buffer_Chunk_Size 32k
  40.         Buffer_Max_Size   64k
  41.         # 跳过长度大于 Buffer_Max_Size 的行，Skip_Long_Lines 若设为Off遇到超过长度的行会停止采集
  42.         Skip_Long_Lines   On
  43.         Refresh_Interval  10
  44.         # 采集文件没有数据库偏移位置记录的，从文件的头部开始读取，日志文件较大时会导致fluent内存占用率升高出现oomkill
  45.         #Read_from_Head    On
  47.     [INPUT]
  48.         Name              tail
  49.         # Tag 标识数据源，用于后续处理流程Filter,output时选择数据
  50.         Tag               ingress_stderr.*
  51.         Path              /var/log/containers/nginx-ingress-controller*.log
  52.         Parser            docker
  53.         DB                /var/log/flb_ingress_stderr.db
  54.         Mem_Buf_Limit     15MB
  55.         Buffer_Chunk_Size 32k
  56.         Buffer_Max_Size   64k
  57.         # 跳过长度大于 Buffer_Max_Size 的行，Skip_Long_Lines 若设为Off遇到超过长度的行会停止采集
  58.         Skip_Long_Lines   On
  59.         Refresh_Interval  10
  60.         # 采集文件没有数据库偏移位置记录的，从文件的头部开始读取，日志文件较大时会导致fluent内存占用率升高出现oomkill
  61.         #Read_from_Head    On
  63.     [INPUT]
  64.         Name              tail
  65.         Tag               kube.*
  66.         Path              /var/log/containers/*_default_*.log,/var/log/containers/*_release_*.log
  67.         Exclude_path     *fluent-bit-*,*mongo-*,*minio-*,*mysql-*
  68.         Parser            docker
  69.         DB                /var/log/flb_kube.db
  70.         Mem_Buf_Limit     15MB
  71.         Buffer_Chunk_Size 1MB
  72.         Buffer_Max_Size   5MB
  73.         # 跳过长度大于 Buffer_Max_Size 的行，Skip_Long_Lines 若设为Off遇到超过长度的行会停止采集
  74.         Skip_Long_Lines   On
  75.         Refresh_Interval  10
  77.     [INPUT]
  78.         Name              tail
  79.         Tag               ego.*
  80.         Path              /var/log/containers/*_default_*.log,/var/log/containers/*_release_*.log
  81.         Exclude_path     *fluent-bit-*,*mongo-*,*minio-*,*mysql-*
  82.         Parser            docker
  83.         DB                /var/log/flb_ego.db
  84.         Mem_Buf_Limit     15MB
  85.         Buffer_Chunk_Size 1MB
  86.         Buffer_Max_Size   5MB
  87.         Skip_Long_Lines   On
  88.         Refresh_Interval  10
  90.   filter-kubernetes.conf: |
  91.     [FILTER]
  92.         Name                kubernetes
  93.         Match               ingress.*
  94.         Kube_URL            https://kubernetes.default.svc:443
  95.         Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  96.         Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
  97.         Kube_Tag_Prefix     ingress.var.log.containers.
  98.         # Merge_Log=On 解析log字段的json内容，提取到根层级, 附加到Merge_Log_Key指定的字段上.
  99.         Merge_Log           Off
  100.         #Merge_Log_Key       log_processed
  101.         #Merge_Log_Trim      On
  102.         # 合并log字段后是否保持原始log字段
  103.         Keep_Log            On
  104.         K8S-Logging.Parser  On
  105.         K8S-Logging.Exclude Off
  106.         Labels              Off
  107.         Annotations         Off
  108.         #Regex_Parser
  110.     [FILTER]
  111.         Name                kubernetes
  112.         Match               ingress_stderr.*
  113.         Kube_URL            https://kubernetes.default.svc:443
  114.         Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  115.         Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
  116.         Kube_Tag_Prefix     ingress_stderr.var.log.containers.
  117.         # Merge_Log=On 解析log字段的json内容，提取到根层级, 附加到Merge_Log_Key指定的字段上.
  118.         Merge_Log           Off
  119.         # 合并log字段后是否保持原始log字段
  120.         Keep_Log            Off
  121.         K8S-Logging.Parser  On
  122.         K8S-Logging.Exclude Off
  123.         Labels              Off
  124.         Annotations         Off
  125.         #Regex_Parser
  128.     [FILTER]
  129.         Name                kubernetes
  130.         Match               kube.*
  131.         Kube_URL            https://kubernetes.default.svc:443
  132.         Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  133.         Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
  134.         Kube_Tag_Prefix     kube.var.log.containers.
  135.         Merge_Log           Off
  136.         Keep_Log            On
  137.         K8S-Logging.Parser  On
  138.         K8S-Logging.Exclude Off
  139.         Labels              Off
  140.         Annotations         Off
  142.     [FILTER]
  143.         Name                kubernetes
  144.         Match               ego.*
  145.         Kube_URL            https://kubernetes.default.svc:443
  146.         Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  147.         Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
  148.         Kube_Tag_Prefix     ego.var.log.containers.
  149.         Merge_Log           Off
  150.         Keep_Log            Off
  151.         K8S-Logging.Parser  On
  152.         K8S-Logging.Exclude Off
  153.         Labels              Off
  154.         Annotations         Off
  158.   filter-modify.conf: |
  159.     [FILTER]
  160.         Name         nest
  161.         Match        *
  162.         Wildcard     pod_name
  163.         Operation    lift
  164.         Nested_under kubernetes
  165.         Add_prefix   kubernetes_
  167.     [FILTER]
  168.         Name            modify
  169.         Match           *
  170.         Rename          time _time_
  171.         Rename          log _log_
  172.         Rename          stream _source_
  173.         Rename          kubernetes_host _node_name_
  174.         Rename          kubernetes_namespace_name _namespace_
  175.         Rename          kubernetes_container_name _container_name_
  176.         Rename          kubernetes_pod_name _pod_name_
  177.         Remove          kubernetes_pod_id
  178.         Remove          kubernetes_docker_id
  179.         Remove          kubernetes_container_hash
  180.         Remove          kubernetes_container_image
  181.         Add             _cluster_ ${CLUSTER_NAME}
  182.         Add             _log_agent_ ${HOSTNAME}
  183.         # ${NODE_IP} 通过daemonset中配置ENV注入
  184.         Add             _node_ip_ ${NODE_IP}
  186.     [FILTER]
  187.         Name    grep
  188.         Match   ingress.*
  189.         #Regex container_name ^nginx-ingress-controller$
  190.         #Regex stream ^stdout$
  191.         Exclude _source_ ^stderr$
  192.         # 排除 TCP 代理日志（日志格式不同影响采集）
  193.         Exclude log ^\[*
  195.     [FILTER]
  196.         Name    grep
  197.         Match   ingress_stderr.*
  198.         Exclude _source_ ^stdout$
  200.     [FILTER]
  201.         Name    grep
  202.         Match   kube.*
  203.         #Regex stream ^stdout$
  204.         Exclude log (ego.sys)
  206.     [FILTER]
  207.         Name    grep
  208.         Match   ego.*
  209.         #Regex lname ^(ego.sys)$
  210.         Regex   log ("lname":"ego.sys")
  212.     # [FILTER]
  213.     #     Name            modify
  214.     #     Match           ego.*
  215.     #     Hard_rename     ts _time_
  217.   output-kafka.conf: |
  218.     [OUTPUT]
  219.         Name           kafka
  220.         Match          ingress.*
  221.         Brokers        ${KAFKA_BROKERS}
  222.         Topics         ingress-stdout-logs-${CLUSTER_NAME}
  223.         #Timestamp_Key  @timestamp
  224.         Timestamp_Key  _time_
  225.         Retry_Limit    false
  226.         # hides errors "Receive failed: Disconnected" when kafka kills idle connections
  227.         rdkafka.log.connection.close false
  228.         # producer buffer is not included in http://fluentbit.io/documentation/0.12/configuration/memory_usage.html#estimating
  229.         rdkafka.queue.buffering.max.kbytes 10240
  230.         # for logs you'll probably want this ot be 0 or 1, not more
  231.         rdkafka.request.required.acks 1
  233.     [OUTPUT]
  234.         Name           kafka
  235.         Match          ingress_stderr.*
  236.         Brokers        ${KAFKA_BROKERS}
  237.         Topics         ingress-stderr-logs-${CLUSTER_NAME}
  238.         #Timestamp_Key  @timestamp
  239.         Timestamp_Key  _time_
  240.         Retry_Limit    false
  241.         # hides errors "Receive failed: Disconnected" when kafka kills idle connections
  242.         rdkafka.log.connection.close false
  243.         # producer buffer is not included in http://fluentbit.io/documentation/0.12/configuration/memory_usage.html#estimating
  244.         rdkafka.queue.buffering.max.kbytes 10240
  245.         # for logs you'll probably want this ot be 0 or 1, not more
  246.         rdkafka.request.required.acks 1
  248.     [OUTPUT]
  249.         Name           kafka
  250.         Match          kube.*
  251.         Brokers        ${KAFKA_BROKERS}
  252.         Topics         app-stdout-logs-${CLUSTER_NAME}
  253.         Timestamp_Key  _time_
  254.         Retry_Limit    false
  255.         # hides errors "Receive failed: Disconnected" when kafka kills idle connections
  256.         rdkafka.log.connection.close false
  257.         # producer buffer is not included in http://fluentbit.io/documentation/0.12/configuration/memory_usage.html#estimating
  258.         rdkafka.queue.buffering.max.kbytes 10240
  259.         # for logs you'll probably want this ot be 0 or 1, not more
  260.         rdkafka.request.required.acks 1
  262.     [OUTPUT]
  263.         Name           kafka
  264.         Match          ego.*
  265.         Brokers        ${KAFKA_BROKERS}
  266.         Topics         ego-stdout-logs-${CLUSTER_NAME}
  267.         Timestamp_Key  _time_
  268.         Retry_Limit    false
  269.         # hides errors "Receive failed: Disconnected" when kafka kills idle connections
  270.         rdkafka.log.connection.close false
  271.         # producer buffer is not included in http://fluentbit.io/documentation/0.12/configuration/memory_usage.html#estimating
  272.         rdkafka.queue.buffering.max.kbytes 10240
  273.         # for logs you'll probably want this ot be 0 or 1, not more
  274.         rdkafka.request.required.acks 1
  276.   parsers.conf: |
  277.     [PARSER]
  278.         Name   apache
  279.         Format regex
  280.         Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
  281.         Time_Key time
  282.         Time_Format %d/%b/%Y:%H:%M:%S %z
  284.     [PARSER]
  285.         Name   apache2
  286.         Format regex
  287.         Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
  288.         Time_Key time
  289.         Time_Format %d/%b/%Y:%H:%M:%S %z
  291.     [PARSER]
  292.         Name   apache_error
  293.         Format regex
  294.         Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
  296.     [PARSER]
  297.         Name   nginx
  298.         Format regex
  299.         Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
  300.         Time_Key time
  301.         Time_Format %d/%b/%Y:%H:%M:%S %z
  303.     [PARSER]
  304.         Name   json
  305.         Format json
  306.         Time_Key time
  307.         Time_Format %d/%b/%Y:%H:%M:%S %z
  309.     [PARSER]
  310.         Name        docker
  311.         Format      json
  312.         Time_Key    time
  313.         Time_Format %Y-%m-%dT%H:%M:%S.%L
  314.         Time_Keep   On
  315.         # 与FILTER 阶段中Merge_Log=On 效果类似，解析log字段的json内容，但无法提到根层级
  316.         #Decode_Field_As escaped_utf8 kubernetes do_next
  317.         #Decode_Field_As json kubernetes
  319.     [PARSER]
  320.         # http://rubular.com/r/tjUt3Awgg4
  321.         Name cri
  322.         Format regex
  323.         Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
  324.         Time_Key    time
  325.         Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  327.     [PARSER]
  328.         Name        syslog
  329.         Format      regex
  330.         Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
  331.         Time_Key    time
  332.         Time_Format %b %d %H:%M:%S
  1. 部署 clickvisual 克隆仓库：
  1. git clone https://github.com/clickvisual/clickvisual.git
  2. cd clickvisual &amp;&amp; cp api/config/default.toml data/helm/clickvisual/default.toml

修改 data/helm/clickvisual/default.toml 中的 mysql、auth 以及其他段配置，将 mysql.dsn 、 auth.redisAddr、auth.redisPassword 替换为你自己的配置。

方法一：[推荐] 使用 helm 直接安装：

  1. helm install clickvisual data/helm/clickvisual --set image.tag=latest --namespac default

如果你已将 clickvisual 镜像推送到你自己的 harbor 仓库，可以通过 —set image.respository 指令修改仓库地址

  1. helm install clickvisual data/helm/clickvisual --set image.repository=${YOUR_HARBOR}/${PATH}/clickvisual --set image.tag=latest --namespace default<br/>

方法二：[可选] 使用 helm 渲染出 yaml 后，手动通过 kubectl 安装：

  1. # 使用 helm template 指令渲染安装的 yaml
  2. helm template clickvisual data/helm/clickvisual --set image.tag=latest > clickvisual.yaml
  4. # 可以使用 "--set image.repository" 来覆盖默认镜像路径
  5. # helm template clickvisual clickvisual --set image.repository=${YOUR_HARBOR}/${PATH}/clickvisual --set image.tag=latest > clickvisual.yaml
  7. # 检查 clickvisual.yaml 是否无误，随后通过 kuebctl apply clickvisual.yaml
  8. kubectl apply -f clickvisual.yaml --namespace default