初试VictoriaLogs集群采集Kubernetes Pod日志

VictoriaLogs 是 VictoriaMetrics 团队推出的开源日志解决方案，旨在替代笨重的 Elasticsearch (ELK) 和架构复杂的 Loki

组件介绍

vlinsert 接收来自 Fluent-bit、Vector 或 Logstash 的日志数据
vlstorage 核心存储，将日志压缩并持久化到磁盘，响应查询请求。
vlselect 查询网关，处理Grafana或CLI的搜索请求。
vmauth 认证网关 - 可选，提供统一入口、负载均衡和身份验证。

Helm部署VictoriaLogs Cluster

这里使用从 Kubernetes 中运行的所有容器收集日志，并将数据发送到已安装的集群版 VictoriaLogs。

vlstorage部署了两个实例，每个实例的保留期为180天，PVC 为200Gi。

首先添加helm仓库

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update

使用环境变量的方式，helm VictoriaLogs部署节点

export RETENTION=180d
export PVC_SIZE=200Gi
export VLSTORAGE_REPLICAS=2
export NAMESPACE=tools

# Install victoria-logs-cluster chart
helm install vlc vm/victoria-logs-cluster --namespace $NAMESPACE --wait \
    --set "vlstorage.retentionPeriod=$RETENTION" --set "vlstorage.persistentVolume.size=$PVC_SIZE" \
    --set vmauth.enabled=true \
    --set vlstorage.replicaCount=$VLSTORAGE_REPLICAS

# Install victoria-logs-collector chart
helm install collector vm/victoria-logs-collector --namespace $NAMESPACE \
    --set "remoteWrite[0].url=http://vlc-victoria-logs-cluster-vmauth:8427"

离线本地下载

export RETENTION=180d
export PVC_SIZE=200Gi
export VLSTORAGE_REPLICAS=2
export NAMESPACE=tools

# 基于本地tgz包安装
helm install vlc ./victoria-logs-cluster-0.0.20.tgz \
  --namespace $NAMESPACE --wait \
  --set "vlstorage.retentionPeriod=$RETENTION" \
  --set "vlstorage.persistentVolume.size=$PVC_SIZE" \
  --set vmauth.enabled=true \
  --set vlstorage.replicaCount=$VLSTORAGE_REPLICAS

# 或基于解压后的目录安装（适合本地修改配置后）
helm install vlc ./victoria-logs-cluster \
  --namespace $NAMESPACE --wait \
  --set "vlstorage.retentionPeriod=$RETENTION" \
  --set "vlstorage.persistentVolume.size=$PVC_SIZE" \
  --set vmauth.enabled=true \
  --set vlstorage.replicaCount=$VLSTORAGE_REPLICAS

如需镜像加速，可使用abcdocker提供镜像加速文档 https://i4t.com/dockerproxy

修改服务暴露，如果没有ingress，可以和我一样修改SVC类型

#修改写的映射
root@k8s-master-01:~# kubectl edit svc  -n tools vlc-victoria-logs-cluster-vlinsert
  type: NodePort
#修改读的映射
root@k8s-master-01:~# kubectl edit svc -n tools vlc-victoria-logs-cluster-vlselect
  type: NodePort

最终效果

root@k8s-master-01:~# kubectl get svc -n tools|grep vlc
vlc-victoria-logs-cluster-vlinsert    NodePort    10.96.3.162   <none>        9481:31072/TCP                                                                                                    29d
vlc-victoria-logs-cluster-vlselect    NodePort    10.96.1.97    <none>        9471:31708/TCP                                                                                                    29d
vlc-victoria-logs-cluster-vlstorage   ClusterIP   None          <none>        9491/TCP                                                                                                          29d
vlc-victoria-logs-cluster-vmauth      ClusterIP   10.96.0.80    <none>        8427/TCP                                                                                                          29d

Grafana 安装插件

首先在Grafana中需要安装victoriametrics-logs-datasource插件

grafana cli plugins install victoriametrics-logs-datasource

初试VictoriaLogs集群采集Kubernetes Pod日志

fluent-bit 采集Kubernetes日志

创建ConfigMap

tools 部署命名空间
采集Pod日志/var/log/containers/*.log路径
写入日志地址 vlc-victoria-logs-cluster-vlinsert
写入日志端口9481

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: tools
  labels:
    app.kubernetes.io/name: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Log_Level    info
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
        Health_Check On

    [INPUT]
        Name            tail
        Tag             kubernetes.*
        Path            /var/log/containers/*.log
        # --- 关键修改在这里 ---
        Exclude_Path    /var/log/containers/fluent-bit-*.log
        Parser          cri
        DB              /var/log/flb_kube.db
        Mem_Buf_Limit   5MB
        Skip_Long_Lines On
        Refresh_Interval 10

    [FILTER]
        Name            kubernetes
        Match           kubernetes.*
        Kube_URL        https://kubernetes.default.svc:443
        Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix kubernetes.var.log.containers.
        Merge_Log       On
        Merge_Log_Key   log_processed
        K8S-Logging.Parser On
        K8S-Logging.Exclude Off

    [OUTPUT]
        Name            http
        Match           *
        Host            vlc-victoria-logs-cluster-vlinsert
        Port            9481
        URI             /insert/jsonline?_msg_field=log&_stream_fields=kubernetes.namespace_name,kubernetes.pod_name,kubernetes.container_name,stream&_time_field=time
        Format          json_lines
        Json_date_key   time
        Json_date_format iso8601
        Header          X-Scope-OrgID 1
        tls             Off
        tls.verify      Off

  parsers.conf: |
    [PARSER]
        Name   cri
        Format regex
        Regex  ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time

创建rbac.yaml文件

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: tools # 建议使用独立的命名空间，如 logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  - pods/logs
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: tools

创建DaemonSet.yaml Pod

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: tools
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
  template:
    metadata:
      labels:
        app.kubernetes.io/name: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: harbor.frps.cn/tools/fluent-bit:3.0.1
        imagePullPolicy: Always
        ports:
        - containerPort: 2020
        env:
        - name: FLB_PROCESSOR
          value: "OFF"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        - name: etcmachineid
          mountPath: /etc/machine-id
          readOnly: true
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 20Mi
        securityContext:
          runAsUser: 0
          privileged: true
      serviceAccountName: fluent-bit
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      - name: etcmachineid
        hostPath:
          path: /etc/machine-id
          type: File
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule

部署完毕后，可以看到服务状态

root@k8s-master-01:~# kubectl get pod -n tools |grep fl
fluent-bit-d9ls4                                      1/1     Running   9 (4d21h ago)     16d
fluent-bit-dmr86                                      1/1     Running   3 (4d20h ago)     9d
fluent-bit-jhs7d                                      1/1     Running   7 (4d21h ago)     14d
fluent-bit-krkh4                                      1/1     Running   9 (4d21h ago)     14d
fluent-bit-t4znj                                      1/1     Running   6 (4d21h ago)     13d
fluent-bit-x9j5c                                      1/1     Running   7 (4d21h ago)     13d

现在我们访问页面就可以看到效果

node_ip:port

图形化访问的是vlselect

vlc-victoria-logs-cluster-vlselect    NodePort    10.96.1.97    <none>        9471:31708/TCP

初试VictoriaLogs集群采集Kubernetes Pod日志

Grafana 配置 victoriametrics-logs-datasource 数据源

Home -->Connections --> Add new connection --> VictoriaLogs -->Add new data source

初试VictoriaLogs集群采集Kubernetes Pod日志

http://vlc-victoria-logs-cluster-vlselect:9471

vlc-victoria-logs-cluster-vlselect    NodePort    10.96.1.97    <none>        9471:31708/TCP

初试VictoriaLogs集群采集Kubernetes Pod日志

保存测试连接
初试VictoriaLogs集群采集Kubernetes Pod日志

默认Grafana没有提供好的模板选择，需要我们自行导入，这里我让Gemini Pro写了个一个简单的查看日志模板，可以直接导入
初试VictoriaLogs集群采集Kubernetes Pod日志

导入下方JSON

下载地址https://d.frps.cn/file/tools/grafana/VictoriaLogs-Dashboard.json
打开JSON文件复制上即可

初试VictoriaLogs集群采集Kubernetes Pod日志

数据源这块别选择❎

初试VictoriaLogs集群采集Kubernetes Pod日志

最后模板就可以看到日志

Namespace: 默认 .* (所有)。想看 tools 就直接输 tools。
Pod: 默认 .* (所有)。想看 blackbox 就直接输 blackbox。
Search: 默认 * (所有)。想搜错误就输 error 或 exception。
时间：默认设为了 Last 15 minutes，这是看实时日志最舒服的范围。

初试VictoriaLogs集群采集Kubernetes Pod日志

推荐订阅源

i4T

Helm部署VictoriaLogs Cluster

Grafana 安装插件

fluent-bit 采集Kubernetes日志

Grafana 配置 victoriametrics-logs-datasource 数据源