RocketMQ k8s部署 4主4从集群

日常 on 打工人日志

2022-08-29 · via 日常 on 打工人日志

RocketMQ k8s 部署 4 主 4 从集群

使用 NFS 配置 StatefulSet 的动态持久化存储

安装 NFS 服务端

1sudo apt update
2sudo apt install nfs-kernel-server nfs-common

安装 NFS 客户端

所有的节点都得执行

sudo apt install nfs-common -y

创建目录

1mkdir -p /data/storage/k8s/rocketmq

使用 NFS 作为StatefulSet持久化存储的操作记录，分别需要创建nfs-provisioner的rbac、storageclass、nfs-client-provisioner和statefulset的pod

创建 nfs 的 rbac

 1---
 2apiVersion: v1
 3kind: ServiceAccount
 4metadata:
 5  name: nfs-provisioner
 6  namespace: sanjiang
 7---
 8kind: ClusterRole
 9apiVersion: rbac.authorization.k8s.io/v1
10metadata:
11  name: nfs-provisioner-runner
12  namespace: sanjiang
13rules:
14  - apiGroups: [""]
15    resources: ["persistentvolumes"]
16    verbs: ["get", "list", "watch", "create", "delete"]
17  - apiGroups: [""]
18    resources: ["persistentvolumeclaims"]
19    verbs: ["get", "list", "watch", "update"]
20  - apiGroups: ["storage.k8s.io"]
21    resources: ["storageclasses"]
22    verbs: ["get", "list", "watch"]
23  - apiGroups: [""]
24    resources: ["events"]
25    verbs: ["list", "watch", "create", "update", "patch"]
26  - apiGroups: [""]
27    resources: ["services", "endpoints"]
28    verbs: ["get", "create", "list", "watch", "update"]
29  - apiGroups: ["extensions"]
30    resources: ["podsecuritypolicies"]
31    resourceNames: ["nfs-provisioner"]
32    verbs: ["use"]
33  - apiGroups: [""]
34    resources: ["endpoints"]
35    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
36---
37kind: ClusterRoleBinding
38apiVersion: rbac.authorization.k8s.io/v1
39metadata:
40  name: run-nfs-provisioner
41subjects:
42  - kind: ServiceAccount
43    name: nfs-provisioner
44    namespace: sanjiang
45roleRef:
46  kind: ClusterRole
47  name: nfs-provisioner-runner
48  apiGroup: rbac.authorization.k8s.io

kubectl apply -f nfs-rbac.yaml

创建 rocketmq 集群的 storageclass

1apiVersion: storage.k8s.io/v1
2kind: StorageClass
3metadata:
4  name: rocketmq-nfs-storage
5  namespace: sanjiang
6provisioner: rocketmq/nfs
7reclaimPolicy: Retain

kubectl apply -f rocketmq-nfs-class.yaml
查看创建情况
kubectl get sc -n sanjiang

创建 rocketmq 集群的 nfs-client-provisioner

PROVISIONER_NAME的值一定要和StorageClass中的provisioner相等

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: rocketmq-nfs-client-provisioner
 5  namespace: sanjiang
 6spec:
 7  replicas: 1
 8  selector:
 9    matchLabels:
10      app: rocketmq-nfs-client-provisioner
11  strategy:
12    type: Recreate
13  template:
14    metadata:
15      labels:
16        app: rocketmq-nfs-client-provisioner
17    spec:
18      serviceAccount: nfs-provisioner
19      containers:
20        - name: rocketmq-nfs-client-provisioner
21          image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner
22          imagePullPolicy: IfNotPresent
23          volumeMounts:
24            - name: nfs-client-root
25              mountPath: /persistentvolumes
26          env:
27            - name: PROVISIONER_NAME
28              value: rocketmq/nfs
29            - name: NFS_SERVER
30              value: 193.0.40.171 #nfs ip
31            - name: NFS_PATH
32              value: /data/storage/k8s/rocketmq
33      volumes:
34        - name: nfs-client-root
35          nfs:
36            server: 193.0.40.171 #nfs ip
37            path: /data/storage/k8s/rocketmq

kubectl apply -f rocketmq-nfs.yml

k8s 部署

生成文件

broker-a-s.properties

 1brokerClusterName=rocketmq-cluster
 2brokerName=broker-a
 3brokerId=1
 4namesrvAddr=rocketmq-0.rocketmq:9876
 5defaultTopicQueueNums=4
 6autoCreateTopicEnable=true
 7autoCreateSubscriptionGroup=true
 8listenPort=20911
 9deleteWhen=04
10fileReservedTime=120
11mapedFileSizeCommitLog=1073741824
12mapedFileSizeConsumeQueue=300000
13diskMaxUsedSpaceRatio=88
14storePathRootDir=/data/rocketmq/store
15maxMessageSize=65536
16brokerRole=SLAVE
17flushDiskType=SYNC_FLUSH

broker-a.properties

 1brokerClusterName=rocketmq-cluster
 2brokerName=broker-a
 3brokerId=0
 4namesrvAddr=rocketmq-0.rocketmq:9876
 5defaultTopicQueueNums=4
 6autoCreateTopicEnable=true
 7autoCreateSubscriptionGroup=true
 8listenPort=20911
 9deleteWhen=04
10fileReservedTime=120
11mapedFileSizeCommitLog=1073741824
12mapedFileSizeConsumeQueue=300000
13diskMaxUsedSpaceRatio=88
14storePathRootDir=/data/rocketmq/store
15maxMessageSize=65536
16brokerRole=MASTER

broker-b-s.properties

 1brokerClusterName=rocketmq-cluster
 2brokerName=broker-b
 3brokerId=1
 4namesrvAddr=rocketmq-0.rocketmq:9876
 5defaultTopicQueueNums=4
 6autoCreateTopicEnable=true
 7autoCreateSubscriptionGroup=true
 8listenPort=20911
 9deleteWhen=04
10fileReservedTime=120
11mapedFileSizeCommitLog=1073741824
12mapedFileSizeConsumeQueue=300000
13diskMaxUsedSpaceRatio=88
14storePathRootDir=/data/rocketmq/store
15maxMessageSize=65536
16brokerRole=SLAVE
17flushDiskType=SYNC_FLUSH

broker-b.properties

 1brokerClusterName=rocketmq-cluster
 2brokerName=broker-b
 3brokerId=0
 4namesrvAddr=rocketmq-0.rocketmq:9876
 5defaultTopicQueueNums=4
 6autoCreateTopicEnable=true
 7autoCreateSubscriptionGroup=true
 8listenPort=20911
 9deleteWhen=04
10fileReservedTime=120
11mapedFileSizeCommitLog=1073741824
12mapedFileSizeConsumeQueue=300000
13diskMaxUsedSpaceRatio=88
14storePathRootDir=/data/rocketmq/store
15maxMessageSize=65536
16brokerRole=MASTER
17flushDiskType=SYNC_FLUSH

broker-c-s.properties

 1brokerClusterName=rocketmq-cluster
 2brokerName=broker-c
 3brokerId=1
 4namesrvAddr=rocketmq-0.rocketmq:9876
 5defaultTopicQueueNums=4
 6autoCreateTopicEnable=true
 7autoCreateSubscriptionGroup=true
 8listenPort=20911
 9deleteWhen=04
10fileReservedTime=120
11mapedFileSizeCommitLog=1073741824
12mapedFileSizeConsumeQueue=300000
13diskMaxUsedSpaceRatio=88
14storePathRootDir=/data/rocketmq/store
15maxMessageSize=65536
16brokerRole=SLAVE
17flushDiskType=SYNC_FLUSH

broker-c.properties

 1brokerClusterName=rocketmq-cluster
 2brokerName=broker-c
 3brokerId=0
 4namesrvAddr=rocketmq-0.rocketmq:9876
 5defaultTopicQueueNums=4
 6autoCreateTopicEnable=true
 7autoCreateSubscriptionGroup=true
 8listenPort=20911
 9deleteWhen=04
10fileReservedTime=120
11mapedFileSizeCommitLog=1073741824
12mapedFileSizeConsumeQueue=300000
13diskMaxUsedSpaceRatio=88
14storePathRootDir=/data/rocketmq/store
15maxMessageSize=65536
16brokerRole=MASTER
17flushDiskType=SYNC_FLUSH

broker-d-s.properties

 1brokerClusterName=rocketmq-cluster
 2brokerName=broker-d
 3brokerId=1
 4namesrvAddr=rocketmq-0.rocketmq:9876
 5defaultTopicQueueNums=4
 6autoCreateTopicEnable=true
 7autoCreateSubscriptionGroup=true
 8listenPort=20911
 9deleteWhen=04
10fileReservedTime=120
11mapedFileSizeCommitLog=1073741824
12mapedFileSizeConsumeQueue=300000
13diskMaxUsedSpaceRatio=88
14storePathRootDir=/data/rocketmq/store
15maxMessageSize=65536
16brokerRole=SLAVE
17flushDiskType=SYNC_FLUSH

broker-d.properties

 1brokerClusterName=rocketmq-cluster
 2brokerName=broker-d
 3brokerId=0
 4namesrvAddr=rocketmq-0.rocketmq:9876
 5defaultTopicQueueNums=4
 6autoCreateTopicEnable=true
 7autoCreateSubscriptionGroup=true
 8listenPort=20911
 9deleteWhen=04
10fileReservedTime=120
11mapedFileSizeCommitLog=1073741824
12mapedFileSizeConsumeQueue=300000
13diskMaxUsedSpaceRatio=88
14storePathRootDir=/data/rocketmq/store
15maxMessageSize=65536
16brokerRole=MASTER
17flushDiskType=SYNC_FLUSH

运行命令

1kubectl create cm rocketmq-config --from-file=broker-a.properties --from-file=broker-a-s.properties --from-file=broker-b.properties --from-file=broker-b-s.properties --from-file=broker-c.properties --from-file=broker-c-s.properties --from-file=broker-d.properties --from-file=broker-d-s.properties -n sanjiang

kubectl get cm -n sanjiang|grep rocketmq

创建配置文件

broker-a-s.yaml

  1apiVersion: v1
  2kind: Service
  3metadata:
  4  labels:
  5    app: broker-a-s
  6  name: broker-a-s
  7  namespace: sanjiang
  8spec:
  9  ports:
 10    - port: 20911
 11      targetPort: 20911
 12      name: broker-port
 13  selector:
 14    app: broker-a-s
 15---
 16apiVersion: apps/v1
 17kind: StatefulSet
 18metadata:
 19  name: broker-a-s
 20  namespace: sanjiang
 21spec:
 22  serviceName: broker-a-s
 23  replicas: 1
 24  selector:
 25    matchLabels:
 26      app: broker-a-s
 27  template:
 28    metadata:
 29      labels:
 30        app: broker-a-s
 31    spec:
 32      affinity:
 33        podAntiAffinity:
 34          requiredDuringSchedulingIgnoredDuringExecution:
 35            - labelSelector:
 36                matchExpressions:
 37                  - key: "app"
 38                    operator: In
 39                    values:
 40                      - broker-a-s
 41              topologyKey: "kubernetes.io/hostname"
 42      containers:
 43        - name: broker-a-s
 44          image: liuyi71sinacom/rocketmq-4.8.0
 45          imagePullPolicy: IfNotPresent
 46          command:
 47            [
 48              "sh",
 49              "-c",
 50              "mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-a-s.properties",
 51            ]
 52          env:
 53            - name: JAVA_OPT
 54              value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
 55              #value: "-XX:MaxRAMPercentage=80.0"
 56          volumeMounts:
 57            - mountPath: /root/logs
 58              name: rocketmq-data
 59              subPath: mq-brokeroptlogs
 60            - mountPath: /data/rocketmq
 61              name: rocketmq-data
 62              subPath: mq-brokeroptstore
 63            - name: broker-config
 64              mountPath: /usr/local/rocketmq-4.8.0/conf/broker-a-s.properties
 65              subPath: broker-a-s.properties
 66          lifecycle:
 67            postStart:
 68              exec:
 69                command: ["/bin/sh", "-c", "touch /tmp/health"]
 70          livenessProbe:
 71            exec:
 72              command: ["test", "-e", "/tmp/health"]
 73            initialDelaySeconds: 5
 74            timeoutSeconds: 5
 75            periodSeconds: 10
 76          readinessProbe:
 77            tcpSocket:
 78              port: 20911
 79            initialDelaySeconds: 15
 80            timeoutSeconds: 5
 81            periodSeconds: 20
 82      volumes:
 83        - name: broker-config
 84          configMap:
 85            name: rocketmq-config
 86            items:
 87              - key: broker-a-s.properties
 88                path: broker-a-s.properties
 89  volumeClaimTemplates:
 90    - metadata:
 91        name: rocketmq-data
 92        namespace: sanjiang
 93        annotations:
 94          volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"
 95      spec:
 96        accessModes:
 97          - ReadWriteMany
 98        resources:
 99          requests:
100            storage: 2Gi
101---
102apiVersion: v1
103kind: PersistentVolume
104metadata:
105  name: broker-a-s-pv
106  namespace: sanjiang
107spec:
108  accessModes:
109    - ReadWriteMany
110  capacity:
111    storage: 2Gi
112  persistentVolumeReclaimPolicy: Retain
113  storageClassName: rocketmq-nfs-storage
114  nfs:
115    path: /data/storage/k8s/rocketmq/broker-a-s
116    server: 193.0.40.171

broker-a.yaml

  1apiVersion: v1
  2kind: Service
  3metadata:
  4  labels:
  5    app: broker-a
  6  name: broker-a
  7  namespace: sanjiang
  8spec:
  9  ports:
 10    - port: 20911
 11      targetPort: 20911
 12      name: broker-port
 13  selector:
 14    app: broker-a
 15---
 16apiVersion: apps/v1
 17kind: StatefulSet
 18metadata:
 19  name: broker-a
 20  namespace: sanjiang
 21spec:
 22  serviceName: broker-a
 23  replicas: 1
 24  selector:
 25    matchLabels:
 26      app: broker-a
 27  template:
 28    metadata:
 29      labels:
 30        app: broker-a
 31    spec:
 32      affinity:
 33        podAntiAffinity:
 34          requiredDuringSchedulingIgnoredDuringExecution:
 35            - labelSelector:
 36                matchExpressions:
 37                  - key: "app"
 38                    operator: In
 39                    values:
 40                      - broker-a
 41              topologyKey: "kubernetes.io/hostname"
 42      containers:
 43        - name: broker-a
 44          image: liuyi71sinacom/rocketmq-4.8.0
 45          imagePullPolicy: IfNotPresent
 46          command:
 47            [
 48              "sh",
 49              "-c",
 50              "mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-a.properties",
 51            ]
 52          env:
 53            - name: JAVA_OPT
 54              value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
 55              #value: "-XX:MaxRAMPercentage=80.0"
 56          volumeMounts:
 57            - mountPath: /root/logs
 58              name: rocketmq-data
 59              subPath: mq-brokeroptlogs
 60            - mountPath: /data/rocketmq
 61              name: rocketmq-data
 62              subPath: mq-brokeroptstore
 63            - name: broker-config
 64              mountPath: /usr/local/rocketmq-4.8.0/conf/broker-a.properties
 65              subPath: broker-a.properties
 66          lifecycle:
 67            postStart:
 68              exec:
 69                command: ["/bin/sh", "-c", "touch /tmp/health"]
 70          livenessProbe:
 71            exec:
 72              command: ["test", "-e", "/tmp/health"]
 73            initialDelaySeconds: 5
 74            timeoutSeconds: 5
 75            periodSeconds: 10
 76          readinessProbe:
 77            tcpSocket:
 78              port: 20911
 79            initialDelaySeconds: 15
 80            timeoutSeconds: 5
 81            periodSeconds: 20
 82      volumes:
 83        - name: broker-config
 84          configMap:
 85            name: rocketmq-config
 86  volumeClaimTemplates:
 87    - metadata:
 88        name: rocketmq-data
 89        namespace: sanjiang
 90        annotations:
 91          volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"
 92      spec:
 93        accessModes:
 94          - ReadWriteMany
 95        resources:
 96          requests:
 97            storage: 2Gi
 98---
 99apiVersion: v1
100kind: PersistentVolume
101metadata:
102  name: broker-a-pv
103  namespace: sanjiang
104spec:
105  accessModes:
106    - ReadWriteMany
107  capacity:
108    storage: 2Gi
109  persistentVolumeReclaimPolicy: Retain
110  storageClassName: rocketmq-nfs-storage
111  nfs:
112    path: /data/storage/k8s/rocketmq/broker-a
113    server: 193.0.40.171

broker-b-s.yaml

  1apiVersion: v1
  2kind: Service
  3metadata:
  4  labels:
  5    app: broker-b-s
  6  name: broker-b-s
  7  namespace: sanjiang
  8spec:
  9  ports:
 10    - port: 20911
 11      targetPort: 20911
 12      name: broker-port
 13  selector:
 14    app: broker-b-s
 15---
 16apiVersion: apps/v1
 17kind: StatefulSet
 18metadata:
 19  name: broker-b-s
 20  namespace: sanjiang
 21spec:
 22  serviceName: broker-b-s
 23  replicas: 1
 24  selector:
 25    matchLabels:
 26      app: broker-b-s
 27  template:
 28    metadata:
 29      labels:
 30        app: broker-b-s
 31    spec:
 32      affinity:
 33        podAntiAffinity:
 34          requiredDuringSchedulingIgnoredDuringExecution:
 35            - labelSelector:
 36                matchExpressions:
 37                  - key: "app"
 38                    operator: In
 39                    values:
 40                      - broker-b-s
 41              topologyKey: "kubernetes.io/hostname"
 42      containers:
 43        - name: broker-b-s
 44          image: liuyi71sinacom/rocketmq-4.8.0
 45          imagePullPolicy: IfNotPresent
 46          command:
 47            [
 48              "sh",
 49              "-c",
 50              "mqbroker -c /usr/local/rocketmq-4.8.0/conf/broker-b-s.properties",
 51            ]
 52          env:
 53            - name: JAVA_OPT
 54              value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
 55              #value: "-XX:MaxRAMPercentage=80.0"
 56          volumeMounts:
 57            - mountPath: /root/logs
 58              name: rocketmq-data
 59              subPath: mq-brokeroptlogs
 60            - mountPath: /data/rocketmq
 61              name: rocketmq-data
 62              subPath: mq-brokeroptstore
 63            - name: broker-config
 64              mountPath: /usr/local/rocketmq-4.8.0/conf/broker-b-s.properties
 65              subPath: broker-b-s.properties
 66          lifecycle:
 67            postStart:
 68              exec:
 69                command: ["/bin/sh", "-c", "touch /tmp/health"]
 70          livenessProbe:
 71            exec:
 72              command: ["test", "-e", "/tmp/health"]
 73            initialDelaySeconds: 5
 74            timeoutSeconds: 5
 75            periodSeconds: 10
 76          readinessProbe:
 77            tcpSocket:
 78              port: 20911
 79            initialDelaySeconds: 15
 80            timeoutSeconds: 5
 81            periodSeconds: 20
 82      volumes:
 83        - name: broker-config
 84          configMap:
 85            name: rocketmq-config
 86            items:
 87              - key: broker-b-s.properties
 88                path: broker-b-s.properties
 89  volumeClaimTemplates:
 90    - metadata:
 91        name: rocketmq-data
 92        namespace: sanjiang
 93        annotations:
 94          volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"
 95      spec:
 96        accessModes:
 97          - ReadWriteMany
 98        resources:
 99          requests:
100            storage: 2Gi
101---
102apiVersion: v1
103kind: PersistentVolume
104metadata:
105  name: broker-b-s-pv
106  namespace: sanjiang
107spec:
108  accessModes:
109    - ReadWriteMany
110  capacity:
111    storage: 2Gi
112  persistentVolumeReclaimPolicy: Retain
113  storageClassName: rocketmq-nfs-storage
114  nfs:
115    path: /data/storage/k8s/rocketmq/broker-b-s
116    server: 193.0.40.171

broker-b.yaml

  1apiVersion: v1
  2kind: Service
  3metadata:
  4  labels:
  5    app: broker-b
  6  name: broker-b
  7  namespace: sanjiang
  8spec:
  9  ports:
 10    - port: 20911
 11      targetPort: 20911
 12      name: broker-port
 13  selector:
 14    app: broker-b
 15---
 16apiVersion: apps/v1
 17kind: StatefulSet
 18metadata:
 19  name: broker-b
 20  namespace: sanjiang
 21spec:
 22  serviceName: broker-b
 23  replicas: 1
 24  selector:
 25    matchLabels:
 26      app: broker-b
 27  template:
 28    metadata:
 29      labels:
 30        app: broker-b
 31    spec:
 32      affinity:
 33        podAntiAffinity:
 34          requiredDuringSchedulingIgnoredDuringExecution:
 35            - labelSelector:
 36                matchExpressions:
 37                  - key: "app"
 38                    operator: In
 39                    values:
 40                      - broker-b
 41              topologyKey: "kubernetes.io/hostname"
 42      containers:
 43        - name: broker-b
 44          image: liuyi71sinacom/rocketmq-4.8.0
 45          imagePullPolicy: IfNotPresent
 46          command:
 47            [
 48              "sh",
 49              "-c",
 50              "mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-b.properties",
 51            ]
 52          env:
 53            - name: JAVA_OPT
 54              value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
 55              #value: "-XX:MaxRAMPercentage=80.0"
 56          volumeMounts:
 57            - mountPath: /root/logs
 58              name: rocketmq-data
 59              subPath: mq-brokeroptlogs
 60            - mountPath: /data/rocketmq
 61              name: rocketmq-data
 62              subPath: mq-brokeroptstore
 63            - name: broker-config
 64              mountPath: /usr/local/rocketmq-4.8.0/conf/broker-b.properties
 65              subPath: broker-b.properties
 66          lifecycle:
 67            postStart:
 68              exec:
 69                command: ["/bin/sh", "-c", "touch /tmp/health"]
 70          livenessProbe:
 71            exec:
 72              command: ["test", "-e", "/tmp/health"]
 73            initialDelaySeconds: 5
 74            timeoutSeconds: 5
 75            periodSeconds: 10
 76          readinessProbe:
 77            tcpSocket:
 78              port: 20911
 79            initialDelaySeconds: 15
 80            timeoutSeconds: 5
 81            periodSeconds: 20
 82      volumes:
 83        - name: broker-config
 84          configMap:
 85            name: rocketmq-config
 86  volumeClaimTemplates:
 87    - metadata:
 88        name: rocketmq-data
 89        namespace: sanjiang
 90        annotations:
 91          volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"
 92      spec:
 93        accessModes:
 94          - ReadWriteMany
 95        resources:
 96          requests:
 97            storage: 2Gi
 98---
 99apiVersion: v1
100kind: PersistentVolume
101metadata:
102  name: broker-b-pv
103  namespace: sanjiang
104spec:
105  accessModes:
106    - ReadWriteMany
107  capacity:
108    storage: 2Gi
109  persistentVolumeReclaimPolicy: Retain
110  storageClassName: rocketmq-nfs-storage
111  nfs:
112    path: /data/storage/k8s/rocketmq/broker-b
113    server: 193.0.40.171

broker-c-s.yaml

  1apiVersion: v1
  2kind: Service
  3metadata:
  4  labels:
  5    app: broker-c-s
  6  name: broker-c-s
  7  namespace: sanjiang
  8spec:
  9  ports:
 10    - port: 20911
 11      targetPort: 20911
 12      name: broker-port
 13  selector:
 14    app: broker-c-s
 15---
 16apiVersion: apps/v1
 17kind: StatefulSet
 18metadata:
 19  name: broker-c-s
 20  namespace: sanjiang
 21spec:
 22  serviceName: broker-c-s
 23  replicas: 1
 24  selector:
 25    matchLabels:
 26      app: broker-c-s
 27  template:
 28    metadata:
 29      labels:
 30        app: broker-c-s
 31    spec:
 32      affinity:
 33        podAntiAffinity:
 34          requiredDuringSchedulingIgnoredDuringExecution:
 35            - labelSelector:
 36                matchExpressions:
 37                  - key: "app"
 38                    operator: In
 39                    values:
 40                      - broker-c-s
 41              topologyKey: "kubernetes.io/hostname"
 42      containers:
 43        - name: broker-c-s
 44          image: liuyi71sinacom/rocketmq-4.8.0
 45          imagePullPolicy: IfNotPresent
 46          command:
 47            [
 48              "sh",
 49              "-c",
 50              "mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-c-s.properties",
 51            ]
 52          env:
 53            - name: JAVA_OPT
 54              value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
 55              #value: "-XX:MaxRAMPercentage=80.0"
 56          volumeMounts:
 57            - mountPath: /root/logs
 58              name: rocketmq-data
 59              subPath: mq-brokeroptlogs
 60            - mountPath: /data/rocketmq
 61              name: rocketmq-data
 62              subPath: mq-brokeroptstore
 63            - name: broker-config
 64              mountPath: /usr/local/rocketmq-4.8.0/conf/broker-c-s.properties
 65              subPath: broker-c-s.properties
 66          lifecycle:
 67            postStart:
 68              exec:
 69                command: ["/bin/sh", "-c", "touch /tmp/health"]
 70          livenessProbe:
 71            exec:
 72              command: ["test", "-e", "/tmp/health"]
 73            initialDelaySeconds: 5
 74            timeoutSeconds: 5
 75            periodSeconds: 10
 76          readinessProbe:
 77            tcpSocket:
 78              port: 20911
 79            initialDelaySeconds: 15
 80            timeoutSeconds: 5
 81            periodSeconds: 20
 82      volumes:
 83        - name: broker-config
 84          configMap:
 85            name: rocketmq-config
 86            items:
 87              - key: broker-c-s.properties
 88                path: broker-c-s.properties
 89  volumeClaimTemplates:
 90    - metadata:
 91        name: rocketmq-data
 92        namespace: sanjiang
 93        annotations:
 94          volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"
 95      spec:
 96        accessModes:
 97          - ReadWriteMany
 98        resources:
 99          requests:
100            storage: 2Gi
101---
102apiVersion: v1
103kind: PersistentVolume
104metadata:
105  name: broker-c-s-pv
106  namespace: sanjiang
107spec:
108  accessModes:
109    - ReadWriteMany
110  capacity:
111    storage: 2Gi
112  persistentVolumeReclaimPolicy: Retain
113  storageClassName: rocketmq-nfs-storage
114  nfs:
115    path: /data/storage/k8s/rocketmq/broker-c-s
116    server: 193.0.40.171

broker-c.yaml

  1apiVersion: v1
  2kind: Service
  3metadata:
  4  labels:
  5    app: broker-c
  6  name: broker-c
  7  namespace: sanjiang
  8spec:
  9  ports:
 10    - port: 20911
 11      targetPort: 20911
 12      name: broker-port
 13  selector:
 14    app: broker-c
 15---
 16apiVersion: apps/v1
 17kind: StatefulSet
 18metadata:
 19  name: broker-c
 20  namespace: sanjiang
 21spec:
 22  serviceName: broker-c
 23  replicas: 1
 24  selector:
 25    matchLabels:
 26      app: broker-c
 27  template:
 28    metadata:
 29      labels:
 30        app: broker-c
 31    spec:
 32      affinity:
 33        podAntiAffinity:
 34          requiredDuringSchedulingIgnoredDuringExecution:
 35            - labelSelector:
 36                matchExpressions:
 37                  - key: "app"
 38                    operator: In
 39                    values:
 40                      - broker-c
 41              topologyKey: "kubernetes.io/hostname"
 42      containers:
 43        - name: broker-c
 44          image: liuyi71sinacom/rocketmq-4.8.0
 45          imagePullPolicy: IfNotPresent
 46          command:
 47            [
 48              "sh",
 49              "-c",
 50              "mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-c.properties",
 51            ]
 52          env:
 53            - name: JAVA_OPT
 54              value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
 55              #value: "-XX:MaxRAMPercentage=80.0"
 56          volumeMounts:
 57            - mountPath: /root/logs
 58              name: rocketmq-data
 59              subPath: mq-brokeroptlogs
 60            - mountPath: /data/rocketmq
 61              name: rocketmq-data
 62              subPath: mq-brokeroptstore
 63            - name: broker-config
 64              mountPath: /usr/local/rocketmq-4.8.0/conf/broker-c.properties
 65              subPath: broker-c.properties
 66          lifecycle:
 67            postStart:
 68              exec:
 69                command: ["/bin/sh", "-c", "touch /tmp/health"]
 70          livenessProbe:
 71            exec:
 72              command: ["test", "-e", "/tmp/health"]
 73            initialDelaySeconds: 5
 74            timeoutSeconds: 5
 75            periodSeconds: 10
 76          readinessProbe:
 77            tcpSocket:
 78              port: 20911
 79            initialDelaySeconds: 15
 80            timeoutSeconds: 5
 81            periodSeconds: 20
 82      volumes:
 83        - name: broker-config
 84          configMap:
 85            name: rocketmq-config
 86  volumeClaimTemplates:
 87    - metadata:
 88        name: rocketmq-data
 89        namespace: sanjiang
 90        annotations:
 91          volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"
 92      spec:
 93        accessModes:
 94          - ReadWriteMany
 95        resources:
 96          requests:
 97            storage: 2Gi
 98---
 99apiVersion: v1
100kind: PersistentVolume
101metadata:
102  name: broker-c-pv
103  namespace: sanjiang
104spec:
105  accessModes:
106    - ReadWriteMany
107  capacity:
108    storage: 2Gi
109  persistentVolumeReclaimPolicy: Retain
110  storageClassName: rocketmq-nfs-storage
111  nfs:
112    path: /data/storage/k8s/rocketmq/broker-c
113    server: 193.0.40.171

broker-d-s.yaml

  1apiVersion: v1
  2kind: Service
  3metadata:
  4  labels:
  5    app: broker-d-s
  6  name: broker-d-s
  7  namespace: sanjiang
  8spec:
  9  ports:
 10    - port: 20911
 11      targetPort: 20911
 12      name: broker-port
 13  selector:
 14    app: broker-d-s
 15---
 16apiVersion: apps/v1
 17kind: StatefulSet
 18metadata:
 19  name: broker-d-s
 20  namespace: sanjiang
 21spec:
 22  serviceName: broker-d-s
 23  replicas: 1
 24  selector:
 25    matchLabels:
 26      app: broker-d-s
 27  template:
 28    metadata:
 29      labels:
 30        app: broker-d-s
 31    spec:
 32      affinity:
 33        podAntiAffinity:
 34          requiredDuringSchedulingIgnoredDuringExecution:
 35            - labelSelector:
 36                matchExpressions:
 37                  - key: "app"
 38                    operator: In
 39                    values:
 40                      - broker-d-s
 41              topologyKey: "kubernetes.io/hostname"
 42      containers:
 43        - name: broker-d-s
 44          image: liuyi71sinacom/rocketmq-4.8.0
 45          imagePullPolicy: IfNotPresent
 46          command:
 47            [
 48              "sh",
 49              "-c",
 50              "mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-d-s.properties",
 51            ]
 52          env:
 53            - name: JAVA_OPT
 54              value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
 55              #value: "-XX:MaxRAMPercentage=80.0"
 56          volumeMounts:
 57            - mountPath: /root/logs
 58              name: rocketmq-data
 59              subPath: mq-brokeroptlogs
 60            - mountPath: /data/rocketmq
 61              name: rocketmq-data
 62              subPath: mq-brokeroptstore
 63            - name: broker-config
 64              mountPath: /usr/local/rocketmq-4.8.0/conf/broker-d-s.properties
 65              subPath: broker-d-s.properties
 66          lifecycle:
 67            postStart:
 68              exec:
 69                command: ["/bin/sh", "-c", "touch /tmp/health"]
 70          livenessProbe:
 71            exec:
 72              command: ["test", "-e", "/tmp/health"]
 73            initialDelaySeconds: 5
 74            timeoutSeconds: 5
 75            periodSeconds: 10
 76          readinessProbe:
 77            tcpSocket:
 78              port: 20911
 79            initialDelaySeconds: 15
 80            timeoutSeconds: 5
 81            periodSeconds: 20
 82      volumes:
 83        - name: broker-config
 84          configMap:
 85            name: rocketmq-config
 86            items:
 87              - key: broker-d-s.properties
 88                path: broker-d-s.properties
 89  volumeClaimTemplates:
 90    - metadata:
 91        name: rocketmq-data
 92        namespace: sanjiang
 93        annotations:
 94          volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"
 95      spec:
 96        accessModes:
 97          - ReadWriteMany
 98        resources:
 99          requests:
100            storage: 2Gi
101---
102apiVersion: v1
103kind: PersistentVolume
104metadata:
105  name: broker-d-s-pv
106  namespace: sanjiang
107spec:
108  accessModes:
109    - ReadWriteMany
110  capacity:
111    storage: 2Gi
112  persistentVolumeReclaimPolicy: Retain
113  storageClassName: rocketmq-nfs-storage
114  nfs:
115    path: /data/storage/k8s/rocketmq/broker-d-s
116    server: 193.0.40.171

broker-d.yaml

  1apiVersion: v1
  2kind: Service
  3metadata:
  4  labels:
  5    app: broker-d
  6  name: broker-d
  7  namespace: sanjiang
  8spec:
  9  ports:
 10    - port: 20911
 11      targetPort: 20911
 12      name: broker-port
 13  selector:
 14    app: broker-d
 15---
 16apiVersion: apps/v1
 17kind: StatefulSet
 18metadata:
 19  name: broker-d
 20  namespace: sanjiang
 21spec:
 22  serviceName: broker-d
 23  replicas: 1
 24  selector:
 25    matchLabels:
 26      app: broker-d
 27  template:
 28    metadata:
 29      labels:
 30        app: broker-d
 31    spec:
 32      affinity:
 33        podAntiAffinity:
 34          requiredDuringSchedulingIgnoredDuringExecution:
 35            - labelSelector:
 36                matchExpressions:
 37                  - key: "app"
 38                    operator: In
 39                    values:
 40                      - broker-d
 41              topologyKey: "kubernetes.io/hostname"
 42      containers:
 43        - name: broker-d
 44          image: liuyi71sinacom/rocketmq-4.8.0
 45          imagePullPolicy: IfNotPresent
 46          command:
 47            [
 48              "sh",
 49              "-c",
 50              "mqbroker  -c /usr/local/rocketmq-4.8.0/conf/broker-d.properties",
 51            ]
 52          env:
 53            - name: JAVA_OPT
 54              value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
 55              #value: "-XX:MaxRAMPercentage=80.0"
 56          volumeMounts:
 57            - mountPath: /root/logs
 58              name: rocketmq-data
 59              subPath: mq-brokeroptlogs
 60            - mountPath: /data/rocketmq
 61              name: rocketmq-data
 62              subPath: mq-brokeroptstore
 63            - name: broker-config
 64              mountPath: /usr/local/rocketmq-4.8.0/conf/broker-d.properties
 65              subPath: broker-d.properties
 66          lifecycle:
 67            postStart:
 68              exec:
 69                command: ["/bin/sh", "-c", "touch /tmp/health"]
 70          livenessProbe:
 71            exec:
 72              command: ["test", "-e", "/tmp/health"]
 73            initialDelaySeconds: 5
 74            timeoutSeconds: 5
 75            periodSeconds: 10
 76          readinessProbe:
 77            tcpSocket:
 78              port: 20911
 79            initialDelaySeconds: 15
 80            timeoutSeconds: 5
 81            periodSeconds: 20
 82      volumes:
 83        - name: broker-config
 84          configMap:
 85            name: rocketmq-config
 86  volumeClaimTemplates:
 87    - metadata:
 88        name: rocketmq-data
 89        namespace: sanjiang
 90        annotations:
 91          volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage"
 92      spec:
 93        accessModes:
 94          - ReadWriteMany
 95        resources:
 96          requests:
 97            storage: 2Gi
 98---
 99apiVersion: v1
100kind: PersistentVolume
101metadata:
102  name: broker-d-pv
103  namespace: sanjiang
104spec:
105  accessModes:
106    - ReadWriteMany
107  capacity:
108    storage: 2Gi
109  persistentVolumeReclaimPolicy: Retain
110  storageClassName: rocketmq-nfs-storage
111  nfs:
112    path: /data/storage/k8s/rocketmq/broker-d
113    server: 193.0.40.171

console.yaml

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  creationTimestamp: null
 5  labels:
 6    app: console
 7  name: console
 8  namespace: sanjiang
 9spec:
10  replicas: 1
11  selector:
12    matchLabels:
13      app: console
14  strategy: {}
15  template:
16    metadata:
17      creationTimestamp: null
18      labels:
19        app: console
20    spec:
21      containers:
22        - image: styletang/rocketmq-console-ng
23          name: rocketmq-console-ng
24          env:
25            - name: JAVA_OPTS
26              value: "-Drocketmq.namesrv.addr=rocketmq:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false"
27          resources: {}
28status: {}
29---
30apiVersion: v1
31kind: Service
32metadata:
33  creationTimestamp: null
34  labels:
35    app: console
36  name: console
37  namespace: sanjiang
38spec:
39  ports:
40    - port: 8080
41      protocol: TCP
42      targetPort: 8080
43      nodePort: 32080
44  selector:
45    app: console
46  type: NodePort

namesrv.yaml

 1apiVersion: v1
 2kind: Service
 3metadata:
 4  labels:
 5    app: mq-namesrv
 6  name: rocketmq
 7  namespace: sanjiang
 8spec:
 9  ports:
10    - port: 9876
11      targetPort: 9876
12      nodePort: 32076
13      name: namesrv-port
14  selector:
15    app: mq-namesrv
16  type: NodePort
17---
18apiVersion: apps/v1
19kind: StatefulSet
20metadata:
21  name: rocketmq
22  namespace: sanjiang
23spec:
24  serviceName: rocketmq
25  replicas: 1
26  selector:
27    matchLabels:
28      app: mq-namesrv
29  template:
30    metadata:
31      labels:
32        app: mq-namesrv
33    spec:
34      affinity:
35        podAntiAffinity:
36          requiredDuringSchedulingIgnoredDuringExecution:
37            - labelSelector:
38                matchExpressions:
39                  - key: "app"
40                    operator: In
41                    values:
42                      - mq-namesrv
43              topologyKey: "kubernetes.io/hostname"
44      containers:
45        - name: mq-namesrv
46          image: liuyi71sinacom/rocketmq-4.8.0
47          imagePullPolicy: IfNotPresent
48          command: ["sh", "/usr/local/rocketmq-4.8.0/bin/mqnamesrv"]
49          ports:
50            - containerPort: 9876
51              protocol: TCP
52          env:
53            - name: JAVA_OPT
54              value: "-server -XX:ParallelGCThreads=1 -Xms1g -Xmx1g -Xmn512m"
55              #value: "-XX:MaxRAMPercentage=80.0"
56          lifecycle:
57            postStart:
58              exec:
59                command: ["/bin/sh", "-c", "touch /tmp/health"]
60          livenessProbe:
61            exec:
62              command: ["test", "-e", "/tmp/health"]
63            initialDelaySeconds: 5
64            timeoutSeconds: 5
65            periodSeconds: 10
66          readinessProbe:
67            tcpSocket:
68              port: 9876
69            initialDelaySeconds: 15
70            timeoutSeconds: 5
71            periodSeconds: 20

ingress.yaml

 1apiVersion: extensions/v1beta1
 2kind: Ingress
 3metadata:
 4  name: rocketmq
 5  namespace: sanjiang
 6spec:
 7  ingressClassName: nginx
 8  rules:
 9    - host: mq.sj.com
10      http:
11        paths:
12          - path: /
13            backend:
14              serviceName: console
15              servicePort: 8080

部署服务

此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。原文来自 — 版权归原作者所有。

推荐订阅源

日常 on 打工人日志