没想到最后把手里内存足够的机子都装了个遍.jpG
0x00.前言
为了之后的性能测试，不得不在各个平台上都装一份，以便进行横向对比
0x01.安装Java
Kafka的运行依赖于Java虚拟机，后者可以直接从各大包管理工具安装，非常方便，并不需要手动下载，再解压配置环境变量之类的……
1.基于RHEL 8系统
自Kafka 3.0.0版本起，The deprecation of support for Java 8 and Scala 2.12，于是选择安装openjdk 11
1
2
3
4
5
| [root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 ~]
openjdk version "11.0.15" 2022-04-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.15+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10-LTS, mixed mode, sharing)
|
2.基于Ubuntu 20.04系统
这里只列出命令，就不再赘述了
1
2
3
4
5
| ubuntu@cn-tx-bj6-u0:~$ sudo apt install openjdk-11-jdk
ubuntu@cn-tx-bj6-u0:~$ java -version
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.20.04.1)
OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.20.04.1, mixed mode, sharing)
|
截至目前，最新版本是3.2.0，然后Scala有2.12和2.13的，这里选择官方推荐的后者，即kafka_2.13-3.2.0.tgz
1.基于RHEL 8系统
和安装Java不同，需要手动下载预编译包，再解压并创建单元文件
1
2
| [root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 ~]
|
创建Kafka和Zookeeper的自定义数据目录，虽然本文并未实际使用到这些目录
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| [root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 kafka]
[root@cn-py-dl-r8 kafka]
[root@cn-py-dl-r8 kafka]
[root@cn-py-dl-r8 kafka]
[root@cn-py-dl-r8 kafka]
.
├── kafka-logs
└── zookeeper
├── data
│ └── myid
└── logs
4 directories, 1 file
|
创建Zookeeper的单元文件，虽然官方计划后续去掉Zookeeper，但现在还是得用到它
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| [root@cn-py-dl-r8 ~]
[Unit]
Description=Zookeeper service
After=network.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/bin/bash /root/kafka_2.13-3.2.0/bin/zookeeper-server-start.sh /root/kafka_2.13-3.2.0/config/zookeeper.properties
ExecStop=/bin/bash /root/kafka_2.13-3.2.0/bin/zookeeper-server-stop.sh
Restart=on-failure
[Install]
WantedBy=multi-user.target
|
创建Kafka的单元文件，在环境变量中设定了JMX_PORT=9988，并增大LimitNOFILE=65535
LimitNOFILE=65535的原因见：记一次 Kafka 未增大文件描述符限制的翻车经历
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| [root@cn-py-dl-r8 ~]
[Unit]
Description=Apache Kafka server (broker)
After=network.target zookeeper.service
[Service]
Type=simple
User=root
Group=root
Environment=JMX_PORT=9988
ExecStart=/bin/bash /root/kafka_2.13-3.2.0/bin/kafka-server-start.sh /root/kafka_2.13-3.2.0/config/server.properties
ExecStop=/bin/bash /root/kafka_2.13-3.2.0/bin/kafka-server-stop.sh
Restart=on-failure
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
|
2.基于Ubuntu 20.04系统
点击此处 ← 查看折叠
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
| ubuntu@cn-tx-bj6-u0:~$ wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz
ubuntu@cn-tx-bj6-u0:~$ tar -zxvf kafka_2.13-3.2.0.tgz
ubuntu@cn-tx-bj6-u0:~$ mkdir kafka
ubuntu@cn-tx-bj6-u0:~$ cd kafka
ubuntu@cn-tx-bj6-u0:~/kafka$ mkdir kafka-logs
ubuntu@cn-tx-bj6-u0:~/kafka$ mkdir -p zookeeper/data
ubuntu@cn-tx-bj6-u0:~/kafka$ echo 0 > zookeeper/data/myid
ubuntu@cn-tx-bj6-u0:~/kafka$ mkdir -p zookeeper/logs
ubuntu@cn-tx-bj6-u0:~$ sudo vim /etc/systemd/system/zookeeper.service
ubuntu@cn-tx-bj6-u0:~$ cat /etc/systemd/system/zookeeper.service
[Unit]
Description=Zookeeper service
After=network.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/bin/bash /home/ubuntu/kafka_2.13-3.2.0/bin/zookeeper-server-start.sh /home/ubuntu/kafka_2.13-3.2.0/config/zookeeper.properties
ExecStop=/bin/bash /home/ubuntu/kafka_2.13-3.2.0/bin/zookeeper-server-stop.sh
Restart=on-failure
[Install]
WantedBy=multi-user.target
ubuntu@cn-tx-bj6-u0:~$ sudo vim /usr/lib/systemd/system/kafka.service
ubuntu@cn-tx-bj6-u0:~$ cat /usr/lib/systemd/system/kafka.service
[Unit]
Description=Apache Kafka server (broker)
After=network.target zookeeper.service
[Service]
Type=simple
User=root
Group=root
Environment=JMX_PORT=9988
ExecStart=/bin/bash /home/ubuntu/kafka_2.13-3.2.0/bin/kafka-server-start.sh /home/ubuntu/kafka_2.13-3.2.0/config/server.properties
ExecStop=/bin/bash /home/ubuntu/kafka_2.13-3.2.0/bin/kafka-server-stop.sh
Restart=on-failure
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
|
CMAK是一款雅虎开源的非常好用的Kafka图形化管理工具，用完之后再也不想用命令行了.jpG，需Java 11
CMAK (Cluster Manager for Apache Kafka, previously known as Kafka Manager)
1.基于RHEL 8系统
下载最新版本预编译包cmak-3.0.0.6.zip，修改配置文件conf/application.conf中的kafka-manager.zkhosts和cmak.zkhosts的值为localhost:2181
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| [root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 cmak-3.0.0.6]
[root@cn-py-dl-r8 cmak-3.0.0.6]
……
kafka-manager.zkhosts="localhost:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
cmak.zkhosts="localhost:2181"
cmak.zkhosts=${?ZK_HOSTS}
……
|
创建CMAK的单元文件，如果是手动解压的Java，需要在ExecStart中添加参数-java-home /root/jdk-11.0.15
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| [root@cn-py-dl-r8 ~]
[Unit]
Description=kafka-manager server service
After=network.target
[Service]
User=root
Group=root
WorkingDirectory=/root/cmak-3.0.0.6
ExecStart=/root/cmak-3.0.0.6/bin/cmak -Dconfig.file=/root/cmak-3.0.0.6/conf/application.conf
Restart=on-failure
RestartSec=60
[Install]
WantedBy=multi-user.target
|
2.基于Ubuntu 20.04系统
点击此处 ← 查看折叠
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
| ubuntu@cn-tx-bj6-u0:~$ wget https://github.com/yahoo/CMAK/releases/download/3.0.0.6/cmak-3.0.0.6.zip
ubuntu@cn-tx-bj6-u0:~$ unzip cmak-3.0.0.6.zip
ubuntu@cn-tx-bj6-u0:~$ cd cmak-3.0.0.6/
ubuntu@cn-tx-bj6-u0:~/cmak-3.0.0.6$ vim conf/application.conf
ubuntu@cn-tx-bj6-u0:~/cmak-3.0.0.6$ cat conf/application.conf
……
kafka-manager.zkhosts="localhost:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
cmak.zkhosts="localhost:2181"
cmak.zkhosts=${?ZK_HOSTS}
……
ubuntu@cn-tx-bj6-u0:~$ sudo vim /usr/lib/systemd/system/kafka-manager.service
ubuntu@cn-tx-bj6-u0:~$ cat /usr/lib/systemd/system/kafka-manager.service
[Unit]
Description=kafka-manager server service
After=network.target
[Service]
User=root
Group=root
WorkingDirectory=/home/ubuntu/cmak-3.0.0.6
ExecStart=/home/ubuntu/cmak-3.0.0.6/bin/cmak -Dconfig.file=/home/ubuntu/cmak-3.0.0.6/conf/application.conf
Restart=on-failure
RestartSec=60
[Install]
WantedBy=multi-user.target
|
0x04.启动服务
因Kafka依赖于Zookeeper，所以需要先启动Zookeeper再启动Kafka
1
2
3
| [root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 ~]
[root@cn-py-dl-r8 ~]
|
0x05.后记
是不是很简单？自己装了四台之后，对于Kafka的安装愈发熟练草……
0x06.引用
Confluent：在 Kafka 上飞驰的数据交换者
Kafka的安装与使用
Kafka集群优化篇-调整broker的堆内存(heap)案例实操
Kafka 两个高性价比的参数调优
Kafka性能调优
Kafka参数调优实战，看这篇文章就够了！
centos8平台使用iostat监控磁盘io
Kafka性能测试及分析
Kafka 高性能吞吐揭秘
