周末花了两天时间入门学习一下 Kafka 的基本概念和使用。在学习Kafka之前,首先搭建Kafka需要的环境以便于之后的学习和项目准备。
docker和docker-compose的搭建和使用我就默认你都会了哈!如果不清楚的先了解一下...
集群搭建
废话不多说,上代码!
version: '3.1'
services:
zookeeper1:
image: zookeeper:3.4.13
container_name: zookeeper1
restart: always
hostname: zookeeper1
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=zookeeper1:2888:3888 server.2=zookeeper2:2888:3888 server.3=zookeeper3:2888:3888
zookeeper2:
image: zookeeper:3.4.13
container_name: zookeeper2
restart: always
hostname: zookeeper2
ports:
- 2182:2181
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zookeeper1:2888:3888 server.2=zookeeper2:2888:3888 server.3=zookeeper3:2888:3888
zookeeper3:
image: zookeeper:3.4.13
container_name: zookeeper3
restart: always
hostname: zookeeper3
ports:
- 2183:2181
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=zookeeper1:2888:3888 server.2=zookeeper2:2888:3888 server.3=zookeeper3:2888:3888
kafka1:
image: xuxiangwork/kafka
container_name: kafka1
ports:
- "9092:9092"
- "1099:1099"
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_HOST_NAME: 172.28.0.1
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2182,zookeeper3:2183
CUSTOM_JMX_PORT: "1099"
JVM_XMS: "256M"
JVM_XMX: "512M"
volumes:
- E:\kafka1:/kafka
restart: always
kafka2:
image: xuxiangwork/kafka
container_name: kafka2
ports:
- "9093:9092"
- "1100:1100"
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
environment:
KAFKA_BROKER_ID: 2
KAFKA_ADVERTISED_HOST_NAME: 172.28.0.2
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2182,zookeeper3:2183
CUSTOM_JMX_PORT: "1100"
JVM_XMS: "256M"
JVM_XMX: "512M"
volumes:
- E:\kafka2:/kafka
restart: always
kafka3:
image: xuxiangwork/kafka
container_name: kafka3
ports:
- "9094:9092"
- "1101:1101"
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
environment:
KAFKA_BROKER_ID: 3
KAFKA_ADVERTISED_HOST_NAME: 172.28.0.3
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2182,zookeeper3:2183
CUSTOM_JMX_PORT: "1101"
JVM_XMS: "256M"
JVM_XMX: "512M"
volumes:
- E:\kafka3:/kafka
restart: always
kafka-manager:
container_name: kafka-manager
image: xuxiangwork/kafka-manager
ports:
- "9000:9000"
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
environment:
ZK_HOSTS: zookeeper1:2181
APPLICATION_SECRET: letmein
KAFKA_MANAGER_AUTH_ENABLED: "true"
KAFKA_MANAGER_USERNAME: 你的控制台帐号
KAFKA_MANAGER_PASSWORD: 控制台密码
restart: always
ps.本地环境为Window10的环境,所以文件映射的路径和linux上搭建不一样,如果搭建在Linux系统或MacOS系统,修改映射文件就可以啦。
值得注意的是,如果为了能够快速将zookeeper和kafka集群搭建起来,建议直接使用以上代码。因为现在使用最新的zookeeper版本好像会搭建失败,具体原因没有细看,看到网上的说法是有可能新版本zookeeper镜像里的文件和之前不太一样,可能需要修改一些配置才可以,但我的目的是快速搭建环境学习Kafka,就不花那么多时间研究版本的问题...
另外,我还加上kafka-manager
服务,这个是一个可视化平台,可以方便的管理kafka节点的情况,访问地址为:http://localhost:9000 。至于如何操作管理平台,可参考 kafka-manager使用
使用docker ps
命令查看是否搭建成功
查看zookeeper节点的状态:
- 进入zookeeper服务:docker exec -it zookeeper1 /bin/bash
- 执行 ./bin/zkServer.sh status
由Mode可知,zookeeper1角色是leader,可以依照相似的命令查看其它zookeeper的角色和判断是否启动成功!
查看kafka节点的状态:
- 查看kafka启动日志:docker logs -f kafka1
-
通过kafka-manager管理平台查看
如果觉得文章不错的话,麻烦点个赞哈!对于文章有哪里不清楚或者有误的地方,欢迎在评论区留言~