目录
环境搭建
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.0/kafka_2.12-2.3.0.tgz
tar xf kafka_2.12-2.3.0.tgz && cd kafka_2.12-2.3.0
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
cp config/server.properties config/server0.properties
# broker.id=0
# listeners=PLAINTEXT://:9092
# log.dirs=/tmp/kafka-logs0
bin/kafka-server-start.sh -daemon config/server0.properties
cp config/server.properties config/server1.properties
# broker.id=1
# listeners=PLAINTEXT://:9093
# log.dirs=/tmp/kafka-logs1
bin/kafka-server-start.sh -daemon config/server1.properties
cp config/server.properties config/server2.properties
# broker.id=2
# listeners=PLAINTEXT://:9094
# log.dirs=/tmp/kafka-logs2
bin/kafka-server-start.sh -daemon config/server2.properties
jps
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic topic-demo --partitions 3 --replication-factor 3
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 -topic topic-demo
# hello kafka
bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094 --topic topic-demo
# >hello kafka
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic topic-demo
# Topic: topic-demo PartitionCount:3 ReplicationFactor:3 Configs:
# Topic: topic-demo Partition: 0 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
# Topic: topic-demo Partition: 1 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
# Topic: topic-demo Partition: 2 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
水平扩展
Producer
- 扩展Topic => Multi Partition
image.png
- Broker Count <font color="#FF0000">MUST >=</font> Partition Factor
Comsumer
- 扩展Consume => Comsumer Group
image.png
- Consumer Count <font color="#00FF00">SHOULD <=</font> Partition Factor
高可用性
ZooKeeper
ZooKeeper Cluster
Server failure detection
Data partitioning
In-sync data replication
Replicas
image.png
数据一致
image.png
AR(所有副本) = ISR(和leader同步的副本) + OSR(未和leader同步的副本)
不是完全同步 => 因为性能较差 & 也不是单纯异步 => 因为一致性较差
