Kafka: 消息中间件 -->分布式流式平台
MQ Redis
Kafka Flume
生产者 source
Broker channel
消费者 sink
正常部署的是： Broker进程
Flume-->Kafka-->Spark streaming 实时流式
Kafka Streaming
发布/订阅
实时app
分布式 副本数 一个
SCALA编写
部署:
kafka_2.11-0.10.0.0.tgz
scala version：2.11
准备环境:
1.JDK
java -version
2.Scala
scala -version
3.Zookeeper
[root@yws85 software]# ln -s zookeeper-3.4.6 zookeeper
[root@yws85 software]# ll
total 462992
lrwxrwxrwx 1 hdfs hdfs 33 Jan 5 2018 flink -> flink1.4.0-hadoop2.6.0-cdh5.12.0/
drwxr-xr-x 11 hdfs hdfs 4096 Jan 5 2018 flink1.4.0-hadoop2.6.0-cdh5.12.0
-rw-r--r-- 1 root root 424555111 Aug 20 2017 hadoop-2.8.1.tar.gz
-rw-r--r-- 1 root root 31831257 Aug 25 2018 kafka_2.11-0.10.0.1.tgz
drwxrwxr-x. 6 root root 46 Sep 12 2017 scala-2.11.8
lrwxrwxrwx 1 root root 15 Aug 17 23:38 zookeeper -> zookeeper-3.4.9
drwxr-xr-x 10 1000 1000 4096 Feb 20 2014 zookeeper-3.4.6 bug
zookeeper-3.4.9
-rw-r--r-- 1 root root 17699306 Jun 20 2017 zookeeper-3.4.6.tar.gz
[root@yws85 software]#
软连接
ln -s 物理文件夹/文件 快捷的文件夹/文件
1.删除 快捷的文件夹/文件 ，增加安全系数
2.多版本管理
3.硬连接
常用命令:
kafka-topics.sh
bin/kafka-topics.sh --create \
--zookeeper hadoop001:2181/kafka \
--replication-factor 1 \
--partitions 3 \
--topic test
高并发写/读
bin/kafka-topics.sh --list \
--zookeeper hadoop001:2181/kafka
生产者：
bin/kafka-console-producer.sh \
--broker-list yws85:9092,yws86:9092,yws87:9092 \
--topic test
消费者
bin/kafka-console-consumer.sh \
--zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \
--topic test \
--from-beginning
bin/kafka-topics.sh \
--describe \
--zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \
--topic test
bin/kafka-topics.sh \
--alter \
--zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \
--topic test \
--partitions 4
干净删除:
bin/kafka-topics.sh --delete \
--zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \
--topic test
rmr /kafka/admin/delete_topics/test
rmr /kafka/config/topics/test
rmr /kafka/brokers/topics/test
rm -rf logs/test-*
概念:
topic：主题
ERP系统log-->erp_topic
OA系统log-->oa_topic
partitions：分区
--partitions 3 3个分区 下标从0开始
--replication-factor 3 一个分区3个副本
单个分区：有序
全局分区: 无序
如何保证生产上 全局有序？
MySQL BINLOG 日志文件 按顺序----有序-----》Kafka test 3个分区
ruozedata.stu
id name age
insert into stu values(1,'jepson',18);
insert into stu values(2,'ruoze',28);
update stu set age=26 where id=1;
delete from stu where id=1;
test-0：
insert into stu values(1,'jepson',18);
test-1：
insert into stu values(2,'ruoze',28);
delete from stu where id=1;
test-2：
update stu set age=26 where id=1;
消费时
insert into stu values(2,'ruoze',28);
delete from stu where id=1;
insert into stu values(1,'jepson',18);
update stu set age=26 where id=1;
核心点: 共性数据发送到同一个topic的1个分区
拼装的Key: ruozedata_stu_id=1 代码 hash(ruozedata_stu_id=1 ) 取模 0,1,2
value: sql
test-0：
insert into stu values(1,'jepson',18);
update stu set age=26 where id=1;
delete from stu where id=1;
test-1：
insert into stu values(2,'ruoze',28);
test-2：
[root@yws86 kafka]# bin/kafka-topics.sh \
> --describe \
> --zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \
> --topic test
Topic:test PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: test Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: test Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
[root@yws86 kafka]#
第一行: 哪个topic 多少个分区 每个分区多少个副本数
Topic: test Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
分区-0
Replicas: 1,2,3 三个副本位于
Leader: 1 读写
Isr: 1,2,3
[root@yws86 kafka]# bin/kafka-topics.sh \
> --describe \
> --zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \
> --topic test
Topic:test PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 2 Replicas: 1,2,3 Isr: 2,3
Topic: test Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: test Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
[root@yws86 kafka]# bin/kafka-topics.sh --describe --zookeeper yws85:2181,yws86:2181,yws87:2181/kafka --topic test
Topic:test PartitionCount:4 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1
Topic: test Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: test Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,2,1
Topic: test Partition: 3 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2
[root@yws86 kafka]#
1 /3 1
2 2
3 0
4 1
5 2
6 0
7
8
9
10 1
HBASE Phoniex 部署
