版本与环境信息
| 组件 | 版本 |
|---|---|
| hbase | 2.3.5 |
| hadoop | 3.2.2 |
| HADOOP_HOME | /usr/local/service/hadoop |
| HBASE_HOME | /usr/local/service/hbase |
前期准备
基于hbase 2.3.5重新编译yarn timeline service
cd /usr/local/service/hadoop/share/hadoop/yarn/timelineservice
export HBASE_CLASSPATH=hadoop-yarn-server-timelineservice-hbase-common-3.2.2.jar:hadoop-yarn-server-timelineservice-hbase-client-3.2.2.jar
hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -create
此时会报错
企业微信截图_47f57947-9cc6-4213-9fb3-a9bfbbb9a147.png
由于上面的jar包是基于hbase 1.x编译的，我们需要基于hbase2.x重新编译
hadoop-yarn-server-timelineservice-hbase这个module，编译之前我们需要改代码，否则就会报NoSuchMethodError，这里很坑，你不改也可以编译过，但是执行时就会报NoSuchMethodError。
image.png
如上图，我们需要将
HTableDescriptor、HColumnDescriptor都改成 TableDescriptor、ColumnFamilyDescriptor，修改完毕后重新编译：
export HADOOP_PROTOC_PATH=/usr/local/protoc2/bin/protoc
mvn package -Dmaven.test.skip=true -Dhbase.profile=2.0 -Dhbase.version=2.3.5
创建hbase表
scp hadoop-yarn-server-timelineservice-hbase-client/target/hadoop-yarn-server-timelineservice-hbase-client-3.2.2.jar hadoop-yarn-server-timelineservice-hbase-common/target/hadoop-yarn-server-timelineservice-hbase-common-3.2.2.jar hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/target/hadoop-yarn-server-timelineservice-hbase-server-2-3.2.2.jar hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/target/hadoop-yarn-server-timelineservice-hbase-server-2-3.2.2-coprocessor.jar root@myip:/root
mkdir bak
mv /usr/local/service/hadoop/share/hadoop/yarn/timelineservice/hadoop-yarn-server-timelineservice-* bak/
chown hadoop:hadoop hadoop-yarn-server-timelineservice-hbase-*.jar
su hadoop && cd
mv hadoop-yarn-server-timelineservice-hbase-*.jar /usr/local/service/hadoop/share/hadoop/yarn/timelineservice/
cd /usr/local/service/hadoop/share/hadoop/yarn/timelineservice/
hadoop fs -put -f hadoop-yarn-server-timelineservice-hbase-server-2-3.2.2-coprocessor.jar /hbase/coprocessor/hadoop-yarn-server-timelineservice.jar
export HBASE_CLASSPATH=hadoop-yarn-server-timelineservice-hbase-client-3.2.2.jar:hadoop-yarn-server-timelineservice-hbase-common-3.2.2.jar
hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -create -skipExistingTable
至此timeline server需要的hbase就创建好了，查看下hbase中的表
image.png
配置timeline service v2
复制jar包到其他NodeManager节点
scp /usr/local/service/hadoop/share/hadoop/yarn/timelineservice/hadoop-yarn-server-timelineservice-hbase-*.jar 172.27.16.89:/usr/local/service/hadoop/share/hadoop/yarn/timelineservice/
- 配置yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,timeline_collector</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.timeline_collector.class</name>
<value>org.apache.hadoop.yarn.server.timelineservice.collector.PerNodeTimelineCollectorsAuxService</value>
</property>
<property>
<name>yarn.timeline-service.reader.webapp.address</name>
<value>0.0.0.0:13000</value>
</property>
<property>
<name>yarn.timeline-service.version</name>
<value>2</value>
</property>
<property>
<name>yarn.timeline-service.http-cross-origin.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.system-metrics-publisher.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.rm.system-metrics-publisher.emit-container-events</name>
<value>true</value>
</property>
<property>
<name>yarn.timeline-service.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.timeline-service.hbase.configuration.file</name>
<value>/usr/local/service/hbase/conf/hbase-site.xml</value>
</property>
<property>
<name>yarn.timeline-service.bind-host</name>
<value>0.0.0.0</value>
</property>
启动 timeserver reader
先添加一些jar包
cd /usr/local/service/hadoop/share/hadoop/yarn/timelineservice
cp -r lib lib2
cd lib2 && rm -f hbase-* jcodings-* joni-* metrics-core-*
cd /usr/local/service/hbase/lib
cp hbase-annotations-2.3.5.jar shaded-clients/hbase-shaded-client-2.3.5.jar hbase-common-2.3.5.jar hbase-protocol-2.3.5.jar jcodings-1.0.18.jar joni-2.1.11.jar metrics-core-3.2.6.jar /usr/local/service/hadoop/share/hadoop/yarn/timelineservice/lib2
- 修改
/usr/local/service/hadoop/bin/yarn，将timelineservice/lib修改为timelineservice/lib2 - 执行命令
/usr/local/service/hadoop/sbin/yarn-daemon.sh start timelinereader，报找不到org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer，把包拷贝上去
scp hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/target/hadoop-yarn-server-timelineservice-3.2.2.jar root@42.193.20.242:/usr/local/service/hadoop/share/hadoop/yarn/
启动 timelinereader
export YARN_LOGFILE=$USER-timelinereader.log
/usr/local/service/hadoop/sbin/yarn-daemon.sh start timelinereader
在浏览器中通过外网访问下
image.png
验证ResourceManager和NodeManager
将刚刚的修改（yarn-site.xml、/usr/local/service/hadoop/share/hadoop/yarn/timelineservice/lib2、/usr/local/service/hadoop/bin/yarn、hadoop-yarn-server-timelineservice-3.2.2.jar）同步到其他节点上
scp /usr/local/service/hadoop/bin/yarn root@172.27.16.6:/usr/local/service/hadoop/bin/
scp -r /usr/local/service/hadoop/share/hadoop/yarn/timelineservice/lib2 root@172.27.16.6:/usr/local/service/hadoop/share/hadoop/yarn/timelineservice
scp /usr/local/service/hadoop/share/hadoop/yarn/hadoop-yarn-server-timelineservice-3.2.2.jar root@172.27.16.6:/usr/local/service/hadoop/share/hadoop/yarn/
- 重启ResourceManager
- 重启NodeManager
- 提交任务
hadoop jar /usr/local/service/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2-tests.jar org.apache.hadoop.mapreduce.SleepJob -Dmapreduce.job.emit-timeline-data=true -m 2 -mt 1000000 -r 0
通过浏览器来测试一下:
企业微信截图_f7fc894b-151f-4256-811c-1d8deb4b275a.png
http://42.193.20.242:13000/ws/v2/timeline/users/hadoop/flows/Sleep%20job/runs/
企业微信截图_e2a2550b-67db-4677-85b6-418d3d0c6d08.png
http://42.193.20.242:13000/ws/v2/timeline/users/hadoop/flows/Sleep%20job/runs/1687315891974
image.png
http://42.193.20.242:13000/ws/v2/timeline/users/hadoop/flows/Sleep%20job/runs/1687315891974/apps
image.png
http://42.193.20.242:13000/ws/v2/timeline/apps/application_1687315771573_0001
image.png
http://42.193.20.242:13000/ws/v2/timeline/apps/application_1687315771573_0001/entity-types
企业微信截图_4014e2d9-c3ca-4723-b952-0994a28ce299.png
image.png
image.png
