需求
历史任务基于 Spark1. 新任务计划转移到 Spark2. 需要 Oozie 同时支持两个版本.
步骤
1 配置 sharelib
参照 Hortonworks 文档 创建 spark2 sharelib:
hdfs dfs -mkdir /user/oozie/share/lib/lib_<ts>/spark2
hdfs dfs -put \
/usr/hdp/<version>/spark2/jars/* \
/user/oozie/share/lib/lib_<ts>/spark2/
hdfs dfs -cp \
/user/oozie/share/lib/lib_<ts>/spark/oozie-sharelib-spark-<version>.jar \
/user/oozie/share/lib/lib_<ts>/spark2/
hdfs dfs -cp \
/user/oozie/share/lib/lib_<ts>/spark/hive-site.xml \
/user/oozie/share/lib/lib_<ts>/spark2/
hdfs dfs -put \
/usr/hdp/<version>/spark2/python/lib/py* \
/user/oozie/share/lib/lib_<ts>/spark2/
oozie admin –sharelibupdate
检查是否创建成功:
oozie admin –shareliblist spark2
2 配置 job.properties
设置以下属性值:
oozie.action.sharelib.for.spark=spark2
2 删除重复的 jar 包
完成上述配置后运行 workflow 会报重复加载文件的错误:
WARN SparkActionExecutor:523 - SERVER[] USER[admin] GROUP[-] TOKEN[] APP[Workflow2] JOB[0000012-170717153234639-oozie-oozi-W] ACTION[0000012-170717153234639-oozie-oozi-W@spark_1] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Attempt to add (hdfs://:8020/user/oozie/share/lib/lib_20170613110051/oozie/aws-java-sdk-core-1.10.6.jar) multiple times to the distributed cache.
2017-07-19 12:36:53,275 WARN SparkActionExecutor:523 - SERVER[] USER[admin] GROUP[-] TOKEN[] APP[Workflow2] JOB[0000012-170717153234639-oozie-oozi-W] ACTION[0000012-170717153234639-oozie-oozi-W@spark_1] Launcher exception: Attempt to add (hdfs://:8020/user/oozie/share/lib/lib_20170613110051/oozie/aws-java-sdk-core-1.10.6.jar) multiple times to the distributed cache.
java.lang.IllegalArgumentException: Attempt to add (hdfs://:8020/user/oozie/share/lib/lib_20170613110051/oozie/aws-java-sdk-core-1.10.6.jar) multiple times to the distributed cache.
原因
/user/oozie/share/lib/lib_<TS>/oozie/ 和 /user/oozie/share/lib/lib_<TS>/spark2/ 下存在同名 jar 包.
解决方法
删除 spark2(尽量减少对其他非 spark2 任务的影响) 下的同名 jar 包.
3 新旧版本的 jar 包兼容
完成上述配置后运行 workflow 会报以下错误:
[0000591-171111105051439-oozie-oozi-W] ACTION[0000591-171111105051439-oozie-oozi-W@spark-b645] Launcher exception: com.fasterxml.jackson.databind.JavaType.isReferenceType()Z
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JavaType.isReferenceType()Z
at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByLookup(BasicSerializerFactory.java:302)
原因
spark2 加载了 /user/oozie/share/lib/lib_<TS>/oozie/ 下 版本过旧的 jar 包:
jackson-annotations-2.4.0.jar (?)
oozie/jackson-core-2.4.4.jar
jackson-databind-2.4.4.jar
解决方法
由于以上版本的 jar 被Spark1任务依赖, 不能直接更新或删除. 因为这个 Oozie 示例只运行 spark action, 简单地把这些
jar move() 到 /user/oozie/share/lib/lib_<TS>/spark/ 目录即可.
此时, spark1 任务加载的是 /user/oozie/share/lib/lib_<TS>/spark/ 下的
jackson-annotations-2.4.0.jar
oozie/jackson-core-2.4.4.jar
jackson-databind-2.4.4.jar
此时, spark2 任务加载的是 /user/oozie/share/lib/lib_<TS>/spark2/ 下的
jackson-annotations-2.6.5.jar
oozie/jackson-core-2.6.5.jar
jackson-databind-2.6.5.jar
其他解决方法
hortonworks 有提到可以重新生成oozie sharelib:
/usr/hdp/2.3.2.0-2950/oozie/bin/oozie-setup.sh sharelib create -locallib /usr/hdp/<version>/oozie/oozie-sharelib.tar.gz -fs hdfs://<namenode-host>:8020
oozie admin -oozie http://<oozie-host>:11000/oozie -sharelibupdate
