spark 2.2.1 client

1. JAVA_HOME not set

vim spark-env.sh 

export JAVA_HOME=/usr/local/jdk
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_HOME=/usr/hdp/current/hadoop-client

2. Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig

cp /usr/hdp/current/hadoop-client/client/jersey-core-1.9.jar /usr/local/spark-2.2.1/jars/
cp /usr/hdp/current/hadoop-client/client/jersey-client-1.9.jar /usr/local/spark-2.2.1/jars/

rm -rf /usr/local/spark-2.2.1/jars/jersey-client-2.22.2.jar

3. java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found (Centos7出现的问题)

从ambari上修改yarn-site配置

搜索关键字: yarn.client.failover-proxy-provider 
修改为: org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider

4. bad substitution

/hadoop/yarn/local/usercache/wormhole/appcache/application_1527591700659_0001/container_e05_1527591700659_0001_01_000001/launch_container.sh: line 30: PWD:PWD/spark_conf:PWD/__spark_libs__/*:/usr/hdp/2.6.4.0-91/hadoop/conf:/usr/hdp/2.6.4.0-91/hadoop/*:/usr/hdp/2.6.4.0-91/hadoop/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*:PWD/mr-framework/hadoop/share/hadoop/mapreduce/:PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:PWD/mr-framework/hadoop/share/hadoop/common/:PWD/mr-framework/hadoop/share/hadoop/common/lib/*:PWD/mr-framework/hadoop/share/hadoop/yarn/:PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:PWD/mr-framework/hadoop/share/hadoop/hdfs/:PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:PWD/mr-framework/hadoop/share/hadoop/tools/lib/:/usr/hdp/{hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.{hdp.version}.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/: bad substitution

step 1:

vim java-opts 

-Dhdp.version=2.6.4.0-91

step 2:
1. Go to 'Ambari -> YARN -> configs' and go to 'Advanced' tab.
2. scroll down the page to till end, there will find an option to add custom property for yarn-site
3. click on 'add property' and enter 'hdp.version' and the version value.

hdp.version=2.6.4.0-91

4. save the changes and restart the required services. It will deploy the hdp.verion property in yarn-site.xml

注意：之前帮客户解决问题的时候，hdp.version=2.5.0.0-1245版本时按照上述方式设置不能解决问题，貌似HDP 2.6以下版本需要按照下面的方式解决。

step 1:

vim java-opts 

-Dhdp.version=2.6.4.0-91

step 2:

vim spark-defaults.conf

spark.driver.extraJavaOptions   -Dhdp.version=2.5.0.0-1245
spark.yarn.am.extraJavaOptions  -Dhdp.version=2.5.0.0-1245

注意：用户提交spark作业时，若重新设置这两个参数的值，会覆盖默认配置，所以用户在配置这两个变量时需要加上hdp.version的配置。

5. spark applicaiton executor ui显示问题

vi spark-default.conf 
spark.hadoop.yarn.timeline-service.enabled false