Spark On Yarn 远程idea提交运行(不是调试)
1. 需要注意的问题
1.1 centos搭建的集群会出现is running beyond virtual memory limits的问题
Current usage: xx MB of xxGB physical memory used; xx GB of xx GB virtual memory used.
解决方法:
# yarn-site.xml中添加以下属性
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
1.2 在linux下使用idea连接docker搭建的集群，之间虽然能够互相ping通，但是还是有防火墙依然会让集群不能访问宿主机
19/01/21 16:44:16 INFO Client: Application report for application_1548058747747_0006 (state: ACCEPTED)
程序运行一直出现这个记录, 解决办法：关闭防火墙
1.3 宿主机占不到集群，一直使用0.0.0.0:8032端口(这一步设置很重要)
这是因为没有把resource资源文件设置成资源文件, 解决方案:
右键点击resource文件，选择Mark Directory as >> Resources root
2. 最终文件形式(src部分)
在idea新建项目, sbt构建项目, sbt版本随意, scala版本选择2.11.8, 因为我的集群中没有专门配置scala,因此用spark-2.3.1-bin-hadoop2.7自带的scala, 其版本号就是2.11.8， src目录如下
# 右键点击resource选择Mark Directory as >> Resources root， 或者去project struct设置
src
├── main
│ ├── resource
│ │ ├── core-site.xml
│ │ ├── hdfs-site.xml
│ │ └── yarn-site.xml
│ └── scala
│ ├── SparkPI.scala
│ └── WordCount.scala
└── test
└── scala
2.1 以提交wordcount为例子
单单这些代码是不能运行的，还需要设置集群，1) 添加集群jars包, 2) 使用sbt打包
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
object WordCount {
def main(args: Array[String]): Unit = {
System.setProperty("HADOOP_USER_NAME", "root")
System.setProperty("user.name", "root")
val conf = new SparkConf().setAppName("WordCount").setMaster("yarn")
.set("deploy-mode", "client")
.set("spark.yarn.jars", "hdfs:/user/root/jars/*") //集群的jars包,是你自己上传上去的
.setJars(List("/home/lee/IdeaProjects/test/target/scala-2.11/test_2.11-0.1.jar")) //这是sbt打包后的文件
.setIfMissing("spark.driver.host", "192.168.1.9") //设置你自己的ip
val sc = new SparkContext(conf)
val rdd = sc.textFile("hdfs:/input/README.txt")
val count = rdd.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_)
count.collect().foreach(println)
}
}
2.2 依赖
# build.sbt中添加一下内容
// https://mvnrepository.com/artifact/org.apache.spark/spark-yarn
libraryDependencies += "org.apache.spark" %% "spark-yarn" % "2.3.1"
3. 步骤
3.1 设置jars
注意 wordcount中conf中的.set("spark.yarn.jars", "hdfs:/user/root/jars/*"),这里面由于没有在本地添加spark的jars包，因此直接使用集群中的jars包, 这个包需要在集群里面提交
# 在docker环境下, 可以使用如下指令
docker exec spark-master /opt/module/hadoop/bin/hdfs dfs -mkdir /input
docker exec spark-master /opt/module/hadoop/bin/hdfs dfs -mkdir /user
docker exec spark-master /opt/module/hadoop/bin/hdfs dfs -mkdir /user/root
docker exec spark-master /opt/module/hadoop/bin/hdfs dfs -mkdir /user/root/jars
docker exec spark-master /opt/module/hadoop/bin/hdfs dfs -put /opt/module/spark/jars/* /user/root/jars
docker exec spark-master /opt/module/hadoop/bin/hdfs dfs -put /opt/module/hadoop/README.txt /input
# /opt/module/hadoop/ 是你自己的hadoop目录
# /opt/module/spark/ 是你自己的spark目录
# 在集群中，假如环境都设置好了，那么就可以
hdfs dfs -mkdir /input
hdfs dfs -mkdir /user
hdfs dfs -mkdir /user/root
hdfs dfs -mkdir /user/root/jars
hdfs dfs -put your_spark_path/jars/* /user/root/jars
hdfs dfs -put /opt/module/hadoop/README.txt /input
当然如果你不喜欢用/user/root目录来放jars，那么也可以自定义，当然在wordcount里面就要做出对应改变了。
3.3 选用本地jars包(与3.1二选一)
如果不想提交spark的jars包到集群，那么可以把spark的jars可以复制到项目里
ls /opt/module/spark
bin conf data examples jars kubernetes LICENSE licenses logs NOTICE python R README.md RELEASE sbin work yarn
对就是SPARK_HOME目录下的jars文件夹, 复制到项目, 最终你的 your_project/jars里面应该是下面这些内容
activation-1.1.1.jar hadoop-yarn-client-2.7.3.jar metrics-graphite-3.1.5.jar
......
zstd-jni-1.3.2-2.jar
hadoop-yarn-api-2.7.3.jar metrics-core-3.1.5.jar
选择file>>project structure>>module, 选择name方框下的dependecies，在点击该栏目右上方的+号， 选择1. jars and Directories, 再弹出框中选择 your_project/jars
3.3 打包
在idea底部选择sbt shell
第一次输入clean
第二次输入package
如果选择其他的打包方式，那就需要修改conf的setJars。
4. 运行
19/01/21 16:44:41 INFO YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool
19/01/21 16:44:41 INFO DAGScheduler: ResultStage 1 (collect at WordCount.scala:20) finished in 0.827 s
19/01/21 16:44:41 INFO DAGScheduler: Job 0 finished: collect at WordCount.scala:20, took 6.945556 s
(under,1)
(this,3)
(distribution,2)
(Technology,1)
(country,1)
(is,1)
(Jetty,1)
(currently,1)
(permitted.,1)
(check,1)
(have,1)
(Security,1)
(U.S.,1)
(with,1)
(BIS,1)
(This,1)
(mortbay.org.,1)
((ECCN),1)
(using,2)
(security,1)
(Department,1)
(export,1)
(reside,1)
(any,1)
(algorithms.,1)
(from,1)
(re-export,2)
(has,1)
(SSL,1)
(Industry,1)
(Administration,1)
(details,1)
(provides,1)
(http://hadoop.apache.org/core/,1)
(country's,1)
(Unrestricted,1)
(740.13),1)
(policies,1)
(country,,1)
(concerning,1)
(uses,1)
(Apache,1)
(possession,,2)
(information,2)
(our,2)
(as,1)
(,18)
(Bureau,1)
(wiki,,1)
(please,2)
(form,1)
(information.,1)
(ENC,1)
(Export,2)
(included,1)
(asymmetric,1)
(Commodity,1)
(Software,2)
(For,1)
(it,1)
(The,4)
(about,1)
(visit,1)
(website,1)
(<http://www.wassenaar.org/>,1)
(performing,1)
(Section,1)
(on,2)
((see,1)
(http://wiki.apache.org/hadoop/,1)
(classified,1)
(following,1)
(in,1)
(object,1)
(cryptographic,3)
(which,2)
(See,1)
(encryption,3)
(Number,1)
(and/or,1)
(software,2)
(for,3)
((BIS),,1)
(makes,1)
(at:,2)
(manner,1)
(Core,1)
(latest,1)
(your,1)
(may,1)
(the,8)
(Exception,1)
(includes,2)
(restrictions,1)
(import,,2)
(project,1)
(you,1)
(use,,2)
(another,1)
(if,1)
(or,2)
(Commerce,,1)
(source,1)
(software.,2)
(laws,,1)
(BEFORE,1)
(Hadoop,,1)
(License,1)
(written,1)
(code,1)
(Regulations,,1)
(software,,2)
(more,2)
(software:,1)
(see,1)
(regulations,1)
(of,5)
(libraries,1)
(by,1)
(exception,1)
(Control,1)
(code.,1)
(eligible,1)
(both,1)
(to,2)
(Foundation,1)
(Government,1)
(functions,1)
(and,6)
(5D002.C.1,,1)
((TSU),1)
(Hadoop,1)
19/01/21 16:44:42 INFO SparkContext: Invoking stop() from shutdown hook
19/01/21 16:44:42 INFO SparkUI: Stopped Spark web UI at http://192.168.1.9:4040
19/01/21 16:44:42 INFO YarnClientSchedulerBackend: Interrupting monitor thread
19/01/21 16:44:42 INFO YarnClientSchedulerBackend: Shutting down all executors
19/01/21 16:44:42 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
19/01/21 16:44:42 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
19/01/21 16:44:42 INFO YarnClientSchedulerBackend: Stopped
19/01/21 16:44:42 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/01/21 16:44:42 INFO MemoryStore: MemoryStore cleared
19/01/21 16:44:42 INFO BlockManager: BlockManager stopped
19/01/21 16:44:42 INFO BlockManagerMaster: BlockManagerMaster stopped
19/01/21 16:44:42 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/01/21 16:44:42 INFO SparkContext: Successfully stopped SparkContext
19/01/21 16:44:42 INFO ShutdownHookManager: Shutdown hook called
19/01/21 16:44:42 INFO ShutdownHookManager: Deleting directory /tmp/spark-88c6c289-4d49-4035-96d7-19ba6410ef8a
