Flink1.8 集群搭建完全指南(5)：Flink on Yarn

在准备好Hadoop的环境后，可以开始搭建Flink了。

这里我们参考的是这篇教程，其中包括很全面的安装步骤：https://files.alicdn.com/tpsservice/4824447b829149c86bedd19424d05915.pdf

这里我们的服务器列表和之前Hadoop是相同的：
我们现在有3台服务器，服务器列表如下：

hostname	ip	作用
master	10.16..	JobManager, NodeManager
slave1	10.16..	NodeManager
slave2	10.16..	NodeManager

1. 下载并配置Flink

1.1 下载Flink

在Flink的官网下载Binary的安装包到master的/data目录下，并解压，下载Hadoop的扩展库：

$ wget http://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.8.0/flink-1.8.0-bin-scala_2.12.tgz
$ tar -xvf flink-1.8.0-bin-scala_2.12.tgz
$ cd flink-1.8.0/lib
$ wget https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar

1.2 修改Flink的配置文件

Flink的配置文件在目录：/data/flink-1.8.0/conf

修改masters：

master

修改slaves：

master
slave1
slave2

修改flink-conf.yaml：

jobmanager.rpc.address: master

1.3 拷贝Flink到其他机器

将master的flink目录拷贝到其他机器的相同目录：

scp -r /data/flink-1.8.0 root@slave1:/data
scp -r /data/flink-1.8.0 root@slave2:/data

2. 启动Flink

如果想通过Yarn实现资源管理，需要先在Yarn启动Flink，然后再执行Flink的程序。

2.1 启动Flink Yarn Session

$ ./bin/yarn-session.sh -n 4 -jm 1024m -tm 4096m
2019-07-01 22:43:04,154 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, ads-data-web-online012-bjdxt9p
2019-07-01 22:43:04,155 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2019-07-01 22:43:04,155 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
2019-07-01 22:43:04,155 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
2019-07-01 22:43:04,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-07-01 22:43:04,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2019-07-01 22:43:04,607 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-07-01 22:43:04,719 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to hdfs/master@HADOOP.COM (auth:KERBEROS)
2019-07-01 22:43:04,775 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at /master:8032
2019-07-01 22:43:04,913 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument n is deprecated in will be ignored.
2019-07-01 22:43:05,099 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2019-07-01 22:43:05,152 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=4096, numberTaskManagers=4, slotsPerTaskManager=1}
2019-07-01 22:43:05,595 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The configuration directory ('/data/flink-1.8.0/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-07-01 22:43:07,827 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Adding delegation token to the AM container..
2019-07-01 22:43:07,836 INFO  org.apache.hadoop.hdfs.DFSClient                              - Created HDFS_DELEGATION_TOKEN token 3 for hdfs on 10.16.195.254:9000
2019-07-01 22:43:07,848 INFO  org.apache.hadoop.mapreduce.security.TokenCache               - Got dt for hdfs://10.16.195.254:9000; Kind: HDFS_DELEGATION_TOKEN, Service: 10.16.195.254:9000, Ident: (HDFS_DELEGATION_TOKEN token 3 for hdfs)
2019-07-01 22:43:07,848 INFO  org.apache.flink.yarn.Utils                                   - Attempting to obtain Kerberos security token for HBase
2019-07-01 22:43:07,848 INFO  org.apache.flink.yarn.Utils                                   - HBase is not available (not packaged with this application): ClassNotFoundException : "org.apache.hadoop.hbase.HBaseConfiguration".
2019-07-01 22:43:07,857 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1562035367237_0001
2019-07-01 22:43:08,384 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1562035367237_0001
2019-07-01 22:43:08,384 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2019-07-01 22:43:08,386 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2019-07-01 22:43:14,953 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
2019-07-01 22:43:15,564 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest client endpoint started.
Flink JobManager is now running on master:1315 with leader id 00000000-0000-0000-0000-000000000000.
JobManager Web Interface: http://master:1315

启动后，可以打开JobManager的管理页面，地址在日志中。在yarn的管理页面中也可以看到一个运行中的应用：

yarn

2.2 执行示例程序

执行Flink的examples程序：

$ /bin/flink run examples/streaming/WordCount.jar --input hdfs:///user/hdfs/input_dir --output hdfs:///user/hdfs/output_dir
2019-07-01 22:46:45,084 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-root.
2019-07-01 22:46:45,084 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-root.
2019-07-01 22:46:45,603 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - YARN properties set default parallelism to 4
2019-07-01 22:46:45,603 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - YARN properties set default parallelism to 4
YARN properties set default parallelism to 4
2019-07-01 22:46:45,636 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at /10.16.195.254:8032
2019-07-01 22:46:45,774 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-07-01 22:46:45,774 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-07-01 22:46:45,778 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2019-07-01 22:46:45,960 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Found application JobManager host name 'host name' and port '1315' from supplied application id 'application_1562035367237_0001'
Starting execution of program
Program execution finished
Job with JobID f1a63f47e44521d457773c3e0acab80b has finished.
Job Runtime: 256 ms

以上，便完成了Flink在Yarn上的安装和部署，更多关于Flink安装的细节，可以参考文章开始的文档，其中的内容讲解很详细。