一、地址和端口问题
spark官网 说默认端口是 4040，使用spark.ui.port 进行配置spark配置。
我们使用的是CDH，使用 spark.ui.port 查看没有找到该配置
image.png
-
搜索
port，找到如下几个端口，看着也就18088比较像
image.png 使用如下命令访问
curl http://xxx.xxx.xxx.xxx:18088/api/v1/applications?limit=2
得到如下结果，看着挺像那么回事，仔细一看，怎么都是历史的任务？
[ {
"id" : "application_1618400751843_125338",
"name" : "cat3-title-vec",
"attempts" : [ {
"startTime" : "2021-05-13T13:33:24.965GMT",
"endTime" : "2021-05-13T13:34:41.551GMT",
"lastUpdated" : "2021-05-13T13:34:46.298GMT",
"duration" : 76586,
"sparkUser" : "supdev",
"completed" : true,
"appSparkVersion" : "2.4.0-cdh6.2.0",
"startTimeEpoch" : 1620912804965,
"lastUpdatedEpoch" : 1620912886298,
"endTimeEpoch" : 1620912881551
} ]
}, {
"id" : "application_1618400751843_125318",
"name" : "re-list-demo",
"attempts" : [ {
"startTime" : "2021-05-13T13:26:58.297GMT",
"endTime" : "2021-05-13T13:28:58.500GMT",
"lastUpdated" : "2021-05-13T13:28:58.570GMT",
"duration" : 120203,
"sparkUser" : "supdev",
"completed" : true,
"appSparkVersion" : "2.4.0-cdh6.2.0",
"startTimeEpoch" : 1620912418297,
"lastUpdatedEpoch" : 1620912538570,
"endTimeEpoch" : 1620912538500
} ]
} ]
- 加上状态
curl "http://sj-20-207-14:18088/api/v1/applications?limit=2&status=running"
还是历史
[ {
"id" : "application_1618400751843_99465",
"name" : "Hive on Spark (hiveSessionId = 4f556621-4e68-40d5-a416-8f1fc6a3ea9e)",
"attempts" : [ {
"attemptId" : "1",
"startTime" : "2021-05-08T09:50:00.273GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"lastUpdated" : "2021-05-08T10:36:31.654GMT",
"duration" : 0,
"sparkUser" : "jiahang.liu",
"completed" : false,
"appSparkVersion" : "2.4.0-cdh6.2.0",
"startTimeEpoch" : 1620467400273,
"lastUpdatedEpoch" : 1620470191654,
"endTimeEpoch" : -1
} ]
}, {
"id" : "application_1618400751843_93669",
"name" : "Hive on Spark (hiveSessionId = f31a0dec-abbe-4418-bf7f-98428a7cb44a)",
"attempts" : [ {
"attemptId" : "1",
"startTime" : "2021-05-07T13:30:08.725GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"lastUpdated" : "2021-05-07T14:58:18.949GMT",
"duration" : 0,
"sparkUser" : "supdev",
"completed" : false,
"appSparkVersion" : "2.4.0-cdh6.2.0",
"startTimeEpoch" : 1620394208725,
"lastUpdatedEpoch" : 1620399498949,
"endTimeEpoch" : -1
} ]
} ]
未完待续
