Spark（2.3.1）集群安装【一篇就够】

比眉伴天荒 2022-04-10 01:51 255阅读 0赞

[https://blog.csdn.net/sjmz30071360/article/details/82561867][https_blog.csdn.net_sjmz30071360_article_details_82561867]

**操作系统：Linux（CenOS7）**

**Spark集群至少需要有JDK、Hadoop、Scala、Spark**

**一、安装Spark依赖的Scala（每个节点都安装）**

**1.1 下载Scala（2.11.12）**

wget "https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz"

**1.2 解压、配置环境变量**

vi /etc/profile 编辑文件，追加如下内容：

\# scala environment  
export SCALA\_HOME=/usr/scala-2.11.12  
export PATH=$PATH:$SCALA\_HOME/bin

**1.3 验证Scala安装成功**

\[root@master usr\]\# source /etc/profile <== 使配置生效  
\[root@master usr\]\# scala -version  
Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL

**二、安装Spark（每个节点都安装）**

**2.1 下载Spark（2.3.1）**

wget "http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz"

**2.2 解压、配置环境变量**

vi /etc/profile 编辑文件，追加如下内容：

\# spark environment  
export SPARK\_HOME=/usr/spark-2.3.1-bin-hadoop2.7  
export PATH=$PATH:$SPARK\_HOME/bin <==因为$SPARK\_HOME/sbin目录下一些文件名和$HADOOP\_HOME/bin目 录下一些文件名相同，为了避免文件名冲突，这里不添加 $SPARK\_HOME/sbin目录

**2.3 配置conf目录下文件**

**2.3.1 新建spark-env.sh文件**

\[root@master conf\]\# cp spark-env.sh.template spark-env.sh

\[root@master conf\]\# vi spark-env.sh 编辑文件，增加如下内容：

JAVA\_HOME=/usr/java/jdk1.8.0\_162

HADOOP\_HOME=/usr/hadoop/hadoop-2.9.0  
HADOOP\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop

SCALA\_HOME=/usr/scala-2.11.12

SPARK\_HOME=/usr/spark-2.3.1-bin-hadoop2.7  
SPARK\_MASTER\_IP=master  
SPARK\_EXECUTOR\_MEMORY=1G

**2.3.2 新建slaves文件**

\[root@master conf\]\# cp slaves.template slaves  
\[root@master conf\]\# vi slaves 编辑文件，增加如下内容：

\#  
\# Licensed to the Apache Software Foundation (ASF) under one or more  
\# contributor license agreements. See the NOTICE file distributed with  
\# this work for additional information regarding copyright ownership.  
\# The ASF licenses this file to You under the Apache License, Version 2.0  
\# (the "License"); you may not use this file except in compliance with  
\# the License. You may obtain a copy of the License at  
\#  
\# http://www.apache.org/licenses/LICENSE-2.0  
\#  
\# Unless required by applicable law or agreed to in writing, software  
\# distributed under the License is distributed on an "AS IS" BASIS,  
\# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  
\# See the License for the specific language governing permissions and  
\# limitations under the License.  
\#

\# A Spark Worker will be started on each of the machines listed below.  
slave1  
slave2

**三、启动Spark集群**

因为Spark是依赖于Hadoop提供的分布式文件系统的，所以在启动Spark之前，先确保Hadoop在正常运行。（[Hadoop安装与启动][Hadoop]）

在Hadoop正常运行的情况下，在master（Hadoop的NameNode，即Spark的master节点）上，

/usr/spark-2.3.1-bin-hadoop2.7/sbin目录下执行：

\[root@master sbin\]\# ./start-all.sh <==前面必须有 **./** ，否则会执行$HADOOP\_HOME/bin下的start-all.sh脚本  
starting org.apache.spark.deploy.master.Master, logging to /usr/spark-2.3.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out  
slave1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/spark-2.3.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.out  
slave2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/spark-2.3.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.out

**访问Spark集群URL：**

![70][]

**四、运行Spark提供的计算圆周率的示例程序**

**4.1 使用单机local模式提交任务**

local模式也就是本地模式，即在本地机器上单机执行程序。使用这个模式，并不需要启动Hadoop集群，也不需要启动Spark集群，只要有一台机器上安装了JDK、Scala、Spark即可运行。

\[root@master sbin\]\# spark-submit --class org.apache.spark.examples.SparkPi --master local /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples\_2.11-2.3.1.jar  
2018-09-09 21:21:21 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
2018-09-09 21:21:22 INFO SparkContext:54 - Running Spark version 2.3.1  
2018-09-09 21:21:22 INFO SparkContext:54 - Submitted application: Spark Pi  
2018-09-09 21:21:22 INFO SecurityManager:54 - Changing view acls to: root  
2018-09-09 21:21:22 INFO SecurityManager:54 - Changing modify acls to: root  
2018-09-09 21:21:22 INFO SecurityManager:54 - Changing view acls groups to:  
2018-09-09 21:21:22 INFO SecurityManager:54 - Changing modify acls groups to:  
2018-09-09 21:21:22 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()  
2018-09-09 21:21:23 INFO Utils:54 - Successfully started service 'sparkDriver' on port 44606.  
2018-09-09 21:21:23 INFO SparkEnv:54 - Registering MapOutputTracker  
2018-09-09 21:21:23 INFO SparkEnv:54 - Registering BlockManagerMaster  
2018-09-09 21:21:23 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information  
2018-09-09 21:21:23 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up  
2018-09-09 21:21:23 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-d76c7c4a-0859-4c11-aba0-7d8e88efdec6  
2018-09-09 21:21:24 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB  
2018-09-09 21:21:24 INFO SparkEnv:54 - Registering OutputCommitCoordinator  
2018-09-09 21:21:24 INFO log:192 - Logging initialized @6994ms  
2018-09-09 21:21:24 INFO Server:346 - jetty-9.3.z-SNAPSHOT  
2018-09-09 21:21:24 INFO Server:414 - Started @7429ms  
2018-09-09 21:21:25 INFO AbstractConnector:278 - Started ServerConnector@e3cee7b\{HTTP/1.1,\[http/1.1\]\}\{0.0.0.0:4040\}  
2018-09-09 21:21:25 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6ab72419\{/jobs,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51684e4a\{/jobs/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6ce1f601\{/jobs/job,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e886a5b\{/jobs/job/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@d816dde\{/stages,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e33c391\{/stages/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6c451c9c\{/stages/stage,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@47747fb9\{/stages/stage/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3113a37\{/stages/pool,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@213e3629\{/stages/pool/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4e9658b5\{/storage,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a7b6f69\{/storage/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@20312893\{/storage/rdd,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@70eecdc2\{/storage/rdd/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c41709a\{/environment,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7db0565c\{/environment/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54ec8cc9\{/executors,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@52eacb4b\{/executors/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5528a42c\{/executors/threadDump,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a551a63\{/executors/threadDump/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a6f5124\{/static,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77b21474\{/,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@52d10fb8\{/api,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22680f52\{/jobs/job/kill,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@60d84f61\{/stages/stage/kill,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:25 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040  
2018-09-09 21:21:25 INFO SparkContext:54 - Added JAR file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples\_2.11-2.3.1.jar at spark://master:44606/jars/spark-examples\_2.11-2.3.1.jar with timestamp 1536499285929  
2018-09-09 21:21:26 INFO Executor:54 - Starting executor ID driver on host localhost  
2018-09-09 21:21:26 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46568.  
2018-09-09 21:21:26 INFO NettyBlockTransferService:54 - Server created on master:46568  
2018-09-09 21:21:26 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy  
2018-09-09 21:21:26 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 46568, None)  
2018-09-09 21:21:27 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:46568 with 413.9 MB RAM, BlockManagerId(driver, master, 46568, None)  
2018-09-09 21:21:27 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 46568, None)  
2018-09-09 21:21:27 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 46568, None)  
2018-09-09 21:21:27 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@18a645fd\{/metrics/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:21:30 INFO SparkContext:54 - Starting job: reduce at SparkPi.scala:38  
2018-09-09 21:21:30 INFO DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions  
2018-09-09 21:21:30 INFO DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)  
2018-09-09 21:21:30 INFO DAGScheduler:54 - Parents of final stage: List()  
2018-09-09 21:21:30 INFO DAGScheduler:54 - Missing parents: List()  
2018-09-09 21:21:30 INFO DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD\[1\] at map at SparkPi.scala:34), which has no missing parents  
2018-09-09 21:21:31 INFO MemoryStore:54 - Block broadcast\_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)  
2018-09-09 21:21:32 INFO MemoryStore:54 - Block broadcast\_0\_piece0 stored as bytes in memory (estimated size 1181.0 B, free 413.9 MB)  
2018-09-09 21:21:32 INFO BlockManagerInfo:54 - Added broadcast\_0\_piece0 in memory on master:46568 (size: 1181.0 B, free: 413.9 MB)  
2018-09-09 21:21:32 INFO SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039  
2018-09-09 21:21:32 INFO DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD\[1\] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))  
2018-09-09 21:21:32 INFO TaskSchedulerImpl:54 - Adding task set 0.0 with 2 tasks  
2018-09-09 21:21:32 INFO TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS\_LOCAL, 7853 bytes)  
2018-09-09 21:21:32 INFO Executor:54 - Running task 0.0 in stage 0.0 (TID 0)  
2018-09-09 21:21:32 INFO Executor:54 - Fetching spark://master:44606/jars/spark-examples\_2.11-2.3.1.jar with timestamp 1536499285929  
2018-09-09 21:21:33 INFO TransportClientFactory:267 - Successfully created connection to master/192.168.11.128:44606 after 146 ms (0 ms spent in bootstraps)  
2018-09-09 21:21:33 INFO Utils:54 - Fetching spark://master:44606/jars/spark-examples\_2.11-2.3.1.jar to /tmp/spark-3fae86fe-d3da-4b87-b0e1-c94efab840c5/userFiles-5292d290-2b31-4dc6-bfd6-c85e4cdaaae6/fetchFileTemp6719743492164655300.tmp  
2018-09-09 21:21:34 INFO Executor:54 - Adding file:/tmp/spark-3fae86fe-d3da-4b87-b0e1-c94efab840c5/userFiles-5292d290-2b31-4dc6-bfd6-c85e4cdaaae6/spark-examples\_2.11-2.3.1.jar to class loader  
2018-09-09 21:21:34 INFO Executor:54 - Finished task 0.0 in stage 0.0 (TID 0). 867 bytes result sent to driver  
2018-09-09 21:21:34 INFO TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS\_LOCAL, 7853 bytes)  
2018-09-09 21:21:34 INFO Executor:54 - Running task 1.0 in stage 0.0 (TID 1)  
2018-09-09 21:21:34 INFO TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 2050 ms on localhost (executor driver) (1/2)  
2018-09-09 21:21:34 INFO Executor:54 - Finished task 1.0 in stage 0.0 (TID 1). 867 bytes result sent to driver  
2018-09-09 21:21:34 INFO TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 318 ms on localhost (executor driver) (2/2)  
2018-09-09 21:21:34 INFO TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool  
2018-09-09 21:21:34 INFO DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 3.901 s  
2018-09-09 21:21:34 INFO DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 4.356085 s  
Pi is roughly 3.146475732378662  
2018-09-09 21:21:35 INFO AbstractConnector:318 - Stopped Spark@e3cee7b\{HTTP/1.1,\[http/1.1\]\}\{0.0.0.0:4040\}  
2018-09-09 21:21:35 INFO SparkUI:54 - Stopped Spark web UI at http://master:4040  
2018-09-09 21:21:35 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!  
2018-09-09 21:21:35 INFO MemoryStore:54 - MemoryStore cleared  
2018-09-09 21:21:35 INFO BlockManager:54 - BlockManager stopped  
2018-09-09 21:21:35 INFO BlockManagerMaster:54 - BlockManagerMaster stopped  
2018-09-09 21:21:35 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!  
2018-09-09 21:21:35 INFO SparkContext:54 - Successfully stopped SparkContext  
2018-09-09 21:21:35 INFO ShutdownHookManager:54 - Shutdown hook called  
2018-09-09 21:21:35 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-3fae86fe-d3da-4b87-b0e1-c94efab840c5  
2018-09-09 21:21:35 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-ba114487-f80c-4684-a962-870139ca248d  
\[root@master sbin\]\#

**4.2 使用独立的Spark集群模式提交任务**

这种模式也就是Standalone模式，使用独立的Spark集群模式提交任务，需要先启动Spark集群，但是不需要启动Hadoop集群。  
启动Spark集群的方法是进入$SPARK\_HOME/sbin目录下，执行start-all.sh脚本，启动成功后，可以访问下面的地址看是否成功：http://master:8080/

\[root@master sbin\]\# spark-submit --class org.apache.spark.examples.SparkPi --master spark://master:7077 /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples\_2.11-2.3.1.jar  
2018-09-09 21:33:01 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
2018-09-09 21:33:02 INFO SparkContext:54 - Running Spark version 2.3.1  
2018-09-09 21:33:02 INFO SparkContext:54 - Submitted application: Spark Pi  
2018-09-09 21:33:02 INFO SecurityManager:54 - Changing view acls to: root  
2018-09-09 21:33:02 INFO SecurityManager:54 - Changing modify acls to: root  
2018-09-09 21:33:02 INFO SecurityManager:54 - Changing view acls groups to:  
2018-09-09 21:33:02 INFO SecurityManager:54 - Changing modify acls groups to:  
2018-09-09 21:33:02 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()  
2018-09-09 21:33:03 INFO Utils:54 - Successfully started service 'sparkDriver' on port 44732.  
2018-09-09 21:33:03 INFO SparkEnv:54 - Registering MapOutputTracker  
2018-09-09 21:33:03 INFO SparkEnv:54 - Registering BlockManagerMaster  
2018-09-09 21:33:03 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information  
2018-09-09 21:33:03 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up  
2018-09-09 21:33:03 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-8089bd51-572f-4c01-93a8-b0a9d5d78014  
2018-09-09 21:33:03 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB  
2018-09-09 21:33:03 INFO SparkEnv:54 - Registering OutputCommitCoordinator  
2018-09-09 21:33:04 INFO log:192 - Logging initialized @6786ms  
2018-09-09 21:33:04 INFO Server:346 - jetty-9.3.z-SNAPSHOT  
2018-09-09 21:33:04 INFO Server:414 - Started @7198ms  
2018-09-09 21:33:04 INFO AbstractConnector:278 - Started ServerConnector@6b9267b\{HTTP/1.1,\[http/1.1\]\}\{0.0.0.0:4040\}  
2018-09-09 21:33:04 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4fdfa676\{/jobs,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@38875e7d\{/jobs/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e886a5b\{/jobs/job,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e33c391\{/jobs/job/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6c451c9c\{/stages,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31c269fd\{/stages/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@372b0d86\{/stages/stage,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@213e3629\{/stages/stage/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4e9658b5\{/stages/pool,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a7b6f69\{/stages/pool/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@20312893\{/storage,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@70eecdc2\{/storage/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c41709a\{/storage/rdd,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7db0565c\{/storage/rdd/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54ec8cc9\{/environment,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@52eacb4b\{/environment/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5528a42c\{/executors,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a551a63\{/executors/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a6f5124\{/executors/threadDump,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:04 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1edb61b1\{/executors/threadDump/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@ec2bf82\{/static,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41c07648\{/,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1fe8d51b\{/api,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@39c11e6c\{/jobs/job/kill,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@324dcd31\{/stages/stage/kill,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:05 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040  
2018-09-09 21:33:05 INFO SparkContext:54 - Added JAR file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples\_2.11-2.3.1.jar at spark://master:44732/jars/spark-examples\_2.11-2.3.1.jar with timestamp 1536499985512  
2018-09-09 21:33:06 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://master:7077...  
2018-09-09 21:33:06 INFO TransportClientFactory:267 - Successfully created connection to master/192.168.11.128:7077 after 139 ms (0 ms spent in bootstraps)  
2018-09-09 21:33:07 INFO StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180909213306-0000  
2018-09-09 21:33:07 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40542.  
2018-09-09 21:33:07 INFO NettyBlockTransferService:54 - Server created on master:40542  
2018-09-09 21:33:07 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy  
2018-09-09 21:33:07 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180909213306-0000/0 on worker-20180909210806-192.168.11.129-41138 (192.168.11.129:41138) with 1 core(s)  
2018-09-09 21:33:07 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180909213306-0000/0 on hostPort 192.168.11.129:41138 with 1 core(s), 1024.0 MB RAM  
2018-09-09 21:33:07 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180909213306-0000/1 on worker-20180909210808-192.168.11.130-41969 (192.168.11.130:41969) with 1 core(s)  
2018-09-09 21:33:07 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180909213306-0000/1 on hostPort 192.168.11.130:41969 with 1 core(s), 1024.0 MB RAM  
2018-09-09 21:33:07 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180909213306-0000/0 is now RUNNING  
2018-09-09 21:33:08 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180909213306-0000/1 is now RUNNING  
2018-09-09 21:33:08 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 40542, None)  
2018-09-09 21:33:08 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:40542 with 413.9 MB RAM, BlockManagerId(driver, master, 40542, None)  
2018-09-09 21:33:08 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 40542, None)  
2018-09-09 21:33:08 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 40542, None)  
2018-09-09 21:33:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@56ccd751\{/metrics/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:33:10 INFO StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0  
2018-09-09 21:33:13 INFO SparkContext:54 - Starting job: reduce at SparkPi.scala:38  
2018-09-09 21:33:13 INFO DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions  
2018-09-09 21:33:13 INFO DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)  
2018-09-09 21:33:13 INFO DAGScheduler:54 - Parents of final stage: List()  
2018-09-09 21:33:13 INFO DAGScheduler:54 - Missing parents: List()  
2018-09-09 21:33:14 INFO DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD\[1\] at map at SparkPi.scala:34), which has no missing parents  
2018-09-09 21:33:15 INFO MemoryStore:54 - Block broadcast\_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)  
2018-09-09 21:33:16 INFO MemoryStore:54 - Block broadcast\_0\_piece0 stored as bytes in memory (estimated size 1181.0 B, free 413.9 MB)  
2018-09-09 21:33:16 INFO BlockManagerInfo:54 - Added broadcast\_0\_piece0 in memory on master:40542 (size: 1181.0 B, free: 413.9 MB)  
2018-09-09 21:33:16 INFO SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039  
2018-09-09 21:33:16 INFO DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD\[1\] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))  
2018-09-09 21:33:16 INFO TaskSchedulerImpl:54 - Adding task set 0.0 with 2 tasks  
2018-09-09 21:33:19 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.11.129:49616) with ID 0  
2018-09-09 21:33:19 INFO TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, 192.168.11.129, executor 0, partition 0, PROCESS\_LOCAL, 7857 bytes)  
2018-09-09 21:33:20 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.11.130:40952) with ID 1  
2018-09-09 21:33:20 INFO TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, 192.168.11.130, executor 1, partition 1, PROCESS\_LOCAL, 7857 bytes)  
2018-09-09 21:33:20 INFO BlockManagerMasterEndpoint:54 - Registering block manager 192.168.11.129:34978 with 413.9 MB RAM, BlockManagerId(0, 192.168.11.129, 34978, None)  
2018-09-09 21:33:20 INFO BlockManagerMasterEndpoint:54 - Registering block manager 192.168.11.130:44746 with 413.9 MB RAM, BlockManagerId(1, 192.168.11.130, 44746, None)  
2018-09-09 21:33:23 INFO BlockManagerInfo:54 - Added broadcast\_0\_piece0 in memory on 192.168.11.129:34978 (size: 1181.0 B, free: 413.9 MB)  
2018-09-09 21:33:23 INFO BlockManagerInfo:54 - Added broadcast\_0\_piece0 in memory on 192.168.11.130:44746 (size: 1181.0 B, free: 413.9 MB)  
2018-09-09 21:33:24 INFO TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 4487 ms on 192.168.11.129 (executor 0) (1/2)  
2018-09-09 21:33:25 INFO TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 5012 ms on 192.168.11.130 (executor 1) (2/2)  
2018-09-09 21:33:25 INFO TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool  
2018-09-09 21:33:25 INFO DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 10.560 s  
2018-09-09 21:33:25 INFO DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 11.549968 s  
Pi is roughly 3.1398956994784974  
2018-09-09 21:33:25 INFO AbstractConnector:318 - Stopped Spark@6b9267b\{HTTP/1.1,\[http/1.1\]\}\{0.0.0.0:4040\}  
2018-09-09 21:33:25 INFO SparkUI:54 - Stopped Spark web UI at http://master:4040  
2018-09-09 21:33:25 INFO StandaloneSchedulerBackend:54 - Shutting down all executors  
2018-09-09 21:33:25 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asking each executor to shut down  
2018-09-09 21:33:25 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!  
2018-09-09 21:33:25 INFO MemoryStore:54 - MemoryStore cleared  
2018-09-09 21:33:25 INFO BlockManager:54 - BlockManager stopped  
2018-09-09 21:33:25 INFO BlockManagerMaster:54 - BlockManagerMaster stopped  
2018-09-09 21:33:25 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!  
2018-09-09 21:33:25 INFO SparkContext:54 - Successfully stopped SparkContext  
2018-09-09 21:33:25 INFO ShutdownHookManager:54 - Shutdown hook called  
2018-09-09 21:33:25 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-ca417e29-82e1-4084-b518-46fde5110317  
2018-09-09 21:33:25 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-bc263b74-64a0-426c-840c-74c6084d131a  
\[root@master sbin\]\#  
![70 1][]

**4.3 用yarn-client模式执行计算程序**

这种模式也叫On-Yarn模式，主要包括yarn-Client和yarn-Cluster两种模式。在这种模式下提交任务，需要先启动Hadoop集群，然后再启动Spark集群。

\[root@master sbin\]\# spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples\_2.11-2.3.1.jar  
2018-09-09 21:51:06 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
2018-09-09 21:51:06 INFO SparkContext:54 - Running Spark version 2.3.1  
2018-09-09 21:51:06 INFO SparkContext:54 - Submitted application: Spark Pi  
2018-09-09 21:51:07 INFO SecurityManager:54 - Changing view acls to: root  
2018-09-09 21:51:07 INFO SecurityManager:54 - Changing modify acls to: root  
2018-09-09 21:51:07 INFO SecurityManager:54 - Changing view acls groups to:  
2018-09-09 21:51:07 INFO SecurityManager:54 - Changing modify acls groups to:  
2018-09-09 21:51:07 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()  
2018-09-09 21:51:07 INFO Utils:54 - Successfully started service 'sparkDriver' on port 37707.  
2018-09-09 21:51:07 INFO SparkEnv:54 - Registering MapOutputTracker  
2018-09-09 21:51:08 INFO SparkEnv:54 - Registering BlockManagerMaster  
2018-09-09 21:51:08 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information  
2018-09-09 21:51:08 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up  
2018-09-09 21:51:08 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-f4bcc309-8d7a-4238-bcfa-5587921eec3b  
2018-09-09 21:51:08 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB  
2018-09-09 21:51:08 INFO SparkEnv:54 - Registering OutputCommitCoordinator  
2018-09-09 21:51:09 INFO log:192 - Logging initialized @7824ms  
2018-09-09 21:51:09 INFO Server:346 - jetty-9.3.z-SNAPSHOT  
2018-09-09 21:51:09 INFO Server:414 - Started @8272ms  
2018-09-09 21:51:09 INFO AbstractConnector:278 - Started ServerConnector@2c444798\{HTTP/1.1,\[http/1.1\]\}\{0.0.0.0:4040\}  
2018-09-09 21:51:09 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e392345\{/jobs,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1bdaa23d\{/jobs/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@79f227a9\{/jobs/job,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@50d68830\{/jobs/job/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e53135d\{/stages,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7674a051\{/stages/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a7704c\{/stages/stage,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@323e8306\{/stages/stage/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@a23a01d\{/stages/pool,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4acf72b6\{/stages/pool/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7561db12\{/storage,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3301500b\{/storage/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@24b52d3e\{/storage/rdd,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@15deb1dc\{/storage/rdd/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e9c413e\{/environment,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@57a4d5ee\{/environment/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5af5def9\{/executors,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a45c42a\{/executors/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@36dce7ed\{/executors/threadDump,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@47a64f7d\{/executors/threadDump/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@33d05366\{/static,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54a3ab8f\{/,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1968a49c\{/api,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:09 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c20be82\{/jobs/job/kill,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:10 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@13c612bd\{/stages/stage/kill,null,AVAILABLE,@Spark\}  
2018-09-09 21:51:10 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040  
2018-09-09 21:51:10 INFO SparkContext:54 - Added JAR file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples\_2.11-2.3.1.jar at spark://master:37707/jars/spark-examples\_2.11-2.3.1.jar with timestamp 1536501070145  
2018-09-09 21:51:13 INFO RMProxy:98 - Connecting to ResourceManager at master/192.168.11.128:8032  
2018-09-09 21:51:14 INFO Client:54 - Requesting a new application from cluster with 2 NodeManagers  
2018-09-09 21:51:14 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)  
2018-09-09 21:51:14 INFO Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead  
2018-09-09 21:51:14 INFO Client:54 - Setting up container launch context for our AM  
2018-09-09 21:51:15 INFO Client:54 - Setting up the launch environment for our AM container  
2018-09-09 21:51:15 INFO Client:54 - Preparing resources for our AM container  
2018-09-09 21:51:20 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK\_HOME.  
2018-09-09 21:51:52 INFO Client:54 - Uploading resource file:/tmp/spark-4adf63c4-82a7-4cf7-bf6f-0abf78282f74/\_\_spark\_libs\_\_7942925809039923414.zip -> hdfs://master:9000/user/root/.sparkStaging/application\_1536498065211\_0002/\_\_spark\_libs\_\_7942925809039923414.zip  
2018-09-09 21:52:39 INFO Client:54 - Uploading resource file:/tmp/spark-4adf63c4-82a7-4cf7-bf6f-0abf78282f74/\_\_spark\_conf\_\_6224445360548244890.zip -> hdfs://master:9000/user/root/.sparkStaging/application\_1536498065211\_0002/\_\_spark\_conf\_\_.zip  
2018-09-09 21:52:40 INFO SecurityManager:54 - Changing view acls to: root  
2018-09-09 21:52:40 INFO SecurityManager:54 - Changing modify acls to: root  
2018-09-09 21:52:40 INFO SecurityManager:54 - Changing view acls groups to:  
2018-09-09 21:52:40 INFO SecurityManager:54 - Changing modify acls groups to:  
2018-09-09 21:52:40 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()  
2018-09-09 21:52:40 INFO Client:54 - Submitting application application\_1536498065211\_0002 to ResourceManager  
2018-09-09 21:52:40 INFO YarnClientImpl:273 - Submitted application application\_1536498065211\_0002  
2018-09-09 21:52:40 INFO SchedulerExtensionServices:54 - Starting Yarn extension services with app application\_1536498065211\_0002 and attemptId None  
2018-09-09 21:52:41 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:41 INFO Client:54 -  
client token: N/A  
diagnostics: AM container is launched, waiting for AM container to Register with RM  
ApplicationMaster host: N/A  
ApplicationMaster RPC port: -1  
queue: default  
start time: 1536501160345  
final status: UNDEFINED  
tracking URL: http://master:8088/proxy/application\_1536498065211\_0002/  
user: root  
2018-09-09 21:52:42 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:43 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:44 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:45 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:46 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:47 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:48 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:49 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:50 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:51 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:52 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:53 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:54 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:55 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:56 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:57 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:58 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:52:59 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:00 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:01 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:02 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:03 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:04 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:05 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:06 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:07 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:08 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:09 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:10 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:11 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:12 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:13 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:14 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:15 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:16 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:17 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:18 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:19 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:20 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:21 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:22 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:23 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:24 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:25 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:26 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:27 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:28 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:29 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:30 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:31 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:32 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:33 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:34 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:35 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:36 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:37 INFO YarnClientSchedulerBackend:54 - Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY\_HOSTS -> master, PROXY\_URI\_BASES -> http://master:8088/proxy/application\_1536498065211\_0002), /proxy/application\_1536498065211\_0002  
2018-09-09 21:53:37 INFO JettyUtils:54 - Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter  
2018-09-09 21:53:37 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:38 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: ACCEPTED)  
2018-09-09 21:53:39 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:54 - ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)  
2018-09-09 21:53:39 INFO Client:54 - Application report for application\_1536498065211\_0002 (state: RUNNING)  
2018-09-09 21:53:39 INFO Client:54 -  
client token: N/A  
diagnostics: N/A  
ApplicationMaster host: 192.168.11.129  
ApplicationMaster RPC port: 0  
queue: default  
start time: 1536501160345  
final status: UNDEFINED  
tracking URL: http://master:8088/proxy/application\_1536498065211\_0002/  
user: root  
2018-09-09 21:53:39 INFO YarnClientSchedulerBackend:54 - Application application\_1536498065211\_0002 has started running.  
2018-09-09 21:53:39 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39617.  
2018-09-09 21:53:39 INFO NettyBlockTransferService:54 - Server created on master:39617  
2018-09-09 21:53:39 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy  
2018-09-09 21:53:40 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 39617, None)  
2018-09-09 21:53:40 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:39617 with 413.9 MB RAM, BlockManagerId(driver, master, 39617, None)  
2018-09-09 21:53:40 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 39617, None)  
2018-09-09 21:53:40 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 39617, None)  
2018-09-09 21:53:40 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2c2c3947\{/metrics/json,null,AVAILABLE,@Spark\}  
2018-09-09 21:53:41 INFO YarnClientSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)  
2018-09-09 21:54:01 INFO SparkContext:54 - Starting job: reduce at SparkPi.scala:38  
2018-09-09 21:54:01 INFO DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions  
2018-09-09 21:54:01 INFO DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)  
2018-09-09 21:54:01 INFO DAGScheduler:54 - Parents of final stage: List()  
2018-09-09 21:54:01 INFO DAGScheduler:54 - Missing parents: List()  
2018-09-09 21:54:01 INFO DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD\[1\] at map at SparkPi.scala:34), which has no missing parents  
2018-09-09 21:54:14 INFO MemoryStore:54 - Block broadcast\_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)  
2018-09-09 21:54:14 INFO MemoryStore:54 - Block broadcast\_0\_piece0 stored as bytes in memory (estimated size 1181.0 B, free 413.9 MB)  
2018-09-09 21:54:14 INFO BlockManagerInfo:54 - Added broadcast\_0\_piece0 in memory on master:39617 (size: 1181.0 B, free: 413.9 MB)  
2018-09-09 21:54:14 INFO SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039  
2018-09-09 21:54:14 INFO DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD\[1\] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))  
2018-09-09 21:54:14 INFO YarnScheduler:54 - Adding task set 0.0 with 2 tasks  
2018-09-09 21:54:32 WARN YarnScheduler:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources  
2018-09-09 21:54:44 WARN YarnScheduler:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources  
2018-09-09 21:54:46 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.11.129:59958) with ID 2  
2018-09-09 21:54:46 INFO TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, slave1, executor 2, partition 0, PROCESS\_LOCAL, 7864 bytes)  
2018-09-09 21:54:46 INFO BlockManagerMasterEndpoint:54 - Registering block manager slave1:39770 with 413.9 MB RAM, BlockManagerId(2, slave1, 39770, None)  
2018-09-09 21:54:48 INFO BlockManagerInfo:54 - Added broadcast\_0\_piece0 in memory on slave1:39770 (size: 1181.0 B, free: 413.9 MB)  
2018-09-09 21:54:50 INFO TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, slave1, executor 2, partition 1, PROCESS\_LOCAL, 7864 bytes)  
2018-09-09 21:54:50 INFO TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 3931 ms on slave1 (executor 2) (1/2)  
2018-09-09 21:54:50 INFO TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 338 ms on slave1 (executor 2) (2/2)  
2018-09-09 21:54:50 INFO YarnScheduler:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool  
2018-09-09 21:54:50 INFO DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 48.440 s  
2018-09-09 21:54:50 INFO DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 48.819130 s  
Pi is roughly 3.144115720578603  
2018-09-09 21:54:50 INFO AbstractConnector:318 - Stopped Spark@2c444798\{HTTP/1.1,\[http/1.1\]\}\{0.0.0.0:4040\}  
2018-09-09 21:54:50 INFO SparkUI:54 - Stopped Spark web UI at http://master:4040  
2018-09-09 21:54:50 INFO YarnClientSchedulerBackend:54 - Interrupting monitor thread  
2018-09-09 21:54:50 INFO YarnClientSchedulerBackend:54 - Shutting down all executors  
2018-09-09 21:54:50 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Asking each executor to shut down  
2018-09-09 21:54:51 INFO SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices  
(serviceOption=None,  
services=List(),  
started=false)  
2018-09-09 21:54:51 INFO YarnClientSchedulerBackend:54 - Stopped  
2018-09-09 21:54:51 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!  
2018-09-09 21:54:51 INFO MemoryStore:54 - MemoryStore cleared  
2018-09-09 21:54:51 INFO BlockManager:54 - BlockManager stopped  
2018-09-09 21:54:51 INFO BlockManagerMaster:54 - BlockManagerMaster stopped  
2018-09-09 21:54:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!  
2018-09-09 21:54:51 INFO SparkContext:54 - Successfully stopped SparkContext  
2018-09-09 21:54:51 INFO ShutdownHookManager:54 - Shutdown hook called  
2018-09-09 21:54:51 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-4adf63c4-82a7-4cf7-bf6f-0abf78282f74  
2018-09-09 21:54:51 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-f085e710-596c-4c23-b436-c879d61d4a01  
\[root@master sbin\]\#  
![70 2][]

**4.4 用yarn-cluster模式执行计算程序**

\[root@master sbin\]\# spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples\_2.11-2.3.1.jar  
2018-09-09 22:00:59 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
2018-09-09 22:01:03 INFO RMProxy:98 - Connecting to ResourceManager at master/192.168.11.128:8032  
2018-09-09 22:01:04 INFO Client:54 - Requesting a new application from cluster with 2 NodeManagers  
2018-09-09 22:01:04 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)  
2018-09-09 22:01:04 INFO Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead  
2018-09-09 22:01:04 INFO Client:54 - Setting up container launch context for our AM  
2018-09-09 22:01:04 INFO Client:54 - Setting up the launch environment for our AM container  
2018-09-09 22:01:04 INFO Client:54 - Preparing resources for our AM container  
2018-09-09 22:01:09 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK\_HOME.  
2018-09-09 22:01:21 INFO Client:54 - Uploading resource file:/tmp/spark-351d275a-b939-41da-ba44-35f2eba3ba25/\_\_spark\_libs\_\_624052493228396690.zip -> hdfs://master:9000/user/root/.sparkStaging/application\_1536498065211\_0003/\_\_spark\_libs\_\_624052493228396690.zip  
2018-09-09 22:02:25 INFO Client:54 - Uploading resource file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples\_2.11-2.3.1.jar -> hdfs://master:9000/user/root/.sparkStaging/application\_1536498065211\_0003/spark-examples\_2.11-2.3.1.jar  
2018-09-09 22:02:40 INFO Client:54 - Uploading resource file:/tmp/spark-351d275a-b939-41da-ba44-35f2eba3ba25/\_\_spark\_conf\_\_5330471617982538290.zip -> hdfs://master:9000/user/root/.sparkStaging/application\_1536498065211\_0003/\_\_spark\_conf\_\_.zip  
2018-09-09 22:02:40 INFO SecurityManager:54 - Changing view acls to: root  
2018-09-09 22:02:40 INFO SecurityManager:54 - Changing modify acls to: root  
2018-09-09 22:02:40 INFO SecurityManager:54 - Changing view acls groups to:  
2018-09-09 22:02:40 INFO SecurityManager:54 - Changing modify acls groups to:  
2018-09-09 22:02:40 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()  
2018-09-09 22:02:40 INFO Client:54 - Submitting application application\_1536498065211\_0003 to ResourceManager  
2018-09-09 22:02:40 INFO YarnClientImpl:273 - Submitted application application\_1536498065211\_0003  
2018-09-09 22:02:41 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:41 INFO Client:54 -  
client token: N/A  
diagnostics: AM container is launched, waiting for AM container to Register with RM  
ApplicationMaster host: N/A  
ApplicationMaster RPC port: -1  
queue: default  
start time: 1536501760869  
final status: UNDEFINED  
tracking URL: http://master:8088/proxy/application\_1536498065211\_0003/  
user: root  
2018-09-09 22:02:42 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:43 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:44 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:46 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:47 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:48 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:49 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:50 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:51 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:52 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:53 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:54 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:55 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:56 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:57 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:58 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:02:59 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:00 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:01 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:02 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:03 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:04 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:05 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:06 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:07 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:08 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:09 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:10 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:11 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:12 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:13 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:14 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:15 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:16 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:17 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:18 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:19 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:20 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:21 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:22 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:23 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:24 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:25 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:26 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:27 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:28 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:29 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:30 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:31 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:32 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:33 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:34 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:35 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:36 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:37 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:38 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:39 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:40 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:41 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:42 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:43 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:44 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:45 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:46 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:47 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:48 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:49 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:50 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:51 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:52 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:53 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:54 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:55 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:56 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:57 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:58 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:03:59 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:00 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:01 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:02 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:03 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:04 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:05 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:06 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:07 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:08 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:09 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: ACCEPTED)  
2018-09-09 22:04:10 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:10 INFO Client:54 -  
client token: N/A  
diagnostics: N/A  
ApplicationMaster host: 192.168.11.130  
ApplicationMaster RPC port: 0  
queue: default  
start time: 1536501760869  
final status: UNDEFINED  
tracking URL: http://master:8088/proxy/application\_1536498065211\_0003/  
user: root  
2018-09-09 22:04:11 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:12 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:13 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:14 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:15 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:16 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:17 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:18 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:19 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:20 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:21 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:22 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:23 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:24 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:25 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:26 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:27 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:28 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:29 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:30 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:31 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:32 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:33 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:34 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:35 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:36 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:37 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:38 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:39 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:40 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:41 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:42 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:43 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:46 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:47 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:48 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:49 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:50 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:51 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:53 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:54 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:55 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:56 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:57 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:58 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:04:59 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:00 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:01 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:02 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:03 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:04 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:05 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:06 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:07 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:08 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:10 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:11 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:12 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:13 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:14 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:15 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:16 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:17 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:18 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:19 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:20 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:21 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:22 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:23 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: RUNNING)  
2018-09-09 22:05:24 INFO Client:54 - Application report for application\_1536498065211\_0003 (state: FINISHED)  
2018-09-09 22:05:24 INFO Client:54 -  
client token: N/A  
diagnostics: N/A  
ApplicationMaster host: 192.168.11.130  
ApplicationMaster RPC port: 0  
queue: default  
start time: 1536501760869  
final status: SUCCEEDED  
tracking URL: http://master:8088/proxy/application\_1536498065211\_0003/  
user: root  
2018-09-09 22:05:24 INFO ShutdownHookManager:54 - Shutdown hook called  
2018-09-09 22:05:24 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-0f174e46-416d-45dd-bb59-51c652b4a70d  
2018-09-09 22:05:24 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-351d275a-b939-41da-ba44-35f2eba3ba25  
\[root@master sbin\]\#  
使用yarn-cluster模式计算，结果没有输出在控制台，结果写在了Hadoop集群的日志中

![70 3][]

**点击红框application id**

![70 4][]

![70 5][]

**点击Logs**

![70 6][]

**点击stdout**

![70 7][]

[https_blog.csdn.net_sjmz30071360_article_details_82561867]: https://blog.csdn.net/sjmz30071360/article/details/82561867
[Hadoop]: https://blog.csdn.net/sjmz30071360/article/details/79889055
[70]: /images/20220404/8203c9a22115445a877dc71e65d29c22.png
[70 1]: /images/20220404/c8d1685ff3494ea790b274e4b3152fdd.png
[70 2]: /images/20220404/ec503af1bdcb4433b6a40a4f21484bb5.png
[70 3]: /images/20220404/e9d3d9f2f56b47c99f0b3b2870c6c08c.png
[70 4]: /images/20220404/54e79324890e4e4282cfb624864b461e.png
[70 5]: /images/20220404/f97490ab9e734bb498cdeb7f9fbf1fab.png
[70 6]: /images/20220404/579bc22d77c446ddb2be4725bcb36a6f.png
[70 7]: /images/20220404/2af361ffe3da4634a6cb0b12c1eac681.png