目的
3台虚拟机实现1个namenode，2个datanode的hadoop集群。
环境
硬件：MAC * 1；Parallels Desktop虚拟机*3：CentOS-7
使用ssh客户端操作3台机器
JDK版本
[keon@centos-linux bin]$ /usr/jdk1.8.0_101/bin/java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
HADOOP版本
[keon@centos-linux bin]$ ./hdfs version
Hadoop 2.7.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41
Compiled by jenkins on 2016-01-26T00:08Z
Compiled with protoc 2.5.0
From source with checksum d0fda26633fa762bff87ec759ebe689c
This command was run using /usr/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar
配置过程
1.修改host
[keon@centos-linux bin]$ vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.211.55.6 h1
10.211.55.7 h3
10.211.55.8 h2
2.配置免密码
在h1生成密匙
[keon@centos-linux bin]$ ssh-keygen -t rsa
[keon@centos-linux ~]$ ls .ssh/
id_rsa id_rsa.pub known_hosts
在h2,h3拷贝公匙
scp keon@h1:/home/keon/.ssh/id_rsa.pub /home/keon/.ssh/authorized_keys
在h1验证是否免密码成功
[keon@centos-linux ~]$ ssh h2
Last login: Tue Aug 2 10:34:14 2016 from h1
[keon@centos-linux ~]$ exit
logout
Connection to h2 closed.
[keon@centos-linux ~]$ ssh h3
Last login: Tue Aug 2 10:34:25 2016 from h1
3.修改配置文件
[keon@centos-linux hadoop-2.7.2]$ pwd
/usr/hadoop-2.7.2
(1)etc/hadoop/hadoop-env.sh
[keon@centos-linux hadoop-2.7.2]$ vim etc/hadoop/hadoop-env.sh
修改java环境地址
export JAVA_HOME=/usr/jdk1.8.0_101
(2)etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://h1:9000/</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/hadoop-2.7.2/tmp</value>
</property>
</configuration>
(3)etc/hadoop/hdfs-site.xml
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/hadoop-2.7.2/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/hadoop-2.7.2/datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
(4)etc/hadoop/mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
修改完后，推送到每一个节点
启动测试
1.格式化namenode
[keon@centos-linux hadoop-2.7.2]$ ./bin/hdfs namenode -format
16/08/02 11:25:45 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = h1/10.211.55.6
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.2
STARTUP_MSG: classpath = /usr/hadoop-2.7.2/etc
.
.
.
16/08/02 11:25:47 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/08/02 11:25:47 INFO util.GSet: VM type = 64-bit
16/08/02 11:25:47 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
16/08/02 11:25:47 INFO util.GSet: capacity = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /usr/hadoop-2.7.2/namenode ? (Y or N) y
16/08/02 11:25:49 INFO namenode.FSImage: Allocated new BlockPoolId: BP-566776757-10.211.55.6-1470108349176
16/08/02 11:25:49 INFO common.Storage: Storage directory /usr/hadoop-2.7.2/namenode has been successfully formatted.
16/08/02 11:25:49 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/08/02 11:25:49 INFO util.ExitUtil: Exiting with status 0
16/08/02 11:25:49 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at h1/10.211.55.6
************************************************************/
2.启动服务
启动hdfs
./sbin/start-dfs.sh
Starting namenodes on [h1]
h1: starting namenode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-namenode-centos-linux.shared.out
h3: starting datanode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-datanode-centos-linux.shared.out
h2: starting datanode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-datanode-centos-linux.shared.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-secondarynamenode-centos-linux.shared.out
启动yarn
./sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop-2.7.2/logs/yarn-keon-resourcemanager-centos-linux.shared.out
h3: starting nodemanager, logging to /usr/hadoop-2.7.2/logs/yarn-keon-nodemanager-centos-linux.shared.out
h2: starting nodemanager, logging to /usr/hadoop-2.7.2/logs/yarn-keon-nodemanager-centos-linux.shared.out
3.验证服务
h1
[keon@centos-linux hadoop-2.7.2]$ /usr/jdk1.8.0_101/bin/jps
2930 Jps
1442 SecondaryNameNode
1045 NameNode
2443 ResourceManager
h2
[keon@centos-linux hadoop-2.7.2]$ /usr/jdk1.8.0_101/bin/jps
7703 DataNode
29240 Jps
8286 NodeManager
h3
[root@centos-linux keon]# /usr/jdk1.8.0_101/bin/jps
11584 NodeManager
12273 Jps
10346 DataNode
相关进程已启动
