MAC本地搭建Hadoop、Hbase、kafka、zookeeper环境

背景

由于项目中需要用到实时处理框架，基于现有架构选择spark streaming消费kafka使用scala语言进行开发。出于服务器资源紧张以及登录测试服务器需要动态验证，故在本地搭建环境进行测试。

环境准备

配置免密登录并修改/etc/hosts
- 127.0.0.1 localhost
- 127.0.0.1 hadoop000
- sudo scutil --set HostName hadoop000 -- 修改终端登录显示
下载安装JDK1.8:
- https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
- 配置环境变量: vim ~/.bash_profile

export MAVEN_HOME="/Users/xieyin/soft/apache-maven-3.6.1/bin"
export PATH="$MAVEN_HOME/bin:$PATH"
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home
export SCALA_HOME=/Users/xieyin/Soft/scala-2.11.8
export PATH=$PATH:$SCALA_HOME/bin:$JAVA_HOME/bin:/usr/local/Cellar/unrar/5.7.5/bin
export PATH="/usr/local/Cellar/redis/5.0.5/bin:$PATH"
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"
export KE_HOME=/Users/xieyin/Soft/kafka-eagle/kafka-eagle-web-1.3.9
export PATH=$KE_HOME/bin:$PATH
export HADOOP_HOME=/Users/xieyin/Soft/hadoop-2.6.0-cdh5.15.1
export PATH=$HADOOP_HOME/bin:$PATH
export HBASE_HOME=/Users/xieyin/Soft/hbase-1.2.0-cdh5.15.1
export PATH=$HBASE_HOME/bin:$PATH

Mac默认安装最新版本kafka2.3.1、zookeeper3.4.14

# 安装
brew install zookeeper
brew install kafka
# 启动
brew services start zookeeper
brew services start kafka

下载安装hadoop
- http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.15.1.tar.gz
- 修改配置文件

# site-core.xml
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hadoop000:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/Users/xieyin/tmp/hadoop2.6</value>
  </property>

# hdfs-site.xml
<property>
  <name>dfs.replication</name>
    <value>1</value>
    <description>设置副本数</description>
</property>

<property>
   <name>dfs.namenode.name.dir</name>
   <value>/Users/xieyin/tmp/hadoop2.6/dfs/name</value>
   <description>设置存放NameNode的文件路径</description>
</property>

<property>
   <name>dfs.datanode.data.dir</name>
   <value>/Users/xieyin/tmp/hadoop2.6/dfs/data</value>
  <description>设置存放DataNode的文件路径</description>
</property>

<property>
    <name>dfs.permissions</name>
    <value>false</value>
    <description>权限还是不要的好</description>
</property>


# copy一份cp mapred-site.xml.template mapred-site.xml 
# mapred-site.xml
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

# yarn-site.xml
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    <description>由于我的测试机内存少，所以就关闭虚拟内存检测</description>
</property>

下载安装HBASE
- http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.15.1.tar.gz
- 修改配置文件

# hbase-env.sh
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home
export HBASE_MANAGES_ZK=false
# hbase-site.xml
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://hadoop000:9000/hbase</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>hadoop000:2181</value>
</property>

启动hadoop

格式化namenode
- $HADOOP_HOME/bin/hadoop namenode -format
启动HDFS
- $HADOOP_HOME/sbin/start-dfs.sh
打开web UI
http://hadoop000:50070/

image.png

启动HBASE

start-hbase.sh
- 日志: /Users/xieyin/Soft/hbase-1.2.0-cdh5.15.1/logs/hbase-xieyin-master-hadoop000.log
web UI
- http://hadoop000:60010/
  
  image.png