Hadoop2.7.7版本安装（CentOs7环境）
作者：互联网
Hadoop-2.7.7集群安装
下载地址，可选择版本：https://archive.apache.org/dist/hadoop/common/
只在Master节点上安装
安装步骤
- 解压安装
- 用户授权
$ cd ~/Downloads
$ rz
# 上传下载到windows内的hadoop文件
$ sudo tar -zxf ./hadoop-2.7.7.tar.gz -C /usr/local
$ cd /usr/local
# 更改文件夹名称便于后面的配置
$ sudo mv ./hadoop-2.2.7/ ./hadoop
$ sudo chown -R qinphy ./hadoop
环境配置
$ vi ~/.bashrc
配置内容
export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/hadoop/sbin
$ source ~/.bashrc
# 检验安装及环境配置
$ cd ~
$ hadoop version
hadoop-2.2.7
...
集群配置
$ vi slaves
workers文件是指定从节点
每行一个
Slave1 Slave2
$ vi core-site.xml
core-site.xml是全局参数配置文件
主要是用于配置NameNode与DataNode通信地址
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://Master:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/hadoop/tmp</value> </property> </configuration>
$ vi hdfs-site.xml
hdfs-site.xml配置内容：声明
dfs.replication是从节点个数
<configuration> <property> <name>dfs.namenode.http-address</name> <value>Master:50090</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>Slave1:50070</value> </property> <property> <name>dfs.replication</name> <value>4</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/usr/local/hadoop/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/usr/local/hadoop/tmp/dfs/data</value> </property> </configuration>
$ cp ./mapred-site.xml.template ./mapred-site.xml
$ rm -f ./mapred-site.xml.template
$ vi mapred-site.xml
mapred-site.xml配置
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>Master:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>Master:19888</value> </property> </configuration>
$ vi yarn-site.xml
yarn-site配置
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>Master:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>Master:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>Master:8031</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>Master:8030</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>Master:8100</value> </property> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> </configuration>
配置好从节点
# 转发到每个结点
$ cd /usr/local
$ sudo tar -zcf ~/hadoop.master.tar.gz ./hadoop
$ scp ~/hadoop.master.tar.gz qinphy@Slave1:/home/qinphy
$ scp ~/hadoop.master.tar.gz qinphy@Slave2:/home/qinphy
# 每个子节点上解压
$ sudo tar -zxf ~/hadoop.master.tar.gz -C /usr/local
# 授予权限
$ sudo chown -R qinphy /usr/local/hadoop
初始化
$ hdfs namenode -format
启动与关闭
$ start-all.sh
# 等同于start-dfs.sh+start-yarn.sh
查看效果：
# 在每个节点上
$ jps
# 在Master结点上
$ hdfs dfsadmin -report
# 停止Hadoop
$ stop-all.sh
jps结果：
# Master $ jps 10433 Jps 10317 ResourceManager 9967 NameNode # Slave1 $ jps 9796 NodeManager 9881 Jps 9722 SecondaryNameNode 9661 DataNod # 其余Slave结点 $ jps 9634 DataNode 9778 Jps 9715 NodeManagerreport结果：能够看到从节点信息
$ hdfs dfsadmin -report Configured Capacity: 36477861888 (33.97 GB) Present Capacity: 30168911872 (28.10 GB) DFS Remaining: 30168903680 (28.10 GB) DFS Used: 8192 (8 KB) DFS Used%: 0.00% Replicated Blocks: Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 Low redundancy blocks with highest priority to recover: 0 Pending deletion blocks: 0 Erasure Coded Block Groups: Low redundancy block groups: 0 Block groups with corrupt internal blocks: 0 Missing block groups: 0 Low redundancy blocks with highest priority to recover: 0 Pending deletion blocks: 0 ------------------------------------------------- Live datanodes (2): Name: 192.168.1.121:9866 (Slave1) Hostname: Slave1 Decommission Status : Normal Configured Capacity: 18238930944 (16.99 GB) DFS Used: 4096 (4 KB) Non DFS Used: 3154427904 (2.94 GB) DFS Remaining: 15084498944 (14.05 GB) DFS Used%: 0.00% DFS Remaining%: 82.70% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri May 01 08:42:26 CST 2020 Last Block Report: Fri May 01 08:41:45 CST 2020 Num of Blocks: 0 Name: 192.168.1.122:9866 (Slave2) Hostname: Slave2 Decommission Status : Normal Configured Capacity: 18238930944 (16.99 GB) DFS Used: 4096 (4 KB) Non DFS Used: 3154522112 (2.94 GB) DFS Remaining: 15084404736 (14.05 GB) DFS Used%: 0.00% DFS Remaining%: 82.70% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri May 01 08:42:26 CST 2020 Last Block Report: Fri May 01 08:41:37 CST 2020 Num of Blocks: 0查看web：192.168.1.120:50090
标签：xml,hadoop,DFS,site,CentOs7,Master,yarn,版本,Hadoop2.7 来源： https://blog.csdn.net/weixin_42631782/article/details/115438008