Hadoop Pseudo-Distributed Mode Deployment
作者：互联网
Pre-condition and software
Hadoop version: hadoop-2.6.0-cdh5.7.0
JDK version：jdk-8u45-linux-x64
Linux install ssh service
1.Create user and upload hadoop installation package
wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
rz (upload command)
[root@10-9-15-140 ~]# useradd hadoop
[root@10-9-15-140 ~]# su - hadoop
[hadoop@10-9-15-140 ~]$
[hadoop@10-9-15-140 ~]$ mkdir app
[hadoop@10-9-15-140 ~]$ cd app/
[hadoop@10-9-15-140 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
或
[hadoop@10-9-15-140 app]$ rz
2.jdk deployment
CDH environment:
mkdir /usr/java jdk deployment
mkdir /usr/share/java CDH deployment need mysql jdbc jar package
rz upload jdk-8u45-linux-x64.gz
unzip
[root@10-9-15-140 java]# tar -xzvf jdk-8u45-linux-x64.gz
[root@10-9-15-140 java]# ll
total 169216
drwxr-xr-x 8 uucp 143 4096 Apr 11 2015 jdk1.8.0_45
-rw-r--r-- 1 hadoop hadoop 173271626 Jun 30 23:59 jdk-8u45-linux-x64.gz
Change permission
[root@10-9-15-140 java]# chown -R root:root jdk1.8.0_45
[root@10-9-15-140 java]# ll
total 169216
drwxr-xr-x 8 root root 4096 Apr 11 2015 jdk1.8.0_45
-rw-r--r-- 1 hadoop hadoop 173271626 Jun 30 23:59 jdk-8u45-linux-x64.gz
[root@10-9-15-140 java]# vi /etc/profile
#env
export JAVA_HOME=/usr/share/java/jdk1.8.0_45
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JER_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JER_HOME/bin:$PATH
[root@10-9-15-140 java]# source /etc/profile
[root@10-9-15-140 java]# which java
/usr/share/java/jdk1.8.0_45/bin/java
3.unzip hadoop package
[hadoop@10-9-15-140 app]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@10-9-15-140 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@10-9-15-140 app]$ ll
total 304288
drwxr-xr-x 14 hadoop hadoop 4096 Mar 24 2016 hadoop-2.6.0-cdh5.7.0
-rw-r--r-- 1 hadoop hadoop 311585484 Jun 30 23:59 hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@10-9-15-140 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ ll
total 76
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24 2016 bin *execution script
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24 2016 bin-mapreduce1
drwxr-xr-x 3 hadoop hadoop 4096 Mar 24 2016 cloudera
drwxr-xr-x 6 hadoop hadoop 4096 Mar 24 2016 etc *configuration menu(conf)
drwxr-xr-x 5 hadoop hadoop 4096 Mar 24 2016 examples
drwxr-xr-x 3 hadoop hadoop 4096 Mar 24 2016 examples-mapreduce1
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24 2016 include
drwxr-xr-x 3 hadoop hadoop 4096 Mar 24 2016 lib *jar package munu
drwxr-xr-x 2 hadoop hadoop 4096 Mar 24 2016 libexec
-rw-r--r-- 1 hadoop hadoop 17087 Mar 24 2016 LICENSE.txt
-rw-r--r-- 1 hadoop hadoop 101 Mar 24 2016 NOTICE.txt
-rw-r--r-- 1 hadoop hadoop 1366 Mar 24 2016 README.txt
drwxr-xr-x 3 hadoop hadoop 4096 Mar 24 2016 sbin *hadoop component‘s start and stop script
drwxr-xr-x 4 hadoop hadoop 4096 Mar 24 2016 share
drwxr-xr-x 17 hadoop hadoop 4096 Mar 24 2016 src
4.Configuration
Use the following:
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
5.Config ssh localhost non password trusted connection
[hadoop@10-9-15-140 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
53:21:86:1c:af:cd:f3:c4:45:a4:ca:72:ea:83:58:37 hadoop@10-9-15-140
The key's randomart image is:
+--[ RSA 2048]----+
| ..oo ..o |
| oo . + |
| . o . |
| = + . |
| o S o |
| . E+ = |
| o o.. . |
| . ... |
| .. |
+-----------------+
[hadoop@10-9-15-140 ~]$ cd .ssh
[hadoop@10-9-15-140 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Jul 2 01:18 id_rsa
-rw-r--r-- 1 hadoop hadoop 400 Jul 2 01:18 id_rsa.pub
[hadoop@10-9-15-140 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@10-9-15-140 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop 400 Jul 2 01:18 authorized_keys
-rw------- 1 hadoop hadoop 1675 Jul 2 01:18 id_rsa 私钥
-rw-r--r-- 1 hadoop hadoop 400 Jul 2 01:18 id_rsa.pub 公钥
ssh localhost date (need to input password, but this user do not need the password.
We need to setup the trusted connetion without password.
Change permission
[hadoop@10-9-15-140 .ssh]$ chmod 600 authorized_keys
[hadoop@10-9-15-140 .ssh]$ ssh localhost date
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 68:d4:3b:cb:d1:f9:dc:23:65:81:a9:a0:fd:e9:ec:3b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Tue Jul 2 01:19:38 CST 2019
#ssh test successfully
[hadoop@10-9-15-140 .ssh]$ ssh localhost date
Tue Jul 2 01:19:53 CST 2019
6.Format
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format
7.Start
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
19/07/02 09:48:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-10-9-15-140.out
localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-10-9-15-140.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-10-9-15-140.out
19/07/02 09:48:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ jps
29539 Jps
29222 DataNode
29405 SecondaryNameNode
29134 NameNode
Two issues:
Issue One:
start-dfs.sh启动出现Error:cannot find configuration directory:/etc/hadoop
将hadoop-env.sh中的
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
改为绝对路径
export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
Issue Two:
Hadoop安装完后，启动时报Error: JAVA_HOME is not set and could not be found.
解决办法：
修改/etc/hadoop/hadoop-env.sh中设JAVA_HOME。
应当使用绝对路径。
export JAVA_HOME=$JAVA_HOME //错误，原来就这样的 不该就报错
export JAVA_HOME=/usr/share/java/jdk1.8.0_45/bin/java //正确，应该这么改
8.环境变量
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ cat ~/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_PREFIX/bin:$PATH
9.正式使用
通过浏览器查看NameNode
http://NameNodeIP机器IP地址:50070/
注意：云主机需要在防火墙添加端口
10.停止NameNode和DataNode
$ sbin/stop-dfs.sh
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ sbin/stop-dfs.sh
19/07/03 00:10:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
19/07/03 00:10:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ jps
23642 Jps
参考资料：
apache hadoop： hadoop.apache.org
cdh hadoop: http://archive.cloudera.com/cdh5/cdh/5/
标签：10,cdh5.7,15,140,Distributed,Pseudo,hadoop,Deployment,2.6 来源： https://blog.csdn.net/weixin_41158277/article/details/94892674