其他分享
首页 > 其他分享> > HDFS体系结构

HDFS体系结构

作者:互联网

HDFS支持主从结构,主节点称为NameNode,是因为主节点上运行的有NameNode的进程,NameNode支持多个
从节点称为DataNode,是因为从节点上运行的有DataNode进程。
HDFS还包含一个SecondaryNameNode进程。
HDFS体系结构如下图

NameNode介绍

NameNode:主要维护整个文件系统的文件目录树,文件/目录的信息和每个文件对应的数据块列表,并且还负责接收用户的操作请求。

NameNode中包含的文件

在hadoop安装包中,有hdfs-default.xml文件(hadoop-3.2.0/share/hadoop/hdfs/hadoop-hdfs-3.2.0.jar),hdfs-site.xml属于default的扩展文件,如果hdfs-site中配置了,则优先生效hdfs-site.xml

<property>
  <name>dfs.namenode.name.dir</name>
  <value>file://${hadoop.tmp.dir}/dfs/name</value>
  <description>Determines where on the local filesystem the DFS name node
      should store the name table(fsimage).  If this is a comma-delimited list
      of directories then the name table is replicated in all of the
      directories, for redundancy. </description>
</property>


<!--dataNode->
<property>
  <name>dfs.datanode.data.dir</name>
  <value>file://${hadoop.tmp.dir}/dfs/data</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices. The directories should be tagged
  with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
  storage policies. The default storage type will be DISK if the directory does
  not have a storage type tagged explicitly. Directories that do not exist will
  be created if local filesystem permission allows.
  </description>
</property>

file://${hadoop.tmp.dir}/dfs/name目录下存储Namenode相关信息。

[root@bigdata02 name]# cd current/
You have new mail in /var/spool/mail/root
[root@bigdata02 current]# ls
edits_0000000000000000001-0000000000000000002
edits_0000000000000000003-0000000000000000004
edits_0000000000000000005-0000000000000000005
edits_0000000000000000006-0000000000000000007
edits_0000000000000000008-0000000000000000009
edits_0000000000000000010-0000000000000000011
edits_0000000000000000012-0000000000000000013
edits_0000000000000000014-0000000000000000015
edits_0000000000000000016-0000000000000000017
edits_0000000000000000018-0000000000000000019
edits_0000000000000000020-0000000000000000033
edits_0000000000000000034-0000000000000000035
edits_0000000000000000036-0000000000000000037
edits_0000000000000000038-0000000000000000039
edits_0000000000000000040-0000000000000000041
edits_0000000000000000042-0000000000000000043
edits_0000000000000000044-0000000000000000045
edits_0000000000000000046-0000000000000000047
edits_0000000000000000048-0000000000000000049
edits_0000000000000000050-0000000000000000051
edits_0000000000000000052-0000000000000000053
edits_0000000000000000054-0000000000000000055
edits_0000000000000000056-0000000000000000057
edits_inprogress_0000000000000000058
fsimage_0000000000000000000
fsimage_0000000000000000000.md5
seen_txid
VERSION
[root@bigdata02 current]# pwd
/data/hadoop_repo/dfs/name/current

如查看fsimage_0000000000000000000,-i后表示输入地址,-o表示输出地址)

[root@bigdata02 current]# hdfs oiv -p XML -i fsimage_0000000000000000000 -o fsimage_0.xml
2021-04-03 10:10:18,910 INFO offlineImageViewer.FSImageHandler: Loading 2 strings
2021-04-03 10:10:18,959 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2021-04-03 10:10:18,959 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2021-04-03 10:10:18,959 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2021-04-03 10:10:18,959 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
``` xml
<?xml version="1.0"?>
<fsimage><version><layoutVersion>-65</layoutVersion><onDiskVersion>1</onDiskVersion><oivRevision>e97acb3bd8f3befd27418996fa5d4b50bf2e17bf</oivRevision></version>
<NameSection><namespaceId>425898831</namespaceId><genstampV1>1000</genstampV1><genstampV2>1000</genstampV2><genstampV1Limit>0</genstampV1Limit><lastAllocatedBlockId>1073741824</lastAllocatedBlockId><txid>0</txid></NameSection>
<ErasureCodingSection>
<erasureCodingPolicy>
<policyId>5</policyId><policyName>RS-10-4-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs</codecName><dataUnits>10</dataUnits><parityUnits>4</parityUnits></ecSchema>
</erasureCodingPolicy>

<erasureCodingPolicy>
<policyId>2</policyId><policyName>RS-3-2-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs</codecName><dataUnits>3</dataUnits><parityUnits>2</parityUnits></ecSchema>
</erasureCodingPolicy>

<erasureCodingPolicy>
<policyId>1</policyId><policyName>RS-6-3-1024k</policyName><cellSize>1048576</cellSize><policyState>ENABLED</policyState><ecSchema>
<codecName>rs</codecName><dataUnits>6</dataUnits><parityUnits>3</parityUnits></ecSchema>
</erasureCodingPolicy>

<erasureCodingPolicy>
<policyId>3</policyId><policyName>RS-LEGACY-6-3-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs-legacy</codecName><dataUnits>6</dataUnits><parityUnits>3</parityUnits></ecSchema>
</erasureCodingPolicy>

<erasureCodingPolicy>
<policyId>4</policyId><policyName>XOR-2-1-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>xor</codecName><dataUnits>2</dataUnits><parityUnits>1</parityUnits></ecSchema>
</erasureCodingPolicy>

</ErasureCodingSection>

<INodeSection><lastInodeId>16385</lastInodeId><numInodes>1</numInodes><inode><id>16385</id><type>DIRECTORY</type><name></name><mtime>0</mtime><permission>root:supergroup:0755</permission><nsquota>9223372036854775807</nsquota><dsquota>-1</dsquota></inode>
</INodeSection>
<INodeReferenceSection></INodeReferenceSection><SnapshotSection><snapshotCounter>0</snapshotCounter><numSnapshots>0</numSnapshots></SnapshotSection>
<INodeDirectorySection></INodeDirectorySection>
<FileUnderConstructionSection></FileUnderConstructionSection>
<SecretManagerSection><currentId>0</currentId><tokenSequenceNumber>0</tokenSequenceNumber><numDelegationKeys>0</numDelegationKeys><numTokens>0</numTokens></SecretManagerSection><CacheManagerSection><nextDirectiveId>1</nextDirectiveId><numDirectives>0</numDirectives><numPools>0</numPools></CacheManagerSection>
</fsimage>

查看edits文件

[root@bigdata02 current]# hdfs oev -i  edits_0000000000000000074-0000000000000000081 -o edits.xml
[root@bigdata02 current]# cat edits.xml 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<EDITS>
  <EDITS_VERSION>-65</EDITS_VERSION>
  <RECORD>
    <OPCODE>OP_START_LOG_SEGMENT</OPCODE>
    <DATA>
      <TXID>74</TXID>
    </DATA>
  </RECORD>
  <RECORD>
    <OPCODE>OP_ADD</OPCODE>
    <DATA>
      <TXID>75</TXID>
      <LENGTH>0</LENGTH>
      <INODEID>16387</INODEID>
      <PATH>/README.txt._COPYING_</PATH>
      <REPLICATION>2</REPLICATION>
      <MTIME>1617416075722</MTIME>
      <ATIME>1617416075722</ATIME>
      <BLOCKSIZE>134217728</BLOCKSIZE>
      <CLIENT_NAME>DFSClient_NONMAPREDUCE_-138837667_1</CLIENT_NAME>
      <CLIENT_MACHINE>192.168.21.102</CLIENT_MACHINE>
      <OVERWRITE>true</OVERWRITE>
      <PERMISSION_STATUS>
        <USERNAME>root</USERNAME>
        <GROUPNAME>supergroup</GROUPNAME>
        <MODE>420</MODE>
      </PERMISSION_STATUS>
      <ERASURE_CODING_POLICY_ID>0</ERASURE_CODING_POLICY_ID>
      <RPC_CLIENTID>c00ba5aa-d2b7-431c-914d-2a4c23209297</RPC_CLIENTID>
      <RPC_CALLID>3</RPC_CALLID>
    </DATA>
  </RECORD>
  <RECORD>
    <OPCODE>OP_ALLOCATE_BLOCK_ID</OPCODE>
    <DATA>
      <TXID>76</TXID>
      <BLOCK_ID>1073741828</BLOCK_ID>
    </DATA>
  </RECORD>
  <RECORD>
    <OPCODE>OP_SET_GENSTAMP_V2</OPCODE>
    <DATA>
      <TXID>77</TXID>
      <GENSTAMPV2>1004</GENSTAMPV2>
    </DATA>
  </RECORD>
  <RECORD>
    <OPCODE>OP_ADD_BLOCK</OPCODE>
    <DATA>
      <TXID>78</TXID>
      <PATH>/README.txt._COPYING_</PATH>
      <BLOCK>
        <BLOCK_ID>1073741828</BLOCK_ID>
        <NUM_BYTES>0</NUM_BYTES>
        <GENSTAMP>1004</GENSTAMP>
      </BLOCK>
      <RPC_CLIENTID/>
      <RPC_CALLID>-2</RPC_CALLID>
    </DATA>
  </RECORD>
  <RECORD>
    <OPCODE>OP_CLOSE</OPCODE>
    <DATA>
      <TXID>79</TXID>
      <LENGTH>0</LENGTH>
      <INODEID>0</INODEID>
      <PATH>/README.txt._COPYING_</PATH>
      <REPLICATION>2</REPLICATION>
      <MTIME>1617416075930</MTIME>
      <ATIME>1617416075722</ATIME>
      <BLOCKSIZE>134217728</BLOCKSIZE>
      <CLIENT_NAME/>
      <CLIENT_MACHINE/>
      <OVERWRITE>false</OVERWRITE>
      <BLOCK>
        <BLOCK_ID>1073741828</BLOCK_ID>
        <NUM_BYTES>1361</NUM_BYTES>
        <GENSTAMP>1004</GENSTAMP>
      </BLOCK>
      <PERMISSION_STATUS>
        <USERNAME>root</USERNAME>
        <GROUPNAME>supergroup</GROUPNAME>
        <MODE>420</MODE>
      </PERMISSION_STATUS>
    </DATA>
  </RECORD>
  <RECORD>
    <OPCODE>OP_RENAME_OLD</OPCODE>
    <DATA>
      <TXID>80</TXID>
      <LENGTH>0</LENGTH>
      <SRC>/README.txt._COPYING_</SRC>
      <DST>/README.txt</DST>
      <TIMESTAMP>1617416075939</TIMESTAMP>
      <RPC_CLIENTID>c00ba5aa-d2b7-431c-914d-2a4c23209297</RPC_CLIENTID>
      <RPC_CALLID>8</RPC_CALLID>
    </DATA>
  </RECORD>
  <RECORD>
    <OPCODE>OP_END_LOG_SEGMENT</OPCODE>
    <DATA>
      <TXID>81</TXID>
    </DATA>
  </RECORD>
</EDITS>

查看 seentxid

[root@bigdata02 current]# cat seen_txid 
100

查看 VERSION(格式化之后内容会发生变化)

cat VERSION 
#Sat Mar 27 14:16:52 CST 2021
namespaceID=425898831
clusterID=CID-eb08d57b-dd80-4f37-8941-690b25f34c13
cTime=1616825812250
storageType=NAME_NODE
blockpoolID=BP-694070002-192.168.21.102-1616825812250
layoutVersion=-65

fsimage

fsimages:元数据镜像文件,存储某一时刻NameNode内存中的元数据信息,类似于定时快照。[此处原信息指文目录树、文件/目录的信息、每个文件对应的数据块列表]

edits

操作日志文件[事务文件],这里面会实时记录用户的所有操作

seentxid

seentxid:存放transactionId文件,format之后是0,它代表的是namenode里面的edits文件的尾数,namenode重启的时候,会按照seen_txid的数字,顺序从头到尾跑edits_0000001~到seen_txid的数字,如果根据对应的seen_txid无法加载到对应的文件,NameNode进程将不会完成启动以保护数据的一致性。

version

VERSION:保存集群的版本信息

SecondaryNameNode

SecondaryNameNode主要负责定期的把edits文件中的内容合并到fsimage中,这个合并操作称为checkpoint,在合并的时候会对edits中的内容进行转换,生成新的内容保存到fsimage文件中。
注意:在NameNode HA 架构中没有SecondaryNameNode进程,文件合并操作会友standby NameNode负责实现。
因此在Hadoop集群中,SecondaryNameNode不是必须的

DataNode介绍。

DataNode是提供真实文件数据的存储服务。
查看DataNode文件目录

<property>
  <name>dfs.datanode.data.dir</name>
  <value>file://${hadoop.tmp.dir}/dfs/data</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices. The directories should be tagged
  with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
  storage policies. The default storage type will be DISK if the directory does
  not have a storage type tagged explicitly. Directories that do not exist will
  be created if local filesystem permission allows.
  </description>
</property>

查看某个block文件


[root@bigdata03 subdir0]# cat blk_1073741828
For the latest information about Hadoop, please visit our website at:

   http://hadoop.apache.org/

and our wiki, at:

   http://wiki.apache.org/hadoop/

This distribution includes cryptographic software.  The country in 
which you currently reside may have restrictions on the import, 
possession, use, and/or re-export to another country, of 
encryption software.  BEFORE using any encryption software, please 
check your country's laws, regulations and policies concerning the
import, possession, or use, and re-export of encryption software, to 
see if this is permitted.  See <http://www.wassenaar.org/> for more
information.

The U.S. Government Department of Commerce, Bureau of Industry and
Security (BIS), has classified this software as Export Commodity 
Control Number (ECCN) 5D002.C.1, which includes information security
software using or performing cryptographic functions with asymmetric
algorithms.  The form and manner of this Apache Software Foundation
distribution makes it eligible for export under the License Exception
ENC Technology Software Unrestricted (TSU) exception (see the BIS 
Export Administration Regulations, Section 740.13) for both object 
code and source code.

The following provides more details on the included cryptographic
software:
  Hadoop Core uses the SSL libraries from the Jetty project written 
by mortbay.org.

Block

HDFS会按照固定的大小,顺序对文件进行划分并编号,划分好的每一个块称为一个block,HDFS默认Block大小是128MB
Block块是HDFS读写数据的基本单位

如果一个文件小于一个数据块的大小,并不会占用整个数据块的存储空间。

副本数

默认副本数是3,副本保证数据安全
配置地址hdfs-site.xml中进行配置,dfs.replication

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>bigdata02:50090</value>
    </property>
</configuration>

而在hdfs-default.xml中可看到默认副本数为3

标签:hdfs,10,HDFS,dfs,edits,文件,root,体系结构
来源: https://www.cnblogs.com/shine-rainbow/p/hdfs-ti-xi-jie-gou.html