centos7部署lustre文件系统
作者：互联网
1. 概述
最新更新：2022年8月12日 21:16:00
本文使用kvm虚拟机node[19-24]在centos7.9操作系统，部署lustre文件系统2.12
集群规划如下：
| 节点名称 | 磁盘数目 | 服务 |
|---|---|---|
| node19/node20 | 共享/dev/vbd | mgt/mgs |
| node21/node22 | 共享/dev/vd{b,c} | mdt/mds |
| node23/node24 | 共享/dev/vd{b,c,d,e,f,g} | oss/ost |
| node25 | client |
2. 虚拟机准备
说明：本篇使用kvm的克隆功能，实现快速部署。
也可以创建多个节点以后，使用批处理命令完成操作
2.1 创建一个lustre内核操作系统模板
virsh start node19
启动node19节点，创建yum源：vim /etc/yum.repo.d/whamcloud.repo
[lustre_latest_el7_client]
name=whamcloud_lustre_client
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.12.8/el7.9.2009/client/
enabled=1
priority=5
gpgcheck=0
[lustre_latest_el7_server]
name=whamcloud_lustre_server
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.12.8/el7.9.2009/server/
enabled=1
priority=5
gpgcheck=0
[e2fsprogs_latest]
name=whamcloud_e2fsprogs
baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/
enabled=1
priority=5
gpgcheck=0
执行安装yum插件：
yum -y install yum-plugin-priorities
执行安装lustre相关软件：
yum -y install e2fsprogs e2fsprogs-libs libss libcom_err kernel-*lustre kmod-lustre kmod-lustre-osd-ldiskfs kmod-lustre-tests lustre-osd-ldiskfs lustre-osd-ldiskfs-mount lustre-modules lustre-ldiskfs-dkms lustre
编辑内核配置文件：vim /etc/modprobe.d/lnet.conf
optinos lnet networks=tcp0(eth0)
编辑启动脚本vim /etc/rc.local
modprobe lnet
modprobe lustre
modprobe ldiskfs
chmod +x /etc/rc.local
关闭node19节点：virsh destroy node19
2.2 使用模板克隆生成新节点
for node in {20..24}
do
virt-clone -o node19 -n node${node} -f /var/lib/libvirt/images/node${node}.qcow2 -m 00:00:00:80:00:${node}
done
2.3 为虚拟机添加共享磁盘
本篇添加共享磁盘的方式为：virsh edit xxx。
也可以通过修改xyz.xml后，使用
virsh define /etc/libvirt/qemu/xyz.xml导入修改
先创建虚拟磁盘：
# 1. 创建lustre-mgt-000.img，挂载给node19和node20,盘符/dev/vdb
qemu-img create -f raw lustre-mgt-000.img 10G
# 2. 创建lustre-mdt-000.img和lustre-mdt-001.img，挂载给node21和node22,盘符/dev/vd{b,c}
qemu-img create -f raw lustre-mdt-000.img 10G
qemu-img create -f raw lustre-mdt-001.img 10G
# 3. 创建lustre-ost-000.img - lustre-ost-006.img，挂载给node23,node24，盘符/dev/vd{b,c,d,e,f,g}
for i in {000..006}
do
qemu-img create -f raw lustre-ost-${i}.img 10G
done
为node19和node20添加共享磁盘
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/lustre-mgt-000.img'/>
<target dev='vdb' bus='virtio'/>
<shareable/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x0'/>
</disk>
为node21和node22添加共享磁盘
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/lustre-mdt-000.img'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x0'/>
<shareable/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/lustre-mdt-001.img'/>
<target dev='vdc' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
<shareable/>
</disk>
为node23和node24添加共享磁盘
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/lustre-ost-000.img'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x0'/>
<shareable/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/lustre-ost-001.img'/>
<target dev='vdc' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
<shareable/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/lustre-ost-002.img'/>
<target dev='vdd' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x07' function='0x0'/>
<shareable/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/lustre-ost-003.img'/>
<target dev='vde' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x07' function='0x0'/>
<shareable/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/lustre-ost-004.img'/>
<target dev='vdf' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x07' function='0x0'/>
<shareable/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/lustre-ost-005.img'/>
<target dev='vdg' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x07' function='0x0'/>
<shareable/>
</disk>
3.创建文件系统
3.1 mgs服务创建
在node19上执行：
# 格式化设备
mkfs.lustre --fsname=mylustre --reformat --mgs --servicenode=192.168.80.19@tcp0 --servicenode=192.168.80.20@tcp0 /dev/vdb
# 创建挂载点
mkdir -p /mnt/mgt000
# 挂载设备
mount.lustre /dev/vdb /mnt/mgt000
node20上面不用创建，直接挂载即可
3.2 mds服务创建
在node21上执行：
# 格式化设备
mkfs.lustre --fsname=mylustre --reformat --mdt --index=0 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.21@tcp0 --servicenode=192.168.80.22@tcp0 /dev/vdb
mkfs.lustre --fsname=mylustre --reformat --mdt --index=1 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.21@tcp0 --servicenode=192.168.80.22@tcp0 /dev/vdc
# 创建挂载点
mkdir -p /mnt/mdt{000,001}
# 挂载设备
mount.lustre /dev/vdb /mnt/mdt000
在node22上执行：
# 创建挂载点
mkdir -p /mnt/mdt{000,001}
# 挂载设备
mount.lustre /dev/vdc /mnt/mdt001
3.3 oss服务创建
在node23上执行：
#!/bin/bash
#/dev/vdb
mkfs.lustre --fsname=mylustre --reformat --ost --index=0 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdb
#/dev/vdc
mkfs.lustre --fsname=mylustre --reformat --ost --index=1 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdc
#/dev/vdd
mkfs.lustre --fsname=mylustre --reformat --ost --index=2 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdd
# mkdir
mkdir -p /mnt/{ost000,ost001,ost002,ost003,ost004,ost005}
# mount
mount.lustre /dev/vdb /mnt/ost000
mount.lustre /dev/vdc /mnt/ost001
mount.lustre /dev/vdd /mnt/ost002
在node24上执行：
#!/bin/bash
#/dev/vde
mkfs.lustre --fsname=mylustre --reformat --ost --index=3 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vde
#/dev/vdf
mkfs.lustre --fsname=mylustre --reformat --ost --index=4 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdf
#/dev/vdg
mkfs.lustre --fsname=mylustre --reformat --ost --index=5 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdg
# mkdir
mkdir -p /mnt/{ost000,ost001,ost002,ost003,ost004,ost005}
# mount
mount.lustre /dev/vde /mnt/ost003
mount.lustre /dev/vdf /mnt/ost004
mount.lustre /dev/vdg /mnt/ost005
4. 创建客户端虚拟机并挂载
node25执行yum -y install lustre-client
创建挂载节点：mkdir -p /home/export/online1
挂载文件系统:mount.lustre 192.168.80.19@tcp0:192.168.80.20@tcp0:/mylustre /home/export/online1/
执行命令：lfs df -h
UUID bytes Used Available Use% Mounted on
mylustre-MDT0000_UUID 5.5G 2.0M 5.0G 1% /home/export/online1[MDT:0]
mylustre-MDT0001_UUID 5.5G 2.0M 5.0G 1% /home/export/online1[MDT:1]
mylustre-OST0000_UUID 9.2G 1.4M 8.7G 1% /home/export/online1[OST:0]
mylustre-OST0001_UUID 9.2G 1.4M 8.7G 1% /home/export/online1[OST:1]
mylustre-OST0002_UUID 9.2G 1.4M 8.7G 1% /home/export/online1[OST:2]
mylustre-OST0003_UUID 9.2G 1.4M 8.7G 1% /home/export/online1[OST:3]
mylustre-OST0004_UUID 9.2G 1.4M 8.7G 1% /home/export/online1[OST:4]
mylustre-OST0005_UUID 9.2G 1.4M 8.7G 1% /home/export/online1[OST:5]
filesystem_summary: 55.1G 8.1M 52.0G 1% /home/export/online1
标签：--,lustre,文件系统,192.168,centos7,tcp0,dev,mylustre 来源： https://www.cnblogs.com/liwanliangblog/p/16581385.html