openstack虚拟机磁盘io error问题
作者：互联网
1：客户虚拟机出现readonly问题，最初操作是对磁盘进行了重新挂载，但是无效。
mount -o remount rw /
2：客户进行了重启虚拟机，之后进入了单用户模式提示是fstab问题，注释掉/etc/fstab中的vdb，之后再次重启正常。
3：进入虚拟机对vdb进行挂载操作提示io error。怀疑是ceph集群集群出了问题，查看集群状态正常，物理磁盘无异常，使用如下命令对磁盘进行修复：
xfs_repair -v -L /dev/vdb
之后提示磁盘io error，如下：
[ 164.966649] blk_update_request: I/O error, dev vdb, sector 0
[ 164.967942] Buffer I/O error on dev vdb, logical block 0, lost async page write
[ 164.968885] blk_update_request: I/O error, dev vdb, sector 16
[ 164.969652] Buffer I/O error on dev vdb, logical block 2, lost async page write
[ 164.970633] Buffer I/O error on dev vdb, logical block 3, lost async page write
[ 164.971711] Buffer I/O error on dev vdb, logical block 4, lost async page write
[ 164.972838] Buffer I/O error on dev vdb, logical block 5, lost async page write
[ 164.974001] Buffer I/O error on dev vdb, logical block 6, lost async page write
[ 164.975050] Buffer I/O error on dev vdb, logical block 7, lost async page write
[ 164.976249] Buffer I/O error on dev vdb, logical block 8, lost async page write
[ 164.977188] Buffer I/O error on dev vdb, logical block 9, lost async page write
[ 164.978043] Buffer I/O error on dev vdb, logical block 10, lost async page write
[ 164.978930] blk_update_request: I/O error, dev vdb, sector 29360128
[ 164.983874] blk_update_request: I/O error, dev vdb, sector 29360208
[ 164.992586] blk_update_request: I/O error, dev vdb, sector 29360264
[ 165.000701] blk_update_request: I/O error, dev vdb, sector 29366384
[ 165.010514] blk_update_request: I/O error, dev vdb, sector 29368784
[ 165.018809] blk_update_request: I/O error, dev vdb, sector 29435544
[ 165.026458] blk_update_request: I/O error, dev vdb, sector 29452288
[ 165.034674] blk_update_request: I/O error, dev vdb, sector 33554432
4：卸载磁盘，尝试通过为磁盘拍快照，并通过快照创建磁盘的方式再次挂载
快照创建失败，提示卷为只读模式，如下：
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server File "rbd.pyx", line 3484, in rbd.Image.create_snap
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server ReadOnlyImage: [errno 30] error creating snapshot snapshot-0aa66c79-6c0f-4735-9f77-da9ade0e11fa from volume-6b3c1c7d-4337-4301-b224-3746686dec05
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server
5：查看卷状态，如下：
[root@bjyz-icontrol01 ~]# cinder show 6b3c1c7d-4337-4301-b224-3746686dec05
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-02-04T03:10:53.000000 |
| description | |
| encrypted | False |
| group_id | None |
| id | 6b3c1c7d-4337-4301-b224-3746686dec05 |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | 公共服务-2 |
| os-vol-host-attr:host | control@tencent#tencent |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | fe2aeba987624a66864f5aa6992c64a7 |
| provider_id | None |
| replication_status | None |
| service_uuid | 43314489-49f4-43e3-8d5e-6512a7ab3953 |
| shared_targets | False |
| size | 1024 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2021-09-17T10:19:15.000000 |
| user_id | 6fbcd30ba47b4480ba57f56fd5fa45e0 |
| volume_type | tencent |
+--------------------------------+--------------------------------------+
通过ceph查看卷状态，如下：
(ceph-mon)[root@bjyz-icontrol01 /]# rbd info tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
rbd image 'volume-6b3c1c7d-4337-4301-b224-3746686dec05':
size 1 TiB in 262144 objects
order 22 (4 MiB objects)
snapshot_count: 1
id: d301e3babf00f2
block_name_prefix: rbd_data.d301e3babf00f2
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Thu Feb 4 11:10:53 2021
access_timestamp: Fri Sep 17 18:58:02 2021
modify_timestamp: Fri Sep 17 15:33:18 2021
###########################################
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock ls tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
There is 1 exclusive lock on this image.
Locker ID Address
client.98051222 auto 94571522251008 192.168.2.31:0/1375643388
(ceph-mon)[root@bjyz-icontrol01 /]#
ssh到192.168.2.31主机，发现并没有此进程，
需要把此读写锁删除，如下：
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock rm tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05 "auto 94571522251008" client.98051222
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock ls tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
6：再次挂载卷正常
标签：17,虚拟机,dev,vdb,2021,io,error,block 来源： https://blog.csdn.net/weixin_40579389/article/details/120875351