其他分享
首页 > 其他分享> > 分布式存储Ceph(七) 对象存储RadosGW使用

分布式存储Ceph(七) 对象存储RadosGW使用

作者:互联网

七 对象存储RadosGW使用

7.1 RadosGW 对象存储简介

7.2 对象存储的特点

7.2.1 对象存储的特点

  1. 通过对象存储将数据存储为对象,每个对象除了包含数据,还包含数据自身的元数据。
  2. 对象通过Object ID来检索,无法通过普通文件系统的方式通过文件路径名称操作直接访问对象,只能通过API来访问,或者第三方客户端。
  3. 对象存储中的对象不整理到目录树中,而是存储在扁平的命名空间中,Amazon S3将这个扁平名称空间称为bucket,而swift则将其称为容器。
  4. 无论bucket还是容器,都不能嵌套。
  5. ceph使用bucket作为存储桶,bucket需要被授权才能访问到。实现对象数据的存储和多用户隔离,数据存储在bucket中,用户的权限也是针对bucket进行授权,可以设置用户对不同的bucket拥有不同的权限,以实现权限管理。
  6. 方便横向扩展、快速检索数据。
  7. 不支持客户端挂载,且需要客户端在访问的时候指定文件名称。
  8. 不是很适用于文件过于频繁修改及删除的场景。

7.2.2 bucket 特性

  1. 存储空间是用于存储对象的容器,所有的对象都必须隶属于某个存储空间,可以设置和修改存储空间属性来控制地域、访问权限、生命周期等,这些属性设置直接作用于该存储空间内所有对象,因此您可以通过灵活创建不同的存储空间来完成不同的管理功能。
  2. 同一个存储空间内部都是扁平的,没有文件系统的目录概念,所有的对象都直接隶属于其对应的存储空间。
  3. 每个用户都可以拥有多个存储空间。
  4. 存储空间的名称在OSS范围内必须是全局唯一的,一旦创建之后无法修改名称。
  5. 存储空间内部的对象数目没有限制。

7.2.3 bucket命名规范

7.3 对象存储访问对比

7.4 部署RadosGW服务

7.4.1 安装radosgw服务

root@ceph-mgr-01:~# apt -y install radosgw
root@ceph-mgr-02:~# apt -y install radosgw

7.4.2 初始化radosgw

7.4.2.1 初始化ceph-mgr-01节点rgw服务

点击查看代码
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy rgw create ceph-mgr-01
[ceph_deploy.conf][DEBUG ] found configuration file at: /var/lib/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph-mgr-01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-mgr-01', 'rgw.ceph-mgr-01')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc0a912a1e0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0x7fc0a99d26d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-mgr-01:rgw.ceph-mgr-01
ceph@ceph-mgr-01's password: 
[ceph-mgr-01][DEBUG ] connection detected need for sudo
ceph@ceph-mgr-01's password: 
[ceph-mgr-01][DEBUG ] connected to host: ceph-mgr-01 
[ceph-mgr-01][DEBUG ] detect platform information from remote host
[ceph-mgr-01][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-mgr-01
[ceph-mgr-01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-mgr-01][WARNIN] rgw keyring does not exist yet, creating one
[ceph-mgr-01][DEBUG ] create a keyring file
[ceph-mgr-01][DEBUG ] create path recursively if it doesn't exist
[ceph-mgr-01][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-mgr-01 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-mgr-01/keyring
[ceph-mgr-01][INFO  ] Running command: sudo systemctl enable ceph-radosgw@rgw.ceph-mgr-01
[ceph-mgr-01][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-mgr-01.service → /lib/systemd/system/ceph-radosgw@.service.
[ceph-mgr-01][INFO  ] Running command: sudo systemctl start ceph-radosgw@rgw.ceph-mgr-01
[ceph-mgr-01][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-mgr-01 and default port 7480

7.4.2.2 初始化ceph-mgr-02节点rgw服务

点击查看代码
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy rgw create ceph-mgr-02
[ceph_deploy.conf][DEBUG ] found configuration file at: /var/lib/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph-mgr-02
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-mgr-02', 'rgw.ceph-mgr-02')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7efd6c1391e0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0x7efd6c9e16d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-mgr-02:rgw.ceph-mgr-02
ceph@ceph-mgr-02's password: 
12[ceph-mgr-02][DEBUG ] connection detected need for sudo
ceph@ceph-mgr-02's password: 
Permission denied, please try again.
ceph@ceph-mgr-02's password: 
[ceph-mgr-02][DEBUG ] connected to host: ceph-mgr-02 
[ceph-mgr-02][DEBUG ] detect platform information from remote host
[ceph-mgr-02][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-mgr-02
[ceph-mgr-02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-mgr-02][WARNIN] rgw keyring does not exist yet, creating one
[ceph-mgr-02][DEBUG ] create a keyring file
[ceph-mgr-02][DEBUG ] create path recursively if it doesn't exist
[ceph-mgr-02][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-mgr-02 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-mgr-02/keyring
[ceph-mgr-02][INFO  ] Running command: sudo systemctl enable ceph-radosgw@rgw.ceph-mgr-02
[ceph-mgr-02][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-mgr-02.service → /lib/systemd/system/ceph-radosgw@.service.
[ceph-mgr-02][INFO  ] Running command: sudo systemctl start ceph-radosgw@rgw.ceph-mgr-02
[ceph-mgr-02][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-mgr-02 and default port 7480

7.4.3 验证radosgw服务状态

点击查看代码
root@ceph-mgr-01:~# systemctl status ceph-radosgw@rgw.ceph-mgr-01
root@ceph-mgr-02:~# systemctl status ceph-radosgw@rgw.ceph-mgr-02

7.4.4 验证ceph集群rgw状态

点击查看代码
ceph@ceph-deploy:~/ceph-cluster$ ceph -s
  cluster:
    id:     6e521054-1532-4bc8-9971-7f8ae93e8430
    health: HEALTH_OK

services:
mon: 3 daemons, quorum ceph-mon-01,ceph-mon-02,ceph-mon-03 (age 35m)
mgr: ceph-mgr-01(active, since 9d), standbys: ceph-mgr-02
mds: 2/2 daemons up, 2 standby
osd: 9 osds: 9 up (since 3w), 9 in (since 3w)
rgw: 2 daemons active (2 hosts, 1 zones)

data:
volumes: 1/1 healthy
pools: 8 pools, 265 pgs
objects: 253 objects, 24 MiB
usage: 1.5 GiB used, 178 GiB / 180 GiB avail
pgs: 265 active+clean

7.4.5 验证radosgw服务进程

root@ceph-mgr-01:~# ps -ef | grep radosgw
ceph     13642     1  0 17:26 ?        00:00:00 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr-01 --setuser ceph --setgroup ceph

root@ceph-mgr-02:~# ps -ef | grep radosgw
ceph 10660 1 0 17:28 ? 00:00:00 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr-02 --setuser ceph --setgroup ceph

7.4.6 rgw存储池功能

点击查看代码
ceph@ceph-deploy:~/ceph-cluster$ ceph osd pool ls
device_health_metrics
.rgw.root           # 包含realm(领域信息),比如zone和zonegroup
default.rgw.log     # 存储日志信息,用于记录各种log信息
default.rgw.control # 系统控制池,在有数据更新时,通知其它rgw更新缓存
default.rgw.meta    # 元数据存储池,通过不同的名称空间分别存储不同的rados对象。这些名称空间包括用户UID及其bucket映射信息的名称空间users.uid、用户的密钥名称空间user.keys、用户email名称空间users.email、用户的subuser的名称空间users.swift,以及bucket的名称空间root等。
default.rgw.buckets.index # 有数据后会生成,存放bucket到object的索引信息
default.rgw.buckets.data  # 有数据后会生成,存放对象的数据
default.rgw.buckets.non-ec # 有数据后会生成,数据的额外信息存储池

7.4.7 验证rgw zone信息

点击查看代码
ceph@ceph-deploy:~/ceph-cluster$ radosgw-admin zone get --rgw-zone=default
{
    "id": "345c2549-82fd-4331-948f-4efff49c0f0e",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.log:gc",
    "lc_pool": "default.rgw.log:lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "roles_pool": "default.rgw.meta:roles",
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",
    "user_email_pool": "default.rgw.meta:users.email",
    "user_swift_pool": "default.rgw.meta:users.swift",
    "user_uid_pool": "default.rgw.meta:users.uid",
    "otp_pool": "default.rgw.otp",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "default.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "realm_id": "",
    "notif_pool": "default.rgw.log:notif"
}

7.4.8 访问radosgw服务

ceph@ceph-deploy:~/ceph-cluster$ curl http://172.16.10.248:7480
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

7.5 rgw服务配置

7.5.1 自定义端口

7.5.1.1 修改配置文件

点击查看代码
root@ceph-mgr-01:~# vim /etc/ceph/ceph.conf

[client.rgw.ceph-mgr-01]
rgw_host = ceph-mgr-01
rgw_frontends = civetweb port=9900

 7.5.1.2 重启rgw服务

root@ceph-mgr-01:~# systemctl restart ceph-radosgw@rgw.ceph-mgr-01

7.5.1.3 验证rgw服务端口

点击查看代码
root@ceph-mgr-01:~# netstat -tnlp | grep radosgw
tcp        0      0 0.0.0.0:9900            0.0.0.0:*               LISTEN      15949/radosgw 

7.5.2 rgw https

7.6 测试数据读写

7.6.1 创建rgw账户

点击查看代码
ceph@ceph-deploy:~/ceph-cluster$ radosgw-admin user create --uid="wgs01" --display-name="wgs01"
{
    "user_id": "wgs01",
    "display_name": "wgs01",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "wgs01",
            "access_key": "Q4XILSOF3BQ6MRBE2084",
            "secret_key": "Azbspmpx1A5IBARe6OVvjYfTKQsAlNSjzEOGgG76"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

7.6.2 安装s3cmd客户端

点击查看代码
root@ceph-client-01:~# apt -y install s3cmd

7.6.3  配置s3cmd客户端执行环境

7.6.3.1 配置s3cmd客户端域名解析

root@ceph-client-01:~# cat /etc/hosts
127.0.0.1       localhost
# The following lines are desirable for IPv6 capable hosts
::1     localhost       ip6-localhost   ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

172.16.10.225 rgw.wgs.com

7.6.3.2 配置s3cmd命令执行环境

点击查看代码
root@ceph-client-01:~# s3cmd --help
root@ceph-client-01:~# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: Q4XILSOF3BQ6MRBE2084 # 输入用户Access Key
Secret Key: Azbspmpx1A5IBARe6OVvjYfTKQsAlNSjzEOGgG76 # 输入用户Secret Key
Default Region [US]: # region选项

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: rgw.wgs.com:9900 #RGW域名

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: rgw.wgs.com:9900/%(bucket) #bucket域名格式

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: # 是否需要加密
Path to GPG program [/usr/bin/gpg]: # gpg路径

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No # 是否使用https

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: # http代理

New settings:
Access Key: Q4XILSOF3BQ6MRBE2084
Secret Key: Azbspmpx1A5IBARe6OVvjYfTKQsAlNSjzEOGgG76
Default Region: US
S3 Endpoint: rgw.wgs.com:9900
DNS-style bucket+hostname:port template for accessing a bucket: rgw.wgs.com:9900/%(bucket)
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y # 测试连接rgw是否正常
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine

标签:ceph,存储,s3,rgw,--,Ceph,mgr,default,RadosGW
来源: https://www.cnblogs.com/wangguishe/p/15340543.html