Ceph对象存储之RGW

2017-02-05 阅读: Ceph

Ceph RGW简介

Ceph RGW(即RADOS Gateway)是Ceph对象存储网关服务,是基于LIBRADOS接口封装实现的FastCGI服务,对外提供存储和管理对象数据的Restful API。 对象存储适用于图片、视频等各类文件的上传下载,可以设置相应的访问权限。目前Ceph RGW兼容常见的对象存储API,例如兼容绝大部分Amazon S3 API,兼容OpenStack Swift API。

Ceph RGW

部署Ceph RGW

Ceph RGW的FastCGI支持多种Web服务器作为前端,例如Nginx、Apache2等。 从Ceph Hammer版本开始,使用ceph-deploy部署时将会默认使用内置的civetweb作为前端。

这里选择和我们线上环境一致的nginx部署,继续使用前面的实验环境:

192.168.61.30 c0 - admin-node, deploy-node
192.168.61.31 c1 - mon
192.168.61.32 c2 - mon osd.1
192.168.61.33 c3 - mon osd.2

可以把Ceph RGW简单理解成Ceph集群的一个客户端,用户通过它间接的访问Ceph集群,因此我们选择先在实验环境的c1服务器上部署一个RGW节点。

192.168.61.30 c0 - admin-node, deploy-node
192.168.61.31 c1 - mon radosgw nginx
192.168.61.32 c2 - mon osd.1
192.168.61.33 c3 - mon osd.2

安装ceph-radosgw和nginx

在c1服务器上安装ceph-radosgw:

sudo yum install -y ceph-radosgw

在c1服务器上安装nginx,这里采用npm方式安装:

/etc/yum.repos.d/nginx.repo

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/x86_64/
gpgcheck=0
enabled=1
sudo yum install -y nginx

启动nginx,并设置开机启动:

sudo systemctl start nginx
sudo systemctl enable nginx

创建RGW用户和keyring

在c1服务器上创建keyring:

sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring

生成ceph-radosgw服务对应的用户和key:

sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key

为用户添加访问权限:

sudo ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring

导入keyring到集群中:

sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring

创建资源池

由于RGW要求专门的pool存储数据,这里手动创建这些Pool,在admin-node上执行:

ceph osd pool create .rgw 128 128
ceph osd pool create .rgw.root 128 128
ceph osd pool create .rgw.control 128 128
ceph osd pool create .rgw.gc 128 128
ceph osd pool create .rgw.buckets 128 128
ceph osd pool create .rgw.buckets.index 128 128
ceph osd pool create .rgw.buckets.extra 128 128
ceph osd pool create .log 128 128
ceph osd pool create .intent-log 128 128
ceph osd pool create .usage 128 128
ceph osd pool create .users 128 128
ceph osd pool create .users.email 128 128
ceph osd pool create .users.swift 128 128
ceph osd pool create .users.uid 128 128

列出pool信息确认全部成功创建:

rados lspools
......
.rgw
.rgw.root
.rgw.control
.rgw.gc
.rgw.buckets
.rgw.buckets.index
.rgw.buckets.extra
.log
.intent-log
.usage
.users
.users.email
.users.swift
.users.uid

RGW配置

在c1服务器的/etc/ceph.conf上添加以下内容:

[client.radosgw.gateway]
rgw frontends=fastcgi socket_port=9000 socket_host=0.0.0.0
host=c1
keyring=/etc/ceph/ceph.client.radosgw.keyring
log file=/var/log/radosgw/client.radosgw.gateway.log
rgw print continue=false
rgw content length compat = true

更多配置查阅CEPH OBJECT GATEWAY CONFIG REFERENCE

nginx配置

在/etc/nginx/nginx.conf中http段落中添加如下内容:


server {
    listen   80 default;
    server_name c1;
    location / {
        fastcgi_pass_header Authorization;
        fastcgi_pass_request_headers on;
        fastcgi_param QUERY_STRING  $query_string;
        fastcgi_param REQUEST_METHOD $request_method;
        fastcgi_param CONTENT_LENGTH $content_length;
        fastcgi_param CONTENT_LENGTH $content_length;
        if ($request_method = PUT) {
                rewrite ^ /PUT$request_uri;
        }
        include fastcgi_params;
        fastcgi_pass 127.0.0.1:9000;
    }
    location /PUT/ {
        internal;
        fastcgi_pass_header Authorization;
        fastcgi_pass_request_headers on;
        include fastcgi_params;
        fastcgi_param QUERY_STRING  $query_string;
        fastcgi_param REQUEST_METHOD $request_method;
        fastcgi_param CONTENT_LENGTH $content_length;
        fastcgi_param  CONTENT_TYPE $content_type;
        fastcgi_pass 127.0.0.1:9000;
    }
}

启动RGW

sudo mkdir /var/log/radosgw
sudo chown ceph:ceph /var/log/radosgw

sudo cp /usr/lib/systemd/system/ceph-radosgw@.service /usr/lib/systemd/system/ceph-radosgw@radosgw.gateway.service 
sudo systemctl start ceph-radosgw@radosgw.gateway
sudo systemctl enable ceph-radosgw@radosgw.gateway

查看radosgw服务状态:

sudo systemctl status ceph-radosgw@radosgw.gateway
....
[/usr/lib/systemd/system/ceph-radosgw@radosgw.gateway.service:17] Unknown lvalue 'TasksMax' in section 'Service'

报了Unknown lvalue ‘TasksMax’ in section ‘Service’的错误,升级systemd:

sudo yum install systemd-*

重启再次查看状态,没有问题了:

sudo systemctl restart ceph-radosgw@radosgw.gateway
sudo systemctl status ceph-radosgw@radosgw.gateway
ps -ef | grep radosgw
ceph      3344     1  1 22:41 ?        00:00:00 /usr/bin/radosgw -f --cluster ceph --name client.radosgw.gateway --setuser ceph --setgroup ceph

restart nginx服务:

sudo systemctl restart nginx

使用curl服务状态,如果出现下面的内容则服务正常:

curl c1
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

使用radosgw-admin管理RGW

创建S3用户

radosgw-admin是RGW服务的命令行管理工具。我们已经知道了RGW兼容绝大部分Amazon S3 API,下面我们先使用radosgw-admin来创建一个S3用户:

radosgw-admin user create --uid=foo --display-name=foo --email=foo@foo.com
{
    "user_id": "foo",
    "display_name": "foo",
    "email": "foo@foo.com",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "foo",
            "access_key": "W0FE3UJ375EAHMT0660A",
            "secret_key": "HeHbboOIfpc6N1lAxlCmW0S69owWBCBRQnL4BI6m"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

注意到上面的命令输出中的access_key和secret_key:

"user": "foo",
"access_key": "W0FE3UJ375EAHMT0660A",
"secret_key": "HeHbboOIfpc6N1lAxlCmW0S69owWBCBRQnL4BI6m"

使用S3 API需要使用access_key和secret_key。access_key用于标识客户端身份;secret_key作为私钥保存在客户端服务器,不会在网络中传输,通常用于作为计算请求签名的密钥。 使用access_key进行身份识别,使用secret_key进行签名,完成客户端的接入、认证和授权。

删除S3用户

radosgw-admin  user rm --uid=foo

使用Admin Ops REST接口管理RGW

Admin OPERATIONS是RGW提供的一套REST接口,可以用来管理S3用户、Bucket、配额等信息,具体可查看ADMIN OPERATIONS

为了使用这套接口,需要创建一个s3用户,并针对不同的接口赋予需要的权限。

这里创建一个admin用户:

radosgw-admin user create --uid=admin --display-name=admin
{
    "user_id": "admin",
    "display_name": "admin",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "admin",
            "access_key": "IQDZQF92L8E9YT8YNUY3",
            "secret_key": "BLUAVMwCMWGdINXXme6lETgjePqB8gcuYnJQ7XGN"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

允许admin读写users信息:

radosgw-admin caps add --uid=admin --caps="users=*"

允许admin读写所有的usage信息:

radosgw-admin caps add --uid=admin --caps="usage=read,write"

参考

标题:Ceph对象存储之RGW
本文链接:https://blog.frognew.com/2017/02/ceph-rgw.html
转载请注明出处。

目录