1.集群管理
1.1 启动和停止各组件
查看当前节点的systemd unit服务：
systemctl list-units 'ceph*' --type=service
UNIT LOAD ACTIVE SUB DESCRIPTION
[email protected] loaded active running Ceph cluster manager daemon
[email protected] loaded active running Ceph cluster monitor daemon
[email protected] loaded active running Ceph object storage daemon osd.0
[email protected] loaded active running Ceph rados gateway查看某个服务是否开机启动：
systemctl is-enabled [email protected]
enabled确认了相关的systemd unit后就可以使用systemd在Ceph集群各个节点上启动或停止各个Ceph组件。
1.2 查看集群状态
查看集群的概要状态：
ceph health
HEALTH_OK
ceph -s
cluster 83d9e421-46bf-4d64-af15-af0e2c381b88
health HEALTH_WARN
clock skew detected on mon.node2
Monitor clock skew detected
monmap e2: 3 mons at {node1=192.168.61.41:6789/0,node2=192.168.61.42:6789/0,node3=192.168.61.43:6789/0}
election epoch 114, quorum 0,1,2 node1,node2,node3
mgr active: node2 standbys: node1, node3
osdmap e138: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds,require_kraken_osds
pgmap v9909: 152 pgs, 12 pools, 2686 bytes data, 214 objects
133 MB used, 284 GB / 284 GB avail
152 active+clean1.3 查看集群空间使用
ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
284G 284G 133M 0.05
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 97185M 0
.rgw.root 1 1681 0 97185M 4
default.rgw.control 2 0 0 97185M 8
default.rgw.data.root 3 603 0 97185M 2
default.rgw.gc 4 0 0 97185M 32
default.rgw.lc 5 0 0 97185M 32
default.rgw.log 6 0 0 97185M 128
default.rgw.users.uid 7 350 0 97185M 2
default.rgw.users.email 8 8 0 97185M 1
default.rgw.users.keys 9 8 0 97185M 1
default.rgw.buckets.index 10 0 0 97185M 1
default.rgw.buckets.data 11 36 0 97185M 3查看每个OSD的存储空间
ceph osd df
ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 1.80170 1.00000 1844G 28903M 1816G 1.53 0.98 152
1 1.80170 1.00000 1844G 29313M 1816G 1.55 1.00 152
2 1.80170 1.00000 1844G 29873M 1815G 1.58 1.02 152
TOTAL 5534G 88089M 5448G 1.55
MIN/MAX VAR: 0.98/1.02 STDDEV: 0.022.pool管理
2.1 删除pool
ceph osd pool delete poolname poolname --yes-i-really-really-mean-it删除pool的命令比较危险，需要重复输入两次pool的名字，并且带上--yes-i-really-really-mean-it参数
2.2 设置Pool的配额
ceph osd pool set-quota cephfs_data max_bytes $((100 * 1024 * 1024 * 1024))
ceph osd pool get-quota cephfs_data
quotas for pool 'cephfs_data':
max objects: N/A
max bytes : 100GiB取消配额限制只需要把对应值设为0即可。
3.MON节点命令
查看MON节点状态：
ceph quorum_status
{
"election_epoch": 114,
"quorum": [
0,
1,
2
],
"quorum_names": [
"node1",
"node2",
"node3"
],
"quorum_leader_name": "node1",
"monmap": {
"epoch": 2,
"fsid": "83d9e421-46bf-4d64-af15-af0e2c381b88",
"modified": "2017-04-06 19:52:00.882973",
"created": "2017-04-06 19:51:47.569731",
"features": {
"persistent": [
"kraken"
],
"optional": []
},
"mons": [
{
"rank": 0,
"name": "node1",
"addr": "192.168.61.41:6789/0",
"public_addr": "192.168.61.41:6789/0"
},
{
"rank": 1,
"name": "node2",
"addr": "192.168.61.42:6789/0",
"public_addr": "192.168.61.42:6789/0"
},
{
"rank": 2,
"name": "node3",
"addr": "192.168.61.43:6789/0",
"public_addr": "192.168.61.43:6789/0"
}
]
}
}MON使用Paxos算法进行选举，上面的输出中election_epoch表示一共进行的投票轮次数量，quorum和quorum_names表示参与投票者的编号和名称，quorum_leader_name表示当前的leader名称，rank表示每个MON节点的权重，权重越小在选举时月容易得到支持。
4.OSD节点命令
4.1 查看OSD节点状态
ceph osd stat
osdmap e138: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds,require_kraken_osds使用ceph osd dump可查看osd节点的详细信息。
4.2 查看OSD节点的分布
ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.27809 root default
-2 0.09270 host node1
0 0.09270 osd.0 up 1.00000 1.00000
-3 0.09270 host node2
1 0.09270 osd.1 up 1.00000 1.00000
-4 0.09270 host node3
2 0.09270 osd.2 up 1.00000 1.000005.rbd镜像命令
5.1 删除rbd镜像
删除RBD镜像的命令是rbd rm {pool-name}/{image-name}
rbd rm kube/kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175删除时报错：
......
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.查看正在使用的客户端：
rados -p kube listwatchers kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175.rbd
watcher=192.168.1.4:0/3418766042 client.284106 cookie=75到客户端所在主机取消内核映射：
rbd showmapped | grep ubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175
3 kube kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175 - /dev/rbd3
rbd unmap /dev/rbd3再次删除：
rados -p kube listwatchers kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175.rbd
rbd rm kube/kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175