Ceph运维常用命令

2017-04-07 阅读: Ceph

1.集群管理

1.1 启动和停止各组件

查看当前节点的systemd unit服务:

systemctl list-units 'ceph*' --type=service
UNIT                           LOAD   ACTIVE SUB     DESCRIPTION
ceph-mgr@node1.service         loaded active running Ceph cluster manager daemon
ceph-mon@node1.service         loaded active running Ceph cluster monitor daemon
ceph-osd@0.service             loaded active running Ceph object storage daemon osd.0
ceph-radosgw@rgw.node1.service loaded active running Ceph rados gateway

查看某个服务是否开机启动:

systemctl is-enabled ceph-mon@node1.service
enabled

确认了相关的systemd unit后就可以使用systemd在Ceph集群各个节点上启动或停止各个Ceph组件。

1.2 查看集群状态

查看集群的概要状态:

ceph health
HEALTH_OK


ceph -s
    cluster 83d9e421-46bf-4d64-af15-af0e2c381b88
     health HEALTH_WARN
            clock skew detected on mon.node2
            Monitor clock skew detected
     monmap e2: 3 mons at {node1=192.168.61.41:6789/0,node2=192.168.61.42:6789/0,node3=192.168.61.43:6789/0}
            election epoch 114, quorum 0,1,2 node1,node2,node3
        mgr active: node2 standbys: node1, node3
     osdmap e138: 3 osds: 3 up, 3 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v9909: 152 pgs, 12 pools, 2686 bytes data, 214 objects
            133 MB used, 284 GB / 284 GB avail
                 152 active+clean

1.3 查看集群空间使用

ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    284G      284G         133M          0.05
POOLS:
    NAME                          ID     USED     %USED     MAX AVAIL     OBJECTS
    rbd                           0         0         0        97185M           0
    .rgw.root                     1      1681         0        97185M           4
    default.rgw.control           2         0         0        97185M           8
    default.rgw.data.root         3       603         0        97185M           2
    default.rgw.gc                4         0         0        97185M          32
    default.rgw.lc                5         0         0        97185M          32
    default.rgw.log               6         0         0        97185M         128
    default.rgw.users.uid         7       350         0        97185M           2
    default.rgw.users.email       8         8         0        97185M           1
    default.rgw.users.keys        9         8         0        97185M           1
    default.rgw.buckets.index     10        0         0        97185M           1
    default.rgw.buckets.data      11       36         0        97185M           3

查看每个OSD的存储空间

ceph osd df
ID WEIGHT  REWEIGHT SIZE  USE    AVAIL %USE VAR  PGS
 0 1.80170  1.00000 1844G 28903M 1816G 1.53 0.98 152
 1 1.80170  1.00000 1844G 29313M 1816G 1.55 1.00 152
 2 1.80170  1.00000 1844G 29873M 1815G 1.58 1.02 152
              TOTAL 5534G 88089M 5448G 1.55
MIN/MAX VAR: 0.98/1.02  STDDEV: 0.02

2.pool管理

2.1 删除pool

ceph osd pool delete poolname poolname --yes-i-really-really-mean-it

删除pool的命令比较危险,需要重复输入两次pool的名字,并且带上--yes-i-really-really-mean-it参数

3.MON节点命令

查看MON节点状态:

ceph quorum_status

{
  "election_epoch": 114,
  "quorum": [
    0,
    1,
    2
  ],
  "quorum_names": [
    "node1",
    "node2",
    "node3"
  ],
  "quorum_leader_name": "node1",
  "monmap": {
    "epoch": 2,
    "fsid": "83d9e421-46bf-4d64-af15-af0e2c381b88",
    "modified": "2017-04-06 19:52:00.882973",
    "created": "2017-04-06 19:51:47.569731",
    "features": {
      "persistent": [
        "kraken"
      ],
      "optional": []
    },
    "mons": [
      {
        "rank": 0,
        "name": "node1",
        "addr": "192.168.61.41:6789/0",
        "public_addr": "192.168.61.41:6789/0"
      },
      {
        "rank": 1,
        "name": "node2",
        "addr": "192.168.61.42:6789/0",
        "public_addr": "192.168.61.42:6789/0"
      },
      {
        "rank": 2,
        "name": "node3",
        "addr": "192.168.61.43:6789/0",
        "public_addr": "192.168.61.43:6789/0"
      }
    ]
  }
}

MON使用Paxos算法进行选举,上面的输出中election_epoch表示一共进行的投票轮次数量,quorumquorum_names表示参与投票者的编号和名称,quorum_leader_name表示当前的leader名称,rank表示每个MON节点的权重,权重越小在选举时月容易得到支持。

4.OSD节点命令

4.1 查看OSD节点状态

ceph osd stat
     osdmap e138: 3 osds: 3 up, 3 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds

使用ceph osd dump可查看osd节点的详细信息。

4.2 查看OSD节点的分布

ceph osd tree
ID WEIGHT  TYPE NAME      UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.27809 root default
-2 0.09270     host node1
 0 0.09270         osd.0       up  1.00000          1.00000
-3 0.09270     host node2
 1 0.09270         osd.1       up  1.00000          1.00000
-4 0.09270     host node3
 2 0.09270         osd.2       up  1.00000          1.00000

5.rbd镜像命令

5.1 删除rbd镜像

删除RBD镜像的命令是rbd rm {pool-name}/{image-name}

rbd rm kube/kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175

删除时报错:

......
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

查看正在使用的客户端:

rados -p kube listwatchers kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175.rbd
watcher=192.168.1.4:0/3418766042 client.284106 cookie=75

到客户端所在主机取消内核映射:

rbd showmapped | grep ubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175
3  kube kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175 -    /dev/rbd3

rbd unmap /dev/rbd3

再次删除:

rados -p kube listwatchers kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175.rbd
rbd rm kube/kubernetes-dynamic-pvc-d6f0802e-9dd5-11e7-a66f-1866da8c6175

参考

标题:Ceph运维常用命令
本文链接:https://blog.frognew.com/2017/04/ceph-command.html
转载请注明出处。

目录