etcd 3.1 高可用集群搭建

2017-04-24

环境信息 #

CentOS 7.3

1192.168.61.11 node1
2192.168.61.12 node2
3192.168.61.13 node3

TLS密钥和证书 #

这里部署的etcd集群使用TLS证书对证书通信进行加密，并开启基于CA根证书签名的双向数字证书认证。

下面介绍使用cfssl生成所需要的私钥和证书.

安装cfssl #

cfssl是使用Go语言开发的工具，如果系统中安装了Go，可以使用直接go get安装cfssl:

1go get -u github.com/cloudflare/cfssl/cmd/...

会在$GOPATH/bin下安装cfssl, cfssjosn, mkbundle等工具。

CA证书和私钥 #

创建ca-config.json：

 1{
 2  "signing": {
 3    "default": {
 4      "expiry": "87600h"
 5    },
 6    "profiles": {
 7      "frognew": {
 8        "usages": [
 9            "signing",
10            "key encipherment",
11            "server auth",
12            "client auth"
13        ],
14        "expiry": "87600h"
15      }
16    }
17  }
18}

ca-config.json中可以定义多个profile，分别设置不同的expiry和usages等参数。如上面的ca-config.json中定义了名称为frognew的profile，这个profile的expiry 87600h为10年，useages中：

signing表示此CA证书可以用于签名其他证书，ca.pem中的CA=TRUE
server auth表示TLS Server Authentication
client auth表示TLS Client Authentication

创建CA证书签名请求配置ca-csr.json：

 1{
 2  "CN": "frognew",
 3  "key": {
 4    "algo": "rsa",
 5    "size": 2048
 6  },
 7  "names": [
 8    {
 9      "C": "CN",
10      "ST": "BeiJing",
11      "L": "BeiJing",
12      "O": "frognew",
13      "OU": "cloudnative"
14    }
15  ]
16}

下面使用cfss生成CA证书和私钥：

1cfssl gencert -initca ca-csr.json | cfssljson -bare ca

1ls
2ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

ca-key.pem和ca.pem需要保存在一个安全的地方，后边会用到。

etcd证书和私钥 #

创建etcd证书签名请求配置etcd-csr.json：

 1{
 2    "CN": "frognew",
 3    "hosts": [
 4      "127.0.0.1",
 5      "192.168.61.11",
 6      "192.168.61.12",
 7      "192.168.61.13",
 8      "node1",
 9      "node2",
10      "node3"
11    ],
12    "key": {
13        "algo": "rsa",
14        "size": 2048
15    },
16    "names": [
17        {
18            "C": "CN",
19            "ST": "BeiJing",
20            "L": "BeiJing",
21            "O": "frognew",
22            "OU": "cloudnative"
23        }
24    ]
25}

注意上面配置hosts字段中制定授权使用该证书的IP和域名列表，因为现在要生成的证书需要被etcd集群各个节点使用，所以这里指定了各个节点的IP和hostname。

下面生成etcd的证书和私钥：

1cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew etcd-csr.json | cfssljson -bare etcd
2
3ls etcd*
4etcd.csr  etcd-csr.json  etcd-key.pem  etcd.pem

对生成的证书可以使用cfssl或openssl查看：

1cfssl-certinfo -cert etcd.pem
2
3openssl x509  -noout -text -in  etcd.pem

安装etcd #

将CA证书ca.pem, etcd秘钥etcd-key.pem, etcd证书etcd.pem拷贝到各节点的/etc/etcd/ssl目录中。

下载etcd二进制文件包：

1wget https://github.com/coreos/etcd/releases/download/v3.1.6/etcd-v3.1.6-linux-amd64.tar.gz

解压缩etcd-v3.1.6-linux-amd64.tar.gz，将其中的etcd和etcdctl两个可执行文件复制到各节点的/usr/bin目录。

在各节点创建etcd的数据目录：

1mkdir -p /var/lib/etcd

在每个节点上创建etcd的systemd unit文件/usr/lib/systemd/system/etcd.service，注意替换ETCD_NAME和INTERNAL_IP变量的值：

 1export ETCD_NAME=node1
 2export INTERNAL_IP=192.168.61.11
 3cat > /usr/lib/systemd/system/etcd.service <<EOF
 4[Unit]
 5Description=etcd server
 6After=network.target
 7After=network-online.target
 8Wants=network-online.target
 9
10[Service]
11Type=notify
12WorkingDirectory=/var/lib/etcd/
13EnvironmentFile=-/etc/etcd/etcd.conf
14ExecStart=/usr/bin/etcd \
15  --name ${ETCD_NAME} \
16  --cert-file=/etc/etcd/ssl/etcd.pem \
17  --key-file=/etc/etcd/ssl/etcd-key.pem \
18  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
19  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
20  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
21  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
22  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \
23  --listen-peer-urls https://${INTERNAL_IP}:2380 \
24  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \
25  --advertise-client-urls https://${INTERNAL_IP}:2379 \
26  --initial-cluster-token etcd-cluster-1 \
27  --initial-cluster node1=https://192.168.61.11:2380,node2=https://192.168.61.12:2380,node3=https://192.168.61.13:2380 \
28  --initial-cluster-state new \
29  --data-dir=/var/lib/etcd
30Restart=on-failure
31RestartSec=5
32LimitNOFILE=65536
33
34[Install]
35WantedBy=multi-user.target
36EOF

上面在启动参数中指定了etcd的工作目录和数据目录是/var/lib/etcd
--cert-file和--key-file分别指定etcd的公钥证书和私钥
--peer-cert-file和--peer-key-file分别指定了etcd的Peers通信的公钥证书和私钥。
--trusted-ca-file指定了客户端的CA证书
--peer-trusted-ca-file指定了Peers的CA证书
--initial-cluster-state new表示这是新初始化集群，--name指定的参数值必须在--initial-cluster中

启动etcd #

在各节点上启动etcd：

1systemctl daemon-reload
2systemctl enable etcd
3systemctl start etcd
4systemctl status etcd

检查集群是否健康，在任一节点执行：

 1etcdctl \
 2  --ca-file=/etc/etcd/ssl/ca.pem \
 3  --cert-file=/etc/etcd/ssl/etcd.pem \
 4  --key-file=/etc/etcd/ssl/etcd-key.pem \
 5  --endpoints=https://node1:2379,https://node2:2379,https://node3:2379 \
 6  cluster-health
 7  
 82017-04-24 19:53:40.545148 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
 92017-04-24 19:53:40.546127 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
10member 4f2f99d70000fc19 is healthy: got healthy result from https://192.168.61.12:2379
11member 99a756f799eb4163 is healthy: got healthy result from https://192.168.61.11:2379
12member a9aff19397de2e4e is healthy: got healthy result from https://192.168.61.13:2379
13cluster is healthy

确保输出cluster is healthy的信息。

参考 #

Clustering Guide
[Generate self-signed certificates](Generate self-signed certificates)
cloudflare/cfssl