重学容器21: 手动启动containerd容器并连接到calico网络
2021-07-08
前面一节完成了calico node集群的部署，本节将在各个节点上部署calico cni插件， 并在两个主机节点上各启动一个连接到calico网络的containerd容器，测试两个容器网络是否是打通的。
部署容器运行时Containerd和CNI #
在各个服务器节点上安装容器运行时Containerd。
创建/etc/modules-load.d/containerd.conf配置文件:
1cat << EOF > /etc/modules-load.d/containerd.conf
2overlay
3br_netfilter
4EOF
执行以下命令使配置生效:
1modprobe overlay
2modprobe br_netfilter
创建/etc/sysctl.d/containerd.conf配置文件：
1cat << EOF > /etc/sysctl.d/containerd.conf
2net.bridge.bridge-nf-call-ip6tables = 1
3net.bridge.bridge-nf-call-iptables = 1
4net.ipv4.ip_forward = 1
5user.max_user_namespaces=28633
6EOF
执行以下命令使配置生效:
1sysctl -p /etc/sysctl.d/containerd.conf
下载Containerd的二进制包:
1wget https://github.com/containerd/containerd/releases/download/v1.5.2/cri-containerd-cni-1.5.2-linux-amd64.tar.gz
cri-containerd-cni-1.5.2-linux-amd64.tar.gz压缩包中已经按照官方二进制部署推荐的目录结构布局好。
里面包含了systemd配置文件，containerd以及cni的部署文件。
将解压缩到系统的根目录/中:
1tar -zxvf cri-containerd-cni-1.5.2-linux-amd64.tar.gz -C /
2
3etc/
4etc/systemd/
5etc/systemd/system/
6etc/systemd/system/containerd.service
7etc/crictl.yaml
8etc/cni/
9etc/cni/net.d/
10etc/cni/net.d/10-containerd-net.conflist
11usr/
12usr/local/
13usr/local/sbin/
14usr/local/sbin/runc
15usr/local/bin/
16usr/local/bin/critest
17usr/local/bin/containerd-shim
18usr/local/bin/containerd-shim-runc-v1
19usr/local/bin/ctd-decoder
20usr/local/bin/containerd
21usr/local/bin/containerd-shim-runc-v2
22usr/local/bin/containerd-stress
23usr/local/bin/ctr
24usr/local/bin/crictl
25......
26opt/cni/
27opt/cni/bin/
28opt/cni/bin/bridge
29......
接下来生成containerd的配置文件:
1mkdir -p /etc/containerd
2containerd config default > /etc/containerd/config.toml
根据文档Container runtimes 中的内容，对于使用systemd作为init system的Linux的发行版，使用systemd作为容器的cgroup driver可以确保服务器节点在资源紧张的情况更加稳定，因此这里配置各个节点上containerd的cgroup driver为systemd。
修改前面生成的配置文件/etc/containerd/config.toml：
1[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
2 ...
3 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
4 SystemdCgroup = true
再修改/etc/containerd/config.toml中的
1[plugins."io.containerd.grpc.v1.cri"]
2 ...
3 # sandbox_image = "k8s.gcr.io/pause:3.5"
4 sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.4.1"
配置containerd开机启动，并启动containerd
1systemctl enable containerd
2systemctl start containerd
使用crictl测试一下，确保可以打印出版本信息并且没有错误信息输出:
1crictl version
2Version: 0.1.0
3RuntimeName: containerd
4RuntimeVersion: v1.5.2
5RuntimeApiVersion: v1alpha2
安装containerd的nerdctl客户端:
1wget https://github.com/containerd/nerdctl/releases/download/v0.10.0/nerdctl-0.10.0-linux-amd64.tar.gz
2tar -zxvf nerdctl-0.10.0-linux-amd64.tar.gz nerdctl && mv nerdctl /usr/local/bin/
3
4nerdctl version
5Client:
6 Version: v0.10.0
7 Git commit: ad836c2f0f53084a762112be58379d8d795812a5
8
9Server:
10 containerd:
11 Version: v1.5.2
12 Revision: 36cc874494a56a253cd181a1a685b44b58a2e34a
部署calico cni plugin #
在各个节点上下载calico cni plugin二进制文件到/opt/cni/bin/:
1cd /opt/cni/bin/
2curl -o calico -O -L "https://github.com/projectcalico/cni-plugin/releases/download/v3.19.1/calico-amd64"
3curl -o calico-ipam -O -L "https://github.com/projectcalico/cni-plugin/releases/download/v3.19.1/calico-ipam-amd64"
4chmod +x calico
5chmod +x calico-ipam
在各个节点上创建calico cni网络配置列表配置文件:
1cat > /etc/cni/net.d/10-calico-containerd.conflist << EOF
2{
3 "name": "calico-containerd-network",
4 "cniVersion": "0.3.1",
5 "plugins": [
6 {
7 "type": "calico",
8 "log_level": "info",
9 "log_file_path": "/var/log/calico/cni/cni.log",
10 "datastore_type": "etcdv3",
11 "etcd_endpoints": "https://192.168.100.101:2379,https://192.168.100.102:2379,https://192.168.100.103:2379",
12 "etcd_key_file": "/etc/calico/ssl/client.key",
13 "etcd_cert_file": "/etc/calico/ssl/client.crt",
14 "etcd_ca_cert_file": "/etc/calico/ssl/ca.crt",
15 "nodename": "$HOSTNAME",
16 "mtu": 0,
17 "ipam": {
18 "type": "calico-ipam"
19 },
20 "container_settings": {
21 "allow_ip_forwarding": true
22 }
23 },
24 {
25 "type": "portmap",
26 "snat": true,
27 "capabilities": {"portMappings": true}
28 },
29 {
30 "type": "bandwidth",
31 "capabilities": {"bandwidth": true}
32 }
33 ]
34}
35EOF
使用nerdctl打印下可以连接的网络:
1nerdctl network ls
2NETWORK ID NAME FILE
30 bridge
4 calico-containerd-network /etc/cni/net.d/10-calico-containerd.conflist
5 host
6 none
启动containerd容器并连接到calico网络 #
calico的默认网络策略是各节点不通的，为了测试简单创建下面的全部允许的全局网络策略:
1calicoctl apply -f - <<EOF
2apiVersion: projectcalico.org/v3
3kind: GlobalNetworkPolicy
4metadata:
5 name: allow-all
6spec:
7 selector: all()
8 ingress:
9 - action: Allow
10 egress:
11 - action: Allow
12EOF
在node1, node2上执行下面的命令，各启动一个redis容器连接到calico-containerd-network：
1nerdctl run -d --name redis --network calico-containerd-network redis:alpine3.13
calicoctl查看IPPool确实分配了2个IP出去:
1calicoctl ipam show
2+----------+--------------+-----------+------------+--------------+
3| GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE |
4+----------+--------------+-----------+------------+--------------+
5| IP Pool | 10.89.0.0/16 | 65536 | 2 (0%) | 65534 (100%) |
6+----------+--------------+-----------+------------+--------------+
通过使用calicoctl查看WorkloadEndpoint资源可以查看容器ip:
1calicoctl get WorkloadEndpoint
2WORKLOAD NODE NETWORKS INTERFACE
3 node1 10.89.3.66/32 calidefault-9f6
4 node2 10.89.33.130/32 calidefault-f4d
在node1上使用nerdctl exec到redis容器，使用这个容器内的redis-cil可以连接上node2上ip为10.89.33.130的redis容器
1[root@node1 ~]# nerdctl exec -it redis sh
2/data # ifconfig
3eth0 Link encap:Ethernet HWaddr C2:3D:23:E0:40:03
4 inet addr:10.89.3.66 Bcast:10.89.3.66 Mask:255.255.255.255
5 inet6 addr: fe80::c03d:23ff:fee0:4003/64 Scope:Link
6 UP BROADCAST RUNNING MULTICAST MTU:1480 Metric:1
7 RX packets:85 errors:0 dropped:0 overruns:0 frame:0
8 TX packets:43 errors:0 dropped:0 overruns:0 carrier:0
9 collisions:0 txqueuelen:0
10 RX bytes:26298 (25.6 KiB) TX bytes:3055 (2.9 KiB)
11
12lo Link encap:Local Loopback
13 inet addr:127.0.0.1 Mask:255.0.0.0
14 inet6 addr: ::1/128 Scope:Host
15 UP LOOPBACK RUNNING MTU:65536 Metric:1
16 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
17 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
18 collisions:0 txqueuelen:1000
19 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
20
21/data # redis-cli -h 10.89.33.130
2210.89.33.130:6379>
说明容器间跨主机通信网络已经打通。
总结 #
本节完成了手动启动containerd容器，通过calico cni插件配置容器使用calico网络，进一步理解了容器运行时与容器网络接口CNI之间的关系。