前面一节完成了calico node集群的部署，本节将在各个节点上部署calico cni插件，
并在两个主机节点上各启动一个连接到calico网络的containerd容器，测试两个容器网络是否是打通的。
部署容器运行时Containerd和CNI
在各个服务器节点上安装容器运行时Containerd。
创建/etc/modules-load.d/containerd.conf配置文件:
1
2
3
4
|
cat << EOF > /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
|
执行以下命令使配置生效:
1
2
|
modprobe overlay
modprobe br_netfilter
|
创建/etc/sysctl.d/containerd.conf配置文件：
1
2
3
4
5
6
|
cat << EOF > /etc/sysctl.d/containerd.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
user.max_user_namespaces=28633
EOF
|
执行以下命令使配置生效:
1
|
sysctl -p /etc/sysctl.d/containerd.conf
|
下载Containerd的二进制包:
1
|
wget https://github.com/containerd/containerd/releases/download/v1.5.2/cri-containerd-cni-1.5.2-linux-amd64.tar.gz
|
cri-containerd-cni-1.5.2-linux-amd64.tar.gz压缩包中已经按照官方二进制部署推荐的目录结构布局好。
里面包含了systemd配置文件，containerd以及cni的部署文件。
将解压缩到系统的根目录/中:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
tar -zxvf cri-containerd-cni-1.5.2-linux-amd64.tar.gz -C /
etc/
etc/systemd/
etc/systemd/system/
etc/systemd/system/containerd.service
etc/crictl.yaml
etc/cni/
etc/cni/net.d/
etc/cni/net.d/10-containerd-net.conflist
usr/
usr/local/
usr/local/sbin/
usr/local/sbin/runc
usr/local/bin/
usr/local/bin/critest
usr/local/bin/containerd-shim
usr/local/bin/containerd-shim-runc-v1
usr/local/bin/ctd-decoder
usr/local/bin/containerd
usr/local/bin/containerd-shim-runc-v2
usr/local/bin/containerd-stress
usr/local/bin/ctr
usr/local/bin/crictl
......
opt/cni/
opt/cni/bin/
opt/cni/bin/bridge
......
|
接下来生成containerd的配置文件:
1
2
|
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
|
根据文档Container runtimes
中的内容，对于使用systemd作为init system的Linux的发行版，使用systemd作为容器的cgroup driver可以确保服务器节点在资源紧张的情况更加稳定，因此这里配置各个节点上containerd的cgroup driver为systemd。
修改前面生成的配置文件/etc/containerd/config.toml：
1
2
3
4
|
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
|
再修改/etc/containerd/config.toml中的
1
2
3
4
|
[plugins."io.containerd.grpc.v1.cri"]
...
# sandbox_image = "k8s.gcr.io/pause:3.5"
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.4.1"
|
配置containerd开机启动，并启动containerd
1
2
|
systemctl enable containerd
systemctl start containerd
|
使用crictl测试一下，确保可以打印出版本信息并且没有错误信息输出:
1
2
3
4
5
|
crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: v1.5.2
RuntimeApiVersion: v1alpha2
|
安装containerd的nerdctl客户端:
1
2
3
4
5
6
7
8
9
10
11
12
|
wget https://github.com/containerd/nerdctl/releases/download/v0.10.0/nerdctl-0.10.0-linux-amd64.tar.gz
tar -zxvf nerdctl-0.10.0-linux-amd64.tar.gz nerdctl && mv nerdctl /usr/local/bin/
nerdctl version
Client:
Version: v0.10.0
Git commit: ad836c2f0f53084a762112be58379d8d795812a5
Server:
containerd:
Version: v1.5.2
Revision: 36cc874494a56a253cd181a1a685b44b58a2e34a
|
部署calico cni plugin
在各个节点上下载calico cni plugin二进制文件到/opt/cni/bin/:
1
2
3
4
5
|
cd /opt/cni/bin/
curl -o calico -O -L "https://github.com/projectcalico/cni-plugin/releases/download/v3.19.1/calico-amd64"
curl -o calico-ipam -O -L "https://github.com/projectcalico/cni-plugin/releases/download/v3.19.1/calico-ipam-amd64"
chmod +x calico
chmod +x calico-ipam
|
在各个节点上创建calico cni网络配置列表配置文件:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
cat > /etc/cni/net.d/10-calico-containerd.conflist << EOF
{
"name": "calico-containerd-network",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "calico",
"log_level": "info",
"log_file_path": "/var/log/calico/cni/cni.log",
"datastore_type": "etcdv3",
"etcd_endpoints": "https://192.168.100.101:2379,https://192.168.100.102:2379,https://192.168.100.103:2379",
"etcd_key_file": "/etc/calico/ssl/client.key",
"etcd_cert_file": "/etc/calico/ssl/client.crt",
"etcd_ca_cert_file": "/etc/calico/ssl/ca.crt",
"nodename": "$HOSTNAME",
"mtu": 0,
"ipam": {
"type": "calico-ipam"
},
"container_settings": {
"allow_ip_forwarding": true
}
},
{
"type": "portmap",
"snat": true,
"capabilities": {"portMappings": true}
},
{
"type": "bandwidth",
"capabilities": {"bandwidth": true}
}
]
}
EOF
|
使用nerdctl打印下可以连接的网络:
1
2
3
4
5
6
|
nerdctl network ls
NETWORK ID NAME FILE
0 bridge
calico-containerd-network /etc/cni/net.d/10-calico-containerd.conflist
host
none
|
启动containerd容器并连接到calico网络
calico的默认网络策略是各节点不通的，为了测试简单创建下面的全部允许的全局网络策略:
1
2
3
4
5
6
7
8
9
10
11
12
|
calicoctl apply -f - <<EOF
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: allow-all
spec:
selector: all()
ingress:
- action: Allow
egress:
- action: Allow
EOF
|
在node1, node2上执行下面的命令，各启动一个redis容器连接到calico-containerd-network：
1
|
nerdctl run -d --name redis --network calico-containerd-network redis:alpine3.13
|
calicoctl查看IPPool确实分配了2个IP出去:
1
2
3
4
5
6
|
calicoctl ipam show
+----------+--------------+-----------+------------+--------------+
| GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE |
+----------+--------------+-----------+------------+--------------+
| IP Pool | 10.89.0.0/16 | 65536 | 2 (0%) | 65534 (100%) |
+----------+--------------+-----------+------------+--------------+
|
通过使用calicoctl查看WorkloadEndpoint资源可以查看容器ip:
1
2
3
4
|
calicoctl get WorkloadEndpoint
WORKLOAD NODE NETWORKS INTERFACE
node1 10.89.3.66/32 calidefault-9f6
node2 10.89.33.130/32 calidefault-f4d
|
在node1上使用nerdctl exec到redis容器，使用这个容器内的redis-cil可以连接上node2上ip为10.89.33.130的redis容器
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
[root@node1 ~]# nerdctl exec -it redis sh
/data # ifconfig
eth0 Link encap:Ethernet HWaddr C2:3D:23:E0:40:03
inet addr:10.89.3.66 Bcast:10.89.3.66 Mask:255.255.255.255
inet6 addr: fe80::c03d:23ff:fee0:4003/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1480 Metric:1
RX packets:85 errors:0 dropped:0 overruns:0 frame:0
TX packets:43 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:26298 (25.6 KiB) TX bytes:3055 (2.9 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/data # redis-cli -h 10.89.33.130
10.89.33.130:6379>
|
说明容器间跨主机通信网络已经打通。
总结
本节完成了手动启动containerd容器，通过calico cni插件配置容器使用calico网络，进一步理解了容器运行时与容器网络接口CNI之间的关系。
参考
次阅读