前面一节完成了calico node集群的部署,本节将在各个节点上部署calico cni插件, 并在两个主机节点上各启动一个连接到calico网络的containerd容器,测试两个容器网络是否是打通的。

部署容器运行时Containerd和CNI

在各个服务器节点上安装容器运行时Containerd。

创建/etc/modules-load.d/containerd.conf配置文件:

1cat << EOF > /etc/modules-load.d/containerd.conf
2overlay
3br_netfilter
4EOF

执行以下命令使配置生效:

1modprobe overlay
2modprobe br_netfilter

创建/etc/sysctl.d/containerd.conf配置文件:

1cat << EOF > /etc/sysctl.d/containerd.conf
2net.bridge.bridge-nf-call-ip6tables = 1
3net.bridge.bridge-nf-call-iptables = 1
4net.ipv4.ip_forward = 1
5user.max_user_namespaces=28633
6EOF

执行以下命令使配置生效:

1sysctl -p /etc/sysctl.d/containerd.conf

下载Containerd的二进制包:

1wget https://github.com/containerd/containerd/releases/download/v1.5.2/cri-containerd-cni-1.5.2-linux-amd64.tar.gz

cri-containerd-cni-1.5.2-linux-amd64.tar.gz压缩包中已经按照官方二进制部署推荐的目录结构布局好。 里面包含了systemd配置文件,containerd以及cni的部署文件。 将解压缩到系统的根目录/中:

 1tar -zxvf cri-containerd-cni-1.5.2-linux-amd64.tar.gz -C /
 2
 3etc/
 4etc/systemd/
 5etc/systemd/system/
 6etc/systemd/system/containerd.service
 7etc/crictl.yaml
 8etc/cni/
 9etc/cni/net.d/
10etc/cni/net.d/10-containerd-net.conflist
11usr/
12usr/local/
13usr/local/sbin/
14usr/local/sbin/runc
15usr/local/bin/
16usr/local/bin/critest
17usr/local/bin/containerd-shim
18usr/local/bin/containerd-shim-runc-v1
19usr/local/bin/ctd-decoder
20usr/local/bin/containerd
21usr/local/bin/containerd-shim-runc-v2
22usr/local/bin/containerd-stress
23usr/local/bin/ctr
24usr/local/bin/crictl
25......
26opt/cni/
27opt/cni/bin/
28opt/cni/bin/bridge
29......

接下来生成containerd的配置文件:

1mkdir -p /etc/containerd
2containerd config default > /etc/containerd/config.toml

根据文档Container runtimes 中的内容,对于使用systemd作为init system的Linux的发行版,使用systemd作为容器的cgroup driver可以确保服务器节点在资源紧张的情况更加稳定,因此这里配置各个节点上containerd的cgroup driver为systemd。

修改前面生成的配置文件/etc/containerd/config.toml

1[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
2  ...
3  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
4    SystemdCgroup = true

再修改/etc/containerd/config.toml中的

1[plugins."io.containerd.grpc.v1.cri"]
2  ...
3  # sandbox_image = "k8s.gcr.io/pause:3.5"
4  sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.4.1"

配置containerd开机启动,并启动containerd

1systemctl enable containerd
2systemctl start containerd

使用crictl测试一下,确保可以打印出版本信息并且没有错误信息输出:

1crictl version
2Version:  0.1.0
3RuntimeName:  containerd
4RuntimeVersion:  v1.5.2
5RuntimeApiVersion:  v1alpha2

安装containerd的nerdctl客户端:

 1wget https://github.com/containerd/nerdctl/releases/download/v0.10.0/nerdctl-0.10.0-linux-amd64.tar.gz
 2tar -zxvf nerdctl-0.10.0-linux-amd64.tar.gz nerdctl && mv nerdctl /usr/local/bin/
 3
 4nerdctl version
 5Client:
 6 Version:	v0.10.0
 7 Git commit:	ad836c2f0f53084a762112be58379d8d795812a5
 8
 9Server:
10 containerd:
11  Version:	v1.5.2
12  Revision:	36cc874494a56a253cd181a1a685b44b58a2e34a

部署calico cni plugin

在各个节点上下载calico cni plugin二进制文件到/opt/cni/bin/:

1cd /opt/cni/bin/
2curl -o calico -O -L  "https://github.com/projectcalico/cni-plugin/releases/download/v3.19.1/calico-amd64" 
3curl -o calico-ipam -O -L "https://github.com/projectcalico/cni-plugin/releases/download/v3.19.1/calico-ipam-amd64"
4chmod +x calico
5chmod +x calico-ipam

在各个节点上创建calico cni网络配置列表配置文件:

 1cat > /etc/cni/net.d/10-calico-containerd.conflist << EOF
 2{
 3  "name": "calico-containerd-network",
 4  "cniVersion": "0.3.1",
 5  "plugins": [
 6    {
 7      "type": "calico",
 8      "log_level": "info",
 9      "log_file_path": "/var/log/calico/cni/cni.log",
10      "datastore_type": "etcdv3",
11      "etcd_endpoints": "https://192.168.100.101:2379,https://192.168.100.102:2379,https://192.168.100.103:2379",
12      "etcd_key_file": "/etc/calico/ssl/client.key",
13      "etcd_cert_file": "/etc/calico/ssl/client.crt",
14      "etcd_ca_cert_file": "/etc/calico/ssl/ca.crt",
15      "nodename": "$HOSTNAME",
16      "mtu": 0,
17      "ipam": {
18          "type": "calico-ipam"
19      },
20      "container_settings": {
21          "allow_ip_forwarding": true
22      }
23    },
24    {
25      "type": "portmap",
26      "snat": true,
27      "capabilities": {"portMappings": true}
28    },
29    {
30      "type": "bandwidth",
31      "capabilities": {"bandwidth": true}
32    }
33  ]
34}
35EOF

使用nerdctl打印下可以连接的网络:

1nerdctl network ls
2NETWORK ID    NAME                         FILE
30             bridge
4              calico-containerd-network    /etc/cni/net.d/10-calico-containerd.conflist
5              host
6              none

启动containerd容器并连接到calico网络

calico的默认网络策略是各节点不通的,为了测试简单创建下面的全部允许的全局网络策略:

 1calicoctl apply -f - <<EOF
 2apiVersion: projectcalico.org/v3
 3kind: GlobalNetworkPolicy
 4metadata:
 5  name: allow-all
 6spec:
 7  selector: all()
 8  ingress:
 9  - action: Allow
10  egress:
11  - action: Allow
12EOF

在node1, node2上执行下面的命令,各启动一个redis容器连接到calico-containerd-network

1nerdctl run -d --name redis --network calico-containerd-network redis:alpine3.13

calicoctl查看IPPool确实分配了2个IP出去:

1calicoctl ipam show
2+----------+--------------+-----------+------------+--------------+
3| GROUPING |     CIDR     | IPS TOTAL | IPS IN USE |   IPS FREE   |
4+----------+--------------+-----------+------------+--------------+
5| IP Pool  | 10.89.0.0/16 |     65536 | 2 (0%)     | 65534 (100%) |
6+----------+--------------+-----------+------------+--------------+

通过使用calicoctl查看WorkloadEndpoint资源可以查看容器ip:

1calicoctl get WorkloadEndpoint
2WORKLOAD   NODE    NETWORKS          INTERFACE
3           node1   10.89.3.66/32     calidefault-9f6
4           node2   10.89.33.130/32   calidefault-f4d

在node1上使用nerdctl exec到redis容器,使用这个容器内的redis-cil可以连接上node2上ip为10.89.33.130的redis容器

 1[root@node1 ~]# nerdctl exec -it redis sh
 2/data # ifconfig
 3eth0      Link encap:Ethernet  HWaddr C2:3D:23:E0:40:03
 4          inet addr:10.89.3.66  Bcast:10.89.3.66  Mask:255.255.255.255
 5          inet6 addr: fe80::c03d:23ff:fee0:4003/64 Scope:Link
 6          UP BROADCAST RUNNING MULTICAST  MTU:1480  Metric:1
 7          RX packets:85 errors:0 dropped:0 overruns:0 frame:0
 8          TX packets:43 errors:0 dropped:0 overruns:0 carrier:0
 9          collisions:0 txqueuelen:0
10          RX bytes:26298 (25.6 KiB)  TX bytes:3055 (2.9 KiB)
11
12lo        Link encap:Local Loopback
13          inet addr:127.0.0.1  Mask:255.0.0.0
14          inet6 addr: ::1/128 Scope:Host
15          UP LOOPBACK RUNNING  MTU:65536  Metric:1
16          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
17          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
18          collisions:0 txqueuelen:1000
19          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
20
21/data # redis-cli -h 10.89.33.130
2210.89.33.130:6379>

说明容器间跨主机通信网络已经打通。

总结

本节完成了手动启动containerd容器,通过calico cni插件配置容器使用calico网络,进一步理解了容器运行时与容器网络接口CNI之间的关系。

参考