前面一节完成了calico node集群的部署,本节将在各个节点上部署calico cni插件, 并在两个主机节点上各启动一个连接到calico网络的containerd容器,测试两个容器网络是否是打通的。

部署容器运行时Containerd和CNI

在各个服务器节点上安装容器运行时Containerd。

创建/etc/modules-load.d/containerd.conf配置文件:

1
2
3
4
cat << EOF > /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

执行以下命令使配置生效:

1
2
modprobe overlay
modprobe br_netfilter

创建/etc/sysctl.d/containerd.conf配置文件:

1
2
3
4
5
6
cat << EOF > /etc/sysctl.d/containerd.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
user.max_user_namespaces=28633
EOF

执行以下命令使配置生效:

1
sysctl -p /etc/sysctl.d/containerd.conf

下载Containerd的二进制包:

1
wget https://github.com/containerd/containerd/releases/download/v1.5.2/cri-containerd-cni-1.5.2-linux-amd64.tar.gz

cri-containerd-cni-1.5.2-linux-amd64.tar.gz压缩包中已经按照官方二进制部署推荐的目录结构布局好。 里面包含了systemd配置文件,containerd以及cni的部署文件。 将解压缩到系统的根目录/中:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
tar -zxvf cri-containerd-cni-1.5.2-linux-amd64.tar.gz -C /

etc/
etc/systemd/
etc/systemd/system/
etc/systemd/system/containerd.service
etc/crictl.yaml
etc/cni/
etc/cni/net.d/
etc/cni/net.d/10-containerd-net.conflist
usr/
usr/local/
usr/local/sbin/
usr/local/sbin/runc
usr/local/bin/
usr/local/bin/critest
usr/local/bin/containerd-shim
usr/local/bin/containerd-shim-runc-v1
usr/local/bin/ctd-decoder
usr/local/bin/containerd
usr/local/bin/containerd-shim-runc-v2
usr/local/bin/containerd-stress
usr/local/bin/ctr
usr/local/bin/crictl
......
opt/cni/
opt/cni/bin/
opt/cni/bin/bridge
......

接下来生成containerd的配置文件:

1
2
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

根据文档Container runtimes 中的内容,对于使用systemd作为init system的Linux的发行版,使用systemd作为容器的cgroup driver可以确保服务器节点在资源紧张的情况更加稳定,因此这里配置各个节点上containerd的cgroup driver为systemd。

修改前面生成的配置文件/etc/containerd/config.toml

1
2
3
4
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

再修改/etc/containerd/config.toml中的

1
2
3
4
[plugins."io.containerd.grpc.v1.cri"]
  ...
  # sandbox_image = "k8s.gcr.io/pause:3.5"
  sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.4.1"

配置containerd开机启动,并启动containerd

1
2
systemctl enable containerd
systemctl start containerd

使用crictl测试一下,确保可以打印出版本信息并且没有错误信息输出:

1
2
3
4
5
crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.5.2
RuntimeApiVersion:  v1alpha2

安装containerd的nerdctl客户端:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
wget https://github.com/containerd/nerdctl/releases/download/v0.10.0/nerdctl-0.10.0-linux-amd64.tar.gz
tar -zxvf nerdctl-0.10.0-linux-amd64.tar.gz nerdctl && mv nerdctl /usr/local/bin/

nerdctl version
Client:
 Version:	v0.10.0
 Git commit:	ad836c2f0f53084a762112be58379d8d795812a5

Server:
 containerd:
  Version:	v1.5.2
  Revision:	36cc874494a56a253cd181a1a685b44b58a2e34a

部署calico cni plugin

在各个节点上下载calico cni plugin二进制文件到/opt/cni/bin/:

1
2
3
4
5
cd /opt/cni/bin/
curl -o calico -O -L  "https://github.com/projectcalico/cni-plugin/releases/download/v3.19.1/calico-amd64" 
curl -o calico-ipam -O -L "https://github.com/projectcalico/cni-plugin/releases/download/v3.19.1/calico-ipam-amd64"
chmod +x calico
chmod +x calico-ipam

在各个节点上创建calico cni网络配置列表配置文件:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
cat > /etc/cni/net.d/10-calico-containerd.conflist << EOF
{
  "name": "calico-containerd-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "calico",
      "log_level": "info",
      "log_file_path": "/var/log/calico/cni/cni.log",
      "datastore_type": "etcdv3",
      "etcd_endpoints": "https://192.168.100.101:2379,https://192.168.100.102:2379,https://192.168.100.103:2379",
      "etcd_key_file": "/etc/calico/ssl/client.key",
      "etcd_cert_file": "/etc/calico/ssl/client.crt",
      "etcd_ca_cert_file": "/etc/calico/ssl/ca.crt",
      "nodename": "$HOSTNAME",
      "mtu": 0,
      "ipam": {
          "type": "calico-ipam"
      },
      "container_settings": {
          "allow_ip_forwarding": true
      }
    },
    {
      "type": "portmap",
      "snat": true,
      "capabilities": {"portMappings": true}
    },
    {
      "type": "bandwidth",
      "capabilities": {"bandwidth": true}
    }
  ]
}
EOF

使用nerdctl打印下可以连接的网络:

1
2
3
4
5
6
nerdctl network ls
NETWORK ID    NAME                         FILE
0             bridge
              calico-containerd-network    /etc/cni/net.d/10-calico-containerd.conflist
              host
              none

启动containerd容器并连接到calico网络

calico的默认网络策略是各节点不通的,为了测试简单创建下面的全部允许的全局网络策略:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
calicoctl apply -f - <<EOF
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-all
spec:
  selector: all()
  ingress:
  - action: Allow
  egress:
  - action: Allow
EOF

在node1, node2上执行下面的命令,各启动一个redis容器连接到calico-containerd-network

1
nerdctl run -d --name redis --network calico-containerd-network redis:alpine3.13

calicoctl查看IPPool确实分配了2个IP出去:

1
2
3
4
5
6
calicoctl ipam show
+----------+--------------+-----------+------------+--------------+
| GROUPING |     CIDR     | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+--------------+-----------+------------+--------------+
| IP Pool  | 10.89.0.0/16 |     65536 | 2 (0%)     | 65534 (100%) |
+----------+--------------+-----------+------------+--------------+

通过使用calicoctl查看WorkloadEndpoint资源可以查看容器ip:

1
2
3
4
calicoctl get WorkloadEndpoint
WORKLOAD   NODE    NETWORKS          INTERFACE
           node1   10.89.3.66/32     calidefault-9f6
           node2   10.89.33.130/32   calidefault-f4d

在node1上使用nerdctl exec到redis容器,使用这个容器内的redis-cil可以连接上node2上ip为10.89.33.130的redis容器

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@node1 ~]# nerdctl exec -it redis sh
/data # ifconfig
eth0      Link encap:Ethernet  HWaddr C2:3D:23:E0:40:03
          inet addr:10.89.3.66  Bcast:10.89.3.66  Mask:255.255.255.255
          inet6 addr: fe80::c03d:23ff:fee0:4003/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1480  Metric:1
          RX packets:85 errors:0 dropped:0 overruns:0 frame:0
          TX packets:43 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:26298 (25.6 KiB)  TX bytes:3055 (2.9 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

/data # redis-cli -h 10.89.33.130
10.89.33.130:6379>

说明容器间跨主机通信网络已经打通。

总结

本节完成了手动启动containerd容器,通过calico cni插件配置容器使用calico网络,进一步理解了容器运行时与容器网络接口CNI之间的关系。

参考