Kubernetes Calico网络在OpenStack上的安全组配置

最近跨网段配置k8s calico网络遇到了问题，简短记录一下解决过程。测试环境本地的物理服务器上有一个k8s集群，版本是1.20，calico网络组件版本为3.17。现在需要往这个k8s集群中添加2个新的node节点，新的节点在本地Openstack的虚拟机上。 Openstack虚拟机的网络和测试环境物理服务器之间网络已经打通，但属于不同的网段，已在Openstack中安全组中配置了允许物理机服务器网段所有TCP协议进入。

因为需要实现k8s pod的跨物理网段通信，使用的是calico的ipip模式。新节点加入到k8s集群后，各组件状态显示正常，k8s get node查看所有节点都是Ready的状态。在部署调度了测试的Pod到新节点上，发现新Node中的可以访问集群老的Node上的Pod，集群老的Node上的Pod无法访问新Node上的Pod。

已经将calico的客户端工具calicoctl部署为kubectl的插件，查看calico node的状态均正常:

kubectl calico node status
Calico process is running.

IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+----------------+-------------------+-------+----------+-------------+
| 10.10.100.101  | node-to-node mesh | up    | 10:52:16 | Established |
| 10.10.100.102  | node-to-node mesh | up    | 10:51:28 | Established |
| 10.10.100.103  | node-to-node mesh | up    | 10:53:02 | Established |
......
| 10.10.101.101  | node-to-node mesh | up    | 10:55:01 | Established |
| 10.10.101.102  | node-to-node mesh | up    | 10:53:28 | Established |

IPv6 BGP status
No IPv6 peers found.

IPPool信息如下:

kubectl calico get ipPool -o wide
NAME                  CIDR            NAT    IPIPMODE   VXLANMODE   DISABLED   SELECTOR
default-ipv4-ippool   10.244.0.0/16   true   Always     Never       false      all()

kubectl calico get felixConfiguration -o yml
apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    name: default
  spec:
    bpfLogLevel: ""
    ipipEnabled: true
    logSeverityScreen: Info
    prometheusMetricsEnabled: true
    reportingInterval: 0s
    vxlanEnabled: false
kind: FelixConfigurationList

看起来一切正常的样子，初步还是怀疑Openstack neutron网络上的限制。 Calico文档中关于在k8s集群中部署calico的系统需求文档中关于网络的需求如下:

Configuration	Host(s)	Connection type	Port/protocol
Calico networking (BGP)	All	Bidirectional	TCP 179
Calico networking with IP-in-IP enabled (default)	All	Bidirectional	IP-in-IP, often represented by its protocol number 4
Calico networking with VXLAN enabled	All	Bidirectional	UDP 4789
Calico networking with Typha enabled	Typha agent hosts	Incoming	TCP 5473 (default)
flannel networking (VXLAN)	All	Bidirectional	UDP 4789
All	kube-apiserver host	Incoming	Often TCP 443 or 6443*
etcd datastore	etcd hosts	Incoming	Officially TCP 2379 but can vary

因为新的K8S节点位于OpenStack环境中，会根据主机IP执行源IP检查出口包，为避免基础设施路由无法配置允许交换BGP路由的问题，才将这个k8s集群的Calico启用了IP-in-IP隧道模式，IPIPMODE设置为了Always。从前面的表格中IP-in-IP, often represented by its protocol number 4，可看出还需要允许ip protocol 4进入所有节点的防火墙，肯定是需要在Openstack安全组中配置允许ip protocol 4进入的。于是Web界面中配置如下，问题顺利解决。

参考

Calico Kubernetes - System requirements