VSwitch expansion in Ack cluster Terway network scenario
2021.12.28
Content
- 1、Introduction to Terway Network
- 2、Problem phenomenon
- 3、Expansion operations
- 3.1 Add a switch and configure NAT
- 3.2 Configure the Terway of the cluster
- 3.3 Restart terway
1. Introduction to Terway Network
Terway is Alibaba Cloud's open source VPC-based container network interface CNI (Container Network Interface) plug-in. It supports network policies based on the Kubernetes standard to define access policies between containers. The network intercommunication within the Kubernetes cluster can be realized by using the Terway network plug-in
The Terway network plug-in assigns native elastic network cards to Pods to implement Pod networks, supports network policies based on the Kubernetes standard (Network Policy) to define access policies between containers, and is compatible with Calico's network policies
In the Terway network plug-in, each Pod has its own network stack and IP address. The communication between Pods in the same ECS is directly forwarded through the machine's internal forwarding, and for cross-ECS Pod communication, messages are forwarded directly through the VPC's elastic network card. Since there is no need to use tunnel technology such as VxLAN to encapsulate packets, the Terway mode network has high communication performance
In one sentence, the biggest feature of Terway is that with the help of the characteristics of the ECS server on the cloud, the pod and node networks are leveled, and the ip in the vSwitch under the VPC is used at the same time
2. Problem phenomenon
Due to the use of the Terway network mode, as the number of node machines and pods increases, each allocation of an ip needs to consume the available ip of vsw under vpc. If the business grows rapidly in a short period of time, causing the pod to consume a large amount of available IP, this time may be insufficient due to insufficient pre-planning, resulting in insufficient available IP for the vSwitch
At this time, the status of the newly created pod is ContainerCreating. Describe and view the pod and prompt error allocate ip.... At this time, check the Terway log of the node where the Pod is located, and the following content will appear
Message: The specified VSwitch "vsw-xxxxx" has not enough IpAddress. It prompts that there are not enough ip addresses. At this time, it is basically because the ip of the switch is not enough. You can log in to the console of the switch to view the available ip of the switch where the node is located. If the number is very small or even 0, it means that the capacity needs to be expanded
3. Expansion operations
3.1 Add a switch and configure NAT
Create a new vSwitch in the VPC corresponding to the VPC management console. The vSwitch must be in the same area as the vSwitch with insufficient IP resources. This is because the strategy when Terway allocates pod ip is to allocate the ip corresponding to the vSwitch in the zone where the node is located. Therefore, expansion requires expansion of switches in the same zone.
It should be considered when initializing the cluster's new switch and expanding the switch. As the density of Pods is getting larger and larger, in order to meet the increasing demand of Pods for IP addresses, it is recommended that the network bits of the vSwitch used by the Pod are less than or equal to 19, that is, each The network segment contains at least 8192 available IP addresses
After the vSwitch is created, you need to configure a NAT policy for this vSwitch to access the external network.
3.2 Configure the Terway of the cluster
- kubectl -n kube-system edit cm eni-config
For configuration examples, refer to Terway Configuration Reference [1], and part of the content is described as follows:
- apiVersion: v1
- kind: ConfigMap
- metadata:
- name: eni-config
- namespace: kube-system
- data:
- 10-terway.conf: |-
- {
- "cniVersion": "0.3.0",
- "name": "terway",
- "type": "terway"
- }
- disable_network_policy: "true"
- eni_conf: |-
- {
- "version": "1", # 版本
- "max_pool_size": 80, # 资源池最大水位
- "min_pool_size": 20, # 资源池最小水位
- "credential_path": "/var/addon/token-config",
- "vswitches": {"cn-shanghai-f":["vsw-AAA", "vsw-BBB"]}, # 关联的虚拟交换机(ENI多IP模式)，添加vsw-BBB到VSwitches部分，其中vsw-AAA是已经存在的且IP资源不足的VSwitch
- "eni_tags": {"ack.aliyun.com":"xxxxxxxxx"},
- "service_cidr": "172.16.0.0/16", # 服务CIDR
- "security_group": "sg-xxxxxxx", # 安全组ID
- "vswitch_selection_policy": "ordered"
- }
In the above configuration parameters, the configuration value of the water level of the resource pool. Terway uses the underlying network resources of the underlying virtualization to open up the container network. The creation and release of network resources requires a series of API calls. If the API is called frequently when the Pod is created and destroyed, it will take a long time to configure the Pod. Terway caches resources in a pooled manner. When it is less than the minimum water level of the resource pool, it automatically replenishes resources, and starts to release resources when it is greater than the maximum water level of the resource pool, which ensures efficient resource utilization and allocation efficiency.
It is equivalent to pre-allocated ip, and the specific settings can be flexibly set in consideration of the maximum number of eni auxiliary network cards supported by the machine node specification and the maximum number of pods.
3.3 Restart terway
Restart all Terway pods to quickly refresh the cache to take effect.
- # kubectl -n kube-system delete pod -l app=terway-eniip
- # kubectl -n kube-system get pod | grep terway
After restarting, check whether the abnormal pod has obtained the ip normally.
When troubleshooting issues related to the ip allocation of a pod, you can also enter the gateway pod of the node where you are located and execute the command line to view the current allocated ip status and the temporarily idle ip status after it has been allocated from the vSwitch .
- # terway-cli mapping
- Status | Pod Name | Res ID | Factory Res ID
- Normal | node-problem-detector-l5h52 | 00:16:10:48:3e:37.10.244.18.167 | 00:16:10:48:3e:37.10.244.18.167
- ...
- Idle | | 00:16:10:48:3e:37.10.244.18.132 | 00:16:10:48:3e:37.10.244.18.132
- Idle | | 00:16:10:48:3e:37.10.244.18.18 | 00:16:10:48:3e:37.10.244.18.18
- Idle | | 00:16:10:48:3e:37.10.244.18.54 | 00:16:10:48:3e:37.10.244.18.54
See you ~
Configure Terway of the cluster and add the vSwitch created above to the ConfigMap configuration of Terway.
- kubectl -n kube-system edit cm eni-config