Background

The development convenience brought by microservice architecture significantly shortens the development cycle of business functions, and makes the continuous integration and delivery of business functions more agile through the native optimization of cloud computing platform architecture. But at the same time, microservice architecture also introduces many problems of service governance: an application consists of multiple services, each service has several instances, and the running state of each instance changes in real time, which gives rise to the emergence of inter-service communication layers. The communication layer is not coupled to the application code, but also captures the dynamic changes of the underlying environment and makes appropriate adjustments to avoid a single point of failure for the business.

1.Introduction to ServiceMesh

1.1 Introduction to Service Mesh

A service network is an infrastructure layer that handles inter-service communication. Cloud-native applications have complex service topologies where service meshes are responsible for the reliable delivery of requests. In practice, service meshes are typically implemented as a set of lightweight network proxies that are deployed with but transparent to the application.

Locally, service mesh technology deploys a proxy on an application node, and the application sends requests to the proxy, which completes point-to-point routing and forwarding.

In the above figure, if you remove the application in the left diagram, only the call relationship between the proxies and them is presented (i.e. the right figure). At this time, the Service The concept of mesh will be clear: the proxy and call relationships form a complete network, representing the complex call relationships between services and carrying all applications in the system.

1.2 Characteristics and advantages of service network architecture

1) Peer-to-peer communication: no central bottleneck.

2) No intrusion into the application: It can support the integration of heterogeneous technology products. At the same time, it is transparent to the application, and application development no longer needs to care about complex network communication implementation, and can focus on the implementation of business logic.

2.Istio及Envoy简介

Istio是一个由Google，IBM和Lyft团队合作开发的开源Service Mesh框架。目前已成为ServiceMesh的事实技术标准，被广泛应用于各个行业的IT架构。

Envoy is used A high-performance proxy developed in C++ with built-in functions such as service discovery, load balancing, TLS termination, HTTP/2, GRPC proxy, circuit breaker, health check, grayscale publishing based on percentage traffic splitting, fault injection, etc., to coordinate inbound and outbound traffic for all services in the service mesh.

3. Principles of Envoy traffic management

3.1 Introduction to Iptables

Istio calls iptables in Linux for traffic management. iptables is an application running in user space, which manages the flow and forwarding of network packets by controlling the Linux kernel netfilter module, in fact, netfilter is the real security framework of the firewall. netfilter is the cornerstone of the Linux network security building, which provides a complete set of hook function mechanisms, and the 5 hook points of the IP layer correspond to the 5 built-in chains of iptables:

• PREROUTING: In this DNAT.
• POSTROUTING: In this SNAT.
• INPUT: Handles packets entered into the local process.
• OUTPUT: Handles the packet output of the local process.
• FORWARD: Handles packets forwarded to other machines and other network namespaces.

3.2 IP packets regarding network inbound

THE IP PACKET INBOUND FROM THE NETWORK ENTERS THE TREOUTING CHAIN FIRST, AND THEN THE ROUTING JUDGMENT IS MADE:

1) If the packet routing destination is local: enter the INPUT chain and send it to the local process.

2) If the packet routing destination is not local and IP forwarding is enabled, it enters the FORWARD chain, then passes through the POSTROUTING chain, and finally sends away through the network interface.

3) For packets sent by local processes to the protocol stack, they are first sent through the OUTPUT chain, then through the POSTROUTING chain, and finally through the network interface.

3.3 About custom chains

In addition, we can also customize the chain, but the custom chain can only be called as an action by a default chain to work.

In Kubernetes, Istio envoys Envoy through the mechanism of admission webhooks Sidecars are automatically injected and run in the same pod as the application container, in which case they will share the network namespace and therefore use the same network stack.

The configuration that Istio injects into application pods mainly includes:

1) Init container istio-init: Used to set iptables port forwarding in Pods.

2) Sidecar container istio-proxy: Run Envoy Sidecar proxies.

3.4Iptables configuration rules

After the container is initialized, we go to the sidecar container to switch to root and view the configured iptables rules.

iptables -t nat -S
1.

ISTIO_INBOUND chain: All traffic entering the pod but not the specified port (such as 22) is redirected to port 15006 (Envoy ingress traffic port) to intercept it.

ISTIO_OUTPUT chain: will be used by istio-proxy Outbound traffic from pods sent out of user space and destined for localhost is all redirected to port 15001 (envoy egress traffic port). All other traffic is released directly to the next POSTROUTING chain without being intercepted by Envoy.

The overall traffic flow diagram is shown in the following figure:

1) The inbound traffic entering the pod is first intercepted and processed by the PREROUTING chain.

2) When TCP requests enter the PREROUTING chain, all are handled by ISTIO_INBOUND.

-A PREROUTING -p tcp -j ISTIO_INBOUND
1.

3) TCP requests with a destination port other than 15008/22/15090/15021/15020 are all handled by the ISTIO_IN_REDIRECT.

-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
1.

4) Redirect all TCP requests sent to this to port 15006 (Envoy ingress traffic port)

-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
1.

5) After Envoy's internal processing, it is decided to forward the packet to the application, which is an egress traffic for Envoy and will be intercepted by netfilter and forwarded to the egress traffic OUTPUT chain.

6) Outbound requests, when TCP requests enter the OUTPUT chain, are all handed over to ISTIO_OUTPUT processing.

-A OUTPUT -p tcp -j ISTIO_OUTPUT
1.

7) MATCH THE OUTBOUND REQUEST CORRESPONDING RULE, REQUEST THE LOCAL SERVICE, THE EXIT IS THE LO NETWORK CARD AND COME FROM ISTIO-PROXY USER SPACE, AND THE TRAFFIC RETURNS TO ITS CALL POINT IN THE NEXT CHAIN EXECUTION, THAT IS, THE POSTROUTING CHAIN 。

-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN -A ISTIO_OUTPUT -m owner 
--gid-owner 1337 -j RETURN
1.
2.

8) The request made by the sidecar arrives at the target application.

9) The target application responds to the sidecar after processing the business logic, which is an egress traffic for the application, and is intercepted and forwarded to the egress traffic OUTPUT chain by netfilter again.

10) Outbound requests, when TCP requests enter the OUTPUT chain, are all handled by ISTIO_OUTPUT.

-A OUTPUT -p tcp -j ISTIO_OUTPUT
1.

11) Request the next service/response request, i.e. request a non-local service and not from the istio-proxy user space, and the traffic is forwarded to the ISTIO_REDIRECT chain.

-A ISTIO_OUTPUT -j ISTIO_REDIRECT
1.

12) Redirect all TCP protocol request traffic redirected to port 15001 (Envoy egress traffic port).

-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
1.

13) After Envoy's internal processing, it is decided to forward the packet externally, which is an egress traffic for Envoy and will be intercepted and forwarded to the egress traffic OUTPUT chain by netfilter.

14) Outbound requests, when TCP requests enter the OUTPUT chain, are all handed over to ISTIO_OUTPUT processing.

-A OUTPUT -p tcp -j ISTIO_OUTPUT
1.

15) REQUEST NON-LOCAL SERVICES, EXIT NOT LO NETWORK CARD AND FROM ISTIO-PROXY USER SPACE SKIPS ISTIO_REDIREC PROCESSING AND RETURNS DIRECTLY TO THE NEXT CHAIN, THAT IS, THE POSTROUTING CHAIN

-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN -A ISTIO_OUTPUT -m owner 
--gid-owner 1337 -j RETURN
1.
2.

16) After the POSTROUTING chain processing is completed, select the appropriate network card according to the routing table to send Outbound traffic.

4. Summary

Envoy's core job is to transparently intercept requests and manage all incoming and outgoing traffic. The intercepted requests are processed by certain rules for security access control, access control, flow control and many other aspects, and then sent to the application. By using Envoy, developers can focus on developing application functions without having to worry about complex network communication.