background introduction

The development convenience brought by the microservice architecture shortens the development cycle of business functions significantly. Through the native optimization of the cloud computing platform architecture, the continuous integration and delivery of business functions is made more agile. But at the same time, the microservice architecture also introduces many problems in service governance: an application is composed of multiple services, each service has several instances, and the running status of each instance changes in real time, which gave birth to the emergence of the inter-service communication layer. The communication layer is neither coupled with the application code, but also can capture the dynamic changes of the underlying environment and make appropriate adjustments to avoid single point failures in the business.

1. Introduction to Service Mesh

1.1 Introduction to Service Mesh

The service mesh is an infrastructure layer that handles inter-service communication. Cloud-native applications have complex service topologies, and service grids are responsible for reliable delivery of requests in these topologies. In practice, service meshes are typically implemented as a set of lightweight network proxies that are deployed alongside, but transparent to, applications.

Partially, service grid technology is to deploy agents on application nodes, applications send requests to agents, and agents complete point-to-point routing and forwarding.

In the above figure, if the application program in the left figure is removed, only the agent and the calling relationship between them (that is, the right figure) will be presented. At this time, the concept of Service Mesh will be clear: the agent and the call relationship form a complete network, representing the complex call relationship between services, and carrying all the applications in the system.

1.2 Features and advantages of service network architecture

1) Point-to-point communication: There is no central bottleneck.

2 ) No intrusion to applications: it can support the integration of heterogeneous technology products. At the same time, it is transparent to the application, and application development no longer needs to care about the realization of complex network communication, and can focus on the realization of business logic.

2. Introduction to Istio and Envoy

Istio is an open source Service Mesh framework jointly developed by Google, IBM and Lyft teams. At present, it has become the de facto technical standard of ServiceMesh and is widely used in the IT architecture of various industries.

Envoy is a high-performance proxy developed in C++ language. It has built-in functions such as service discovery, load balancing, TLS termination, HTTP/2, GRPC proxy, circuit breaker, health check, grayscale release based on percentage traffic splitting, fault injection, etc. Used to coordinate inbound and outbound traffic for all services in the service mesh.

3. The principle of Envoy traffic management

3.1 Introduction to Iptables

Istio calls iptables in Linux for traffic management. Iptables is an application software running in user space. It manages the flow and transfer of network data packets by controlling the Linux kernel netfilter module. In fact, netfilter is the real security framework of the firewall. Netfilter is the cornerstone of the Linux network security building. It provides a complete set of hook (Hook) function mechanism. The five hook points of the IP layer correspond to the five built-in chains of iptables:

• PREROUTING: Here DNAT.
• POSTROUTING: SNAT here.
• INPUT: Processes packets input to the local process.
• OUTPUT: Handle the packets output by the local process.
• FORWARD: Process packets forwarded to other machines and other network namespaces.

3.2 Regarding the inbound IP packets of the network

Inbound IP packets from the network first enter the TREOUTING chain, and then make routing judgments:

1) If the packet routing destination is the local machine: enter the INPUT chain, and then send it to the local process.

2 ) If the routing destination of the packet is not the local machine, and IP forwarding is enabled, it will enter the FORWARD chain, then pass through the POSTROUTING chain, and finally send it out through the network interface.

3 ) For the packet sent by the local process to the protocol stack, it first passes through the OUTPUT chain, then passes through the POSTROUTING chain, and finally sends it away through the network interface.

3.3 About custom chain

In addition, we can also customize the chain, but the custom chain can only work if it is called as an action by a default chain.

In Kubernetes, Istio automatically injects Envoy Sidecar through the Admission webhook mechanism, and runs in the same Pod as the application container. In this case, they will share the network namespace and therefore use the same network protocol stack.

The configuration that Istio injects into the application Pod mainly includes:

1) Init container istio-init: used to set iptables port forwarding in Pod.

2) Sidecar container istio-proxy: run Envoy Sidecar proxy.

3.4 Iptables configuration rules

After the container is initialized, we enter the Sidecar container and switch to the root user to view the configured iptables rules.

iptables -t nat -S
1.

ISTIO_INBOUND chain: All traffic entering the Pod but not the specified port (such as 22) is redirected to port 15006 (Envoy ingress traffic port) for interception.

ISTIO_OUTPUT chain: Redirect all outbound traffic from pods that are sent from istio-proxy user space and whose destination is not localhost to port 15001 (envoy egress traffic port). All other traffic is directly released to the next POSTROUTING chain without being intercepted by Envoy.

The overall flow flow diagram is shown in the figure below:

1) Inbound traffic entering the Pod is first intercepted and processed by the PREROUTING chain.

2 ) When the TCP request enters the PREROUTING chain, it is all handed over to ISTIO_INBOUND for processing.

-A PREROUTING -p tcp -j ISTIO_INBOUND
1.

3 ) All TCP requests whose destination ports are not 15008/22/15090/15021/15020 are handed over to ISTIO_IN_REDIRECT for processing.

-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
1.

4 ) Redirect all TCP requests sent here to port 15006 (Envoy ingress traffic port)

-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
1.

5) After Envoy's internal processing, it is decided to forward the data packet to the application. This step is egress traffic for Envoy and will be intercepted by netfilter and forwarded to the OUTPUT chain of the egress traffic.

6 ) Outbound requests, when TCP requests enter the OUTPUT chain, all are handed over to ISTIO_OUTPUT for processing.

-A OUTPUT -p tcp -j ISTIO_OUTPUT
1.

7 ) Match the corresponding rule of the outbound request, request the local service, the exit is the lo network card and come from the istio-proxy user space at the same time, and the traffic returns to the next chain in its call point for execution, that is, the POSTROUTING chain.

-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN -A ISTIO_OUTPUT -m owner 
--gid-owner 1337 -j RETURN
1.
2.

8 ) The request sent by Sidecar reaches the target application.

9 ) The target application responds to Sidecar after processing the business logic. This step belongs to the egress traffic for the application, and is intercepted and forwarded to the OUTPUT chain of the egress traffic by netfilter again.

10 ) Outbound requests, when TCP requests enter the OUTPUT chain, all are handed over to ISTIO_OUTPUT for processing.

-A OUTPUT -p tcp -j ISTIO_OUTPUT
1.

11 ) To request the next service/response request, that is, to request a non-local service and not come from the istio-proxy user space, the traffic is forwarded to the ISTIO_REDIRECT chain.

-A ISTIO_OUTPUT -j ISTIO_REDIRECT
1.

12 ) Redirect all TCP protocol request traffic redirected here to port 15001 (Envoy egress traffic port).

-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
1.

13 ) After Envoy's internal processing, it is decided to forward the data packet to the outside world. This step is egress traffic for Envoy, and will be intercepted and forwarded to the egress traffic OUTPUT chain by netfilter.

14 ) Outbound requests, when TCP requests enter the OUTPUT chain, all are handed over to ISTIO_OUTPUT for processing.

-A OUTPUT -p tcp -j ISTIO_OUTPUT
1.

15 ) When requesting a non-local service, the export is not from the lo network card and at the same time it comes from the istio-proxy user space, the ISTIO_REDIREC processing is skipped, and the RETURN directly goes to the next chain, that is, the POSTROUTING chain

-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN -A ISTIO_OUTPUT -m owner 
--gid-owner 1337 -j RETURN
1.
2.

16 ) After the processing of the POSTROUTING chain is completed, select the appropriate network card to send outbound traffic according to the routing table.

4. Summary

The core work of Envoy is to intercept business transparent requests and manage all incoming and outgoing traffic. After certain rules are applied to the intercepted request, it is processed in many aspects such as security access control, access control, flow control, etc., and then sent to the application program. By using Envoy, developers can focus on the development of application functions without considering complex network communication.