Part 01

demand background 

In a privatization project on the OneNET platform, the requirement of the project party is to obtain the real IP address of the device, and then count the number of devices in each province according to the IP of the device and display it on the large screen.

Part 02

find a solution 

Taking MQTT device access as an example, since the outer load used by the project party is the Nginx soft load, and the MQTT protocol is based on TCP, it can only forward messages in layer 4. When Nginx forwards messages, it will forward the source TCP connection The IP address is rewritten to its own intranet IP address. It cannot directly forward the source address of the device to the back-end service like the hard load of F5. Therefore, it is not possible to directly configure Nginx to allow the MQTT access service to obtain the device. If the source IP address is not used, the project party's needs cannot be realized.

After searching for related solutions online, I found an Internet protocol called proxy protocol (reference: https://www.jianshu.com/p/cc8d592582c9 ), which can be transmitted by adding a small header information to the TCP packet. Client information (protocol stack, source IP, destination IP, source port, destination port, etc.) is very useful when the network situation is complicated and the real IP of the user needs to be obtained. Its essence is that after the three-way handshake ends, the proxy inserts a data packet carrying the quadruple information of the original connection into the connection.

picture

proxy protocol agreement process

Check the format of the proxy protocol message as shown in the figure below, which contains information such as the source address and port of the client, which can meet our needs.

proxy protocol message format

To obtain this special message, the backend service also needs to configure and enable the proxy_protocol protocol on Nginx, as shown in the figure below.

Configure and enable the proxy_protocol protocol on Nginx

Through the wireshark tool to capture packets, we also found that the packet structure is consistent with the obtained information. The next thing we need to do is to parse this special packet and save the real IP of the client when the MQTT protocol is parsed.

Wireshark captures proxy_protocol protocol packets

Part 03

practical operation  

Just do it, we modified the source code of the MQTT access service based on the Netty framework, so we added a real IP decoder when encoding and decoding, as shown in the figure below.

Real IP Decoder

Then we parse the original message in the decode method of the decoder to determine whether there is a proxy protocol message, then parse the message and extract the real source IP address and port of the device inside, and keep it in the ChannelAttribute in Netty In the context, it is convenient for subsequent acquisition.

Note that the proxy protocol message here and the message in the MQTT protocol are glued together, so we need to extract the source address and then separate the remaining MQTT protocol packets to the subsequent MQTT protocol decoder for processing. This is overall processing flow.

Real IP decoder source code

Part 04

Test certification 

After repackaging and deploying the service, we can see from the logs that the source IP information of the test device can be obtained normally through the Nginx load method, which meets the requirements.

For the layer 4 UDP protocol to obtain the device source IP, you can also refer to this solution.