​This article focuses on the TCP​ protocol. Let’s first review the characteristics of the TCP protocol:

• TCP is a connection-oriented transport layer protocol.
• Each TCP connection has only two endpoints, and each TCP connection can only be point-to-point (one-to-one).
• TCP provides a reliable delivery service that ensures that the transmitted data is error-free, lost, duplicated, and ordered.
• TCP​ provides full-duplex communication. TCP​ allows application processes on both sides of the communication to send data at any time. For this reason, both ends of the TCP connection are equipped with send buffers and receive buffers to temporarily store data for two-way communication.
• TCP​ is byte stream oriented. (focus of this article)

Although the interaction between the application and TCP is one block of data (of varying sizes) at a time, TCP treats the data handed over by the application as just a series of unstructured byte streams.

The origin of sticky police? Origin of sticky bag?

Sticky Pack Police, the term was first seen in v2​. The sticky package police believe that the word "sticky package" insults TCP​, and it is a false proposition to discuss "sticky package" under TCP​. Conversely, sticky packetists consider "sticky packets" to be a TCP​ problem. Then the sticky policeman frequently appeared under the "TCP​sticky package" post, trying to correct this prejudice, reminding everyone: TCP is oriented to byte streams.

A little story about the origin of sticky bags:

It is said that there used to be a group of programmers with weak foundation who often used VC to write various Windows client programs, and liked to use UDP programming (VC's UDP programming, the code is simple, the sending and receiving logic is simple and clear). Because of the complexity and requirements of communication applications, they tried to send multiple pieces of data in one UDP packet, and encountered the "sticky packet problem". At the same time, they began to contact and use TCP​. The inertial thinking applies the previous UDP​ programming method to use TCP​, and it is very easy to encounter the so-called "sticky packet problem". With hardware upgrades, the popularization of multi-physical-core CPUs, and the beginning of multi-threading and parallel programming processes, higher requirements are placed on the basic skills of programmers. These people are still programming in parallel programs using serial thinking, and they must encounter "sticky packages". question". So this group of people summed up this problem and called it the "sticky bag problem".

What is sticking/unpacking?

The so-called sticky package: It means that several data packets are glued together. If you want to process them, you must unpack them first.

The so-called unpacking: is to receive a batch of data packet fragments, these fragments must be glued together to form a complete data packet.

For example, when the client sends data to the server, the following five situations may occur:

• Chestnut 1: The client sends the complete data packets A and B respectively, and the server receives the complete data packet A first, and there is no unpacking/sticking problem.
• Chestnut 2: The client sends the data packets that A and B stick together one by one, and the server receives the data packets. The server needs to parse out A and B, and there is a sticky packet problem.
• Chestnut 3: The client sends A|B-1​ data packets and B-2 data packets, the server first receives the complete A and B part of the data packet B-1, the server needs to parse the complete A, and wait Read the complete B packet, there is a sticking/unpacking problem.
• Chestnut 4: The client sends the A-1​ data packet and the B|A-2 data packet, and the server receives a part of the data packet A-1 of A. At this time, it needs to wait for the complete A data packet to be received, and there is an unpacking problem. .
• Chestnut 5: The data packet A is large, the client sends the data packet A in sections, and the server needs several times to receive the data packet A, and there is a problem of unpacking.

Summary: Due to the unpacking/sticking problem, how to identify a complete packet becomes a problem? The difficulty lies in how to define the boundaries of a packet.

Why do some people say TCP sticky packets?

Let's first look at the process of the application using TCP​ sockets: Corresponding to the TCP/IP 4-layer protocol:

• When an application calls write, the kernel copies all data from the application's buffer to the send buffer of the written socket.
• Local TCP delivers data to IP in MSS-sized or smaller chunks.
• The TCP segment plus the IP header forms the IP packet, and looks up the routing table entry according to its destination IP address to determine the outgoing interface, and then delivers the datagram to the corresponding data link.

MSS​ and MTU are explained here:

• MTU (Maxitum Transmission Unit​) is the size of the maximum transmission data at the link layer at one time. Generally speaking, the size is 1500byte.
• MSS (Maximum Segement Size​) refers to the maximum segment length of TCP, which is the size of the largest data sent by the transport layer at one time.

The general calculation relationship between MTU​ and MSS​ is: MSS​ = MTU​ - IP​ header - TCP header.

"Packaging Scientists" believe that there are three reasons for TCP sticking/unpacking:

• The size of bytes written by the application write is larger than the socket send buffer size.
• MSS​ +TCP​ header+IP​ header>MTU​, TCP segmentation is required.
• If the payload of the Ethernet frame​ is greater than the MTU​, IP fragmentation is required.

To put it bluntly, the "sticker" thinks what I gave you, you should give it back to me.

The "sticky police" thinks that this is not TCP's pot at all:

• TCP is byte-oriented: there is no concept of packets at all, what about sticking/unpacking.
• The essential problem of "sticking/unpacking" is: how to extract data from the binary stream and how to define the boundaries of the data.

To put it bluntly, the "sticky packet police" believes that how to parse the data is the problem of your application layer, and TCP only transmits and provides reliable delivery services.

Extension: Nagle's Algorithm

The Nagle​ algorithm was defined by Ford Aviation and Communications as a TCP/IP​ congestion control method in 1984, which enabled Ford to operate the earliest dedicated TCP/IP networks for congestion control reduction, and the method has been widely used ever since.

Advantage: Avoid flooding the network with many small data blocks in order to send as large blocks of data as possible.

If the data that needs to be sent each time is only 1 byte, plus the IP header of 20 bytes and the TCP header of 20 bytes, the size of the data packet sent each time is 41 bytes, but only 1 byte is valid information, which is a huge waste of information.

The rules of the Nagle​ algorithm (refer to the tcp_nagle_check function note in the tcp_output.c​ file):

• If the packet length reaches MSS, it is allowed to send;
• If this contains a FIN, it is allowed to send;
• If the TCP_NODELAY option is set, it is allowed to send;
• When the TCP_CORK​ option is not set, if all outgoing small data packets (packet length less than MSS) are confirmed, it is allowed to send;
• None of the above conditions are met, but a timeout (usually 200ms) occurs, and it will be sent immediately.

Linux​ turns on the Nagle algorithm by default, which can effectively reduce network overhead in scenarios with a large number of small packets.

• The Nagle algorithm can be disabled via the TCP_NODELAY​ parameter provided by Linux.
• In Netty​, the Nagle algorithm is disabled by default in order to minimize the data transmission delay.

Tips:​ There is also a delayed ACK (Delay ACK​). When TCP​ sends an ACK, there are the following regulations:

• When there is response data sent, ACK will be sent along with the data.
• .If there is no response data, ACK will have a delay to wait for whether there is a response data sent in one piece, but this delay is generally between 40ms and 500ms, usually around 40ms.
• If the second data arrives while waiting for the ACK to be sent, the ACK should be sent immediately.

Extension: Why is UDP not fragmented?

Let's review the characteristics of UDP first:

• UDP does not require a connection to be established.
• No connection status.
• Packet header overhead is small. (the first 8 bytes)
• UDP is packet-oriented. (emphasis)

The sender UDP​ delivers the packets delivered by the application layer to the IP​ layer after adding the header, neither merging nor splitting, but retaining the boundaries of these packets; receiver UDP​ to IP​ The ​layer hands over the UDP​ user datagram, and after removing the header, it is delivered to the upper-layer application process intact, and a complete message is delivered at a time. Therefore, the message is indivisible and is the smallest unit of UDP datagram processing.

Look at the UDP datagram format:

It can be seen that the maximum length of user data that a UDP datagram can carry is: 2^16 - 8 = 65535 - 8 = 65527 (B)

Summary Why is UDP not segmented?

• UDP​ protocol features: packet-oriented. 16-bit UDP length. No segmentation ability: the ability to mark the sequence of segmentation, that is, the number (ID​), the identifier of the tail number (Flag)
• UDP​ Application characteristics: It is often used in network applications that transmit a relatively small amount of data at one time, such as DNS, SNMP​, etc.

When the DNS​ query exceeds 512 bytes, the TC​ flag of the protocol has a delete flag, and then uses TCP​ to send. Usually traditional UDP​ packets are generally no larger than 512 bytes.

Unpacking/Stick Packing Solutions

It can be seen from the above that we need a definition to define the boundary of the data packet, which is also the only way to solve the unpacking/sticking: define the communication protocol of the application layer.

The mainstream protocol solutions are:

• fixed message length
• specific delimiter
• message length + message content

Netty supports three common frame sealing methods:

fixed message length

The class FixedLengthFrameDecoder is provided in Netty​:

• Each datagram requires a fixed length.
• When the receiver accumulatively reads the fixed-length messages, it considers that a complete message has been obtained.
• When the sender's data is less than the fixed length, it needs to be filled with blanks.

# 举个栗子：假定固定消息长度是 3字节，当你收到如下报文：
+---+----+------+----+
| A | BC | DEFG | HI |
+---+----+------+----+

# 将它们解码成以下 3个固定长度的数据包：
+-----+-----+-----+
| ABC | DEF | GHI |
+-----+-----+-----+
1.
2.
3.
4.
5.
6.
7.
8.
9.

Project address: Corresponding code:

ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
    .channel(NioServerSocketChannel.class)
    .childHandler(new ChannelInitializer<SocketChannel>() {
        @Override
        public void initChannel(SocketChannel ch) {
            ch.pipeline().addLast(new FixedLengthFrameDecoder(3));
            //... ...
        }
    });
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

Access via telnet​: telnet localhost 8088

Advantages and disadvantages:

• Advantages: The fixed-length message method is very simple to use
• Disadvantages: The fixed-length value cannot be set well. If the length is too large, it will cause a waste of bytes, and if the length is too small, it will affect the message transmission. Therefore, the fixed-length message method will not be used in general.

special separator

Since the receiver cannot distinguish the boundary of the message, a specific separator can be added to the end of each sent message, and the receiver can split the message according to the special separator.

DelimiterBasedFrameDecoder automatically completes the decoding of messages ending with a delimiter:

# 举个栗子：以下报文根据特定分隔符 `\n` 按行解析
+--------------+
| ABC\nDEF\r\n |
+--------------+

# 解析后得到：
+-----+-----+
| ABC | DEF |
+-----+-----+
1.
2.
3.
4.
5.
6.
7.
8.
9.

Project address: code

ServerBootstrap b = new ServerBootstrap();

b.group(bossGroup, workerGroup)
    .channel(NioServerSocketChannel.class)
    .childHandler(new ChannelInitializer<SocketChannel>() {

        @Override
        public void initChannel(SocketChannel ch) {
            // 以 & 为分隔符
            ByteBuf delimiter = Unpooled.copiedBuffer("&".getBytes());
            // 10 表示单条消息的最大长度，当达到该长度后扔没有查找到分隔符，就抛出异常
            // TooLongFrameException，防止由于异常码流失分隔符导致的内存溢出
            ch.pipeline().addLast(new DelimiterBasedFrameDecoder(10, delimiter));
            // ... ...
        }
    });
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.

Access via telnet​: telnet localhost 8088

The recommended practice is to encode the message, such as base64 encoding, and then choose a character other than the 64 encoded characters as a specific delimiter.

The specific delimiter method is more efficient when the message protocol is simple enough. Redis uses the newline delimiter in the communication process.

• Communication after Redis 2.0​ is unified as RESP​ protocol (Redis Serialization Protocol)
• RESP​ is a binary secure text protocol that works over the TCP​ protocol. RESP​ uses lines as the unit, and the commands or data sent by the client and the server are always \r\n (CRLF) as newlines.

message length + message content

Message length + message content is the most commonly used protocol in project development. The basic format of the protocol is shown below.

+--------|----------+
|消息头    |消息体    |
+--------|----------+
| Length | Content  |
+--------|----------+
1.
2.
3.
4.
5.

The total length of the message is stored in the message header, for example, a 4-byte int value is used to record the length of the message, and the actual binary byte data of the message body.

When the receiver parses the data:

• First read the length field Len of the message header
• Then read the byte data of length Len, and the data is judged as a complete data packet

Still taking the raw byte data mentioned above as an example, the result of encoding using this protocol is as follows:

+-----|-------|-------|----|-----+
| 2AB | 4CDEF | 4GHIJ | 1K | 2LM |
+-----|-------|-------|----|-----+
1.
2.
3.

The use of message length + message content is very flexible, and there are no obvious shortcomings of the fixed message length method and the specific delimiter method.

Of course, the message header is not only limited to storing the length of the message, but also other necessary extension fields can be customized:

• message version
• Algorithm type
• and many more