Detailed explanation of the principle of TCP sticking and unpacking!

In computer networks, TCP (Transmission Control Protocol) is a connection-oriented, reliable, byte-stream-based transport layer protocol. Since it treats data as a continuous stream of bytes, rather than independent messages or packets, it may encounter problems with sticking and unpacking. In this article, we will explain in detail how these two phenomena work and why they are.

1. Basic features of TCP

Byte-stream-oriented: TCP doesn't care about the boundaries of application-layer data, and the data is treated as a continuous byte stream.
Reliable transmission: Ensure the reliability and sequence of data through serial number, acknowledgment response, retransmission mechanism, etc.
Flow Control vs. Congestion Control: Adjust the transmission rate to prevent network congestion and receiver overflow.

Because of these features, TCP does not preserve the message boundaries of the application layer when transferring data, which directly leads to the problem of sticking and unpacking.

2. Packet sticking (packet sticking)

(1) Definitions

A sticky packet is a packet sent independently by multiple application layers that is combined into a single TCP packet during transmission and reaches the receiver, and the receiver cannot distinguish whether it is one packet or multiple packets.

(2) Causes

The sender sends data too fast: The application layer sends small data multiple times, and TCP combines them into one large packet to send them to improve transmission efficiency.
Network latency and buffering: TCP's send and receive buffers stage data and are sent all at once when the buffer accumulates to a certain level or reaches the send window.
Nagle algorithm: To reduce the number of small packets, the Nagle algorithm combines multiple small packets into a single packet send.

(3) Examples

Suppose the application layer sends two small messages in a row: "Hello" and "World", which may be combined into a single packet "HelloWorld" to reach the receiver during TCP transmission.

3. Unpacking (Packet Splitting)

(1) Definitions

Unpacket unpacking means that a packet sent by an application layer is split into multiple TCP packets to reach the receiver, and the receiver needs to reassemble these fragmented data to obtain the original message in its entirety.

(2) Causes

A single packet is too large: The amount of data sent by the application layer exceeds the maximum TCP packet segment length (MSS), causing the data to be split.
Changes in network conditions: For example, network congestion, packet loss, etc., TCP may retransmit and split data.
Receiver buffer limitation: The receiver buffer is not processed in a timely manner, resulting in fragmented data reception.

(3) Examples

The application layer sends a large message, "HelloWorld", which may be split into two TCP packets, "Hello" and "World", which need to be reassembled after reaching the receiver.

4. How to deal with sticky packs and unpacks

Since sticking and unpacking is due to the streaming nature of TCP, there are some strategies that the application layer needs to adopt to address this issue. Common methods are:

(1) Fixed-length protocol

Each message is of a fixed length, and the receiver reads the data by a fixed number of bytes.

Pros: Simple and easy to implement. Cons: Not flexible enough, wasted bandwidth, or unable to accommodate longer messages.
Example: Each message is fixed at 10 bytes, and the receiver reads 10 bytes at a time as a complete message.

(2) Separator protocol

A specific delimiter is added between messages, and the receiver differentiates the message based on the separator.

Advantages: It is suitable for variable-length messages and is simple and easy to implement. Disadvantages: The message content cannot contain delimiters, or the delimiters need to be escaped.
Example: Using \n as the message separator, send "Hello\nWorld\n" and the receiver splits the message based on \n.

(3) Length Field Protocol

Each message is preceded by a field that indicates the length of the message, and the receiver reads the length field first, and then reads the full message based on the length field.

Pros: Flexible and efficient, with the ability to know exactly the size of each message. Disadvantages: You need to process the parsing of length fields, which increases the complexity of the protocol.
Example: Send a 4-byte integer to represent the message length, and then send the actual message content. For example:

[0x00 0x00 0x00 0x05] "Hello" [0x00 0x00 0x00 0x05] "World"1.

(4) Based on the application layer protocol

Message boundaries are handled using existing application-layer protocols such as HTTP, Protobuf, JSON-RPC, etc., which typically already define their own message formats and parsing methods.

Pros: Leverage existing and proven protocols, reducing development effort.

Disadvantages: It may increase the complexity and overhead of protocol parsing.

5. Code Samples

Here's an example of a simple sticky and unpacking process based on the length field protocol (using Python as an example).

(1) Sender

import socket
import struct

def send_message(sock, message):
    # 将消息编码为字节
    encoded_message = message.encode('utf-8')
    # 获取消息长度
    message_length = len(encoded_message)
    # 使用 struct 打包长度为 4 字节的网络字节序
    sock.sendall(struct.pack('!I', message_length) + encoded_message)

# 示例使用
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('localhost', 12345))
send_message(sock, "Hello")
send_message(sock, "World")
sock.close()1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.

(2) Receiver

import socket
import struct

def recv_message(sock):
    # 首先接收 4 字节的长度
    raw_length = recvall(sock, 4)
    if not raw_length:
        return None
    message_length = struct.unpack('!I', raw_length)[0]
    # 接收实际的消息内容
    return recvall(sock, message_length).decode('utf-8')

def recvall(sock, n):
    data = b''
    while len(data) < n:
        packet = sock.recv(n - len(data))
        if not packet:
            return None
        data += packet
    return data

# 示例使用
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('localhost', 12345))
sock.listen(1)
conn, addr = sock.accept()
with conn:
    while True:
        message = recv_message(conn)
        if message is None:
            break
        print("Received:", message)
sock.close()1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.

6. Summary

TCP, as a streaming protocol, does not have a built-in message boundary mechanism, which leads to the problem of sticky and unpacket.
Sticky packets are when multiple messages are merged into a single packet, and unpacket is when a message is split into multiple packets.
The key to solving the problem of sticking and unpacking lies in the design of the application layer protocol, which defines the boundaries of messages by means of fixed length, separators, or length fields.

In practical applications, choosing the appropriate protocol design method can effectively avoid the problems caused by sticking and unpacking, and ensure the correct transmission and analysis of data.

By understanding the streaming characteristics of TCP and the principles of packet sticking and unpacking, developers can design appropriate application-layer protocols to achieve stable and reliable network communication.

新聞