In the early days of the web, people sent files only a few KB in size. By 2023, we're enjoying high-resolution MB-level images and watching 4K (and soon 8K) video in gigabytes.

Even with a good internet connection, downloading a 5GB file still takes some time. If you own an Xbox or PlayStation, you know this feeling.

We have three ways to reduce the time it takes to send large amounts of data over HTTP:

• Compress data
• Send chunked data
• Request data within a selection range

They are not mutually exclusive. You can use all methods together depending on your use case.

Compress data

1*_un0bHBemgCSDocQmucK5Q.png

To compress data, we need compression algorithms.

When sending a request, the browser includes a header called Accept-Encoding, which contains a list of supported compression algorithms, including gzip (GZIP), compress, deflate, and br (Brotli).

Next, the server selects its supported algorithm from the list and sets the algorithm name in the Content-Encoding header.

When the browser receives the response, it knows how to parse the data in the body.

Among these algorithms, the most popular is GZIP. It is an excellent choice for compressing text data such as HTML, CSS and JavaScript.

Brotli is another algorithm worth mentioning. It performs even better than GZIP at compressing HTML.

These efficient algorithms have some limitations.

They work well for compressing text, but are not sufficient for compressing images or videos. After all, the medium is already optimized.

Try compressing a video file on your computer. You can hardly see much difference before and after compression.

Furthermore, it is almost impossible to compress a 5GB video to a few KB without losing quality.

Compression is good, but we need a better solution - send the file in chunks and assemble the partial data on the client side.

Send chunked data

1*0WLNkzfgw9faLpTUXkk3tg.png

In version 1.1, HTTP introduced chunked data to handle big data situations.

When sending the response, the server adds a header Transfer-Encoding: chunked to let the browser know that the data is transmitted in chunks.

1*Nwlp0QqhEsvWl4fw-x0X7Q.png

Each chunk of data has the following components:

• A length block marker that marks the length of the current chunked data
• Chunked data block
• CRLF delimiter at the end of each block

Wondering what CRLF is?

1*s_-5lmT9176ymCAaaGCE2w.png

CR followed by LF (CRLF, \r\n, or 0x0D0A) moves the cursor to the next line, then to the beginning of the line. You can find more details in the Further reading section at the end of this article. Here you can simply think of it as a separator.

The server continues streaming chunked data to the browser. When the end of the data stream is reached, it appends an end tag containing the following part:

• A block of length with number 0 and CRLF at the end
• an additional CRLF

On the browser side, it waits for all chunks of data until the end tag is reached. Then, it removes the chunked encoding, including CRLF and length information.

Next, it combines the chunked data into a whole. Therefore, on Chrome DevTools, you only see assembled data, not chunked data.

Eventually, you receive a chunk of the entire data.

1*oChWIlysG3PQD3vy8ctVxw.png

Chunking data is useful. However, for a 5GB video, the complete data will still take some time to arrive.

Can we get selected chunks of data and request additional chunks when needed?

HTTP says yes.

Request data within selected range

1*LOGONes_KpmSN6zXaz9DhA.png

Open a video on YouTube and you'll see a gray progress bar moving forward.

What you just saw is YouTube requesting data for a selected range.

This feature allows you to jump anywhere in the timeline. When you click somewhere on the progress bar, the browser requests a specific range of video data.

Implementing range requests on the server is optional. If implemented, you can see Accept-Ranges: bytes in the response header.

1*MWd4AGP8lLRIQw5mketXew.png

Here is an example of a YouTube request. You can find this header in any "playback" request.

The range request header looks like `Range:bytes=0-80`, which is a 0-based index.

This head is a very cleverly designed head with excellent flexibility.

Suppose a data has a total of 100 bytes.

• Range: bytes=20 requests the range from 20 to the end, which is equal to Range: bytes=20-99.
• Range: bytes=-20 The last 20 bytes of the requested data are equal to Range: bytes=80-99.

If the requested range is valid, the server will send a response with a Content-Range header, verifying the data range and total length, for example Content-Range: bytes 70-80/100.

Range requests are widely used in video streaming and file download services.

Have you ever continued a file download after an internet outage? That's a range request.

Additionally, range requests support multiple ranges.

For example, you can request two ranges from a file, such as Range: bytes=20-45, 70-80.

Multi-range bodies look similar to chunked data. Each data block has the following parts:

• A boundary block, identifying the boundaries of different data blocks, starting with -- and ending with CRLF
• The two headers, Content-Type and Content-Range, display the attributes of the corresponding data block and end with CRLF
• An extra CRLF to tell the client that the real data is coming
• Finally, the data block ending with CRLF

The boundary is just a random string that looks like 3d6b6a416f9b5, marking the boundary between different chunks of data.

Finally, the body ends in a boundary block, starting with -- and ending with -- and CRLF. This section tells the browser that the multipart has ended.

Let's bring it all together. The structure of the response body is as follows.

Summarize

HTTP helps us transfer large amounts of data through compression, chunked data, and range data.

The idea here is to send the data we need when needed, and then other data when needed. You can try the same idea when you encounter problems designing similar systems.

By combining these three methods, we can send compressed chunked data range data.