Experiments and implementations · Introductory

Recover messages from a byte stream with length framing

Use a local stream socket pair to implement length-prefixed messages, handle incomplete reads, and reject oversized payloads.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Test framing rather than Internet TCP packets
  2. Read a length and then the declared body
  3. Recover two messages from one send
  4. Identify the remaining production requirements
  5. Ask Logeia about this articleCheck your understanding

Test framing rather than Internet TCP packets

TCP presents a byte stream to an application. The number of send calls is not a contract for the number of recv calls or for message boundaries. This experiment uses socket.socketpair locally to test the application-side handling of that property. It does not capture packets on a real network TCP connection.

A socket pair provides connected endpoints on one machine, with implementation details depending on the platform. Here it is a local stream surrogate. It does not reproduce retransmission, congestion control, or path delay. The narrower setup lets the example focus on framing without exposing an external listening service.

Read a length and then the declared body

The protocol places a four-byte body length before each body, using network byte order for the integer. The recvn function loops until it has the requested number of bytes. It does not assume that a single recv fills the request or preserves a sender call boundary.

Treat a received length as untrusted input and enforce a payload limit before proceeding. End of stream inside a header or body is an error, not a successful empty message. An explicitly declared zero-length body is different and can be represented without ambiguity.

python
import socket
import struct

MAX_PAYLOAD = 1024 * 1024

def recvn(sock, size):
    data = bytearray()
    while len(data) < size:
        part = sock.recv(size - len(data))
        if not part:
            raise EOFError("Stream ended inside a frame")
        data.extend(part)
    return bytes(data)

def receive_frame(sock):
    size = struct.unpack("!I", recvn(sock, 4))[0]
    if size > MAX_PAYLOAD:
        raise ValueError("Frame exceeds configured limit")
    return recvn(sock, size)

left, right = socket.socketpair()
with left, right:
    left.settimeout(2)
    right.settimeout(2)
    messages = [b"hello", b"world"]
    wire = b"".join(struct.pack("!I", len(m)) + m for m in messages)
    left.sendall(wire)
    received = [receive_frame(right), receive_frame(right)]
    assert received == messages
    print(received)

Recover two messages from one send

The sender concatenates the length-prefixed hello and world bodies and invokes sendall once. The receiver recovers two bodies by reading two protocol frames. The protocol’s lengths, rather than the call boundaries in sender code, define where one message ends and another starts.

Successful execution does not prove that recv happened to split the data into small pieces. Data may arrive together in this run. The important property is that the reading structure handles either case. Use an additional controlled fake source if a test must force particular fragment sizes or an exact sequence of incomplete reads.

Identify the remaining production requirements

The example sets socket timeouts and closes both endpoints through context managers. It does not implement a whole-message deadline against a peer sending slowly, admission limits, authentication, encryption, or protocol version negotiation. The timeout of each individual call and the deadline for a complete operation are separate decisions.

Correct framing also does not prove meaningful content. If the recovered body is decoded as JSON, validate the syntax and required fields at that layer. Test maximum sizes, truncated input, empty bodies, and consecutive frames. Make the protocol contract explicit before adding more features to its implementation.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 10

Choose a learning path