How Go's net/http Package Manages TCP Connections

Preface

Before we start, let's briefly discuss two concepts—Socket and file descriptor—for easier understanding later.

What is a Socket?

A Socket is the fundamental abstraction for network communication. It provides a standard interface for applications to access the network protocol stack. Simply put, a Socket is an endpoint for network communication, allowing programs on different computers to exchange data over the network.

Main features of Sockets:

It serves as the interface between the application layer and the transport layer.
It can be regarded as a special type of file that supports read and write operations.
There are different types: TCP Sockets (connection-oriented), UDP Sockets (connectionless), etc.

What is a File Descriptor?

A file descriptor is an integer value used by the operating system to identify and manage open files. In Unix/Linux systems, everything is considered a file, including regular files, directories, devices, and even network connections.

Key points about file descriptors:

It is a non-negative integer, usually starting from 0 (0 is standard input, 1 is standard output, 2 is standard error).
In the OS kernel, the file descriptor is an index pointing to a file table entry.
Each process has its own file descriptor table.

Relationship Between Sockets and File Descriptors

In Unix/Linux systems, Sockets are also regarded as a special kind of file, so they also have corresponding file descriptors. When you create a Socket:

The operating system allocates a file descriptor.
This file descriptor can be used for subsequent network operations (read, write, close, etc.).
The application interacts with the Socket through this file descriptor.

TCP Connection Establishment Process

Socket Creation

// Implementation inside the net package
fd, err := socket(family, syscall.SOCK_STREAM, syscall.IPPROTO_TCP)

This step creates a socket file descriptor through a system call.

Server Binding (Bind) and Listening (Listen)

// Simplified server-side code flow
bind(fd, addr)
listen(fd, backlog)

The server binds the socket to a specific address and port, and then starts listening for connection requests.

Accepting Connections (Accept)

// net/http/server.go simplified version
func (srv *Server) Serve(l net.Listener) error {
    for {
        rw, err := l.Accept()  // Accept a new connection
        if err != nil {
            // Handle the error
            continue
        }
        go srv.newConn(rw).serve(ctx)  // Create a new goroutine for each connection
    }
}

Client Connection (Connect)

// net/http/transport.go simplified version
func (t *Transport) dialConn(ctx context.Context, addr string) (*conn, error) {
    // Create a TCP connection
    netConn, err := t.dial(ctx, "tcp", addr)
    if err != nil {
        return nil, err
    }
    // Wrap it as an HTTP connection
    return &conn{
        conn: netConn,
        // ... other fields
    }, nil
}

Data Transmission:

// Read data
n, err := syscall.Read(fd, buf)
// Write data
n, err := syscall.Write(fd, data)

Closing the Connection:

err := syscall.Close(fd)

Key Implementation Details

Multiplexing

HTTP/1.1 uses the Keep-Alive mechanism to reuse TCP connections.
HTTP/2 implements multiplexing through streams, allowing multiple HTTP requests to share a single TCP connection.

Connection Pool Management

// net/http/transport.go
type Transport struct {
    // Idle connection pool
    idleConn     map[connectMethodKey][]*persistConn
    // Maximum number of idle connections
    maxIdleConns int
    // ... other fields
}

Timeout Control

// Set connection timeout
conn.SetDeadline(time.Now().Add(timeout))

Error Handling and Retry Mechanism

// Simplified retry logic
for retry := 0; retry < maxRetries; retry++ {
    conn, err := dial()
    if err == nil {
        return conn
    }
    // Wait and retry
    time.Sleep(backoff)
}

Workflow

When the client initiates an HTTP request:

First, check whether there is an available connection in the connection pool.
If not, create a new TCP connection.
Send the HTTP request data.
Wait for and read the response.

When the server processes a request:

The Accept loop receives new connections.
Create a goroutine for each connection.
Parse the HTTP request.
Process the request and return the response.

This is the core mechanism by which the net/http package implements HTTP connections on top of the TCP protocol. Through abstraction and encapsulation, developers do not need to directly handle the underlying details of TCP connections, while efficient connection management and reuse mechanisms are provided.

We are Leapcell, your top choice for hosting Go projects.

Leapcell is the Next-Gen Serverless Platform for Web Hosting, Async Tasks, and Redis:

Multi-Language Support

Develop with Node.js, Python, Go, or Rust.

Deploy unlimited projects for free

pay only for usage — no requests, no charges.

Unbeatable Cost Efficiency

Pay-as-you-go with no idle charges.
Example: $25 supports 6.94M requests at a 60ms average response time.

Streamlined Developer Experience

Intuitive UI for effortless setup.
Fully automated CI/CD pipelines and GitOps integration.
Real-time metrics and logging for actionable insights.

Effortless Scalability and High Performance

Auto-scaling to handle high concurrency with ease.
Zero operational overhead — just focus on building.

Explore more in the Documentation!