Eliminating Network Socket Leaks in Parallel Test Execution

A test suite running across twelve parallel workers passes nine times in CI and fails on the tenth. The failure output shows EADDRINUSE on an OS-assigned ephemeral port, or a client handshake that hangs until timeout. Run the suite locally with a single worker, and every test passes.

The standard response is to label the suite as suffering from flaky tests and add retry logic to the CI job. Retries mask the failure mode without addressing it. In most parallel test suites, assertion logic is not the source of non-determinism. The failure occurs because parallel processes leak socket handles into shared operating system state.

Port Zero and the TIME_WAIT Trap

The common pattern for parallel network tests is binding a server to port 0. The operating system kernel selects an unused ephemeral port from its range, returns the file descriptor, and avoids hardcoded port collisions.

This approach fails under high concurrency because closing a TCP socket is not instantaneous. When a test process shuts down a TCP connection, the socket enters the TIME_WAIT state to ensure late-arriving packets are not misrouted to a subsequent connection. On Linux, TIME_WAIT typically lasts 60 seconds.

When hundreds of tests spin up and tear down local HTTP or gRPC servers in seconds, ephemeral port pools are exhausted rapidly. The kernel begins reallocating port numbers that still have sockets lingering in TIME_WAIT. If an incoming test binds immediately, or if lingering child processes inherit open socket file descriptors across a fork(), the new listener either fails to bind or receives stray traffic meant for the previous test run.

// Anti-pattern: Relying solely on port 0 leaves socket state to the OS kernel
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
    t.Fatalf("failed to bind: %v", err)
}
defer ln.Close() // Leaves lingering TIME_WAIT state in the host networking stack

When building repeatable software, relying on the host OS network stack to clean up state between parallel subroutines creates an uncontrolled ambient dependency.

Isolating Transports to In-Memory Boundaries

To eliminate socket-level flakes, eliminate TCP from tests that do not strictly require kernel-level network validation.

Most integration tests verify serialization, routing, middleware execution, and handler responses. These layers do not require an active TCP stack. Replacing OS network bindings with in-memory duplex streams eliminates port exhaustion, firewall prompts, and TIME_WAIT accumulation entirely.

// Pattern: In-memory listener without host port allocation
type MemoryListener struct {
    net.Listener
    conns chan net.Conn
}

func NewMemoryListener() *MemoryListener {
    return &MemoryListener{conns: make(chan net.Conn, 256)}
}

func (l *MemoryListener) Dial() (net.Conn, error) {
    serverSide, clientSide := net.Pipe()
    l.conns <- serverSide
    return clientSide, nil
}

If tests require actual file descriptor semantics—such as verifying keep-alive behavior, connection pool resets, or low-level TLS handshakes—use isolated Unix domain sockets with unique paths per test worker instead of global TCP ports:

socketPath := filepath.Join(t.TempDir(), "worker.sock")
ln, err := net.Listen("unix", socketPath)

Because Unix domain sockets are bounded by the filesystem hierarchy, cleaning up the temporary directory removes the transport address immediately, with no lingering state in the kernel's TCP tables.

Enforcing Socket Lifecycles in the Harness

Where full TCP testing against localhost is unavoidable, network interfaces must be treated as explicit, bounded lifecycles managed directly by your test harness.

Three rules prevent network bleed between parallel test routines:

1. Explicit socket tracking. A server abstraction must track every active client connection and forcibly close them (Close() or SO_LINGER = 0) before the listener itself closes. A listener that closes while connections remain open leaves orphan sockets alive in background workers.
2. Worker-scoped loopback addresses. Linux maps the entire 127.0.0.0/8 block to the loopback interface. Rather than having all workers bind to 127.0.0.1, assign each parallel worker a distinct loopback address derived from its worker ID (such as 127.0.0.2 through 127.0.0.254). Because the TCP tuple includes the destination IP, TIME_WAIT sockets on worker 2 cannot collide with port allocations on worker 3.
3. Process-group descriptor reclamation. Sockets remain open in the kernel as long as any running process holds the file descriptor. Child processes spawned during tests must run inside their own process groups. On test completion or timeout, the harness must signal the entire process group to ensure no orphan process keeps an inherited socket alive.