Lesson 1.4
The 40 ms floor - Nagle and delayed ACK
A kernel-level latency floor which never shows up in code review - why it happens, how to see it yourself, and the one line that already defeats it.
- Explain Nagle's algorithm and delayed ACK, and how the two of them interlock into a latency floor.
- Reproduce the ~40 ms floor on Linux and show that it vanishes with setNoDelay(true).
- Explain why a raw net socket needs this but an http.Server socket doesn't.
The previous lesson changed the bytes in a response. This lesson keeps those bytes valid and instead measures a delay that TCP introduces. Latency is the elapsed time for one operation - here it means the time from writing one request till the complete response arrives. This delay can reach tens of milliseconds on Linux, when Nagle's algorithm interacts with delayed acknowledgements. And this is exactly why setNoDelay(true) was the first line of your connection handler.
Two policies that are each reasonable alone
Nagle's algorithm is a TCP sending policy described in RFC 896. If a connection has already sent some data which the peer hasn't acknowledged yet, the kernel can hold back another small write. It sends the held bytes after an acknowledgement arrives, or after enough bytes accumulate for a full TCP segment. Combining several small writes into a larger segment is called coalescing. This policy reduces protocol overhead, but it can increase latency.
Delayed ACK is a receiving policy described in RFC 1122. ACK is short for acknowledgement. Instead of acknowledging every segment immediately, the receiver may wait for a short time - either for another segment, or for some response data on which it can include the ACK. Including the acknowledgement along with outgoing data is called piggybacking.
Each policy is sensible on its own. But put one on each end of the same connection, and they get in each other's way.
How the two policies interlock
Think of a server that writes its response in two small pieces - the head first and then the body. That's quite a natural way to write a response writer. Here is what the two policies do with it.
Figure 1.3 - Nagle holds the small body write while the peer delays its acknowledgement, creating a latency floor even though both policies are reasonable on their own.
The server's kernel holds the body while it waits for an ACK of the head. The client's kernel delays that ACK while it waits for more data. Both sides keep waiting for each other, till the delayed-ACK timer finally expires and the body leaves. Throughput is the number of completed operations in a unit of time. A repeated delay like this reduces throughput, even though every response stays valid.
socket.setNoDelay(true) sets the TCP_NODELAY socket option, which disables Nagle for that socket. A socket option is a setting which the operating system stores with one socket. With this option enabled, the second small write can leave even while earlier data is unacknowledged. A later chapter also assembles each response into one buffer, so the framework can do just one write operation.
Why you must do this and node:http does not
In Node 26, the http.Server constructor defaults its noDelay option to true. It disables Nagle immediately after receiving a connection. A raw net.Server doesn't do this for accepted sockets, so your Server has to call socket.setNoDelay(true) itself.
The operating system enables Nagle on a TCP connection by default. Calling setNoDelay(true) changes that default. Your connection callback does it before registering the other listeners, so every later write uses the selected policy.
A note on platforms
Delayed-ACK behaviour depends on the operating system and the network interface. macOS loopback often doesn't produce any visible delay in this experiment. Loopback is the virtual network interface used when a computer connects to itself through 127.0.0.1.
If both your macOS runs produce similar numbers, run the experiment in a Linux container. A container runs a process with an isolated filesystem and network environment while sharing the host kernel. Docker can start the Node 26 Linux image and mount the current project directory at /work.
docker run --rm -it -v "$PWD:/work" -w /work node:26-slim bash--rm deletes the container after it exits. -it attaches your terminal. -v makes the current directory available inside the container. -w selects the working directory.
Build a small experiment that isolates the mechanism. It has two pieces.
- A two-write demo server on a raw
netsocket which, for each request, writes the response head and the body in two separatesocket.writecalls. Gate the NODELAY setting on an environment variable i.e callsocket.setNoDelay(true)only when, say,process.env.NODELAY === "1". This way the same code path runs both times, and the only thing that changes is the kernel policy. - A persistent-connection timing client that opens one connection, sends a request, waits for all 52 response bytes, records the request-to-response time, and then sends the next request. An HTTP persistent connection carries several request and response exchanges over one TCP connection. Print the arithmetic mean i.e the sum of the recorded times divided by their count.
Don't treat one client 'data' event as one response. Count the bytes across events till all 52 bytes have arrived. The client sends its next request only after that count reaches 52, so bytes from two responses can never share the counter. This keeps the experiment consistent with TCP's lack of message markers.
Run the server once with NODELAY=0 and once with NODELAY=1, driving each run with the client.
Reveal these one at a time, and only when you are genuinely stuck. Each one tells you a little more than the last - none of them is the answer.
The verification produces two Linux measurements from the same server code.
NODELAY=0- a larger mean round-trip, commonly in the tens of milliseconds for this Linux experiment.NODELAY=1- a much smaller mean round-trip on the same machine.
Record the operating system and both values. Exact times vary with the kernel and the machine, so compare the two runs instead of expecting one fixed number. Your real src/server.js sets setNoDelay(true) unconditionally.
- Both runs are fast on your machine. You are almost surely on macOS loopback, where the heuristic hides the floor. Use a Linux container to see it, or just trust the mechanism.
- You wrote the response in one
writeand saw no difference. Obviously. The floor needs the two-write (head then body) pattern, and that's why the demo splits the write on purpose. - A fresh connection per request shows noise instead of a clean floor. Reuse one connection across the whole timing loop. The floor is a property of an established connection with data in flight.
- You concluded that Nagle is "bad" and should be off everywhere. No, it's a reasonable default for many workloads. For a request/response server it is wrong, and that's why you disable it per socket.
The chapter ends with a Server that binds and closes, a 52-byte HTTP response, three recorded framing failures, and a two-run Nagle measurement. Chapter 2 starts parsing the request bytes.