Lab 07 · Node Runtime Labs

Your own binary protocol gateway.

Build a local TCP gateway that accepts custom binary frames, parses partial socket chunks, rejects malformed clients, applies sink pressure, forwards events, exposes metrics, and writes reports from real load runs.

Read the phase plan Get the lab

16Phases

ExpertDifficulty

CoreModules only

Gateway commandsProtocol gateway session

01Format
$ npm run protocol
Prints the binary header and validation rules.
02Server
$ npm run server -- --port 41234
Starts the TCP listener with explicit flags.
03Client
$ npm run client -- --port 41234 --name user.login
Sends one valid frame through the parser.
04Pressure
$ npm run server -- --sink file --slow-sink-ms 10
Makes sink backpressure part of the run.
05Metrics
$ npm run gateway -- metrics --port 9090
Checks the runtime counters from the outside.

Print the frame layout, run the server, then stress the client path.

What this lab builds

The finished project is a protocol gateway.

Binary protocol surface

Define a fixed frame header, frame types, byte order, payload rules, checksum policy, protocol documentation, and generated layout output.

Encoder, decoder, and TCP tools

Build client tooling that writes frames, server parsing that reads them, and test paths that prove round trips, partial chunks, and combined buffers.

Bounded sink behavior

Forward normalized events to NDJSON files or HTTP targets while tracking queue pressure, drain events, retries, timeouts, drops, and socket pause/resume counts.

Metrics, load, and reports

Expose live gateway state, persist metrics samples, run multi-client load, track ACK latency when enabled, and generate Markdown plus HTML reports from actual run data.

Complete phase plan

Every phase in Lab 07.

Each phase adds one observable gateway capability and one artifact to inspect. Click any phase to expand it.

00Create the projectCreate the protocol gateway project shell with runnable server and client modes.

Tasks

Create the package manifest, set ESM mode, and state the Node.js v24 runtime floor.
Create src, client, data, reports, docs, and test folders so source files, generated data, protocol docs, and evidence stay separated.
Create the CLI entrypoint as the command router with overview output and a successful help path.
Add placeholder command modes for server, client, protocol, encode, decode, receiver, load, and report.
Add npm scripts for every gateway mode plus the test runner so the command surface stays stable through the whole lab.
Reject unknown overview flags with stderr output and a failure exit code.

Artifact

A runnable gateway shell with command modes, help output, source folders, report folders, and strict top-level flag handling.

01Define the binary protocolDefine the binary frame layout, publish it in docs/protocol.md, and print the layout from code.

Tasks

Create the protocol constants module with magic bytes, version, header size, frame type values, and max payload size.
Define event, heartbeat, ACK, and error frame types so routing does not depend on payload parsing.
Document offset, width, field name, type, meaning, and validation rule for every header field.
Choose big-endian numeric fields, unsigned 64-bit request ids, and UTF-8 JSON payloads for event frames.
Define checksum calculation over the header with the checksum field zeroed plus the payload bytes.
Generate protocol layout output from shared constants so docs, encoder, decoder, and tests stay aligned.

Artifact

A documented 24-byte binary frame contract with shared constants and generated protocol layout output.

02Encode client framesBuild a client-side encoder that turns an event object into one valid binary frame.

Tasks

Create the encoder module and expose one frame encoding function for later client tools.
Build event payloads from a name, request id, timestamp, and optional attributes.
Convert payload JSON to UTF-8 bytes once and use that byte length in the frame header.
Allocate the frame, write fixed header fields at the defined offsets, and copy payload bytes after the header.
Compute the checksum after frame bytes exist and then fill the checksum field.
Add an encode command that prints offsets, hex bytes, ASCII preview, computed length, and expected length.
Add encoder tests for header fields, payload placement, frame length, request id handling, and checksum stability.

Artifact

A client encoder, hex inspection path, and tests that prove the frame bytes match the protocol contract.

03Decode complete framesParse one complete binary frame from a Buffer and return the normalized event.

Tasks

Create the complete-frame decoder and return frame metadata, decoded payload, and consumed byte count.
Reject buffers shorter than the fixed header before reading any field.
Validate magic bytes, version, frame type, and flags with distinct protocol failure paths.
Validate payload length against the cap and verify checksum before accepting the frame.
Decode event, ACK, and error payloads as UTF-8 JSON, while keeping heartbeat handling separate.
Add a decode command that accepts copied hex and prints the decoded frame data.
Add round-trip tests that connect the client encoder and server decoder, including a bad-checksum case.

Artifact

A complete-buffer decoder with controlled protocol errors and round-trip coverage against encoded frames.

04Build the TCP serverAccept TCP socket connections and log connection lifecycle events.

Tasks

Create the server module and let the CLI start it without embedding socket logic in the router.
Add host, port, sink, and idle timeout flags with defaults visible in help output.
Listen with node:net and print the actual bound address after the listening event.
Create structured JSON logs for connection.open and connection.close events.
Track bytes read from every socket before frame parsing exists.
Handle server bind errors and per-socket errors so expected network failures become logged lifecycle events.

Artifact

A TCP server that accepts clients, records opens and closes, counts raw bytes, and reports bind or socket failures cleanly.

05Send frames from a TCP clientSend one encoded frame from a TCP client to the server.

Tasks

Create a send client that opens a real TCP connection to the gateway.
Reuse the frame encoder so the network client and hex tool produce the same protocol bytes.
Add host, port, event name, request id, count, and payload size flags.
Write the encoded frame to the socket and end the connection after the write callback.
Compare the client frame length with the server's per-connection byte count.

Artifact

A working end-to-end byte path from client encoder to TCP server with matching sent and received byte totals.

06Handle partial packetsParse frames split across multiple TCP chunks.

Tasks

Create the connection parser module with per-connection chunk input.
Store carry bytes, frames read, bytes read, last activity, and protocol error count on each connection.
Merge carry bytes with new chunks only when an incomplete tail exists.
Wait for a complete header before reading payload length, then wait for the full payload before decoding.
Emit normalized decoded events back into the server path after a full frame exists.
Add chunk size and chunk delay client flags to stress arbitrary socket boundaries.
Add deterministic tests that feed one encoded frame one byte at a time.

Artifact

A stateful parser that survives arbitrary chunk boundaries and emits exactly one event for one complete frame.

07Handle multiple frames in one chunkParse several frames from one received Buffer.

Tasks

Update the parser loop so it keeps decoding while the buffer contains another complete frame.
Preserve only the unconsumed incomplete tail after decoded frames are released.
Add client modes for count, burst, and same-write sends.
Track frames received, events emitted, and frames per connection on the server.
Add tests for two complete frames in one buffer and one-and-a-half frames across two pushes.

Artifact

A parser that handles coalesced frames, preserves incomplete tails, and reports frame counts that match client sends.

08Validate malformed inputReject bad magic bytes, unsupported versions, invalid lengths, unknown frame types, and checksum failures.

Tasks

Create protocol error classes with stable error names and codes.
Route decoder and parser failures through the same protocol error shape.
Add max protocol error and malformed policy flags for close or quarantine behavior.
Enforce close and quarantine behavior in the connection path, with one logged failure shape.
Add malformed client modes for bad magic, bad version, huge length, bad checksum, and bad JSON.
Append protocol failures to protocol-errors JSONL evidence.
Test every error class and malformed frame mode.

Artifact

A gateway that rejects malformed frames through controlled errors while keeping the server process available.

09Add connection lifecycle managementTrack connection state, idle timeouts, and close reasons.

Tasks

Create a connection state module with helpers for creation, updates, and finalization.
Store connection id, remote address, timestamps, bytes, frames, protocol errors, pause counts, and close reason.
Use socket idle timeouts and make the timeout handler choose a close reason.
Finalize each connection once across normal close, error close, protocol close, and timeout close.
Append close records to connections JSONL and update in-memory metrics.
Add idle client testing with a hold-open option.

Artifact

Connection reports with close reason, duration, counters, error counts, and idle-timeout behavior.

10Add a backpressure-aware file sinkWrite normalized events to disk while respecting writable stream backpressure.

Tasks

Create a file sink module with a write-event boundary and close path.
Normalize decoded frames into event records with receive time, connection id, request id, type, name, attributes, and payload size.
Write one normalized event per line into NDJSON output.
Pause socket reads when the file stream signals backpressure and resume after drain.
Track file writes, drain count, buffered bytes, socket pauses, socket resumes, and sink write latency.
Add a slow-sink option so pressure can be observed in small local runs.

Artifact

A disk sink that bounds JavaScript buffering and reports file pressure through pause, resume, drain, and latency counters.

11Add HTTP forwarding modeForward normalized events to a local HTTP endpoint with concurrency limits, timeouts, retries, and failure counts.

Tasks

Create an HTTP sink with the same write-event and close shape as the file sink.
Add target URL, concurrency, timeout, retry, queue limit, and overflow policy flags.
Build a local HTTP receiver that accepts event posts and stores received JSON lines.
Forward with bounded concurrency and timeout-controlled requests.
Retry transient failures, count timeouts, and treat client-side HTTP failures as final.
Apply the overflow policy when the HTTP queue reaches its configured limit.
Report periodic HTTP sink snapshots for queue depth, in-flight work, delivery counts, and failures.

Artifact

An HTTP forwarding path with bounded queues, retry accounting, timeout accounting, and visible outbound pressure.

12Add per-connection metricsExpose runtime metrics while the gateway is running.

Tasks

Create a metrics module for counters, gauges, latency samples, and snapshots.
Start a metrics HTTP server from the gateway process on a configurable port.
Return gateway-wide metrics for uptime, connections, bytes, frames, events, protocol errors, pause/resume counts, sink pressure, HTTP state, latency, and memory.
Include per-connection snapshots with bytes, frames, last activity, and paused state.
Persist one metrics sample per second to metrics JSONL for later reporting.
Add an optional metrics CLI that fetches and prints the metrics endpoint.

Artifact

Live and persisted metrics that line up with small load tests, sink counters, and connection state.

13Add graceful shutdownStop accepting new connections, drain existing work, close sockets according to policy, and write a final report.

Tasks

Create a shutdown module that coordinates listener, sockets, timers, metrics server, sinks, and reports.
Handle SIGINT and SIGTERM through one cleanup path.
Stop accepting new connections, stop metrics sampling, close the metrics server, and stop reading new frames.
Add shutdown grace, drain-or-close policy, and ACK policy flags.
Define ACK frame behavior for accepted events, persisted events, disabled ACKs, sink failures, dropped events, and malformed input.
Await sink close, close or destroy sockets by policy, and force close remaining sockets after the deadline.
Write a final report with shutdown timing, reason, active connections, dropped frames, drained events, forced closes, and sink close time.

Artifact

A final shutdown report that records what drained, what closed, what dropped, and which resources were forced closed.

14Build a load test clientStress the gateway with many connections, frame counts, frame rates, payload sizes, and malformed ratios.

Tasks

Create a load client that opens one socket per configured client.
Add flags for host, port, clients, frames, frame size, rate, malformed ratio, chunk size, duration, ACK timeout, report path, and metrics URL.
Generate deterministic request ids that stay unique across all clients.
Mix valid and malformed frames in one run using the configured malformed ratio.
Measure ACK latency only when ACK frames are observed, and parse ACK or error responses per socket.
Track connected state, frames sent, bytes sent, ACKs, ACK timeouts, injected protocol errors, and socket errors per client.
Poll gateway metrics during load when a metrics URL is configured.
Write a load report with throughput, latency, errors, memory, and backpressure data.

Artifact

A load report that ties client-side throughput and errors to gateway-side metrics, pressure, and malformed-frame policy.

15Generate the final protocol reportProduce a Markdown and HTML report that summarizes protocol design, parser behavior, error policy, backpressure behavior, shutdown behavior, and load results.

Tasks

Create a report generator and connect it to the report command.
Read protocol docs, latest load report, final shutdown report, connection JSONL, protocol error JSONL, and metrics JSONL.
Summarize header layout, frame types, parser state, and malformed input policy.
Summarize connection lifecycle, file sink backpressure, HTTP forwarding, metrics during load, and graceful shutdown.
Cite measured totals for frames, accepted events, malformed frames, protocol errors, throughput, latency, active connections, file buffering, HTTP queue depth, and dropped frames.
Render Markdown and HTML reports for inspection and sharing.

Artifact

A final protocol gateway report in Markdown and HTML, backed by actual load, metric, connection, error, and shutdown data.

Protocol evidence

The reports prove the gateway behavior.

Lab 07 turns protocol rules, connection state, malformed input, sink pressure, HTTP delivery, metrics, shutdown, and load results into files a developer can inspect after the run.

docs/protocol.mdwire format, frame types, and validation rules

data/events.ndjsonnormalized file-sink event records

data/http-received.ndjsonevents accepted by the local receiver

connections.jsonlconnection lifecycle and close reasons

protocol-errors.jsonlmalformed frame evidence by error code

metrics.jsonlgateway samples captured during runtime

load-latest.jsonthroughput, latency, errors, and pressure

protocol-gateway.htmlfinal human-readable gateway report

Choose your
NodeBook package.

Buy a single volume or lock in every volume at once. Switch between one-reader pricing and team licenses for up to 25 members.

Choose volume

Individual pricing is for one reader and one personal purchase record.

Downloadable book bundle

Digital Bundle

Volume I as EPUB, light and dark PDFs, slides, cheatsheets, and future updates.

$19.99$49.99

One-time purchase

Volume I EPUB for offline reading
Light and dark PDF editions
Slide decks for chapter review
Cheatsheets for quick lookup
Future Digital Bundle updates
Lifetime access to the files

Get Digital Bundle

This is the downloadable Volume I study bundle. It does not include Node Runtime Labs.

Best value

Everything for this volume

NodeBook Pro

Volume I Labs plus its downloadable bundle in one purchase. Save $9.99 vs buying the Digital Bundle and Labs separately.

$49.99$99.99

One-time purchase

Node Runtime LabsDigital Bundle

Everything in Node Runtime Labs
Everything in the Digital Bundle
Volume I labs and book bundle
Future updates for both products
Lifetime access to purchased files

Get NodeBook Pro

Includes both paid products for Volume I.

Premium labs

Complex runtime projects

Node Runtime Labs

Volume I long-form builds with checkpoints, hints, debugging notes, and expected output.

$39.99$79.99

One-time purchase

Volume I runtime lab projects
Phase-by-phase build instructions
Checkpoints, hints, and rubrics
Debugging notes and expected output
Reflection questions
Future lab updates

Get Labs Bundle

This is the paid labs bundle for Volume I. It does not include EPUB, PDFs, slides, or cheatsheets.

See complete pricing breakdown

Other labs in the bundle

Lab 07 sits with six other runtime builds.

The bundle includes seven runtime projects covering process observation, binary storage, streams, module resolution, file watching, async orchestration, and custom protocols.

Lab 01