thenodebook.com
Lab 07 / Node Runtime Labs

Custom Binary Protocol Gateway.

Build a local TCP gateway that accepts custom binary frames, parses partial socket chunks, rejects malformed clients, applies sink pressure, forwards events, exposes metrics, and writes reports from real load runs.

Read the phase planGet the lab
Gateway commandsProtocol gateway session
  • 01Format
    $ npm run protocol

    Prints the binary header and validation rules.

  • 02Server
    $ npm run server -- --port 41234

    Starts the TCP listener with explicit flags.

  • 03Client
    $ npm run client -- --port 41234 --name user.login

    Sends one valid frame through the parser.

  • 04Pressure
    $ npm run server -- --sink file --slow-sink-ms 10

    Makes sink backpressure part of the run.

  • 05Metrics
    $ npm run gateway -- metrics --port 9090

    Checks the runtime counters from the outside.

Print the frame layout, run the server, then stress the client path.
16Phases
ExpertDifficulty
Core modulesPackage scope
What this lab builds

The finished project is a protocol gateway.

Binary protocol surface

Define a fixed frame header, frame types, byte order, payload rules, checksum policy, protocol documentation, and generated layout output.

Encoder, decoder, and TCP tools

Build client tooling that writes frames, server parsing that reads them, and test paths that prove round trips, partial chunks, and combined buffers.

Bounded sink behavior

Forward normalized events to NDJSON files or HTTP targets while tracking queue pressure, drain events, retries, timeouts, drops, and socket pause/resume counts.

Metrics, load, and reports

Expose live gateway state, persist metrics samples, run multi-client load, track ACK latency when enabled, and generate Markdown plus HTML reports from actual run data.

Complete phase plan

Every phase in Lab 07.

The build starts with a clean command surface and ends with a generated protocol report. Each phase adds one observable gateway capability and one artifact to inspect.

00

Create the project

Create the protocol gateway project shell with runnable server and client modes.

  • Create the package manifest, set ESM mode, and state the Node.js v24 runtime floor.
  • Create src, client, data, reports, docs, and test folders so source files, generated data, protocol docs, and evidence stay separated.
  • Create the CLI entrypoint as the command router with overview output and a successful help path.
  • Add placeholder command modes for server, client, protocol, encode, decode, receiver, load, and report.
  • Add npm scripts for every gateway mode plus the test runner so the command surface stays stable through the whole lab.
  • Reject unknown overview flags with stderr output and a failure exit code.
A runnable gateway shell with command modes, help output, source folders, report folders, and strict top-level flag handling.
01

Define the binary protocol

Define the binary frame layout, publish it in docs/protocol.md, and print the layout from code.

  • Create the protocol constants module with magic bytes, version, header size, frame type values, and max payload size.
  • Define event, heartbeat, ACK, and error frame types so routing does not depend on payload parsing.
  • Document offset, width, field name, type, meaning, and validation rule for every header field.
  • Choose big-endian numeric fields, unsigned 64-bit request ids, and UTF-8 JSON payloads for event frames.
  • Define checksum calculation over the header with the checksum field zeroed plus the payload bytes.
  • Generate protocol layout output from shared constants so docs, encoder, decoder, and tests stay aligned.
A documented 24-byte binary frame contract with shared constants and generated protocol layout output.
02

Encode client frames

Build a client-side encoder that turns an event object into one valid binary frame.

  • Create the encoder module and expose one frame encoding function for later client tools.
  • Build event payloads from a name, request id, timestamp, and optional attributes.
  • Convert payload JSON to UTF-8 bytes once and use that byte length in the frame header.
  • Allocate the frame, write fixed header fields at the defined offsets, and copy payload bytes after the header.
  • Compute the checksum after frame bytes exist and then fill the checksum field.
  • Add an encode command that prints offsets, hex bytes, ASCII preview, computed length, and expected length.
  • Add encoder tests for header fields, payload placement, frame length, request id handling, and checksum stability.
A client encoder, hex inspection path, and tests that prove the frame bytes match the protocol contract.
03

Decode complete frames

Parse one complete binary frame from a Buffer and return the normalized event.

  • Create the complete-frame decoder and return frame metadata, decoded payload, and consumed byte count.
  • Reject buffers shorter than the fixed header before reading any field.
  • Validate magic bytes, version, frame type, and flags with distinct protocol failure paths.
  • Validate payload length against the cap and verify checksum before accepting the frame.
  • Decode event, ACK, and error payloads as UTF-8 JSON, while keeping heartbeat handling separate.
  • Add a decode command that accepts copied hex and prints the decoded frame data.
  • Add round-trip tests that connect the client encoder and server decoder, including a bad-checksum case.
A complete-buffer decoder with controlled protocol errors and round-trip coverage against encoded frames.
04

Build the TCP server

Accept TCP socket connections and log connection lifecycle events.

  • Create the server module and let the CLI start it without embedding socket logic in the router.
  • Add host, port, sink, and idle timeout flags with defaults visible in help output.
  • Listen with node:net and print the actual bound address after the listening event.
  • Create structured JSON logs for connection.open and connection.close events.
  • Track bytes read from every socket before frame parsing exists.
  • Handle server bind errors and per-socket errors so expected network failures become logged lifecycle events.
A TCP server that accepts clients, records opens and closes, counts raw bytes, and reports bind or socket failures cleanly.
05

Send frames from a TCP client

Send one encoded frame from a TCP client to the server.

  • Create a send client that opens a real TCP connection to the gateway.
  • Reuse the frame encoder so the network client and hex tool produce the same protocol bytes.
  • Add host, port, event name, request id, count, and payload size flags.
  • Write the encoded frame to the socket and end the connection after the write callback.
  • Compare the client frame length with the server's per-connection byte count.
A working end-to-end byte path from client encoder to TCP server with matching sent and received byte totals.
06

Handle partial packets

Parse frames split across multiple TCP chunks.

  • Create the connection parser module with per-connection chunk input.
  • Store carry bytes, frames read, bytes read, last activity, and protocol error count on each connection.
  • Merge carry bytes with new chunks only when an incomplete tail exists.
  • Wait for a complete header before reading payload length, then wait for the full payload before decoding.
  • Emit normalized decoded events back into the server path after a full frame exists.
  • Add chunk size and chunk delay client flags to stress arbitrary socket boundaries.
  • Add deterministic tests that feed one encoded frame one byte at a time.
A stateful parser that survives arbitrary chunk boundaries and emits exactly one event for one complete frame.
07

Handle multiple frames in one chunk

Parse several frames from one received Buffer.

  • Update the parser loop so it keeps decoding while the buffer contains another complete frame.
  • Preserve only the unconsumed incomplete tail after decoded frames are released.
  • Add client modes for count, burst, and same-write sends.
  • Track frames received, events emitted, and frames per connection on the server.
  • Add tests for two complete frames in one buffer and one-and-a-half frames across two pushes.
A parser that handles coalesced frames, preserves incomplete tails, and reports frame counts that match client sends.
08

Validate malformed input

Reject bad magic bytes, unsupported versions, invalid lengths, unknown frame types, and checksum failures.

  • Create protocol error classes with stable error names and codes.
  • Route decoder and parser failures through the same protocol error shape.
  • Add max protocol error and malformed policy flags for close or quarantine behavior.
  • Enforce close and quarantine behavior in the connection path, with one logged failure shape.
  • Add malformed client modes for bad magic, bad version, huge length, bad checksum, and bad JSON.
  • Append protocol failures to protocol-errors JSONL evidence.
  • Test every error class and malformed frame mode.
A gateway that rejects malformed frames through controlled errors while keeping the server process available.
09

Add connection lifecycle management

Track connection state, idle timeouts, and close reasons.

  • Create a connection state module with helpers for creation, updates, and finalization.
  • Store connection id, remote address, timestamps, bytes, frames, protocol errors, pause counts, and close reason.
  • Use socket idle timeouts and make the timeout handler choose a close reason.
  • Finalize each connection once across normal close, error close, protocol close, and timeout close.
  • Append close records to connections JSONL and update in-memory metrics.
  • Add idle client testing with a hold-open option.
Connection reports with close reason, duration, counters, error counts, and idle-timeout behavior.
10

Add a backpressure-aware file sink

Write normalized events to disk while respecting writable stream backpressure.

  • Create a file sink module with a write-event boundary and close path.
  • Normalize decoded frames into event records with receive time, connection id, request id, type, name, attributes, and payload size.
  • Write one normalized event per line into NDJSON output.
  • Pause socket reads when the file stream signals backpressure and resume after drain.
  • Track file writes, drain count, buffered bytes, socket pauses, socket resumes, and sink write latency.
  • Add a slow-sink option so pressure can be observed in small local runs.
A disk sink that bounds JavaScript buffering and reports file pressure through pause, resume, drain, and latency counters.
11

Add HTTP forwarding mode

Forward normalized events to a local HTTP endpoint with concurrency limits, timeouts, retries, and failure counts.

  • Create an HTTP sink with the same write-event and close shape as the file sink.
  • Add target URL, concurrency, timeout, retry, queue limit, and overflow policy flags.
  • Build a local HTTP receiver that accepts event posts and stores received JSON lines.
  • Forward with bounded concurrency and timeout-controlled requests.
  • Retry transient failures, count timeouts, and treat client-side HTTP failures as final.
  • Apply the overflow policy when the HTTP queue reaches its configured limit.
  • Report periodic HTTP sink snapshots for queue depth, in-flight work, delivery counts, and failures.
An HTTP forwarding path with bounded queues, retry accounting, timeout accounting, and visible outbound pressure.
12

Add per-connection metrics

Expose runtime metrics while the gateway is running.

  • Create a metrics module for counters, gauges, latency samples, and snapshots.
  • Start a metrics HTTP server from the gateway process on a configurable port.
  • Return gateway-wide metrics for uptime, connections, bytes, frames, events, protocol errors, pause/resume counts, sink pressure, HTTP state, latency, and memory.
  • Include per-connection snapshots with bytes, frames, last activity, and paused state.
  • Persist one metrics sample per second to metrics JSONL for later reporting.
  • Add an optional metrics CLI that fetches and prints the metrics endpoint.
Live and persisted metrics that line up with small load tests, sink counters, and connection state.
13

Add graceful shutdown

Stop accepting new connections, drain existing work, close sockets according to policy, and write a final report.

  • Create a shutdown module that coordinates listener, sockets, timers, metrics server, sinks, and reports.
  • Handle SIGINT and SIGTERM through one cleanup path.
  • Stop accepting new connections, stop metrics sampling, close the metrics server, and stop reading new frames.
  • Add shutdown grace, drain-or-close policy, and ACK policy flags.
  • Define ACK frame behavior for accepted events, persisted events, disabled ACKs, sink failures, dropped events, and malformed input.
  • Await sink close, close or destroy sockets by policy, and force close remaining sockets after the deadline.
  • Write a final report with shutdown timing, reason, active connections, dropped frames, drained events, forced closes, and sink close time.
A final shutdown report that records what drained, what closed, what dropped, and which resources were forced closed.
14

Build a load test client

Stress the gateway with many connections, frame counts, frame rates, payload sizes, and malformed ratios.

  • Create a load client that opens one socket per configured client.
  • Add flags for host, port, clients, frames, frame size, rate, malformed ratio, chunk size, duration, ACK timeout, report path, and metrics URL.
  • Generate deterministic request ids that stay unique across all clients.
  • Mix valid and malformed frames in one run using the configured malformed ratio.
  • Measure ACK latency only when ACK frames are observed, and parse ACK or error responses per socket.
  • Track connected state, frames sent, bytes sent, ACKs, ACK timeouts, injected protocol errors, and socket errors per client.
  • Poll gateway metrics during load when a metrics URL is configured.
  • Write a load report with throughput, latency, errors, memory, and backpressure data.
A load report that ties client-side throughput and errors to gateway-side metrics, pressure, and malformed-frame policy.
15

Generate the final protocol report

Produce a Markdown and HTML report that summarizes protocol design, parser behavior, error policy, backpressure behavior, shutdown behavior, and load results.

  • Create a report generator and connect it to the report command.
  • Read protocol docs, latest load report, final shutdown report, connection JSONL, protocol error JSONL, and metrics JSONL.
  • Summarize header layout, frame types, parser state, and malformed input policy.
  • Summarize connection lifecycle, file sink backpressure, HTTP forwarding, metrics during load, and graceful shutdown.
  • Cite measured totals for frames, accepted events, malformed frames, protocol errors, throughput, latency, active connections, file buffering, HTTP queue depth, and dropped frames.
  • Render Markdown and HTML reports for inspection and sharing.
A final protocol gateway report in Markdown and HTML, backed by actual load, metric, connection, error, and shutdown data.
Protocol evidence

The reports prove the gateway behavior.

Lab 07 turns protocol rules, connection state, malformed input, sink pressure, HTTP delivery, metrics, shutdown, and load results into files a developer can inspect after the run.

docs/protocol.mdwire format, frame types, and validation rules
data/events.ndjsonnormalized file-sink event records
data/http-received.ndjsonevents accepted by the local receiver
connections.jsonlconnection lifecycle and close reasons
protocol-errors.jsonlmalformed frame evidence by error code
metrics.jsonlgateway samples captured during runtime
load-latest.jsonthroughput, latency, errors, and pressure
protocol-gateway.htmlfinal human-readable gateway report

Choose your NodeBook package.

Switch between one-reader pricing and team licenses for up to 25 team members.

Individual pricing is for one reader and one personal purchase record.TEAM LICENSES INCLUDE UP TO 25 MEMBERS. DOWNLOADED CONTENT MAY BE
SHARED INTERNALLY WITH UP TO 30 PEOPLE TOTAL.
Downloadable book bundle

Digital Bundle

Volume I as EPUB, light/dark PDFs, slides, cheatsheets, and future updates.

$19.99$49.99
One-time purchase
  • Volume I EPUB for offline reading
  • Light and dark PDF editions
  • Slide decks for chapter review
  • Cheatsheets for quick lookup
  • Future Digital Bundle updates
  • Lifetime access to the files
Get Digital Bundle
This is the downloadable Volume I study bundle. It does not include Node Runtime Labs.
Best value
Everything together

NodeBook Pro

All Labs plus the downloadable Volume I bundle in one purchase. Save $9.99 vs buying separately.

$49.99$99.99
One-time purchase
Node Runtime Labs+Digital Bundle
  • Everything in Node Runtime Labs
  • Everything in the Digital Bundle
  • Seven included lab projects
  • Three upcoming labs when released
  • Future updates for both products
  • Lifetime access to purchased files
Get NodeBook Pro
This includes both paid products: Node Runtime Labs and the Digital Bundle.
Premium labs
Complex runtime projects

Node Runtime Labs

Seven long-form builds: recorder, binary store, stream workbench, resolver, watcher, task runtime, and protocol gateway.

$39.99$79.99
One-time purchase
  • Node Runtime Flight Recorder
  • Binary File Store / Append-Only Log Database
  • Stream Processing Workbench
  • Module Resolution Inspector
  • Atomic File Watcher + Incremental Build Cache
  • Async Task Runtime / Local Job Orchestrator
  • Custom Binary Protocol Gateway
  • Three more labs upcoming
Get Labs Bundle
This is the paid labs bundle. It does not include EPUB, PDFs, slides, or cheatsheets.
Team downloadable files

Digital Bundle Team

Volume I files, slides, cheatsheets, and updates for a small engineering team.

$59.99
One-time team license
  • Everything in the Digital Bundle
  • Supports up to 25 team members
  • Share internally with up to 30 people
  • Single purchase for the team
  • Future Digital Bundle updates
  • Lifetime access to purchased files
Get Team Bundle
Covers up to 25 team members and internal sharing with up to 30 people.
Team value
Everything for the team

NodeBook Pro Team

Node Runtime Labs plus the downloadable Volume I bundle in one team license.

$149.99
One-time team license
Node Runtime Labs+Digital Bundle
  • Everything in NodeBook Pro
  • Supports up to 25 team members
  • Share internally with up to 30 people
  • One receipt and license record
  • Future updates for both products
  • Save $29.99 vs team products separately.
Get Pro Team
Covers up to 25 team members and internal sharing with up to 30 people.
Team labs
Runtime training projects

Node Runtime Labs Team

Seven long-form builds for onboarding, study groups, and internal training.

$119.99
One-time team license
  • Node Runtime Flight Recorder
  • Binary File Store / Append-Only Log Database
  • Stream Processing Workbench
  • Module Resolution Inspector
  • Supports up to 25 team members
  • Share internally with up to 30 people
  • Three more labs upcoming
Get Team Labs
Covers up to 25 team members and internal sharing with up to 30 people.
See complete pricing breakdown
Other labs in the bundle

Lab 07 sits with six other runtime builds.

The bundle includes seven runtime projects covering process observation, binary storage, streams, module resolution, file watching, async orchestration, and custom protocols.

Custom Binary Protocol Gateway Lab | NodeBook Runtime Labs