Lab 05 · Node Runtime Labs

Atomic file watcher + build cache.

Build a watch-mode CLI that treats filesystem events as prompts to verify state. It scans a directory, fingerprints inputs, persists cache data, normalizes noisy saves, queues rebuilds, and writes reports that explain the session.

Read the phase plan Get the lab

15Phases

8.5/10Difficulty

0 depsPackage installs

Watcher commandsIncremental build checks

01Scan
$ npm run scan -- watched --metadata --hash
Captures file identity, content hash, and cache input.
02Watch
$ npm run watch -- watched
Starts the native watcher path.
03Poll
$ npm run watch -- watched --poll --poll-interval 1000
Exercises the fallback watcher with measured work.
04Batch
$ npm run watch -- watched --debounce 50 --concurrency 2 --rebuild-delay-ms 500
Makes rebuild batching and queue limits observable.
05Symlinks
$ npm run scan -- watched --metadata --hash --follow-symlinks
Compares link treatment with the default scan.

Scan the tree, watch edits, then tune fallback behavior.

What this lab builds

The finished tool verifies filesystem state.

Directory inventory

Create a scanner that walks the watched tree, applies ignore policy, records relative paths, counts files and directories, and writes repeatable scan reports.

Metadata and hash cache

Store file type, size, timestamps, permissions, device and inode identity, symlink targets, SHA-256 hashes, and persistent cache data.

Watch event pipeline

Receive native watcher events, add polling fallback, normalize uncertainty, group atomic writes, debounce bursts, and verify state through rescans.

Incremental rebuild system

Classify added, changed, deleted, and unchanged paths, queue rebuild jobs with bounded concurrency, skip cache hits, clean deleted outputs, and report the session.

Complete phase plan

Every phase in Lab 05.

The build starts with a tiny command surface and ends with a complete watch-session report. Each phase adds one observable mechanism and one artifact to inspect.

00Create the projectCreate the watcher project shell and a CLI that accepts a directory argument.

Tasks

Create a private ESM package with an engine requirement for Node.js v24 or newer.
Add npm scripts for scan, watch, demo-watch, and simulate-atomic.
Create src, watched, .cache, and reports so source files, watched input, persistent state, and evidence stay separate.
Build the CLI entry point that dispatches scan and watch commands from raw argv strings.
Print help output with the planned metadata, hash, poll, debounce, concurrency, rebuild delay, symlink, and cache flags.
Accept a watched directory, default to watched, and reject unknown commands, flags, and invalid directories with a failure exit code.

Artifact

A project shell with package metadata, folder boundaries, help output, directory parsing, and strict input handling.

01Scan the directory treeBuild the initial file inventory for the watched tree.

Tasks

Create src/scan.js and expose scanTree(root, options) as the shared inventory boundary.
Walk nested directories with fs.promises.readdir and Dirent objects, then sort entries for deterministic output.
Ignore .git, node_modules, .cache, and reports by default while recording ignored paths as evidence.
Store portable relative POSIX-style paths instead of machine-specific absolute paths.
Count files, directories, ignored paths, and scan errors during traversal.
Write reports/scan.json with root, generatedAt, counts, files, ignored paths, and errors.

Artifact

A scan report that describes every regular file under watched plus ignored paths and traversal errors.

02Record file metadataStore size, mtime, mode, inode, device, and file type for each scanned entry.

Tasks

Add the --metadata flag so extra filesystem calls are opt-in and visible.
Use lstat for each entry so symlinks are reported as links by default.
Record size, mtimeMs, ctimeMs, birthtimeMs, mode, permissions, dev, ino, and nlink.
Create one file type classifier for files, directories, symlinks, FIFOs, sockets, block devices, character devices, and unknown entries.
Read symlink targets and keep target strings separate from link metadata.
Record metadata failures with path, code, and phase so races do not crash the scan.

Artifact

A metadata-rich inventory that can separate content changes, permission changes, identity changes, and symlink state.

03Add content hashingCompute a content hash for every regular file.

Tasks

Add the --hash flag so content reads are explicit in scan and watch runs.
Create src/hash.js with a helper that hashes one file and returns algorithm, digest, and byte count.
Stream file bytes into crypto.createHash('sha256') instead of buffering whole files.
Hash regular files only and skip directories, symlinks, and special files by policy.
Preserve path-sorted report output even if hashing later runs with concurrency.
Record ENOENT, EACCES, EPERM, and other hash failures as report errors.

Artifact

A content fingerprint layer that gives the cache a stable input hash for ordinary files.

04Build the persistent cacheSave the latest inventory to .cache/index.json and load it on the next run.

Tasks

Create src/cache.js with loadCache(cachePath) and saveCacheAtomic(cachePath, inventory).
Define a cache schema with version, root, createdAt, updatedAt, options, and filesByPath.
Load the previous cache before scanning so diffing has the prior state.
Save through a same-directory temporary file and rename over the target cache path.
Support --cache <path> and --no-cache to make cache location and bypass mode observable.
Report cache.path, cache.loaded, cache.loadMs, and cache.saveMs in reports/scan.json.

Artifact

An atomic cache file that survives between runs and becomes the previous inventory for later diffs.

05Detect added, changed, and deleted filesCompare the current inventory against the cached inventory and classify paths.

Tasks

Create src/diff.js and export diffInventories(previous, current).
Compare filesByPath keys to classify added and deleted paths.
Compare hash, size, mtimeMs, ctimeMs, mode, identity, and type to explain changed paths.
Count unchanged paths and keep unchanged details compact.
Write reports/diff.json with counts, added, changed, deleted, and unchanged data.
Preserve the order: load cache, scan, diff, write reports, then save the current cache.

Artifact

A diff report that explains rebuild decisions with added, changed, deleted, and unchanged classifications.

06Add fs.watchReceive live filesystem events from the watched directory tree.

Tasks

Create src/watch.js so long-running watch state stays inside a session module.
Run the existing scan path before registering live watchers to capture an initial baseline.
Register directory watchers for the root and nested directories, using recursive watching where the platform supports it.
Normalize every raw event into id, time, eventType, filename, watchRoot, absolutePath, relativePath, source, and uncertainty notes.
Keep a bounded event buffer and append normalized events to reports/events.jsonl.
Record watcher registration and delivery errors as structured events with path, operation, phase, code, and message.

Artifact

A live watch session that records raw filesystem events without pretending they are final logical changes.

07Add fs.watchFile fallbackSupport polling mode with --poll.

Tasks

Add --poll and --poll-interval <ms> flags for watch mode.
Register fs.watchFile pollers for regular files discovered by the startup baseline.
Run periodic tree discovery so polling mode can notice newly created files.
Convert curr and prev Stats snapshots into the same event stream used by native watching.
Track listener functions and call fs.unwatchFile during cleanup or deleted-file handling.
Write reports/watch-state.json with mode, interval, registered files, estimated stat checks, callbacks, discovery scans, and event counts.

Artifact

A polling observation mode whose latency and filesystem cost are visible in the watch-state report.

08Handle atomic writesDetect editor-style temporary write and rename patterns as one logical file change.

Tasks

Create src/simulate-atomic-write.js to write a same-directory temp file and rename it over the target.
Create src/atomic.js as the event burst grouping layer above raw watcher events.
Recognize temp name patterns such as .name.tmp, name~, .swp, and .tmp.
Group rename bursts by directory and a short time window such as 100ms.
After the burst settles, rescan affected paths and compare metadata identity and hash.
Write reports/logical-events.json with logical events, rawEventIds, tempPaths, and final path state.

Artifact

Logical atomic-replace events that preserve raw event evidence while reporting one target-file change.

09Debounce rebuildsGroup rapid events into one rebuild cycle after the event stream settles.

Tasks

Add --debounce <ms> with a default interval such as 150ms.
Create src/debounce.js to collect events and fire after quiet time.
Store affected paths in a Set and mark the whole watched root for rescan when filename data is missing.
When the timer fires, scan with metadata and hashes, diff against cache, enqueue rebuild work, and save cache after scheduling.
Append batch records to reports/batches.jsonl with batch timing, event counts, paths, and diff counts.
Confirm reports and cache files stay outside watched input or remain ignored.

Artifact

Batch records that show how raw and logical events become verified rebuild cycles.

10Add a rebuild queueRun rebuild jobs under a concurrency limit.

Tasks

Create src/queue.js with a concurrency-limited task queue that tracks active and pending work.
Add --concurrency <n> with a default such as 2 and reject values below 1.
Represent each rebuild job with id, paths, createdAt, startedAt, finishedAt, and status.
Create src/rebuild.js with placeholder async rebuild work that writes per-path output records under reports/build-output.
Coalesce duplicate affected paths into pending jobs while preserving active job history.
Append queue samples to reports/queue.jsonl whenever active or pending counts change.

Artifact

A rebuild queue that exposes pressure, enforces the concurrency cap, and records job state changes.

11Add incremental build cacheSkip unchanged files during rebuild.

Tasks

Add a buildCache section to .cache/index.json separate from filesByPath.
Store inputHash, inputMetadata, outputPath, outputHash, builtAt, and builderVersion for each built source.
Warm the build cache during watch startup before relying on unchanged-file hits.
Rebuild added and changed paths, then validate unchanged paths against input fingerprint, builder version, and output existence.
Clean generated outputs and cache records for deleted source files.
Append rebuild summaries to reports/rebuilds.jsonl with cache hits, misses, rebuilt paths, skipped paths, and deleted outputs.

Artifact

A build cache that can prove when work was skipped and when one changed file caused one rebuilt output.

12Handle symlinks and permissionsDefine and implement policy for symlinks, unreadable files, and special files.

Tasks

Add --follow-symlinks to scan and watch modes while keeping the default as record links and skip targets.
Record skipped entries for symlinks, special files, and unsupported file types with path, kind, action, and reason.
When requested, resolve symlink targets with realpath, stat the target, and hash followed regular files.
Track visited realpaths to report symlink cycles instead of traversing forever.
Handle EACCES, EPERM, ENOENT, and ELOOP during scan, metadata, hash, watch, and rebuild operations.
Write platform notes when symlink or permission behavior changes what the learner can verify locally.

Artifact

An edge-case policy report that makes skipped, followed, errored, and platform-specific filesystem behavior visible.

13Add graceful shutdownClose watchers and save cache state on SIGINT and SIGTERM.

Tasks

Create src/shutdown.js with a registry of cleanup functions from watch mode.
Handle SIGINT and SIGTERM as the normal control path for stopping a watch process.
Make shutdown idempotent with states such as running, stopping, and forced.
Stop event intake, close FSWatcher handles, unwatch polling paths, and clear timers.
Flush pending debounce work or mark active and pending rebuild jobs with honest final state.
Save cache atomically and write reports/shutdown.json with signal, reason, closed resources, cache result, queue state, and duration.

Artifact

A shutdown report that records how the watcher stopped and what state was preserved for the next run.

14Generate watch reportProduce a final Markdown report for a complete watch session.

Tasks

Create src/report.js to read JSON and JSONL artifacts without reaching into watcher internals.
Implement src/demo-watch.js to reset a fixture, start watch mode, edit files, simulate an atomic write, delete a file, and stop through shutdown.
Clear generated report artifacts at demo startup while preserving reports/reflections.md.
Generate reports/watch-report.md with options, root, scan counts, raw events, logical groups, batches, rebuild jobs, queue high-water mark, cache hits, misses, skipped files, errors, and shutdown reason.
Optionally generate reports/watch-report.html from the same data model.
End the Markdown report with one concrete filesystem rule written from the observed evidence.

Artifact

A final watch report that connects raw events, verified diffs, rebuild work, cache results, and shutdown state.

Watch evidence

Reports are part of the runtime tool.

Lab 05 treats reports as build output. The learner can inspect the scan baseline, cache load, diff reasons, raw events, logical groups, queue pressure, rebuild decisions, shutdown state, and the final watch report.

scan.jsonbaseline inventory, metadata, hash, cache timing, and scan errors

diff.jsonadded, changed, deleted, unchanged, and reason fields

events.jsonlraw native or polling watch events in append order

logical-events.jsonatomic write groups and verified final state

batches.jsonldebounced rebuild batches with affected paths

queue.jsonlactive and pending rebuild queue samples

rebuilds.jsonlcache hits, misses, rebuilt paths, and deleted outputs

watch-state.jsonwatch mode, poll counters, registrations, and event totals

shutdown.jsonsignal, cleanup, cache save, and queue state at exit

watch-report.mdthe final session report from scan through shutdown

Choose your
NodeBook package.

Buy a single volume or lock in every volume at once. Switch between one-reader pricing and team licenses for up to 25 members.

Choose volume

Individual pricing is for one reader and one personal purchase record.

Downloadable book bundle

Digital Bundle

Volume I as EPUB, light and dark PDFs, slides, cheatsheets, and future updates.

$19.99$49.99

One-time purchase

Volume I EPUB for offline reading
Light and dark PDF editions
Slide decks for chapter review
Cheatsheets for quick lookup
Future Digital Bundle updates
Lifetime access to the files

Get Digital Bundle

This is the downloadable Volume I study bundle. It does not include Node Runtime Labs.

Best value

Everything for this volume

NodeBook Pro

Volume I Labs plus its downloadable bundle in one purchase. Save $9.99 vs buying the Digital Bundle and Labs separately.

$49.99$99.99

One-time purchase

Node Runtime LabsDigital Bundle

Everything in Node Runtime Labs
Everything in the Digital Bundle
Volume I labs and book bundle
Future updates for both products
Lifetime access to purchased files

Get NodeBook Pro

Includes both paid products for Volume I.

Premium labs

Complex runtime projects

Node Runtime Labs

Volume I long-form builds with checkpoints, hints, debugging notes, and expected output.

$39.99$79.99

One-time purchase

Volume I runtime lab projects
Phase-by-phase build instructions
Checkpoints, hints, and rubrics
Debugging notes and expected output
Reflection questions
Future lab updates

Get Labs Bundle

This is the paid labs bundle for Volume I. It does not include EPUB, PDFs, slides, or cheatsheets.

See complete pricing breakdown

Other labs in the bundle

Lab 05 sits inside the runtime lab track.

The bundle includes seven runtime projects covering process observation, binary storage, streams, module resolution, file watching, async orchestration, and custom protocols.

Lab 01