Node.js File I/O: Whole-File Helpers, Streams, Random Access, and Durability

File I/O is the path between storage and process memory. The choice of API is less about which method looks convenient and more about how much data is buffered, how operations are ordered, and what durability guarantee the write needs. Whole-file helpers fit small files. Streams fit large files and pipelines. Descriptor calls fit random access. Flush operations count when a successful write must still be present after a crash.

Reading and Writing Files

The snippets in this chapter use CommonJS with node: built-in specifiers. Examples that use await are async-function fragments. If you run them with top-level await in an ES module, rewrite require() calls as import declarations.

File content reads and writes eventually operate through OS file descriptors or platform handles, which the previous subchapter covered. Not every path operation exposes a descriptor to user code: calls such as stat, rename, unlink, mkdir, and watch can work at the path level. Once the work is about file contents, though, the fs module gives you several layers of control: whole-file helpers (readFile, writeFile, appendFile), stream APIs (createReadStream, createWriteStream), descriptor-level byte calls (fs.read, fs.write), line readers, and durability calls such as fsync(). Beneath most asynchronous filesystem work, libuv's thread-pool dispatch path shapes the performance profile.

Reading Entire Files with readFile

The most direct reading API is fs.readFile(). It accepts a path and returns the contents through a callback.

const fs = require('node:fs');
fs.readFile('./config.json', 'utf8', (err, data) => {
  if (err) throw err;
  const config = JSON.parse(data);
});

When you pass a path, Node opens the file, buffers the full contents, closes the descriptor it opened internally, and hands you the result. That does not mean Node performs one giant OS read. For regular files, Node v24 reads in 512 KiB chunks; when it cannot determine the size up front, as with some non-regular files, it uses smaller chunks. If you pass an encoding such as 'utf8', Node converts the buffer to a string before returning it. Without an encoding, the result is a raw Buffer.

The promise-based version changes the JavaScript shape without changing that underlying behavior:

const data = await fs.promises.readFile('./config.json', 'utf8');
const config = JSON.parse(data);

The filesystem work still follows the same libuv path. The completion surface changes from callback invocation to promise resolution, which makes it fit naturally into async/await code.

readFileSync and Blocking

The synchronous counterpart performs the same kind of work while blocking the main thread:

const data = fs.readFileSync('./config.json', 'utf8');

While the read is in progress, everything else in the process waits. No timers fire. No incoming requests get processed. No microtasks run. The event loop is frozen until the syscall sequence returns.

That cost is acceptable when the process is still starting. Loading configuration before a server begins accepting connections delays readiness, but it does not serialize live request handling. CLI tools that read a file, transform it, and exit are similar: the cost is command latency, not contention with concurrent work.

// At startup - fine
const config = JSON.parse(fs.readFileSync('./config.json', 'utf8'));
const app = createServer(config);
app.listen(3000);

Inside a request handler, the same call has a very different effect. A 10 ms blocking read ties up the process for those 10 ms before any other request, timer, or microtask can run. If every request depends on that synchronous read, throughput is bounded by the blocking read latency, and network-mounted filesystems or overloaded storage make the bound worse.

Memory Implications

readFile loads the entire file into one buffer. A 10 KB configuration file uses 10 KB. A 500 MB log file uses 500 MB. The relationship is linear and unforgiving.

The buffer itself lives outside V8's managed heap, as covered in Chapter 2, but V8 still tracks it through external memory accounting. Large external allocations can increase garbage collection frequency and pause time. During those pauses, every connection stalls, even though the bytes are not stored inside ordinary JavaScript objects.

Concurrency multiplies the same pressure. Ten requests that each read a 100 MB file allocate about a gigabyte of buffer memory at the same time. In a container with a smaller memory limit, the process can exceed that limit before the reads complete. Even when memory is available, the extra external memory pressure can hurt latency.

The point where readFile stops being practical depends on the memory limit, concurrency level, and latency target of the process. Small configuration files are usually fine. Large or unpredictable files belong on streams or explicit chunked reads.

Error Handling

File operations fail for ordinary reasons: the path might not exist, permissions might be wrong, or the disk might be full or disconnected. The error object's .code property is the stable place to inspect the cause.

try {
  const data = await fs.promises.readFile(path, 'utf8');
  return JSON.parse(data);
} catch (err) {
  if (err.code === 'ENOENT') return {};  // file doesn't exist
  if (err.code === 'EACCES') throw new Error('permission denied');
  throw err;
}

ENOENT means the path does not exist, which is often expected for a cache file that has not been created yet or an optional config file. Returning a default value is reasonable in that case. EACCES means the process does not have read permission. On macOS, Linux, and Windows, EISDIR means you accidentally passed a directory path; FreeBSD can return a representation of the directory contents instead. EMFILE means the process has hit its file descriptor limit, which the previous subchapter covered.

The delivery mechanism changes with the API style. Callback APIs receive the error as the first argument, promise APIs reject, and sync calls throw exceptions. The error codes themselves are the same.

The AbortSignal Option

An in-flight readFile can be cancelled with an AbortSignal:

const controller = new AbortController();
setTimeout(() => controller.abort(), 500);

try {
  const data = await fs.promises.readFile('./huge.bin', {
    signal: controller.signal
  });
} catch (err) {
  if (err.name === 'AbortError') console.log('read cancelled');
}

Cancellation here means Node stops the readFile operation it is coordinating. If the abort fires before completion, Node rejects with an AbortError and closes any descriptor it opened for a path-based readFile. It does not cancel an individual OS read that is already in progress; it stops Node's internal buffering between those requests. If you pass an existing file descriptor or FileHandle, you still own that resource's lifetime.

Writing Entire Files with writeFile

fs.writeFile() is the write-side counterpart to readFile. You pass a path and some data:

await fs.promises.writeFile('./output.json', JSON.stringify(data));

Node opens the file, creating it if it does not exist, truncates it to zero length, writes all the bytes, and closes the descriptor. The default flag is 'w': open for writing, create if needed, and truncate if the file already exists.

The data can be a string, a Buffer, a TypedArray, a DataView, an Iterable, an AsyncIterable, or a Stream. Strings are encoded as 'utf8' by default, though you can choose a different encoding when needed:

await fs.promises.writeFile('./output.txt', content, 'latin1');

Exclusive Creation with 'wx'

Changing the flag to 'wx' makes creation atomic by failing if the file already exists.

await fs.promises.writeFile('./lock.pid', process.pid.toString(), {
  flag: 'wx'
});

At the syscall level, this sets O_CREAT | O_WRONLY | O_EXCL. The kernel either creates and opens the file, or returns EEXIST. There is no race window between "check if exists" and "create," so if two processes compete to create the same lock file, exactly one wins.

File Permissions on Creation

When writeFile creates a new file, the mode option supplies the permissions:

await fs.promises.writeFile('./secret.key', keyData, {
  mode: 0o600  // owner read/write only
});

The default mode is 0o666, modified by the process umask. With a typical umask of 0o022, that results in 0o644. The subtle part is that mode only applies when the file is created. If the file already exists, its permissions stay unchanged. To enforce permissions regardless of whether the file was new, call chmod after writing.

The Truncation Problem

The convenience of 'w' comes with a failure mode: the file is truncated before the new data is fully written. If the process crashes between truncation and write completion, the file can be empty or partially written. For configuration files, state files, or anything where corruption breaks the application on restart, that risk is consequential.

For local POSIX filesystems, the durable version of the temp-file-and-rename pattern looks like this:

const crypto = require('node:crypto');
const fs = require('node:fs/promises');
const path = require('node:path');

async function replaceFileAtomic(targetPath, data) {
  const dir = path.dirname(targetPath);
  const base = path.basename(targetPath);
  const suffix = crypto.randomBytes(6).toString('hex');
  const tmpPath = path.join(dir, `.${base}.${process.pid}.${suffix}.tmp`);

  let file;
  try {
    file = await fs.open(tmpPath, 'wx');
    await file.writeFile(data);
    await file.sync();
    await file.close();
    file = undefined;

    await fs.rename(tmpPath, targetPath);

    const directory = await fs.open(dir, 'r');
    try {
      await directory.sync();
    } finally {
      await directory.close();
    }
  } catch (err) {
    if (file) await file.close().catch(() => {});
    await fs.unlink(tmpPath).catch(() => {});
    throw err;
  }
}

The temp file is created in the same directory as the target so the later rename() can replace the target atomically. If the write fails, the original file remains untouched. If the write succeeds, POSIX readers see either the old content or the new content, never a half-written replacement. The file sync() makes the new file contents durable before the rename. The parent directory sync() then makes the name change durable after the rename on platforms that support opening and syncing directories. Without that directory sync, the replacement can become atomically visible before it is crash-durable.

Figure 4.4 — A durable replacement writes the new bytes to a temporary file in the target directory, flushes that file, renames it into place, and then flushes the directory entry.

writeFileSync

writeFileSync has the same blocking profile as readFileSync: it uses main-thread syscalls and freezes the event loop until the write completes. That is appropriate for startup code and scripts. In hot server paths, it serializes concurrent work behind the write.

try {
  fs.writeFileSync('./output.json', JSON.stringify(result));
} catch (err) {
  console.error('write failed:', err.code);
}

Errors are thrown as exceptions. If an ENOSPC (disk full) or EACCES exception escapes uncaught, the process terminates.

Appending to Files

Appending is a different write mode rather than a different storage mechanism. fs.appendFile() opens the file with the O_APPEND flag and writes at the end:

const entry = `${new Date().toISOString()} Server started\n`;
await fs.promises.appendFile('./server.log', entry);

If the file does not exist, it is created. If it does exist, the new data goes after the existing content.

O_APPEND changes how the kernel chooses the write position. On local POSIX filesystems, before each write(2) call, the kernel moves the file offset to the end and performs the write as one atomic step. That protects the file position from races between processes. It does not mean every high-level append call becomes a single syscall, and it does not make network filesystems such as NFS safe for concurrent appenders. Keep log entries small, write each record as one payload when possible, and treat ordering between processes as nondeterministic.

You get the same behavior by passing { flag: 'a' } to writeFile. appendFile exists for clarity: it names the intent at the call site.

When Append Falls Short

Append mode fits log files, CSV files that grow over time, and audit trails. It does not fit structured formats that require the whole file to remain valid. You cannot append to a JSON file and still have valid JSON if the closing bracket is already there. You would need to read the file, parse it, modify the data structure, and write the whole thing back, ideally with atomic replacement.

For high-throughput logging, repeatedly calling appendFile is also inefficient because each call opens and closes the file. A write stream in append mode keeps the descriptor open:

const fs = require('node:fs');
const { once } = require('node:events');

const log = fs.createWriteStream('./app.log', { flags: 'a' });

if (!log.write('entry one\n')) await once(log, 'drain');
if (!log.write('entry two\n')) await once(log, 'drain');

log.end();
await once(log, 'finish');

The stream keeps the file descriptor open and uses writable-stream backpressure when writes cannot keep up with the rate at which the application produces log records.

Stream-Based File I/O

As file size grows, whole-file buffering stops being the natural default. Streams process data in chunks instead. For file read streams, those chunks are typically up to 64 KB by default, so memory is bounded by stream buffers, transform state, queued writes, and application concurrency rather than by the total file size.

createReadStream

const stream = fs.createReadStream('./access.log', 'utf8');
stream.on('data', (chunk) => {
  // chunk is a string, typically ~64 KB
});
stream.on('end', () => console.log('done'));
stream.on('error', (err) => console.error('read failed:', err.code));

Node opens the file, requests up to highWaterMark bytes at a time, emits a 'data' event with each chunk, and repeats until EOF. For file read streams, the default highWaterMark is 64 KB. After the final chunk, the stream emits 'end' and closes the file descriptor.

Passing 'utf8' makes chunks arrive as strings. Without an encoding, they arrive as Buffers. For binary files such as images, videos, and archives, leave the encoding off and work with the raw bytes.

The highWaterMark option controls the requested chunk size:

const stream = fs.createReadStream('./file.bin', {
  highWaterMark: 256 * 1024  // 256 KB chunks
});

Larger chunks usually mean fewer read requests and fewer JavaScript callbacks, but they also mean more memory per stream. On NVMe drives, larger chunks can improve throughput when overhead is dominated by JavaScript processing rather than disk I/O. In memory-constrained environments that read many files concurrently, smaller chunks, such as 4-16 KB, reduce per-stream memory. The default 64 KB is a reasonable middle point for most workloads.

Byte Range Reads

createReadStream can also read a slice of a file with start and end:

const stream = fs.createReadStream('./video.mp4', {
  start: 1048576,   // byte 1 MB
  end: 2097151      // byte 2 MB - 1 (inclusive)
});

Node requests bytes from start through end, then stops. It does not request the rest of the file, although the operating system may still perform filesystem or cache readahead. HTTP range requests for video seeking or download resumption use this pattern: the server reads only the requested byte range and sends it to the client.

createWriteStream

The write-side stream has the same chunked character:

const fs = require('node:fs');
const { once } = require('node:events');

const out = fs.createWriteStream('./output.txt');
out.write('first chunk\n');
out.write('second chunk\n');
out.end('final chunk\n');
await once(out, 'finish');

Each write() call can buffer data inside the writable stream. highWaterMark is a backpressure threshold, not a hard memory cap. When the buffered length reaches or crosses that threshold, write() returns false, telling you to pause until 'drain'. If you need the exact threshold, inspect out.writableHighWaterMark or set highWaterMark explicitly; defaults have changed across Node releases. In practice, you rarely handle this manually because pipeline manages backpressure for connected streams.

end() flushes remaining data, closes the descriptor, and emits 'finish'. To append instead of overwrite, use the 'a' flag:

const log = fs.createWriteStream('./server.log', { flags: 'a' });
log.end('started\n');

Piping with pipeline

Streams become most useful when you connect them:

const { pipeline } = require('node:stream/promises');
await pipeline(
  fs.createReadStream('./input.bin'),
  fs.createWriteStream('./output.bin')
);

Data flows chunk by chunk from source to destination. pipeline wires up backpressure, so if the write side cannot keep up, the read side pauses until the buffer drains. If any stream errors, pipeline destroys the connected streams and rejects the promise. Use pipeline over bare pipe() for most file-to-file or transform chains; pipe() does not propagate errors, so a write failure can leave the read stream hanging open and leak a file descriptor. HTTP responses are the main caveat: destroying the response socket after a partial response is user-visible, so handle that case deliberately.

Transform chains follow the same flow:

const zlib = require('node:zlib');
await pipeline(
  fs.createReadStream('./data.json'),
  zlib.createGzip(),
  fs.createWriteStream('./data.json.gz')
);

Here the chunks move from disk through compression to a new file. Memory stays bounded by stream buffers and transform state, not by the total input size.

Serving Files over HTTP

A common HTTP handler streams a file directly to the response:

const fs = require('node:fs');
const http = require('node:http');

http.createServer((req, res) => {
  const stream = fs.createReadStream('./static/bundle.js');
  stream.on('open', () => {
    res.writeHead(200, { 'content-type': 'application/javascript' });
    stream.pipe(res);
  });
  stream.on('error', () => {
    if (res.headersSent) return res.destroy();
    res.writeHead(404);
    res.end();
  });
}).listen(3000);

The file moves to the client without first being buffered as a whole in process memory. Ten concurrent downloads of a 50 MB file keep memory proportional to active stream buffers instead of allocating 500 MB of file contents up front.

Buffer-All vs. Stream - The Decision

The decision is mainly about whether the program really needs the whole file at once.

The tradeoff is complexity for efficiency. Streams mean chunks, async events, and flow control. For large files, use streaming or an explicit chunked fs.read loop. Calling readFile on a 10 GB file either fails allocation or creates memory pressure far outside what a request path should tolerate.

Figure 4.5 — Whole-file helpers collect the complete file before processing, while streams move bounded chunks forward and let backpressure slow later reads.

Low-Level Byte Operations with fs.read and fs.write

fs.read() and fs.write() move you below whole-file helpers and streams. You allocate the buffer, specify the offset in the file, and control exactly how many bytes transfer. Higher-level helpers such as readFile, streams, and readline use the same lower-level filesystem read/write machinery, but they own buffer allocation, descriptor lifetime, and control flow for you.

Reading at Specific Positions

The traditional callback form takes six parameters:

fs.read(fd, buffer, offset, length, position, callback);

You provide the file descriptor, a pre-allocated buffer, the position inside that buffer to start writing, the number of bytes to read, and the position in the file to read from. The callback receives (err, bytesRead, buffer).

The promise-based FileHandle version is easier to read:

const fd = await fs.promises.open('./data.bin', 'r');
try {
  const buf = Buffer.allocUnsafe(64);
  const { bytesRead } = await fd.read(buf, 0, 64, 0);
} finally {
  await fd.close();
}

This reads 64 bytes from position 0 of the file into buf, starting at offset 0 in the buffer. bytesRead tells you how many bytes were actually read. It can be less than 64 if the file is shorter, or zero if the position is already at EOF.

The position parameter is where random access enters the picture. Pass a number, and Node reads at that byte offset. Pass null, and it reads from the current file position and advances it. Explicit positions let you jump around inside a file without reading sequentially from the start.

Parsing a Binary Header

Many binary formats begin with a fixed-size header containing metadata. PNG files start with an 8-byte signature. ZIP files have a central directory at the end. Database files store record counts, version numbers, and data offsets in the header.

Here is a hypothetical format with a 64-byte header:

let dataOffset;
const fd = await fs.promises.open('./database.db', 'r');
try {
  const header = Buffer.allocUnsafe(64);
  const { bytesRead } = await fd.read(header, 0, 64, 0);

  if (bytesRead < 64) throw new Error('Incomplete header');

  const magic = header.readUInt32BE(0);    // 4-byte signature
  const version = header.readUInt32LE(4);  // format version
  const recordCount = header.readUInt32LE(8);
  dataOffset = header.readUInt32LE(12);
} finally {
  await fd.close();
}

The four readUInt32* calls read from the header buffer already in memory; they are not additional file reads. Once the header has told you where records begin, record number 5 in a fixed-width record layout can be read directly:

const fd = await fs.promises.open('./database.db', 'r');
try {
  const recordBuf = Buffer.allocUnsafe(128);
  const pos = dataOffset + (5 * 128);
  const { bytesRead } = await fd.read(recordBuf, 0, 128, pos);

  if (bytesRead < 128) throw new Error('Incomplete record');
} finally {
  await fd.close();
}

That is two file reads: 64 bytes for the header and 128 bytes for the record. If the file is 10 GB with millions of records, the program has still read only 192 bytes. readFile would try to allocate the whole 10 GB.

Figure 4.6 — Explicit positions let one descriptor read a small header, calculate a record offset, and fetch only the requested record instead of loading surrounding bytes.

Writing at Specific Positions

fs.write() works the same way in reverse:

const fd = await fs.promises.open('./database.db', 'r+');
try {
  const buf = Buffer.allocUnsafe(4);
  buf.writeUInt32LE(42, 0);
  await fd.write(buf, 0, 4, 16);  // write at byte 16
} finally {
  await fd.close();
}

The file is opened in read-write mode with 'r+', which preserves existing content. Only 4 bytes at offset 16 are replaced; the rest of the file is untouched. This is how databases update individual records without rewriting entire files.

The flag is the important part. 'w' would truncate the file to zero length before the write, destroying the existing data. 'r+' opens for both reading and writing without truncation.

Buffer Reuse in Read Loops

Low-level reads also let you reuse one buffer across many operations:

const buf = Buffer.allocUnsafe(4096);
let position = 0;
let bytesRead;
do {
  ({ bytesRead } = await fd.read(buf, 0, 4096, position));
  if (bytesRead > 0) processChunk(buf.subarray(0, bytesRead));
  position += bytesRead;
} while (bytesRead > 0);

One 4 KB buffer is reused for every read. In a tight loop over a large file, that reduces garbage collection pressure compared with readFile, which allocates one massive buffer, or streams, which allocate a new buffer for each chunk. The subarray call creates a view into the existing buffer without copying, as covered in Chapter 2.

allocUnsafe is safe in this loop because the read immediately overwrites the bytes you process. The important constraint is to process only buf.subarray(0, bytesRead), not the full buffer. Bytes beyond bytesRead contain uninitialized memory, possibly from a previous allocation.

When to Go Low-Level

Use fs.read() and fs.write() when the higher-level APIs hide control you actually need:

• Byte-level precision. Reading specific byte ranges from binary formats, file headers, fixed-size records, or length-prefixed protocols.
• Random access. Jumping to calculated offsets based on an index or metadata, reading only the parts you need.
• Buffer reuse. Allocating once and reusing across many reads in performance-sensitive loops.
• Custom abstractions. Building something that readFile and streams do not cover, like a paged database engine or a binary protocol parser.

Most application code never touches these APIs. They are for cases where higher-level abstractions do not fit: binary file formats, database internals, and network protocol implementations.

Flushing to Disk with fsync

When a write "succeeds," the data may still be in the OS buffer cache rather than on physical storage. The kernel batches disk writes for performance. On many Linux systems, dirty page expiration defaults are measured in tens of seconds, but that is not a durability contract and it differs by platform and configuration.

If the machine loses power or the kernel panics before the flush, the data may be lost. The write callback returned successfully, or the promise resolved, but the bytes may never have reached the disk.

fsync() asks the kernel to flush buffered writes and file metadata for a descriptor to the storage device:

const fd = await fs.promises.open('./ledger.dat', 'w');
try {
  await fd.write(buf, 0, buf.length, 0);
  await fd.sync();
} finally {
  await fd.close();
}

The sync() call blocks a thread-pool worker until the OS reports that the transfer has completed. That is slow relative to buffered writes, and the real guarantee still depends on the filesystem, mount options, storage device, and whether the device honors flushes honestly.

Node v20.10 and later, including Node v24, expose flush: true on whole-file helpers and write streams:

await fs.promises.writeFile('./ledger.dat', data, { flush: true });
await fs.promises.appendFile('./audit.log', entry, { flush: true });

const out = fs.createWriteStream('./ledger.dat', { flush: true });
out.end(data);

Those options flush the file descriptor before closing it. They do not sync the containing directory after a rename.

For the temp-file-and-rename pattern with durability, the file and the directory need separate flushes:

const fd = await fs.promises.open(tmpPath, 'w');
try {
  await fd.writeFile(data);
  await fd.sync();
} finally {
  await fd.close();
}

await fs.promises.rename(tmpPath, targetPath);

const dir = await fs.promises.open(path.dirname(targetPath), 'r');
try {
  await dir.sync();
} finally {
  await dir.close();
}

The file sync flushes the new file's contents before the rename makes it visible. On POSIX-style systems where directories can be opened and synced, the directory sync flushes the directory entry update that points the target name at the new file. Without both steps, you may get atomic visibility without crash durability.

Most application writes do not need fsync. Log files, caches, and temporary files are usually regenerable. fsync is for database transaction logs, financial records, or any state file where loss means silent data corruption.

Line-by-Line Reading with readline

Text files often have line structure: log files, CSVs, configuration files, and JSONL files with one JSON object per line. The readline module parses a readable stream into individual lines with bounded memory. It holds stream chunks and incomplete line data, not the whole file.

const fs = require('node:fs');
const readline = require('node:readline/promises');

const input = fs.createReadStream('./access.log', { encoding: 'utf8' });
const rl = readline.createInterface({
  input,
  crlfDelay: Infinity
});

let inputError;
input.on('error', (err) => {
  inputError = err;
  rl.close();
});

try {
  for await (const line of rl) {
    if (line.includes('ERROR')) console.log(line);
  }
} finally {
  rl.close();
  input.destroy();
}
if (inputError) throw inputError;

crlfDelay: Infinity normalizes line endings. It ensures that a \r\n pair split across two chunks is treated as one line break. Without it, if \r arrives at the end of one chunk and \n arrives in the next chunk more than 100 ms later, which is the default crlfDelay, readline would treat them as two line breaks and produce an unwanted empty line.

The for await loop is an async iterator over lines. It gives sequential application processing, but it is not true line-level backpressure. readline.createInterface() starts consuming the input stream as soon as it is created, and it may read ahead into its internal buffer while your loop awaits work for the current line. If strict backpressure is part of the design, use a stream transform or a manual chunked read loop where you control when the next read happens.

How readline Buffers Lines

Under the hood, readline reads chunks from the input stream, appends them to an internal string buffer, and scans for newline characters. When it finds one, it slices the line out of the buffer and emits it.

Chunk breaks are the awkward part. A 64 KB chunk from the read stream can split a line in half. The first chunk might end with "2024-01-15 request to /api/us" and the next might start with "ers 200 OK\n". readline holds the partial line in the buffer until the next chunk completes it. Complete lines are emitted immediately; partial lines wait.

When the stream ends, whatever remains in the buffer is emitted as the final line, even if that last line does not have a trailing newline.

Early Exit and Searching

You can break out of the loop early:

const input = fs.createReadStream('./access.log', { encoding: 'utf8' });
const rl = readline.createInterface({ input, crlfDelay: Infinity });

try {
  for await (const line of rl) {
    if (line.startsWith('FATAL')) {
      console.log('Found:', line);
      input.destroy();
      break;
    }
  }
} finally {
  rl.close();
}

Breaking out of a for await loop closes the readline interface, but rl.close() does not destroy the input stream. When early exit is important, destroy the file stream yourself. Also remember that readline may already have read ahead, so early exit stops future reads rather than guaranteeing an exact byte count.

Batched Concurrent Processing

Sequential line handling is easy to reason about, but it can be too slow when each line triggers an independent async operation such as an API call or database insert. Batching adds controlled concurrency:

const batch = [];
for await (const line of rl) {
  batch.push(processLine(line));
  if (batch.length >= 20) {
    await Promise.all(batch);
    batch.length = 0;
  }
}
if (batch.length > 0) await Promise.all(batch);

This reads 20 lines, processes them concurrently, waits for all of them to complete, and then moves to the next batch. The await Promise.all(batch) call limits application-level concurrency so the program does not accumulate an unbounded number of in-flight promises. It does not prevent readline from having already buffered more input.

readline vs. Manual Chunk Splitting

You could implement line-by-line reading yourself with createReadStream and split('\n'):

const stream = fs.createReadStream('./file.txt', 'utf8');
let leftover = '';
stream.on('data', (chunk) => {
  const lines = (leftover + chunk).split('\n');
  leftover = lines.pop();
  for (const line of lines) processLine(line);
});
stream.on('end', () => {
  if (leftover) processLine(leftover);
});
stream.on('error', (err) => {
  console.error('read failed:', err.code);
});

Manual splitting handles the simple case. readline handles the cases that make the simple version less reliable: proper \r\n handling, crlfDelay behavior for cross-platform compatibility, and integration with the promises API and async iterators. You still own input stream errors and cleanup. For anything beyond a quick script, readline is usually the better starting point.

How libuv Dispatches File I/O

The libuv dispatch path explains much of the performance profile described earlier.

Node's portable filesystem model comes from libuv. Libuv runs filesystem operations through its thread pool by default. Some libuv versions experimented with io_uring for selected Linux filesystem operations, but libuv reverted to thread-pool-by-default behavior in v1.49. For Node v24, the working model is that async fs calls usually avoid blocking JavaScript by running blocking filesystem work on libuv worker threads.

When you call fs.readFile('./data.json', callback), Node's JavaScript layer first validates your arguments and creates an internal filesystem request object. That request wraps libuv's uv_fs_t type, which records the filesystem operation, path or descriptor, buffers, flags, result storage, and completion callback state. The exact C++ class names are Node implementation details; the useful model is the libuv request that carries the operation across the native handoff.

Node then calls uv_fs_open(). For asynchronous use, libuv schedules that work on its thread pool. One of the pool's worker threads, 4 by default and configurable with UV_THREADPOOL_SIZE up to 1024, picks up the request when it becomes available.

The worker thread executes the blocking open() syscall, or the platform equivalent. The OS traverses directory entries, checks permissions, allocates a file descriptor or handle, and reads metadata from storage if it is not cached. This can take microseconds on a warm cache or milliseconds if it requires disk I/O. Because it happens on a worker thread, the main event loop continues running JavaScript concurrently.

Once open() returns, the worker thread stores the file descriptor or handle, or an error code, in the uv_fs_t result field. For a path-based readFile, Node may issue metadata work such as uv_fs_fstat() to size the file, followed by one or more uv_fs_read() requests. Finally, uv_fs_close() follows the same pattern for descriptors Node opened internally.

After the last operation completes, libuv notifies the event loop thread. Node retrieves the result from the request state and invokes the JavaScript callback or resolves the promise with the data or error.

The full cycle is the shape to remember: JavaScript, native binding, libuv request, worker thread, syscall, stored result, event-loop notification, and JavaScript completion. A path-based readFile triggers open, one or more reads, and close, plus metadata work when Node can use it to size the buffer.

Thread Pool Contention

The default pool has 4 worker threads, so at most 4 thread-pool tasks execute simultaneously unless you change the size. If you start 100 concurrent filesystem calls that all need the pool, 4 run at once and the rest wait in the queue. That queue can become the bottleneck in I/O-heavy applications.

You can increase UV_THREADPOOL_SIZE:

UV_THREADPOOL_SIZE=16 node server.js

Set it before the Node process starts. Assigning process.env.UV_THREADPOOL_SIZE = '16' inside application code is not guaranteed to work because the pool may already exist before your code runs.

More threads mean more memory and more scheduling overhead. Libuv gives worker threads an 8 MB stack by default, with stacks growing lazily on most platforms, and more threads also create more context switching. There is no universal best value; measure the workload you actually run.

The thread pool is shared, which is why filesystem latency can be affected by work that is not file I/O. fs operations share the pool with DNS resolution through dns.lookup, some crypto operations, and zlib compression. If a burst of DNS lookups occupies all 4 threads, file reads queue behind them. Thread-pool contention can produce file I/O latency spikes even when the storage path is healthy.

Sync Variants Skip the Thread Pool

readFileSync, writeFileSync, and other sync calls bypass the thread pool entirely. They call the blocking syscall directly on the main thread. That is why they freeze the event loop: there is no offloading, no background execution, just a direct open() + read() + close() sequence that blocks everything until the disk responds.

Promise-based APIs use the same thread-pool mechanism as callbacks. The only difference is in the JavaScript completion layer: instead of invoking a callback, Node resolves a Promise, which schedules microtasks. The underlying I/O path is identical.

Choosing the Right API

• Small config/JSON at startup: readFileSync. Keeps startup code simple; the cost is startup latency, not request-path contention.
• Small file in a request handler: fs.promises.readFile. Simple async whole-file read; memory still grows with file size and concurrency.
• Large file processing: createReadStream. Processes chunks with bounded buffering rather than one full-file allocation.
• Writing state/config: temp file + sync + rename + directory sync. Gives atomic visibility plus stronger crash-durability behavior where the platform supports the directory sync step.
• Appending log entries: appendFile or a write stream with flags: 'a'. O_APPEND gives per-write append positioning on local POSIX filesystems.
• Large data output: createWriteStream. Provides backpressure-aware output without collecting all data in memory.
• Binary format parsing: fs.read with explicit positions. Gives random access and byte-level control.
• Log analysis or CSV parsing: readline plus a read stream. Parses line by line with bounded buffering, while still allowing read-ahead.
• Durability after write: flush: true or fsync before close. Requests a file flush before completion; use directory sync as well when a rename must survive a crash.

The sync variants belong in startup code, CLI tools, and scripts. In any path that runs while the event loop is handling concurrent work, use the async versions. The promise-based API (fs.promises.*) is usually the cleanest modern surface because async/await syntax, try/catch error handling, and the rest of your async code all compose around it.

There is a spectrum of control. readFile and writeFile handle opening, sizing, reading or writing, and closing. Streams add chunked processing with flow control. fs.read and fs.write expose byte-level position and buffer management. readline adds line-oriented parsing on top of streams, with its own read-ahead behavior. Each step down trades simplicity for precision. Pick the highest abstraction that solves the problem, and drop lower only when you need the control.

The governing tradeoff is memory, ordering, and durability. Buffering buys simple code at the cost of memory proportional to file size. Streams keep memory bounded by accepting chunked control flow. Descriptor calls give ordering and position control. flush, fsync, and directory syncs are the tools for writes that must be visible and still present after a crash.

Related Reading

• Previous subchapter: Node.js File Descriptors: fs.open, FileHandle, Flags, and EMFILE
• Next subchapter: Node.js fs.promises and FileHandle: Async File Operations and Resource Cleanup