Lesson 2.4

Walking the request line

Byte codes instead of characters, half-open [start, i) token bounds, the invisible off-by-one, and the request-line scan that fills method, path, query and version - all inside the parser skeleton it lives in.

35 minsrc/parser.js
By the end you can
  • Scan the request line into method, request-target and version using integer byte compares.
  • Explain why a token is [start, i) and how including the delimiter creates an invisible bug.
  • Split the request-target on the first ? into path and rawQuery, and validate the version by bytes.
  • Return null for WANT_MORE and throw HttpError(400) for malformed framing.

Now comes the parser itself. You'll build it as a state machine that walks the arrival buffer one byte at a time, and this lesson does the first half - the skeleton, plus the scan of the request line into method, path, query and version. The scan has one classic bug, a trailing space which is invisible in the terminal and silently breaks every route match. You'll prevent it by construction, using the token rule below.

In this lesson

Byte codes, not characters

You're comparing raw octets here, so you work in byte codes, not character literals. buf[i] returns a number from 0 to 255. Comparing it with an integer is about as cheap as computing gets, and it keeps the scan loop free from string comparisons, which would allocate and run slower. Name the codes you branch on once, and use those names everywhere.

text
SP    = 0x20   space           '  '
  HTAB  = 0x09   horizontal tab   '\t'
  CR    = 0x0d   carriage return  '\r'
  LF    = 0x0a   line feed        '\n'
  COLON = 0x3a   ':'
  QMARK = 0x3f   '?'

buf[i] === SP is an integer comparison. Walking bytes by code uses these numeric values without creating any strings.

A token is [start, i)

A delimiter is a byte that marks the end of a token. The space after GET is the method delimiter. A half-open range includes its start index and excludes its end index.

text
a token is the half-open range [start, i)
  -- from start up to but NOT INCLUDING the delimiter byte at i

The scan pattern never changes. You remember where a token started, walk forward till you hit its delimiter, and the token is everything you walked over - not the delimiter itself. Both subarray(start, end) and toString(enc, start, end) are end-exclusive i.e they take bytes [start, end). So when your loop stops with i sitting on the space after the method, the method is [0, i).

An incorrect end index can pull in the delimiter also. Say the method scan stops on the space at index 3.

Half-open token bounds around GET and its delimiter

Figure 2.3 - The half-open range ends before the delimiter, keeping the invisible trailing space out of the method token.

"GET " with a trailing space renders exactly same as "GET" in every terminal, it echoes back looking perfect, and it makes every method === "GET" comparison come out silently false. Your router will never match a route, and you won't be able to see why. The same off-by-one on the version scan drags a trailing \r onto the version. So the half-open range is a correctness requirement here. If you get it wrong, the parser looks fine while every comparison keeps failing.

The scan pattern, and what counts as malformed

Every token uses the identical loop. Here is the method in pseudocode.

text
i = 0
  while buf[i] != SP:            # walk to the delimiter
      if buf[i] is CR or LF: reject   # a line break inside a token is malformed
      i++
  method = buf[0 .. i)          # the token: exclusive of the delimiter at i
  i++                           # step over the SP

The target and version use the same loop structure. Walk to the delimiter, take [start, i) as the token, then step over the delimiter. The target ends at the next SP. The version ends at CR, after which the parser confirms the LF.

  • Running out of bytes means WANT_MORE. If i reaches the end of the buffer mid-token (i >= len), the request is incomplete, not malformed. You return null. That is the honest "come back with more bytes".
  • A CR or LF inside a token means malformed. A line break sitting where a token byte should be means the request line is broken - there is no legal newline inside a method, target or version. That throws HttpError(400, "ERR_MALFORMED_REQUEST_LINE"). Same goes for an empty token (two spaces in a row, or a leading space).

Splitting the target, and checking the version

The request-target /users?id=7 contains a path (/users) and a query string (id=7). The router later matches the path, and another chapter parses the query string. url stores the complete target. path stores the bytes before the first ?. rawQuery stores the bytes after that delimiter, excluding the first ?, or "" when the delimiter is absent. Any later ? bytes stay inside rawQuery as it is.

Validate the version without building any string. It must be exactly eight bytes, HTTP/1. followed by 1 or 0. Compare H, T, T, P, /, 1, ., and the final version digit at their known offsets. Reject HTTP/2.0, a truncated version, and anything else. Then materialize the three bytes 1.1 or 1.0 which come after HTTP/.

The scan materializes method, URL, version, path, and a nonempty query. It never creates a whole-request string or a line array.

The parser skeleton

The scan lives inside a Parser class. This lesson builds its skeleton plus the request-line half of next().

The push(chunk) method in this chapter stores just one chunk. It assumes one 'data' event carries one complete request. Chapter 3 replaces it with accumulated bytes and a saved cursor. A cursor is the numeric index of the next byte the parser will inspect.

next() returns a Request when a full header block has arrived, null (WANT_MORE) when it needs more bytes, and it throws HttpError(400) on malformed framing. This lesson builds next() till the end of the request line and up to the header block. After the request line, you recognize the empty header block (an immediate CRLF terminator) and return the Request. A request that actually carries field lines comes in the next lesson, so you'll verify this stage with a header-less request.

Build it

Create src/parser.js.

  • Export the eight state constants with values 0 through 7, in this order - REQUEST_LINE, HEADERS, READ_BODY, CHUNK_SIZE, CHUNK_DATA, CHUNK_CRLF, TRAILERS, DONE. All eight exist from now itself, so later chapters keep the same state names and values.
  • Define the six byte-code constants above as module-private values.
  • Import HttpError from ./errors.js and Request from ./request.js.
  • A class Parser with the members below.
    • constructor(options = {}) - three fields in fixed order. _options (store options; it stays unused for several chapters), _buf = null, _state = REQUEST_LINE.
    • reset() - set _buf = null and _state = REQUEST_LINE.
    • push(chunk) - store this._buf = chunk.
    • next() - walk a local cursor i from 0.
      • If _buf is null, return null.
      • Scan the method to the first SP (rejecting CR/LF inside, and an empty method, as ERR_MALFORMED_REQUEST_LINE; i >= len returns null). Step over the SP.
      • Scan the target to the next SP (same rejects). Step over the SP.
      • Scan the version to CR; then confirm the following byte is LF (i >= len at either point returns null; a CR not followed by LF is malformed). Step over the CRLF.
      • Validate the version token - exactly 8 bytes, HTTP/1.1 or HTTP/1.0, compared byte by byte; anything else is ERR_MALFORMED_REQUEST_LINE.
      • Construct a Request, set _buf on it, and fill method, url, and version (the 1.x part) with buf.toString("latin1", ...).
      • Scan the target's bytes for the first QMARK; set path and rawQuery accordingly (the whole target and "" when there is no ?).
      • Set _state = HEADERS. Then handle the header block terminator only - if the next byte is CR and the one after is LF, step over both, set _state = DONE, and return the Request. Field-line parsing comes in the next lesson.
  • Every malformed case throws new HttpError(400, "ERR_MALFORMED_REQUEST_LINE"), and every "not enough bytes yet" returns null.
Hints

Reveal these one at a time, and only when you are genuinely stuck. Each one tells you a little more than the last - none of them is the answer.

    Verify it

    Feed the parser a header-less request and check what the request-line scan produced.

    bash
    node -e '
  import("./src/parser.js").then(({ Parser }) => {
    const p = new Parser();
    p.push(Buffer.from("GET /users?id=7 HTTP/1.1\r\n\r\n", "latin1"));
    const req = p.next();
    console.log("method  :", JSON.stringify(req.method));
    console.log("path    :", JSON.stringify(req.path));
    console.log("rawQuery:", JSON.stringify(req.rawQuery));
    console.log("url     :", JSON.stringify(req.url));
    console.log("version :", JSON.stringify(req.version));
  })'
    text
    method  : "GET"
path    : "/users"
rawQuery: "id=7"
url     : "/users?id=7"
version : "1.1"

    method is exactly "GET" with no trailing space - that invisible bug, prevented by construction. The path is /users, with the query stripped off into rawQuery. Now confirm the WANT_MORE case - an incomplete chunk returns null, it doesn't throw.

    bash
    node -e '
  import("./src/parser.js").then(({ Parser }) => {
    const p = new Parser();
    p.push(Buffer.from("GET /us", "latin1"));
    console.log("mid-line:", p.next());
  })'
    text
    mid-line: null

    And a bad version throws a 400.

    bash
    node -e '
  import("./src/parser.js").then(({ Parser }) => {
    const p = new Parser();
    p.push(Buffer.from("GET / HTTP/2.0\r\n\r\n", "latin1"));
    try { p.next(); console.log("NO THROW -- bug"); }
    catch (e) { console.log(e.code, e.statusCode); }
  })'
    text
    ERR_MALFORMED_REQUEST_LINE 400
    Common mistakes
    • method === "GET" comes out false even though it echoes as GET - means you sliced [start, i] inclusive and dragged the delimiter space into the token. The token is [start, i), and the delimiter at i is never part of it.
    • The path comes out as /users?id=7 - you split the target on the space but never on the ?. Scan for the first ? inside the target and cut there; everything after it is rawQuery.
    • A truncated GET / HTTP/1 or HTTP/2.0 gets accepted - means you didn't validate the version by bytes. It must be exactly eight octets, HTTP/1.1 or HTTP/1.0.
    • An incomplete request throws a 400 instead of returning null - you treated "ran out of bytes mid-token" as malformed. Out of bytes is WANT_MORE (return null); only a CR/LF inside a token or an empty token is a reject.
    Recap

    Your parser now has a skeleton and it reads a request line (method, url, path, rawQuery and version) by walking each byte only once, materializing just the four tokens it keeps, splitting the query off the path, and validating the version octet by octet. It returns null for want-more and throws a 400 for a broken line. It stops at the header block. The next lesson fills in the field-line loop, including the whitespace-before-colon reject.