Streaming Parser Complete Guide

JSON6.parse wants exactly one value in one string. That is wrong for log files, event feeds, sockets and anything where text arrives in pieces or where one document contains many values. JSON6.begin() creates a parser that keeps its state between calls. The README describes it as a way to get "objects as they are encountered in a stream". Its signature is JSON6.begin(callback[, reviver][, options]), and the parse options and the reviver behave exactly as they do for parse.

The returned object has three public methods. write(string) feeds text in and calls your callback once for each value that becomes complete. value() returns the most recently completed value, which matters only if you drive the lower-level interface. reset([options]) clears an error state so you can keep using the same parser. There are two further members in the declarations, _write and finalError, which are internal; the low-level section shows what _write does.

stream-basic.mjs

Run it:

Output:

Values in a stream are separated by whitespace (including newlines), by comments, or by nothing at all when the previous value ends in a closing bracket or quote. They are not separated by commas: {a:1},{b:2} throws "bad context; excessive commas". That matches how JSON text sequences and line-delimited JSON work, and it is different from an array, which is one value that contains many.

A number, or a bare word like true, has no closing delimiter. When a chunk ends with 12 the parser cannot tell whether the next chunk will start with 3. So it waits. An object, array or quoted string closes itself, but a trailing number does not. There are two ways to finish it: send any whitespace, or call write('') with an empty string, which tells the parser that the input has ended for now.

stream-numbers.mjs

Run it:

Output:

Two outcomes in that table are worth stating plainly. An unterminated object followed by an empty write throws "context stack is not empty at flush", which is the right behavior: the input ended in the middle of a structure. An unterminated string followed by an empty write does not throw and delivers nothing in my test, even though the README says a split string throws on flush. Do not rely on the empty write to detect truncation of a string.

The parser accepts strings. Anything else you pass to write is converted with string conversion first, which means a Buffer is decoded as UTF-8 on every call. That is a trap with multi-byte characters: when a network or file chunk boundary falls inside a character, each half decodes to the replacement character U+FFFD and the data is silently corrupted. In my test a two-byte character cut in half produced two replacement characters followed by the second character, while the same bytes through a streaming decoder came out right. So decode first. Either call stream.setEncoding('utf8'), as the previous example does, or use a TextDecoder with the stream option.

stream-decoding.mjs

Run it:

Output:

Now the full pattern: a file read in tiny chunks. The file below is three records, read four bytes at a time, which splits every record across many chunks.

stream-file.mjs

Run it:

Output:

For real pipelines a Transform stream in object mode is a better fit than loose callbacks, because it plugs into pipeline() and gives you backpressure and error propagation. The class below wraps the parser. It decodes bytes with a StringDecoder, pushes every completed value downstream and flushes the parser when the input ends. Parse errors become stream errors.

stream-transform.mjs

Run it:

Output:

One consequence of the design: the whole value has to be in memory before the callback fires. A giant single array of a million records is one value and will be held in full. The stream parser is for many moderately sized values, not for incrementally walking one enormous document.

Under write() sits _write(string, completeAtEnd), which the README documents as a low-level routine. It returns a number: 0 when no value completed, 1 when a value completed and there is no more input, 2 when a value completed and more input remains. You then call value() to fetch the completed value, and call _write() with no arguments to continue through the buffered input. The leading underscore means it is not part of the stable API, so prefer write with a callback. It is useful when you want a pull-style loop and not a callback. The README prints this loop as the internal implementation of write. In the run below every completed value returned 2.

stream-lowlevel.mjs

Run it:

Output:

Note that JSON6.begin() with no callback is allowed when you use _write, but write() then throws, since write can only hand values to the callback. That behavior was added in 1.1.6; earlier versions required the callback.