Does this use JSON.parse under the hood?

No, it's a hand-written lexer that scans the raw text one character at a time, the same way a parser's front end would, rather than delegating to the browser's built-in JSON.parse; this lets it tokenize text that isn't fully valid JSON, as long as each individual token is well-formed.

What token types can appear in the output?

brace-open/brace-close for { and }, bracket-open/bracket-close for [ and ], colon, comma, string (the full quoted string including its quote marks), number, boolean (for true/false), and null.

Is whitespace between tokens included in the output?

No, whitespace (spaces, tabs, newlines, carriage returns) between tokens is silently skipped and never emitted as its own token, keeping the output focused on meaningful lexical content.

What happens if the input has an invalid character?

The tokenizer stops and reports an error naming the exact unexpected character and its position, translated into a line and column number the same way JSON parse errors are reported elsewhere in this tool suite.

JSON Tokenizer Online - Lexical Token Breakdown

Overview

Introduction

Most JSON tools work with the parsed value, an object graph of nested arrays and objects, but sometimes you need to see the raw lexical structure of the text itself.

This tool exposes that lower level: the exact sequence of tokens the text breaks down into before any parsing into a value happens.

What Is JSON Tokenizer?

A hand-rolled JSON lexer that scans the input character by character (not via JSON.parse) and emits one token per structural character, string, number, or literal it recognizes.

It's the same kind of front-end pass a real JSON parser would run first, exposed directly as a tool rather than hidden inside a parser.

How JSON Tokenizer Works

The lexer walks the input position by position, skipping whitespace, and dispatching on the current character: punctuation characters become single-character tokens, a quote starts an escape-aware string scan, a digit or minus sign starts a number scan, and t/f/n dispatch to literal matchers for true/false/null.

Each recognized token is appended to a flat list as it's found; if a character doesn't match any recognized start (an unquoted word, a stray symbol), the lexer stops immediately and reports the exact character and position.

When To Use JSON Tokenizer

Use it to debug a hand-written JSON parser or template engine by comparing its token stream against this reference tokenization.

It's also useful for teaching or learning how JSON's grammar breaks down text into tokens before any structural parsing happens.

Often used alongside JSON Formatter & Beautifier, JSON Validator and JSON Analyzer.

Features

Advantages

Shows the exact lexical breakdown independent of whether the whole document parses into a valid structure.
Reports precise error location (line, column, and character position) on the first unrecognized character.
Distinguishes booleans and null as their own token types rather than lumping them in with strings or numbers.

Limitations

It only reports the first lexical error it finds and stops there; it doesn't attempt to recover and report multiple errors in one pass.
This is a lexer, not a validator, a token stream that lexes cleanly can still be structurally invalid JSON (e.g. mismatched braces); use JSON Validator for structural checking.

Examples

Tokenizing a small object

Input

{"a":1,"b":true}

Output

brace-open	{
string	"a"
colon	:
number	1
comma	,
string	"b"
colon	:
boolean	true
brace-close	}

Each character or literal maps to exactly one tab-separated type/value token line, in source order.

Reporting an unexpected character

Input

{"a": xyz}

Output

Error: Unexpected character "x" at position 6

The bare word xyz doesn't start with a quote, digit, or a recognized true/false/null prefix, so the lexer fails at the x.

Best Practices & Notes

Best Practices

Use this for debugging lexical issues (stray characters, malformed literals) before reaching for a full structural validator.
Pair with JSON Validator when you need to know whether the document is fully valid, not just lexically clean.
Feed it small snippets when debugging; the flat token list can get long fast on large documents.

Developer Notes

Errors are thrown as plain Error objects with a message ending in "at position N" and then passed through the same locateJsonError() helper the rest of this category's JSON.parse-based tools use, so error line/column reporting stays consistent across every JSON tool in this suite even though this one never calls JSON.parse.

JSON Tokenizer Use Cases

Debugging a hand-written JSON parser or serializer by comparing token streams
Teaching or learning how JSON text breaks down into lexical tokens
Diagnosing exactly which character in a malformed document a lexer chokes on

Common Mistakes

Expecting structural validation (matched braces, correct nesting); this tool only reports lexical, not structural, correctness.
Assuming whitespace tokens appear in the output; they're intentionally skipped and never emitted.

Tips

The reported error position is a zero-based character offset into the original input, matching the position field used by other JSON tools in this suite.
If you just need to know whether JSON is valid, JSON Validator is faster than reading through a full token list.

JSON Tokenizer