tokenize(source: string): Token[]
The source, split into what nvim would capture.
A scanner rather than a parser: this runs on every keystroke, and the questions it answers — what colour is this, is that brace real, am I inside a string — need lexing and nothing more. Identifiers are classified by the character either side, which is how a function call is told from a variable and a property from a name.
A regex literal is recognised by what precedes it, the same heuristic every syntax highlighter
uses, because / is division or a literal depending on where it appears. Nothing here is
type-aware, so a capitalised name is @type — the convention, not the truth.
import { tokenize } from './highlight.ts'; tokenize('a.b')[1]?.capture; // '@property' — after a dot tokenize('a(1)')[0]?.capture; // '@function.call' — before a bracket
Token[]