function tokenize
tokenize(source: string): Token[]

The source, split into what nvim would capture.

A scanner rather than a parser: this runs on every keystroke, and the questions it answers — what colour is this, is that brace real, am I inside a string — need lexing and nothing more. Identifiers are classified by the character either side, which is how a function call is told from a variable and a property from a name.

A regex literal is recognised by what precedes it, the same heuristic every syntax highlighter uses, because / is division or a literal depending on where it appears. Nothing here is type-aware, so a capitalised name is @type — the convention, not the truth.

import { tokenize } from './highlight.ts';

tokenize('a.b')[1]?.capture; // '@property' — after a dot
tokenize('a(1)')[0]?.capture; // '@function.call' — before a bracket

Parameters

source: string

Return Type

Usage

import { tokenize } from "lib/repl/highlight.ts";