Getting started

Tokens

Every renderer takes the same inputs — source plus the token array from highlighter.highlight():

import { createHighlighter } from "@opentf/syntax-highlighter";

const highlighter = await createHighlighter({ language: "javascript" });
const tokens = highlighter.highlight(source);

A token is { type, start, end }: a semantic type such as keyword, string, function, number, comment, or operator, plus exclusive UTF-16 offsets into source. Tokens always cover the full source with no gaps or overlaps, so renderers recover text via source.slice(start, end).

Choosing a language

Pass any of the 102 supported languages by name or alias:

await createHighlighter({ language: "python" });
await createHighlighter({ language: "ts" }); // alias for typescript

getRegisteredLanguages() lists every name up front, and loadLanguage(name) loads a definition on demand. Unknown names throw UnknownLanguageError (with code: "UNKNOWN_LANGUAGE") instead of failing silently.

Custom languages

Register additional languages or aliases without touching the tokenizer or renderers:

import { registerLanguage, getRegisteredLanguages } from "@opentf/syntax-highlighter";

registerLanguage(definition); // LanguageDefinition
console.log(getRegisteredLanguages());

A LanguageDefinition carries the word lists (keywords, booleans, nulls, constants), lexer config (lex: strings, comments, operators), markup config, and opt-in features (parameterBindings, contextStack, declarations, retroactiveRewrite, typeAnnotationAware, propertyKeys, classDetection, lineBreakResetsExpectation, and the structure passes styleSheet, yamlStructure, and shellStructure). contextStack and declarations default to true; the rest default to false.

Next steps

• Renderers — emit the token stream everywhere.

• API reference — the full export list.