Hand-marked tokens

A grammar sees syntax only. A language server also knows that one name is a method and another a local, and a book about it needs to show that. Add the lsp annotation to a block, and wrap text in §token|text§, where token is an LSP semantic token type, optionally followed by modifiers, separated by dots.

```rust,lsp
enum §enum|E§ { A, B }
§function.defaultLibrary|scan§(list, print);
```

In such a block § is the marker: text alternates between code and tokens, and nothing else is read, so it works the same in any language, $ and ${…} included, in strings and comments too. In the rare code that contains a §, write §§ for one:

```bash,lsp
echo $HOME is §variable|home§ and costs §§5
```

An unbalanced § or a token without a valid type| is reported and the block is skipped. To read § this way in every block of a language, set lsp = true for it under [preprocessor.tsitter.languages].

To recolour a token, set the CSS variable named after it, anywhere in your additional-css:

:root {
  --lsp-enum: #d699b6;
  --lsp-method: #e69875;
  --lsp-mod-deprecated: #859289;
}

--lsp-<type> colours a token type, written in lowercase (--lsp-enummember), and --lsp-mod-<modifier> a modifier (--lsp-mod-defaultlibrary). No selectors or specificity to think about, and it works the same in every theme, Everforest included. Variables left unset keep the bundled colours. These names are stable within a major version: the classes lsp-<type> and lsp-mod-<modifier> and the variables --lsp-<type> and --lsp-mod-<modifier> are renamed or removed only in a new one.

The bundled stylesheets know every token type and modifier of m2-ls and rust-analyzer, and the standard ones of the protocol, in every mdBook theme and in Everforest. A token type has the colour of the tree-sitter capture closest to it (an enum is coloured like a type). A modifier leaves the colour of its type alone until you set --lsp-mod-<modifier>; strong, emphasis, link, deprecated and intraDocLink also set bold, italic, underline and strike-through.

Modifiers are what make marks expressive, because they combine with any type. Compound selectors style exactly a type with a modifier:

:root { --lsp-mod-controlflow: var(--ts-orange); }   /* `match`, `if`, `.await` */
.lsp-variable.lsp-mod-mutable { text-decoration: underline; }
.lsp-function.lsp-mod-async { font-style: italic; }
code[data-language="rust"] .lsp-mod-unsafe { color: var(--ts-red); }

Token and modifier names are not limited to the LSP standard; a server’s own types work too. The bundled stylesheets colour the standard ones, and anything else stays plain until you give it a rule. Map it to one of the colour roles (--ts-type, --ts-function, --ts-keyword, --ts-string, --ts-number, --ts-special, …) or a hue (--ts-orange, --ts-blue, …), which every mdBook theme and Everforest redefine, and one line covers all of them:

.lsp-mything { color: var(--ts-special); }

For more than colour, the span is <span class="lsp-function lsp-mod-defaultlibrary"> inside a <code data-language="…">, so ordinary selectors work too: code[data-language="m2"] .lsp-method { font-style: italic; }.

• The markers are removed before the code is parsed, so the grammar, the Copy button and the Run button see plain code.
• A token can span several grammar tokens, such as std::mem::drop, and styles each of them. Tokens do not nest.

Marks and Neovim

tools/lsp-marks is a Neovim plugin (add it to the runtime path) that converts in both directions:

• :[range]LspMarksYank copies the semantic tokens of the language server attached to the buffer as marks. It uses the server of your running editor, with your own settings, and leaves out what a grammar already colours.
• :LspMarksPreview [filetype] shows a buffer written in marks as highlighted code in a split that follows every edit. Markers are stripped first, so tree-sitter parses clean code, and tokens are coloured with @lsp.type.*, @lsp.mod.* and @lsp.typemod.*, as Neovim colours real semantic tokens.

To generate marks without an editor, run nvim --headless -l tools/lsp-marks/headless.lua <file> [first:last]; it starts the server itself. It is a bare LSP client, so it works with any server that speaks semantic tokens.

Example

A grammar sees a trait, a struct and an enum as the same thing: a type name. A language server knows which is which, and so can the page:

trait Area {
    fn area(&self) -> f64;
}

struct Circle(f64);
enum Shape { Round(Circle), Flat }

fn total(shapes: &[Shape]) -> f64 {
    shapes.iter().map(|shape| shape.area()).sum()
}

Marks only change what you see when the book styles them differently from the grammar. The bundled colours follow the nearest tree-sitter capture, so on their own interface, struct and enum all look like type. This book gives each a colour of its own with four lines, using the palette roles every theme defines, so they follow the theme:

:root {
  --lsp-interface: var(--ts-purple);
  --lsp-struct: var(--ts-orange);
  --lsp-enum: var(--ts-red);
  --lsp-method: var(--ts-cyan);
}

Nothing else is read in such a block, so $ and other code need no escaping. A modifier styles what the server knows about a name, here with a dotted underline on readonly:

echo $HOME in $(pwd)