Highlight strings and comments that span line breaks #8

Merged
claude-bot merged 1 commit from fix/multiline-string-highlighting into main 2026-11-20 13:38:12 +00:00
Collaborator

Problem

The highlighter tokenized each line on its own. After a line break inside a '...' string, the rest of the string was colored as SQL code. Multi-line "..." quoted identifiers and /* ... */ block comments broke the same way.

Fix

  • sql_highlight.rs: new LexState (Normal, SingleQuote, DoubleQuote, BlockComment). tokenize_sql_line_from(line, state) continues a run carried over from the previous line and returns the state at the end of the line. tokenize_sql_line is kept as the Normal-start wrapper.
  • line_start_states(lines) computes each line's start state with a byte scan of quotes and comments only. It builds no tokens, and it shares skip_quoted / skip_block_comment with the tokenizer so the two can't drift apart. Scanning bytes is exact because the relevant characters are ASCII, and UTF-8 never uses those byte values inside multi-byte characters.
  • vim_editor.rs: layout_sql_line takes the start state. States are computed before mouse hit-testing and again after keyboard input, and only down to a full screen past the scroll position or cursor.
  • Cost: a release build takes about 8ms per 100k lines for the full scan, and normal files take well under 1ms per frame.
  • Documented in AGENTS.md.

Testing

  • cargo test passes (28 tests), including 5 new highlighter tests: a string across lines, '' escapes across lines, block comments and quoted identifiers across lines, quotes inside -- comments, and agreement between line_start_states and the tokenizer.
  • Not yet checked by hand in the Windows GUI.

🤖 Generated with Claude Code

## Problem The highlighter tokenized each line on its own. After a line break inside a `'...'` string, the rest of the string was colored as SQL code. Multi-line `"..."` quoted identifiers and `/* ... */` block comments broke the same way. ## Fix - `sql_highlight.rs`: new `LexState` (`Normal`, `SingleQuote`, `DoubleQuote`, `BlockComment`). `tokenize_sql_line_from(line, state)` continues a run carried over from the previous line and returns the state at the end of the line. `tokenize_sql_line` is kept as the `Normal`-start wrapper. - `line_start_states(lines)` computes each line's start state with a byte scan of quotes and comments only. It builds no tokens, and it shares `skip_quoted` / `skip_block_comment` with the tokenizer so the two can't drift apart. Scanning bytes is exact because the relevant characters are ASCII, and UTF-8 never uses those byte values inside multi-byte characters. - `vim_editor.rs`: `layout_sql_line` takes the start state. States are computed before mouse hit-testing and again after keyboard input, and only down to a full screen past the scroll position or cursor. - Cost: a release build takes about 8ms per 100k lines for the full scan, and normal files take well under 1ms per frame. - Documented in `AGENTS.md`. ## Testing - `cargo test` passes (28 tests), including 5 new highlighter tests: a string across lines, `''` escapes across lines, block comments and quoted identifiers across lines, quotes inside `--` comments, and agreement between `line_start_states` and the tokenizer. - Not yet checked by hand in the Windows GUI. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The highlighter tokenized each line in isolation, so after a line break
inside a '...' string the rest of the string was colored as code. The
same happened to "..." quoted identifiers and /* ... */ block comments.
Each line now starts in the lexer state the previous one ended in
(LexState). The editor computes per-line start states with a cheap
byte scan of quotes and comments, limited to the lines that can be on
screen, and passes them to tokenize_sql_line_from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude-bot deleted branch fix/multiline-string-highlighting 2026-11-20 13:38:12 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
daniel/qry-duck!8
No description provided.