lexer: fixed syntax highlighting breaks on longer files

[jorbakk]

This patch fixes issue #1253

Problem description

If lexer starts parsing at a file offset determined by the horizon variable and this offset happens to be within a string, then all non-strings are highlighted as strings and vice versa.

This occurs with strings that have identical characters for their opening and closing delimiters, e.g. single quotes ', double quotes " or quotes like """. Paired delimiters that use different types of characters for start and end of the string are not affected. Most programming languages use identical characters as string delimiters. Exceptions are:

• m4 macro processor uses backtick ` at the start and apostrophe ' at the end
• Tcl allows quotes and braces {}
• postscript uses parantheses

The current approach also fails when the horizon lies completely within a very big token. This is for example the case with big comments, when the starting delimiter of the comment is before the horizon. Then the lexer treats the comment as un-commented program text.

The lexer currently fails quite frequently on files with:

• large multi-line strings, e.g. for documentation, SQL statements, statements in embedded DSLs
• stb-style header-only libraries in C/C++, where there is usually a very large comment in the beginning of the file, which contains the API documentation of the library

Patch description

With this patch, if the horizon variable is set to -1, the whole file buffer up to the end of the viewport is run through the lexer. Performance is still reasonably good, as no syntax highlighting takes place outside of the viewport.

However, with this simple patch, performance degrades with big files when the viewport is close to the end of the file. A set of patches will follow shortly that overcome this limitation while also yielding the same correct results as this simple patch. It also achieves generally a big performance boost in almost all cases compared to the current horizon approach.

Edit 2026-09-06: more explicit and accurate problem description, including examples

[jorbakk]

Without the patch, this would just set the horizon to -1 which means that lexing of the file begins at viewport.start - (-1) = viewport.start + 1 which is a one byte offset into the current viewport. With the patch, the exact same command sets the horizon to the beginning of the file. So, you also need this command to set the horizon with the patch to activate it. I made it configurable so that you can opt out if you might have performance issues on very slow machines. Currently, it's deactivated by default and it follows the old behaviour for backward compatibility. However, lexing is fast and otherwise syntax highlighting is simply not correct and I see no other way to fix this issue other than to parse the file from the beginning. So it might be reasonable to activate it by default and remove the horizon variable. I would be happy to send a patch to do so.

…

On 2026-05-30 22:53, Joe M wrote: *4d30* left a comment (martanne/vis#1357) <#1357 (comment)> Cannot the horizon command do this? |:set horizon -1 | — Reply to this email directly, view it on GitHub <#1357?email_source=notifications&email_token=AWXEBLIV3AKRBADWW7ELNLD45NC5HA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJYGQ3TENBXGA22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4584724705>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXEBLO4SZBPGXMCKNYPACL45NC5HAVCNFSM6AAAAACZQQWGF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DKOBUG4ZDINZQGU>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jorbakk]

This next set of patches provides incremental lexing of a file while introducing a cache for already lexed tokens. This makes the process of lexing the text into tokens for syntax highlighting much more efficient.

In order to achieve this, an event is needed that signals changes in the text, in particular where and when these changes occur. Then only the affected lexer tokens need an update, so the lexer can start from the position of the change until the end position of the current viewport. This is the minimal possible change of the token cache that also preserves as much tokens as possible and maintains a correct list of tokens that reflect the document's syntax.

Any newly created tokens should only depend on the position of changes in the document and the position of the viewport. They should not depend on a fixed sized horizon, as this leads to incorrect results and should not be required in any text editor. In addition to a correct token list, this approach also gives a performance boost, e.g. when scrolling down through a freshly opened document. Then only the new viewport has to be lexed. After hitting the bottom of the text, the whole document is in the cache on no lexing is required when moving through it, until the document is changed. If this happens, only the tokens from the cursor, where the changes take place, to the end of the viewport need to be lexed, again.

Each window maintains its own token cache, so a different syntax can be set for the same document in different windows.

[Nomarian]

I don't understand any of this but localize your variables

[jorbakk]

I don't understand any of this but localize your variables

Yep, missed a few, hope I got them all now.

[Nomarian]

run luacheck on it

[jorbakk]

run luacheck on it

Okay, should be clean now. Thanks!

[Nomarian]

put local i repeat here and remove lines 67-70, repeat runs at least once so local i can be unnassigned

[jorbakk]

Great twist, much better!

[fischerling]

Hey thanks for your work!

I will test it and will report back.

Especially, the modification event looks really promising to me. I had this kind of event on the to-do list for vis-lspc for a long time.

[jorbakk]

Hey thanks for your work!

I will test it and will report back.

Especially, the modification event looks really promising to me. I had this kind of event on the to-do list for vis-lspc for a long time.

Thanks! I can imagine that the modification event would be useful for interactive use of an LSP client, like auto-complete. It actually sounds like a basic event to be provided by an editor. Looking forward to your feedback!

[rnpnr]

Now that this was at the top of my inbox again I took a quick look. I don't remember why I didn't comment on the original but this is looking quite good.

One thing I think should be changed is that we should disable lexing for "large" files. I don't see any reasonable way of handling them without doing partial lexing and I don't want your cache to blow up in size just because someone opened a "large" file.

Right now a file being large or not seems to be determined in lua but it's a property that's set during the creation of the window in C. I think it should just be exported in window_index() in vis-lua.c.

Also if you can leave a comment documenting how the cache is actually organized that would be helpful. I have a plugin which maintains a cache and I know that vis-spellcheck also has a token cache and probably we would both prefer to just access this one directly (even if we aren't explicitly making it public API).

EDIT: What I mean by the last part is that both these plugins are trying to avoid re-lexing the file over and over. They want access to the token list directly. I suppose an explicit API might be better but even just access to this cache would be sufficient.

[Nomarian]

I don't know the C side but from what I understand, Disabling syntax highlighting is just done by setting win.syntax = nil, statusbar event already sets a win.large boolean, it could be a one line change to just add a win.syntax = nil to the conditional, the other PR already disables syntax highlighting for the window as well via vis.command"set syntax off"... But I do think a cache would be nice, maybe both must be used, win.huge files and win.large, with huge files disabling many facilities, like the syntax highlighting and etc and large turning on the cache mode.

[jorbakk]

@rnpnr: I agree that lexing and syntax highlighting should be turned off on large files, so I had a closer look to the current way large files are handled:

• there is View.large_file (C) and win.large (lua), they are set independently, albeit using the same threshold: file size > 2^25 or selection.col > 2^16
• there's also a ui option UI_OPTION_LARGE_FILE that sets View.large_file using win_options_set(), however it cannot be set with a :set command.
• the thresholds are hard coded (same value, but in two different places)
• View.large_file and win.large are checked by polling file/line size on each STATUS_BAR update

I think that "large file" should be a property/function of the file, without storing it as additional state that needs to be updated. So I made the following changes:

• removed UI_OPTION_LARGE_FILE property.
• introduced one single function that checks for large files and depends only on the text state.
• made this function available in the lua API.
• introduced thresholds for file and line size that can be set as ui options per window to adjust the large file check.
• Turn off lexing and syntax highlighting when check for large files is true or no syntax is set for the window.

I also added documentation for the token cache.

@Nomarian: do you think there should be two thresholds for large and huge files? For now, I think one adjustable threshold should suffice and is simpler. Keep in mind that the token cache should be enabled whenever syntax highlighting is enabled, for performance reasons.

[rnpnr]

One issue I have found is that when you yank some text from another window and then paste it the cache after the pasted text is not invalidated. For example open 2 of the c files in vis and yank a comment from one and paste it above some other code in the other.

[Nomarian]

@Nomarian: do you think there should be two thresholds for large and huge files? For now, I think one adjustable threshold should suffice and is simpler. Keep in mind that the token cache should be enabled whenever syntax highlighting is enabled, for performance reasons.

If your cache system works then there should be no need for it, I was merely stating this because its a quick addition and fix on the nosyntaxhighlighting PR and it seemed to be what Randy wanted.

[jorbakk]

@Nomarian: do you think there should be two thresholds for large and huge files? For now, I think one adjustable threshold should suffice and is simpler. Keep in mind that the token cache should be enabled whenever syntax highlighting is enabled, for performance reasons.

If your cache system works then there should be no need for it, I was merely stating this because its a quick addition and fix on the nosyntaxhighlighting PR and it seemed to be what Randy wanted.

Okay. I'll need to take a closer look at your PR, but I would guess that this patch here only reads win.syntax and your patch sets it. Let me know if there's any potential conflict, and I'm sure we can fix it.

I also replied the following (with little edits) to your other comment (via email), but it bounced, the github notification bot seems to be busy:

Sorry, but I can't follow. The WIN_STATUS event callback in vis-std.lua did set win.large before I changed win.large: bool to a function win:large() -> bool. I also changed the WIN_STATUS event callback, so there should be no conflict. Please let me know if I missed s.th. here.

[jorbakk]

One issue I have found is that when you yank some text from another window and then paste it the cache after the pasted text is not invalidated. For example open 2 of the c files in vis and yank a comment from one and paste it above some other code in the other.

@rnpnr : Sorry, but I have a hard time to reproduce this ...?! I could not manage to corrupt the token cache by yanking and pasting any code snippet from one window to another ... .

[rnpnr]

@rnpnr : Sorry, but I have a hard time to reproduce this ...?! I could not manage to corrupt the token cache by yanking and pasting any code snippet from one window to another ... .

My apologies, please disregard. I was testing on a different system from usual on top of my personal work branch and I didn't realize that it must have been out of date.

[rnpnr]

I do have a similar example which is still an issue:

1. Open an existing C file
2. Start typing #inclu (this is correctly not recognized)
3. Finish typing so you now have #include (this is incorrectly not recognized until you do something which invalidates the cache).

Note that in order to reproduce the file needs to have more than one token. I you open a new file t.c and only do these steps this first token seems to work as intended.

[jorbakk]

I do have a similar example which is still an issue:

1. Open an existing C file
2. Start typing #inclu (this is correctly not recognized)
3. Finish typing so you now have #include (this is incorrectly not recognized until you do something which invalidates the cache).

Note that in order to reproduce the file needs to have more than one token. I you open a new file t.c and only do these steps this first token seems to work as intended.

@rnpnr Great that you spotted this! I should have stumbled across it also, but the devil's in the details ...

So here's a small patch with a long answer. I'd like to start with your example and the corresponding lexer outputs:

File contents:

int main();

lexer output:

{ type, 4, whitespace.c, 5, function, 9, operator, 10, operator, 11, operator, 12, whitespace.c, 14 }

File contents:

int main();
#inc

lexer output:

{ type, 4, whitespace.c, 5, function, 9, operator, 10, operator, 11, operator, 12, whitespace.c, 13, default, 14, identifier, 17, whitespace.c, 18 }

File contents:

int main();
#include

lexer output with min_token_cache_entries = 4, looking back 2 tokens:

{ type, 4, whitespace.c, 5, function, 9, operator, 10, operator, 11, operator, 12, whitespace.c, 13, default, 14, identifier, 21, whitespace.c, 22 }

The hash sign (now identified as 'default') is ignored and only the string 'include' and the following whitespace are re-lexed. This is why it's not identified as a preprocessor token.

lexer output with min_token_cache_entries = 6, looking back 3 tokens:

{ type, 4, whitespace.c, 5, function, 9, operator, 10, operator, 11, operator, 12, whitespace.c, 13, preprocessor, 21, whitespace.c, 22 }

The three tokens '#' = 'default', 'include' = 'identifier', and whitespace = 'whitespace.c' are re-lexed and correctly identified as 'preprocessor' and 'whitespace.c'.

Note that the incremental lexing performed here always starts at token boundaries, currently two tokens back from the modification position (min_token_cache_entries = 4). The question is how many tokens do we need to go back for incremental lexing with any grammar? This is obviously related to the complexity of the grammar, so I made a quick analysis of all grammars in lua/lexers and came to the conclusion that 8 tokens (that's 16 entries in the token cache) should cover all grammars, even the weird ones.

edit: some small changes

[jorbakk]

While testing it as my daily driver, I found another edge case which I need to take care of. Please bear with me ...