[sunrioa]

Summary

Fixes #6238

Markdown code blocks for Java, Go, Rust, C/C++, C#, and SQL currently render without syntax colors. This PR adds highlighting through Astryx's existing tokenizer extension point.

• Lazily load the selected Refractor grammars and map their tokens to Astryx's existing syntax colors. Reuse the current CodeBlock, copy button, collapse behavior, and light/dark themes.
• Normalize language aliases, including c++ / cpp and c# / csharp. Preserve document-global UTF-16 offsets and split multiline tokens at line boundaries.
• Keep existing built-in highlighting unchanged and update the generated third-party notices for the added dependency.

Scope is shared Markdown code blocks. Editors, diff previews, and standalone tool-output code blocks are unchanged.

Highlighting limits

The custom tokenizer reads prepared tokens rather than doing grammar work during React rendering. Streaming updates are debounced by 120 ms, and obsolete results are discarded when the source or language changes.

Extra-language highlighting is limited to 20,000 UTF-16 code units per block. Larger blocks, unknown languages, and highlighting failures remain readable as plain code without truncation. The optional grammar bundle is approximately 20 KB gzipped and loads only when needed.

Verification

• UI suite: 819 tests passed, including 24 new tokenizer and Markdown integration cases.
• Chromium suite: 7 tests passed, covering actual light/dark colors, aliases, lazy loading, clipboard contents, collapse controls, streaming/source changes, failed chunk loading, and long-code fallback.
• Build, typecheck, lint, format check, Knip for UI/Desktop, renderer architecture checks, third-party notices, and ASF header checks passed locally.
• Compared identical snippets against the unmodified baseline and the patched renderer. The contributor also manually tested the local macOS Desktop build and confirmed that highlighting works.

Not run: the full all-workspace test suite or native Windows/Linux application tests.

Before and after

The screenshots use the same Markdown snippets, Maka theme, and code-block components before and after the change.

Theme	Before	After
Dark
Light

AI use

• No generative tool made a substantive contribution
• Generative tooling made a substantive contribution

Tool(s) and scope: OpenAI Codex assisted with implementation, regression tests, local verification, and this PR description. The contributor manually tested the local Desktop build. This PR is published by Codex on the contributor's behalf after their review and approval.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

简体中文

修改概述

关联并解决 #6238。

目前 Java、Go、Rust、C/C++、C# 和 SQL 的 Markdown 代码块没有语法颜色。本 PR 通过 Astryx 现有的 tokenizer 扩展接口补充高亮。

• 按需加载所选 Refractor 语法定义，将分词结果映射到 Astryx 现有配色。继续复用原来的 CodeBlock、复制按钮、折叠行为和明暗主题。
• 统一识别 c++ / cpp、c# / csharp 等语言别名。保留全文 UTF-16 偏移，并按行拆分多行 token，避免后续行高亮错位。
• 原有内置语言的高亮保持不变，同时更新新增依赖对应的第三方许可声明。

修改范围仅限共享 Markdown 代码块，不涉及编辑器、diff 预览或独立的工具输出代码块。

高亮限制

自定义 tokenizer 只读取预先准备好的分词结果，不在 React 渲染过程中执行语法分析。流式更新使用 120 毫秒防抖；代码内容或语言发生变化后，过期结果不会应用到新内容上。

新增语言的高亮上限为每个代码块 20,000 个 UTF-16 代码单元。超过上限、语言未知或高亮失败时，仍完整显示纯文本代码，不截断原文。新增语法包 gzip 后约 20 KB，仅在需要时加载。

验证结果

• UI 测试：819 项通过，包含新增的 24 项分词器及 Markdown 集成测试。
• Chromium 测试：7 项通过，覆盖明暗主题实际颜色、语言别名、懒加载、复制原文、折叠控件、流式与内容切换、资源加载失败，以及长代码回退。
• 本地构建、类型检查、lint、格式检查、UI/Desktop 的 Knip 检查、渲染器架构检查、第三方许可声明及 ASF 文件头检查均通过。
• 已使用相同代码对比未修改的基线和修改后的渲染结果；贡献者也已在本地 macOS Desktop 中手动测试，确认高亮可用。

未执行：全仓库所有工作区测试，以及 Windows/Linux 原生应用测试。

明暗主题的前后对比截图见上方英文正文。

AI 使用说明

OpenAI Codex 协助完成实现、回归测试、本地验证及 PR 描述。贡献者已手动测试本地 Desktop 构建。本 PR 发布前已由贡献者本人审核并批准，由 Codex 代为发布。

检查清单

• 测试覆盖本次修改，缺少相应实现时无法通过
• 本地 lint、格式、类型检查及受影响测试均通过
• 存在行为变化，已在修改概述中说明

[Astro-Han]

Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.

Reviewed 147b82a452d4ff7a278b8353e627d5b293d4dd92; I found no P0–P2 issue. I read all nine changed files, including the dependency and notice changes. markdown-body.tsx retains the existing CodeBlock and trust boundaries; markdown-code-highlighting.ts normalizes aliases through the AST and prepares tokens after rendering, while markdown-code-tokenizer.ts maps the seven selected Refractor grammars to Astryx categories. The source, clipboard and rendering remain owned by the existing component.

All 819 UI tests and seven Chromium tests passed locally. These cover UTF-16/CRLF and multiline token ranges, punctuated aliases, literal HTML, existing TypeScript highlighting, streaming/source changes, unmounting, lazy-load failure, copying, collapsing and oversized plain-code fallback. I also repeated three browser checks while replacing only the Markdown component with main's actual module: the head passed all three and main failed all three. I inspected paired light/dark screenshots; the requested language colors are visible and the component layout is retained.

Fresh installation and dependency patches, the normal clean root build:test, UI and Desktop type checks, the production renderer build, 168 architecture tests and the strict architecture comparison with main 59f61cf7, surface inventory, generated third-party notices, ASF headers, Biome and whitespace checks passed. The optional production grammar chunk is 20.15 kB gzip. No protocol or storage change is present; the epoch guard passes at 227. The contribution declares OpenAI Codex and its commit includes the corresponding trailer.

The 20,000 UTF-16-unit limit, 120 ms streaming debounce and bounded cache constrain work, but grammar execution still runs synchronously on the renderer thread; they are not a hard latency guarantee. I did not qualify native OS applications, run the full monorepo suite or perform a production performance benchmark. Exact-head CI test, dependency audit, Linux/Windows packaging and Windows recovery have succeeded, and GitHub reports no merge conflict.