Skip to content

Avoid unnecessary CTC vocabulary inference - #993

Merged
Alex-Wengg merged 5 commits into
FluidInference:mainfrom
achembarpu:perf/ctc-candidate-preflight
Oct 9, 2026
Merged

Alex-Wengg merged 5 commits into
FluidInference:mainfrom
achembarpu:perf/ctc-candidate-preflight

Conversation

@achembarpu

@achembarpu achembarpu commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Why is this change needed?

Vocabulary rescoring currently computes CTC evidence even when no transcript
word is eligible for correction. This adds hasCTCRescoringCandidates to the
rescorer and prepared session so callers can skip that work. It uses the existing
matching rules, including aliases, compounds, multiword terms and thresholds.

Empty vocabulary or timings return false. Acoustic rescue conservatively returns
true for nonempty inputs. Callers that need standalone keyword detections must
still run rescoring. Term forms are also prepared once, and transcript words are
normalized once per request.

Initial measurements on one Apple Silicon Mac reduced unnecessary separate CTC
work from about 140–153 ms to 0.3 ms, and shared-head work from about 12 ms to
0.3 ms. These are component timings, not whole-dictation speedups.

Validation

  • 144 selected checks pass, including parity against real CTC evidence across
    12 candidate cases, threshold handling, empty inputs and acoustic rescue.
  • Contribution lint, strict lint for changed Swift files and formatting checks
    pass.
  • The default Intel build passes, as does Intel Release compilation for macOS 14
    with the optional text-processing trait disabled.

The streaming vocabulary test class is excluded: its first-word replacement
failure also reproduces on unchanged main. The full suite and hosted CI have not
been verified green.

@Alex-Wengg

Copy link
Copy Markdown
Member

Looks correct — early-stop placement matches all three candidate loops and the cached forms/set are equivalent to the old builders. One nit: hasCTCRescoringCandidates returns true whenever spotterRescueEnabled (default on), but rescue only runs on the term-centric path under largeVocabThreshold; gating it with !useBKTree && terms.count <= largeVocabThreshold would let large vocabularies skip CTC too.

@achembarpu

Copy link
Copy Markdown
Contributor Author

Thanks for catching that. Fixed in 6c57258: the rescue shortcut now requires !useBKTree and terms.count <= largeVocabThreshold. Larger vocabularies can skip CTC when no text candidates match, even with rescue enabled. Added regression coverage at and above the threshold, including positive candidates and prepared-session parity, and updated the docs. All 103 selected vocabulary checks and formatting checks pass locally.

@Alex-Wengg
Alex-Wengg merged commit 0b0fa2a into FluidInference:main Oct 9, 2026
@achembarpu
achembarpu deleted the perf/ctc-candidate-preflight branch October 9, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants