Skip to content

feat: online voice engines: provider registry, streaming, offline fallback modes - #167

Open
hongkongkiwi wants to merge 21 commits into
aicodeking:mainfrom
hongkongkiwi:voice-online
Open

feat: online voice engines: provider registry, streaming, offline fallback modes#167
hongkongkiwi wants to merge 21 commits into
aicodeking:mainfrom
hongkongkiwi:voice-online

Conversation

@hongkongkiwi

@hongkongkiwi hongkongkiwi commented Oct 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What and why

Stacked on #163 (offline voice dictation) — merge that first. This branch contains #163 plus the online layer; the new diff is the voice commits after 83eb400.

Closes #169.

What

Settings → Voice gains a transcription mode, exactly the requested set:

  • Offline (default) — Apple Speech or downloaded local models; nothing leaves the machine.
  • Online — a provider you configure below transcribes; audio goes only to that provider.
  • Online with offline fallback — online first; if it fails (connect error, timeout, mid-stream drop), the recording is kept and the local quality model settles the transcript, with the result marked fallback: true and a toast.

Language hint on every transport

The dictation language from Settings → Voice rides along when a specific language is picked (ISO-639-1, region stripped): the language multipart field for the OpenAI-compatible endpoints (verified against the official SDK docs: applies to the gpt-4o-transcribe models as well as whisper-1), Deepgram's language query parameter (omit → the provider defaults to en), and the realtime transcription session's language config. Detect automatically sends nothing and lets the provider detect. The dictation picker itself gains Русский and Polski, named natively like the rest.

Provider registry

Presets cover the APIs people actually use; Custom points at any OpenAI-compatible endpoint, self-hosted included (a local whisper.cpp server on localhost works):

Provider Transport Live partials Default model
OpenAI multipart POST /v1/audio/transcriptions — gpt-4o-mini-transcribe
Groq multipart — whisper-large-v3-turbo
OpenRouter multipart (/api/v1/audio/transcriptions) — openai/gpt-4o-mini-transcribe
Together multipart — openai/whisper-large-v3
Deepgram streaming WebSocket (Token auth) ✅ interim results while speaking nova-3
OpenAI Realtime streaming WebSocket (Together-compatible base URL) ✅ deltas while speaking gpt-4o-mini-transcribe
Custom user endpoint — user model

Speed was the design constraint: streaming providers send 16 kHz PCM (Deepgram) or 24 kHz PCM16 (OpenAI Realtime, upsampled from the capture) and partial text lands in the composer while you speak; multipart providers send one WAV on stop and usually return within a second on the turbo models. voice_test_provider exercises the whole configured path with half a second of silence so "Test" in Settings is a real connectivity check.

Keys and settings

  • Keys live in the secret store (voice_set_key/voice_keys — ids only ever reach the webview, never values).
  • Model and base-URL overrides are plain settings (voice.json); a provider switch adopts the preset's default model.
  • A provider-registry command (voice_providers) drives the Settings picker, marked live vs on-stop.
  • Mode normalization is deliberate: online mode stays selectable before a provider is picked (so the form is reachable), and voice_start refuses with a clear message until provider + key exist.

Fallback semantics

Recording always captures locally. Online failures mid-take emit a toast and keep buffering; on stop the buffer is decoded by the offline engine (sherpa::finalize_buffer — the quality pass without the streaming stage). Without fallback mode, the online error is the outcome. Multipart providers can't fail mid-take.

How it was tested

  • cargo clippy --all-targets -- -D warnings clean; cargo test --lib voice:: — 37 passed, 1 ignored (provider URL shapes per preset, WAV encode round-trip, PCM16/upsample math, settings mode normalization, language-hint normalization, Deepgram URL carrying the hint, and loopback WebSocket mocks covering both streaming clients end to end — session config, binary/base64 payloads, interim→final composition, error paths).
  • npx tsc --noEmit, npm run build (typecheck + bundle budget) clean.
  • npx playwright test tests/voice.spec.ts — 5/5 in Chrome.
  • WebKit: tests/voice.spec.ts was missing from the WebKit config's testMatch and grep (zero voice tests ran there); both are now registered. A WebKit run completed 3/5 with 2 timeouts while the machine was under heavy memory pressure (25 GB swap in use from a concurrent perf run); Chrome passes all 5. Treat the WebKit verdict as partial pending a calm-machine rerun.
  • New spec coverage: mode selector, provider switch adopting preset models, key save to the secure store, connectivity test (refuses without a key, succeeds with one).
  • npm test has one pre-existing failure (tool-cards, reproduces on the clean fork tip; verified this session); the three bundle-budget failures in a fresh worktree clear after npm run build.

Also includes a small DX fix: PW_PORT on the Playwright config lets a second worktree run the suite against its own dev server instead of reusing whatever serves 1420 (default behavior unchanged).

  • npm run build (typecheck + bundle)
  • npm test
  • cargo test --manifest-path src-tauri/Cargo.toml (if Rust changed)
  • Relevant Playwright tests in Chrome and WebKit (if UI changed)

Checklist

  • Commit messages follow <type>: <description>
  • Docs in docs/ updated if behavior changed
  • Preview mocks added in src/preview.ts for new IPC commands
  • No telemetry, secrets or personal paths added

@hongkongkiwi
hongkongkiwi force-pushed the voice-online branch 3 times, most recently from a9131f8 to 1e326f5 Compare October 26, 2026 11:09
@hongkongkiwi hongkongkiwi changed the title feat: online voice engines — provider registry, streaming, offline fallback modes feat: online voice engines: provider registry, streaming, offline fallback modes Oct 26, 2026
@hongkongkiwi
hongkongkiwi force-pushed the voice-online branch 6 times, most recently from 47fb4e3 to fd5da56 Compare October 31, 2026 12:17
@hongkongkiwi hongkongkiwi mentioned this pull request Oct 31, 2026
Open
8 tasks done
@hongkongkiwi
feat: voice dictation native core (engines, model downloads, commands)
c83611b 
Pluggable engine layer for speech-to-text: Apple Speech on macOS
(streaming punctuated partials, on-device preferred) and sherpa-onnx
everywhere (streaming zipformer live partials + offline quality pass).
Models are never bundled: Settings will offer a pinned catalog
(Moonshine, SenseVoice, Parakeet, Whisper, live streamers, punctuation)
with size caps, SHA-256 verification and progress events. Commands:
voice_state/save_settings/models/download_model/start/stop/cancel.
@hongkongkiwi
feat: microphone usage strings for the macOS bundle
92b7252 
src-tauri/Info.plist is merged by Tauri into the generated bundle, so
voice dictation prompts for microphone and speech recognition with the
right copy. The engine refuses (instead of crashing) when the running
binary has no bundle — the tauri dev case.
@hongkongkiwi
feat: voice dictation composer UI and Settings section
a692723 
The mic sits beside the Saver leaf once Voice is on: tap or hold (per
setting) records natively, live partial text streams into the prompt
replacing the previous partial, and Done settles the final transcript
for review — never auto-send, Esc cancels, and the recording bar (level
meter, timer) survives composer re-renders from body. Settings → Voice
lists the pinned model catalog with sizes, licenses, download progress
and a quality-model picker; preview mocks cover the whole flow.
@hongkongkiwi
feat: voice dictation in the mobile composer
cfe7590 
expo-speech-recognition drives the phone's own recognizer: tap the mic
to dictate, live partial text streams into the composer, and only the
final transcript is sent — as ordinary text over Connect. On-device
recognition is preferred (iOS requiresOnDeviceRecognition, Android's
on-device service); Settings gains an On-device only switch that
forbids the OS's network service. The mic hides on devices without a
speech service. Nothing is uploaded to PandaCode.
@hongkongkiwi
docs: voice dictation (feature doc, IPC contract, privacy rows)
69bde31 
docs/voice.md covers engines, the pinned model catalog with licenses,
commands/events, interaction and mobile behavior, with a Verification
section. docs/ipc.md gains the voice_* contract; the README privacy
table lists the model download and the mobile OS speech service.
Adds the PANDACODE_MIC live capture test the doc references.
@hongkongkiwi
feat: online voice engines with modes and offline fallback
d1d531f 
Settings → Voice gains a mode: offline (default), online, or
online-with-fallback. A provider registry presets OpenAI, Groq,
OpenRouter, Together, Deepgram, OpenAI Realtime (Together-compatible)
and any custom OpenAI-compatible endpoint; keys live in the secret
store (voice_set_key/voice_keys, never the webview), model and base
URL are settings, voice_test_provider exercises the whole path with a
half second of silence. Streaming providers (Deepgram's WebSocket,
OpenAI Realtime at 24 kHz) give live partials while speaking; multipart
providers send one WAV on stop. A failed online attempt settles on the
local quality model (finalize_buffer) when fallback is on, with the
result marked fallback. Loopback WebSocket mocks cover both streaming
clients end to end (session config, PCM/base64 payloads, interim and
final composition, error paths).
@hongkongkiwi
feat: online modes in Settings → Voice, provider forms, PW_PORT override
015f2b3 
The mode selector (offline / online / online with fallback) sits above
the engine; online shows the provider picker, API-key field (secure
store, write-only from the webview), model list and endpoint override
with a Test button that probes the real service. Preview mocks cover
the whole flow (providers, keys, test, mode normalization keeps online
reachable until a provider is picked); the voice spec asserts key
saving and the connectivity test. PW_PORT lets a second worktree run
the suite against its own dev server instead of port 1420's.
@hongkongkiwi
fix: restore the composer mic button and voice spec after the 0.3.49 …
ae7cb8b 
…rebase
@hongkongkiwi
fix: style the voice API key and base URL inputs
95e491c 
The online provider forms render .voice-key and .voice-base-url inputs that
had no stylesheet, so they showed as bare browser fields inside the styled
setting rows.
@hongkongkiwi
fix: settle a mid-stream voice feed failure on stop instead of orphan…
d93f598 
…ing the recording
@hongkongkiwi
fix: free the voice session slot before a recording settles
52a4af2
@hongkongkiwi
fix: let saving an empty field clear a saved voice API key
63a8048
@hongkongkiwi
chore: regenerate Cargo.lock after the 0.3.79 rebase
f96f3cb
@hongkongkiwi
ci: install libasound2-dev so cpal's alsa-sys build finds ALSA
25ff4e7 
The Linux runner has pkg-config but no ALSA headers, so alsa-sys's build
script panics before the tests compile.
@hongkongkiwi
fix: satisfy clippy on the voice core
086d64f 
Drop a Copy clone, replace a dropped spawn handle with drop(), take &Path
in setup, and build test Settings with struct literals.
@hongkongkiwi
fix: drop the unused SinkExt import in the provider-error test
19a82de
@hongkongkiwi
fix: build the Voice dir from the &Path setup argument
de5994c
@hongkongkiwi
fix: compile and clippy cleanups after the 0.3.79 rebase
f50a86d
@hongkongkiwi
feat: pass the dictation language to online providers
57450f1 
The picked dictation language rides along as an ISO-639-1 hint on every
transport: the multipart language field, Deepgram's language query
parameter, and the realtime session's transcription config. Detect
automatically sends nothing and leaves detection to the provider.
@hongkongkiwi
feat: add Russian and Polish to the dictation language picker
10a1c6b 
Named in their own language like the rest, and the voice spec is now
registered in the WebKit config (it silently ran zero voice tests there).
@hongkongkiwi
feat: add Mistral, Alibaba Qwen and MiniMax online providers
5e29cef
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Voice dictation online modes: OpenAI, Groq, OpenRouter, Together, Deepgram and Realtime providers, with offline fallback

1 participant

@hongkongkiwi