feat(tts): Paradee-8M CoreML backend [beta] by Alex-Wengg · Pull Request #998 · FluidInference/FluidAudio

Alex-Wengg · 2026-10-07T23:37:35Z

Note

Beta — API and hosted model assets may change.

Adds Paradee-8M v1.0 (Kokoro-82M distilled to 8.07M params, af_heart, 24 kHz English) as a TTS backend. Models: FluidInference/paradee-8m-coreml (int8/ 12 MB default, fp32/ 34 MB). Conversion code lives in our private model-lab repo.

ParadeeManager: ParadeeText → host (round-half-even durations, column expansion, seeded source noise) → ParadeeAcoustic
Reuses the KokoroAne English frontend and KokoroAneVocab (Paradee's vocab is identical); applies misaki's ɾ→T, ʔ→t output step, which Paradee was trained on (MiniMax WER 1.76% → 1.20%)
Rejects .all / .cpuAndGPU: the LSTMs abort in MPSGraph (GPURNNOps JIT not supported)
CLI: tts --backend paradee [--variant int8|fp32] [--speed] [--seed] [--phonemes], tts-benchmark --backend paradee

Results (tts-benchmark --corpus minimax-english, M5 Pro, macOS 27): int8 and fp32 both WER 1.20% / CER 0.14%, RTFx ~100 including G2P, p50 77 ms per phrase. Model-only, Core ML runs ~152× real time vs upstream ONNX 20× (1 thread) / 47× (15 threads).

Testing: release build + swift-format clean; unit tests added (ParadeeTests) but not run locally (no XCTest here), so CI is their first run.

🤖 Generated with Claude Code

github-actions · 2026-10-07T23:53:21Z

Parakeet EOU Benchmark Results ✅

Status: Benchmark passed
Chunk Size: 320ms
Files Tested: 100/100

Performance Metrics

Metric	Value	Description
WER (Avg)	7.03%	Average Word Error Rate
WER (Med)	4.17%	Median Word Error Rate
RTFx	11.22x	Real-time factor (higher = faster)
Total Audio	470.6s	Total audio duration processed
Total Time	42.5s	Total processing time

Streaming Metrics

Metric	Value	Description
Avg Chunk Time	0.042s	Average chunk processing time
Max Chunk Time	0.085s	Maximum chunk processing time
EOU Detections	0	Total End-of-Utterance detections

_{Test runtime: 1m9s • 10/07/2026, 10:01 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O}

github-actions · 2026-10-07T23:59:39Z

ASR Benchmark Results ✅

Status: All benchmarks passed

Parakeet v3 (multilingual)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	0.57%	0.00%	4.58x	✅
test-other	1.19%	0.00%	3.01x	✅

Parakeet v2 (English-optimized)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	0.80%	0.00%	4.71x	✅
test-other	1.00%	0.00%	2.77x	✅

Streaming (v3)

Metric	Value	Description
WER	0.00%	Word Error Rate in streaming mode
RTFx	0.53x	Streaming real-time factor
Avg Chunk Time	1.680s	Average time to process each chunk
Max Chunk Time	1.975s	Maximum chunk processing time
First Token	2.074s	Latency to first transcription token
Total Chunks	31	Number of chunks processed

Streaming (v2)

Metric	Value	Description
WER	0.00%	Word Error Rate in streaming mode
RTFx	0.44x	Streaming real-time factor
Avg Chunk Time	1.994s	Average time to process each chunk
Max Chunk Time	2.340s	Maximum chunk processing time
First Token	2.096s	Latency to first transcription token
Total Chunks	31	Number of chunks processed

_{Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming}

_{25 files per dataset • Test runtime: 9m2s • 10/07/2026, 09:59 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)}

Expected RTFx Performance on Physical M1 Hardware:

• M1 Mac: ~28x (clean), ~25x (other)
• CI shows ~0.5-3x due to virtualization limitations

_{Testing methodology follows HuggingFace Open ASR Leaderboard}

github-actions · 2026-10-08T00:06:14Z

Supertonic3 Smoke Test ✅

Check	Result
Build	✅
Model download (incl. `VectorEstimatorVariants/` int4 buckets)	✅
Model load	✅
Synthesis pipeline (`--ve-variant int4`)	✅
Output WAV	✅ (364.7 KB)

_{Runtime: 0m29s}

_{Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf.}

github-actions · 2026-10-08T00:09:16Z

VAD Benchmark Results

Performance Comparison

Dataset	Accuracy	Precision	Recall	F1-Score	RTFx	Files
MUSAN	94.0%	89.3%	100.0%	94.3%	555.7x faster	50
VOiCES	94.0%	89.3%	100.0%	94.3%	411.9x faster	50

Dataset Details

MUSAN: Music, Speech, and Noise dataset - standard VAD evaluation
VOiCES: Voices Obscured in Complex Environmental Settings - tests robustness in real-world conditions

✅: Average F1-Score above 70%

github-actions · 2026-10-08T00:18:43Z

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Optimal clustering with Hungarian algorithm for maximum accuracy

Metric	Value	Target	Status	Description
DER	10.4%	<20%	✅	Diarization Error Rate (lower is better)
RTFx	8.35x	>1.0x	✅	Real-Time Factor (higher is faster)

Offline VBx Pipeline Timing Breakdown

Time spent in each stage of batch diarization

Stage	Time (s)	%	Description
Model Download	21.255	16.9	Fetching diarization models
Model Compile	9.109	7.2	CoreML compilation
Audio Load	0.121	0.1	Loading audio file
Segmentation	32.805	26.1	VAD + speech detection
Embedding	125.328	99.7	Speaker embedding extraction
Clustering (VBx)	0.135	0.1	Hungarian algorithm + VBx clustering
Total	125.661	100	Full VBx pipeline

Speaker Diarization Research Comparison

Offline VBx achieves competitive accuracy with batch processing

Method	DER	Mode	Description
FluidAudio (Offline)	10.4%	VBx Batch	On-device CoreML with optimal clustering
FluidAudio (Streaming)	17.7%	Chunk-based	First-occurrence speaker mapping
Research baseline	18-30%	Various	Standard dataset performance

Pipeline Details:

Mode: Offline VBx with Hungarian algorithm for optimal speaker-to-cluster assignment
Segmentation: VAD-based voice activity detection
Embeddings: WeSpeaker-compatible speaker embeddings
Clustering: PowerSet with VBx refinement
Accuracy: Higher than streaming due to optimal post-hoc mapping

_{🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 158.3s processing • Test runtime: 2m 42s • 10/07/2026, 09:52 PM EST}

github-actions · 2026-10-08T00:33:35Z

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Evaluating "who spoke when" detection accuracy

Metric	Value	Target	Status	Description
DER	15.1%	<30%	✅	Diarization Error Rate (lower is better)
JER	24.9%	<25%	✅	Jaccard Error Rate
RTFx	28.67x	>1.0x	✅	Real-Time Factor (higher is faster)

Diarization Pipeline Timing Breakdown

Time spent in each stage of speaker diarization

Stage	Time (s)	%	Description
Model Download	10.412	28.4	Fetching diarization models
Model Compile	4.462	12.2	CoreML compilation
Audio Load	0.053	0.1	Loading audio file
Segmentation	10.979	30.0	Detecting speech regions
Embedding	18.299	50.0	Extracting speaker voices
Clustering	7.320	20.0	Grouping same speakers
Total	36.604	100	Full pipeline

Speaker Diarization Research Comparison

Research baselines typically achieve 18-30% DER on standard datasets

Method	DER	Notes
FluidAudio	15.1%	On-device CoreML
Research baseline	18-30%	Standard dataset performance

Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:

M2 MacBook Air (2022): Runs at 150 RTFx real-time
Performance scales with Apple Neural Engine capabilities

_{🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 36.6s diarization time • Test runtime: 2m 5s • 10/07/2026, 10:09 PM EST}

github-actions · 2026-10-08T00:36:51Z

Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

Metric	Value	Target	Status
DER	30.3%	<35%	✅
Miss Rate	28.2%	-	-
False Alarm	0.9%	-	-
Speaker Error	1.2%	-	-
RTFx	13.9x	>1.0x	✅
Speakers	4/4	-	-

_{Sortformer High-Latency • ES2004a • Runtime: 3m 30s • 2026-10-08T02:00:17.178Z}

github-actions · 2026-10-08T00:39:00Z

PocketTTS Smoke Test ✅

Check	Result
Build	✅
Model download	✅
Model load	✅
Synthesis pipeline	✅
Output WAV	✅ (150.0 KB)

_{Runtime: 0m9s}

_{Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon.}

Paradee-8M v1.0 (Kokoro-82M distilled to 8.07M params, af_heart, 24 kHz English) via FluidInference/paradee-8m-coreml (int8 default 12 MB, fp32 34 MB). Two CoreML graphs with host-side steps in Swift: - ParadeeText: ids -> durations, prosody features d, text features asr_tok - host: max(1, round(dur / speed)) with half-to-even rounding (torch.round), column expansion, seeded N(0,1) harmonic-source noise (InflectNoise) - ParadeeAcoustic: F0/N predictor + iSTFTNet decoder + phase-lock filter Text path reuses the KokoroAne English frontend (NeMo TN, Misaki lexicon, BART G2P) and KokoroAneVocab (Paradee's vocab is byte-identical to Kokoro's). Sentences are synthesized separately, as upstream does. The frontend's lexicon keeps misaki's raw flap/glottal stop (ɾ, ʔ); misaki rewrites them to T/t for Kokoro v1.0 and Paradee was trained only on that output, so the Paradee text path applies the same rewrite. MiniMax English WER 1.76% -> 1.20%, CER 0.33% -> 0.14% ("kittens", "satellite", "patterns" were garbled). .all/.cpuAndGPU are rejected: the LSTMs abort in MPSGraph (GPURNNOps JIT not supported). cpuOnly default; cpuAndNeuralEngine also supported. CLI: tts --backend paradee [--variant int8|fp32] [--speed] [--seed] [--phonemes]; tts-benchmark --backend paradee. tts-benchmark minimax-english (100 phrases, Parakeet round trip, M5 Pro, macOS 27): int8 and fp32 both WER 1.20% / CER 0.14%, RTFx ~100 (text frontend + both models), synth p50 77 ms; all-ane 95.6x, same WER.

…es (#999) The English frontend reads the misaki lexicon raw, so it keeps the flap `ɾ`. misaki's own `G2P.__call__` rewrites it to `T` before Kokoro v1.0 sees it. Baseline Kokoro read "metal" as "mattle". - Apply `ɾ → T` in the English phonemizer path (also used by `englishPhonemes(for:)`) - Keep the glottal stop `ʔ`: misaki also rewrites it to `t`, but that made Kokoro read "button" as "butt" and "mitten" as "mit" **Results** (`tts-benchmark --backend kokoro-ane`, M5 Pro, macOS 27): | | 12 flap/glottal-dense sentences | minimax-english (100) | |---|---:|---:| | main | WER 3.67% / CER 0.56% | 0.68% / 0.17% | | `ɾ→T` + `ʔ→t` | 1.67% / 1.04% | 0.68% / 0.17% | | **this PR (`ɾ→T`)** | **0.00% / 0.00%** | 0.68% / 0.17% | **Reviewer notes:** the gain is small (one word in a targeted set; MiniMax unchanged). The Mandarin variant's English runs (`englishPhonemes(for:)`) also get the rewrite; that path is untested. Unit test added, not run locally (no XCTest here). Found while adding Paradee (#998), which needs both rewrites. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Alex-Wengg mentioned this pull request Oct 7, 2026

fix(tts/kokoro-ane): apply misaki's ɾ → T flap step to English phonemes #999

Merged

Alex-Wengg force-pushed the feat/paradee-tts branch from 335dddb to 42be1df Compare October 8, 2026 01:41

Alex-Wengg merged commit 4ba3e01 into main Oct 8, 2026
19 checks passed

Alex-Wengg deleted the feat/paradee-tts branch October 8, 2026 02:45

Navigation Menu

Repository navigation

feat(tts): Paradee-8M CoreML backend [beta] - #998

feat(tts): Paradee-8M CoreML backend [beta]#998
Alex-Wengg merged 1 commit into
mainfrom
feat/paradee-tts

Alex-Wengg commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Alex-Wengg commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Parakeet EOU Benchmark Results ✅

Performance Metrics

Streaming Metrics

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ASR Benchmark Results ✅

Parakeet v3 (multilingual)

Parakeet v2 (English-optimized)

Streaming (v3)

Streaming (v2)

Expected RTFx Performance on Physical M1 Hardware:

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Supertonic3 Smoke Test ✅

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VAD Benchmark Results

Performance Comparison

Dataset Details

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Offline VBx Pipeline Timing Breakdown

Speaker Diarization Research Comparison

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Diarization Pipeline Timing Breakdown

Speaker Diarization Research Comparison

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PocketTTS Smoke Test ✅

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading