diff --git a/Documentation/ANE_Profiler.md b/Documentation/ANE_Profiler.md index 65c752919..4ea4b64b2 100644 --- a/Documentation/ANE_Profiler.md +++ b/Documentation/ANE_Profiler.md @@ -1,5 +1,9 @@ # ANE Profiler +The header below describes the original June audit. Later model sections state +their own dates, configurations and metrics; see [Kokoro ANE v3](#kokoro-ane-v3) +for the October M5 Pro routing, latency and telemetry checks. + | | | |---|---| | **Measured** | 2026-06-05 | @@ -237,14 +241,64 @@ Latency **measured on real synthesis**, warm (one short sentence; `tts --backend | Model | Type | ANE | GPU | CPU | ops | Size | Heavy graph → device | |-------|------|----:|----:|----:|----:|-----:|----------------------| -| Kokoro ANE (7-stage) | batch (per utterance) | 75% | 0% | 25% | 1472 | 83 MB | Vocoder → ANE | +| Kokoro ANE (legacy capability profile) | batch (per utterance) | 75% | 0% | 25% | 1472 | 83 MB | Vocoder → ANE | | Supertonic (`--ve-variant fp16`, legacy) | batch (8-step diffusion) | 30% | 0% | 70% | 1365 | 192 MB | VectorEstimator → **CPU** (dynamic shapes can't use ANE) | | Supertonic (default, int4 L-bucketed) | batch (8-step diffusion) | ~90% | 0% | ~10% | 1289 | 102 MB | VectorEstimator → **ANE** (fixed L-buckets) | | PocketTTS (v2.1) | streaming (autoregressive) | ~9% | ~31% | ~60% | 2629 | ~330 MB | flow_decoder_fused → **ANE**; flowlm/cond → GPU; mimi → CPU | **Component detail** -### Kokoro ANE (7-stage) +### Kokoro ANE v3 + +**Separate check: M5 Pro, 24 GiB, macOS 27.0 (26A428), October 7, 2026.** +The historical seven-stage op-count percentages below do not describe v3. + +| Stage | Requested compute policy in v0.17.6 | +| --- | --- | +| Static/dynamic Albert, PostAlbert, Alignment, Prosody, masked decoder | `.cpuAndNeuralEngine` | +| Fast waveform generator | `.cpuAndGPU` | +| Native source/STFT | CPU / Accelerate (outside Core ML) | +| Long-input fallback Vocoder | `.cpuAndNeuralEngine` | +| Long-input fallback Noise / Tail | `.cpuAndGPU` | + +This is hybrid routing, not `.all`. Allowed devices are not a guarantee that +all operations execute on one device; CPU fallback is permitted. There is no +strict ANE-only Core ML compute policy. The public v3 API retains this fixed +routing; the policy overrides below were local diagnostic changes. + +Across the three language variants, `MLComputePlan` preferred ANE for about +**98.8% of estimated operation cost in static Albert-64** and approximately +**100% in DecoderPre-200**; the source/generator plan preferred GPU for about +**100%**. PostAlbert, Alignment and fp32 Prosody preferred CPU in this audit. +These percentages are **estimated cost weights within individual graphs**, not +op counts, measured utilization, wall-time shares or a runtime execution trace. +The 32-token and 120-frame buckets are not covered by those percentages. + +A separate real-input check measured hybrid medians of **49.45 / 35.85 / 53.97 ms** +for English / Japanese / Mandarin. Forcing `.cpuAndNeuralEngine` throughout +measured **511.40 / 474.45 / 584.19 ms**. Both `.all` and `.cpuAndGPU` hit +`GPURNNOps.mm: JIT not supported` during Prosody on the second warmup in all +three cases. See the [timing protocol and raw records](TTS/Benchmarks.md#compute-policy-comparison). + +**Interpreting asitop:** the live demo performs short bursts, while the observed +powermetrics collector sampled roughly once per second. Samples near three demo +clicks reported about **30.0 / 9.7 / 11.7 mW** of system-wide ANE power for +English / Japanese / Mandarin. Those observations are not synchronized per-process +energy measurements and are separate from the compute-policy timing run. +The installed asitop display rounded low power to `0.0 W` / `0%`; a local viewer +showed mW and the maximum observed sample over 60 seconds. That maximum is still +a sampled average, not an instantaneous utilization peak. Neither those readings +nor a compute plan establish which Kokoro operations actually ran on ANE; +process-attributed runtime tracing remains outstanding. +[Compute-plan records](TTS/Measurements/KokoroV3/compute-placement.json) and +[system-wide power samples](TTS/Measurements/KokoroV3/ane-latest-clicks.json) +preserve the separate sources for those observations. + +### Kokoro ANE (legacy seven-stage capability profile) + +Historical June profile under `.cpuAndNeuralEngine` throughout. Percentages are +preferred-device operation counts, not production utilization or v3 routing. + | Component | ANE | GPU | CPU | ops | Size | Lat ms | |-----------|----:|----:|----:|----:|-----:|-------:| | Albert | 94% | 0% | 6% | 310 | 6 MB | 6.5 | diff --git a/Documentation/Benchmarks.md b/Documentation/Benchmarks.md index 8cae0852d..ae8ac7e3b 100644 --- a/Documentation/Benchmarks.md +++ b/Documentation/Benchmarks.md @@ -248,6 +248,21 @@ Derived metrics: ## Text-to-Speech +### Kokoro ANE v3 (FluidAudio v0.17.6) + +Selected M5 Pro / macOS 27.0 checks measured **49.45 / 35.85 / 53.97 ms** +median inference for English / Japanese / Mandarin, producing 3.350 / 3.050 / +3.675 seconds of audio. Each median uses five calls after two warmups; input +phonemes are prepared in advance. Current v3 uses per-stage CPU/ANE and CPU/GPU +policies plus native CPU DSP, rather than `.all` or an ANE-only policy. + +Applying CPU+ANE to every graph measured 511.40 / 474.45 / 584.19 ms; `.all` +and CPU+GPU aborted in Apple's GPU RNN path before completing warmups. +These are bounded local checks, not full-corpus results. See the +[complete protocol, inputs, ranges, raw records and separate live-demo timings](TTS/Benchmarks.md#kokoro-ane-v3-selected-m5-pro-measurements). + +### Historical M4 Pro comparison + We generated the same strings with to generate audio between 1s to ~300s in order to test the speed across a range of varying inputs on Pytorch CPU, MPS, and MLX pipeline, and compared it against the native Swift version with Core ML models. Each pipeline warmed up the models by running through it once with pesudo inputs, and then comparing the raw inference time with the model already loaded. You can see that for the Core ML model, we traded lower memory and very slightly faster inference for longer initial warm-up. diff --git a/Documentation/CLI.md b/Documentation/CLI.md index c6bc46cec..683995363 100644 --- a/Documentation/CLI.md +++ b/Documentation/CLI.md @@ -7,14 +7,22 @@ This guide collects commonly used `fluidaudio` CLI commands for ASR, diarization TTS is built into the CLI. Run it directly: ```bash -# Default Kokoro (CPU+GPU, multi-voice, chunker, custom lexicon) +# Default TTS backend (use --backend to select explicitly) swift run fluidaudiocli tts "Hello from FluidAudio" --output out.wav -# Kokoro ANE (7-stage, ANE-resident, 3-11× RTFx, single voice af_heart) +# Kokoro legacy seven-stage runtime (variant voices; OS-dependent routing) swift run fluidaudiocli tts "Hello from FluidAudio" \ --backend kokoro-ane \ --output out-ane.wav +# Kokoro ANE v3: fixed hybrid CPU/ANE/GPU routing (v0.17.6+) +swift run -c release fluidaudiocli tts "Hello from FluidAudio" \ + --backend kokoro-ane --kokoro-version v3 --voice af_heart --output en.wav +swift run -c release fluidaudiocli tts "こんにちは、世界。" \ + --backend kokoro-ane --kokoro-version v3 --variant ja --voice jf_alpha --output ja.wav +swift run -c release fluidaudiocli tts "你好世界。" \ + --backend kokoro-ane --kokoro-version v3 --variant zh --voice zf_001 --output zh.wav + # PocketTTS (streaming, voice cloning) swift run fluidaudiocli tts "Hello from FluidAudio" \ --backend pocket \ @@ -24,6 +32,10 @@ swift run fluidaudiocli tts "Hello from FluidAudio" \ swift run fluidaudiocli g2p-benchmark ``` +V3 requires macOS 15 / iOS 18. It supports the fixed hybrid route, not arbitrary +global compute policies. See [Kokoro usage and limits](TTS/KokoroAne.md) and +[latency measurement boundaries](TTS/Benchmarks.md#kokoro-ane-v3-selected-m5-pro-measurements). + ## ASR ```bash diff --git a/Documentation/Models.md b/Documentation/Models.md index 5f8c6ee1d..37c5c6536 100644 --- a/Documentation/Models.md +++ b/Documentation/Models.md @@ -68,7 +68,7 @@ maps the supported artifacts to their upstream sources and documents the limits | Model | Description | Context | |-------|-------------|---------| -| **Kokoro ANE (7-stage)** | Kokoro 82M weights split into 7 CoreML stages so the ANE-friendly layers (Albert / Prosody / Vocoder) stay resident on the Neural Engine while PostAlbert / Alignment / Noise / Tail run on CPU. 3-11× RTFx. English (`af_heart`) and Mandarin (`ANE-zh`) variants. ≤510 IPA phonemes per call, no chunker / SSML / custom lexicon. Managed by `KokoroAneManager`. | ANE-optimized variant derived (with permission) from [laishere/kokoro-coreml](https://github.com/laishere/kokoro-coreml). The original single-graph (mono) Kokoro backend was removed in favor of this ANE pipeline; the kokoro repo root is now retained only for shared G2P assets. | +| **Kokoro ANE (v3 / legacy)** | `KokoroAneManager` provides native English, Japanese and Mandarin text synthesis with opt-in `version: .v3` in v0.17.6. V3 uses static CPU/ANE stages, a CPU/GPU generator and native CPU DSP, with legacy fallback beyond the fast buckets. Existing callers keep the legacy runtime; Spanish/French remain legacy-only. See [KokoroAne.md](TTS/KokoroAne.md) for limits and [measured latency](TTS/Benchmarks.md#kokoro-ane-v3-selected-m5-pro-measurements). | Base weights: [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (en/ja), [hexgrad/Kokoro-82M-v1.1-zh](https://huggingface.co/hexgrad/Kokoro-82M-v1.1-zh) (zh). Legacy conversion derives from laishere; v3 adapts Matt Mireles’s optimized GPU graph. | | **PocketTTS** | TTS backend (~155M params). Autoregressive frame-by-frame generation with dynamic audio chunking. No phoneme stage, works directly on text tokens. Managed by `PocketTtsManager`. | Supports streaming, minimal RAM usage, excellent quality | | **Supertonic-3** | Multilingual TTS, 31 languages (`en`, `ko`, `ja`, `ar`, `bg`, `cs`, `da`, `de`, `el`, `es`, `et`, `fi`, `fr`, `hi`, `hr`, `hu`, `id`, `it`, `lt`, `lv`, `nl`, `pl`, `pt`, `ro`, `ru`, `sk`, `sl`, `sv`, `tr`, `uk`, `vi`, plus `na` for numeric/language-agnostic input). 4-stage CoreML pipeline (text_encoder → duration_predictor → vector_estimator → vocoder, ~398 MB). Caller-supplied voice styles loaded from Supertonic preset JSON. 44.1 kHz mono fp32 output. Managed by `Supertonic3Manager`. | CoreML conversion of upstream `Supertone/supertonic-3`; see `Scripts/convert_supertonic3_to_coreml.py`. | | **StyleTTS2 (LibriTTS, iteration_3)** | Reference-audio–driven zero-shot English TTS. 8-stage CoreML pipeline (`text_encoder → bert → ref_encoder → fused_diffusion_sampler → duration_predictor → fused_f0n_har_source → decoder_pre → decoder_upsample`) with 3 lazily-loaded T = 64 / 128 / 256 bucket variants of `bert` / `fused_diffusion_sampler`. 5-step ADPM2 Karras-σ diffusion sampler with α/β style blending against a speaker reference clip. 24 kHz mono fp32 output. Phonemizer reuses Kokoro's Misaki lexicon cache + BART G2P CoreML model with Misaki uppercase diphthong shorthand (`A O I Y W` → `eɪ oʊ aɪ ɔɪ aʊ`) expanded before encoding so the output matches the espeak IPA the model was trained on. Callers with a higher-quality phonemizer can bypass the stack via `StyleTTS2Manager.synthesize(ipa:...)`. See [StyleTTS2.md](TTS/StyleTTS2.md). | Zero-shot voice cloning from a single reference WAV; English only | @@ -106,7 +106,7 @@ Models we converted and tested but are not supported: too large for on-device de | Diarization (Pyannote) | [FluidInference/speaker-diarization-coreml](https://huggingface.co/FluidInference/speaker-diarization-coreml) | | LS-EEND | [FluidInference/ls-eend-coreml](https://huggingface.co/FluidInference/ls-eend-coreml) (per-dataset optimized variants: `/optimized/ami`, `/optimized/ch`, `/optimized/dih2`, `/optimized/dih3`) | | Sortformer | [FluidInference/diar-streaming-sortformer-coreml](https://huggingface.co/FluidInference/diar-streaming-sortformer-coreml) | -| Kokoro ANE (7-stage) | [FluidInference/kokoro-82m-coreml/tree/main/ANE](https://huggingface.co/FluidInference/kokoro-82m-coreml/tree/main/ANE) (English: `/ANE`; Mandarin: `/ANE-zh`; shared G2P assets `G2PEncoder.mlmodelc`, `G2PDecoder.mlmodelc`, `g2p_vocab.json` at the repo root) | +| Kokoro ANE (v3 / legacy) | [FluidInference/kokoro-82m-coreml](https://huggingface.co/FluidInference/kokoro-82m-coreml): v3 English `ANE-v3`, Japanese `ANE-v3/ja`, Mandarin `ANE-v3/zh`; legacy `ANE`, `ANE-ja`, `ANE-zh`. Shared G2P assets remain at the repo root / language asset directories; SDK pins and cache paths are documented in [KokoroAne.md](TTS/KokoroAne.md). | | PocketTTS | [FluidInference/pocket-tts-coreml](https://huggingface.co/FluidInference/pocket-tts-coreml) | | StyleTTS2 (LibriTTS, iteration_3) | [FluidInference/StyleTTS-2-coreml/iteration_3/compiled](https://huggingface.co/FluidInference/StyleTTS-2-coreml/tree/main/iteration_3/compiled) (shared phonemizer assets pulled from [`FluidInference/kokoro-82m-coreml`](https://huggingface.co/FluidInference/kokoro-82m-coreml): `G2PEncoder.mlmodelc`, `G2PDecoder.mlmodelc`, `g2p_vocab.json`, `us_lexicon_cache.json`) | | Supertonic-3 | [FluidInference/supertonic-3-coreml](https://huggingface.co/FluidInference/supertonic-3-coreml) | diff --git a/Documentation/README.md b/Documentation/README.md index 510858832..d6722324d 100644 --- a/Documentation/README.md +++ b/Documentation/README.md @@ -37,7 +37,9 @@ ## Text-to-Speech (TTS) -- [Kokoro ANE (7-stage)](TTS/KokoroAne.md) +- [Kokoro ANE v3 and legacy runtime](TTS/KokoroAne.md) +- [Kokoro v3 latency and compute-policy measurements](TTS/Benchmarks.md#kokoro-ane-v3-selected-m5-pro-measurements) +- [Kokoro ANE placement and telemetry](ANE_Profiler.md#kokoro-ane-v3) - [PocketTTS](TTS/PocketTTS.md) - [StyleTTS2](TTS/StyleTTS2.md) - [SSML](TTS/SSML.md) diff --git a/Documentation/TTS/Benchmarks.md b/Documentation/TTS/Benchmarks.md index 9bb72186e..438a5e79d 100644 --- a/Documentation/TTS/Benchmarks.md +++ b/Documentation/TTS/Benchmarks.md @@ -1,10 +1,103 @@ # TTS Benchmarks +## Kokoro ANE v3: selected M5 Pro measurements + +FluidAudio **v0.17.6** adds the opt-in `KokoroAneManager(version: .v3)` +runtime for English, Japanese and Mandarin. These are small local development +checks on **Apple M5 Pro, 24 GiB, macOS 27.0 (26A428)**, recorded October 7, +2026 (Toronto). They do not replace the historical 100-phrase results below or +establish cross-device performance or audio quality equivalence. + +### Compute-policy comparison + +The released runtime uses **per-stage hybrid routing**, not `.all`: +Albert / PostAlbert / Alignment / Prosody / masked decoder request +`.cpuAndNeuralEngine`; the generator requests `.cpuAndGPU`; native source/STFT +runs on CPU through Accelerate. The long-input fallback requests CPU+ANE for +Vocoder and CPU+GPU for Noise/Tail. Core ML chooses placement within each policy. + +| Policy applied to every Core ML model | English | Japanese | Mandarin | +| --- | ---: | ---: | ---: | +| Released hybrid routing | **49.45 ms** (40.52–66.98) | **35.85 ms** (35.41–36.36) | **53.97 ms** (45.46–58.52) | +| `.cpuAndNeuralEngine` | 511.40 ms (464.74–526.83) | 474.45 ms (451.96–522.91) | 584.19 ms (505.66–596.27) | +| `.all` | SIGABRT | SIGABRT | SIGABRT | +| `.cpuAndGPU` | SIGABRT | SIGABRT | SIGABRT | + +Values are **median (minimum–maximum) of five timed runs after two discarded +warmups**, one isolated process per language/policy, serial execution, rotating +policy order by language. Timed calls within each policy are consecutive; other +desktop apps remained running. A local copy of the released pipeline changed +only the requested compute units and added diagnostic logging outside timed +runs. The public v3 API does not accept these global overrides. Native CPU work +remained unchanged. + +The monotonic timer wraps `pipeline.synthesize(inputIds:style:speed:)` through +complete PCM, including native DSP and array handling, excluding text processing, +phonemization, model loading, WAV encoding and playback. These are complete-audio +latencies, not streaming first-audio measurements. They are not just the sum of +reported per-stage timers. + +| Input | Voice | Audio produced by successful policies | +| --- | --- | ---: | +| The quick brown fox jumps over the lazy dog. | `af_heart` | 3.350 s | +| こんにちは、今日はいい天気ですね。 | `jf_alpha` | 3.050 s | +| 你好，欢迎使用本地语音合成。 | `zf_001` | 3.675 s | + +Each policy receives identical token IDs and the same 256-float voice style for +that language at speed 1.0. Successful outputs were finite and non-silent, with +matching durations; this does not establish perceptual equivalence. Different +sentences and voices mean the rows do not rank languages by intrinsic speed. + +CPU+ANE was approximately **10–13× slower** than hybrid on these inputs. The +English generator alone measured a 498.24 ms median under CPU+ANE versus +26.97 ms under hybrid. This is a routing comparison, not proof of actual ANE +execution: `.cpuAndNeuralEngine` permits CPU fallback, and Core ML has no +strict ANE-only policy. + +Both `.all` and `.cpuAndGPU` aborted during **Prosody on the second warmup** +for every language, with `GPURNNOps.mm: JIT not supported` in Apple's +MetalPerformanceShadersGraph. No valid warm median is reported for those cases. +The current hybrid route completed this bounded check; it is not a claim of +sustained stability on other machines or physical iOS devices. + +Source: SDK commit `ca0cbe77843d7049f0be28db57244e21fae46566` (v0.17.6), model +release `6c0750dd02ef9d981fb82f147d784a3f171755ee`, base model assets +`006395f65025af251858b1ab0a7178a6a1e73f9f`. +[Raw measurements and protocol](Measurements/KokoroV3/comparison.json), +[CSV summary](Measurements/KokoroV3/summary.csv), +[per-call records and crash logs](Measurements/KokoroV3/). + +### Live demo: text to complete audio + +The app wraps public `synthesizeDetailed(text:voice:speed:)`, including text +normalization and phonemization, after model loading and a discarded warmup. +It excludes WAV encoding/playback. The following are **individual screenshot +runs in “One at a time” mode**, not five-run medians +([transcribed screenshot values](Measurements/KokoroV3/live-demo-screenshots.json)): + +| Language | Legacy | V3 | Legacy / V3 | V3 audio | +| --- | ---: | ---: | ---: | ---: | +| English | 149.8 ms | 79.8 ms | 1.88× | 3.35 s | +| Japanese | 305.9 ms | 57.8 ms | 5.29× | 3.05 s | +| Mandarin | 161.7 ms | 71.9 ms | 2.25× | 3.67 s | + +“Together” mode generates with both runtimes concurrently, so its timings also +include shared-resource contention. Do not mix that mode with serial runs or +attribute the gap between backend and app timing to UI rendering. The screenshot +speedups vary between runs; neither 2.3× nor 5.3× is an established overall average. +Earlier conversion-only measurements use different fixtures/runtime stages and +are not interchangeable with these SDK results. + +See [Kokoro setup and limits](KokoroAne.md#ane-version-3) and +[ANE placement and telemetry](../ANE_Profiler.md#kokoro-ane-v3). + +## Historical corpus benchmark setup + > **Setup:** Apple M5 Pro, 24 GB, on AC — Kokoro ANE rows on macOS > 26.6, PocketTTS / Supertonic-3 rows on macOS 26.5 (25F71). -> Kokoro ANE now runs on M5 out of the box — the **default** compute -> routing is the M5-safe placement (tail iSTFT on GPU, RNN stages on -> ANE; see the ᴷ footnote). Only the StyleTTS2 row is carried from the +> These legacy Kokoro ANE results used the **default** compute +> routing for these historical macOS 26.6 runs requested CPU+GPU for tail +> iSTFT and CPU+ANE for RNN stages (see the ᴷ footnote). Only the StyleTTS2 row is carried from the > M2 reference (MacBook Air M2, 16 GB), pending a HF asset fix (see the > ˢ footnote below). > **Corpus:** [MiniMax Multilingual TTS Test Set][minimax] (100 @@ -14,7 +107,7 @@ > **Status:** Kokoro ANE (English + Mandarin + Japanese via > `--phonemes`), PocketTTS (English), and Supertonic-3 (English) all > complete the full 100-phrase MiniMax run on **M5 Pro** (Kokoro on -> its default M5-safe routing). Only **StyleTTS2** is **M2-only** here — its bucketed BERT +> the routing described in footnote ᴷ). Only **StyleTTS2** is **M2-only** here — its bucketed BERT > assets ship without `model.mil`; that row carries its M2 numbers. > > [minimax]: https://huggingface.co/datasets/MiniMaxAI/TTS-Multilingual-Test-Set @@ -35,8 +128,8 @@ feel: this slice `ttft_ms == synth_ms`. **PocketTTS** is wired through its streaming API (`synthesizeStreaming`), so its `ttft_ms` is honest first-frame latency. -3. **Per-stage compute units** — Kokoro ANE is a pipeline of - 7 graphs. Sometimes ANE is *slower per call* but more efficient. +3. **Per-stage compute units** — legacy Kokoro ANE is a pipeline of + 7 graphs; v3 adds static buckets and native DSP. Sometimes ANE is *slower per call* but more efficient. The "right" compute-unit choice differs per stage. 4. **Memory footprint** — drives whether a backend is mobile-viable. 5. **Quality** — RTFx alone tells you nothing about whether the model @@ -173,8 +266,8 @@ prior **MacBook Air, Apple M2 (2022), 8-core CPU / 8-core GPU / 16-core Neural Engine, 16 GB, macOS 26** (`Mac14,2`) reference — it does not run on the M5 host (missing `model.mil` in the shipped bucketed BERT graphs — see footnote ˢ). All M5 rows use `--compute-units -default`; for Kokoro ANE that default is now the M5-safe routing (tail -iSTFT on GPU, RNN stages on ANE — see footnote ᴷ). 100 phrases per +default`; for these legacy Kokoro ANE measurements that requested CPU+GPU +for Noise/Tail and CPU+ANE for the other stages (see footnote ᴷ). 100 phrases per language. Voices are backend defaults (`af_heart` for Kokoro ANE en, `zf_001` for Kokoro ANE zh, `alba` for PocketTTS, LibriTTS iteration_3 for StyleTTS2). English WER / CER via Parakeet @@ -194,19 +287,14 @@ peak RSS, WER, CER) so there is a single source of truth. | StyleTTS2 (M2)ˢ | research | en (LibriTTS iteration_3) | ~0.67 GB¶ | 24 kHz | 256 tokens / pass (≈30 s of audio max) | No | 1574 / 3088 ms | 1574 / 3088 ms | 4.59× | 522 MB | 9.4% | 4.1% | | Supertonic-3 (int4) | Apache-2.0 | en (`M1`, 31-lang) | int4 ~0.10 GB | 44.1 kHz | 128 codepoints / pass (chunker splits ≥70 char Latin / 57 CJK) | No | **81 / 120 ms** | 81 / 120 ms | **94×** | 197 MB | 1.02%ᶜ | 0.31%ᶜ | -ᴷ **Kokoro ANE on M5** runs on the **default** compute routing, which is -now the M5-safe placement: every stage on `.cpuAndNeuralEngine` **except -the tail (iSTFT) → `.cpuAndGPU`**. This is the only routing that runs on -M5 / macOS 26.5 — the *previous* default (prosody/noise/tail on `.all`) -crashed on the 2nd+ prediction in a process (`GPURNNOps … 'JIT not -supported'` on the prosody RNN on the GPU), and `all-ane`/`cpu-only` -crash in `libBNNS` (SIGSEGV) on the tail iSTFT. Keeping the RNN off the -GPU and the iSTFT off BNNS dodges both (the tail was always meant to run -fp32 on CPU/GPU — ANE rejects the exp/sin/iSTFT — so this is faithful, -not a hack). The old `.all` default ran fine on M2 but -its M2 perf under the new default is not re-verified here. The underlying -Apple bug is tracked in [#667][i667]; this routing is -`TtsComputeUnitPreset.default` / `.aneTailGpu`. +ᴷ **Historical legacy routing on macOS 26.6:** `.aneTailGpu` requests +`.cpuAndNeuralEngine` for Albert, PostAlbert, Alignment, Prosody and Vocoder, +and `.cpuAndGPU` for Noise/Tail. Earlier M5/macOS 26.5 testing encountered +GPU RNN `JIT not supported` assertions with Prosody on `.all`, and BNNS +faults with tail iSTFT on CPU/ANE ([#667][i667]). The current legacy default +is OS-dependent: on OS 27+, Noise/Tail request CPU-only. Neither these +historical timings nor the short v3 checks establish long-session stability. +See [current routing and platform advisories](KokoroAne.md). All three Kokoro ANE rows were **re-measured on macOS 26.6 after the KokoroNoise v2 fix** ([#700][i700] — atan2 phase reconstruction, diff --git a/Documentation/TTS/KokoroAne.md b/Documentation/TTS/KokoroAne.md index ae0963675..0deb6916b 100644 --- a/Documentation/TTS/KokoroAne.md +++ b/Documentation/TTS/KokoroAne.md @@ -2,7 +2,7 @@ ## ANE version 3 -Opt in with `version: .v3`. Existing callers continue to use `.legacy`. +Available in **FluidAudio v0.17.6**. Opt in with `version: .v3`. Existing callers continue to use `.legacy`. ANE-v3 requires macOS 15 or iOS 18 and uses a hybrid pipeline: static Albert and masked decoder graphs on CPU/ANE, native Accelerate source/STFT processing, and a CPU/GPU generator adapted from @@ -13,6 +13,25 @@ policy (or `.aneTailGpu`). The v3 route is fixed and includes GPU execution, even when the legacy OS-dependent default would select CPU. Spanish and French remain supported by `.legacy`; requesting them with `.v3` fails before downloading. +### Compute routing and measured latency + +| Stage | Requested policy | +| --- | --- | +| Albert, PostAlbert, Alignment, Prosody, masked decoder | `.cpuAndNeuralEngine` | +| Fast generator; fallback Noise/Tail | `.cpuAndGPU` | +| Fallback Vocoder | `.cpuAndNeuralEngine` | +| Native source/STFT | CPU / Accelerate | + +Hybrid does not mean `.all`, and CPU+ANE still allows CPU fallback. Selected +M5 Pro / macOS 27.0 runs measured 49.45 / 35.85 / 53.97 ms median for the English / +Japanese / Mandarin demo inputs, excluding G2P and loading. CPU+ANE throughout +was 10–13× slower; `.all` and CPU+GPU throughout aborted in the GPU RNN path. +Those overrides were tested in a local harness, not exposed by the v3 public API. +See [protocol and live-demo timing boundaries](Benchmarks.md#kokoro-ane-v3-selected-m5-pro-measurements) +and [placement versus actual ANE activity](../ANE_Profiler.md#kokoro-ane-v3). + +### Usage and assets + ```swift import FluidAudio @@ -91,10 +110,11 @@ confirmed workaround. Physical-device validation remains outstanding for this op ## Legacy seven-stage runtime -Splits the Kokoro 82M graph into 7 CoreML stages so the ANE-friendly layers -(Albert / PostAlbert / Alignment / Vocoder) stay resident on the Neural Engine -while Prosody / Noise / Tail run on CPU+GPU. Yields **3-11× RTFx** on Apple -Silicon. +Splits Kokoro 82M into seven Core ML stages. The legacy default requests +CPU+ANE for Albert, PostAlbert, Alignment, Prosody and Vocoder. Noise/Tail request +CPU+GPU on older OS versions and CPU-only on OS 27+. Core ML decides placement; +these policies do not imply that every eligible stage runs on ANE. See the +[historical benchmark results](Benchmarks.md#historical-corpus-benchmark-setup). Derived from [laishere/kokoro-coreml](https://github.com/laishere/kokoro-coreml), used with the author's permission. Conversion lives in @@ -104,7 +124,7 @@ used with the author's permission. Conversion lives in | Aspect | `KokoroAneManager` | |------------------|-------------------------------------------------| -| Compute | 4 stages on ANE, 3 on GPU | +| Compute | Per-stage policies; OS-dependent legacy routing, fixed hybrid v3 routing | | Voices | Variant catalogs (54 en / 103 zh / 5 ja / 3 es / 1 fr) | | Input length | ≤ 510 phoneme characters / utterance | | Custom lexicon | No | diff --git a/Documentation/TTS/Measurements/KokoroV3/ane-latest-clicks.json b/Documentation/TTS/Measurements/KokoroV3/ane-latest-clicks.json new file mode 100644 index 000000000..b7e5e1934 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/ane-latest-clicks.json @@ -0,0 +1,287 @@ +[ + { + "time": "2026-10-08 00:40:01", + "ane_mW": 0.0, + "interval_s": 1.030301791 + }, + { + "time": "2026-10-08 00:40:02", + "ane_mW": 0.0, + "interval_s": 1.038022666 + }, + { + "time": "2026-10-08 00:40:03", + "ane_mW": 0.0, + "interval_s": 1.035242 + }, + { + "time": "2026-10-08 00:40:04", + "ane_mW": 0.0, + "interval_s": 1.021118833 + }, + { + "time": "2026-10-08 00:40:05", + "ane_mW": 0.0, + "interval_s": 1.021117625 + }, + { + "time": "2026-10-08 00:40:06", + "ane_mW": 0.0, + "interval_s": 1.035186333 + }, + { + "time": "2026-10-08 00:40:07", + "ane_mW": 30.0454, + "interval_s": 1.031771541 + }, + { + "time": "2026-10-08 00:40:08", + "ane_mW": 0.0, + "interval_s": 1.03041975 + }, + { + "time": "2026-10-08 00:40:09", + "ane_mW": 0.0, + "interval_s": 1.021197874 + }, + { + "time": "2026-10-08 00:40:10", + "ane_mW": 0.970231, + "interval_s": 1.030682791 + }, + { + "time": "2026-10-08 00:40:11", + "ane_mW": 0.0, + "interval_s": 1.031136916 + }, + { + "time": "2026-10-08 00:40:12", + "ane_mW": 0.0, + "interval_s": 1.035806666 + }, + { + "time": "2026-10-08 00:40:13", + "ane_mW": 0.0, + "interval_s": 1.030924875 + }, + { + "time": "2026-10-08 00:40:14", + "ane_mW": 0.0, + "interval_s": 1.020524833 + }, + { + "time": "2026-10-08 00:40:15", + "ane_mW": 0.0, + "interval_s": 1.232727083 + }, + { + "time": "2026-10-08 00:40:16", + "ane_mW": 0.0, + "interval_s": 1.033203291 + }, + { + "time": "2026-10-08 00:40:17", + "ane_mW": 3.86755, + "interval_s": 1.034246666 + }, + { + "time": "2026-10-08 00:40:18", + "ane_mW": 0.0, + "interval_s": 1.031142416 + }, + { + "time": "2026-10-08 00:40:19", + "ane_mW": 0.0, + "interval_s": 1.0243685 + }, + { + "time": "2026-10-08 00:40:20", + "ane_mW": 0.0, + "interval_s": 1.134790916 + }, + { + "time": "2026-10-08 00:40:21", + "ane_mW": 14.5062, + "interval_s": 1.034037125 + }, + { + "time": "2026-10-08 00:40:22", + "ane_mW": 0.0, + "interval_s": 1.031369291 + }, + { + "time": "2026-10-08 00:40:23", + "ane_mW": 9.72621, + "interval_s": 1.028149708 + }, + { + "time": "2026-10-08 00:40:25", + "ane_mW": 0.0, + "interval_s": 1.029091374 + }, + { + "time": "2026-10-08 00:40:26", + "ane_mW": 0.96849, + "interval_s": 1.032534791 + }, + { + "time": "2026-10-08 00:40:27", + "ane_mW": 0.0, + "interval_s": 1.039927291 + }, + { + "time": "2026-10-08 00:40:28", + "ane_mW": 0.0, + "interval_s": 1.028888208 + }, + { + "time": "2026-10-08 00:40:29", + "ane_mW": 0.0, + "interval_s": 1.024633666 + }, + { + "time": "2026-10-08 00:40:30", + "ane_mW": 0.0, + "interval_s": 1.032946833 + }, + { + "time": "2026-10-08 00:40:31", + "ane_mW": 15.4759, + "interval_s": 1.033868875 + }, + { + "time": "2026-10-08 00:40:32", + "ane_mW": 0.0, + "interval_s": 1.038721875 + }, + { + "time": "2026-10-08 00:40:33", + "ane_mW": 11.6654, + "interval_s": 1.028683791 + }, + { + "time": "2026-10-08 00:40:34", + "ane_mW": 0.0, + "interval_s": 1.036165083 + }, + { + "time": "2026-10-08 00:40:35", + "ane_mW": 0.0, + "interval_s": 1.031422625 + }, + { + "time": "2026-10-08 00:40:36", + "ane_mW": 0.0, + "interval_s": 1.038142041 + }, + { + "time": "2026-10-08 00:40:37", + "ane_mW": 0.976154, + "interval_s": 1.024428916 + }, + { + "time": "2026-10-08 00:40:38", + "ane_mW": 4.51691, + "interval_s": 1.1069505 + }, + { + "time": "2026-10-08 00:40:39", + "ane_mW": 0.967613, + "interval_s": 1.03347125 + }, + { + "time": "2026-10-08 00:40:40", + "ane_mW": 0.0, + "interval_s": 1.030723416 + }, + { + "time": "2026-10-08 00:40:41", + "ane_mW": 0.0, + "interval_s": 1.021509625 + }, + { + "time": "2026-10-08 00:40:42", + "ane_mW": 0.0, + "interval_s": 1.02023625 + }, + { + "time": "2026-10-08 00:40:43", + "ane_mW": 0.0, + "interval_s": 1.025294458 + }, + { + "time": "2026-10-08 00:40:44", + "ane_mW": 0.0, + "interval_s": 1.022167583 + }, + { + "time": "2026-10-08 00:40:45", + "ane_mW": 0.0, + "interval_s": 1.02165825 + }, + { + "time": "2026-10-08 00:40:46", + "ane_mW": 0.977788, + "interval_s": 1.022716583 + }, + { + "time": "2026-10-08 00:40:47", + "ane_mW": 0.0, + "interval_s": 1.074667166 + }, + { + "time": "2026-10-08 00:40:48", + "ane_mW": 0.0, + "interval_s": 1.023795333 + }, + { + "time": "2026-10-08 00:40:49", + "ane_mW": 0.0, + "interval_s": 1.021059416 + }, + { + "time": "2026-10-08 00:40:50", + "ane_mW": 0.0, + "interval_s": 1.025929333 + }, + { + "time": "2026-10-08 00:40:51", + "ane_mW": 0.0, + "interval_s": 1.021785875 + }, + { + "time": "2026-10-08 00:40:52", + "ane_mW": 0.0, + "interval_s": 1.019317916 + }, + { + "time": "2026-10-08 00:40:53", + "ane_mW": 0.0, + "interval_s": 1.053311124 + }, + { + "time": "2026-10-08 00:40:54", + "ane_mW": 0.0, + "interval_s": 1.029971875 + }, + { + "time": "2026-10-08 00:40:56", + "ane_mW": 0.0, + "interval_s": 1.03295475 + }, + { + "time": "2026-10-08 00:40:57", + "ane_mW": 0.0, + "interval_s": 1.023871791 + }, + { + "time": "2026-10-08 00:40:58", + "ane_mW": 0.0, + "interval_s": 1.028349041 + }, + { + "time": "2026-10-08 00:40:59", + "ane_mW": 0.0, + "interval_s": 1.032043208 + } +] \ No newline at end of file diff --git a/Documentation/TTS/Measurements/KokoroV3/comparison.json b/Documentation/TTS/Measurements/KokoroV3/comparison.json new file mode 100644 index 000000000..4035c8e61 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/comparison.json @@ -0,0 +1,198 @@ +{ + "hardware": "Apple M5 Pro, 24 GiB", + "os": "macOS 27.0 (26A428)", + "sdkMergeCommit": "ca0cbe77843d7049f0be28db57244e21fae46566", + "modelRevision": "6c0750dd02ef9d981fb82f147d784a3f171755ee", + "warmups": 2, + "timedRuns": 5, + "method": "Serial isolated processes per language/policy. Same exact cached input IDs and style vectors. Timer around pipeline.synthesize through complete PCM, including native source/STFT. Excludes model load, G2P, WAV and statistics. Stage logging enabled only during discarded warmups. Policies rotate by language; timed iterations within a policy are consecutive. Existing desktop apps remain running. Local exploratory samples, not official benchmarks.", + "routing": "Local copy of released v3 pipeline. Only MLModelConfiguration.computeUnits is overridden for all models. Native operations remain CPU. Hybrid keeps released per-stage policies. Released v0.17.6 does not accept these global v3 overrides. cpuAndNeuralEngine permits CPU fallback and is not strict ANE-only. Compute-unit requests are not measured hardware attribution.", + "rows": [ + { + "language": "en", + "policy": "hybrid", + "status": "passed", + "medianMs": 49.449666, + "minMs": 40.515875, + "maxMs": 66.98, + "audioSeconds": 3.35, + "peak": 0.5302734375, + "stageMedians": { + "Albert_64": 1.4937919974327087, + "Alignment": 1.232875019311905, + "Decoder_200": 2.793833017349243, + "NativeSource": 3.809750020503998, + "PostAlbert": 4.211416989564896, + "Prosody": 6.195915997028351, + "SourceGenerator": 26.96504098176956 + } + }, + { + "language": "en", + "policy": "cpuAndNeuralEngine", + "status": "passed", + "medianMs": 511.400542, + "minMs": 464.735209, + "maxMs": 526.832541, + "audioSeconds": 3.35, + "peak": 0.544921875, + "stageMedians": { + "Albert_64": 1.2941249907016754, + "Alignment": 0.3946250081062317, + "Decoder_200": 2.697499990463257, + "NativeSource": 2.48920801281929, + "PostAlbert": 2.7586250007152557, + "Prosody": 3.3506669998168945, + "SourceGenerator": 498.2420420050621 + } + }, + { + "language": "en", + "policy": "all", + "status": "failed (-6)", + "medianMs": null, + "minMs": null, + "maxMs": null, + "audioSeconds": null, + "peak": null, + "stageMedians": {}, + "failureEvidence": "/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported'" + }, + { + "language": "en", + "policy": "cpuAndGPU", + "status": "failed (-6)", + "medianMs": null, + "minMs": null, + "maxMs": null, + "audioSeconds": null, + "peak": null, + "stageMedians": {}, + "failureEvidence": "/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported'" + }, + { + "language": "ja", + "policy": "cpuAndNeuralEngine", + "status": "passed", + "medianMs": 474.447791, + "minMs": 451.964542, + "maxMs": 522.908959, + "audioSeconds": 3.05, + "peak": 0.5595703125, + "stageMedians": { + "Albert_64": 1.3007080256938934, + "Alignment": 0.4070829749107361, + "Decoder_200": 2.7335000038146973, + "NativeSource": 2.378792017698288, + "PostAlbert": 2.515082985162735, + "Prosody": 3.142916977405548, + "SourceGenerator": 461.57045900821686 + } + }, + { + "language": "ja", + "policy": "all", + "status": "failed (-6)", + "medianMs": null, + "minMs": null, + "maxMs": null, + "audioSeconds": null, + "peak": null, + "stageMedians": {}, + "failureEvidence": "/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported'" + }, + { + "language": "ja", + "policy": "cpuAndGPU", + "status": "failed (-6)", + "medianMs": null, + "minMs": null, + "maxMs": null, + "audioSeconds": null, + "peak": null, + "stageMedians": {}, + "failureEvidence": "/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported'" + }, + { + "language": "ja", + "policy": "hybrid", + "status": "passed", + "medianMs": 35.851334, + "minMs": 35.414625, + "maxMs": 36.358667, + "audioSeconds": 3.05, + "peak": 0.5419921875, + "stageMedians": { + "Albert_64": 1.21670800447464, + "Alignment": 0.28737500309944153, + "Decoder_200": 2.6525830030441284, + "NativeSource": 2.15412500500679, + "PostAlbert": 2.229374974966049, + "Prosody": 2.8358749747276306, + "SourceGenerator": 24.16687497496605 + } + }, + { + "language": "zh", + "policy": "all", + "status": "failed (-6)", + "medianMs": null, + "minMs": null, + "maxMs": null, + "audioSeconds": null, + "peak": null, + "stageMedians": {}, + "failureEvidence": "/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported'" + }, + { + "language": "zh", + "policy": "cpuAndGPU", + "status": "failed (-6)", + "medianMs": null, + "minMs": null, + "maxMs": null, + "audioSeconds": null, + "peak": null, + "stageMedians": {}, + "failureEvidence": "/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported'" + }, + { + "language": "zh", + "policy": "hybrid", + "status": "passed", + "medianMs": 53.97225, + "minMs": 45.457584, + "maxMs": 58.523583, + "audioSeconds": 3.675, + "peak": 0.6953125, + "stageMedians": { + "Albert_64": 1.4665409922599792, + "Alignment": 0.6295419931411743, + "Decoder_200": 3.028082996606827, + "NativeSource": 4.10979101061821, + "PostAlbert": 3.971457988023758, + "Prosody": 7.614499986171722, + "SourceGenerator": 32.09708300232887 + } + }, + { + "language": "zh", + "policy": "cpuAndNeuralEngine", + "status": "passed", + "medianMs": 584.190834, + "minMs": 505.655042, + "maxMs": 596.269708, + "audioSeconds": 3.675, + "peak": 0.681640625, + "stageMedians": { + "Albert_64": 1.2982079982757568, + "Alignment": 0.33283400535583496, + "Decoder_200": 2.689083009958267, + "NativeSource": 2.761209011077881, + "PostAlbert": 2.2730830013751984, + "Prosody": 3.4774169921875, + "SourceGenerator": 562.4224589765072 + } + } + ] +} \ No newline at end of file diff --git a/Documentation/TTS/Measurements/KokoroV3/compute-placement.json b/Documentation/TTS/Measurements/KokoroV3/compute-placement.json new file mode 100644 index 000000000..a094630a4 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/compute-placement.json @@ -0,0 +1,302 @@ +[ + { + "estimatedCost" : { + + }, + "language" : "en", + "model" : "base\/KokoroAlbert", + "operationsSupportingANE" : 303, + "preferredOperations" : { + "CPU" : 310, + "unknown" : 244 + } + }, + { + "estimatedCost" : { + "CPU" : 0.9999148906586185 + }, + "language" : "en", + "model" : "base\/KokoroPostAlbert", + "operationsSupportingANE" : 74, + "preferredOperations" : { + "CPU" : 98, + "unknown" : 173 + } + }, + { + "estimatedCost" : { + "CPU" : 0.9999999999999997 + }, + "language" : "en", + "model" : "base\/KokoroAlignment", + "operationsSupportingANE" : 12, + "preferredOperations" : { + "CPU" : 19, + "unknown" : 29 + } + }, + { + "estimatedCost" : { + "CPU" : 1 + }, + "language" : "en", + "model" : "base\/KokoroProsody_v2", + "operationsSupportingANE" : 0, + "preferredOperations" : { + "CPU" : 140, + "unknown" : 252 + } + }, + { + "estimatedCost" : { + "ANE" : 0.5203152309015737, + "CPU" : 0.47968476909842467 + }, + "language" : "en", + "model" : "base\/KokoroVocoder", + "operationsSupportingANE" : 648, + "preferredOperations" : { + "ANE" : 648, + "CPU" : 6, + "unknown" : 879 + } + }, + { + "estimatedCost" : { + "ANE" : 0.9881555442435253, + "CPU" : 0.011844455756474118 + }, + "language" : "en", + "model" : "fast\/KokoroAlbert_64", + "operationsSupportingANE" : 295, + "preferredOperations" : { + "ANE" : 295, + "CPU" : 8, + "unknown" : 235 + } + }, + { + "estimatedCost" : { + "ANE" : 1.0000000000000009 + }, + "language" : "en", + "model" : "decoder\/KokoroDecoderPre_200", + "operationsSupportingANE" : 258, + "preferredOperations" : { + "ANE" : 258, + "unknown" : 291 + } + }, + { + "estimatedCost" : { + "GPU" : 1.0000000000000102 + }, + "language" : "en", + "model" : "fast\/KokoroSourceGenerator", + "operationsSupportingANE" : 0, + "preferredOperations" : { + "GPU" : 1089, + "unknown" : 1075 + } + }, + { + "estimatedCost" : { + + }, + "language" : "ja", + "model" : "base\/KokoroAlbert", + "operationsSupportingANE" : 303, + "preferredOperations" : { + "CPU" : 310, + "unknown" : 244 + } + }, + { + "estimatedCost" : { + "CPU" : 0.9999148906586185 + }, + "language" : "ja", + "model" : "base\/KokoroPostAlbert", + "operationsSupportingANE" : 74, + "preferredOperations" : { + "CPU" : 98, + "unknown" : 173 + } + }, + { + "estimatedCost" : { + "CPU" : 0.9999999999999997 + }, + "language" : "ja", + "model" : "base\/KokoroAlignment", + "operationsSupportingANE" : 12, + "preferredOperations" : { + "CPU" : 19, + "unknown" : 29 + } + }, + { + "estimatedCost" : { + "CPU" : 1.0000000000000004 + }, + "language" : "ja", + "model" : "base\/KokoroProsody_v2", + "operationsSupportingANE" : 0, + "preferredOperations" : { + "CPU" : 140, + "unknown" : 252 + } + }, + { + "estimatedCost" : { + "ANE" : 0.5203152309015769, + "CPU" : 0.4796847690984255 + }, + "language" : "ja", + "model" : "base\/KokoroVocoder", + "operationsSupportingANE" : 648, + "preferredOperations" : { + "ANE" : 648, + "CPU" : 6, + "unknown" : 879 + } + }, + { + "estimatedCost" : { + "ANE" : 0.9881555442435253, + "CPU" : 0.011844455756474118 + }, + "language" : "ja", + "model" : "fast\/KokoroAlbert_64", + "operationsSupportingANE" : 295, + "preferredOperations" : { + "ANE" : 295, + "CPU" : 8, + "unknown" : 235 + } + }, + { + "estimatedCost" : { + "ANE" : 1.000000000000001 + }, + "language" : "ja", + "model" : "decoder\/KokoroDecoderPre_200", + "operationsSupportingANE" : 258, + "preferredOperations" : { + "ANE" : 258, + "unknown" : 291 + } + }, + { + "estimatedCost" : { + "GPU" : 1.0000000000000093 + }, + "language" : "ja", + "model" : "fast\/KokoroSourceGenerator", + "operationsSupportingANE" : 0, + "preferredOperations" : { + "GPU" : 1089, + "unknown" : 1075 + } + }, + { + "estimatedCost" : { + + }, + "language" : "zh", + "model" : "base\/KokoroAlbert", + "operationsSupportingANE" : 303, + "preferredOperations" : { + "CPU" : 310, + "unknown" : 244 + } + }, + { + "estimatedCost" : { + "CPU" : 0.9998897507275808 + }, + "language" : "zh", + "model" : "base\/KokoroPostAlbert", + "operationsSupportingANE" : 74, + "preferredOperations" : { + "CPU" : 98, + "unknown" : 173 + } + }, + { + "estimatedCost" : { + "CPU" : 1 + }, + "language" : "zh", + "model" : "base\/KokoroAlignment", + "operationsSupportingANE" : 12, + "preferredOperations" : { + "CPU" : 19, + "unknown" : 29 + } + }, + { + "estimatedCost" : { + "CPU" : 1.0000000000000038 + }, + "language" : "zh", + "model" : "base\/KokoroProsody_v2", + "operationsSupportingANE" : 0, + "preferredOperations" : { + "CPU" : 140, + "unknown" : 252 + } + }, + { + "estimatedCost" : { + "ANE" : 0.5198534448227075, + "CPU" : 0.4801465551772948 + }, + "language" : "zh", + "model" : "base\/KokoroVocoder", + "operationsSupportingANE" : 648, + "preferredOperations" : { + "ANE" : 648, + "CPU" : 6, + "unknown" : 879 + } + }, + { + "estimatedCost" : { + "ANE" : 0.9881555442435249, + "CPU" : 0.011844455756474115 + }, + "language" : "zh", + "model" : "fast\/KokoroAlbert_64", + "operationsSupportingANE" : 295, + "preferredOperations" : { + "ANE" : 295, + "CPU" : 8, + "unknown" : 235 + } + }, + { + "estimatedCost" : { + "ANE" : 1.000000000000001 + }, + "language" : "zh", + "model" : "decoder\/KokoroDecoderPre_200", + "operationsSupportingANE" : 258, + "preferredOperations" : { + "ANE" : 258, + "unknown" : 291 + } + }, + { + "estimatedCost" : { + "GPU" : 1.0000000000000095 + }, + "language" : "zh", + "model" : "fast\/KokoroSourceGenerator", + "operationsSupportingANE" : 0, + "preferredOperations" : { + "GPU" : 1089, + "unknown" : 1075 + } + } +] \ No newline at end of file diff --git a/Documentation/TTS/Measurements/KokoroV3/en-all-crash.txt b/Documentation/TTS/Measurements/KokoroV3/en-all-crash.txt new file mode 100644 index 000000000..4054daa02 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/en-all-crash.txt @@ -0,0 +1,26 @@ +LOAD_MODEL KokoroAlbert units=2 +LOAD_MODEL KokoroPostAlbert units=2 +LOAD_MODEL KokoroAlignment units=2 +LOAD_MODEL KokoroProsody_v2 units=2 +LOAD_MODEL KokoroNoise_v2 units=2 +LOAD_MODEL KokoroVocoder units=2 +LOAD_MODEL KokoroTail_v2 units=2 +LOAD_MODEL KokoroAlbert_32 units=2 +LOAD_MODEL KokoroAlbert_64 units=2 +LOAD_MODEL KokoroDecoderPre_120 units=2 +LOAD_MODEL KokoroDecoderPre_200 units=2 +LOAD_MODEL KokoroSourceGenerator units=2 +START en all -2 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +PREDICT Decoder_200 +PREDICT SourceGenerator +DONE en all -2 ms=2004.485459 frames=134 peak=0.5224609375 +START en all -1 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported' diff --git a/Documentation/TTS/Measurements/KokoroV3/en-cpuAndGPU-crash.txt b/Documentation/TTS/Measurements/KokoroV3/en-cpuAndGPU-crash.txt new file mode 100644 index 000000000..b42b6958e --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/en-cpuAndGPU-crash.txt @@ -0,0 +1,26 @@ +LOAD_MODEL KokoroAlbert units=1 +LOAD_MODEL KokoroPostAlbert units=1 +LOAD_MODEL KokoroAlignment units=1 +LOAD_MODEL KokoroProsody_v2 units=1 +LOAD_MODEL KokoroNoise_v2 units=1 +LOAD_MODEL KokoroVocoder units=1 +LOAD_MODEL KokoroTail_v2 units=1 +LOAD_MODEL KokoroAlbert_32 units=1 +LOAD_MODEL KokoroAlbert_64 units=1 +LOAD_MODEL KokoroDecoderPre_120 units=1 +LOAD_MODEL KokoroDecoderPre_200 units=1 +LOAD_MODEL KokoroSourceGenerator units=1 +START en cpuAndGPU -2 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +PREDICT Decoder_200 +PREDICT SourceGenerator +DONE en cpuAndGPU -2 ms=1955.613 frames=135 peak=0.546875 +START en cpuAndGPU -1 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported' diff --git a/Documentation/TTS/Measurements/KokoroV3/en-cpuAndNeuralEngine.json b/Documentation/TTS/Measurements/KokoroV3/en-cpuAndNeuralEngine.json new file mode 100644 index 000000000..a3d07cb84 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/en-cpuAndNeuralEngine.json @@ -0,0 +1,187 @@ +{ + "inputIds" : [ + 0, + 81, + 51, + 16, + 53, + 65, + 156, + 102, + 53, + 16, + 44, + 123, + 156, + 39, + 56, + 16, + 48, + 156, + 69, + 53, + 61, + 16, + 82, + 156, + 138, + 55, + 58, + 61, + 16, + 156, + 31, + 64, + 83, + 123, + 16, + 81, + 51, + 16, + 54, + 156, + 24, + 68, + 51, + 16, + 46, + 156, + 76, + 92, + 4, + 0 + ], + "language" : "en", + "loadMilliseconds" : 20067.208167, + "policy" : "cpuAndNeuralEngine", + "text" : "The quick brown fox jumps over the lazy dog.", + "trials" : [ + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : -2, + "milliseconds" : 745.651625, + "peak" : 0.544921875, + "rms" : 0.07071478783866116, + "stages" : { + "Albert_64" : 3.1984580159187317, + "Alignment" : 5.128749996423721, + "Decoder_200" : 9.961833983659744, + "NativeSource" : 5.051499992609024, + "PostAlbert" : 53.166415989398956, + "Prosody" : 95.31204199790955, + "SourceGenerator" : 572.8598330020905 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : -1, + "milliseconds" : 480.074834, + "peak" : 0.544921875, + "rms" : 0.07071478783866116, + "stages" : { + "Albert_64" : 1.3792920112609863, + "Alignment" : 0.6544589996337891, + "Decoder_200" : 2.756417006254196, + "NativeSource" : 2.8941250145435333, + "PostAlbert" : 2.4882499873638153, + "Prosody" : 3.322874993085861, + "SourceGenerator" : 466.1417499780655 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 0, + "milliseconds" : 483.677125, + "peak" : 0.544921875, + "rms" : 0.07071478783866116, + "stages" : { + "Albert_64" : 1.2941249907016754, + "Alignment" : 0.4067919850349426, + "Decoder_200" : 2.642500013113022, + "NativeSource" : 2.715041995048523, + "PostAlbert" : 2.7586250007152557, + "Prosody" : 3.3506669998168945, + "SourceGenerator" : 470.2010000050068 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 1, + "milliseconds" : 464.735209, + "peak" : 0.544921875, + "rms" : 0.07071478783866116, + "stages" : { + "Albert_64" : 1.2481250166893005, + "Alignment" : 0.3946250081062317, + "Decoder_200" : 2.638667017221451, + "NativeSource" : 2.5361250042915344, + "PostAlbert" : 2.4040839970111847, + "Prosody" : 3.1896250247955322, + "SourceGenerator" : 452.04262498021126 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 2, + "milliseconds" : 511.400542, + "peak" : 0.544921875, + "rms" : 0.07071478783866116, + "stages" : { + "Albert_64" : 1.2799580097198486, + "Alignment" : 0.3497919738292694, + "Decoder_200" : 2.697499990463257, + "NativeSource" : 2.4698339998722076, + "PostAlbert" : 2.8662079870700836, + "Prosody" : 3.1981250047683716, + "SourceGenerator" : 498.2420420050621 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 3, + "milliseconds" : 523.14175, + "peak" : 0.544921875, + "rms" : 0.07071478783866116, + "stages" : { + "Albert_64" : 1.3252909779548645, + "Alignment" : 0.4409170150756836, + "Decoder_200" : 2.854000002145767, + "NativeSource" : 2.48920801281929, + "PostAlbert" : 2.890042006969452, + "Prosody" : 3.6083329916000366, + "SourceGenerator" : 509.19337499141693 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 4, + "milliseconds" : 526.832541, + "peak" : 0.544921875, + "rms" : 0.07071478783866116, + "stages" : { + "Albert_64" : 2.1475830078125, + "Alignment" : 0.3184170126914978, + "Decoder_200" : 2.7588749825954437, + "NativeSource" : 2.455958992242813, + "PostAlbert" : 2.6078749895095825, + "Prosody" : 3.837917000055313, + "SourceGenerator" : 512.2457499802113 + } + } + ], + "voice" : "af_heart" +} \ No newline at end of file diff --git a/Documentation/TTS/Measurements/KokoroV3/en-hybrid.json b/Documentation/TTS/Measurements/KokoroV3/en-hybrid.json new file mode 100644 index 000000000..ac9f407eb --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/en-hybrid.json @@ -0,0 +1,187 @@ +{ + "inputIds" : [ + 0, + 81, + 51, + 16, + 53, + 65, + 156, + 102, + 53, + 16, + 44, + 123, + 156, + 39, + 56, + 16, + 48, + 156, + 69, + 53, + 61, + 16, + 82, + 156, + 138, + 55, + 58, + 61, + 16, + 156, + 31, + 64, + 83, + 123, + 16, + 81, + 51, + 16, + 54, + 156, + 24, + 68, + 51, + 16, + 46, + 156, + 76, + 92, + 4, + 0 + ], + "language" : "en", + "loadMilliseconds" : 17234.768791, + "policy" : "hybrid", + "text" : "The quick brown fox jumps over the lazy dog.", + "trials" : [ + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : -2, + "milliseconds" : 513.148125, + "peak" : 0.5302734375, + "rms" : 0.07102196727029357, + "stages" : { + "Albert_64" : 3.5204170048236847, + "Alignment" : 1.461542010307312, + "Decoder_200" : 5.392500013113022, + "NativeSource" : 11.429333984851837, + "PostAlbert" : 37.77300000190735, + "Prosody" : 156.0267500281334, + "SourceGenerator" : 296.64258298277855 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : -1, + "milliseconds" : 68.936917, + "peak" : 0.5302734375, + "rms" : 0.07102196727029357, + "stages" : { + "Albert_64" : 1.4530420005321503, + "Alignment" : 2.651416003704071, + "Decoder_200" : 3.2367909848690033, + "NativeSource" : 4.909207999706268, + "PostAlbert" : 4.708834022283554, + "Prosody" : 9.60487499833107, + "SourceGenerator" : 41.64645901322365 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 0, + "milliseconds" : 54.695708, + "peak" : 0.5302734375, + "rms" : 0.07102196727029357, + "stages" : { + "Albert_64" : 1.4937919974327087, + "Alignment" : 3.8802499771118164, + "Decoder_200" : 2.877624988555908, + "NativeSource" : 3.809750020503998, + "PostAlbert" : 5.75220799446106, + "Prosody" : 9.354083985090256, + "SourceGenerator" : 26.96504098176956 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 1, + "milliseconds" : 66.98, + "peak" : 0.5302734375, + "rms" : 0.07102196727029357, + "stages" : { + "Albert_64" : 1.6079999804496765, + "Alignment" : 0.8746669888496399, + "Decoder_200" : 2.793833017349243, + "NativeSource" : 4.659707993268967, + "PostAlbert" : 4.211416989564896, + "Prosody" : 11.298958986997604, + "SourceGenerator" : 40.78362500667572 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 2, + "milliseconds" : 49.449666, + "peak" : 0.5302734375, + "rms" : 0.07102196727029357, + "stages" : { + "Albert_64" : 1.5026249885559082, + "Alignment" : 1.252208024263382, + "Decoder_200" : 2.92637500166893, + "NativeSource" : 3.828915983438492, + "PostAlbert" : 4.635500013828278, + "Prosody" : 5.726624995470047, + "SourceGenerator" : 28.984375 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 3, + "milliseconds" : 44.23875, + "peak" : 0.5302734375, + "rms" : 0.07102196727029357, + "stages" : { + "Albert_64" : 1.357459008693695, + "Alignment" : 1.232875019311905, + "Decoder_200" : 2.7590419948101044, + "NativeSource" : 3.3920409977436066, + "PostAlbert" : 2.6981250047683716, + "Prosody" : 6.195915997028351, + "SourceGenerator" : 26.20254099369049 + } + }, + { + "audioSeconds" : 3.35, + "fast" : true, + "frames" : 134, + "iteration" : 4, + "milliseconds" : 40.515875, + "peak" : 0.5302734375, + "rms" : 0.07102196727029357, + "stages" : { + "Albert_64" : 1.3372500240802765, + "Alignment" : 0.3967079818248749, + "Decoder_200" : 2.7628749907016754, + "NativeSource" : 2.815665990114212, + "PostAlbert" : 2.8588750064373016, + "Prosody" : 3.3930409848690033, + "SourceGenerator" : 26.599665999412537 + } + } + ], + "voice" : "af_heart" +} \ No newline at end of file diff --git a/Documentation/TTS/Measurements/KokoroV3/ja-all-crash.txt b/Documentation/TTS/Measurements/KokoroV3/ja-all-crash.txt new file mode 100644 index 000000000..bd1af9a88 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/ja-all-crash.txt @@ -0,0 +1,26 @@ +LOAD_MODEL KokoroAlbert units=2 +LOAD_MODEL KokoroPostAlbert units=2 +LOAD_MODEL KokoroAlignment units=2 +LOAD_MODEL KokoroProsody_v2 units=2 +LOAD_MODEL KokoroNoise_v2 units=2 +LOAD_MODEL KokoroVocoder units=2 +LOAD_MODEL KokoroTail_v2 units=2 +LOAD_MODEL KokoroAlbert_32 units=2 +LOAD_MODEL KokoroAlbert_64 units=2 +LOAD_MODEL KokoroDecoderPre_120 units=2 +LOAD_MODEL KokoroDecoderPre_200 units=2 +LOAD_MODEL KokoroSourceGenerator units=2 +START ja all -2 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +PREDICT Decoder_200 +PREDICT SourceGenerator +DONE ja all -2 ms=1217.815333 frames=122 peak=0.55908203125 +START ja all -1 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported' diff --git a/Documentation/TTS/Measurements/KokoroV3/ja-cpuAndGPU-crash.txt b/Documentation/TTS/Measurements/KokoroV3/ja-cpuAndGPU-crash.txt new file mode 100644 index 000000000..e350c1f7c --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/ja-cpuAndGPU-crash.txt @@ -0,0 +1,26 @@ +LOAD_MODEL KokoroAlbert units=1 +LOAD_MODEL KokoroPostAlbert units=1 +LOAD_MODEL KokoroAlignment units=1 +LOAD_MODEL KokoroProsody_v2 units=1 +LOAD_MODEL KokoroNoise_v2 units=1 +LOAD_MODEL KokoroVocoder units=1 +LOAD_MODEL KokoroTail_v2 units=1 +LOAD_MODEL KokoroAlbert_32 units=1 +LOAD_MODEL KokoroAlbert_64 units=1 +LOAD_MODEL KokoroDecoderPre_120 units=1 +LOAD_MODEL KokoroDecoderPre_200 units=1 +LOAD_MODEL KokoroSourceGenerator units=1 +START ja cpuAndGPU -2 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +PREDICT Decoder_200 +PREDICT SourceGenerator +DONE ja cpuAndGPU -2 ms=235.355333 frames=122 peak=0.5869140625 +START ja cpuAndGPU -1 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported' diff --git a/Documentation/TTS/Measurements/KokoroV3/ja-cpuAndNeuralEngine.json b/Documentation/TTS/Measurements/KokoroV3/ja-cpuAndNeuralEngine.json new file mode 100644 index 000000000..cbc6c1931 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/ja-cpuAndNeuralEngine.json @@ -0,0 +1,176 @@ +{ + "inputIds" : [ + 0, + 53, + 57, + 114, + 114, + 51, + 21, + 51, + 75, + 43, + 3, + 16, + 53, + 164, + 57, + 158, + 16, + 75, + 43, + 16, + 51, + 158, + 16, + 62, + 47, + 112, + 53, + 164, + 51, + 16, + 46, + 47, + 61, + 101, + 16, + 56, + 47, + 4, + 0 + ], + "language" : "ja", + "loadMilliseconds" : 33982.602125, + "policy" : "cpuAndNeuralEngine", + "text" : "こんにちは、今日はいい天気ですね。", + "trials" : [ + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : -2, + "milliseconds" : 622.046666, + "peak" : 0.5595703125, + "rms" : 0.10189608270346202, + "stages" : { + "Albert_64" : 7.604290992021561, + "Alignment" : 1.1906670033931732, + "Decoder_200" : 7.815625011920929, + "NativeSource" : 2.8563750088214874, + "PostAlbert" : 38.8741250038147, + "Prosody" : 40.71341699361801, + "SourceGenerator" : 522.2260000109673 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : -1, + "milliseconds" : 476.642459, + "peak" : 0.5595703125, + "rms" : 0.10189608270346202, + "stages" : { + "Albert_64" : 1.4755419790744781, + "Alignment" : 0.6910420060157776, + "Decoder_200" : 2.775375008583069, + "NativeSource" : 2.519124984741211, + "PostAlbert" : 2.4829579889774323, + "Prosody" : 3.0218749940395355, + "SourceGenerator" : 463.347000002861 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 0, + "milliseconds" : 463.698958, + "peak" : 0.5595703125, + "rms" : 0.10189608270346202, + "stages" : { + "Albert_64" : 1.3007080256938934, + "Alignment" : 0.5215829908847809, + "Decoder_200" : 2.7360000014305115, + "NativeSource" : 2.53541699051857, + "PostAlbert" : 2.4708750247955322, + "Prosody" : 2.9949580132961273, + "SourceGenerator" : 450.78004202246666 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 1, + "milliseconds" : 522.908959, + "peak" : 0.5595703125, + "rms" : 0.10189608270346202, + "stages" : { + "Albert_64" : 1.3494159877300262, + "Alignment" : 0.5244579911231995, + "Decoder_200" : 3.008958011865616, + "NativeSource" : 2.7812910079956055, + "PostAlbert" : 3.458750009536743, + "Prosody" : 5.3766250014305115, + "SourceGenerator" : 505.8385000228882 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 2, + "milliseconds" : 451.964542, + "peak" : 0.5595703125, + "rms" : 0.10189608270346202, + "stages" : { + "Albert_64" : 1.2719579935073853, + "Alignment" : 0.4070829749107361, + "Decoder_200" : 2.689999997615814, + "NativeSource" : 2.220750004053116, + "PostAlbert" : 2.515082985162735, + "Prosody" : 3.0062499940395355, + "SourceGenerator" : 439.54879200458527 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 3, + "milliseconds" : 474.447791, + "peak" : 0.5595703125, + "rms" : 0.10189608270346202, + "stages" : { + "Albert_64" : 1.3324159979820251, + "Alignment" : 0.3564999997615814, + "Decoder_200" : 2.7335000038146973, + "NativeSource" : 2.378792017698288, + "PostAlbert" : 2.628250002861023, + "Prosody" : 3.142916977405548, + "SourceGenerator" : 461.57045900821686 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 4, + "milliseconds" : 505.083917, + "peak" : 0.5595703125, + "rms" : 0.10189608270346202, + "stages" : { + "Albert_64" : 1.2647500038146973, + "Alignment" : 0.3137499988079071, + "Decoder_200" : 2.6875829994678497, + "NativeSource" : 2.2169169783592224, + "PostAlbert" : 2.4556660056114197, + "Prosody" : 4.21095797419548, + "SourceGenerator" : 491.64208298921585 + } + } + ], + "voice" : "jf_alpha" +} \ No newline at end of file diff --git a/Documentation/TTS/Measurements/KokoroV3/ja-hybrid.json b/Documentation/TTS/Measurements/KokoroV3/ja-hybrid.json new file mode 100644 index 000000000..cd352bc6a --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/ja-hybrid.json @@ -0,0 +1,176 @@ +{ + "inputIds" : [ + 0, + 53, + 57, + 114, + 114, + 51, + 21, + 51, + 75, + 43, + 3, + 16, + 53, + 164, + 57, + 158, + 16, + 75, + 43, + 16, + 51, + 158, + 16, + 62, + 47, + 112, + 53, + 164, + 51, + 16, + 46, + 47, + 61, + 101, + 16, + 56, + 47, + 4, + 0 + ], + "language" : "ja", + "loadMilliseconds" : 1199.288333, + "policy" : "hybrid", + "text" : "こんにちは、今日はいい天気ですね。", + "trials" : [ + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : -2, + "milliseconds" : 94.452625, + "peak" : 0.5419921875, + "rms" : 0.10198565137291829, + "stages" : { + "Albert_64" : 2.4720830023288727, + "Alignment" : 1.076875001192093, + "Decoder_200" : 4.006291002035141, + "NativeSource" : 2.430292010307312, + "PostAlbert" : 20.9755420088768, + "Prosody" : 20.244084000587463, + "SourceGenerator" : 42.83033299446106 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : -1, + "milliseconds" : 36.864458, + "peak" : 0.5419921875, + "rms" : 0.10198565137291829, + "stages" : { + "Albert_64" : 1.2462500035762787, + "Alignment" : 0.6350000202655792, + "Decoder_200" : 2.6949580013751984, + "NativeSource" : 2.2272499799728394, + "PostAlbert" : 2.297084003686905, + "Prosody" : 3.091874986886978, + "SourceGenerator" : 24.40279197692871 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 0, + "milliseconds" : 35.851334, + "peak" : 0.5419921875, + "rms" : 0.10198565137291829, + "stages" : { + "Albert_64" : 1.2448340058326721, + "Alignment" : 0.3157080113887787, + "Decoder_200" : 2.621832996606827, + "NativeSource" : 2.046332985162735, + "PostAlbert" : 2.175583988428116, + "Prosody" : 2.82354199886322, + "SourceGenerator" : 24.382874995470047 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 1, + "milliseconds" : 36.064125, + "peak" : 0.5419921875, + "rms" : 0.10198565137291829, + "stages" : { + "Albert_64" : 1.2053749859333038, + "Alignment" : 0.295540988445282, + "Decoder_200" : 2.645332992076874, + "NativeSource" : 2.15412500500679, + "PostAlbert" : 2.2647919952869415, + "Prosody" : 3.0890829861164093, + "SourceGenerator" : 24.16687497496605 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 2, + "milliseconds" : 36.358667, + "peak" : 0.5419921875, + "rms" : 0.10198565137291829, + "stages" : { + "Albert_64" : 1.2211669981479645, + "Alignment" : 0.27141597867012024, + "Decoder_200" : 2.678167015314102, + "NativeSource" : 2.0985000133514404, + "PostAlbert" : 2.229374974966049, + "Prosody" : 2.8358749747276306, + "SourceGenerator" : 24.763957977294922 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 3, + "milliseconds" : 35.414625, + "peak" : 0.5419921875, + "rms" : 0.10198565137291829, + "stages" : { + "Albert_64" : 1.21670800447464, + "Alignment" : 0.2681249976158142, + "Decoder_200" : 2.73404198884964, + "NativeSource" : 2.2038330137729645, + "PostAlbert" : 2.144457995891571, + "Prosody" : 2.8202910125255585, + "SourceGenerator" : 23.776957988739014 + } + }, + { + "audioSeconds" : 3.05, + "fast" : true, + "frames" : 122, + "iteration" : 4, + "milliseconds" : 35.749834, + "peak" : 0.5419921875, + "rms" : 0.10198565137291829, + "stages" : { + "Albert_64" : 1.1856250166893005, + "Alignment" : 0.28737500309944153, + "Decoder_200" : 2.6525830030441284, + "NativeSource" : 2.210458993911743, + "PostAlbert" : 2.5535829961299896, + "Prosody" : 2.8792499899864197, + "SourceGenerator" : 23.742125004529953 + } + } + ], + "voice" : "jf_alpha" +} \ No newline at end of file diff --git a/Documentation/TTS/Measurements/KokoroV3/live-demo-screenshots.json b/Documentation/TTS/Measurements/KokoroV3/live-demo-screenshots.json new file mode 100644 index 000000000..e418d9f48 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/live-demo-screenshots.json @@ -0,0 +1,28 @@ +{ + "source": "User-supplied live demo screenshots; individual runs, not benchmark medians", + "mode": "One at a time", + "timer": "Public text synthesis through complete PCM; excludes model load, warmup, WAV and playback", + "rows": [ + { + "language": "en", + "legacyMs": 149.8, + "v3Ms": 79.8, + "displayedSpeedup": 1.88, + "v3AudioSeconds": 3.35 + }, + { + "language": "ja", + "legacyMs": 305.9, + "v3Ms": 57.8, + "displayedSpeedup": 5.29, + "v3AudioSeconds": 3.05 + }, + { + "language": "zh", + "legacyMs": 161.7, + "v3Ms": 71.9, + "displayedSpeedup": 2.25, + "v3AudioSeconds": 3.67 + } + ] +} diff --git a/Documentation/TTS/Measurements/KokoroV3/status.json b/Documentation/TTS/Measurements/KokoroV3/status.json new file mode 100644 index 000000000..9275f7ff4 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/status.json @@ -0,0 +1,74 @@ +[ + { + "language": "en", + "policy": "hybrid", + "exitCode": 0, + "elapsedSeconds": 18.455811833 + }, + { + "language": "en", + "policy": "cpuAndNeuralEngine", + "exitCode": 0, + "elapsedSeconds": 23.919849584 + }, + { + "language": "en", + "policy": "all", + "exitCode": -6, + "elapsedSeconds": 12.775751 + }, + { + "language": "en", + "policy": "cpuAndGPU", + "exitCode": -6, + "elapsedSeconds": 3.9040888749999993 + }, + { + "language": "ja", + "policy": "cpuAndNeuralEngine", + "exitCode": 0, + "elapsedSeconds": 37.63165875 + }, + { + "language": "ja", + "policy": "all", + "exitCode": -6, + "elapsedSeconds": 12.419991332999999 + }, + { + "language": "ja", + "policy": "cpuAndGPU", + "exitCode": -6, + "elapsedSeconds": 2.4310372080000064 + }, + { + "language": "ja", + "policy": "hybrid", + "exitCode": 0, + "elapsedSeconds": 1.5749986250000063 + }, + { + "language": "zh", + "policy": "all", + "exitCode": -6, + "elapsedSeconds": 11.817329125 + }, + { + "language": "zh", + "policy": "cpuAndGPU", + "exitCode": -6, + "elapsedSeconds": 3.205860042000012 + }, + { + "language": "zh", + "policy": "hybrid", + "exitCode": 0, + "elapsedSeconds": 14.703979167 + }, + { + "language": "zh", + "policy": "cpuAndNeuralEngine", + "exitCode": 0, + "elapsedSeconds": 22.87929062500001 + } +] \ No newline at end of file diff --git a/Documentation/TTS/Measurements/KokoroV3/summary.csv b/Documentation/TTS/Measurements/KokoroV3/summary.csv new file mode 100644 index 000000000..335db8f49 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/summary.csv @@ -0,0 +1,13 @@ +language,policy,status,medianMs,minMs,maxMs,audioSeconds,peak +en,hybrid,passed,49.449666,40.515875,66.98,3.35,0.5302734375 +en,cpuAndNeuralEngine,passed,511.400542,464.735209,526.832541,3.35,0.544921875 +en,all,failed (-6),,,,, +en,cpuAndGPU,failed (-6),,,,, +ja,cpuAndNeuralEngine,passed,474.447791,451.964542,522.908959,3.05,0.5595703125 +ja,all,failed (-6),,,,, +ja,cpuAndGPU,failed (-6),,,,, +ja,hybrid,passed,35.851334,35.414625,36.358667,3.05,0.5419921875 +zh,all,failed (-6),,,,, +zh,cpuAndGPU,failed (-6),,,,, +zh,hybrid,passed,53.97225,45.457584,58.523583,3.675,0.6953125 +zh,cpuAndNeuralEngine,passed,584.190834,505.655042,596.269708,3.675,0.681640625 diff --git a/Documentation/TTS/Measurements/KokoroV3/zh-all-crash.txt b/Documentation/TTS/Measurements/KokoroV3/zh-all-crash.txt new file mode 100644 index 000000000..eb4b92182 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/zh-all-crash.txt @@ -0,0 +1,26 @@ +LOAD_MODEL KokoroAlbert units=2 +LOAD_MODEL KokoroPostAlbert units=2 +LOAD_MODEL KokoroAlignment units=2 +LOAD_MODEL KokoroProsody_v2 units=2 +LOAD_MODEL KokoroNoise_v2 units=2 +LOAD_MODEL KokoroVocoder units=2 +LOAD_MODEL KokoroTail_v2 units=2 +LOAD_MODEL KokoroAlbert_32 units=2 +LOAD_MODEL KokoroAlbert_64 units=2 +LOAD_MODEL KokoroDecoderPre_120 units=2 +LOAD_MODEL KokoroDecoderPre_200 units=2 +LOAD_MODEL KokoroSourceGenerator units=2 +START zh all -2 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +PREDICT Decoder_200 +PREDICT SourceGenerator +DONE zh all -2 ms=1197.009 frames=147 peak=0.69775390625 +START zh all -1 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported' diff --git a/Documentation/TTS/Measurements/KokoroV3/zh-cpuAndGPU-crash.txt b/Documentation/TTS/Measurements/KokoroV3/zh-cpuAndGPU-crash.txt new file mode 100644 index 000000000..e3e018d53 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/zh-cpuAndGPU-crash.txt @@ -0,0 +1,26 @@ +LOAD_MODEL KokoroAlbert units=1 +LOAD_MODEL KokoroPostAlbert units=1 +LOAD_MODEL KokoroAlignment units=1 +LOAD_MODEL KokoroProsody_v2 units=1 +LOAD_MODEL KokoroNoise_v2 units=1 +LOAD_MODEL KokoroVocoder units=1 +LOAD_MODEL KokoroTail_v2 units=1 +LOAD_MODEL KokoroAlbert_32 units=1 +LOAD_MODEL KokoroAlbert_64 units=1 +LOAD_MODEL KokoroDecoderPre_120 units=1 +LOAD_MODEL KokoroDecoderPre_200 units=1 +LOAD_MODEL KokoroSourceGenerator units=1 +START zh cpuAndGPU -2 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +PREDICT Decoder_200 +PREDICT SourceGenerator +DONE zh cpuAndGPU -2 ms=331.677875 frames=147 peak=0.71044921875 +START zh cpuAndGPU -1 +PREDICT Albert_64 +PREDICT PostAlbert +PREDICT Alignment +PREDICT Prosody +/AppleInternal/Library/BuildRoots/4~CV2iugCS017GRPdEjj32gfBJXr8YaxAcCWYlUuw/Library/Caches/com.apple.xbs/TemporaryDirectory.WK6jPd/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Runtimes/MPSRuntime/Operations/GPURNNOps.mm:1565: failed assertion `JIT not supported' diff --git a/Documentation/TTS/Measurements/KokoroV3/zh-cpuAndNeuralEngine.json b/Documentation/TTS/Measurements/KokoroV3/zh-cpuAndNeuralEngine.json new file mode 100644 index 000000000..103577986 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/zh-cpuAndNeuralEngine.json @@ -0,0 +1,173 @@ +{ + "inputIds" : [ + 0, + 73, + 127, + 172, + 88, + 108, + 169, + 3, + 88, + 161, + 171, + 152, + 172, + 95, + 144, + 169, + 153, + 173, + 30, + 121, + 169, + 40, + 127, + 173, + 137, + 169, + 151, + 171, + 88, + 140, + 172, + 94, + 124, + 172, + 4, + 0 + ], + "language" : "zh", + "loadMilliseconds" : 18777.163208, + "policy" : "cpuAndNeuralEngine", + "text" : "你好，欢迎使用本地语音合成。", + "trials" : [ + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : -2, + "milliseconds" : 688.458708, + "peak" : 0.681640625, + "rms" : 0.0803981773148257, + "stages" : { + "Albert_64" : 6.062207996845245, + "Alignment" : 1.4964579939842224, + "Decoder_200" : 8.78387501835823, + "NativeSource" : 4.0372500121593475, + "PostAlbert" : 7.805417001247406, + "Prosody" : 6.40616700053215, + "SourceGenerator" : 653.0665829777718 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : -1, + "milliseconds" : 506.346041, + "peak" : 0.681640625, + "rms" : 0.0803981773148257, + "stages" : { + "Albert_64" : 1.4184580147266388, + "Alignment" : 1.7653749883174896, + "Decoder_200" : 2.7662919759750366, + "NativeSource" : 3.070792019367218, + "PostAlbert" : 2.392707973718643, + "Prosody" : 4.3141250014305115, + "SourceGenerator" : 490.29012501239777 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 0, + "milliseconds" : 508.036792, + "peak" : 0.681640625, + "rms" : 0.0803981773148257, + "stages" : { + "Albert_64" : 1.3501249849796295, + "Alignment" : 0.3272089958190918, + "Decoder_200" : 2.6805000007152557, + "NativeSource" : 2.6849160194396973, + "PostAlbert" : 2.2730830013751984, + "Prosody" : 3.485042005777359, + "SourceGenerator" : 494.90404200553894 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 1, + "milliseconds" : 505.655042, + "peak" : 0.681640625, + "rms" : 0.0803981773148257, + "stages" : { + "Albert_64" : 1.2982079982757568, + "Alignment" : 0.37587499618530273, + "Decoder_200" : 2.689083009958267, + "NativeSource" : 2.622833013534546, + "PostAlbert" : 2.4422909915447235, + "Prosody" : 3.458083003759384, + "SourceGenerator" : 492.45950001478195 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 2, + "milliseconds" : 592.3945, + "peak" : 0.681640625, + "rms" : 0.0803981773148257, + "stages" : { + "Albert_64" : 1.2698749899864197, + "Alignment" : 0.33283400535583496, + "Decoder_200" : 2.7228749990463257, + "NativeSource" : 2.761209011077881, + "PostAlbert" : 2.2566670179367065, + "Prosody" : 3.4774169921875, + "SourceGenerator" : 579.1142919957638 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 3, + "milliseconds" : 584.190834, + "peak" : 0.681640625, + "rms" : 0.0803981773148257, + "stages" : { + "Albert_64" : 1.4952499866485596, + "Alignment" : 0.660042017698288, + "Decoder_200" : 2.846334010362625, + "NativeSource" : 3.9126660227775574, + "PostAlbert" : 4.417042016983032, + "Prosody" : 7.867041975259781, + "SourceGenerator" : 562.4224589765072 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 4, + "milliseconds" : 596.269708, + "peak" : 0.681640625, + "rms" : 0.0803981773148257, + "stages" : { + "Albert_64" : 1.2270419895648956, + "Alignment" : 0.3241669833660126, + "Decoder_200" : 2.6298750042915344, + "NativeSource" : 2.7687910199165344, + "PostAlbert" : 2.2313750088214874, + "Prosody" : 3.408999979496002, + "SourceGenerator" : 583.4025420248508 + } + } + ], + "voice" : "zf_001" +} \ No newline at end of file diff --git a/Documentation/TTS/Measurements/KokoroV3/zh-hybrid.json b/Documentation/TTS/Measurements/KokoroV3/zh-hybrid.json new file mode 100644 index 000000000..c639f3147 --- /dev/null +++ b/Documentation/TTS/Measurements/KokoroV3/zh-hybrid.json @@ -0,0 +1,173 @@ +{ + "inputIds" : [ + 0, + 73, + 127, + 172, + 88, + 108, + 169, + 3, + 88, + 161, + 171, + 152, + 172, + 95, + 144, + 169, + 153, + 173, + 30, + 121, + 169, + 40, + 127, + 173, + 137, + 169, + 151, + 171, + 88, + 140, + 172, + 94, + 124, + 172, + 4, + 0 + ], + "language" : "zh", + "loadMilliseconds" : 14134.340333, + "policy" : "hybrid", + "text" : "你好，欢迎使用本地语音合成。", + "trials" : [ + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : -2, + "milliseconds" : 112.47575, + "peak" : 0.6953125, + "rms" : 0.08059115548018292, + "stages" : { + "Albert_64" : 2.6017079949378967, + "Alignment" : 1.095124989748001, + "Decoder_200" : 4.570374995470047, + "NativeSource" : 3.9683749973773956, + "PostAlbert" : 5.242749989032745, + "Prosody" : 8.480749994516373, + "SourceGenerator" : 85.87908300757408 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : -1, + "milliseconds" : 46.404708, + "peak" : 0.6953125, + "rms" : 0.08059115548018292, + "stages" : { + "Albert_64" : 1.3400420248508453, + "Alignment" : 0.901625007390976, + "Decoder_200" : 2.9789159893989563, + "NativeSource" : 3.893875002861023, + "PostAlbert" : 2.991084009408951, + "Prosody" : 4.288249999284744, + "SourceGenerator" : 29.589040994644165 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 0, + "milliseconds" : 58.523583, + "peak" : 0.6953125, + "rms" : 0.08059115548018292, + "stages" : { + "Albert_64" : 1.3299579918384552, + "Alignment" : 0.5920830070972443, + "Decoder_200" : 2.8278749883174896, + "NativeSource" : 3.6107909977436066, + "PostAlbert" : 3.5314579904079437, + "Prosody" : 7.614499986171722, + "SourceGenerator" : 38.51012501120567 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 1, + "milliseconds" : 45.457584, + "peak" : 0.6953125, + "rms" : 0.08059115548018292, + "stages" : { + "Albert_64" : 1.483916014432907, + "Alignment" : 0.6162090003490448, + "Decoder_200" : 2.8970830142498016, + "NativeSource" : 3.356750011444092, + "PostAlbert" : 2.9107919931411743, + "Prosody" : 4.826292008161545, + "SourceGenerator" : 28.890082985162735 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 2, + "milliseconds" : 52.730167, + "peak" : 0.6953125, + "rms" : 0.08059115548018292, + "stages" : { + "Albert_64" : 1.690750002861023, + "Alignment" : 0.6295419931411743, + "Decoder_200" : 3.028082996606827, + "NativeSource" : 4.10979101061821, + "PostAlbert" : 4.600250005722046, + "Prosody" : 9.069750010967255, + "SourceGenerator" : 29.027415990829468 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 3, + "milliseconds" : 54.476167, + "peak" : 0.6953125, + "rms" : 0.08059115548018292, + "stages" : { + "Albert_64" : 1.4238329827785492, + "Alignment" : 1.029875010251999, + "Decoder_200" : 3.1336670219898224, + "NativeSource" : 4.337332993745804, + "PostAlbert" : 3.971457988023758, + "Prosody" : 7.756166994571686, + "SourceGenerator" : 32.09708300232887 + } + }, + { + "audioSeconds" : 3.675, + "fast" : true, + "frames" : 147, + "iteration" : 4, + "milliseconds" : 53.97225, + "peak" : 0.6953125, + "rms" : 0.08059115548018292, + "stages" : { + "Albert_64" : 1.4665409922599792, + "Alignment" : 1.156125009059906, + "Decoder_200" : 3.1017920076847076, + "NativeSource" : 4.87679198384285, + "PostAlbert" : 3.976042002439499, + "Prosody" : 5.844582974910736, + "SourceGenerator" : 32.71441701054573 + } + } + ], + "voice" : "zf_001" +} \ No newline at end of file diff --git a/README.md b/README.md index 4e4d80b49..7f4b20225 100644 --- a/README.md +++ b/README.md @@ -677,27 +677,37 @@ for the full language table. ### KokoroAne -ANE-resident Kokoro 82M (4-stage on Neural Engine, 3-stage on GPU). Yields -3-11× RTFx on Apple Silicon vs. the prior single-graph Kokoro path. English -(`af_heart`) and Mandarin variants ship with a built-in G2P pipeline (BART -CoreML for English OOV, jieba + sandhi + G2pW for Mandarin). +Kokoro ANE v3 is available in **v0.17.6** for English, Japanese and Mandarin, +with native text frontends. Opt in with `version: .v3`; existing callers keep +`.legacy`. V3 requires macOS 15 / iOS 18 and combines CPU/ANE model stages, +a CPU/GPU generator and native Accelerate DSP. It is not ANE-only or `.all`. ```swift import FluidAudio -Task { - let manager = KokoroAneManager() - try await manager.initialize() - let samples = try await manager.synthesize(text: "Hello from FluidAudio.") - // `samples` is 24 kHz mono Float32 PCM -} +let manager = KokoroAneManager(version: .v3) +try await manager.initialize() +let result = try await manager.synthesizeDetailed(text: "Hello from FluidAudio.") +// result.samples: 24 kHz mono Float32 PCM + +let japanese = KokoroAneManager(variant: .japanese, version: .v3) +let mandarin = KokoroAneManager(variant: .mandarin, version: .v3) ``` ```bash -swift run fluidaudiocli tts "Hello from FluidAudio." --backend kokoroAne --output out.wav +swift run -c release fluidaudiocli tts "Hello from FluidAudio." \ + --backend kokoro-ane --kokoro-version v3 --output out.wav ``` -Model assets are cached under `~/.cache/fluidaudio/Models/kokoro/`. +Selected M5 Pro demo inputs produced 3.05–3.68 seconds of audio in +**36–54 ms median inference**, excluding text processing and loading. These +are bounded local checks; see [benchmark methodology and live-demo timings](Documentation/TTS/Benchmarks.md#kokoro-ane-v3-selected-m5-pro-measurements). +Models compile/cache on first use. Physical iOS inference remains unvalidated. +See [setup, voices, model pins, caches and platform limits](Documentation/TTS/KokoroAne.md). + +Base weights: [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) +(English/Japanese) and [hexgrad/Kokoro-82M-v1.1-zh](https://huggingface.co/hexgrad/Kokoro-82M-v1.1-zh) +(Mandarin). Core ML assets: [FluidInference/kokoro-82m-coreml](https://huggingface.co/FluidInference/kokoro-82m-coreml/tree/main/ANE-v3). ## Continuous Integration