Hugging Face's logo

FluidInference
/
kokoro-82m-coreml

Text-to-Speech
Core ML
English
Japanese
Chinese
Model card Files
xet
 Community
3
kokoro-82m-coreml / ANE-v3 /README.md
alexwengg's picture
alexwengg
docs: document FluidAudio v0.17.6 routing, latency and profiling
d93c5f3 verified
preview code
|
Raw History Blame Contribute Delete
10.3 kB
metadata 
license: apache-2.0
language:
  - en
base_model:
  - hexgrad/Kokoro-82M
pipeline_tag: text-to-speech
tags:
  - coreml
  - kokoro
  - apple-silicon
  - ane

Kokoro ANE v3 — hybrid ANE + GPU

An opt-in Kokoro Core ML pipeline (English in this directory; Japanese in ja/, Mandarin in zh/) for macOS 15 / iOS 18 or newer. It combines fixed quantized Albert models, a masked ANE decoder, deterministic native Accelerate source/STFT, and Matt Mireles's optimized GPU generator adapted to our parameters. It is a hybrid implementation; the generator is GPU-based. Existing model paths in this repository remain available to older clients.

The base model and pretrained weights are hexgrad/Kokoro-82M, created by hexgrad. This variant converts and optimizes those weights for Core ML.

FluidAudio SDK v0.17.6

Use KokoroAneManager(version: .v3) for English, KokoroAneManager(variant: .japanese, version: .v3) for Japanese, or KokoroAneManager(variant: .mandarin, version: .v3) for Mandarin. All three accept native text and variant voices. Existing callers retain .legacy. See SDK usage, voices and platform limits. Requires macOS 15 / iOS 18; physical iOS inference remains unvalidated.

The SDK pins model release 6c0750dd02ef9d981fb82f147d784a3f171755ee and base revision 006395f65025af251858b1ab0a7178a6a1e73f9f, adopting fp32 KokoroProsody_v2 and COLA-corrected KokoroTail_v2. The bundled standalone conversion runtime and its historical measurements below are separate from this SDK integration. This documentation update does not change model bytes or pins.

SDK compute routing and selected measurements

Hybrid is not .all or ANE-only. Albert, PostAlbert, Alignment, Prosody and masked decoder request .cpuAndNeuralEngine; the fast generator requests .cpuAndGPU; native source/STFT runs on CPU/Accelerate. Fallback Vocoder requests CPU+ANE, Noise/Tail CPU+GPU. Core ML chooses actual placement within each policy.

Policy English Japanese Mandarin
Current hybrid 49.45 ms 35.85 ms 53.97 ms
CPU+ANE throughout 511.40 ms 474.45 ms 584.19 ms
.all throughout SIGABRT SIGABRT SIGABRT
CPU+GPU throughout SIGABRT SIGABRT SIGABRT

Apple M5 Pro, 24 GiB, macOS 27.0 (26A428); October 7, 2026 (Toronto). Median of five calls after two warmups on one selected demo input per language; 3.350 / 3.050 / 3.675 seconds of output. Timer covers prepared tokens/style to complete PCM, excluding text processing, loading and WAV IO. Identical inputs per policy, serial isolated processes, native DSP unchanged. These are local exploratory checks, not corpus or cross-device results. Global overrides were applied in a local harness; the public v3 API retains its fixed hybrid routing. Both .all and CPU+GPU aborted during Prosody on the second warmup with Apple's GPURNNOps: JIT not supported. CPU+ANE permits CPU fallback; it is not ANE-only.

Live-demo screenshots instead time raw text through complete audio and showed 57.8–79.8 ms in selected serial runs. Their 1.88× / 5.29× / 2.25× speedups versus legacy are individual observations, not a benchmark average. Concurrent demo mode also includes shared-resource contention. See inputs, ranges, protocol and raw records.

Compute plans prefer ANE for static Albert/decoder and GPU for the generator, but plan estimates and system-wide asitop power samples do not establish actual per-process execution. See ANE profiling and telemetry limits.

Initial standalone Japanese and Mandarin extension

Japanese and Mandarin bundles include their voices, vocabularies, conversion sources and real WAV samples. Use --language ja or --language zh with the downloader and the commit containing those bundles. They measured 24–28 ms on four short M5 Pro inputs versus 111–138 ms baseline.

The shared native STFT now uses Accelerate SGEMM to match reference convolution summation. English model bytes are unchanged. With this runtime, English warm medians are 22.36 / 32.72 ms, versus 98.15 / 151.94 ms baseline; isolated native-source/generator relative RMSE is 0.07246 / 0.07080, correlation 0.99739 / 0.99750. Its five real-input integration checks still pass. The comparison and fidelity figures below describe the initial English release.

Initial English latency evidence

Warm synthesis Published seven-stage baseline This variant Matt Mireles, M5 routing fallback
Hello world, 1.625 s output 96.54 ms 22.05 ms 27.47 ms
Fox sentence, 3 s output 151.19 ms 32.99 ms 36.01 ms

Apple M5 Pro, 24 GiB RAM, macOS 27.0; optimized Swift; three discarded warmups and five measurements per input/model with rotating serial model order. Timer: tokens/style through complete 24 kHz PCM, excluding G2P, model loading and file IO. Matt's duration and pitch routing was changed to CPU+ANE because its default CPU+GPU execution aborted on this host. These two-clip developer-machine results are exploratory; they do not establish performance on other devices or lengths.

Contents

  • fast/: Albert at 32/64 tokens, fused source/generator/iSTFT, native DSP parameters and real validation fixtures.
  • decoder/: masked decoder models at 120/200 acoustic frames (3/5 seconds).
  • .mlpackage source models and .mlmodelc compiled models are both included. Compiled models were built on macOS 27.0; use source packages and compile on the deployment platform when needed.
  • runtime/: standalone SwiftPM library, sanity checker and bounded benchmark.
  • conversion/: the exporters and validation code used for this variant.
  • validation/: raw timing/fidelity evidence and selected extra fixtures.
  • samples/hello.wav, samples/fox.wav: actual output from the validated runtime.
  • manifest.json: SHA-256/size manifest and pinned fallback dependency.

The downloader retrieves the seven original fallback models from this same repository at revision 006395f65025af251858b1ab0a7178a6a1e73f9f. Requests beyond five seconds use that original vocoder instead of being truncated. Requests above 64 tokens use the original dynamic Albert. The runtime accepts at most 2,000 predicted acoustic frames. Inputs are token IDs and a 256-value voice style; text normalization, G2P and voice selection belong to the caller.

Download and run

Install Python with huggingface_hub and Apple's Swift/Core ML tools. Pin the 40-character Hugging Face commit containing this release as KOKORO_REVISION. After downloading this ANE-v3 directory, run from it:

python tools/download-fast-pipeline.py \
  --revision "$KOKORO_REVISION" --output-dir assets
swift run --package-path runtime -c release kokoro-fast-check \
  assets/base assets/fast assets/decoder \
  assets/validation/kokoro-fast-extra-fixtures.json

# Two selected fixtures; 3 warmups + 5 recorded calls per fixture.
swift run --package-path runtime -c release kokoro-fast-benchmark \
  assets/base assets/fast assets/decoder assets/fast/fixtures.json > latency.json

Use --format mlpackage in the downloader to get source packages instead of precompiled models. The Swift runtime compiles packages when no compiled model is present. Numerical validation additionally requires coremltools 9.0, numpy and soundfile:

python tools/validate-fast-pipeline.py --models-dir assets/fast

The checker emits the native spectra and waveform files consumed by that check. Cache KokoroFastPipeline(baseDirectory:fastDirectory:decoderDirectory:) and call synthesize(inputIds:style:speed:). Its result contains 24 kHz PCM, stage timings and the usedFastVocoder flag. Each instance serializes synthesis calls.

Fidelity and scope

With reference decoder features held fixed, native source plus the adapted GPU graph has waveform correlation 0.99789 / 0.99848 and relative RMSE 0.06489 / 0.05506 on the two clips. Full-chain correlation versus the published baseline is 0.95688 / 0.92139; log1p-mel correlation is 0.99844 / 0.99872. The isolated score is not a full-chain or perceptual equivalence claim. Full-chain correlation versus the unquantized PyTorch reference is approximately 0.75.

Real-input checks cover both measured clips, “Good morning,” a slowed fox sentence using the 200-frame decoder, and a six-second repeated sentence using the original fallback. Invalid inputs are rejected. A pinned Whisper base.en smoke check correctly transcribed six selected baseline/candidate clips without a supplied transcript prompt; it is not a formal quality evaluation.

Provenance and licenses

  • Kokoro weights/voice: hexgrad/Kokoro-82M, revision f3ff3571791e39611d31c381e3a41a3af07b4987 (Apache-2.0).
  • Published quantized Albert: FluidInference/kokoro-82m-coreml, revision 006395f65025af251858b1ab0a7178a6a1e73f9f.
  • Adapted GPU graph: mattmireles/kokoro-coreml, revision 9183566f294ca3f31a63fd2767cf5848bfebfd50 (Apache-2.0).
  • Source inspiration: Matt's repository.
  • Original seven-stage conversion: laishere/kokoro-coreml (MIT).

The adapter checks 202 convolution/linear parameters, replaces 23 small folded-bias differences and two inverse-STFT kernels with our export's values, and preserves the optimized upstream graph. Per-parameter hashes and changed constant names are recorded in fast/generator-provenance.json. This is an explicit upstream graph adaptation. Retain NOTICE, mattmireles-LICENSE and conversion-LICENSE when redistributing it.