Speech synthesis
A text spoken into a sound: a language model writes audio codes from
the text, a vocoder turns the codes into a waveform. This page is the
audit of the published files OKLM runs (OuteTTS 0.2 and 0.3 with
WavTokenizer's decoder, and Soprano with its own), what
oklm speak and include/oklm/mm/speech.h do
with them, the comparison with llama.cpp, and the limits. Speech
recognition, the other way round, is in multimodal.md.
Status: implemented on the CPU: oklm speak, the library
(oklm_speech_*, oklm_vocoder_*), the
catalogue's speech category with
outetts-0.2-500m, outetts-0.3-500m and
soprano-1.1-80m, the server's route, the dashboard's voice
and the chat window's (below); a speed, speaker files chosen by name on
a server, FLAC files. English, one built-in voice. Soprano is the model that speaks faster than it is
heard on a CPU (3 to 4 seconds of sound a second where OuteTTS makes
0.2, a first sound in under a second), with one fixed voice at 32 kHz;
the sections before its own describe OuteTTS. Not there: a device path
for the vocoder, other speech models, MP3 and Opus.
Using it
oklm models pull outetts-0.2-500m # 636 MiB, its licence asked (non-commercial)
oklm speak -m outetts-0.2-500m "Hello world, this is a test of speech synthesis."
oklm models pull outetts-0.3-500m # the same size; a licence that allows commercial use
oklm speak -m outetts-0.3-500m "Hello world, this is a test of speech synthesis."
oklm speak -m outetts-0.2-500m -o chapter.wav < chapter.txt
oklm speak -m OuteTTS-0.2-500M-Q8_0.gguf --vocoder WavTokenizer-Large-75-F16.gguf --voice speaker.json "Good morning."
oklm speak --vocoder WavTokenizer-Large-75-F16.gguf --codes codes.txt -o codes.wav
oklm speak -m outetts-0.2-500m --speed 1.25 --first-words 5 -o reply.flac "A first sound sooner, a little faster, in a FLAC file."
oklm models pull soprano-1.1-80m # 197 MiB, Apache-2.0: faster than real time on a CPU, one fixed voice
oklm speak -m soprano-1.1-80m "It costs $2.47 and arrives on Monday, Dr. Smith."
oklm speak writes a mono 16-bit WAV file at 24 kHz (32
kHz with Soprano, which takes the same options
but --voice, --prompt-words and
--no-guide; -o, speech.wav by
default) from the text given, or read from standard input.
-m takes the language model's GGUF file or the catalogue's
name, whose vocoder is then found by itself; with a file,
--vocoder names the vocoder. --voice FILE
replaces the built-in voice by a speaker file of OuteTTS (below).
--temp, --top-k, --repeat-penalty
and --seed set how the codes are sampled,
--no-guide leaves each word's first token to the model,
--words N sets how many words one pass of the model speaks
(24), --prompt-words N how many the model speaks from one
prompt (64: the passes that hold in it share the prompt),
--first-words N cuts the first pass shorter for an earlier
first sound, --speed S (0.25 to 4) changes the pace and
keeps the pitch (passes,
prompts and the first sound, speed). A name
ending in .flac given to -o writes a FLAC
file, the same samples compressed without loss (FLAC). --codes FILE skips the language
model and gives the vocoder audio codes read from a file (integers, 75 a
second). --backend cpu|opencl|vulkan|auto says where the
language model computes (cpu by default); the vocoder's
network follows it there (on a device).
A server speaks for its clients with the same models:
oklm serve --speak-model outetts-0.2-500m answers
POST /v1/audio/speech (OpenAI's route) with a WAV file, or
with raw samples sent pass by pass while they are made (server.md). The server's dashboard plays
them: a Speak button on each reply of its chat, /speak, and
a switch that reads every reply aloud, sentence by sentence while it is
written (server.md);
with a transcription model beside it, the dashboard holds a spoken
conversation, the microphone listened to until a pause and the reply
interrupted by speaking (server.md).
A request names its voice among those the server offers (the one it was
started with, the built-in one, the speaker files of its voice
directories and of the model's folder:
GET /v1/audio/voices), its speed, and
wav, pcm or flac. The chat window
(oklm-gui) has the same three, and plays the samples
through the system's sound device: from its own server started with
--speak-model, from a server at an address that has a
voice, or with the library in the window itself, the model loaded at the
first reading (gui.md). The
window reads a reply while it is written, a sentence a call, and chooses
among speaker files (~/.config/oklm/voices/*.json); with
its Talk box it holds a spoken conversation, the microphone listened to
until a pause and the reply interrupted by speaking (gui.md).
While it runs the command says how much is spoken, then how fast it was:
$ oklm speak -m outetts-0.2-500m -o cat.wav "The catalogue name finds both files."
[2.6 s of speech after 6.0 s]
first sound after 6.00 s; 1 prompt of 901 tokens (0 kept in the cache) in 1.21 s, 262 tokens written in 4.61 s, the vocoder 0.18 s
cat.wav: 2.60 s of speech in 6.00 s (0.43 s of sound per second)
$ oklm transcribe -m whisper-base cat.wav
The catalog name finds both files.
The second line is oklm_speech_last_stats(): when the
first pass was handed over, the prompts evaluated and their tokens,
those the context's cache already held, the tokens the model wrote, and
the time of each part.
The two files
Those llama.cpp's tts example names
(--tts-oute-default):
| Language model | Vocoder | |
|---|---|---|
| Repository | OuteAI/OuteTTS-0.2-500M-GGUF |
ggml-org/WavTokenizer (converted from
novateur/WavTokenizer-large-speech-75token) |
| File | OuteTTS-0.2-500M-Q8_0.gguf |
WavTokenizer-Large-75-F16.gguf |
| Size | 536,666,752 bytes | 130,186,688 bytes |
| SHA-256 | 2b23a91e20202f1e4a5904078c69a01bc8a404b8b0f8477c0f7a0c61eb6ac23a |
2356baa8631cc2995ea3465196a017a2733600d849a91180c0f97fa7fb375bbe |
general.architecture |
qwen2 (Qwen2.5-0.5B retrained) |
wavtokenizer-dec |
| Shape | 24 blocks, width 896, 14 heads over 2 key/value heads, FFN 4864, RoPE base 1e6, context 32768 | code width 512, stacks of width 768, FFN 2304, output rows of 1282, context 8192 |
| Tensors | 290 (499 M parameters; Q8_0, the norms and biases F32) | 161 (matrices and convolutions F16, norms and biases F32) |
| Vocabulary | 157,696 tokens, byte-level BPE with the qwen2
pre-tokenizer, no BOS added, EOS <|im_end|>
(151645) |
4096 codes (tokenizer.ggml.model =
none) |
| Licence | CC-BY-NC-4.0 (the model card; the catalogue asks for agreement) | MIT (general.license, and the source repository's
card) |
Other quantizations of the language model are published in the same
repository, from Q2_K (344 MB) to F16 (1.0 GB); Q4_K_M is 403 MB. The
vocoder also exists as Q5_1 (73 MB), which OKLM's vocoder does not read
(F32 and F16 only). The language model needs nothing new: OKLM's
qwen2 runtime loads it (oklm info reads it,
oklm run generates from it), and the audio tokens are
ordinary vocabulary entries.
OuteTTS 0.3 is the catalogue's
outetts-0.3-500m, with the same vocoder file:
| Language model | |
|---|---|
| Repository, file | OuteAI/OuteTTS-0.3-500M-GGUF,
OuteTTS-0.3-500M-Q8_0.gguf |
| Size, SHA-256 | 536,666,880 bytes,
e6cb54453b2de1eaee85bd46b67de9c8cdd3c2dfe117cc3e3da794ddc64dad85 |
| Shape | the same qwen2 model: 24 blocks, width 896, 290
tensors, 499 M parameters, Q8_0 |
| Vocabulary | 157,696 tokens: 0.2's (<|text_sep|>,
<|code_start|>, <|code_end|> and
the codes keep their ids), with <|space|> (156777)
and punctuation marks (<|period|> 156779,
<|comma|> 156782, ...) among the entries 0.2 left
unused |
tokenizer.chat_template |
outetts-0.3 (what llama.cpp's example reads the version
from) |
| Licence | CC-BY-SA-4.0 (the model card): commercial use allowed, changes shared alike |
The text and the prompt
The text is normalized as OuteTTS's prompt processor does, with
numbers spelled as llama.cpp's tts example spells them (OuteTTS itself
uses the inflect package; romanization of Chinese, Japanese
and Korean with uroman and MeCab is in neither llama.cpp
nor OKLM):
- each run of digits, with its decimals when a dot and digits follow,
is spelled in English:
42isforty two,3.14isthree point one four,1234567isone million two hundred thirty four thousand five hundred sixty seven; above 2,147,483,647 the digits are read one by one (llama.cpp drops such a number). Nothing separates the spelled number from letters around it:2ndistwond, and1,000isone zero, in both; - upper case becomes lower case (ASCII only);
-,_,/,,,.and\become spaces;- every character that is not a letter
atozor white space is dropped, accented letters included (caféiscaf); - the words are separated by
<|text_sep|>.
The prompt of one pass, the voice being a recording's words with the duration and the audio codes of each:
<|im_start|>
<|text_start|>the<|text_sep|>overall<|text_sep|>...<|text_sep|>lovely<|text_sep|>hello<|text_sep|>world<|text_end|>
<|audio_start|>
the<|t_0.08|><|code_start|><|257|><|740|><|636|><|913|><|788|><|1703|><|code_end|>
overall<|t_0.36|><|code_start|><|127|><|201|>...<|code_end|>
...
lovely<|t_0.56|><|code_start|><|634|>...<|1677|><|code_end|>
The voice's words come first in the text, then the words to speak;
the audio part holds the voice's words only, and the model continues it:
a line feed, then each word of the text as its letters, a duration token
<|t_S.SS|>, <|code_start|>, its
codes, <|code_end|>, a line feed between words, and
<|audio_end|>, a line feed and
<|im_end|> after the last. The model was trained on
exactly this layout (get_training_prompt() of OuteTTS's
prompt_processor.py).
OuteTTS 0.3's layout separates the words by
<|space|>, which also ends a word's codes, and has no
<|code_start|>:
<|im_start|>
<|text_start|>the<|space|>overall<|space|>...<|space|>lovely<|space|>hello<|space|>world<|text_end|>
<|audio_start|>
the<|t_0.08|><|257|><|740|><|636|><|913|><|788|><|1703|><|space|>
overall<|t_0.36|><|127|><|201|>...<|space|>
OKLM reads the layout from the vocabulary (a model that has the token
<|space|> is 0.3; oklm_speech_format()),
builds the prompt the way llama.cpp's example derives it from 0.2's
(<|text_sep|> becomes <|space|>,
<|code_start|> goes, <|code_end|>
becomes <|space|>), and counts a word written at
<|space|>. The same voice, the same normalization,
the same sampling and guide. OuteTTS's own prompt processor for 0.3
keeps the punctuation as tokens (<|period|>, ...);
llama.cpp's example does not, and neither does OKLM: no punctuation
reaches the model. The prompt is 30 tokens shorter (858 before the first
word of the text: no <|code_start|> for the voice's
30 words).
The voice. The built-in one is
en_male_1, a default speaker of OuteTTS, as llama.cpp's
example embeds it: 30 words ("the overall package from just two people
is pretty remarkable ..."), 658 codes, 8.8 seconds. It is compiled in
(src/mm/speech_voice.inc). --voice (and
oklm_speech_params.voice) reads a speaker file of OuteTTS
instead: a JSON object with words, an array of
{"word": "...", "duration": 0.24, "codes": [..]}, as
interface.save_speaker() of the outetts
package writes it. Making such a file from a recording needs
WavTokenizer's encoder and a word aligner, which are not here: the
outetts package does it. A voice costs its codes in every
prompt: the built-in one makes the prompt 888 tokens long before the
first word of the text.
The tokens. Everything between <|
and |> above is one token:
| Tokens | Ids |
|---|---|
<|im_start|>, <|im_end|> |
151644, 151645 |
<|text_start|>, <|text_end|>,
<|audio_start|>,
<|audio_end|> |
151665 to 151668 |
<|code_start|>, <|code_end|>,
<|text_sep|> |
151669, 151670, 151671 |
the audio codes <|0|> to
<|4099|> |
151672 to 155771 |
the durations <|t_0.00|>,
<|t_0.01|>, ... |
from 155772 |
An audio code is told from the rest by its id: token 151672 + c is
code c. The vocabulary has 4100 of them and the vocoder's codebook 4096:
OKLM finds the first by its name (<|0|>), checks that
the codes are consecutive, and keeps the codes the vocoder knows.
(llama.cpp's example keeps ids 151672 to 155772, one duration token
included, and would give the vocoder a code above 4095 if the model
wrote one.)
Sampling and the end. llama.cpp's example samples
among the 4 most likely tokens at temperature 1 (top_k = 4,
no other sampler), up to 4096 tokens, and stops at an end-of-generation
token. The model card advises temperature 0.1 with a repetition penalty
of 1.1. OKLM's default is llama.cpp's (see what is heard for why), with the guide
tokens llama.cpp has as an option (--tts-use-guide-tokens)
turned on: the first token of each word is set to the text's own, which
keeps the model on the text. A pass ends at
<|audio_end|> or the end token, when as many words
are written as the text has, or after 160 tokens per word.
Passes. A text is cut into passes of at most 24
words (--words): a sentence each, and a sentence that is
too long alone after a comma, a semicolon or a colon, then anywhere.
Sentences are not joined in a pass, however short: a sentence's end is
then always a pass's end, where the pause between two sentences is made
(below). Each pass's
sound is handed over as soon as it is made (the callback of
oklm_speech_synthesize_chunks(), a chunk of the server's
stream). How the passes share prompts is the next section.
Passes, prompts and the first sound
What a prompt costs, and what can be kept of it. The prompt of OuteTTS is the header, the voice's words, the text's words, then the voice's codes: with the built-in voice 62 tokens before the text's first word and 826 after its last. A language model's cache holds what comes before a change: from one text to another, the 62 tokens of the header and the voice's words stay valid; the voice's 658 codes, which are most of the prompt, come after the text and have to be evaluated again with every new text. Keeping "the voice" in the cache is therefore not possible with this layout, which is the one the model was trained on; what is possible is to evaluate fewer prompts.
One prompt for several passes. The passes of a text
are written one after the other in the same context while they hold 64
words together (--prompt-words,
oklm_speech_options.prompt_words; fewer when the context is
smaller or the voice longer: 48 tokens are counted per word): their
words are all in the prompt's text, and the model goes on from one
pass's last word to the next pass's first as it does inside a pass. The
prompt is evaluated once per group instead of once per pass, and the
codes are the ones a single pass of all those words would give. Each
pass is given to the vocoder when its last word is written:
- the vocoder reads the pass with the 75 codes before it (a second) as context, whose samples are dropped;
- the last 8 codes of a pass that is not the group's last wait for the next pass, which gives the vocoder what follows them;
- a pass begins by a cross-fade of 10 ms from what the vocoder wrote there with the pass before;
- a pass that ends a sentence (
.,!,?or a line feed) is given whole, nothing waiting, with 0.3 second of silence after it: the model is given no punctuation and pauses little between two sentences by itself, where a new prompt begins with a pause.
The samples are not those of one call of the vocoder on all the
codes: its group normalizations and its attention read every frame they
are given, so a pass decoded with a second of context differs from the
same codes decoded with the whole text, as two passes of their own
always did (57 words in 11 passes against one call: 14.4 dB of signal to
difference; in 4 passes 17.2 dB). What the joins are held to is
continuity: the largest step between two samples is 0.265 in the 11
passes and 0.258 in the one call, and Whisper base hears the same words
in both. The quarter of a second silenced at the start is silenced once
per prompt. When the model ends a group early (its end token before the
last word, or a context that is full), the pass it was in is given as it
is and the passes after it get a prompt of their own. With
--prompt-words equal to --words every pass has
its own prompt, as before.
From one prompt to the next, and from one call to the next (a server's requests), the tokens the cache already holds at the start of the new prompt are kept: 62 with the built-in voice, 70 when a text begins with the same word, the whole prompt but its last token when the same text is spoken again (another seed, another speed).
The first sound. It comes after the prompt's
evaluation and the codes of the first pass. --first-words N
(first_words) cuts the first pass at N words, inside a
sentence when it must: since the passes of a group share a prompt and
are joined as above, a short first pass costs no prompt of its own and
no audible seam. The chat window used to ask for a reply's first five
words as a synthesis of their own, at the price of a second prompt and
of a break in the voice between the two.
The short first pass is a piece of the first sentence, or that
sentence whole when it has N words or fewer: it moves no sentence's end,
and the 0.3 second of silence comes after every sentence that another
follows under the same prompt, with the option and without it. A text is
therefore as long either way. (Until 2026-11-15 sentences that held
together in 24 words were one pass, with no silence between them, and a
text of short sentences was 0.3 second longer with
--first-words than without.) The tests' tiny model (two
codes a word, four samples a code) shows it in bytes: "Hello there,
world. One two three." is a WAV file of 14 540 bytes, 48 samples of
words and 7200 of silence after the third word, whether
--first-words is absent, 2 (a first pass that ends at the
comma: three passes) or 5 (two passes) (test_speech_route).
The published OuteTTS 0.2 on the 64-word text of the table below, its
language model on OpenCL (RX 7900 XT): 1 016 684 bytes, 21.18 s of
speech, 2 prompts of 1913 tokens both ways; five passes and a first
sound after 0.87 s without the option, six passes and a first sound
after 0.58 s with --first-words 5; the whole text after 5.8
s (the table below has the CPU and Vulkan).
A model that ends the sound at a sentence's end leaves the sentences
after it to a prompt of their own, as before; they used to be lost when
they were in the same pass. OuteTTS 0.3 does so on "Hello. Yes.": with
the default sampling the commit before this cut (built and run again on
2026-11-15) gave 1.71 s of sound under one prompt, in which Whisper
large-v3 hears "Hello.", and the code since gives two prompts and 2.77
s, heard "Hello. Yes.". At temperature 0 the same text shows the second
prompt and nothing more: 0.3 then writes codes that are close to
silence, before the cut (one prompt, 107 tokens, 1.32 s) as after it
(two prompts of 1720 tokens, 156 tokens written, 2.22 s), and Whisper
base, large-v3 and large-v3-turbo hear no word in either
(test_published_03 checks the prompts and the pause, not
what is heard).
Measured on 2026-11-15, with every sentence a pass of its own: a text
of five sentences and 64 words, 20.8 to 22.1 seconds of speech, OuteTTS
0.2 Q8_0. On the CPU, OKLM_THREADS=4 on the Core i7-11700K
while another job kept the load average at 3.1 to 5.1; on the RX 7900 XT
the language model and the vocoder's network on the device (load 1.1 to
2.9). Two or three runs of each, one after the other: the token counts
are the same from run to run, the times within the ranges given.
| Backend | Prompts | Their tokens (kept in the cache) | Time in the prompts | Tokens written | Sound | First sound after | The whole text after | |
|---|---|---|---|---|---|---|---|---|
--prompt-words 24: a prompt per pass, its start
kept |
CPU | 4 | 3687 (210) | 4.1 s | 1913 | 21.39 s | 6.5 to 6.6 s | 43.5 to 43.7 s |
| OpenCL | 4 | 3687 (210) | 0.19 s | 1923 | 21.53 s | 0.89 to 0.92 s | 5.9 to 6.0 s | |
| Vulkan | 4 | 3687 (210) | 0.16 to 0.17 s | 1967 | 22.11 s | 0.69 to 0.70 s | 4.65 to 4.67 s | |
| the default: the passes that hold in 64 words under one prompt | CPU | 2 | 1913 (70) | 2.2 s | 1828 | 20.86 s | 6.6 s | 40.9 to 41.0 s |
| OpenCL | 2 | 1913 (70) | 0.10 s | 1852 | 21.18 s | 0.86 to 0.87 s | 5.76 to 5.80 s | |
| Vulkan | 2 | 1913 (70) | 0.09 s | 1820 | 20.75 s | 0.65 to 0.66 s | 4.31 to 4.34 s | |
the same with --first-words 5 |
CPU | 2 | 1913 (70) | 2.2 s | 1828 | 20.86 s | 4.6 s | 40.8 to 40.9 s |
| OpenCL | 2 | 1913 (70) | 0.10 s | 1852 | 21.18 s | 0.57 to 0.58 s | 5.75 to 5.78 s | |
| Vulkan | 2 | 1913 (70) | 0.09 s | 1820 | 20.75 s | 0.44 s | 4.34 to 4.35 s | |
the same with --first-words 8 --words 12 (eight
passes) |
CPU | 2 | 1913 (70) | 2.2 s | 1828 | 20.86 s | 6.6 to 6.8 s | 41.1 to 41.2 s |
| OpenCL | 2 | 1913 (70) | 0.10 s | 1852 | 21.18 s | 0.86 to 0.88 s | 5.79 to 5.84 s | |
| Vulkan | 2 | 1913 (70) | 0.09 s | 1820 | 20.75 s | 0.65 s | 4.35 s |
(The text is 64 words once its number is spelled, two more than the
62 a context of 4096 tokens is counted to hold with this voice: two
prompts. Its first sentence has eight words: the first sound without
--first-words comes after that sentence, and
--first-words 8 changes nothing to it. The sound is the
same with and without --first-words on a backend, to the
byte: 1 001 324 bytes on the CPU, 1 016 684 on OpenCL, 996 204 on
Vulkan; it differs from one backend to another as on a device says, and with the prompts, which
are other contexts. The series this table replaces was made before the
sentences were passes of their own, under a load three times heavier:
its first pass was the first 24 words, not the first sentence.)
The prompts are a small part of the time: on the CPU the 1913 tokens of the two prompts are evaluated in 2.2 seconds, where the model writes 49 tokens a second one at a time, and a second of speech is about 88 tokens. One prompt for the group saves what the other prompts cost (1.9 seconds of 43 on this text); the first sound comes earlier with a short first pass, by the tokens of the first sentence's other words (6.6 s to 4.6 s); and neither makes the synthesis faster than it is heard: on this CPU the language model writes a second of speech in two, which only a device (3.7 seconds of sound a second on OpenCL, 4.8 on Vulkan) or a smaller model changes: Soprano.
Heard. Six sentences of 57 words as one text, Whisper base listening (the table of listening by proxy spoke them one by one):
| Seed | A prompt per pass (three prompts) | One prompt, a first pass of 6 words |
|---|---|---|
| 1234 | 56 of 57 ("vocoder" heard "bar2"), 17.13 s | 55 of 57 ("on" heard "a", "vocoder"), 14.93 s |
| 7 | 56 of 57 ("vocoder"), 16.81 s | 52 of 57 ("quick" heard "quit", "and arrives on" heard "in a rinds and", "vocoder"), 14.89 s |
| 99 | 53 of 57 ("and arrives on" heard "in a rites and", "vocoder"), 16.29 s | 55 of 57 ("vocoder turns" heard "vacay returns"), 15.37 s |
| Words right | 165 of 171 (96.5%) | 162 of 171 (94.7%) |
("sea shells" heard "seashells" counted right, "$42" as "forty two dollars".) The same slip ("and arrives on") is heard under both, with different seeds: three words between the two columns are within what a seed changes. Under one prompt the model pauses less between sentences (it is given no punctuation, and nothing but a line feed separates two words), so the same text was two seconds shorter in these runs, made before the 0.3 second of silence after a sentence was added: that silence gives the pause back between passes, and changes no code.
The vocoder
WavTokenizer's decoder is a Vocos network with an attention stack in front; llama.cpp's converter names its tensors as below. The activations are [channels, frames], one frame per code.
- Codes to features: row c of
token_embd.weight(4096 x 512). - Input convolution:
conv1d(kernel 7, 512 to 768 channels, half padding). - "posnet", six blocks, their kind told by the
tensors they have:
- blocks 0, 1, 3, 4, ResNet: group normalization (
norm1: 32 groups over the channels of a group and every frame, epsilon 1e-6), swish (x * sigmoid(x)),conv1(kernel 3, 768 to 768),norm2, swish,conv2, and the input added; - block 2, attention:
attn_norm(group normalization), three 1x1 convolutionsattn_q,attn_k,attn_v, one head over every frame in both directions with the scores divided by sqrt(768),attn_output(1x1), and the input added; - block 5:
attn_normalone, a group normalization.
- blocks 0, 1, 3, 4, ResNet: group normalization (
token_embd_norm: a layer normalization over the channels (epsilon 1e-6).- ConvNeXt, twelve blocks (
convnext.N): a depthwise convolutiondw(kernel 7), a layer normalizationnorm, a linear layerpw1(768 to 2304), GELU,pw2(2304 to 768), a per-channel scalegamma, and the input added. output_norm(layer normalization) andoutput, a linear layer to 1282 values per frame: the log-magnitudes of 641 frequency bins, then their phases.- The inverse short-time Fourier transform
(
oklm_istft()): the magnitude is exp() of the first half, at most 100; the complex spectrum goes through an inverse real transform of 1280 points; each frame is multiplied by a periodic Hann window of 1280 samples; the frames are added 320 samples apart and the sum divided by the sum of the squared windows; 480 samples are dropped at each end ("same" padding). A code is 320 samples: 75 codes a second at 24 kHz (the rate is not in the file: it is the model's, and llama.cpp's example writes it as a constant too).
src/mm/vocoder.c binds the tensors where the file is
mapped (the matrices in F16, multiplied as they are; biases, norms, the
code table and the depthwise kernels as F32), and runs the convolutions
as products over unfolded frames with oklm_matmul(), the
attention with oklm_attention(), the layer normalizations,
swish and GELU with the core ops, as the Whisper encoder does; the group
normalization, the depthwise convolution and the inverse transform (a
mixed-radix transform: 1280 = 2^8 x 5) are plain C. The transform's
frames are computed on the worker team, 64 at a time, and added to the
sound in their order (the samples of one thread, to the bit: 52 ms
became 9 ms for 6.4 seconds of sound). Two choices differ from
llama.cpp, both towards the model's definition in PyTorch:
- GELU is the exact one (erf), as
torch.nn.GELU(); ggml computes the tanh approximation through a half-precision table; - the inverse real transform is the true one. llama.cpp's example sums the one-sided spectrum and divides by 641, which scales the signal by 1280/1282 and adds the constant and the highest bin a second time.
Comparison with llama.cpp
The reference. The pinned llama.cpp of
build/llama.cpp (8642541b) still has the
wavtokenizer-dec architecture, but its
llama-tts was rewritten for other models in August 2026 and
no longer runs OuteTTS. The reference is the last commit before that,
built for the CPU in a scratch directory:
1c3c9674de4d455f1e571bed808252af54932767
(2026-08-04), cmake -DGGML_NATIVE=ON -DLLAMA_CURL=OFF,
target llama-tts, with a few lines added to
tools/tts/tts.cpp that write the prompt's ids, the
generated ids, the vocoder's output rows and the samples to files. It
ran with --top-k 1 (the most likely token each time) on
three sentences: "Hello world, this is a test of speech synthesis.",
"The quick brown fox jumps over the lazy dog." and "It costs 42 dollars
and arrives on Monday morning."
The prompt's token ids: identical for the three sentences (906, 906 and 913 ids).
The audio codes. Chosen greedily by each program, they are the same for the first 75, 104 and 29 codes, then differ: the codes are close calls by nature (the model is meant to be sampled among four), and the two programs do not round alike. Fed llama.cpp's own tokens, OKLM's most likely token is llama.cpp's at 258 of 262 steps, 252 of 254 and 289 of 300; where it is not, OKLM's logits of the two tokens differ by 0.033 at most (0.007 and 0.019 for the other two sentences).
The vocoder, on llama.cpp's codes (212, 204 and 240
of them). The third implementation is
tools/gen_speech_fixtures.py --run: numpy in double
precision, written from the model's definition.
| Compared, sentence 1 | Largest difference | Mean difference | Signal to difference |
|---|---|---|---|
| Rows (log-magnitude, phase): OKLM against numpy | 1.1e-4 | 6.3e-7 | 131.2 dB |
| Rows: llama.cpp against numpy with ggml's GELU | 6.5e-2 | 8.4e-4 | 71.1 dB |
| Rows: OKLM against llama.cpp | 4.9e-1 | 3.3e-3 | 57.4 dB |
| Rows: OKLM built with ggml's GELU, against llama.cpp | 6.5e-2 | 8.4e-4 | 71.1 dB |
Samples: OKLM against numpy (irfft) |
1.1e-6 | 2.5e-8 | 124.7 dB |
| Samples: OKLM against llama.cpp | 4.4e-3 | 5.6e-4 | 40.9 dB |
| Samples: llama.cpp's transform applied to OKLM's rows, against llama.cpp | 3.6e-3 | 1.4e-4 | 50.8 dB |
| Samples: the true transform against llama.cpp's, both on llama.cpp's rows | 3.6e-3 | 5.3e-4 | 41.5 dB |
The samples are in [-1, 1] with a root mean square of 0.09. Sentences 2 and 3 give 51.7 and 49.4 dB on the rows against llama.cpp (70.5 and 65.9 dB with ggml's GELU) and 39.4 and 39.6 dB on the samples. Read together: OKLM's network is the definition's to 131 dB; llama.cpp's is the same network to 71 dB, the rest being its half-precision GELU table; the 41 dB between the two programs' samples is the difference between the two inverse transforms, which the last line measures alone.
OuteTTS 0.3 was not run against llama.cpp: the build
of commit 1c3c9674 was a scratch build, and the pinned llama.cpp no
longer speaks OuteTTS. What stands in for it: the source of that
commit's tools/tts/tts.cpp (the version read from the chat
template, the separator in process_text() and in the guide,
the three replacements applied to the built-in voice,
audio_text_from_speaker() and
audio_data_from_speaker() for a speaker file), which the
test follows (0.3's prompt is 0.2's after those replacements, for the
built-in voice and a speaker file); the prompt of the first sentence is
876 tokens, the 906 that are llama.cpp's for 0.2 less the 30
<|code_start|>; the vocoder is the same file; and
what is heard, below.
Listening by proxy
Six sentences spoken by oklm speak, transcribed by
oklm transcribe with Whisper, and the words compared (lower
case, numbers spelled; "$42" read as "forty two dollars"). The default
settings, seed 1234:
| Sentence | Heard by Whisper base | Heard by Whisper small |
|---|---|---|
| Hello world, this is a test of speech synthesis. | the same, 9 of 9 | the same, 9 of 9 |
| The quick brown fox jumps over the lazy dog. | "... over the lazy dobs.", 8 of 9 | the same, 9 of 9 |
| It costs 42 dollars and arrives on Monday morning. | "It costs $42 and arrives on Monday morning.", 10 of 10 | the same words, 10 of 10 |
| Please open the window before you leave the room. | the same, 9 of 9 | "PZAP in the window before you leave the room.", 7 of 9 |
| The model writes audio codes and the vocoder turns them into sound. | "... and the VARTA turns them into sound.", 11 of 12 | "... and the Vita turns them into sound.", 11 of 12 |
| She sells sea shells by the sea shore. | "She sells seashells by the sea shore.", 6 of 8 | "She sells seashells by the seashore.", 4 of 8 |
| Words right | 53 of 57 (93.0%); the first three sentences 27 of 28 | 50 of 57 (87.7%); the first three 28 of 28 |
"seashells" and "seashore" are the same sounds written as one word: counted right, the figures are 55 and 54 of 57. What is left is one ending ("dog"), the start of one sentence for one of the two listeners, and "vocoder", a word neither the speech model nor Whisper knows.
The same six sentences with other settings, heard by Whisper base:
| Settings | Words right |
|---|---|
| temperature 1, top-k 4 (llama.cpp's; the default), with or without the guide | 53 of 57 (93.0%) |
| temperature 0.4, top-k 40, repeat penalty 1.1, guide | 46 of 57 (80.7%) |
| the most likely token, no guide | 45 of 57 (78.9%) |
| temperature 0.1, repeat penalty 1.1 (the model card's), no guide | 42 of 57 (73.7%) |
| the default with seed 7, heard by Whisper small | 49 of 57 (86.0%) |
| the same without the guide, heard by Whisper small | 42 of 57 (73.7%) |
At low temperatures the model slurs words ("and arrives on" heard as "in a rise on") and, without the guide, sometimes begins by repeating the voice's last words ("and it looks lovely. Please open the window before you"): the repeat penalty and a low temperature both work against a model whose codes repeat by nature. With seed 1234 the guide changed nothing; with seed 7 the run without it began one sentence with the voice's last words again, which is what the guide is for. "sea shells" heard as "seashells" counts as two words wrong in every row (four with "seashore" for Whisper small).
OuteTTS 0.3, the six sentences as one text under one prompt (seed 1234, a first pass of 6 words), heard by Whisper base: "Hello world, this is a test of speech synthesis. The quick brown fox jumps over the lazy dog. It costs $42 and a rise on Monday morning. Please open the window before you leave the room. The model writes audio codes and a vaker that turns them into sound. She sells seashells by the seashore.": 54 of 57 ("arrives", "the", "vocoder"), where 0.2 gave 55 with the same settings (above); with seed 7, 52 of 57 ("quid" for "quick", "door" for "dog", "Peace out in" for "Please open", "vocoder"), as 0.2 with that seed. 0.3 speaks slower: 19.5 s and 19.2 s for 0.2's 14.9 s, 1658 and 1634 tokens for 1417. In one run of the 64-word text with a first pass of 5 words, its first word was not heard ("Synthesis turns a written text into sound ..."): 0.3 can begin to speak inside the quarter of a second silenced after a prompt, which is llama.cpp's for both versions and was kept.
A longer text, five sentences of 64 words, took four passes (7.7, 4.3, 5.1 and 3.7 seconds of speech; 20.8 seconds in all) and came back from Whisper small as "Speech synthesis turns a written text in the sound, the language model reads the words and writes audio codes 75 for each second. The Vata then turns the codes into a waveform at 24 kHz. A long text is cut into passes of a few sentences and each pass is spoken on its own. This paragraph has five sentences and takes several passes.": "into sound" and "vocoder" misheard, the rest right.
Limits
- English only. The model card lists Chinese,
Japanese and Korean as experimental; they need a romanization OKLM does
not have, and the normalization drops every character outside
atoz. OuteTTS 0.3's card adds French and German. Tried once with what is here (the built-in English voice, texts without accents, Whisper small told the language): of "Guten Morgen, das ist ein Test der Sprachsynthese. Das Wetter ist heute sehr gut, und wir gehen in die Stadt." the first 13 words of 19 are heard and the end is not ("um die Gaming gestartet"); of a French sentence of 17 words, 5. Another language takes a voice recorded in it, the accented letters kept as OuteTTS's own processor keeps them, and numbers spelled in it: not done. - One built-in voice (a man's,
en_male_1). Other voices are speaker files made elsewhere; OKLM cannot make one from a recording: cloning a voice takes WavTokenizer's encoder (OKLM has its decoder only) to turn the recording into codes, and an aligner to cut them word by word. The chat window lists the speaker files it finds and speaks with the one chosen (gui.md). A server offers the voice it was started with (--speak-voice), the built-in one and the speaker files of its voice directories (--speak-voices) and of the model's folder: a request names one (server.md), andoklm_speech_set_voice()changes the voice of an open synthesizer. - No pronunciation control: no punctuation reaches the model, so no pause is asked for inside a pass; abbreviations, symbols and digits stuck to letters are read as their letters allow.
- Length. A text of any length is spoken, pass after
pass. One pass takes the prompt (888 tokens with the built-in voice, two
or so per word of the text) and what the model writes (about 75 tokens
per second of speech and four per word) in a context of 4096 tokens:
about 40 seconds;
--wordstakes up to 64. The default of 24 words keeps a pass near 10 seconds, about the length of the voice's own recording. The vocoder takes 8192 codes a call (109 seconds). - Not streamed inside a pass.
oklm_speech_synthesize_chunks()gives each pass's samples when the pass is done; the first sound comes after the prompt and the first pass's codes. A client that wants it early asks for a short first pass (first_words), which costs no prompt (above). The vocoder's attention reads every frame it is given, so the end of a pass is written with the few codes after it, or with none when it is the last. - The voice's codes are evaluated with every prompt: they follow the text in the prompt, so the cache keeps the 62 tokens before the text and no more. A shorter voice makes a shorter prompt.
- Slower than real time with OuteTTS (Soprano is not) on this machine's CPU (Core i7-11700K, 8 cores). The best run spoke 2.6 seconds in 6.0 seconds, 0.43 second of sound per second; the runs of the tables above, made on 2026-11-15 with four threads while another job kept the load at 3 to 5, gave 0.49 to 0.51 (the series before them, while other builds held every core, 0.12 to 0.38). Nearly all of it is the language model: the prompt's 890 tokens once per group of passes (about a second in those runs), then about 90 tokens per second of speech, written at 49 a second; the vocoder and the inverse transform took 1.1 second for 20.9 seconds of sound. llama.cpp's example, run in turn with OKLM under that load, took 28 to 74 seconds for the sentence OKLM spoke in 19 to 25: the same order, the load deciding. Peak memory was 1.65 GiB.
- The vocoder's inverse transform on the CPU.
--backend opencl|vulkanputs the language model on a device, where nearly all the time was, and the vocoder's network with it (on a device). - Licence: OuteTTS 0.2 is CC-BY-NC-4.0, not for commercial use; OuteTTS 0.3 is CC-BY-SA-4.0, which allows it.
- A sampled run is repeatable with its seed on one machine, not across machines: near-equal logits can round differently.
Speed
--speed S, oklm_speech_options.speed and
the route's speed (0.25 to 4; 1 is the model's own pace)
change how long the sound lasts and keep its pitch. OuteTTS has no
control of its pace (the durations in its prompt are the voice's, and
the model writes those of the text itself), so the change is made on the
samples, pass by pass, by oklm_speech_stretch(): waveform
similarity overlap-add. Frames of 40 ms are taken from the input every
20 ms times the speed, each moved by at most 10 ms to the place where it
is most like what would have followed the frame before (the largest
normalized correlation), and added 20 ms apart under a Hann window. The
first frame is the input's own start, so two passes still join.
A 220 Hz tone comes out at 220 Hz (216 to 224 counted on its zero crossings) at speeds 0.5, 0.75, 1.5, 2 and 4, at 0.85 to 1.15 of its level and of the length asked to the sample (the test). The 64-word text above at 1.5 is 14.12 s of sound for 21.18 s, at 0.75 28.24 s (OpenCL, 2026-11-15, the 0.3 s after each sentence stretched with it); Whisper base heard the same words in the three before those pauses were added. Above 2 speech is hard to follow: the frames skip more than they hold.
FLAC
src/mm/flac.c (include/oklm/mm/flac.h)
writes FLAC files of mono 16-bit sound without a library: the stream
marker, a STREAMINFO block, frames of 4096 samples (the last one
shorter), each one subframe: a constant, the samples as they are, or one
of the format's five fixed predictors (orders 0 to 4, the one whose
residual is smallest) with its residual in Rice codes, cut into the 1 to
256 partitions that take the fewest bits, each with its own parameter;
the two checksums of every frame. No linear prediction (LPC) and no MD5
of the samples (the header's field is zero, which the format reads as
"not computed": flac -t says so and checks the rest).
oklm speak -o NAME.flac, the route's"response_format": "flac"(whole, its length in the header, or streamed frame by frame with"stream": true, the length left unknown),oklm_flac_encode()andoklm_flac_writerin the library.- Checked with the reference tools:
flac -taccepts the files,flac -dgives back the WAV fileoklm speakwrites byte for byte, andffmpegdecodes the same samples. 5.3 s of the vocoder's sound: 151,235 bytes for the 256,044 of the WAV file (59%), whereflac -8, with linear prediction, makes 142,653 (56%). oklm_flac_decode()reads what the encoder writes (one channel, 16 bits, those subframes), for the tests and for reading a file back; it is not a general FLAC reader (Allegro is, foroklm transcribe).
MP3, Opus and AAC are not written, and the route
refuses them (400, saying so). They are lossy codecs whose encoders are
whole programs: MP3 needs a polyphase filter bank, an MDCT, a
psychoacoustic model and its bit allocation loops and Huffman tables;
Opus is two codecs (SILK and CELT) and a range coder, some tens of
thousands of lines in its reference implementation. Neither fits a
contained, tested module of this size, and OKLM links no codec library
to write sound. FLAC is the compressed form: about half the size of the
WAV, exact, read by browsers (<audio>,
decodeAudioData()), ffmpeg and every player.
Soprano
Soprano 1.1 (ekwek/Soprano-1.1-80M, Apache-2.0) is the
model that speaks faster than it is heard on a CPU. It has no voice in
its prompt, so none of the cost measured above, and it writes 16
tokens for a second of speech where OuteTTS writes 90. English, one
fixed voice, 32 kHz.
oklm models pull soprano-1.1-80m # 197 MiB, Apache-2.0
oklm speak -m soprano-1.1-80m "Hello world, this is a test of speech synthesis."
oklm speak -m soprano-1.1-80m --first-words 5 -o reply.flac < reply.txt
oklm speak -m Soprano-1.1-80M-soprano-tok.F16.gguf --vocoder codec-F32.gguf "Dr. Smith paid $2.47 at 8:05."
oklm serve -m qwen3-8b --speak-model soprano-1.1-80m # POST /v1/audio/speech at 32 kHz
$ oklm speak -m soprano-1.1-80m -o cat.wav "The catalogue name finds both files."
[2.3 s of speech after 0.3 s]
first sound after 0.27 s; 1 prompt of 29 tokens (0 kept in the cache) in 0.01 s, 37 tokens written in 0.16 s, the vocoder 0.10 s
cat.wav: 2.30 s of speech in 0.27 s (8.42 s of sound per second)
$ oklm transcribe -m whisper-base cat.wav
The catalog name finds both files.
(The best run seen, four threads; the same sentence took OuteTTS 0.2 6.00 s at its best, above.)
oklm speak, the library and the route are those of
OuteTTS; what a caller sees change is the sample rate
(oklm_speech_rate() is 32000), the voice (none to choose:
--voice, oklm_speech_params.voice and
oklm_speech_set_voice() with a file are refused, saying
"Soprano speaks with one fixed voice: a speaker file does not apply")
and the sampling's defaults (below).
Its two files
| Language model | Decoder | |
|---|---|---|
| Original | ekwek/Soprano-1.1-80M (revision 27b5a5f5):
model.safetensors, 159,449,112 bytes, bfloat16 |
the same repository's decoder.pth, 121,448,447 bytes,
SHA-256
11245487e4588b0f4128d6b4c82035212bad42ffb26f36e059e9c7f861c1f8c4:
a PyTorch checkpoint |
| What the catalogue installs | BricksDisplay/Soprano-1.1-80M-GGUF (revision
d4f2e374),
Soprano-1.1-80M-soprano-tok.Q8_0.gguf |
the same repository's codec-F32.gguf |
| Size, SHA-256 | 84,921,376 bytes,
b16d98166e83b3e5f4740b25e5b14f09346dffd4ffab1660ac787768aaf414a5 |
121,423,904 bytes,
6004d7a8cbd04a6ae14172a6b78dfac2114e4a75a5dacb7ed3a454ea5ec329a1 |
general.architecture |
qwen3 |
soprano (the layout of codec.cpp's converter) |
| Shape | 17 blocks, width 512, 4 heads over 1 key/value head of 128, FFN 2304, RoPE base 10000, context 1024 | hidden states of width 512 in, an input layer to 768 channels, 8
ConvNeXt blocks (depthwise kernel 3, FFN 2304), a head of 2050 values
per frame; n_fft 2048, hop 512, 4 frames per state |
| Tensors | 190 (79.7 M parameters; Q8_0, the norms F32) | 81 (30.4 M values, F32) |
| Vocabulary | 8192 tokens: [UNK], [TEXT],
[START], [STOP] (0 to 3), the audio tokens
[0] to [7999] (4 to 8003), and 188 text tokens
(8004 to 8191): 53 characters (the space,
!"$%&'*+,-./<>?_, the digits, a to
z) and 135 merges (th, in, ...
gonna); tokenizer.ggml.pre =
soprano; the beginning, the end and the padding are all
token 3 |
none |
The same repository has the language model as F16 (159,632,416 bytes,
2f7f6821..., the checkpoint's own precision) and in smaller
quantizations, and files without -soprano-tok in their name
whose tokenizer.ggml.pre is gpt-2: those are
for codec.cpp and are cut into the wrong tokens by OKLM and by
llama.cpp. The catalogue takes Q8_0 because it is the faster here
(below) and is heard the same.
The decoder's file is the checkpoint.
decoder.pth is a zip archive of a pickle (the tensors'
names, shapes and places) and one raw file per tensor.
tools/gen_soprano_fixtures.py reads it with Python's
standard library and numpy, without PyTorch (its unpickler resolves the
tensor constructors of a state dictionary and nothing else):
--check-pth decoder.pth codec-F32.gguf: the 81 tensors of the published GGUF file are the checkpoint's, byte for byte, and its metadata is what the checkpoint's shapes give;--convert decoder.pth out.ggufwrites that GGUF file from a checkpoint: fordecoder.pththe result iscodec-F32.ggufbyte for byte (cmp), so a decoder trained elsewhere converts the same way;--mmproj decoder.pth out.ggufwrites the projector file llama.cpp'sllama-ttsreads, for the comparison below.
From the text to the prompt
The cleaner. Soprano's package cleans a text before
its tokenizer sees it (clean_text() of
soprano/utils/text_normalizer.py, adapted there from
tortoise-tts); llama.cpp leaves that to the caller. OKLM does it
(src/mm/soprano_text.c,
oklm_speech_normalize_soprano()), in the package's
order:
- ASCII:
éise,ßisss, the typographic quotes and dashes the plain ones, a long dash a dash between spaces; - every line is a sentence: a line that does not end with
.,!or?gets a period; - numbers:
#1is "number one",5K"five thousand",1,234loses its comma,12/25/2024is "twelve dash twenty-five dash twenty twenty-four",(555) 123-4567its digits one by one,8:05"eight oh five",$2.47"two dollars, forty-seven cents",3.14"three point one four",2 * 3,6 / 2,2 + 2and-5"times", "over", "plus" and "minus",21st"twenty-first",mp3"mp three", and what is left in words: 1001 to 2999 as a year ("nineteen eighty-four", "two thousand nine", "nineteen oh five"), the others as theinflectpackage writes them ("one million, two hundred thirty-four thousand, five hundred sixty-seven"); https://spelled, a dash between spaces a comma, a period between two letters "dot", brackets commas;- abbreviations:
Dr."doctor",Mrs."misess",St."saint", ...;Hz,MB,API,CPU,Mon,Jan,etc, ... in their own case; - a word with capitals inside is cut before each ("LMDeploy" is "l m deploy"; "TPUs" is "tpu's");
- signs:
@"at",&"and",%"percent",:a period,;a comma,+,\,~,<,>,=,/in words,_and*spaces; - lower case; only the letters, the space and
!$%&'*,-.?are kept; - one space for several, none before punctuation;
..is.,!!is!,?!is?, three periods stay an ellipsis; a letter written three times in a row is written twice.
It is written by hand, each regular expression of the package as a
scan of the text, and checked on 386 texts against the package's
expressions applied with Python's re
(clean_text() of
tools/gen_soprano_fixtures.py, numbers spelled by a
rewriting of inflect's rules): the two give the same text
for all of them, and for 80,000 random texts put together from the
pieces the rules look for (digits, signs, brackets, abbreviations,
capitals). The package's own cleaner was not run: it
needs the inflect and unidecode packages.
Where OKLM differs from it, on purpose:
- characters outside the table of step 1 are dropped;
unidecodewould write other scripts in Latin letters; £20is "twenty pounds": the package turns£intoPSbefore its rule for pounds can see it;- a time with seconds reads its minutes as they are ("3:45:10" is
"three forty-five ten"); the package writes a Python set there
(
{'45'}); - a tab is a space; the package drops it and glues the words;
- a number above the decillions is read digit by digit, and
$,5is five dollars;inflectand the package raise an error.
It keeps the package's oddities too: "1001" is "ten oh one", "9-5" "nine minus five", "the 1990s" "the nineteen ninety s".
The tokenizer (tokenizer.json:
Lowercase and \s+ to a space, then Digits with each digit
alone and a Split on \s+|\w+|[^\w\s]+, then BPE without
byte encoding, [UNK] for a piece the vocabulary lacks) is
the GGUF's tokenizer.ggml.pre = soprano, which
OKLM's tokenizer had; the published files do not all carry
tokenizer.ggml.unknown_token_id, and OKLM takes
[UNK] when they do not (the pinned llama.cpp writes the id
-1 for such a piece with these files). On the eight texts of the
repository's tokenizer-tests.md, whose counts are the
Hugging Face tokenizer's, OKLM gives the same ids from the GGUF file and
from tokenizer.json, and the table's count for seven; for
the eighth, a line of emoji and signs, OKLM and the pinned llama.cpp
both write 29 tokens where the table says 30 (not explained: the Hugging
Face tokenizer is not on this machine). A cleaned text has no character
outside the vocabulary, so [UNK] is not written in a
synthesis.
The prompt is [STOP],
[TEXT], the cleaned text's tokens, [START] (3,
1, ..., 2): "hello world, this is a test of speech synthesis." is 36
tokens. The model writes audio tokens until [STOP].
What goes to the decoder is not the tokens but the
final normalized hidden state that predicted each of them: the state at
[START] for the first, the state at each token written for
the next, the one that predicts [STOP] left out
(outputs.hidden_states[j][-1] of the package's
generate(); llama_get_embeddings_ith() in
llama.cpp; oklm_context_params.hidden and
oklm_get_hidden() here). oklm_speech_states()
gives them for a text.
Sampling. The package's: the most likely token (a
temperature of 0.001 under top-p 0.95) with a repetition penalty of 1.2
over the prompt and everything written, at most 512 tokens (32.7 s).
OKLM's defaults for this model are those
(oklm_speech_options_for():
--temp 0 --top-k 0 --repeat-penalty 1.2; the prompt's
tokens count for the penalty, which holds [STOP], the
prompt's first token, back a little), where OuteTTS's are temperature 1
and top-k 4. --temp above 0 samples under top-p 0.95. A
prompt is also cut at 32 tokens plus three per character of its text (a
token is 64 ms, a character of speech about as much): a model that does
not stop is stopped there. The package's detector of runs of similar
states (its retries) is not there.
Passes. As the package does: every sentence is a
prompt of its own, and a sentence whose cleaned text is shorter than 30
characters goes with the one before it (the first one with the one
after). The model was trained on such prompts: given two long sentences
in one prompt it repeated the end of the second ("... by the seashells
by the seashore", heard by both listeners), which it does not when each
has its prompt. The sentences are found as for OuteTTS (.,
! or ? followed by a space, or a line feed), a
sentence of more words than --words (24) is cut after a
comma, a semicolon or a colon, then anywhere, and a prompt's text of
more than 300 characters is cut again at a space (the package's splitter
cuts at 300 too). A prompt costs three tokens beside its text:
--prompt-words is not read, and nothing is kept in the
cache from one prompt to the next.
The first sound. --first-words N does
not cut the text: the sound of the first prompt's first N words, counted
as 6 tokens (0.38 s) a word, is handed over as soon as those tokens and
the 4 after them are written, then the rest of the sentence when it is
done. The decoder is convolutions and a transform of overlapping
windows: a sample depends on the states within 4 of its own (the
package's RECEPTIVE_FIELD), so the two pieces are
the samples of the whole sentence, bit for bit (checked
on the three sentences below and in the tests), with no seam to
hide.
The decoder
src/mm/vocoder.c reads it beside WavTokenizer's
(oklm_vocoder_load() tells them by
general.architecture;
oklm_vocoder_state_width() is 512 for this one and 0 for a
vocoder of codes):
- the n states, [512] each, are interpolated linearly to 4 (n - 1) + 1
frames, the first and the last frame being the first and the last state
(
interpolate(mode='linear', align_corners=True)); sop.decode.embed, a convolution of kernel 1 to 768 channels, and a layer normalization (epsilon 1e-6);- eight ConvNeXt blocks (
sop.decode.cnx.N): the depthwise convolutiondw(kernel 3),ln,pw1(768 to 2304), the exact GELU,pw2, the scalegamma, and the input added: the blocks of WavTokenizer's decoder, run by the same code; sop.decode.flnandsop.decode.head.out: 2050 values per frame, the log-magnitudes of 1025 bins (at most 100 once raised) and their phases;oklm_istft_center(): the constant and the highest bin of every frame set to zero (the head's ownspec[:, 0] = 0; spec[:, -1] = 0), an inverse real transform of 2048 points, a periodic Hann window (the file'ssop.decode.istft.windowis checked to be one), frames added 512 samples apart, the sum divided by the sum of the squared windows, 1024 samples dropped at each end astorch.istft(center=True)does: (frames - 1) x 512 samples. A state is 2048 samples: 15.6 states a second at 32 kHz.
Comparison with llama.cpp and with a double-precision decoder
The reference. The pinned llama.cpp
(build/llama.cpp, 8642541b) runs Soprano in
llama-tts. That one tool was built from the pinned sources
for the CPU in a directory of its own
(cmake -DGGML_NATIVE=ON -DLLAMA_CURL=OFF, the tree used as
a subdirectory so that nothing is written in it), with a copy of
tools/tts/tts.cpp that also writes the text's ids, the
tokens sampled, the states handed to its decoder and the samples as
floats. Its files: the same language model file as OKLM, and the
projector file tools/gen_soprano_fixtures.py --mmproj
writes from decoder.pth (what llama.cpp's converter writes
with PyTorch: the same F32 values, the depthwise kernels in half
precision because ggml's convolution reads no other). Both programs ran
the F16 language model with the most likely token and no repeat penalty,
four threads, on the cleaned texts "hello world, this is a test of
speech synthesis.", "the quick brown fox jumps over the lazy dog." and
"it costs forty-two dollars and arrives on monday morning.". The third
implementation is the numpy decoder of
tools/gen_soprano_fixtures.py --run, in double precision,
written from the package's soprano/vocos/*.py.
| Compared | Sentence 1 | Sentence 2 | Sentence 3 |
|---|---|---|---|
| The text's token ids, OKLM and llama.cpp | the same 33 | the same 34 | the same 38 |
| The tokens written | the same 48 | the same 42 | the same 52 |
| Hidden states, OKLM against llama.cpp (root mean square 1) | 63.9 dB, largest difference 0.007 | 65.0 dB, 0.004 | 65.2 dB, 0.006 |
| Rows (log-magnitude, phase): OKLM against numpy, the same states | 134.8 dB | 126.3 dB | 124.8 dB |
| Samples: OKLM against numpy, the same states | 122.0 dB, largest difference 1.0e-6 | 110.2 dB, 4.7e-6 | 107.4 dB, 1.1e-5 |
Samples: the synthesis with first_words (two pieces)
against one call of the decoder |
identical | identical | identical |
| Samples: llama.cpp against numpy on llama.cpp's states, the two bins kept and the depthwise kernels in half precision | 57.3 dB | 43.6 dB | 39.6 dB |
| Samples: numpy with the two bins kept against numpy without | 38.9 dB | 39.5 dB | 39.9 dB |
| Samples: numpy on OKLM's states against numpy on llama.cpp's | 46.4 dB | 30.1 dB | 23.6 dB |
| Samples: OKLM against llama.cpp, each from its own states | 37.8 dB | 27.9 dB | 21.5 dB |
The samples are in [-1, 1] with a root mean square of 0.08. Read together: the language model is llama.cpp's to the token (142 tokens of 142) and to 64 dB on the states; OKLM's decoder is the definition's to 107 dB or better. llama.cpp's decoder is the same network (mean difference 4e-5 to 1.4e-4 of a sample, a few frames further off) with one difference of definition: it keeps the constant and the highest frequency bin, which the model's head sets to zero; that alone is the 39 dB of the eighth row. The last two rows are not a decoder's error: the decoder turns phases into samples, and states that differ at 64 dB come out 24 to 46 dB apart whoever computes them (ninth row, one decoder on both programs' states).
The Q8_0 file the catalogue installs: its states are 35 to 37 dB from the F16 file's in both programs (the quantization, not the program), the two programs write the same first 17, 42 and 28 tokens of the three sentences and then choose differently between near-equal tokens, as they do for OuteTTS.
What was not compared: the package itself (PyTorch
is not on this machine). The decoder's reference is its definition
rewritten in numpy; the language model's is llama.cpp; the cleaner's is
the package's expressions under Python's re.
Heard
Six sentences spoken one by one by
oklm speak -m soprano-1.1-80m (the defaults), transcribed
by oklm transcribe:
| Sentence | Heard by Whisper base | Heard by Whisper small |
|---|---|---|
| Hello world, this is a test of speech synthesis. | the same, 9 of 9 | the same, 9 of 9 |
| The quick brown fox jumps over the lazy dog. | the same, 9 of 9 | the same, 9 of 9 |
| It costs 42 dollars and arrives on Monday morning. | "It costs $42 and arrives on Monday morning.", 10 of 10 | the same words, 10 of 10 |
| Please open the window before you leave the room. | the same, 9 of 9 | the same, 9 of 9 |
| The model writes audio codes and the vocoder turns them into sound. | "... and the VACOTER turns them into sound.", 11 of 12 | "... and the Vockoder turns them into sound.", 11 of 12 |
| She sells sea shells by the sea shore. | "She sells seashells by the seashore.", 4 of 8 | the same, 4 of 8 |
| Words right | 52 of 57 (91.2%); 56 with "seashells" and "seashore" counted right | 52 of 57; 56 the same way |
Every word is heard but "vocoder", which neither Whisper knows;
OuteTTS 0.2 on the same sentences is heard 53 and 50 of 57 (above), with a wrong ending and a wrong
start that Soprano does not have. The six sentences as one text (17.2 s
of sound, six prompts) are heard the same way by both: every word but
"vocoder" ("Vacoter", "Vockoder"), and with
--temp 0.3 --seed 7 too. The 64-word text of the table above comes back from
Whisper small as "Speech synthesis turns a written text into sound. The
language model reads the words and writes audio code 75 for each second.
The Valkoder then turns the codes into a waveform at 24 kHz. A long text
is cut into passes of a few sentences and each pass is spoken on its
own. This paragraph has five sentences and takes several passes.":
"codes," and "vocoder" misheard. And "Dr. Smith paid $2.47 on 3/4/2025
at 8:05 for the 2nd API key, i.e. about 50% of 1,200 MB." (cleaned to
"doctor smith paid two dollars, forty-seven cents on three dash four
dash twenty twenty-five at eight oh five for the second a p i key, i dot
e. about fifty percent of twelve hundred megabyte.") as "Dr. Smith paid
$2.47 on 3-4-2025-805 for the second API key I.E. about 50% of 1200
megabytes."
Measured against OuteTTS 0.2
The text of five sentences and 64 words of the table above, measured on
2026-11-15 with OuteTTS's sentences passes of their own: on the CPU,
OKLM_THREADS=4 on the Core i7-11700K (8 cores, 16 threads)
while another job kept the load average at 4.1 to 5.1; on the RX 7900
XT, the language model and the decoder's network on the device. Three
runs of each, one after the other:
| Backend | Prompts, their tokens | Tokens written | Sound | First sound after | The whole text after | Sound per second | |
|---|---|---|---|---|---|---|---|
| OuteTTS 0.2 (Q8_0), the defaults | CPU | 2, 1913 | 1828 | 20.86 s | 6.6 s | 40.9 to 41.0 s | 0.51 |
| OpenCL | 2, 1913 | 1852 | 21.18 s | 0.86 to 0.87 s | 5.76 to 5.80 s | 3.65 to 3.67 | |
| Vulkan | 2, 1913 | 1820 | 20.75 s | 0.65 to 0.66 s | 4.31 to 4.34 s | 4.78 to 4.82 | |
OuteTTS 0.2 with --first-words 5 |
CPU | 2, 1913 | 1828 | 20.86 s | 4.6 s | 40.8 to 40.9 s | 0.51 |
| OpenCL | 2, 1913 | 1852 | 21.18 s | 0.57 to 0.58 s | 5.75 to 5.78 s | 3.66 to 3.68 | |
| Vulkan | 2, 1913 | 1820 | 20.75 s | 0.44 s | 4.34 to 4.35 s | 4.77 to 4.78 | |
| Soprano 1.1 (Q8_0, the catalogue's), the defaults | CPU | 5, 275 | 308 | 19.39 s | 0.20 to 0.21 s | 1.48 to 1.54 s | 12.6 to 13.1 |
| OpenCL | 5, 275 | 312 | 19.65 s | 0.06 to 0.07 s | 0.50 to 0.51 s | 38.8 to 39.6 | |
| Vulkan | 5, 275 | 313 | 19.71 s | 0.06 s | 0.42 to 0.45 s | 44.2 to 47.1 | |
Soprano 1.1 (Q8_0) with --first-words 5 |
CPU | 5, 275 | 308 | 19.39 s | 0.17 s | 1.49 to 1.62 s | 12.0 to 13.0 |
| OpenCL | 5, 275 | 312 | 19.65 s | 0.05 s | 0.50 to 0.51 s | 38.4 to 39.5 | |
| Vulkan | 5, 275 | 313 | 19.71 s | 0.05 s | 0.42 to 0.44 s | 45.3 to 46.6 | |
| Soprano 1.1 (F16), the defaults: an earlier series | CPU, load 15 to 18 | 5, 276 | 319 | 20.10 s | 1.71 s, 1.58 s, 1.25 s | 9.5 s, 11.4 s, 7.9 s | 2.12, 1.76, 2.56 |
Soprano 1.1 (F16) with --first-words 5: an earlier
series |
CPU, load 15 to 18 | 5, 276 | 319 | 20.10 s | 1.28 s, 1.08 s, 1.34 s | 8.2 s, 11.2 s, 8.1 s | 2.44, 1.80, 2.47 |
(The F16 file is not on the machine of 2026-11-15: its two rows are those of the series before, made under six other jobs, where the Q8_0 file gave 2.7 to 4.0 seconds of sound a second and OuteTTS 0.2 0.08 to 0.23. With nothing else running OuteTTS 0.2 made 0.43 second of sound a second at best in the runs of Limits.)
Soprano makes 12.6 to 13.1 seconds of sound a second on the CPU where
OuteTTS 0.2 makes 0.51 under the same load: 25 times faster, and faster
than it is heard with room to spare. Its first sound comes after 0.2 s
without asking for anything (its first prompt is the first sentence,
eight words here), 0.17 s with --first-words 5, against 6.6
s, or 4.6 s with --first-words 5, for OuteTTS. On a device
both are faster than they are heard: OuteTTS 3.7 seconds of sound a
second on OpenCL and 4.8 on Vulkan, its first sound after 0.9 and 0.7 s
(0.6 and 0.4 with --first-words 5); Soprano 39 and 44 to
47, its first sound after 0.06 s. Where the time goes for the Q8_0 file
on the CPU: the five prompts 0.07 s, the 308 tokens 0.86 to 0.97 s (320
to 360 a second), the decoder and its transform 0.54 to 0.57 s. A
sentence alone is spoken at 2.5 to 4.6 seconds of sound a second (the
six of the table above, under the earlier load).
A second series of the Q8_0 file alone, later (load 19 to 21): the
text after 4.3 s, 4.4 s and 3.9 s (4.5 to 5.0 seconds of sound a
second), its first sound after 0.76 s, 0.46 s and 0.50 s; with
--first-words 5 after 3.7 s and 3.8 s, the first sound
after 0.39 s and 0.45 s. Peak memory is 397 MiB for this text, where
OuteTTS takes 1.65 GiB.
Which model the examples name
OKLM has no default speech model: oklm speak and
oklm serve --speak-model take the model they are given. The
examples name one model, outetts-0.2-500m:
those of README.md, of this page, of server.md and of gui.md, and the hints the chat window and the
dashboard's page give when nothing speaks. Another model is named only
on a line that is about it (OuteTTS 0.3, Soprano), and wherever an
example starts a server or a window with OuteTTS, a line beside it does
the same with Soprano. Until 2026-11-15 the Speech section of server.md
and the README's device example named outetts-0.3-500m.
Why OuteTTS 0.2 and not Soprano, from what is weighed below: the
examples about voices (--speak-voice,
--speak-voices, the Voice chooser, the route's
voice) only run with a model that takes speaker files; the
transcripts and figures of gui.md and server.md were made with it and
stay true as they are; a client written for OpenAI's route assumes 24
kHz; and since 2026-11-14 its language model runs on a device (3.7
seconds of sound a second on the RX 7900 XT with
--backend opencl, the 64-word text above), which takes away
most of what speed argued for where there is a GPU. Soprano is the one
to take on a CPU alone, or where a licence must allow commercial use:
the lines beside the examples say so.
- For: 2.7 to 4.0 seconds of sound a second on four threads of a loaded CPU against 0.08 to 0.23, and a first sound in about a second against half a minute (the table above); a licence that allows commercial use (Apache-2.0; OuteTTS 0.2 is non-commercial); 197 MiB against 636; punctuation, numbers, dates and abbreviations read; heard at least as well.
- Against: one voice and no speaker files (OuteTTS takes any voice made by its package); another voice than the one users of OuteTTS know; 32 kHz where a client written for OuteTTS may assume 24 kHz; a catalogue file converted by a third party (its decoder checked byte for byte against the author's checkpoint, its language model against llama.cpp, not against the author's weights).
- What switching would touch: the examples of
README.md,docs/speech.md,docs/server.mdanddocs/gui.mdand the quick start (--speak-model soprano-1.1-80m); the hints of the chat window and of the dashboard's page, and the tests that read them; the chat window's Voice chooser, which would list one voice (the window plays at the rate of the model that speaks, and picks the installed speech model by itself); nothing else in the dashboard, whose player reads the rate fromX-Sample-Rateandspeak_stats.sample_rate; the tests that nameoutetts-0.2-500mas the installed model (tests/server/test_server.c,tests/mm/test_speech.c), which stay as tests of OuteTTS.
Limits of Soprano
- One fixed voice, English. No speaker file, no
cloning (the model has no reference in its prompt); the package's
Soprano-Encoderrepository is for training. - The package itself was not run (see above): three implementations agree, none of them is PyTorch's.
- A sentence at a time. The model's prosody is a sentence's; between two sentences there is the pause the model ends and begins with, nothing added.
- The cleaner is English and literal: "9-5" is "nine minus five", years are guessed from the number alone, other scripts are dropped.
- Mispronounced words: the model card gives 1,000 hours of training speech; rare words ("vocoder") come out wrong, as with OuteTTS.
- The decoder's inverse transform on the CPU.
--backend opencl|vulkanputs the language model and the decoder's network on a device (on a device). - The decoder is about two fifths of the time on this CPU (F32 products of 768 x 2304, 62 frames a second): it was not tuned.
- Soprano 1.0 (
ekwek/Soprano-80M, a decoder of 58 MB with other sizes) was not tried: its checkpoint would go through--convert, whose metadata is read from the tensors' shapes.
The library
include/oklm/mm/speech.h:
int oklm_speech_open(const char* model, const char* vocoder, const oklm_speech_params* p,
oklm_speech** out, char* err, size_t err_size);
int oklm_speech_synthesize(oklm_speech* s, const char* text, const oklm_speech_options* o,
float** samples, size_t* n);
int oklm_speech_synthesize_chunks(oklm_speech* s, const char* text, const oklm_speech_options* o,
oklm_speech_chunk_fn fn, void* user);
int oklm_speech_set_voice(oklm_speech* s, const char* voice, char* err, size_t err_size);
int oklm_speech_voice_check(const char* path);
void oklm_speech_last_stats(const oklm_speech* s, oklm_speech_stats* out);
int oklm_speech_stretch(const float* samples, size_t n, int rate, float speed, float** out, size_t* n_out);
int oklm_speech_rate(const oklm_speech* s);
void oklm_speech_free(oklm_speech** s);
int oklm_speech_wav(const float* samples, size_t n, int rate, unsigned char** wav, size_t* size);
void oklm_speech_pcm16(const float* samples, size_t n, int16_t* out);
int oklm_speech_format(const oklm_speech* s); /* OKLM_SPEECH_OUTETTS_0_2, _0_3 or OKLM_SPEECH_SOPRANO */
oklm_speech_options oklm_speech_options_for(int format); /* the sampling a model is meant to run with */
int oklm_speech_probe(const char* model, const char* vocoder, int* format, int* rate);
int oklm_speech_normalize_soprano(const char* text, char** out);
int oklm_speech_states(oklm_speech* s, const char* text, const oklm_speech_options* o, float** states, size_t* n, int32_t** tokens);
int oklm_vocoder_decode_states(oklm_vocoder* v, const float* states, size_t n, float** samples, size_t* n_samples);oklm_speech_params holds the backend of the language
model ("cpu", "opencl", "vulkan",
"auto"), a speaker file and the context;
oklm_speech_options the sampling settings, the guide, the
words per pass (max_words), per prompt
(prompt_words) and of the first pass
(first_words), the speed, and a stop flag read
between tokens. oklm_speech_set_voice() gives an open
synthesizer another speaker file, or the built-in voice again;
oklm_speech_voice_check() says whether a file is a speaker
file. oklm_speech_last_stats() gives what the last
synthesis cost (oklm_speech_stats: prompts,
passes, prompt_tokens,
cached_tokens, generated_tokens,
prompt_seconds, generate_seconds,
vocoder_seconds, first_seconds,
seconds, audio_seconds). The samples are mono
floats in about [-1, 1] at oklm_speech_rate() (24000);
oklm_speech_wav() makes a 16-bit WAV file in memory and
oklm_speech_wav_write() writes one;
include/oklm/mm/flac.h has oklm_flac_encode(),
oklm_flac_writer_new() / _add() /
_free() and oklm_flac_decode(). A synthesizer
runs one synthesis at a time: callers that share one serialize their
calls. The pieces are public too: oklm_speech_normalize(),
oklm_speech_split(),
oklm_speech_split_first(),
oklm_speech_prompt_text(),
oklm_speech_prompt_tokens(),
oklm_speech_codes() (the language model's half) and the
vocoder alone (oklm_vocoder_load(),
oklm_vocoder_spectrum(),
oklm_vocoder_decode(), oklm_istft()).
A client written for OuteTTS runs Soprano unchanged when it reads the
rate from oklm_speech_rate() (32000) instead of
OKLM_SPEECH_RATE, takes its options from
oklm_speech_options_for(oklm_speech_format(s)) (or passes
none) instead of oklm_speech_default_options(), which are
OuteTTS's, and offers no speaker file when the format is
OKLM_SPEECH_SOPRANO. oklm_speech_probe() gives
the format and the rate from the files' headers, before anything is
loaded (what the server does). oklm_speech_codes() has no
meaning for Soprano (OKLM_ERR_UNSUPPORTED): its counterpart
is oklm_speech_states(), with
oklm_vocoder_spectrum_states(),
oklm_vocoder_decode_states() and
oklm_istft_center() for the decoder alone, and
oklm_vocoder_probe() to tell the two kinds of vocoder file
apart.
Tests
tests/mm/test_speech.c, with fixtures written by
tools/gen_speech_fixtures.py (numpy only): the
normalization of eleven texts and the prompts of three with a small
speaker file, against the script's own implementation; the cut into
passes and its shorter first pass; the change of speed on a tone
(length, pitch, level, the first samples); FLAC files of ten lengths and
five kinds of sound read back sample for sample, a stream written in
pieces, damaged files refused; the inverse transform against numpy's
irfft for sizes 40, 28 and 1280 (factors 2, 5 and 7); a
tiny random vocoder in the published file's layout (the same tensor
names, shapes and types) against the script's double-precision network,
rows and samples; WAV files read back; the refusals (a backend the
machine does not have, a file that is not a vocoder, a language model
that is not a speech model, codes outside the codebook). When
models/outetts-0.2-500m is there: the prompt's ids against
llama.cpp's (their count and hash), the vocoder's rows and samples on 32
codes against the numpy implementation run on the published file, and a
word spoken end to end. The vocoder's network on each device backend
found against the CPU's samples, on the tiny file
(test_vocoder_devices) and on the published one with the
codes of "Hello world" (test_published_devices): the ratio
of the sound to the difference, a shorter then a longer call, a second
call, the vocoder back on the CPU, a device with no room and
OKLM_MM_DEVICE_ENDS=0; and, on the tests' counting device
with a bound (tests/mm/count_backend.h, the CPU's kernels
behind a device's interface), a device with room for the network's
weights and not for a call's activations: the call is computed on the
CPU, to its bits, the network staying there. The passes of a text under
one prompt and under one each (their counts, the tokens kept in the
cache, the same sound), a voice changed and one refused are in
tests/server/test_server.c
(test_speech_route), on the tiny speech model that test
builds, with the route's flac, speed,
voice and GET /v1/audio/voices.
Soprano: tests/mm/test_soprano.c, with fixtures written
by tools/gen_soprano_fixtures.py (numpy only, in
tests/fixtures/mm/tiny-soprano/): the cleaner on 386 texts
against the script's; the inverse transform with "center" padding
against numpy's irfft for sizes 16, 40 and 2048; a tiny
random decoder in the published file's layout against the script's
double-precision network, rows and samples, and its sound in pieces
against its sound in one call; a tiny language model with Soprano's kind
of vocabulary that writes twelve tokens after any prompt, on which the
whole synthesis runs: the prompt's ids, the states, a prompt per
sentence, the short sentences put together, a long one cut, the first
sound early with the same samples, the speed, a stop, and the refusals
(a speaker file, codes, a language model and a vocoder that do not go
together). When models/soprano-1.1-80m is there: the
prompt's ids and the first tokens written against llama.cpp's, the
decoder on the states (and on each device backend against the CPU's
samples, as the tiny decoder is by test_decoder_devices), a
sentence spoken with its first sound early.
test_speech_route_soprano of
tests/server/test_server.c runs the route on the tiny
files: 32 kHz in the headers and the WAV file, the synthesizer's own
samples as a file and streamed, the one voice.
Other models
What running each would take, from their model cards and repositories; no code here.
Soprano is no longer here: it runs (Soprano).
Kokoro (hexgrad/Kokoro-82M; Apache-2.0;
8 languages, 54 voices, 24 kHz). 82 M parameters as a PyTorch checkpoint
(kokoro-v1_0.pth, 327 MB) and one tensor file per voice;
built on StyleTTS 2 with an ISTFTNet vocoder. It reads phonemes, not
text: its package uses the misaki grapheme-to-phoneme
library with eSpeak NG behind it. It takes a converter from the
checkpoint, kernels OKLM does not have (bidirectional LSTMs for
durations and prosody, transposed convolutions, adaptive instance
normalization), and above all a grapheme-to-phoneme front end per
language. No GGUF and no llama.cpp support: the reference would be its
Python package.
Piper (rhasspy/piper-voices; the
voices' licences vary with their datasets, the repository's card says
MIT; more than 30 languages). One VITS model per voice, published as
ONNX (.onnx and a .onnx.json with the phoneme
table), 16 or 22.05 kHz, fast on a CPU. It also reads phonemes, made by
eSpeak NG (GPL-3.0), which OKLM would have to call as a program or
replace. It takes an ONNX reader or a converter, and VITS's blocks: a
transformer text encoder with relative positions, a duration predictor,
normalizing flows, a HiFi-GAN decoder (transposed convolutions). The
reference would be ONNX Runtime.
OuteTTS 1.0 (Llama-OuteTTS-1.0-1B,
CC-BY-NC-SA-4.0) changed the audio codec (DAC) and is another vocoder.
OuteTTS 0.3 also exists as a 1B model
(OuteAI/OuteTTS-0.3-1B-GGUF, CC-BY-NC-SA-4.0 with its Llama
base), which the same code would open and the catalogue does not
list.
On a device
oklm speak --backend opencl|vulkan|auto
(oklm_speech_params::backend) creates the language model's
context on a device: the prompt, the voice kept in the cache and every
token written are computed there as for any model of its architecture,
the logits (Soprano: the hidden states too) read back a token. The
vocoder's network (WavTokenizer's decoder, Soprano's decoder) computes
there too since 2026-11-14 (the
vocoder on a device); the inverse transform stays on the CPU.
Where the time of a sentence of 7 seconds goes (RX 7900 XT, measured
again on 2026-11-15, load average 1.9 to 3.7: a sentence of 23 words,
--seed 1, two runs each, seconds; the prompt, the tokens
written, the vocoder, then the whole command's synthesis; in the two
device columns the vocoder's network is on the device, its group
normalizations and attention block included, and the inverse transform
is on the worker team in all three):
| Model | CPU | OpenCL | Vulkan | |
|---|---|---|---|---|
| OuteTTS 0.3 500M Q8_0 | prompt of 907 tokens | 0.56 to 0.59 | 0.05 | 0.05 |
| 603 to 648 tokens written | 12.86 | 1.77 to 1.79 | 1.39 | |
| vocoder | 0.22 | 0.02 | 0.02 | |
| the sentence | 13.67 to 13.69 | 1.90 to 1.92 | 1.48 | |
| Soprano 1.1 80M Q8_0 | 109 to 113 tokens written | 0.30 to 0.31 | 0.18 | 0.13 |
| decoder | 0.11 | 0.02 | 0.02 to 0.03 | |
| the sentence | 0.43 to 0.44 | 0.21 to 0.22 | 0.17 |
On the CPU the language model is 94 % of OuteTTS's time and the vocoder 1.6 %; on a device the sentence is 3.5 to 3.6 s of sound a second of work on OpenCL and 4.7 on Vulkan against 0.54 on the CPU, and the vocoder is 1 % of it. While the vocoder stayed on the CPU (the same day's first table: 0.31 to 0.33 s of a sentence of 1.83 to 2.25 s) it was 15 to 17 % of the sentence. Soprano speaks 15.6 to 15.9 s of sound a second on the CPU, 32.9 to 33.5 on OpenCL and 42.4 to 43.1 on Vulkan; its decoder is a quarter of its time on the CPU and a tenth on a device.
The sound is not the CPU's sample for sample: a device's products sum
in another order, the codes of a sound are many near ties, and a code
that differs changes every code after it. Measured
(test_published_devices in
tests/mm/test_speech.c): on the prompt of "Hello world" and
the 96 tokens the CPU then writes greedily, the device's best token is
the CPU's in 92 of 96 steps for OuteTTS 0.2, 94 or 95 for OuteTTS 0.3
and 95 for Soprano, and where they differ the CPU's margin between the
two tokens is 0.063 at most; the codes of a word written greedily are
the CPU's for the first 42 (OuteTTS 0.2) and 6 or 8 (OuteTTS 0.3), of
the same count. Soprano's tiny model gives the CPU's hidden states on
both backends (test_tiny_devices in
tests/mm/test_soprano.c).
Limits: a backend the machine does not have is refused.
The vocoder on a device
oklm_vocoder_use_backend() copies a vocoder's network to
a device (oklm speak --backend and a server's
--speak-backend do it when the language model has layers
there), and oklm_vocoder_spectrum() and
oklm_vocoder_spectrum_states() then compute a call's rows
there, one pass on the device a call. Where a call's time went on the
CPU (RX 7900 XT host, 6.4 seconds of sound in two passes, milliseconds):
WavTokenizer's network 97 and 150, its inverse transform 20 and 32;
Soprano's network 39 and 53, its inverse transform 26 and 37.
The network is products, norms and activations the backends have, and
convolutions, which they have not as an op. No op was added
(src/mm/vocoder.c, vd_forward()):
- A convolution of kernel k is k products: the kernel
is copied to the device as one matrix a position, and each product reads
a view of the rows that position reads, the first written, the others
added (
OKLM_OP_MATMULaccumulating), with the activations at F32's precision (oklm_matmul_wide()). An activation a convolution reads is kept between four rows of zeros at each end: what the kernel reads past the frames (kernels up to nine positions; a longer one leaves the vocoder on the CPU). - The depthwise convolution of a ConvNeXt block is, for each position of its kernel, the rows that position reads times that position's weights, a vector, added to the bias then to the sum so far: the CPU's order.
- Group normalization (WavTokenizer's "posnet" stack, ten of them a call) reduces over a group's channels and every frame. A frame holds a group's channels side by side, so a view of the stream [channels a group, frames, groups] names them, and a copy through it lays each group out as one row; a layer normalization of those rows is the group's mean and variance; a copy through the same view of the destination puts the values back, and the scale and the shift by channel are a product and a sum (five launches, six with the swish).
- The attention block of that stack has one head of 768 values, wider than Vulkan's attention shaders take (256). It is computed as products on both backends: the keys read as a matrix times the queries are the scores, their softmax the weights, and the values copied by channel (a transposed view) times the weights the output. The products run over the frames rounded up to 32, the keys and the values past the frames rows of zeros.
- The lookup of the codes, Soprano's interpolation of the states and the inverse transform stay in host memory, the last on the worker team (above).
Measured on 2026-11-14 (RX 7900 XT, --seed 1, the
sentence of 23 words of the test, 6.3 to 6.5 seconds of sound, two runs
each; the vocoder's time, then the whole synthesis, with the language
model on the device in every column):
| Model | Vocoder on the CPU, as before | The transform on the team | The network on the device | |
|---|---|---|---|---|
| OuteTTS 0.3 500M Q8_0 with WavTokenizer | Vulkan | 0.30 s, 1.67 s | 0.25 to 0.26 s, 1.61 to 1.62 s | 0.06 s, 1.42 s |
| OpenCL | not run | 0.26 s, 2.07 to 2.10 s | 0.05 s, 1.89 to 1.90 s | |
| Soprano 1.1 80M Q8_0 | Vulkan | 0.16 s, 0.28 s | 0.10 s, 0.23 s | 0.02 s, 0.14 s |
| OpenCL | not run | 0.10 s, 0.25 s | 0.02 s, 0.16 to 0.17 s |
WavTokenizer's network takes 23 and 29 ms on Vulkan and 17 and 25 ms on OpenCL for the two passes (97 and 150 on the CPU), Soprano's 4.8 and 5.2 ms and 3.5 and 3.9 ms (39 and 53). Of WavTokenizer's, the "posnet" stack with its group normalizations on the CPU was 16 and 21 ms on Vulkan and 10 and 16 ms on OpenCL. With the group normalizations and the attention block on the device (2026-11-14, a sentence of 23 words in one pass of 507 to 527 frames, two runs each) the network takes 14.2 to 14.3 ms on Vulkan where it took 43.9 to 44.6 ms, and 13.5 to 13.7 ms on OpenCL where it took 32.9 to 33.8 ms; the 16-bit files differ from those of the normalizations on the CPU by one step in 207 and 272 of 168 640 and 162 240 samples (99 and 97 dB). For that sentence the command's vocoder time went from 0.05 s to 0.02 s on Vulkan and from 0.04 s to 0.02 s on OpenCL, and the sentence from 1.51 s to 1.47 to 1.48 s and from 1.90 to 1.95 s to 1.88 to 1.92 s; the table above is from before.
The sound against the CPU's vocoder on the same codes or states: the
16-bit files of the sentence above differ by one step at most, in 263 to
1 267 of 153 280 to 200 704 samples (90 to 97 dB). In floats
(test_vocoder_devices, test_published_devices
and test_decoder_devices): WavTokenizer on the codes of
"Hello world" within 1.2e-6 on OpenCL and 5.9e-6 on Vulkan (125 and 110
dB), Soprano's decoder on a sentence's states within 3.5e-6 and 3.3e-6
(112 and 113 dB), the tiny files 117 to 132 dB; the tests ask for 80 dB,
a second call to give the first's bits, and the vocoder back on the CPU
the CPU's. The convolutions' and the depthwise form's ops, the group
normalization's and the one head's are checked against the CPU's
arithmetic by test_conv_views
(tests/core/test_opencl.c,
tests/core/test_vulkan_gemm.c; RX 7900 XT and Intel UHD
750). The language model's tokens are what they were: its figures above
are measured again by the same test with the vocoder on the device (92
of 96, 94 or 95, 95).
Limits: the device holds the network's weights (WavTokenizer's file
is its size) and a call's activations, which a longer call asks for when
it comes (oklm_vocoder_device_bytes()); a device without
room for the weights leaves the vocoder on the CPU with a notice, one
without room for a call's activations leaves that call to the CPU
(WavTokenizer's attention adds a row of the frames a frame to them: 17
MiB at the 2 048 frames a server counts, 1 MiB for a sentence of 7 s). A
call reads nothing back but its spectrum.
OKLM_MM_DEVICE_ENDS=0 keeps the network on the CPU (build.md). On the Intel UHD 750 the tiny files ran
on both backends (118 to 131 dB), no published vocoder.
A server's synthesizer takes a device with
--speak-backend (cpu by default): each token
of its language model is a pass on the device, and each call of its
vocoder, that takes its turn with the scheduler's steps (server.md); a request
streaming a completion meanwhile saw no pause (its tokens at most 0.02
to 0.03 s apart, 0.007 to 0.011 s usually), and a sentence of 2.9 s was
spoken in 0.76 s (OuteTTS 0.2, Vulkan) and 0.90 s (OpenCL), one of 2.6 s
in 0.13 to 0.16 s by Soprano on either (measured with the vocoder on the
CPU). With the vocoder's network on the device, a pass of 4 to 29 ms a
call that lets a waiting pass in after its first stack, the stream's
tokens were at most 0.03 s apart for OuteTTS 0.2 and 0.02 s for Soprano
on both backends (0.04 to 0.07 s and 0.02 s before the passes ran in the
order they ask; 2026-11-14). The server leaves the synthesizer its room
on the device before it sizes its cache
(oklm_speech_device_bytes(): the language model's weights,
cache and activations, the vocoder's network and the activations of its
longest call): 903 and 259 MiB counted for OuteTTS 0.2 with
WavTokenizer, 122 and 226 MiB for Soprano, where the device gave 1 070
to 1 078 and 270 to 274 MiB after a first sentence (2026-11-15; 903 and
243, 122 and 244 counted before WavTokenizer's attention had its rows
there and the activations were counted as twelve) (server.md).