Multimodal
Images, sounds and speech given to a language model: decoding, the vision and audio encoders whose embeddings a model reads in place of tokens, speech recognition with Whisper, and how each is checked against llama.cpp. Video containers follow adr/0007-video-containers.md.
Status: implemented: images (SigLIP encoders with the idefics3
projector: SmolVLM 256M, 500M, SmolVLM2 2.2B; with the gemma3 projector:
Gemma 3 4B; the CLIP encoder with the mlp projector: LLaVA 1.5 7B, and
LLaVA 1.6 Mistral 7B with its grid of tiles; the encoders of Qwen2-VL
and Qwen2.5-VL at the image's own resolution: Qwen2-VL 2B, Qwen2.5-VL
7B), sounds (Ultravox), both through the server too, speech recognition
(Whisper), documents (text, JSON, HTML, DOCX, PDF), videos (Ogg Theora
natively, other containers through ffmpeg when allowed),
oklm describe, oklm transcribe,
oklm ingest, on Linux and Windows.
Using it
oklm describe -m models/smolvlm-256m/SmolVLM-256M-Instruct-Q8_0.gguf \
--mmproj models/smolvlm-256m/mmproj-SmolVLM-256M-Instruct-f16.gguf \
-p "What does the newspaper say?" photo.jpg
oklm describe -m models/gemma3-4b/gemma-3-4b-it-Q4_K_M.gguf \
--mmproj models/gemma3-4b/mmproj-model-f16.gguf \
-p "What does the newspaper say?" photo.jpg
oklm describe -m models/llava-1.5-7b/ggml-model-q4_k.gguf \
--mmproj models/llava-1.5-7b/mmproj-model-f16.gguf \
-p "Describe the picture in detail." photo.jpg
oklm describe -m models/llava-1.6-mistral-7b/llava-v1.6-mistral-7b.Q4_K_M.gguf \
--mmproj models/llava-1.6-mistral-7b/mmproj-model-f16.gguf \
-p "What does the newspaper say?" photo.jpg
oklm describe -m models/qwen2.5-vl-7b/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
--mmproj models/qwen2.5-vl-7b/mmproj-F16.gguf \
-p "What does the newspaper say?" photo.jpg
oklm describe -m models/ultravox-1b/Llama-3.2-1B-Instruct-Q8_0.gguf \
--mmproj models/ultravox-1b/mmproj-ultravox-v0_5-llama-3_2-1b-f16.gguf \
-p "What is said?" speech.wav
oklm transcribe -m models/whisper-base speech.wav
oklm transcribe -m models/whisper-small --format srt talk.ogg > talk.srt
oklm transcribe -m models/whisper-small --language fr --translate fable.ogg
oklm describe takes images, sounds and videos (by
extension: .wav, .mp3, .flac,
.ogg, .opus are sounds; .ogv,
.mp4, .mkv, .webm,
.mov, .avi, ... videos), puts each where the
prompt has the marker <__media__> (or before the
prompt; a sound for Ultravox after it, see the table below), and
generates -n tokens (256). Without -p the
question is Describe this image.,
Describe this video. when a video is among a vision model's
files, and Describe this sound. for an audio model (it was
the image's for all three until 2026-11-14). --tiles splits
large images into tiles plus an overview, as SmolVLM was trained;
without it the overview alone is encoded. Gemma 3 reads one image of 896
pixels and LLaVA 1.5 one of 336, whatever the option says. LLaVA 1.6
reads an overview of 336 pixels and the tiles of the image fitted in the
resolution of its file's list that suits it best (two to four tiles: 577
rows for the overview, then up to 2 352 more), whatever SIDE says,
except that a SIDE up to 336 gives the overview alone (577 rows: a
prompt four times shorter, without the detail the model was trained to
read). Qwen2-VL and Qwen2.5-VL read one image at its own size, as many
embeddings as it has squares of 28 pixels, up to 4 096 (3.2 megapixels,
llama.cpp's limit; a larger picture is reduced to that);
--tiles SIDE caps the largest area at SIDE x SIDE pixels
instead (--tiles 1002 for 1 280 embeddings, the default
before 2026-11-14: the encoder's attention grows with the square of
their number, 10 s for a picture of 5 megapixels on an RX 7900 XT and
one to three minutes on the CPU). Without -c,
oklm describe makes the context as long as the pictures'
rows need. oklm transcribe writes what a sound file says
(speech recognition below):
| Option | |
|---|---|
--language CODE |
the language spoken (en, fr,
de, ...: the codes of the model's language tokens; an
unknown one prints the list). auto, the default: the model
finds it from the first 30 seconds, and the line
[language: fr (0.99)] on the error stream says which, with
its probability |
--translate |
write English whatever the language spoken (Whisper's translation task; not large-v3-turbo's strength: it was trained without it) |
--format text |
the words (the default) |
--format timed, --timestamps |
a line a segment:
[00:00:06.000 --> 00:00:11.360] text |
--format srt, --format vtt |
subtitles (SubRip, WebVTT) |
--format json |
task, language,
language_probability (when found), duration,
text and segments (id,
seek, start, end,
text, avg_logprob,
no_speech_prob): the route's verbose_json |
--backend cpu|opencl|vulkan|auto |
where the model computes (cpu by default): on a device
the encoder's blocks and the decoder run there (on a device) |
--no-timestamps |
decode without timestamp tokens: a cut every 30 seconds wherever that falls, one segment a cut |
--detect-language |
print the language and its probability (fr 0.997) and
transcribe nothing: one pass of the encoder, one decoder step |
The other way, a text spoken into a sound with
oklm speak, is in speech.md.
oklm serve --mmproj FILE takes the same images and
sounds in chat messages, as base64 image_url and
input_audio parts, their rows cached by the media's content
hash (server.md). A turn's
text around its media is written by one function for both,
oklm_mm_content(): each image or sound a marker, joined to
the parts around it as the projector's model writes them, so
oklm describe -p QUESTION picture and a chat request
holding the picture then the question give the model the same tokens
(tests/server/test_server.c, on the tiny projector of each
family: as many rows, the same greedy answer, and describe's prompt sent
as a raw completion finds every row but the last in the prefix cache).
The retrieval's captions (rag.md) are written by
it too. What follows a marker that another part follows, by
projector:
| Projector | After a marker | The reference |
|---|---|---|
idefics3 (SmolVLM) |
nothing | the template in the model's file, given content in parts:
User:<image>What is this? |
gemma3 |
nothing | the template in the model's file, given content in parts:
<start_of_image>What is this? |
qwen2vl_merger, qwen2.5vl_merger |
nothing | the template in the model's file:
<|vision_start|><|image_pad|><|vision_end|>What is this? |
mlp (LLaVA 1.5, LLaVA 1.6) |
a new line | the model's own conversations: <image>\n before
the question. The files' templates take a string, not parts (LLaVA 1.5's
files have none) and say nothing of it |
ultravox |
a new line; and a space before a sound that comes right after a text | the model's own pipeline (ultravox_pipeline.py): the
sound at the end of the user's turn, a space after the text,
Transcribe this. <|audio|>. The language model's
template takes a string and says nothing of a sound; a text after a
sound, which the pipeline never writes, keeps the new line |
| any other | a new line | none: LLaVA's form |
oklm describe -p QUESTION puts the pictures before the
question, and for Ultravox the question before the sound, the pipeline's
order. The order decides the answer: asked "Transcribe this." about
jfk.wav, Ultravox v0.5 with Llama-3.2-1B (Q8_0, CPU, greedy) writes the
sentence when the sound follows the question ("Here's the transcribed
text: "And so, my fellow Americans ask not what your country can do for
you, ask what you can do for your country.""), and quotes "My Country,
'Tis of Thee" when the question follows the sound, which is what
oklm describe wrote until 2026-11-14. A chat request keeps
the order of its parts: the question first, then the
input_audio part, is describe's prompt (228 tokens by
either way, the same answer; test_sound_describe in
tests/server/test_server.c on the tiny encoder: the same
rows).
Otherwise nothing is put before a marker, and nothing between two
texts. llama.cpp at the pinned revision joins a marker and a text with
nothing for every projector (common/chat.cpp;
mtmd-cli writes the markers right before the prompt): for
SmolVLM, Gemma 3 and Qwen it writes what the templates write, for LLaVA
it leaves out the model's new line. Gemma 3 has a second reference,
which is not followed: the Transformers processor replaces the image's
place in the rendered template by two new lines, the image's tokens and
two more new lines, an empty line on each side of the image that neither
the template nor llama.cpp writes. OKLM writes the template's form, and
llama.cpp's recorded answer for Gemma 3 4B is made from it
(tools/gen_mm_ref.sh). oklm_mm_build() gives a
prompt's rows without evaluating them (tokens, and each media row's
content hash and index, with the embeddings; an item may be named by the
hash an earlier build gave it, oklm_mm_item.hash, and is
then built from the embeddings kept of it with no pixel or sample read,
which is how the server skips decoding a part it has seen),
oklm_mm_eval_rows() evaluates a prompt and names its rows
the same way, without keeping the embeddings, and
oklm_image_decode() / oklm_audio_decode() read
files held in memory, oklm_video_decode() a video (written
to a private temporary file under TMPDIR for the time of
the call: the decoders read a path).
A conversation kept in a file
oklm describe -m MODEL --mmproj MMPROJ -p "Describe this image." --kv-save moon.kv moon.jpg
oklm describe -m MODEL --mmproj MMPROJ --kv-load moon.kv -p "What colours are in it?"
oklm describe -m MODEL --mmproj MMPROJ -p "Describe this image." --then "What colours are in it?" moon.jpg
oklm describe -m MODEL --mmproj MMPROJ --kv-load moon.kv -p "Is this a part of it?" crater.jpg
oklm describe -m MODEL --mmproj MMPROJ -p "Describe this image." --then "Is this a part of it?" --then-with crater.jpg moon.jpg--kv-save FILE writes the conversation's cache after the
answer: the rows of the prompt, the image's among them, and of the
answer, as a KV file
(oklm_kv_save_media(): each row with its token or its media
hash, and for Qwen2-VL and Qwen2.5-VL the positions of its rows).
--kv-load FILE -p QUESTION, in a later run, takes the rows
back (oklm_kv_load_media()) and asks the question as the
conversation's next turn: no picture is given and none is encoded, the
model evaluates the question's tokens only. --then QUESTION
asks a second question after the first answer in the same run. The three
combine (--kv-load a.kv -p ... --kv-save a.kv goes on turn
by turn). A next turn may bring media of its own (2026-11-14; it was
text only): the files named with --kv-load go with its
question, and --then-with FILE (repeatable) gives one to
the question of --then; a picture, a sound or a video, laid
out as in a first turn (the items beside the question, in the model's
order; a video's frames announced for an encoder of still images). Their
rows take the positions that follow the conversation's: for Qwen2-VL and
Qwen2.5-VL the three-part positions from the sequence's counter, which
the file restores. Without -c the context is sized for the
pictures of both questions of one run; after --kv-load it
is the default (4 096) or -c. The next turn's text is what
the model's chat template writes after an answer for a new user message;
a template that writes a longer conversation another way is refused with
a message. SmolVLM's own format, which ends its prompt at
Assistant: and writes a kept answer after a space, was
refused so until 2026-11-14, with --then and with
--kv-load alike: the space is the one the model's answer
begins with, and the conversation goes on (SmolVLM-256M: a second
question, with and without a picture of its own, answered from a file as
in one run). Each answer is sampled from a sampler of its own, so a
second answer does not depend on whether the first was generated in the
same run.
Run on Qwen2-VL 2B (Q4_K_M, the CPU, --temp 0 -n 64, the
picture of test_mm_public's answer case, a newspaper's
front page): the first answer, saved (480 rows, 391 of them the image's,
13.8 MB), then the second question in a new process from the file, give
the 64 and the 8 tokens' text of the one run with --then,
byte for byte; the new process evaluated 16 tokens where the prompt had
416 rows. The file is told which rows are the media's by the evaluation
itself (oklm_mm_eval_rows(): the prompt evaluated, each row
named by its token or its item's hash and index as
oklm_mm_build() names it), so the encoders run once: the
same picture with -n 16 and --kv-save took
19.5 s before that, the encoders run a second time for the names, and
takes 10.4 s, the file the same bytes (434 rows, 12.5 MB).
A next turn with a picture, the same model and settings (the front
page at 448 x 336, What is the headline?, then its top left
corner enlarged,
Is this second picture a part of the first one? Say what it shows.):
saved after the first answer (227 rows, 6.5 MB), the second question and
its picture in a new process (192 embeddings, 220 rows evaluated after
the 227 loaded) give the text of the one run with
--then ... --then-with, byte for byte ("The headline is
"Men Walk on Moon."", "Yes, this is a part of the first image."). With
the clip as the next
turn's media (What happens in this video?, 384 embeddings
in 402 rows after the 225 loaded): the same text either way again ("In
this video, we see a newspaper with the headline "Men Walk on Moon."
[...]"). On the tiny qwen2vl model
(test_kv_file's test_next_turn_media): a turn
with a picture saved, then a text, a second picture or a video and a
text after the load, contiguous and paged either way, give the logits
and the position counter of the context that never left memory, bit for
bit.
In the terminal chat
oklm chat -m MODEL --mmproj MMPROJ # then: /attach moon.jpg, and a message
oklm chat -m smolvlm-256m # a catalogue model that carries a projector: no optionoklm chat takes the same projector
(--mmproj FILE|NAME|none, --tiles,
--fps, --frames, --ffmpeg) since
2026-11-15, and its /attach FILE a picture, a sound or a
video that goes with the next message, read by the code that reads a
question's file here (a video is its sound, one item, or announced
frames, by the encoder). The chat's turn is the conversation rendered
whole and made rows by oklm_mm_build(); the rows are
compared with the context's, a media row by its item's hash and its
index within it, and what follows the common prefix is evaluated: tokens
through oklm_eval(), an item's rows through
oklm_eval_embd(), oklm_eval_embd_mutual(),
oklm_eval_embd_grid() or
oklm_eval_embd_video() as its model reads them. A media
item is never cut by the prefix: it is in the cache whole, or evaluated
from its first row. Each item is encoded once: its embeddings are kept
by its hash for the later turns' prompts, and the pixels or samples are
freed. --kv-save writes the rows as --kv-save
does here (oklm_kv_save_media(), appended turn by turn) and
--resume takes them back
(oklm_kv_load_media()), the conversation's file naming the
media files to encode again. The commands, what they do with media in
the conversation and the measured runs (SmolVLM-256M, Qwen2-VL 2B, Gemma
3 4B, Ultravox) are in inference.md.
Video
A video becomes frames for the vision encoder or its sound for the
audio encoder, whichever the projector file has (adr/0007-video-containers.md).
For a vision encoder each frame is announced by its time
(Frame at 0:03:), the question after them. For an audio
encoder (Ultravox) the video is its sound track and nothing else: one
sound, placed where the model's own order puts a sound, after the
question and a space (the table above), so
oklm describe -p "Transcribe this." talk.ogv is the prompt
of the same command on the sound alone; a video without sound is refused
there. Until 2026-11-14 the sound had lines of its own before the
question (Video talk.ogv (11.0 s):, Sound: ),
which is not the order of Ultravox's own prompt: jfk.wav as the sound
track of an Ogg video, asked "Transcribe this." (Ultravox v0.5 1B, the
CPU, greedy), was 243 prompt tokens and an answer that spoke of the
quote before giving a version of it ("The given text appears to be a
quote from John F. Kennedy's inaugural address [...]"); it is 228
tokens, the count of the sound file alone, and the sentence ("[...] "And
so, my fellow Americans ask not what your country can do for you, ask
what you can do for your country." This is a famous quote from John F").
No model here is given a video's frames and its sound together: a
projector file holds one encoder, and Qwen2-VL and Qwen2.5-VL have no
audio one. Qwen2-VL and Qwen2.5-VL read a video as their own processors
and code define one, the frames two by two and no text between them (below):
oklm describe -m models/smolvlm-256m/SmolVLM-256M-Instruct-Q8_0.gguf \
--mmproj models/smolvlm-256m/mmproj-SmolVLM-256M-Instruct-f16.gguf \
--frames 4 -p "What happens in this video?" clip.ogv
oklm describe ... --ffmpeg -p "What does the speaker say?" talk.mp4
- Ogg Theora with Vorbis sound (
.ogv) is decoded by OKLM (libtheora, libvorbis and libogg, the libraries of Allegro's video addon), as fast as they decode, without an audio device. - Any other container goes through
ffmpeg, only with--ffmpegorOKLM_FFMPEG=1(orOKLM_FFMPEG=/path/to/ffmpeg): frames as PPM images on a pipe, the sound as 16 kHz PCM, an argument vector and no shell. Without it, such a file is refused with both ways out named. --fps F(1) frames a second at most,--frames N(8, at most 32) frames kept: past N, the interval doubles and every other frame goes, so memory stays bounded; the frames kept are the first, the last and evenly spaced ones between.- A reader may bound what it decodes
(
oklm_video_params.max_pixels): the frames asked for times a frame's width and height, checked when the stream's headers give the size (the first picture's, through ffmpeg), before a frame is decoded; above it the video is refused (OKLM_ERR_SHAPE) with the figures ("8 frames of 3840 x 2160 are 66.36 megapixels, above the limit of 16.00: fewer frames or a smaller video").oklm describesets none; the server does (server.md). - The lines that give the frames to an encoder of still images
(
Video clip.ogv (4.0 s):, thenFrame at 0:03:and the frame's marker, a line each) are written by one function,oklm_mm_video_text(), foroklm describeand for the server's chat routes, which have no file name and writeVideo (4.0 s):.
SmolVLM-256M reads the headline of a still Theora video of llama.cpp's test image from two frames; Ultravox quotes the speech of an MP4 read through ffmpeg.
The server takes a video the same way, as a video_url
part of a chat request (/v1/chat/completions; a
video block in /v1/messages), base64 in a
data: URL: the same decoder, the same sampling by default
with fps and frames on the part, the same
items and the same text for each kind of projector, so a request holding
a video and a question is the prompt oklm describe writes
for them, and, for Qwen2-VL and Qwen2.5-VL, the same rows. The decoded
frames are bounded there (server.md).
Documents
oklm_ingest_file() turns a document into the text a
model reads, by its extension, then its content:
| Document | Text |
|---|---|
| Plain text, Markdown, CSV, TSV, code | as it is, without a byte order mark or carriage returns |
| JSON | checked, indented two spaces a level |
| HTML | rendered by nilorea's n_html: scripts and styles dropped, entities decoded, blocks on their lines |
| DOCX | the paragraphs of word/document.xml (read by nilorea's
n_zip): headings as # lines, list items as -
lines, tables as Markdown rows, text boxes once |
| the text of each page in order, lines and spaces from where the glyphs land, pages apart by a blank line (below) |
oklm ingest report.docx
oklm run -m model.gguf --doc report.docx -p "Summarize this report."
--doc puts the text before the prompt, between
<document> tags. The agent's Read tool shows a Word
document's text the same way, and a PDF's pages with its
pages argument (Edit and Write refuse both). A PDF's text
is read (below); scanned pages, images without text,
are not.
Decoding
| Input | Without Allegro 5 | With Allegro 5 |
|---|---|---|
| Images | PNM (P5, P6) | PNG, JPEG, BMP, TGA, PCX (allegro_image) |
| Sounds | PCM WAV (8, 16, 24, 32-bit integer, 32-bit float) | FLAC, Ogg Vorbis, Opus, MP3 (allegro_acodec) |
Allegro decodes into memory bitmaps and samples, without a display.
The Windows build gets it from the Allegro project's MinGW-w64 release:
make windows-deps downloads it and its dependency pack into
external/mingw (checksums checked), after which
make MINGW=1 builds with the POSIX-threads MinGW-w64
compiler and Allegro (oklm-gui.exe too), and
make windows-dist gathers the executables and every DLL
they need in build/mingw/dist/oklm. Every multimodal test
passes under Wine. Sounds are mixed to mono and resampled to 16 kHz with
a Kaiser-windowed sinc (32 zero crossings, unit gain at DC). An output
sample sits at one of a few places between two input samples (160 from
44.1 kHz, one from 48 kHz), and the filter's weights depend on that
place alone: they are computed once per place, not once per sample
(reading 52 seconds at 44.1 kHz: 17 s before, about one now; the same
values within 1e-6). Images are resized with a separable Lanczos-3
filter (bilinear and bicubic also), widened when shrinking.
Vision
oklm_vision loads a llama.cpp mmproj GGUF with a vision
encoder:
- SigLIP: patches embedded as one matrix product, learned positions, pre-norm blocks with GELU (tanh), a final norm; a tile side that is not a multiple of the patch loses its last pixels, as the unpadded convolution does (SmolVLM2-2.2B: 384 pixels, 27 patches of 14);
- the idefics3 projector: a pixel shuffle (each output token concatenates a scale x scale group of patches) and one linear layer;
- the gemma3 projector: the mean of each 4 x 4 group of patches, an
RMS norm with a weight (
mm.soft_emb_norm), one linear layer (mm.input_projection, stored with its sides exchanged); - CLIP (LLaVA 1.5): a class token before the patches, patches without a bias, a norm before the blocks, quick-GELU (x sigmoid(1.702 x)), no norm after the blocks;
- the mlp projector: the patches without the class token through two
linear layers (
mm.0,mm.2) with the exact GELU between them; in a file that lists resolutions (clip.vision.image_grid_pinpoints, LLaVA 1.6), the same for an overview and for each tile, the embeddings then joined in one grid (oklm_vision_pack(), below); - the encoders of Qwen2-VL and Qwen2.5-VL
(
qwen2vl_merger,qwen2.5vl_merger): an image of any size in units of 28 pixels, two patch kernels summed, two-dimensional rotary positions in the attention, for 2.5 RMS norms, a gated MLP and attention in windows; a merger joining each 2 x 2 group of patches (below).
Any other clip.projector_type is refused when the file
is loaded, and so is an mlp file with chosen feature layers
(clip.vision.feature_layer) or one whose tiles are joined
otherwise than LLaVA 1.6's (clip.vision.mm_patch_merge_type
other than spatial_unpad).
The tiling is SmolVLM's: the image is resized so its longest side is
the largest the file allows, rounded up to whole tiles, cut into tiles,
and an overview of the whole image is added. The prompt is idefics3's:
each tile between
<fake_token_around_image><row_R_col_C> markers,
a new line per row, then <global-img> and the
overview. SmolVLM's own chat template reads only structured message
content, so oklm_mm_format() writes its turns
(User: ... <end_of_utterance>
Assistant:) itself.
The embeddings go into the language model through
oklm_eval_embd(): their rows replace the token lookup,
after which the model runs as usual.
The encoder on a device. An encoder computes on the
CPU unless oklm_vision_use_backend() gives it a device: its
weights are then copied there (one buffer, with another for a tile's
activations), and oklm_vision_encode() computes each tile
on it through the same ops (norms, products, bidirectional attention,
the activation): the tile's patches are uploaded and the projector's
rows read back. The patch embedding before the blocks and the projector
after them compute there with the blocks since 2026-11-14 (below; the
figures of this section are from before, the blocks alone on the
device). Two things differ from the CPU's pass: CLIP's activation x
sigmoid(1.702 x) is three element-wise ops there, and the MLP's inside
width is rounded up to a multiple of 32 with zeros (SigLIP's 4 304
becomes 4 320: a device's matrix products take rows of a multiple of 32,
and handed that product back otherwise). The encoders of Qwen2-VL and
Qwen2.5-VL, which read an image at its own size, go through the same
blocks (below):
their rows are turned by their patch's place through the rotary op, and
a block with windowed attention is one attention op, each row given its
window's rows as its span of keys. oklm describe does it
when the language model has layers on a device, for every encoder
(SmolVLM, Gemma 3, LLaVA 1.5 and 1.6, Qwen2-VL, Qwen2.5-VL); the
retrieval's encoder computes on the CPU. oklm serve does it
too when its model has layers on a device: a request's encoder runs in
its own thread, each tile or image as one pass on the device
(oklm_backend_pass_begin()), which lets the scheduler's
steps in between two of its blocks
(oklm_backend_pass_yield()): a request that is decoding
waits for a block, 0.04 to 0.08 s with its own step where it waited 0.25
to 0.70 s for the pass on Vulkan (server.md).
Measured on 2026-11-14 (RX 7900 XT, llama.cpp's test image, the
language model's layers on the device in each case): the prompt of
oklm describe, encoder included, and one tile through the
encoder alone (test_mm_public, a second tile, the kernels
built):
| Model | Prompt, encoder on the CPU: OpenCL, Vulkan | Prompt, blocks on the device: OpenCL, Vulkan | One tile: CPU | OpenCL | Vulkan |
|---|---|---|---|---|---|
| LLaVA 1.5 7B (577 rows, 23 blocks) | 1.40 s, 1.50 s | 0.40 s, 0.41 to 0.42 s | 1.16 s | 0.13 s | 0.11 s |
| LLaVA 1.6 Mistral 7B (five tiles) | 7.83 s, not run | 1.85 s, 1.89 s | |||
| Gemma 3 4B (4 096 rows, 27 blocks) | 21.0 s, 21.1 s | 0.89 to 0.90 s, 0.84 s | 22.0 s | 0.77 s | 0.73 s |
| SmolVLM2-2.2B (five tiles) | 10.6 s, not run | 0.72 s, 0.60 s | |||
| SmolVLM-256M | not run | 0.33 s, 0.29 to 0.31 s |
On the CPU alone the same prompts take 7.0 s (LLaVA 1.5), 23.3 s
(Gemma 3), 12.0 s (SmolVLM2-2.2B) and 3.3 s (SmolVLM-256M). With the
blocks on OpenCL the embeddings are within 1.4e-5 to 4.3e-5 (relative
norm) of the CPU's (LLaVA 1.5 and Gemma 3, two pictures each), on Vulkan
within 1.2e-5 to 5.0e-5: its matrix units take a prompt's activations
rounded to F16, and an encoder's products are computed there without
that rounding since 2026-11-14, which is what Vulkan's columns above are
with (below). OpenCL's
column is with its bidirectional attention on tiles of its own
(attention_bi64 / 80 / 128, opencl.md; the figures of the day's end,
with the CPU's, are below); on
the kernel of one work-group per head and token Gemma 3's 4 096 rows
took 3.03 s a tile and its prompt 3.15 s, SmolVLM2-2.2B's prompt 0.98 s
and SmolVLM-256M's 0.51 s. Gemma 3's row is with those tiles' keys and
values in one buffer of local memory (2026-11-14): with a tile of each
its prompt took 1.05 to 1.11 s and its 4 096 rows alone 0.91 s on the
same day. Of the 0.82 s of kernel time of that prompt on OpenCL
(OKLM_CL_PROFILE=1) the encoder's attention is 0.40 s (0.50
s before), in float arithmetic, and its matrix products 0.23 s, in F32
where Vulkan's matrix units take F16: with OKLM_CL_WMMA=1,
which gives OpenCL's products those units and their rounding, the prompt
takes 0.72 to 0.73 s.
The patch embedding and the projector on a device
With its blocks on a device, an encoder computed its patch embedding before them and its projector after them in host memory, the stream going up and coming back between the two. Timed on 2026-11-14 (RX 7900 XT, llama.cpp's test image, Vulkan, milliseconds of one tile or image):
| Encoder | Patch embedding | Positions, norms | Projector | The encoder |
|---|---|---|---|---|
| Gemma 3 4B (4 096 patches) | 3.4 cutting the patches, 18.1 the product | 3.4 | 1.9 the pool and its norm, 5.0 the projection | 721 |
| Qwen2-VL 2B (1 564 patches, 391 embeddings) | 0.8, 6.8 | 0.9 | 79.3 the merger | 341 |
| LLaVA 1.5 7B (576 patches) | 0.5, 2.2 | 0.4 | 67.0 the two layers | 108 |
| SmolVLM-256M (a tile of 1 024 patches) | 1.3, 4.3 | 0.6 | 0.2 the shuffle, 3.9 the projection | 42 |
| SmolVLM2-2.2B (a tile of 729 patches) | 0.6, 3.4 | 0.6 | 0.2, 11.5 | 89 |
The two parts were 4 % of Gemma 3's encoder and 65 % of LLaVA 1.5's.
Both are products, a norm and an activation, which the backends have:
they moved to the device without a kernel or an op more
(src/mm/vision.c, dev_stream() and
dev_project()).
- The patch embedding is a convolution whose stride
is its kernel: a product over the patches' rows. The host still cuts the
image into rows of 3 x patch^2 values and uploads them in place of the
stream (588 values a patch of 14 pixels where SigLIP's stream has 1
152). The kernel is copied to the device with its rows widened with
zeros to a multiple of 32 values, which a device's products ask for (588
becomes 608), and the product is the blocks' own, with the activations
at F32's precision (
oklm_matmul_wide()). SigLIP's bias, the positions and CLIP's norm are ops after it; CLIP's class token is uploaded as the stream's first row and the product is written into the rows after it. For a pair of frames (Qwen2-VL's and Qwen2.5-VL's video) each frame's patches meet their own kernel and the second product is added to the first. - Two layers (LLaVA, and the mergers of Qwen2-VL and Qwen2.5-VL): two products with the exact GELU between them, LLaVA's over the rows after the class token (a view), the mergers' over each group of four rows read as one (the rows of a group already follow each other there), after the last norm.
- The pixel shuffle (SmolVLM) and the pool (Gemma 3) join the patches of a square group, which are not neighbours in reading order. The device therefore takes a tile's rows group by group (a group's rows, then its columns), the positions copied to it in that order: every row sees every row in the blocks, so the order changes the order of the sums and nothing else. The shuffle is then the stream read with each group's rows as one, and the pool fifteen additions of views of every sixteenth row, in the host's order, and a scale; Gemma 3's norm and the projections follow.
OKLM_MM_DEVICE_ENDS=0 keeps these parts in host memory,
the blocks alone on the device (build.md);
oklm_vision_device_parts() tells which it is.
Measured on 2026-11-14 (RX 7900 XT, one build, the blocks alone on
the device -> the whole encoder; the prompt of
oklm describe on llama.cpp's test image with the language
model on the same device, three runs each; then the encoder alone on a
second image and its embeddings against the CPU's,
test_mm_public, two pictures):
| Model | Prompt, OpenCL | Prompt, Vulkan | Encoder alone, OpenCL | Encoder alone, Vulkan | Embeddings from the CPU's (OpenCL; Vulkan) |
|---|---|---|---|---|---|
| Gemma 3 4B | 0.89 to 0.90 s -> 0.86 s | 0.83 to 0.84 s -> 0.80 s | 0.772 s -> 0.736 s | 0.733 s -> 0.689 s | 1.4e-5, 4.3e-5; 1.2e-5, 2.8e-5 |
| LLaVA 1.5 7B | 0.40 to 0.42 s -> 0.33 to 0.36 s | 0.40 to 0.41 s -> 0.33 to 0.34 s | 0.127 to 0.129 s -> 0.058 s | 0.109 s -> 0.040 to 0.042 s | 1.4e-5, 2.1e-5; 2.5e-5, 4.8e-5 |
| LLaVA 1.6 Mistral 7B (five tiles) | 1.82 to 1.88 s -> 1.49 to 1.52 s | 1.87 to 1.89 s -> 1.53 to 1.54 s | |||
| Qwen2-VL 2B | 0.47 s -> 0.38 s | 0.44 s -> 0.35 s | 0.254 s -> 0.188 s | 0.234 to 0.235 s -> 0.171 to 0.172 s | 6.4e-5, 5.1e-5; 2.1e-5, 3.0e-5 |
| Qwen2.5-VL 7B | 0.60 to 0.61 s -> 0.48 to 0.51 s | 0.55 s -> 0.45 s | 0.232 to 0.245 s -> 0.155 s | 0.199 to 0.202 s -> 0.124 to 0.125 s | 1.5e-5, 1.1e-5; 8.0e-6, 9.8e-6 |
| SmolVLM-256M | 0.33 s -> 0.28 s | 0.29 to 0.30 s -> 0.25 s | 0.044 s -> 0.035 s (a tile) | 0.037 s -> 0.028 s | 2.7e-6; 4.0e-6 (one tile) |
| SmolVLM2-2.2B (five tiles) | 0.70 to 0.71 s -> 0.59 to 0.60 s | 0.60 to 0.61 s -> 0.49 s |
The embeddings are as far from the CPU's as with the blocks alone on the device (within 1.3e-5 of those figures in every row). SmolVLM's embeddings on a device are compared by the test since (the overview tile of SmolVLM-256M). The greedy answer about the newspaper from the device encoder's embeddings and from the CPU's agree on 46 of Qwen2-VL 2B's 48 choices with the encoder on OpenCL and 47 on Vulkan (the two tokens 0.18 apart where they part; 48 and 47 with the blocks alone there), and on 48 and 48 for Qwen2.5-VL 7B (47 and 47): near ties that fall either way with the order of a sum.
Checks: test_vision runs every tiny encoder (the
generated SigLIP with a pixel shuffle, Gemma 3's, LLaVA's, Qwen2-VL's
and Qwen2.5-VL's, their sized images and their pair of frames) on OpenCL
and Vulkan both ways, the whole encoder on the device and its blocks
alone, each within 8e-7 of the CPU's embeddings on values up to 2
(9.6e-7 on the Intel UHD 750), and back on the CPU to the CPU's bits;
test_mm_public the published files, both ways too.
test_encoder_types writes the generated encoder with its
blocks' matrices in F16 and in BF16 (no file on disk is BF16), checks it
against the reference in doubles and runs it through the device path on
the tests' counting device, which computes with the CPU's kernels: the
path's copies and widened rows in a type of 16 bits, not a device's
kernels for it (those are the op tests'). An encoder with quantized
matrices in its blocks is not put on a device at all
(oklm_vision_use_backend() answers that it does not take
them); no tiny file has such matrices (the encoder the test writes is 16
values wide, half a quantized block, and
tools/gen_vision_fixtures.py writes floats), so that answer
has no test. The forms the ops take there (a product into the rows after
the first, additions of views of every n-th row) are checked as ops
against the CPU's arithmetic by test_conv_views in
tests/core/test_opencl.c and
tests/core/test_vulkan_gemm.c, on the RX 7900 XT and the
Intel UHD 750. None of them is handed back to the CPU on the published
files (the log of OKLM_LOG_LEVEL=INFO names the ops a
backend hands back).
Limits. The device holds the patch kernels, the positions, the norms
and the projector's matrices beside the blocks, as the file holds them
(oklm_vision_device_bytes() counts them and the larger
activations); a device with room for the blocks and not for these keeps
them in host memory, with a notice
(the device has no room for the patch embedding and the projector):
test_vision gives each tiny encoder a device with room for
the blocks to the byte (the tests' counting device with a bound,
computing with the CPU's kernels, tests/mm/count_backend.h)
and finds the notice, the bytes held and the embeddings of
OKLM_MM_DEVICE_ENDS=0, bit for bit, then with one byte less
the refusal and the CPU's bits. So does an encoder whose projector's
matrices are quantized or have rows that are not a multiple of 32 values
(none on disk). The image is still cut into patches and normalized on
the host. Gemma 3 gains 4 %: its 4 096 rows of attention are the
encoder.
The tables measured again
At the end of the day's changes (2026-11-15, RX 7900 XT, one job at a
time, load average 1.1 to 5.0): the whole encoder on the device,
Vulkan's products at F32's precision, the question written as each
model's template has it. The prompt of oklm describe on
llama.cpp's test image, the language model on the same backend, three
runs on each device and one on the CPU; then the encoder alone on a
second image (test_mm_public, two pictures, the kernels
built) and its embeddings against the CPU's:
| Model (embeddings of the prompt) | Prompt: CPU | OpenCL | Vulkan | Encoder alone: CPU | OpenCL | Vulkan | Embeddings from the CPU's (OpenCL; Vulkan) |
|---|---|---|---|---|---|---|---|
| LLaVA 1.5 7B (576) | 7.17 s | 0.33 to 0.35 s | 0.34 s | 1.12 to 1.61 s | 0.058 s | 0.040 s | 1.4e-5, 2.1e-5; 2.5e-5, 4.8e-5 |
| LLaVA 1.6 Mistral 7B (five tiles, 2 340) | 30.2 s | 1.48 to 1.51 s | 1.53 to 1.54 s | ||||
| Gemma 3 4B (256, from 4 096 rows) | 12.3 s | 0.85 s | 0.80 s | 10.3 to 10.7 s | 0.737 s | 0.689 s | 1.4e-5, 4.3e-5; 1.2e-5, 2.8e-5 |
| SmolVLM-256M (five tiles, 320) | 2.98 s | 0.28 s | 0.24 to 0.26 s | 0.40 s (a tile) | 0.034 s | 0.028 s | 2.7e-6; 4.0e-6 (one tile) |
| SmolVLM2-2.2B (five tiles, 405) | 9.64 s | 0.59 s | 0.49 to 0.51 s | ||||
| Qwen2-VL 2B (391; the encoder alone on 273) | 6.57 s | 0.38 s | 0.35 s | 4.14 to 4.22 s | 0.186 s | 0.171 s | 6.4e-5, 5.1e-5; 2.1e-5, 3.0e-5 |
| Qwen2.5-VL 7B (391; 273) | 8.93 s | 0.48 to 0.52 s | 0.45 s | 3.22 to 3.64 s | 0.154 s | 0.125 to 0.126 s | 1.5e-5, 1.1e-5; 8.0e-6, 9.8e-6 |
The devices' figures are those of the table of the patch embedding and the projector to within a hundredth of a second: nothing after it changed an image encoder's path. The CPU's are new: its encoder was taken on the same day for the large pictures (below), and Gemma 3's prompt, 23.3 s in the first table of this section, is 12.3 s.
A picture of 2 000 x 2 000 pixels, the two Qwen files, three runs on
each device: at 1 225 embeddings (--tiles 980: the largest
size until the default went to 4 096) the prompt of Qwen2-VL 2B takes
1.60 s on OpenCL and 1.67 to 1.68 s on Vulkan (1.89 to 1.91 s and 1.78
to 1.96 s when last measured, the blocks alone on the device), and
Qwen2.5-VL 7B's 1.35 to 1.38 s and 1.22 s (1.72 to 1.76 s and 1.40 to
1.42 s then); at the default largest size, 4 096 embeddings and a prompt
of 4 121 rows, 11.1 to 11.2 s and 13.4 s for Qwen2-VL 2B, 5.51 to 5.53 s
and 5.58 s for Qwen2.5-VL 7B.
The encoders' precision on Vulkan
Vulkan's matrix units (cooperative matrices, vulkan.md) take the activations
of a prompt's products rounded to F16. A language model's prompt bears
it; an encoder's blocks do not: their activations reach tens (80 in
Qwen2-VL's), the terms of a product cancel, and 23 to 32 blocks add the
rounding up, to embeddings 2.6e-3 to 1.5e-2 from the CPU's where
OpenCL's F32 products give 1.4e-5 to 5.3e-5 (3.1e-4 for Ultravox). The
blocks of the vision and audio encoders therefore ask for their products
with the activations at F32's precision
(oklm_matmul_wide()), which Vulkan computes on the same
units with each activation as two F16 parts, the rounded value and what
the rounding dropped (vulkan.md: where the error
enters, product by product, and the forms measured). The CPU and OpenCL
compute them as before.
Measured on 2026-11-14 on the RX 7900 XT (RADV), one build, the
activations rounded as before (OKLM_VK_WIDE=0) -> the
default: the embeddings against the CPU's (relative norm; two pictures a
vision encoder, jfk.wav for Ultravox) and the encoder alone
(test_mm_public, a second image or sound), then the prompt
of oklm describe on llama.cpp's test image or jfk.wav, the
language model on Vulkan:
| Encoder | Embeddings from the CPU's | Encoder alone | Prompt |
|---|---|---|---|
| Qwen2-VL 2B (273 embeddings; the prompt's picture 391) | 1.5e-2, 1.1e-2 -> 1.9e-5, 3.0e-5 | 0.18 s -> 0.23 to 0.24 s | 0.37 to 0.38 s -> 0.43 to 0.45 s |
| Qwen2.5-VL 7B | 6.3e-3, 4.8e-3 -> 1.1e-5, 9.6e-6 | 0.15 to 0.16 s -> 0.20 s | 0.48 to 0.50 s -> 0.55 to 0.56 s |
| Gemma 3 4B (SigLIP, 4 096 rows) | 2.6e-3, 5.1e-3 -> 1.2e-5, 2.8e-5 | 0.63 s -> 0.73 s | 0.74 s -> 0.84 s |
| LLaVA 1.5 7B (CLIP, 577 rows) | 5.6e-3, 7.2e-3 -> 2.3e-5, 5.0e-5 | 0.11 s -> 0.11 s | 0.40 to 0.41 s -> 0.41 to 0.42 s |
| LLaVA 1.6 Mistral 7B (five tiles) | 1.82 to 1.83 s -> 1.89 s | ||
| SmolVLM2-2.2B (five tiles) | 0.49 to 0.50 s -> 0.60 s | ||
| SmolVLM-256M | 0.26 s -> 0.29 to 0.31 s | ||
| Ultravox v0.5 (Whisper large-v3-turbo's encoder, jfk.wav) | 9.6e-3 (largest difference 8.1e-2) -> 5.5e-5 (8.1e-4) | 0.23 s -> 0.30 s | 0.27 s -> 0.33 s |
Whisper large-v3-turbo, oklm transcribe of jfk.wav |
0.24 to 0.25 s -> 0.31 to 0.32 s | ||
| Whisper large-v3, the same | 0.41 s -> 0.49 s |
An encoder takes 1.0 to 1.3 times what it took, a prompt 1.03 to 1.22
times, a transcription of jfk.wav 1.2 to 1.3 times: 0.01 to 0.10 s an
image or a 30-second window. With OKLM_VK_WIDE=f32 the same
products run on the F32 tiles instead: Qwen2-VL 2B's embeddings 5.3e-5
and 4.6e-5 from the CPU's, its encoder 0.28 to 0.29 s and its prompt
0.50 s, Gemma 3's prompt 0.95 to 0.96 s, Ultravox's 0.38 to 0.39 s,
jfk.wav in 0.37 s (large-v3-turbo) and 0.55 to 0.56 s (large-v3): no
closer and slower, so the two parts are the default.
OKLM_VK_WIDE=0 gives the earlier speed and rounding (build.md).
The answers do not move with it more than with any change of a sum's order. With the language model on Vulkan (its own prompt products keep their F16 rounding, to the bit), the newspaper answer from the Vulkan encoder's embeddings and from the CPU's agree on 47 of 48 greedy choices for Qwen2-VL 2B (47 before; the two tokens 0.24 apart where they part) and on 47 for Qwen2.5-VL 7B (48 before; 0.11 apart), as OpenCL's encoder does (48 and 47): a difference of 1e-5 in the embeddings is rounded to F16 in the language model's first products like one of 1e-2, so the logits of the two runs are as far apart as before (within 1.07 and 0.93, 0.90 and 1.02 before) and a near tie falls either way. Whisper's texts for jfk.wav are the same on both settings; large-v3-turbo's probability of English is 0.9499 where the CPU gives 0.9498 and the rounded products 0.9508.
Gemma 3
Gemma 3 (4B, 12B, 27B; the 1B and 270M read text only) was audited
against llama.cpp's tools/mtmd at the pinned revision,
which OKLM follows where the two differ from the model's own
processor:
- The picture. One image whatever its size, no tiles:
fitted in a square of 896 pixels keeping its proportions (the sides
rounded up), centred, the rest black, with a bilinear filter; then each
channel to (x / 255 - 0.5) / 0.5. The resampling is Pillow's, in its
fixed-point arithmetic (
oklm_image_resize(),oklm_image_fit()): the bytes are llama.cpp's. The model's own processor stretches the picture to the square instead, and its "pan and scan" (crops of a large or elongated picture given as further images) is optional: neither llama.cpp nor OKLM does it. - The encoder. The SigLIP encoder above at 896 pixels in patches of 14: 4 096 patches through 27 blocks, every block and the final norm used, the tanh GELU.
- The projector. The patches as a 64 x 64 grid, the
mean of each 4 x 4 group (row by row: 256 embeddings), an RMS norm
(epsilon 1e-6) times
mm.soft_emb_norm, thenmm.input_projection. The files do not say the group's side: 4, as for every size of the family. - The prompt.
<start_of_image>, the 256 embeddings,<end_of_image>, where the marker is in the turn Gemma's template renders. llama.cpp adds no new line around them at this revision (the model's processor writes two before and two after). - Rows that see each other. Gemma 3 was trained with
the image's embeddings attending to one another in both directions, and
llama.cpp evaluates them so.
oklm_eval_embd_mutual()andoklm_eval_batch_mutual()do it: each of these rows sees the rows before the image and every row of the image, the window of a windowed layer still counted from its own position; the tokens after the image see it as usual.oklm describeand the server use them when the projector asks (oklm_vision_mutual()); the server gives such an item to the model in one piece, waiting a step rather than cutting it, so that an answer does not depend on what else is being served. On a device the attention takes the last key these rows all see: OpenCL'sattentionkernel and Vulkan'sattn_splitshader, the ones of one work-group per head and token, on the contiguous and the paged cache (Verification has the runs). OpenCL's tiled prompt kernels take that last key too since 2026-11-14 (every block of queries then walks the keys up to it), where they take causal rows: heads up to 128 (Gemma 3 27B's). The 4B's and 12B's heads of 256 stay on theattentionkernel, which measured faster for them: 20.0 ms of a prompt's 0.95 s of kernel time for the image's 256 rows and the text after them in the 4B's 34 layers, 25.6 to 35.5 ms on the tiles (opencl.md).
LLaVA 1.5
LLaVA 1.5 7B (Vicuna 7B, architecture llama, with CLIP
ViT-L/14 at 336 pixels) runs from the files llama.cpp's documentation
names (mys/ggml_llava-v1.5-7b, 2023). Audited against
llama.cpp's tools/mtmd at the pinned revision and against
the model's definition (CLIP's and LLaVA's own code), which differ here.
OKLM computes the encoder as the model defines it and keeps doing so (which of the
two is right):
- The file. It names no
clip.projector_type; a file that hasclip.has_llava_projectoris taken formlp, as llama.cpp took it when the file was made. The pinned llama.cpp refuses it; the test reference reads a copy with the key added (tools/mmproj_add_type.py). The conversion left the last of CLIP's 24 blocks out, LLaVA reading the output of the one before it: the file holds 23. - The picture. Fitted in a square of 336 pixels keeping its proportions, centred, the rest the mean colour of CLIP's training images (122, 116, 104), with a bicubic filter; then CLIP's mean and deviation from the file. The bytes are llama.cpp's.
- The encoder, as the model defines it, which is what OKLM computes. The class token first, with the first position; the patches after it; a norm; all 23 blocks with quick-GELU; no final norm; the class token's row dropped: 576 rows.
- The encoder, as the pinned llama.cpp computes it.
Three things differ: the class token is placed after the patches, so
patch i takes the position of patch i - 1 and the class token the last;
22 blocks are run, one fewer; and the row dropped is the first, a patch,
the class token's being kept as the last. A float64 pass written both
ways (
tools/vision_ref.py,--as-llama-cpp) shows these three are the whole difference: llama.cpp's embeddings are within 6e-3 and 1.5e-2 of the pass written its way, and 0.99 and 0.79 from the model's (mostly the rows being one patch apart). OKLM does not follow llama.cpp here. - The projector.
mm.0, GELU,mm.2. The model's GELU is the exact one, which OKLM uses; llama.cpp's is the tanh form. - The prompt. The 576 embeddings where the marker is,
nothing around them. The file has no chat template: the turns are
Vicuna's as llama.cpp's
vicunatemplate writes them (a system text and an empty line;USER:andASSISTANT:lines;ASSISTANT:to answer), inoklm describeand in a server with this projector. LLaVA's own prompt joins the turns with spaces and opens with a sentence about a curious human and an assistant; give it with--systemif wanted. Without the projector the same turns come from the catalogue, which namesvicunafor this model:oklm chat -m llava-1.5-7b(or the file by its path in the model's folder) chats in them, and--chat-template vicunagives them to a copy kept elsewhere (inference.md).
The CLIP encoder: the model's definition, or llama.cpp's
The pinned llama.cpp reads a LLaVA file with the class token after the patches, one block fewer, and the first patch dropped where the model drops the class token. The model's definition is the right one, on three grounds that do not depend on each other, and OKLM keeps it, for LLaVA 1.5 and 1.6 alike (the same encoder):
- The model's code.
CLIPVisionEmbeddings.forward()(transformers,modeling_clip.py) concatenates the class embedding and then the patches, and adds the position table row for row. LLaVA'sCLIPVisionTower.feature_select()(llava/model/multimodal_encoder/clip_encoder.py) takeshidden_states[mm_vision_select_layer], -2 in the models' configurations, and withpatchdrops its first row;LlavaNextForConditionalGeneration.get_image_features()does the same (vision_feature_layer-2,[:, 1:]). - The file's position table. The positions of the
patches were learned as a grid, so the table shows where its rows are
without any code being run (
tools/clip_positions.py): of the 577 rows ofv.position_embd.weight, row 0 resembles none (its cosine with row 1 is 0.040) while the last two are neighbours (0.961, the median of consecutive rows being 0.980); and consecutive rows are least alike each time the first of the two is a multiple of 24 (0.92 against 0.97 to 0.99), which is where a row of 24 patches ends if the patches start at row 1. Row 0 is the class token's position and rows 1 to 576 the patches', in both files. Placed after the patches, the class token takes the last patch's position, and each patch its left neighbour's (the first of each row, the last of the row above; the very first, the class token's). - The blocks the file holds. CLIP ViT-L/14 has 24
blocks and a final norm;
hidden_states[-2]is what 23 of them give, without that norm. The conversion kept exactly that: 23 blocks, nov.post_ln. Running 22 of them giveshidden_states[-3], which no configuration of the model asks for.
What it costs to differ from llama.cpp: the embeddings of one image are not llama.cpp's (0.99 and 0.79 apart in relative norm on the two test pictures, mostly the rows being one patch apart), so an answer may read differently in the two programs; and llama.cpp cannot be the reference of this encoder, which is held to a float64 pass of the definition instead (Verification).
LLaVA 1.6
LLaVA 1.6 ("LLaVA-NeXT") Mistral 7B (Mistral 7B Instruct v0.2,
architecture llama, with the encoder of LLaVA 1.5) runs
from the published files (cjpais/llava-1.6-mistral-7b-gguf,
catalogue llava-1.6-mistral-7b). Its definition here is the
model's own processor and code as transformers has them
(image_processing_llava_next.py,
modeling_llava_next.py, read at v4.49, with LLaVA's own
llava/mm_utils.py and
llava/model/llava_arch.py beside them), not llama.cpp: the
pinned llama.cpp gives a file that lists resolutions to its own slicing,
an overview and tiles of 576 embeddings each, without the cut nor the
ends of rows described below.
- The file.
clip.projector_typeismlp;clip.vision.image_grid_pinpointslists five resolutions (336 x 672, 672 x 336, 672 x 672, 1 008 x 336, 336 x 1 008), each a whole number of tiles of 336 pixels;clip.vision.mm_patch_merge_typeisspatial_unpad; the tensormodel.image_newlineis one embedding of the language model's width. The encoder's tensors are LLaVA 1.5's (23 blocks, no final norm). - The resolution. The one that keeps the most of the
image's pixels once the image is scaled into it without changing its
proportions, and, among equals, the one with the least area left over;
the first of the list at a tie (
select_best_resolution()). The pairs are read as transformers reads them, a height then a width. LLaVA's own code reads the same list as widths then heights: the list being symmetric, only a tie differs, and a picture small enough to fit whole in every resolution (under 336 pixels both ways) gets the first of the list, two tiles side by side here, one above the other there. - The tiles. The image is resized into that resolution keeping its proportions (the shorter side rounded up, bicubic), pasted centred on black (the offset rounded down), and cut in tiles of 336 pixels row by row. The overview is the whole image stretched to 336 x 336 (bicubic), without the mean-coloured bars of LLaVA 1.5. Each is then normalized with CLIP's mean and deviation. transformers v4.49 pads both ends by the offset rounded down, which leaves the canvas a pixel short when the padding is odd; later versions put the pixel at the end, as LLaVA's own paste does and as OKLM does.
- The encoder and the projector. LLaVA 1.5's, on the overview and on each tile: 576 embeddings each.
- The rows (
pack_image_features(),oklm_vision_pack()). The overview's 576 embeddings first. Then the tiles' embeddings as one grid of features, 24 to a tile's side (48 x 48 for four tiles), from which the part that covers the black padding is cut (unpad_image()): the image's proportions against the grid's say which side was padded; the image's size on that side, in features, is rounded down, and half the difference, rounded down, is cut from each end, so that one row or column of padding stays when the difference is odd. The grid is read row by row, and after each row comes theimage_newlineembedding. A picture of 640 x 488 pixels gives 576 + 36 x (48 + 1) = 2 340 rows; the most is 2 928 (a square picture of 672 pixels or more). The overview alone (--tiles 336or less) is the model's own form for an image of one tile: its 576 embeddings and that embedding once. - The prompt. The rows where the marker is, nothing
around them. The file carries Mistral's chat template, which is used:
[INST] <image>\nQUESTION [/INST], the form LLaVA's ownmistral_instructconversation writes. That template refuses a system message, as Mistral 7B Instruct v0.2's does (LLaVA's conversation for this model has none either). A LLaVA 1.6 file without a template (the Vicuna variants) gets Vicuna's turns, as LLaVA 1.5 does; no such file was run. - Room. An image takes up to 2 928 rows of the
context: the default of 4 096 holds one with its answer, and a second
image in a conversation needs
-c 8192.
Qwen2-VL and Qwen2.5-VL
Qwen2-VL (2B run here) and Qwen2.5-VL (3B, 7B, 32B and 72B run here:
the other sizes) were
audited against the models' own code (transformers:
modeling_qwen2_vl.py,
image_processing_qwen2_vl.py,
modeling_qwen2_5_vl.py, read at v4.49) and against
llama.cpp's tools/mtmd at the pinned revision
(models/qwen2vl.cpp, clip.cpp,
mtmd.cpp), on the published files
(ggml-org/Qwen2-VL-2B-Instruct-GGUF,
unsloth/Qwen2.5-VL-7B-Instruct-GGUF). The two agree on the
picture's rows and their positions; they differ on the limits of the
picture's size, in arithmetic (Verification)
and, in the language model, on the epsilon of the RMS norms, which the
pinned llama.cpp does not read for this architecture (Qwen2.5-VL 7B and
llama.cpp's RMS norm).
- The picture. One image at its own resolution, no
tiles and no padding: each side rounded to the nearest multiple of 28
pixels (a patch of 14 times the merger's 2; Python's
round(), halves to the even neighbour), then, when the area passes the maximum, both sides divided by sqrt(area / maximum) and rounded down to a multiple of 28, or, when it is under the minimum, multiplied by sqrt(minimum / area) and rounded up. The picture is stretched to that size with a bicubic filter (the proportions change by the rounding only), then CLIP's mean and deviation from the file. The bytes are llama.cpp's. - The limits of the area are where the references
differ, the files holding neither key
(
clip.vision.image_min_pixels,image_max_pixels, read when present). The processor class's defaults are 3 136 and 1 003 520 pixels (4 to 1 280 embeddings). Thepreprocessor_config.jsonpublished with every size of the two models (read on 2026-11-14: Qwen2-VL 2B, Qwen2.5-VL 3B, 7B, 32B, 72B) keeps the minimum and raises the maximum to 12 845 056 pixels (16 384 embeddings); llama.cpp at the pinned revision takes 8 and 4 096 embeddings (6 272 and 3 211 264 pixels). OKLM takes the processor's minimum and llama.cpp's maximum, 4 to 4 096 embeddings, since 2026-11-14 (1 280 before): a picture between 6 272 and 3 211 264 pixels gets llama.cpp's size, a larger one is reduced to 3.2 megapixels as llama.cpp reduces it. The models' own 16 384 is not reachable (the owner's decision, 2026-11-14): no reference encodes it, its 65 536 patches attend each to each (sixteen times the work of 4 096 embeddings), and the rows are four times the default context.--tiles SIDE(ofoklm describeand ofoklm serve) caps the area under the default, SIDE x SIDE pixels. Where the rows must fit a context of a fixed length the default is bounded by it (oklm_vision_limit_tokens()):oklm serveand the captioner ofoklm rag indexgive an image at most three quarters of a sequence's context (3 072 embeddings with-c 4096; the whole 4 096 from-c 5464on), so a large photograph is reduced rather than its request refused for its length;--tilesreplaces that bound by the area asked. A prompt's rows name a picture by its content and by the bound in force, so embeddings kept under another bound (a server whose context was resized) are not taken for it.oklm describewithout-csizes its context to the pictures instead (oklm_vision_image_tokens()gives an image's embeddings before a pixel is read): 5 120 tokens for a picture of 4 015 embeddings. - The patches. The model embeds two frames at once
with a three-dimensional convolution without bias (kernel 2 x 14 x 14);
a still image is given twice, so its embedding is the sum of the two
kernels (
v.patch_embd.weight,v.patch_embd.weight.1) applied to the same patch: the kernels are summed when the file is loaded. No class token, no learned positions, no norm before the blocks. - The order of the rows. The patches are not in reading order: the four patches of each 2 x 2 group follow each other (the merger joins consecutive rows), groups in reading order; Qwen2.5-VL orders the groups by window as well. Attention does not depend on the order, each row carrying its own position: OKLM orders the groups window by window for both (one window holding everything for Qwen2-VL), so a window is a run of rows, and puts the embeddings back in reading order at the end.
- Positions. Rotary, two-dimensional, in every block's attention: with a head of d values, pair i (values i and i + d/2) is rotated by the patch's row times 10000^(-i / (d/4)) for i under d/4, and by its column times 10000^(-(i - d/4) / (d/4)) from d/4 on. Rows and columns are counted in patches.
- The blocks of Qwen2-VL (32, width 1 280, 16 heads):
layer norm (epsilon 1e-6), attention with biases, every row seeing every
row; layer norm, a linear layer to 5 120, quick-GELU (x sigmoid(1.702
x)), a linear layer back. The file names the first of the two
ffn_down, and gives 1 536, the language model's width, asclip.vision.feed_forward_length: the MLP's width is read from the tensors. - The blocks of Qwen2.5-VL (32, width 1 280, 16
heads): RMS norms with a weight and no bias; a gated MLP with biases,
down(SiLU(gate(x)) times up(x)), 3 420 wide; and windowed attention: the
merged grid is cut into windows of 4 x 4 embeddings (8 x 8 patches, 112
pixels,
clip.vision.window_sizewhen the file has it; the windows at the right and bottom edges are smaller), a row sees its window only, except in the blocks with full attention: 7, 15, 23 and 31 (fullatt_block_indexesin the model's configuration; the file saysclip.vision.n_wa_pattern = 8, every eighth block, which is the same list). - The merger. A norm (
v.post_ln: layer norm for Qwen2-VL, RMS for 2.5), the four rows of each group joined into one of 5 120,mm.0, GELU,mm.2to the language model's width (1 536 for the 2B, 3 584 for the 7B). The model's GELU is the exact one, which OKLM uses; llama.cpp's is the tanh form. An image of w x h pixels becomes (w / 28) x (h / 28) embeddings in reading order. - The prompt.
<|vision_start|>, the embeddings,<|vision_end|>where the marker is in the turn the file's ChatML template renders.oklm describeand the server's chat routes put the question right after the image, as the model's own template does (the table of the projectors). - Positions in the language model (M-RoPE).
general.architectureisqwen2vl: the tensors and every other key areqwen2's, plusqwen2vl.rope.dimension_sections = [16, 24, 24, 0]. A row has three positions: time, height, width. Of the 64 rotated pairs of a head (pair i is values i and i + 64, frequency 1000000^(-i / 64) as inqwen2), the first 16 turn by the time position, the next 24 by the height, the last 24 by the width. A token has the three equal, so a text is rotated exactly asqwen2rotates it. An image whose first embedding would take position p has, for the embedding at row r and column c of its grid, time p, height p + r, width p + c; the text after it continues from p + max(rows, columns), the largest position used plus one, not from p plus the number of embeddings: the 391 embeddings of a 644 x 476 picture take 23 positions. llama.cpp does the same at the pinned revision.
How the language side does it (src/model/mrope.c):
oklm_eval_embd_grid()andoklm_eval_batch_grid()take an image's rows with the columns of its grid, and the index within the image of the first row given, so an image may come in pieces (the server's); the positions follow from that and from what the sequence holds.oklm_mm_eval()and the server call them for these projectors (oklm_mm_media.grid_w).oklm_seq_n_pos()gives the position the next token takes, which is no longer the number of cached rows.- The context keeps two numbers per cached row, its time position and
the position a token after it takes. A cut
(
oklm_seq_truncate()) anywhere, in the middle of an image too, leaves the rows before it with theirs, and the rest of the image can be given again; a copy of a sequence copies them; a sequence swapped out keeps them. A paged cache keeps them per block, so the prefix cache gives a sequence the rows another computed, an image's included, with their positions (equal rows after equal rows have equal positions). A KV file of a sequence that holds an image or a video keeps the two per row too (version 2 of the format, inference.md): the context that loads it has the positions of the one that saved, and a token after the image takes the position it would have taken there. Whether a sequence needs that version is told row by row: an image of one column ends at the position its row count gives, as tokens would, and still needs it. - A batch whose rows all have three equal positions (text, in a
conversation that holds images too) goes through the usual rotary
operation. A batch holding an image's rows goes through the same
operation given three positions a row and the bands
(
oklm_rope_parts(): the first pairs of a head turned by the first position, the next by the second, the others by the third), where the block runs: the CPU kernel, OpenCL'sropekernel and Vulkan'snorm_ropeshader read the position of a pair's band, and compute its angle as the one-position rotation does at that position (on the CPU bit for bit,test_ops). The three positions of a batch's rows follow its one-part positions in the context's position buffer of each device. OpenCL's fused launches (the rotation with the head norms, or with the cache writes) take one position a row: such a batch is rotated by the plain kernel. - The labels of the template (2026-11-14). The chat
template in the files of Qwen2-VL and Qwen2.5-VL counts the pictures and
the videos of a conversation and, when its
add_vision_idvariable is set, writesPicture 1:,Picture 2:right before each picture andVideo 1:before each video, each kind counted through the whole conversation. Nothing sets the variable unless the caller does: the template renders no label without it, and the pinned llama.cpp never sets it (it gives the template a string with its markers in it), so no label is the default here too, and the answers compared with llama.cpp's are unchanged. The Transformers processor passes the variable through when its caller gives it (its code is not installed here: the default was read in the template, not in the processor). The labelled form is offered:oklm describe --vision-id, andchat_template_kwargs: {"add_vision_id": true}(oradd_vision_id: truebeside the messages) in a chat request tooklm serve, both written byoklm_mm_content_numbered(), whichoklm_mm_content()calls without a numbering.test_vision'stest_vision_idrenders four conversations (two pictures, a picture between two texts, two turns with a video and two pictures, two videos) with the files' template and compares them, with and without labels, with jinja2's rendering of the same conversations given in parts (tools/qwen_vision_id_ref.py, the same output for the 2B's and the 7B's file). Run on Qwen2-VL 2B (Q4_K_M, the CPU,--temp 0, two pictures of 448 x 336, a newspaper's front page and its top left corner,Describe Picture 2 in one sentence.): 425 prompt tokens with--vision-idagainst 415 (5 tokens a label), and an answer either way ("The New York Times newspaper from July 20, 1969, reports on the historic event of men walking on the moon." with the labels, "[...] featuring the headline "Men Walk on Moon."" without); asked which picture shows the whole page, it answersAwith the labels andBwithout, so the labels are read but this small model does not answer by them. For the other projectors the option changes nothing (their templates have no such variable):oklm describesays so, a request is answered as without it.--vision-idis refused with--kv-load: the file does not say how many pictures and videos the conversation holds, so the count cannot go on. - Not covered. Several images at once in the model's own batch.
Qwen2.5-VL 7B and llama.cpp's RMS norm
The 7B's logits part from llama.cpp's as a prompt takes pictures, the
same embeddings given to both (Verification), where the 2B's do not. The
cause, found on 2026-11-14, is one number: the pinned llama.cpp
evaluates a qwen2vl file's RMS norms with an epsilon of
0, and this model has rows that need the file's.
- The model.
Qwen2RMSNormdivides a row by sqrt(mean(x^2) + epsilon), and the decoder's norms takeconfig.rms_norm_eps, 1e-6 in the publishedconfig.json(modeling_qwen2.py,modeling_qwen2_5_vl.py, read at v4.57). The GGUF files carry it (qwen2vl.attention.layer_norm_rms_epsilon = 1e-6, the 2B's and the 7B's), and OKLM reads it as for every architecture (src/io/hf_config.c). - llama.cpp (commit
8642541b,src/models/qwen2vl.cpp):load_arch_hparams()of this architecture reads the rotary sections and nothing else, sof_norm_rms_epsis never set (qwen2.cppandqwen3vl.cppread the key; the file is the same on its master branch on 2026-11-14). Its first norm gives every row the norm sqrt(3 584) = 59.867 before the weights, which is an epsilon of 0; with the file's an ordinary token's row has 59.77 and<|im_start|>'s 10.78. - Why the 7B and not the 2B. The epsilon matters for
a row whose mean square is near it. The 7B's embeddings of
<|im_start|>and<|im_end|>have norms 0.0110 and 0.0149 (mean squares 3.4e-8 and 6.2e-8) where an ordinary token's is near 1.05 (3.1e-4): the model's first norm leaves them 0.18 and 0.24 of what a norm without epsilon gives, and llama.cpp reads them 5.5 and 4.1 times too large. 4 248 of the vocabulary's 152 064 rows have a mean square under 1e-5. The 2B's two tokens have norms 0.50 and 0.56 (a factor 0.997), and none of its rows is under 1e-5: both programs give it the same norms to 0.3 %. - What it changes inside. Every chat prompt starts
with
<|im_start|>. With an epsilon of 0 that first row is the one whose residual stream grows to a norm of thousands from the fourth block on (2 825 after it, the others 26 to 55; Q4_K_M, the one-picture prompt); with the model's epsilon it is the third row, the new line aftersystem(7 330), and the first stays at 41. The rows after attend to a different row for it, and the image rows are most of them.
The arbiter is tools/qwen2vl_ref.py: the language model
of a qwen2vl GGUF in numpy float64, written from the
transformers code (the norm above, get_rope_index()'s three
positions, apply_multimodal_rotary_pos_emb(), grouped-query
attention, SwiGLU), the weights through gguf-py's dequantizers, on a
prompt's rows and embeddings as test_mm_public writes them;
--eps 0 is the same pass as a program that ignores the key
computes it. On the BF16 file (unsloth's, 14.2 GiB, fetched
for this and removed), whose products are float products everywhere, the
residual stream after a block of the one-picture prompt (418 rows;
relative difference over all the rows, then of the last row) and the
logits of the answer's first token:
| Qwen2.5-VL 7B BF16, one picture | OKLM from the model's pass | llama.cpp from the model's pass | llama.cpp from the pass with an epsilon of 0 |
|---|---|---|---|
| after block 0 | 3.0e-4, 2.6e-3 | 3.7e-2, 0.10 (its first row: 0.99) | 7.2e-4, 3.1e-3 |
| after block 3 | 7.2e-4, 1.8e-3 | 1.06, 0.25 (its first row: 70) | 9.2e-3, 2.9e-3 |
| after block 10 | 5.6e-4, 1.2e-3 | 1.10, 0.31 | 5.4e-3, 4.6e-3 |
| after block 26 | 5.4e-4, 6.0e-4 | 0.52, 0.11 | 3.2e-3, 2.3e-3 |
| after block 27, the last row | 9.8e-4 | 0.21 | 3.4e-3 |
| logits: largest, mean | 0.014, 0.0020 | 3.60, 0.85 | 0.089, 0.014 |
The difference is there after the first block, in the rows of the
special tokens, and does not come from the positions: the two programs
give every row the same three (the text before at 0 to 14, the image's
23 x 17 embeddings at time 15, rows 15 to 31, columns 15 to 37, the text
after from 38; with three pictures the texts after them from 38, 53 and
76), which are get_rope_index()'s, and the same bands (16,
24 and 24 pairs of a head of 128).
The logits of the first token on the four prompts (largest difference over the vocabulary, then the mean; llama.cpp given OKLM's embeddings):
| Prompt (rows) | OKLM from the model's pass | llama.cpp from the model's pass | llama.cpp from the pass with an epsilon of 0 |
|---|---|---|---|
| BF16: a text alone (38) | 0.007, 0.0006 | 1.13, 0.27 | 0.055, 0.011 |
| one picture (418) | 0.014, 0.0020 | 3.60, 0.85 | 0.089, 0.014 |
| two pictures, one turn (518) | 0.149, 0.072 | 6.42, 2.42 | 0.150, 0.022 |
| three pictures, one turn (797) | 0.013, 0.0021 | 4.87, 2.45 | 0.228, 0.139 |
| Q4_K_M: a text alone (38) | 0.84, 0.13 | 1.32, 0.18 | 1.14, 0.33 |
| one picture (418) | 1.03, 0.19 | 2.86, 0.50 | 0.80, 0.12 |
| two pictures, one turn (518) | 1.25, 0.27 | 3.29, 0.51 | 1.43, 0.35 |
| three pictures, one turn (797) | 1.31, 0.45 | 8.53, 4.14 | 1.36, 0.42 |
OKLM is the model's on the float file (the 0.149 of
the two-picture prompt is the keys kept as half floats: 2.2e-4 with an
F32 cache), and llama.cpp is the model with an epsilon of 0 (OKLM with
its epsilon set to 0 in a test harness, not an option of the runtime,
comes within 0.05, 0.09, 0.13 and 0.17 of llama.cpp on the four BF16
prompts). On the quantized file the two programs' 8-bit activations cost
up to 1.3 of a logit against a pass that rounds nothing, and llama.cpp's
distance stands out of that with pictures. The best token is the same in
the three computations of the eight cases; with three pictures the model
gives it 21.10, OKLM 21.09 and llama.cpp 25.18 (BF16). Nothing was
changed in OKLM, and the tests keep llama.cpp as their reference with
the bounds they had (test_qwen25vl_several_images: 85 % of
its choices, near ties of 1.5).
Whole answers, and the 3B. With
--answer the pass takes the 48 tokens of a recorded answer
after the prompt (teacher forcing: one causal pass gives the logits
before each of them, 80 s for the 7B's 418 rows and 48 tokens, 42 s for
the 3B's), and compares every position. On the Q4_K_M files, each
program fed the same rows and the tokens of llama.cpp's answer (largest
difference over the vocabulary and the 48 positions, the mean, and the
positions whose best token is the pass's):
| Q4_K_M, 48 tokens | OKLM from the model's pass | llama.cpp from the model's pass | llama.cpp from the pass with an epsilon of 0 |
|---|---|---|---|
| 7B, one picture (418 rows) | 1.26, 0.169, 47 | 2.96, 0.403, 47 | 1.12, 0.143, 48 |
| 7B, a text alone (38 rows) | 1.03, 0.156, 46 | 2.30, 0.233, 47 | not run |
| 3B, one picture (418 rows) | 4.58, 0.220, 46 | 4.21, 0.211, 47 | 4.26, 0.211, 47 |
The 7B's later tokens say what its first did: OKLM stays within 1.26 of the model at every position (0.51 to 1.26 a position) where llama.cpp is 0.73 to 2.96 from it, and llama.cpp is the pass with an epsilon of 0 (OKLM is 3.24 from that one, 0.358 on average). Where OKLM's best token is not the pass's, the pass has the two within 0.055 (the picture) and 0.133 and 0.175 (the text).
The 3B does not show the effect: its embeddings of
<|im_start|> and <|im_end|> have
mean squares of 1.91e-4 and 1.99e-4, a thousand times the epsilon (its
first norm leaves them 0.997 and 0.998 of what a norm without epsilon
gives, as the 2B's), and llama.cpp is as far from the pass with the
file's epsilon as from the one with 0 (the first token: 0.970 and 0.968;
OKLM 1.12). Its largest differences are not the epsilon's: over the
first 37 tokens OKLM is within 1.61 of the pass and llama.cpp within
1.50, and from the 38th on single logits move by up to 4.58 (OKLM) and
4.21 (llama.cpp) at the same tokens, the mean staying at 0.22: both
programs round the activations of this file's products to 8 bits, and
the pass rounds nothing.
test_qwen25vl_answer and test_qwen25vl_text
hold OKLM's CPU logits to that pass where its recording is present
(build/ref/mm/<tag>_same_answer_f64.bin and
<tag>_text_answer_f64.bin, 96 KiB each: the pass's
256 largest logits and their tokens before each of the 48, written by
tools/qwen2vl_ref.py --record;
tools/gen_mm_ref.sh runs it for the 3B and the 7B). This is
the bound that can be tight, the weights being the file's in both: over
those logits OKLM is within 0.943 (0.157 on average) on the 7B's
picture, 0.689 (0.148) on its text and 3.17 (0.225) on the 3B's picture,
bounded at 1.5, 1.5 and 5.0 with a mean of 0.35; llama.cpp, printed
beside it and not bounded, is 2.31, 1.95 and 4.21 from the same logits.
The comparison with llama.cpp keeps its bounds.
Not measured: the 32B and the 72B against the float64 pass (their files, 18.5 and 22.1 GiB, were removed and do not fit beside the other work on this disk; the pass itself would take their 64 and 80 layers in minutes), so whether their special tokens' rows are as small as the 7B's is not known; whole answers on the BF16 file (14.2 GiB, removed), where the first token alone was compared; whole answers of the prompts with several pictures.
The other sizes of Qwen2.5-VL
The 3B, the 32B and the 72B have the 7B's encoder (32 blocks of width
1 280, windowed attention) with a merger as wide as their language model
(2 048, 5 120 and 8 192), and the same qwen2vl architecture
(36, 64 and 80 layers). Each was fetched on 2026-11-14, run as the 7B
was and removed, one file on the disk at a time
(tests/public_models_large.txt has the URLs and SHA-256
sums; the catalogue has them as qwen2.5-vl-3b,
qwen2.5-vl-32b and qwen2.5-vl-72b):
ggml-org's Q4_K_M for the 3B (1.8 GiB) and the 32B (18.5
GiB), and for the 72B bartowski's IQ1_M (22.1 GiB, 1.75
bits a weight), since no 2-bit or 3-bit file of it (23.7 GiB and more)
left room on a disk with 26.8 GiB free; the F16 projector each time.
test_mm_public runs the three as it runs the 7B: two
pictures through the encoder against a float64 pass of the model's
definition and against llama.cpp's embeddings, then the answer about
llama.cpp's test image on llama.cpp's greedy tokens (48 for the 3B, 16
for the two large files), from the same embeddings and against its whole
path, then on OpenCL and on Vulkan.
| 3B Q4_K_M | 32B Q4_K_M | 72B IQ1_M | |
|---|---|---|---|
| Embeddings from the float64 pass (two pictures) | 3.5e-6, 4.8e-6 | 1.2e-5, 2.9e-6 | 2.7e-6, 3.8e-6 |
| from llama.cpp's | 4.2e-3, 6.8e-3 | 1.2e-2, 3.4e-2 | 4.1e-3, 9.6e-3 |
| The encoder on OpenCL, from the CPU's; time | 8.2e-6, 1.0e-5; 0.15 s | 4.8e-5, 1.4e-5; 0.15 s | 1.3e-5, 2.7e-5; 0.15 s |
| on Vulkan | 1.6e-5, 1.2e-5; 0.12 s | 1.7e-4, 8.5e-5; 0.12 s | 8.7e-6, 2.1e-5; 0.12 s |
| Rows of prompt, as llama.cpp's | 418 | 418 | 418 |
| Choices llama.cpp's: same embeddings, whole path | 47, 47 of 48 | 16, 16 of 16 | 15, 15 of 16 |
| Logits from llama.cpp's: largest (mean) | 4.93 (0.19), 1.96 (0.18) | 3.40 (0.25), 3.39 (0.25) | 1.18 (0.19), 1.29 (0.15) |
| CPU: prompt rows/s, tokens/s | 241, 14.5 | 23, 1.5 | 8, 0.7 |
| OpenCL, the layers that fit: choices the CPU's, logits; rows/s, tokens/s | 36 of 36: 47, 4.18; 3 460, 141 | 61 of 64: 16, 0.72; 253, 13.7 | 62 of 80: 14, 0.83; 29, 2.7 |
| OpenCL, half the layers, paged cache | 47, 4.34; 493, 24.0 | 16, 0.73; 43, 2.8 | 15, 0.78; 15, 1.2 |
| Vulkan, the layers that fit | 48, 3.42; 2 458, 193 | 16, 0.87; 227, 14.9 | 14, 0.84; 29, 2.7 |
| Vulkan, half the layers, paged cache | 48, 1.49; 500, 28.5 | 16, 0.74; 42, 2.9 | 15, 0.80; 15, 1.3 |
oklm describe, the prompt: CPU, OpenCL, Vulkan |
8.72 s, 0.37 s, 0.32 s | not run, 3.50 s, 3.70 s | not run, 21.0 s, 18.0 s |
The 3B was also run with half its layers and the contiguous cache (47
and 48 of 48) and with the CPU's half streamed for the prompt (48 and
48, 1 163 and 1 192 rows/s). In the first of its layouts' rows the two
large files have what oklm_context_params.n_gpu_layers = -1
places on the 20 GiB card, the other layers on the CPU;
oklm describe with its context of 4 096 and the encoder on
the device places 59 of the 32B's 64. The 72B's rates were measured
while another test ran on the machine. Where the 72B's choices part from
the CPU's (14 of 16) the two tokens are 0.18 apart on the CPU: the tests
do not hold choices that part at such near ties against a device on
these short answers. The 1.75-bit 72B reads the page ("Men Walk on Moon"
/ The subheadlines include: / "Astronauts Land on Plain [...]"); the 32B
names the paper and its date ("The newspaper shown is the New
York Times, dated Monday, July 21, 1969
[...]"). The 32B's embeddings are the furthest of the four from the
CPU's on a device (1.1e-4 on OpenCL with the blocks alone there, 1.7e-4
on Vulkan, for one of the two pictures: its projector's values reach
46), which the tests' bound of 2e-4 holds. Not run: the other
quantizations, a video and several pictures on these three sizes, the
32B and the 72B on the CPU through oklm describe.
A video for Qwen2-VL and Qwen2.5-VL
oklm describe gives these models a video
(.ogv, or another container through --ffmpeg)
as their processors and code do
(video_processing_qwen2_vl.py,
modeling_qwen2_vl.py and
modeling_qwen2_5_vl.py of transformers), where the other
projectors get the frames as still images:
oklm describe -m models/qwen2.5-vl-7b/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
--mmproj models/qwen2.5-vl-7b/mmproj-F16.gguf \
--fps 2 --frames 8 -p "What happens in this video?" clip.ogv
- Frame pairs. The patch embedding is a convolution
over two frames: the file's first kernel
(
v.patch_embd.weight) reads the first frame of a pair and the second (v.patch_embd.weight.1) the second, one grid of embeddings a pair (oklm_vision_encode_pair()). A still image is the same frame twice, which is the two kernels summed: the image path, unchanged. An odd count repeats its last frame, as the processor does: that pair is the last frame as a still image, bit for bit. Each pair then goes through the blocks and the merger on its own, as an image does: in the models' code a row sees the rows of its own grid only, in the blocks with full attention as in Qwen2.5-VL's windows. - The frames' size. Every frame is stretched
(bicubic) to one size, that of the first within the limits of the
models' video processor: 128 to 768 embeddings a frame (an image's are 4
to 4 096), sides in units of 28 pixels
(
oklm_vision_frame_size(),oklm_vision_frame()).--tiles SIDEsets the largest area to SIDE x SIDE pixels instead; under 128 embeddings the smallest area is an image's, so smaller frames can be asked for (a prompt that fits a small context, or the CPU's patience). - The prompt.
<|vision_start|>, the grids of all the pairs one after the other,<|vision_end|>, right before the question as an image is: the processor writes one<|video_pad|>an embedding (pairs x rows x columns) between the two. No text gives a frame's time: Qwen2.5-VL's "absolute time" is a position, not a token. - The time position. A row of the video has, as
height and width, the video's start p plus its row and column in its own
grid, and as time p plus what its grid has advanced: its pair's index
for Qwen2-VL, whatever the frames' times; for Qwen2.5-VL the index times
the seconds a pair covers times two
(
vision_config.tokens_per_secondof Qwen2.5-VL-7B-Instruct's configuration; the mmproj file has no key for it), the fraction dropped. A pair covers twice the distance between frames: at the two frames a second of the model's processor a pair is one second and two positions; atoklm describe's default of one frame a second, four. The text after the video continues from the largest of the three positions used plus one, which the time can be: three pairs of Qwen2.5-VL at one frame a second end at p + 8 when the grid is narrower than that.oklm_vision_time_step()gives the step,oklm_eval_embd_video()andoklm_eval_batch_video()take the rows with the grid's columns, rows and step (a video in pieces, a cut in its middle, a copy, the paged cache and its prefix cache as for an image: the context's two numbers a row are enough, a video's start being its last row's time less what its grid has advanced). One or two frames are one grid: an image's positions. - transformers after 4.49. Qwen2.5-VL's time is taken here as the model was published (4.49.0: index x seconds x 2, then truncated). transformers 4.57.1 truncates the seconds a pair first: the same positions whenever a pair covers whole seconds (its processor's default), and every pair at the first one's time when it covers less than one (four frames a second or more). OKLM keeps the published form there, by the owner's decision (2026-11-14).
- Frames taken.
--fps(1) and--frames(8) choose the frames as for the other models; the processors' own sampling (two frames a second, 4 to 768 frames) is not applied (the owner's decision, 2026-11-14: these defaults stay), and the time position follows the distance between the frames kept. Eight frames of the smallest size are 512 embeddings, of the largest 3 072: give the context room with-c. - Through the server. A
video_urlpart of a chat request is this video item (server.md): the scheduler gives its rows to the model in the pieces its steps cut, each with the grid's columns, rows and time step (oklm_eval_batch_video()), and the item's hash (frames, count, spacing,--tiles) keys its embeddings in the server's cache and its rows in the prefix cache. The clip of the run below (four frames a second apart of 448 x 336) throughoklm serveand throughoklm describe --temp 0on Qwen2-VL 2B (Q4_K_M, the CPU): 384 embeddings and 418 prompt tokens by either way and the same 64 tokens ("The video starts with a close-up shot of a newspaper with the headline "Men Walk on Moon" and the text "Astronauts Land on Plain, Collect Rocks, Plant Flag." It then cuts to a black and white image of a man walking on the moon. [...]"); the request sent again finds 416 of its 418 rows in the prefix cache, the video's embeddings kept;/v1/messageswith the video as avideoblock gives the same text. - Not covered. The processors' rule of at least four frames (by decision, above); a video's sound (these files have no audio encoder); several videos at once in the model's own batch.
The encoder of Qwen2-VL and Qwen2.5-VL on a device
With oklm_vision_use_backend() the 32 blocks of these
encoders run on the device, as the other encoders' do, and with them,
since 2026-11-14, the patch embedding before them and the merger after
them (above);
the CPU's pass is unchanged (its rotation is still the host's table, the
same bits as before).
- Positions. The rows' two-dimensional rotation is
the rotary op with three positions a row and a band of pairs each
(
oklm_rope_parts_factors()): a row's positions are its patch's row, its column and 0, the first quarter of a head's values turns by the first, the second quarter by the second, no pair is left to the third. The op's frequencies run on across the bands, the model's start again at each: the op is given a base of 1 (every pair's angle is then its band's position) and a factor a pair, 10000^(i / (d/4)), which divides it. The angle is the host table's to the last place (the table multiplies by the inverse). - Windows. A block with windowed attention (28 of
Qwen2.5-VL's 32) is one attention op,
oklm_attention_spans(): bidirectional attention with a table of two values a row, the first row of its window and one past the last, uploaded once a picture beside the positions (the rows are ordered window by window already, so a window is a run of rows). Both backends attend it on their bidirectional tiles in one launch a block (opencl.md, vulkan.md): a block of 32 rows walks the tiles of keys from the first row any of its rows sees to the last, each row keeping the scores of its own window, so windows of any sizes are taken, the smaller ones at the right and bottom edges included. A picture of 273 embeddings has 24 windows, one of 391 has 30, one of 1 225 has 81: that many launches a block before. Where a backend's tiles do not take the operands (OpenCL:OKLM_CL_ATT_BLOCKS=0, fewer than 16 rows; Vulkan: fewer rows than half a block of the tiles; no published file's shapes) the op returnsOKLM_ERR_UNSUPPORTEDand the block runs the attention op once a window, on views of the window's rows in the queries, keys, values and output, as it did. On the CPU the pass is the host loop it was. The launches were a small part of the time: of the 0.455 s of kernel time of the test image's prompt on OpenCL (OKLM_CL_PROFILE=1, 2026-11-14) the windowed blocks' attention was 0.029 s in 840 launches and is 0.006 s in 28; Vulkan recorded its dispatches in one pass already and its times did not move. Qwen2.5-VL's encoder is faster than Qwen2-VL's, whose 32 blocks attend over every row. - The MLP. Qwen2-VL's quick-GELU is three element-wise ops, as CLIP's; Qwen2.5-VL's gate is a product, SiLU and a multiplication, and its inside width of 3 420 is padded to 3 424 for the second product (the padded values stay zero).
- Memory. The blocks' weights (most of the file's 1.3
GB) and a buffer of activations for a picture of 1 280 embeddings (5 120
rows: 0.49 GiB for Qwen2-VL with the patch embedding and the merger on
the device, a little less for Qwen2.5-VL, the rows' positions and spans
included), which
oklm_vision_device_bytes()reports for what shares the device: 1.74 GiB and 1.71 GiB with the weights. A larger picture, up to the 4 096 embeddings of the default, gets a larger buffer when it comes: 0.99 GiB of activations at 2 552 embeddings and 1.55 GiB at 4 015 for Qwen2-VL, the device giving up 2.26 GiB and 2.62 to 2.80 GiB in all for the encoder (OpenCL's count of its free memory on the RX 7900 XT, two runs; 2.14 GiB and 2.65 GiB for Qwen2.5-VL).oklm_vision_device_bytes_at()counts the buffer of a picture of a given size, and a server reserves that of the largest picture it accepts before it sizes its cache (three quarters of a sequence's context: 3 072 embeddings at-c 4096, 2.43 GiB with the weights where 1.74 GiB were kept; server.md).oklm describereserves nothing: it has no cache to size. When the device has no room for it, that picture is encoded on the CPU, with a notice ("vision: the device has no room for the 1010 MiB of activations of a picture of 10208 patches, the encoder computes it on the CPU": 140 s then for the 2B's prompt, on a busy machine, where the device takes 5 s), and the next picture asks the device again.OKLM_MM_DEVICE_ACT_BYTES=Nbounds the buffer for a test of that path (test_vision: the picture's embeddings are then the CPU's to the bit, the next one's the device's). - Who uses it.
oklm describe,oklm serve --mmprojand the captioner ofoklm rag index --caption-model(with--gen-backend, rag.md) put an image encoder on the device their language model's layers are on.
Measured on 2026-11-14 (RX 7900 XT, the language model's layers on
the device in each case): the prompt of oklm describe on
llama.cpp's test image (391 embeddings, a prompt of 418 rows), before
(the encoder on the CPU) and after; and the encoder alone on the tests'
picture of 273 embeddings (test_mm_public, a second image,
the kernels built):
| Model | Prompt, encoder on the CPU: OpenCL, Vulkan | Prompt, blocks on the device: OpenCL, Vulkan | One image: CPU | OpenCL | Vulkan |
|---|---|---|---|---|---|
| Qwen2-VL 2B | 8.32 s, 8.57 s | 0.48 s, 0.43 to 0.45 s | 4.84 to 5.67 s | 0.26 s | 0.23 to 0.24 s |
| Qwen2.5-VL 7B | 7.20 s, 8.71 s | 0.58 to 0.59 s, 0.55 to 0.56 s | 4.43 to 4.73 s | 0.23 s | 0.20 s |
Vulkan's columns are with the encoder's products at F32's precision
(above: 0.37 s and 0.48
s for the prompts, 0.18 s and 0.15 s for the encoders with the
activations rounded to F16, OKLM_VK_WIDE=0); the Vulkan
figures of the next two paragraphs were measured with that rounding, and
again without it at the day's end (the tables measured again: a
picture of 1 225 embeddings in 1.67 to 1.68 s for Qwen2-VL 2B and 1.22 s
for Qwen2.5-VL 7B on Vulkan, 1.60 s and 1.35 to 1.38 s on OpenCL).
Qwen2.5-VL's row is with its windowed blocks in one launch each; with a
launch a window, on the same day and build, its prompt took 0.61 to 0.63
s on OpenCL and 0.48 to 0.49 s on Vulkan, and its encoder alone 0.25 s
and 0.14 to 0.15 s. Its prompt on a picture of 2 000 x 2 000 pixels (1
225 embeddings, 81 windows a block) went from 1.80 to 1.83 s to 1.72 to
1.76 s on OpenCL and stayed at 1.40 to 1.42 s on Vulkan.
A picture of 2 000 x 2 000 pixels (1 225 embeddings at the default
largest size, 4 900 rows) takes Qwen2-VL 2B's prompt from 43.4 s to 1.79
s on Vulkan and to 1.89 to 1.91 s on OpenCL (6.66 s on its kernel of one
work-group per head and token, 2.06 s on the first form of its
bidirectional tiles: opencl.md; Vulkan 1.78 to
1.96 s in the session of the last figure, 0.72 of OpenCL's 1.48 s of
kernel time the encoder's attention and 0.42 s its products). Through
oklm serve a request that is decoding waits 0.18 s (Vulkan)
and 0.30 s (OpenCL) for the encoder of the test image (server.md).
With the blocks on OpenCL the embeddings are within 5.0e-5 to 5.3e-5
(relative norm) of the CPU's for Qwen2-VL 2B and 1.7e-5 for Qwen2.5-VL
7B (1.3e-5 to 1.7e-5 with a launch a window); on Vulkan within 1.9e-5 to
3.0e-5 and 9.6e-6 to 1.1e-5 (above: 1.1e-2 to 1.5e-2 and
4.8e-3 to 6.3e-3 while its matrix units took the encoder's activations
rounded to F16. Qwen2-VL's activations are large: its embeddings reach
80; llama.cpp's own are 2.0e-2 to 2.4e-2 from the CPU's for that model).
The tiny files give the CPU's embeddings within 5.4e-7 on OpenCL and
6.0e-7 on Vulkan (test_vision). The answer about the
newspaper is the one the CPU's embeddings give, the language model on
the same device both times: 48 (OpenCL) and 47 (Vulkan) of 48 greedy
choices for Qwen2-VL 2B (parting where the two tokens were 0.24 apart),
47 and 47 of 48 for Qwen2.5-VL 7B (0.19 and 0.11 apart; 47 and 48 with
the rounded activations, 48 and 47 with a launch a window: the near ties
fall as the sums' order does; test_qwen2vl_encoder_devices,
test_qwen25vl_encoder_devices). oklm describe
writes the same 48 tokens before and after for the 2B on both backends;
the 7B's answers part at one near tie ("and" against "and below it,"),
as its two backends already did with the encoder on the CPU. Qwen2-VL
2B's wording moves at such ties with whatever changes a sum's order: of
five pictures asked about at once through oklm serve,
beside transcriptions, streams and speech, one (Vulkan) and three
(OpenCL) answers were oklm describe's to the token, the
others a reading of the same picture in other words.
Audio
The spectrogram is whisper.cpp's: a reflected half window at the start, 30 seconds of silence at the end, a 400-sample periodic Hann window every 160 samples, a Slaney mel filterbank from 0 to 8 kHz (area normalized), log10 clamped 8 below the maximum and scaled to (x + 4) / 4, cut into 30-second chunks of 3000 frames.
The Whisper encoder: two convolutions (as products over unfolded frames), GELU, positions, pre-norm blocks with bidirectional attention, a final norm.
Its products are an encoder's: 1500 positions (3000 for the first
convolution) against float matrices, nearly all the time of a window
with the large checkpoints. They go through the blocked kernel
(oklm_matmul_dense(), core.md). The decoder's products are one
token against whole matrices, so they stay oklm_matmul()'s.
The spectrogram's frames are computed on the worker team, and a frame of
zeros (the 30 seconds of padding) without its transform.
oklm_whisperloads a Hugging Face Whisper directory (config.json, model.safetensors, tokenizer.json): speech recognition below.oklm_audioloads an mmproj GGUF with an audio encoder. Theultravoxprojector stacks 8 frames, applies an RMS norm, a SwiGLU MLP and another RMS norm. The last group of frames is padded, as the original model does (llama.cpp drops it: 188 embeddings for 30 seconds against 187).
The audio encoder on a device
With the language model's layers on a device
(--backend opencl|vulkan, --gpu-layers),
oklm describe and oklm serve --mmproj put the
blocks of the Ultravox encoder there too
(oklm_audio_use_backend()), through the path of Whisper's
encoder (Whisper on a device): the
same blocks, the same ops, no kernel of its own. The blocks' weights and
the final norm's are copied to the device; each 30-second chunk's stream
is uploaded, its blocks and the final norm are one pass there
(oklm_backend_pass_begin()), and their output is read back
for the projector. The spectrogram, the two convolutions before the
blocks and the projector (two RMS norms, the SwiGLU MLP, a linear layer
over 188 rows) stayed on the host; since 2026-11-14 the convolutions and
the projector compute on the device too (the convolutions on a device),
and what is read back is the projector's 188 rows. The figures below are
from before; with the whole encoder on the device and Vulkan's products
at F32's precision the prompt of oklm describe on jfk.wav
(228 rows) takes 0.33 s on OpenCL and 0.26 s on Vulkan against 5.35 s on
the CPU (2026-11-15, three runs each on the devices, the same figure
each time), and the encoder alone on a second sound 0.295 s and 0.228 s
against 4.68 s (test_mm_public), its embeddings 2.4e-4 and
7.2e-5 from the CPU's.
Measured on 2026-11-14 (RX 7900 XT, Ultravox v0.5 with Llama-3.2-1B Q8_0 and its F16 projector, the 32 blocks of Whisper large-v3-turbo's encoder; jfk.wav, 188 embeddings; the language model on the device in each case, the encoder on the CPU before; another job was using the CPU):
| OpenCL | Vulkan | |
|---|---|---|
the prompt of oklm describe (228 rows), before ->
after |
5.77 s -> 0.41 s | 5.32 s -> 0.26 s (0.33 s since the encoder's products keep F32's precision) |
a chat request of oklm serve with a new sound, 48
tokens, before -> after |
5.80 to 5.88 s -> 0.57 to 0.60 s | 5.65 to 5.79 s -> 0.42 to 0.43 s |
the encoder alone, a second sound (test_mm_public; the
CPU 4.8 to 5.9 s) |
0.36 to 0.38 s | 0.23 s (0.30 s since) |
| the embeddings against the CPU's, relative (largest difference) | 3.1e-4 (3.1e-3) | 9.6e-3 (8.1e-2); 5.5e-5 (8.1e-4) since |
The answers are the ones the CPU's embeddings give: the 48 tokens of
oklm describe and of three chat requests through the server
(the recording, its first 8 and its first 5 seconds) are the same text
before and after on each backend. Vulkan's 9.6e-3 was the rounding of
its matrix units, which took the encoder's activations as F16, as for
the vision encoders and as llama.cpp's own (the CPU's embeddings are
1.1e-2 from llama.cpp's); the encoder's products keep F32's precision
there since 2026-11-14, for 0.07 s a sound (the encoders' precision on
Vulkan; a chat request with a new sound and 8 tokens takes 0.28 s on
Vulkan and 0.36 s on OpenCL alone since, 0.40 s and 0.50 s among a
stream: server.md).
OpenCL's is larger than a vision encoder's (1e-5): it comes with the
tiled products (gm_f16; 3.1e-5 with
OKLM_CL_GEMM_TILES=0), through 32 blocks. It is rounding,
not a wrong term: each output of a tile is one running sum over the 1
280 or 5 120 terms of a product, and the same kernel with a partial sum
every 32 terms gives 4.5e-5, every 128 terms 1.1e-5, for 7 % more time
in the products (opencl.md); the
single sum is kept. On the tiny file of the tests
(test_audio_enc_devices in
tests/mm/test_audio.c: a random encoder of 2 blocks, 32
values wide, two chunks) the embeddings are 1.9e-6 (OpenCL) and 1.7e-6
(Vulkan) from the CPU's on the RX 7900 XT, 2.0e-6 and 1.8e-6 on the
Intel UHD 750.
Limits: the device takes a copy of the blocks' weights (about the
file's 1.4 GB) and 0.1 GB of activations
(oklm_audio_device_bytes()); a device without that room
leaves the encoder on the CPU with a notice
(the audio encoder runs on the CPU), and a file with
quantized blocks stays there without one, as a vision encoder's does. A
sound longer than 30 seconds is one pass a chunk. The spectrogram is
host work (24 ms for jfk.wav). No published file was run on the UHD
750.
Speech recognition
oklm_whisper (include/oklm/mm/audio.h)
loads a Hugging Face Whisper directory and decodes greedily with
cross-attention and a KV cache. Everything that differs between the
checkpoints is read from the files: the mel bin count (80, or 128 for
the large-v3 family, the filter bank built for either), the layer counts
of the encoder and of the decoder (32 and 4 for large-v3-turbo), and the
ids of the special tokens, looked up by name in the tokenizer: the
languages are the tokens between the start token and the tasks'
(large-v3 has one more, <|yue|>, which moves the
tokens after it), the timestamps those after
<|notimestamps|>. The weights are F32 (base, small)
or F16 (large-v3, large-v3-turbo): the matrices of an F16 checkpoint,
the token embeddings among them, are read where the file is mapped, and
its biases, norms and positions are widened to F32 once at load.
Whisper on a device
oklm_whisper_use_backend(w, backend)
(oklm transcribe --backend opencl|vulkan) copies the
weights of the encoder's blocks, of the decoder and the output matrix to
a device, and a window is then computed there through the same ops as on
the CPU: the blocks of the encoder (bidirectional attention over the 1
500 positions), its final norm, the keys and values of the decoder's
cross-attention, and for each token the decoder's blocks (causal
attention over a cache of its own in the device's memory, the keys and
values written where the cache keeps them, cross-attention over the
encoder's output) and the product with the token embeddings. The host
keeps the spectrogram and the choice of each token with its timestamp
rules: the first convolution's windows are uploaded once a window, the
logits are read back once a token. The two convolutions before the
blocks were host work too (0.08 to 0.11 s a full window) until
2026-11-14 (the convolutions on
a device); the table below is with them there, the paragraph after
it from before.
Where a 30-second window's time goes (RX 7900 XT, measured again on
2026-11-15 with the convolutions on the device and Vulkan's products at
F32's precision, load average 3.7 to 5.6; one window of 69 tokens,
--no-timestamps, two runs on each device and one on the
CPU, seconds; the encoder is waited for before the decoder starts; the
spectrogram, 0.045 s, is in the window and in neither part):
| Model | CPU | OpenCL | Vulkan | |
|---|---|---|---|---|
| large-v3-turbo (32 encoder blocks, 4 decoder blocks) | encoder | 4.93 | 0.27 | 0.20 to 0.21 |
| decoder | 1.10 | 0.13 to 0.14 | 0.11 | |
| the window | 6.07 | 0.45 | 0.36 | |
| large-v3 (32 and 32) | encoder | 5.09 | 0.27 | 0.20 to 0.21 |
| decoder | 6.25 | 0.60 | 0.46 | |
| the window | 11.4 | 0.92 | 0.71 | |
| small | the window (58 to 66 tokens) | 2.21 | 0.29 | 0.24 |
| base | the window | 0.55 | 0.17 | 0.14 |
jfk.wav (11 seconds, with timestamps) in the same session: 0.27 s on Vulkan and 0.35 s on OpenCL with large-v3-turbo against 5.19 s on the CPU, 0.44 s and 0.59 s with large-v3 (7.86 s), 0.13 s and 0.16 s with small (1.33 s), 0.08 s and 0.09 s with base (0.37 s). The day's first table, before the convolutions went to the device, had the encoder at 0.34 to 0.36 s on OpenCL (timed with every kernel waited for) and 0.17 s on Vulkan (its activations rounded to F16), the windows at 0.47 to 0.49 s and 0.33 s for large-v3-turbo, 0.97 s and 0.66 s for large-v3.
On the CPU the encoder is four fifths of large-v3-turbo's window and
the decoder more than half of large-v3's: both are on the device.
OpenCL's encoder and windows are with its bidirectional attention on
tiles of its own (opencl.md); on the kernel of
one work-group per head and token the encoder took 0.66 s (that kernel
69 % of a run's kernel time) and the windows 0.83 and 1.32 s. Its
encoder is timed with every kernel waited for
(OKLM_CL_PROFILE=1), its windows without. Of the kernel
time of jfk.wav's window (0.28 s) the encoder's attention is 0.085 s and
the matrix products (gm_f16) 0.14 s: these are F32 products
where Vulkan's matrix units take F16, and what is left of the distance
between the two (jfk.wav transcribed in 0.39 to 0.40 s on OpenCL, 0.30
to 0.32 s on Vulkan, 0.35 to 0.36 s on OpenCL with
OKLM_CL_WMMA=1, its products then 0.035 s). Vulkan's column
and these figures are with the encoder's activations rounded to F16 in
the matrix units; since 2026-11-14 its products keep F32's precision (the encoders' precision on
Vulkan), as do the keys and values of the cross-attention, and
jfk.wav goes from 0.24 to 0.25 s to 0.31 to 0.32 s with large-v3-turbo
and from 0.41 to 0.49 s with large-v3 (one session;
OKLM_VK_WIDE=0 for the first figure of each); the table
above is with them. Whisper small took 3.14, 0.43 and 0.35 s for the
same 30 seconds decoded with timestamps (before the convolutions went to
the device).
The text is the CPU's on the samples the tests use: the tiny
checkpoints' texts, segments, times and languages
(test_whisper_devices in
tests/mm/test_audio.c, on the RX 7900 XT and the Intel UHD
750, both backends), and jfk.wav with base, small, large-v3-turbo and
large-v3 (tests/mm/test_mm_public.c). A device's products
sum in another order than the CPU's, so a token whose two best
candidates are all but tied may differ on another recording; none did
here.
A server's recognizer takes a device with
--transcribe-backend (cpu by default): a
window's encoder and each token of the decoder are passes on the device
(oklm_backend_pass_begin()) that take their turn with the
scheduler's steps, the encoder's between two of its blocks: the streams
wait 0.015 s (Vulkan) or 0.02 s (OpenCL) for a block of large-v3-turbo's
encoder, where they waited 0.20 s and 0.28 s for its pass (server.md).
Limits: widths that are not a multiple of 32 values (the tiny checkpoints' 16) have their products handed back to the CPU, the rest of the pass staying on the device; the device takes a copy of the blocks' weights (about the file's size: 1.6 GB for large-v3-turbo, 3.1 GB for large-v3) and 0.2 to 0.8 GB of activations, and a device without that room leaves the recognizer on the CPU with a notice; on the Intel UHD 750 Whisper base takes 7.4 s (OpenCL) and 2.7 s (Vulkan) for jfk.wav where the CPU takes 0.6 s.
The decoder multiplies one token by each of its matrices and by the token embeddings (the logits): products bound by the memory they read, about 550 MiB a token for Whisper small in F32. OpenAI trained and published its checkpoints in half precision, and the F32 files of base and small hold nothing but half-precision values (every one of their 245 and 479 tensors converts to F16 and back unchanged). At load, a decoder matrix stored in F32 whose values are all halves gets an F16 copy, and the products read that: the same values (the dot product widens the halves back to the same floats) from half the memory. The decoder of small then takes 1.7 to 2.2 s for the 66 tokens of a window instead of 2.5 to 3.2 s, that of base 0.5 to 0.7 s for 58 tokens instead of 0.9 to 1.5 s (eight threads, four alternations on a loaded machine). A matrix with one value that is not a half's keeps the file's floats.
The language. The logits after the start token,
taken over the language tokens, are the model's opinion of the language
spoken: oklm_whisper_detect_language() returns the most
probable one and its probability among them, and a recognition without a
language (--language auto, the default) uses the one of its
first window for the whole recording.
oklm_whisper_n_languages() and
oklm_whisper_language() list the codes a model knows (99
for base and small, 100 for the large-v3 family),
oklm_whisper_describe() reads them from a directory's
tokenizer without loading the weights. Translation is
the other task token: <|translate|> in place of
<|transcribe|> makes the model write English
(--translate).
Timestamps. oklm_whisper_run() decodes
with Whisper's timestamp tokens (<|0.00|> to
<|30.00|>, one every 20 ms) under the rules of the
reference implementation (openai/whisper's
ApplyTimestampRules):
- a window opens with a timestamp, of at most one second;
- timestamps come in pairs, the end of a segment and the start of the next, except a single one right before the end of text: after one timestamp that follows text, no text; after two, no timestamp;
- timestamps never decrease, and a segment ends after it starts;
- when the timestamps together are more probable than the most probable other token, the next token is a timestamp.
The tokens of a window are then cut at each pair of timestamps into
segments with a start, an end and a text
(oklm_whisper_segment), and the next window starts
where the last closed segment ended, not 30 seconds further:
the model closes its segments at pauses, so the cuts fall in silences
and the sentence a window's end interrupts is decoded again, whole, by
the next. A window that ends on a single timestamp (the speech stopped
inside it), or holds no pair, is followed by the next 30 seconds. A
window whose no-speech probability (the <|nospeech|>
token after the start token) is above 0.6 and whose tokens have a mean
log-probability under -1 writes nothing.
oklm_whisper_transcribe() and
--no-timestamps are the plain form: no timestamp tokens, a
window every 30 seconds wherever that falls (a recording of exactly 30
seconds is one window: nothing is decoded from the padding after it), so
the sentence spoken across a cut can lose words. The chat window's
dictation uses it for its short recordings.
What it does not do: beam search and the temperature fallback (the decoding is greedy: a window the reference would decode again at a higher temperature keeps its first text), a prompt from the text before the window (so a name is not spelled the same way from one window to the next by construction, and a first word that starts exactly at a window's start can be missed), word-level times, and a voice-activity detector before the model.
Speed. Seconds for one window: a recording of exactly 30 seconds, decoded without timestamps so that it is one window (58 to 69 tokens), the model's loading left out. Eight threads on an i7-11700K (8 cores, 16 threads) that five other jobs shared, so each figure is three alternations of the two binaries and the load average is said:
| Model | before | after | load |
|---|---|---|---|
| base | 1.45 to 1.97 | 0.78 to 1.42 | 9 to 11 |
| small | 4.61 to 7.17 | 2.48 to 3.98 | 9 to 11 |
| large-v3-turbo | 11.48 to 17.45 | 7.78 to 11.94 | 10 to 11 |
| large-v3 | 17.92 to 21.19 | 13.58 to 15.71 | 10 to 11 |
"Before" is the encoder's products computed one dot product at a time and the decoder reading F32 matrices; "after" the blocked product, the spectrogram on the worker team, the decoder's query, key and value as one loop and its matrices read as halves. Under a heavier load (16 to 19) the same alternation gave 20.9 to 39.2 s then 14.6 to 30.4 s for large-v3-turbo, and 55.7 to 87.2 s then 24.1 to 40.0 s for large-v3 (and once 151 s, for a cause not found: the page cache was full): what else the machine does counts more than anything here. On one thread large-v3 took 132.6 s of CPU before and 73.6 s after (load 8 to 10).
Where a window's time goes now (eight threads, load 10): for large-v3-turbo 11.4 s of 13.4 in the encoder (8 s its products, 2.7 s its attention) and 2 s in the decoder; for large-v3 13.5 s in the encoder and 12 s in the decoder, whose 32 layers read 1.5 GiB of matrices for each of the 69 tokens: that part is bound by memory, and only smaller (quantized) matrices would shorten it.
With timestamps a window starts before the end of the one it follows, so a recording can take more windows than its length in half minutes: the 127-second address takes five either way (small's start at 0, 29.1, 58.3, 85.6 and 114.7 s instead of every 30 s) and 7.7 s with base, 24 s with small (18 s without timestamps: fewer tokens) and 47 s with large-v3-turbo (44 s); a recording of exactly 30 seconds whose last sentence the window cuts takes two.
The convolutions on a device
With the blocks of Whisper's encoder on a device, the host still computed what stands before them, and for Ultravox what stands after. Timed on 2026-11-14 on jfk.wav (11 seconds: the frames after the sound are silence; RX 7900 XT, Vulkan, large-v3-turbo's encoder, milliseconds): the spectrogram 24.0; the first convolution 6.4 with 0.6 for its windows, its GELU 1.8; the second convolution 23.7 with 2.7 for its windows, its GELU 1.4; then the blocks, 200. For Ultravox the projector after them, 28.0. So 37 ms of convolutions and 28 ms of projector stood beside 0.20 s of blocks.
- The first convolution (kernel 3, stride 1) is, as on the CPU, a product over the windows the host unfolds from the spectrogram (3 x 128 or 3 x 80 values a frame, widened with zeros to a multiple of 32: 80 bins give 240, and the device's copy of the kernel has rows of 256). The windows are uploaded in place of the stream.
- The second (kernel 3, stride 2) reads the first
one's output, which is on the device. It is not unfolded there: the
kernel is copied as three matrices, one a position, and the convolution
is three products, each over a view of every second row from that
position on, the first written and the two others added to it
(
OKLM_OP_MATMULaccumulating). The first convolution writes its rows after one row of zeros, which is the padding the first position reads. The products are the blocks' (oklm_matmul_wide()), the GELUs and the positions are ops: no kernel and no op was added. - Ultravox's projector reads the encoder's output where it is: the 1 500 rows and four rows of zeros after them as 188 rows of eight frames (a reshape), the RMS norm, the first product, the SwiGLU of its two halves (views), the second norm and product. The 188 rows are read back, where the 1 500 of the output were.
src/mm/audio.c: wenc_convs_place() copies
the kernels, wenc_device_convs() computes a chunk,
ultravox_project_device() the projector.
OKLM_MM_DEVICE_ENDS=0 keeps the three in host memory (build.md);
oklm_whisper_device_parts() and
oklm_audio_device_parts() tell which it is.
Measured on 2026-11-14 (RX 7900 XT, one build, the convolutions and the projector in host memory -> on the device, three runs each):
| OpenCL | Vulkan | |
|---|---|---|
oklm transcribe of jfk.wav, large-v3-turbo |
0.38 to 0.40 s -> 0.35 s | 0.31 s -> 0.27 s |
| large-v3 | 0.63 s -> 0.59 s | 0.48 to 0.49 s -> 0.44 to 0.45 s |
| base | 0.10 s -> 0.09 s | 0.08 to 0.09 s -> 0.08 s |
the prompt of oklm describe with Ultravox, jfk.wav |
0.40 to 0.41 s -> 0.33 s | 0.33 s -> 0.26 s |
Ultravox's encoder alone, a second sound
(test_mm_public) |
0.363 s -> 0.298 s | 0.298 s -> 0.230 s |
| its embeddings against the CPU's, relative | 3.1e-4 -> 2.4e-4 | 5.5e-5 -> 7.2e-5 |
The texts of jfk.wav are the same before and after, and the CPU's,
with the four checkpoints on both backends
(test_mm_public), and Ultravox starts its answer with the
same words. The tiny checkpoints' texts, segments, times and languages
are the reference's both ways on both backends, and the tiny Ultravox
encoder's embeddings are within 1.9e-6 of the CPU's both ways
(test_whisper_devices and
test_audio_enc_devices in
tests/mm/test_audio.c; on the Intel UHD 750 too, within
2.0e-6). The second convolution's form (accumulated products over views
of every second row) is checked as an op against a convolution in
doubles, with Whisper's 1 280 channels among the shapes
(test_conv_views, both backends, the RX 7900 XT and the
Intel UHD 750: within 1.8e-7 of the sum of the terms' magnitudes).
Limits. The spectrogram stays on the host: 24 ms of jfk.wav's 0.27 s
on Vulkan, in doubles on the worker team, a transform of 400 points a
frame that has sound. A transform on a device was not written: it would
be a product by a table of cosines at F32's precision, and a logarithm,
which no backend has as an op. The device holds the two kernels (the
second as its three matrices), the biases and the positions beside the
blocks; a device without that room keeps the convolutions in host memory
with a notice, the blocks staying on it, and likewise for Ultravox's
projector (oklm_audio_device_bytes() counts both). The room
is asked for in this order: the blocks' weights, the projector's, the
activations, the convolutions' weights. Where the activations do not
fit, the buffer is asked again without the first convolution's windows
(the larger part for a narrow encoder, the blocks' for the published
ones), then with the projector given back: a device that holds what
OKLM_MM_DEVICE_ENDS=0 asks for holds the blocks, whatever
the rest takes. test_whisper_small_device and
test_audio_enc_devices run the tiny recognizer and the tiny
Ultravox encoder on a device with that room to the byte (the tests'
counting device with a bound, computing with the CPU's kernels,
tests/mm/count_backend.h): the notices, the bytes held and
the result of OKLM_MM_DEVICE_ENDS=0; with room for all but
the convolutions, the projector on the device; with a byte less than the
blocks need, the refusal and the CPU's result. No published file was run
on the UHD 750.
src/mm/pdf.c, with nothing but zlib (nilorea's
copy):
- Objects are found by scanning the file for
n g obj, a later definition replacing an earlier one (incremental updates), so a damaged or missing cross-reference table does not matter; object streams are unpacked. - The page tree is walked in order with inherited resources (every page object in file order when there is no tree). Each page's content runs through an interpreter of the text, font and graphics state operators; form XObjects are entered, inline images skipped.
- Strings are decoded by the font's ToUnicode CMap (bfchar, bfrange
and its array form, surrogate pairs), else its simple encoding:
StandardEncoding, WinAnsi or MacRoman, the encoding built into an
embedded Type 1 program, then /Differences glyph names (the Adobe Glyph
List's Latin, Greek and typographic names,
uniXXXX). Composite fonts with the Identity CMap need a ToUnicode map; UCS-2 CMaps are read directly. Ligatures are spelled out. - A space goes where the gap to the previous glyph passes 0.15 em and 0.4 of the font's own space (letter-spaced text stays one word); a line break where the baseline moves half a line, a blank line past two.
- Streams: Flate, LZW, ASCII85, ASCIIHex, RunLength, PNG and TIFF predictors.
- Encrypted documents open when their user password is empty, the
usual case of a document encrypted only to restrict printing or editing:
the standard security handler's revisions 2 to 6, RC4 40 to 128 bits,
AES-128, AES-256 (
src/mm/pdf_crypt.c: MD5, RC4, AES, SHA-384/512 of its own, nilorea's SHA-256). Others are refused (OKLM_ERR_UNSUPPORTED). - Bounded for untrusted files: nesting 64, 64 MB per decoded stream, 1 GB over a document, 100 million operators, 100 000 pages, forms 8 deep.
- Not read: scanned pages (images, no text without OCR), Type 3 bitmap fonts without a ToUnicode map, CFF fonts' built-in encodings, reading order beyond the content's own (columns and tables come as the content draws them).
Against poppler's pdftotext (a test-only reference),
tools/check_pdf.sh compares the words of both texts on
every PDF it finds: on the 618 PDFs of this machine's
/usr/share and /opt (TeX Live's documentation,
mostly; 4.1 million words), 96.5 % of pdftotext's words are found and
97.8 % of OKLM's are in pdftotext's text, weighted by words; most of the
rest is order (pdftotext reorders columns and tables), math set in TeX's
own glyph names, Type 3 bitmap fonts where neither reader gets text, and
code pdftotext splits into letters where OKLM keeps the words. The
largest document (222 000 words) takes 0.3 s.
Verification
tests/mm/test_ingest.c checks the documents (a DOCX
written by the test included). tests/mm/test_pdf.c reads
the PDF fixtures of tools/gen_pdf_fixtures.py (each feature
above, a damaged table, an incremental update), the encrypted copies
qpdf made of one of them at every revision against its plain text,
refuses one with a user password, checks the hashes and AES against
published vectors, and feeds damaged and hostile inputs: every
truncation of a fixture, byte flips, nesting past the limit, a 200 MB
decompression bomb. fuzz/fuzz_pdf.c fuzzes the reader.
tests/mm/test_video.c decodes a 2-second Theora fixture
(frames, times, sampling, the 440 Hz tone) and, with ffmpeg installed,
the same stream remuxed into Matroska through ffmpeg (mean difference
0.55 of 255 against the native frames); the same file held in memory
gives the file's frames byte for byte, and the limit on decoded pixels
refuses the frames asked for one pixel under their size and passes them
at it, on both paths. tests/mm/test_vision.c and
tests/mm/test_audio.c check every piece on small random
encoders written by the tests against double-precision references, image
operations, WAV formats, resampling and the spectrogram.
tests/mm/test_vision.c also loads the tiny projector files
of tools/gen_vision_fixtures.py (random weights in the
layout of a published file) and requires the embeddings its numpy
float64 implementation gives, and the bytes Pillow gives for the
projector's resize of a test picture; for the two Qwen layouts, the
embeddings of a wide and of a tall image (windows cut at the edges for
2.5) and the picture at the size the model's own rule gives it; and a
video of three frames (the first two through the two kernels, the last
one twice: within 6e-7 of the reference on values up to 2.1), the size
of a video's frame, and the video in a prompt: the rows the processor
writes placeholders for, between the two vision texts, the grid and time
step on the item, one frame the rows of that frame as an image.
tests/model/test_model.c runs a tiny qwen2vl
model on the cases of tools/gen_mrope_fixture.py (a numpy
pass with three positions a row: tokens, wide and tall images, two
images, an image at either end, a column of one embedding a row, and
videos: Qwen2-VL's step of one a grid, Qwen2.5-VL's of two, four, one
and a half and a half, a video then an image; the layout of each is
asserted there against transformers' get_rope_index()
written again in numpy, as are 98 videos of the published models' grid
of 23 x 17 at eight frame rates) seven ways each: one call an image,
pieces of 4 rows, a batch with another sequence's tokens on a contiguous
and on a paged cache, after a cut in the middle of an image, on a copy
of the sequence, and from rows found in the prefix cache (a cut inside
an image for the longer cases): within 3e-5 of the reference's logits,
the next position the reference's; a text gives bit for bit the logits
of the same tensors as qwen2.
tests/core/test_attention.c,
tests/model/test_model.c (cases of
tools/gen_mutual_fixture.py, a numpy pass of the gemma3
architecture with a window of 4: within 8e-7 on a contiguous cache, in a
batch and on a paged cache) and tests/server/test_server.c
(the server's answer is oklm_mm_eval()'s, and not the one
the same rows give read one after the other) check the rows that see
each other. tests/mm/test_mm_public.c (part of
make test-public) compares with llama.cpp, whose outputs
tools/gen_mm_ref.sh records with
tools/mtmd_ref.cpp from the very inputs OKLM computes. A
whole answer is asked about the picture kept beside its recording
(build/ref/mm/<tag>_answer.ppm), so a tree that holds
the recordings without the llama.cpp build compares the same pixels:
| Check | Result |
|---|---|
| SmolVLM-256M, overview tile of llama.cpp's test image | max difference 0.1 on values up to 109 |
| Gemma 3 4B, two pictures (one drawn by the test, 600 x 360; llama.cpp's test image, 640 x 488): the 896 x 896 pixels after the resize | llama.cpp's, byte for byte |
Gemma 3 4B, the 256 embeddings of each, against a float64 pass of
the encoder (tools/vision_ref.py) |
relative difference 8.9e-6 and 1.6e-5 (max 0.0013 on values up to 33) |
| the same against llama.cpp | relative difference 3.2e-3 and 6.9e-3 (max 0.22 and 0.77): llama.cpp is as far from the float64 pass, 3.2e-3 and 6.9e-3 |
| LLaVA 1.5 7B, the same two pictures: the 336 x 336 pixels after the resize | llama.cpp's, byte for byte |
| LLaVA 1.5 7B, the 576 embeddings of each, against the float64 pass of the model's definition | relative difference 5.4e-6 and 6.9e-6 (max 0.00033 on values up to 38) |
| LLaVA 1.5 7B Q4_K, "Describe the picture in detail." about llama.cpp's test image, llama.cpp's language model given OKLM's embeddings | the prompt's 595 rows are llama.cpp's (its own tokens around 576 image rows); on its 48 greedy tokens, 47 choices equal, logits within 0.38; they part at the fifth token, where llama.cpp's two best are 0.007 apart |
LLaVA 1.6 Mistral 7B, the same two pictures: the tiles of each (2 x
2 and the overview), against the model's own processor written again
with Pillow (tools/llava_next_ref.py) |
the reference's bytes, normalized: every pixel equal |
| LLaVA 1.6, the 576 embeddings of each of the ten tiles, against the float64 pass of the model's definition on the reference's pixels | relative difference 2.1e-6 to 7.6e-6 (max 0.0002 on values up to 37) |
| LLaVA 1.6, the rows of each image (1 948 and 2 340), against the model's own code written again with numpy, given the same ten tiles' embeddings | the same rows: no value differs |
| LLaVA 1.6 Mistral 7B Q4_K_M, "What does the newspaper say?" about llama.cpp's test image, llama.cpp's language model given OKLM's rows | 2 357 rows (2 340 of the image); on its 48 greedy tokens, 48 choices equal, logits within 0.32 |
| Gemma 3 4B Q4_K_M, "What does the newspaper say?" about llama.cpp's test image, llama.cpp's language model given OKLM's embeddings | the prompt's 273 rows are llama.cpp's (256 of the image); on llama.cpp's 48 greedy tokens, 47 choices equal, the first 12 in a row, logits within 3.78; where they part llama.cpp's two best are 0.84 apart |
| the same against llama.cpp's whole path (its own encoder) | 44 of 48 choices equal, the first 10 in a row, logits within 3.85; where they part llama.cpp's two best are 0.21 apart, 1.11 at the widest of the four choices that differ. llama.cpp given OKLM's embeddings leaves its own answer at that eleventh token too. With a new line after the image, the prompt until 2026-11-14 (274 rows), the whole path gave 47 of 48, the first 23 in a row, logits within 3.64, 0.32 apart |
| Qwen2-VL 2B and Qwen2.5-VL 7B, two pictures (the one drawn by the test, 600 x 360, and the same turned a quarter, 360 x 600): the 588 x 364 and 364 x 588 pixels after the resize | llama.cpp's, byte for byte |
Qwen2-VL 2B, the 273 embeddings of each (21 x 13, 13 x 21), against
the float64 pass of the model's definition
(tools/vision_ref.py) |
relative difference 1.3e-5 and 1.3e-5 (max 0.0012 on values up to 82) |
| the same against llama.cpp | 2.4e-2 and 2.0e-2 (max 4.7): llama.cpp is as far from the float64 pass |
| Qwen2.5-VL 7B, the 273 embeddings of each, against the float64 pass | relative difference 2.9e-6 and 2.4e-6 (max 1.3e-4 on values up to 17) |
| the same against llama.cpp | 3.0e-3 and 1.0e-2 (max 0.75): llama.cpp is as far from the float64 pass |
Qwen2-VL 2B Q4_K_M, a question without a picture (architecture
qwen2vl), llama.cpp's 48 greedy tokens fed one by one |
the prompt's 38 tokens are llama.cpp's; 48 choices equal, logits within 0.79 |
| Qwen2-VL 2B, "What does the newspaper say?" about llama.cpp's test image, llama.cpp's language model given OKLM's embeddings at mtmd's positions | the prompt's 418 rows are llama.cpp's (391 of the image, a grid of 23 x 17); 46 of 48 choices equal, the first 26 in a row, logits within 1.76; where they part llama.cpp's two best are 0.09 apart |
| the same against llama.cpp's whole path (its own encoder) | 45 of 48 choices equal, the first 26 in a row, logits within 2.34; where they part llama.cpp's two best are 0.24 apart |
| Qwen2.5-VL 7B Q4_K_M, a question without a picture | 47 of 48 choices equal, logits within 2.24; where they part llama.cpp's two best are 0.23 apart |
| Qwen2.5-VL 7B, the newspaper question, llama.cpp's language model given OKLM's embeddings | 418 rows (391 of the image); 46 of 48 choices equal, logits within 3.16; they part at the third token, where llama.cpp's two best are 0.40 apart |
| the same against llama.cpp's whole path | 46 of 48 choices equal, logits within 3.45; llama.cpp's two best 0.26 apart where they part |
| jfk.wav spectrogram (128 bins) | max difference 2.6e-5 |
| Ultravox on jfk.wav (Whisper large-v3-turbo encoder) | relative difference 0.011 over the 187 common rows (llama.cpp rounds activations to F16) |
| the same with the encoder's blocks on a device, against the CPU's | relative difference 3.1e-4 on OpenCL, 5.5e-5 on Vulkan (the audio encoder on a device) |
| Whisper base, small, large-v3-turbo and large-v3 on jfk.wav | the exact transcript of each (large-v3-turbo writes a comma after "And so") |
Speech recognition has transformers as its reference, llama.cpp
having no Whisper decoder: the same model.safetensors and
the same 16 kHz WAV, greedy, no timestamps, each 30 seconds on its own.
jfk.wav comes out word for word with the four checkpoints, and a
127-second radio address (five chunks) word for word with base (375
words) and small (365). With large-v3-turbo OKLM writes 357 of
transformers' 364 words, and with large-v3 214 of 215 on the first 75
seconds: each time a few words of the sentence a 30-second cut falls in.
The chunk in question transcribed alone gives transformers' text
exactly; inside a recording OKLM's spectrogram is the whole recording's
(its maximum sets the floor and the frames at a cut see the samples
across it, as in whisper.cpp), where transformers computes each chunk's.
tests/mm/test_audio.c also decodes a 35-second test signal
with two tiny random checkpoints (the layout of base, and that of
large-v3-turbo: 128 bins, F16, a shorter decoder, one more language
token) and requires the text a double-precision numpy implementation
decodes (tools/gen_whisper_fixtures.py), without timestamps
and with them: the language found and its probability, each segment's
times and text and so every window's start, for a given language and for
the translation task. That reference writes the timestamp rules, the
cutting and the seeking from openai/whisper's decoding.py
and transcribe.py, on its own.
With timestamps there is no reference implementation on the machine to compare real recordings with; what was looked at instead, on the CPU:
The cuts. The same 127-second address, with timestamps (the windows start where the last closed segment ended) and without (a cut every 30 seconds), words counted:
Model with timestamps a cut every 30 s what the cuts lost base 372 375 nothing: the three words differ both ways (one word missed with timestamps, "Voting", at a window's start) small 375 365 a sentence of 11 words large-v3-turbo 375 357 three fragments, 18 words large-v3 375 364 two fragments, 11 words The texts of large-v3 and large-v3-turbo with timestamps are the same word for word, and small's differs from them by two places ("and we" for "Americans", and a missing "In" at a window's start).
The language. French, German and Spanish readings and the English address, the first 30 seconds of each: the four models name the right language, with a probability of 0.985 to 1.000.
The times. A French fable of 52 seconds read with pauses: base, small and large-v3 close their segments at the pauses (large-v3: 0.88 to 5.56, 6.44 to 10.92, 11.98 to 17.90 ...), and the second window starts at one (25.16 s for base). The 11-second jfk.wav is one segment from 0.00 to 11.00 for base and small, three for large-v3.
Translation. The fable in English: large-v3 writes it ("Master crow on a perched tree held in his beak a cheese..."), small roughly, base only English words; large-v3-turbo writes French, as its authors say it does (it was trained without the task).
What goes wrong. Greedy decoding without the reference's fallbacks keeps a window's first text. Seen: large-v3-turbo closing a segment at the window's end (29.98 s) and so skipping the sentence spoken from 25 to 30 seconds of the fable; large-v3 adding "Ah!" after the sentence of jfk.wav when 20 seconds of silence surround it; and, with 20 seconds of silence before the speech, every model starting its first segment at 0.00 (a window must open within its first second), so that segment's start is the window's, not the speech's.
Gemma 3's encoder is where the two references part. llama.cpp gives F16 matrices their activations rounded to F16 (27 blocks here); a numpy pass that rounds them so moves 2.6e-3 from the float64 one, the distance llama.cpp is at. OKLM keeps them in F32 and stays within 2e-5 of the float64 pass, so the test holds it to that pass (1e-4) and to llama.cpp more loosely (2e-2).
The answer was read three ways on the same picture and question,
greedy: oklm describe and oklm serve --mmproj
(an image_url part and the question in a chat request)
write the same 64 tokens ("Here's the text from the New York Times
newspaper [...] Headline: MEN WALK ON MOON [...]
ASTRONAUTS LAND ON PLAIN; COLLECT ROCKS, PLANT FLAG"), and llama.cpp's
own run writes the same headline and subheading, its sentence parting
from OKLM's at the token counted above. Not run: a second image in one
prompt, a prompt long enough for the window of 1 024 positions to cut an
image's rows (the tiny model's window of 4 does, on the CPU and on the
devices).
On OpenCL and on Vulkan (2026-11-14, RX 7900 XT) the same 273 rows
and 48 tokens, against the CPU's logits and against llama.cpp's answer
from the same embeddings (test_mm_public; each layout on a
model loaded for it):
| Gemma 3 4B, layout | Choices the CPU's, of 48 | Logits from the CPU's: largest, mean | Choices llama.cpp's | Logits from llama.cpp's | Prompt, rows/s | Tokens/s |
|---|---|---|---|---|---|---|
| CPU | 47 | 3.78 | 184 | 10.7 | ||
| OpenCL, every layer, contiguous cache | 46 | 3.67, 0.353 | 46 | 3.50 | 2 809 | 104 |
| OpenCL, 17 of 34 layers, paged cache | 46 | 3.18, 0.298 | 46 | 4.32 | 391 | 21.8 |
| Vulkan, every layer, contiguous cache | 45 | 3.35, 0.323 | 45 | 3.47 | 2 740 | 146 |
| Vulkan, 17 of 34 layers, paged cache | 47 | 2.86, 0.288 | 46 | 4.16 | 389 | 22.8 |
oklm describe --temp 0 writes the same headline and
subheading on the three backends (Vulkan the CPU's 48 tokens but one,
OpenCL parting from them at the fourteenth). The tiny model's five cases
run three ways each on OpenCL and on each Vulkan device (RX 7900 XT, UHD
750), every block there and the last one alone, within 1e-6 of the numpy
pass, and no attention is handed back to the CPU. The server with an
image runs on a device since (server.md, Gemma 3 4B among
its files). Not run on a device: the 12B and 27B files.
LLaVA 1.5 has no whole-path comparison with llama.cpp, its encoder
being read differently there: the encoder is held to the float64 pass,
and the language side to llama.cpp's own model fed the embeddings OKLM
computed (tools/mtmd_ref.cpp --answer-rows).
oklm describe and oklm serve --mmproj write
the same 64 tokens about the newspaper ("The image is a vintage
newspaper clipping featuring a story about the Apollo 11 moon landing.
The headline reads "Men Walk on Moon." [...]"). Asked what the newspaper
says, llama.cpp with its own encoder answers "Men Walked on Moon" and
with OKLM's embeddings "Men Walk on Moon", the headline. The dashboard's
scripted check (tools/check_dashboard.sh) was run with
LLaVA 1.5 as its vision model: a pasted picture is sent and answered.
Not run: LLaVA 1.5 13B; a file made by a newer conversion (which names
its projector and may hold all 24 blocks: OKLM runs every block a file
holds).
LLaVA 1.6 has no reference in llama.cpp, which slices such a file its
own way, and transformers could not be run here (no PyTorch on the
machine). Its reference is the model's own processor and code written
again from transformers' sources, function by function, in
tools/llava_next_ref.py (numpy and Pillow), and what is
compared with what is this, in three parts that follow each other from
the picture to the rows:
- The tiles. From the picture alone, the reference chooses the resolution, resizes, pads and cuts; OKLM's pixels are its bytes normalized. Nothing of OKLM enters the reference.
- Each tile's embeddings. The float64 pass of the
encoder (
tools/vision_ref.py, the pass LLaVA 1.5 is held to) on the reference's pixels, one tile at a time; OKLM's embeddings of the ten tiles are within 7.6e-6 of it. - The rows. The reference places the embeddings of
the tiles as
pack_image_features()does; the embeddings it is given are the ones OKLM computed, so this part checks the placement alone (which feature goes where, what is cut, the ends of rows), and the test asks for equal values.
What these do not cover: transformers itself run on the same picture
(the port could carry a misreading of its sources), and the choices
where transformers' versions differ, which are listed in the script. The
same checks run in make test on a tiny file, with pictures
of five sizes that take every branch (one row of two or three tiles, a
square of four; a width cut, a height cut; an odd difference):
tests/mm/test_vision.c.
The language side is held to llama.cpp's own model fed the rows OKLM
computed (tools/mtmd_ref.cpp --answer-rows): the 48 greedy
tokens are the same. oklm describe --temp 0 and
oklm serve --mmproj (an image_url part and the
question, greedy) write the same 64 tokens about the newspaper ("The
newspaper headline reads: "MEN WALK ON MOON ASTRONAUTS LAND ON PLAIN;
COLLECT ROCKS, PLANT FLAG VOICE FROM MOON: 'EAGLE HAS LANDED' A POWDERY
SURFACE IS"): the headline and the subheadings under it, as printed.
With the overview alone (--tiles 336, 577 rows) the same
model reads the headline and gives the last subheading as "A POWERFUL
SURFACE". Asked to describe the picture drawn by the test, it names a
large blue circle, a red square, a yellow circle and a checkered
background. The server's second request for the same picture finds 2 352
of its 2 355 rows in the prefix cache (in that run the server put the
question right after the image; it writes a new line between them since,
as oklm describe and the model's own prompt do). Not run: a
LLaVA 1.6 with another language model (the Vicuna and 34B variants: the
same projector layout, and for Vicuna the turns LLaVA 1.5 gets); two
images in one prompt on the published file (the context must then hold
up to 5 856 rows); a picture that takes three tiles in a row on the
published file (the tiny file has the case).
The Qwen encoders are held to the float64 pass of the model's
definition (1e-4), and to llama.cpp loosely (5e-2), for what llama.cpp's
distance is made of. ggml gives a product with an F16 matrix its operand
rounded to F16; a float64 pass that rounds each operand so and uses the
tanh GELU (tools/vision_ref.py --as-llama-cpp) comes within
2.9e-3 and 5.2e-3 of llama.cpp's embeddings for Qwen2-VL, from 2.4e-2
and 2.0e-2. The distance is not spread evenly: the median embedding of
llama.cpp is 0.2 to 0.8 % from the exact one, and a few rows carry the
rest, the same rows in llama.cpp and in the rounded pass (one of the 273
moves by 36 to 79 % in three of the four cases). For the second picture
with Qwen2.5-VL the rounded pass is as far from llama.cpp (1.1e-2) as
the exact one, for that reason: both differ from the exact pass mostly
on one row, not by the same amount.
The language side was checked three ways. A text alone gives
llama.cpp's choices (all 48 with the 2B; 47 with the 7B, the other at a
tie of 0.23). With the same embeddings on both sides
(tools/mtmd_ref.cpp --answer with the embeddings OKLM
computed, which mtmd places at its three-part positions), the choices
part only where llama.cpp's two best logits are close: 0.09 for the 2B,
0.40 for the 7B, whose logits are further from llama.cpp's on a text
already (2.24: the two compute the quantized products differently, and
llama.cpp's norms lack the file's epsilon, Qwen2.5-VL 7B and
llama.cpp's RMS norm). As a control, the same rows given the
positions of a text, one after the other, move the 2B's logits by 9.8
and change a choice where llama.cpp's two best are 1.8 apart: the
positions are what the agreement rests on.
oklm describe --temp 0 and oklm serve --mmproj
(an image_url part and the question, greedy) write the same
64 tokens with both models ("The newspaper says, "Men Walk on Moon.
Astronauts Land on Plain; Collect Rocks, Plant Flag.""; the 7B: "The
newspaper in the image is from The New York Times, dated Monday, July
21, 1969. The headline reads: "MEN WALK ON MOON"
[...]"). The server's second request for the same picture finds 416 of
its 418 rows in the prefix cache, and a second turn of the conversation
(the 7B's) 480 of 497. A picture of 1 920 x 1 464 pixels (the test image
enlarged) is reduced to 1 240 embeddings and read the same by the 2B,
its prompt taking 66 s; no reference was run at that size. Not run:
several images in one prompt on the published files (the tiny model has
the case), Qwen2.5-VL 3B (its checks are in the test, skipped without
the files).
A video on the published files (2026-11-14, CPU, greedy): four frames a second apart of 448 x 336, the test image for two and its top left corner enlarged for two, are 2 pairs of 192 embeddings, a prompt of 423 tokens (8.7 s for the 2B, 10.8 s for the 7B). Asked what the video shows at its start and at its end, the 2B answers "The video starts with a close-up shot of a newspaper, specifically the New York Times, and then transitions to a black and white image of a man walking on the moon. The video ends with a shot of the moon."; asked to describe its two parts, the 7B reads the page ("The first part of the video shows a newspaper front page from The New York Times dated July 20, 1969. The headline reads "MEN WALK ON MOON," with a subheading that says "Astronauts Land on Plain; Collect Rocks, Plant Flag." [...]") and had not reached the second part after 110 tokens. No reference exists for a video on these files (llama.cpp has no video path for them): the embeddings, the positions and the layout are checked on the tiny models.
On OpenCL and on Vulkan (2026-11-14, RX 7900 XT) the same rows and
tokens give, against the CPU's logits and against llama.cpp's
(test_mm_public, the language model given OKLM's
embeddings; each layout on a model loaded for it):
| Model, layout | Choices the CPU's, of 48 | Logits from the CPU's: largest, mean | Choices llama.cpp's | Logits from llama.cpp's | Prompt, rows/s | Tokens/s |
|---|---|---|---|---|---|---|
| Qwen2-VL 2B, CPU | 46 | 1.76 | 483 | 28.1 | ||
| OpenCL, every layer, contiguous cache | 46 | 2.16, 0.121 | 46 | 2.59 | 6 131 | 202 |
| OpenCL, 14 of 28 layers, paged cache | 47 | 1.12, 0.094 | 47 | 1.82 | 1 019 | 50.6 |
| Vulkan, every layer, contiguous cache | 47 | 2.47, 0.141 | 45 | 2.64 | 6 183 | 303 |
| Vulkan, 14 of 28 layers, paged cache | 48 | 1.38, 0.114 | 46 | 1.84 | 1 008 | 55.0 |
| Qwen2.5-VL 7B, CPU | 46 | 3.16 | 111 | 6.5 | ||
| OpenCL, every layer, contiguous cache | 47 | 1.27, 0.162 | 46 | 3.00 | 1 910 | 91.2 |
| OpenCL, 14 of 28 layers, paged cache | 47 | 1.14, 0.150 | 46 | 3.17 | 236 | 12.7 |
| Vulkan, every layer, contiguous cache | 47 | 1.38, 0.180 | 47 | 2.95 | 1 754 | 115 |
| Vulkan, 14 of 28 layers, paged cache | 46 | 1.11, 0.160 | 48 | 3.18 | 233 | 13.1 |
The rates are of one run each, the prompt's the 418 rows through the
language model alone (Vulkan's first prompt includes its pipelines: 3
581 rows/s on another run).
oklm describe --temp 0 --backend opencl and
--backend vulkan write the CPU's sentence with the 2B; with
the 7B OpenCL writes the CPU's 48 tokens and Vulkan parts from them at
the tenth. The tiny model runs its seven cases seven ways each on OpenCL
and on each Vulkan device (RX 7900 XT, UHD 750), every block on the
device and the last one alone, within 6e-5 of the numpy pass (the paged
cache, its prefix cache, cuts and copies among them).
Layers split between the device and the CPU
(2026-11-14, RX 7900 XT, answer_devices() of
test_mm_public). An image's rows then get their three-part
positions on both sides: the CPU's kernel in the first layers, the
device's in the last. Beside the paged cache above, the same rows and
tokens with the contiguous cache, and with the CPU's layers run on the
device for the prompt, their weights streamed
(--gpu-stream,
oklm_context_params.stream_batch: the image's rows are then
rotated on the device in every layer and the tokens after them on the
CPU in the first 14), each on a model loaded for it:
| Model, layout (14 of 28 layers on the device) | Choices the CPU's, of 48 | Logits from the CPU's: largest, mean | Choices llama.cpp's | Logits from llama.cpp's | Prompt, rows/s | Tokens/s |
|---|---|---|---|---|---|---|
| Qwen2-VL 2B, OpenCL, contiguous cache | 47 | 1.12, 0.094 | 47 | 1.82 | 989 | 52.4 |
| OpenCL, the CPU's layers streamed for the prompt | 45 | 1.71, 0.113 | 47 | 1.78 | 2 256 | 52.4 |
| Vulkan, contiguous cache | 48 | 1.38, 0.114 | 46 | 1.84 | 987 | 54.7 |
| Vulkan, the CPU's layers streamed for the prompt | 46 | 2.73, 0.130 | 45 | 2.63 | 2 067 | 55.1 |
| Qwen2.5-VL 7B, OpenCL, contiguous cache | 47 | 1.14, 0.150 | 46 | 3.17 | 225 | 12.8 |
| OpenCL, the CPU's layers streamed for the prompt | 47 | 1.20, 0.157 | 47 | 3.01 | 534 | 12.8 |
| Vulkan, contiguous cache | 46 | 1.11, 0.160 | 48 | 3.18 | 223 | 13.0 |
| Vulkan, the CPU's layers streamed for the prompt | 47 | 1.26, 0.170 | 47 | 3.03 | 488 | 13.0 |
The split's logits are the paged cache's to the digits printed: the
cache's layout changes no sum. A video goes the same four ways
(test_qwen_video_devices: four frames of 448 x 336 cut from
the test image, each 32 pixels to the right of the one before and half a
second after it, two grids of 192 embeddings, a prompt of 411 rows,
asked what the newspaper says; the CPU's own 48 greedy tokens, no
reference existing for a video):
| Model | Layout | Choices the CPU's, of 48 | Logits from the CPU's: largest, mean | Prompt, rows/s | Tokens/s |
|---|---|---|---|---|---|
| Qwen2-VL 2B | CPU | 475 | 27.2 | ||
| OpenCL, every layer | 48 | 1.73, 0.123 | 6 672 | 199 | |
| OpenCL, 14 of 28, paged | 48 | 1.00, 0.100 | 1 010 | 50.9 | |
| OpenCL, 14 of 28, contiguous | 48 | 1.00, 0.100 | 993 | 52.4 | |
| OpenCL, 14 of 28, streamed for the prompt | 48 | 1.54, 0.115 | 2 296 | 52.7 | |
| Vulkan, every layer | 48 | 1.81, 0.129 | 6 763 | 304 | |
| Vulkan, 14 of 28, paged | 48 | 1.04, 0.106 | 914 | 46.3 | |
| Vulkan, 14 of 28, contiguous | 48 | 1.04, 0.106 | 925 | 55.5 | |
| Vulkan, 14 of 28, streamed for the prompt | 48 | 1.76, 0.124 | 1 917 | 55.0 | |
| Qwen2.5-VL 7B | CPU | 110 | 6.5 | ||
| OpenCL, every layer | 47 | 1.62, 0.163 | 2 157 | 92.2 | |
| OpenCL, 14 of 28, paged | 48 | 1.08, 0.153 | 235 | 12.7 | |
| OpenCL, 14 of 28, contiguous | 48 | 1.08, 0.153 | 221 | 12.8 | |
| OpenCL, 14 of 28, streamed for the prompt | 48 | 2.25, 0.176 | 552 | 12.8 | |
| Vulkan, every layer | 47 | 1.29, 0.159 | 2 011 | 115 | |
| Vulkan, 14 of 28, paged | 48 | 1.08, 0.142 | 235 | 12.6 | |
| Vulkan, 14 of 28, contiguous | 48 | 1.08, 0.142 | 225 | 12.9 | |
| Vulkan, 14 of 28, streamed for the prompt | 46 | 1.71, 0.162 | 560 | 13.1 |
Where a choice parts from the CPU's the two tokens were 0.04 to 0.35
apart on the CPU. Asked "What happens in this video?" instead, the 2B
parted from the CPU's answer at 2 to 5 of the 48 tokens (43 on Vulkan
with every layer on the device, under the test's 90 %), its logits as
far from the CPU's as in the table (1.9 at most, 0.14 on average): the
2B's wording moves at near ties, and the test asks the question whose
answer is the page's text. oklm describe --backend opencl
and --backend vulkan with
--gpu-layers 14 --gpu-stream 16 write the sentence the CPU
writes for the 2B (the encoder on the device, a prompt of 418 tokens in
0.53 s and 0.54 s).
On the Intel UHD 750 (2026-11-14;
OKLM_CL_DEVICE=UHD OKLM_VK_DEVICE=Intel with
test_mm_public in a directory holding these files alone).
The published vision files that fit it run there on both backends,
slower than the eight cores beside it: nothing is refused, nothing
fails, and the answers are the CPU's to near ties.
| File | Encoder: CPU, OpenCL, Vulkan | from the CPU's embeddings | Language model, every layer: choices the CPU's of 48, logits; prompt rows/s, tokens/s (CPU; OpenCL; Vulkan) | Half the layers, paged cache (OpenCL; Vulkan) |
|---|---|---|---|---|
| SmolVLM-256M Q8_0 (one tile) | 2.3 s, 10.7 s, 3.2 s | 2.6e-6, 2.3e-6 | 2 545, 157; 48, 1.85; 69, 28.7; 47, 1.25; 148, 42.9 | 48, 0.94; 130, 41.3; 47, 1.09; 397, 62.1 |
| SmolVLM-500M Q8_0 | 1 145, 68.8; 47, 1.80; 25, 13.1; 47, 1.58; 100, 24.1 | 48, 1.00; 48, 19.5; 48, 0.88; 192, 34.6 | ||
| Qwen2-VL 2B Q4_K_M (273 embeddings) | 6.9 s, 104 to 122 s, 25 to 32 s | 5.1e-5 to 5.5e-5, 4.9e-5 to 6.2e-5 | 353, 26.9; 46, 2.50; 4, 3.1; 46, 1.83; 35, 14.4 | 48, 1.22; 10, 5.1; 48, 1.15; 66, 18.2 |
Qwen2-VL 2B's answer on the UHD 750 gives 46 of llama.cpp's 48 tokens
on both backends, as the CPU does, and 46 of 48 of the choices its CPU
embeddings give when the encoder's blocks are there too (the two tokens
0.25 apart). With the contiguous cache and half the layers the logits
are the paged cache's; with the CPU's half streamed for the prompt the
choices are 46 to 48 of 48 for the three files and the prompt is slower
than unstreamed (6 against 12 rows/s for the 2B on OpenCL, 36 against 66
on Vulkan): the device is the slow side. The rates above were taken
while other runs used the machine; alone,
oklm describe --temp 0 on the test image (391 embeddings,
the encoder on the device) writes the same sentence three ways: the
prompt of 418 tokens in 9.78 s on the CPU, 40.4 s on the UHD 750 through
Vulkan and 196 s through OpenCL, the 23 tokens after it in 0.82 s, 1.50
s and 5.69 s. The time is mostly the encoder's (1 564 rows through 32
blocks), where Intel's OpenCL takes four times Vulkan's. Where OpenCL's
196 s go there (OKLM_CL_PROFILE=1, 2026-11-14, 195.5 s of
kernel time in 3 034 launches): the matrix products, 92.5 % (the
encoder's F16 products on gm_f16 115.0 s in 194 launches,
the language model's prompt on gm_q4_k 55.0 s and
gm_q6_k 10.7 s); the encoder's bidirectional tiles are 8.3
s (attention_bi80, 32 launches), the language model's
attention 0.75 s, and no op falls back to the CPU. It is not the tiles'
shape: with OKLM_CL_GEMM_TILES=0 the same products on the
row kernels take as long (mm_f16 127.9 s,
mm_q4_k 48.3 s, the prompt 195.6 s either way). The
encoder's products are about 1 970 GFLOP, so both kernel families run at
about 17 GFLOP a second on this device, whatever their work-groups;
nothing cheap changes that, and the CPU beside it is twenty times
faster. Not run there: Qwen2.5-VL (its 7B is 4.4 GiB of weights on
shared memory), Gemma 3, LLaVA, SmolVLM2-2.2B, a picture above 273 to
391 embeddings, several pictures, a video.
Pictures above a megapixel (2026-11-14;
test_qwen_large_pictures). llama.cpp's test image enlarged
(bicubic) to 1 624 x 1 238 pixels, 2.0 megapixels, is 2 552 embeddings
(58 x 44); enlarged to 2 600 x 1 982, 5.2 megapixels, it is reduced to 2
044 x 1 540 and 4 015 embeddings (73 x 55), the most its proportions
give under 4 096. llama.cpp gives both the same sizes and its prompts
the same rows (2 579 and 4 042). Its 16 greedy tokens fed here on the
CPU, from the same embeddings and from its whole path; then the encoder
on each device against the CPU's embeddings, and the language model with
every layer there:
| Model, embeddings | Choices llama.cpp's, of 16: same embeddings, whole path | Logits from llama.cpp's: largest (mean) | Encoder, CPU | OpenCL: time, from the CPU's | Vulkan | Language model on OpenCL: choices the CPU's, logits; prompt rows/s | on Vulkan |
|---|---|---|---|---|---|---|---|
| Qwen2-VL 2B, 2 552 | 16, 16 | 0.94 (0.11), 0.80 (0.11) | 215 s | 4.36 s, 1.3e-4 | 5.03 s, 7.7e-5 | 16, 1.13; 6 469 | 16, 0.97; 5 290 |
| Qwen2-VL 2B, 4 015 | 16, 16 | 0.93 (0.12), 1.03 (0.12) | 467 s | 9.87 s, 2.9e-4 | 11.6 s, 9.8e-5 | 16, 0.80; 5 366 | 16, 0.88; 3 952 |
| Qwen2.5-VL 7B, 2 552 | 15, 15 | 3.92 (0.49), 3.69 (0.42) | 90 s | 1.63 s, 1.1e-4 | 1.39 s, 7.8e-5 | 16, 1.44; 2 156 | 16, 1.29; 1 933 |
| Qwen2.5-VL 7B, 4 015 | 15, 15 | 3.60 (0.33), 3.67 (0.32) | 169 s | 2.92 s, 2.1e-4 | 2.63 s, 1.7e-4 | 15, 1.15; 1 858 | 15, 1.34; 1 582 |
The devices' columns were measured again on 2026-11-15 by the same test on the last code and are the table's: the encoder at 2 552 embeddings in 4.36 s (OpenCL) and 5.04 s (Vulkan) for Qwen2-VL 2B and 1.62 s and 1.39 s for Qwen2.5-VL 7B, at 4 015 in 9.82 s and 11.6 s and in 2.92 s and 2.63 s, 7.7e-5 to 2.9e-4 from the CPU's embeddings.
The CPU's encoder times of the table are of one pass each while other
work ran on the machine, before its kernels were looked into
(llama.cpp's whole path, its own encoder on the CPU with the prompt and
the 16 tokens, took 70 s and 159 s for the 2B and 76 s and 206 s for the
7B on the same day, busy as well). Where the time went, at 4 015
embeddings of Qwen2-VL (16 060 patch rows through 32 blocks, 389 s in
the ops): 290 s in the attention, every block's rows seeing all of them,
each tile of 32 rows gathering its head's 16 060 keys and values from
the tensors again, and 95 s in the products, whose F16 weights had a
one-row dot product. Since then (core.md): the attention of such rows runs
head by head on copies of the head's keys and values, the products of
many rows over F16 weights take eight rows a weight row (AVX-512; four
on AVX2) and are cut by blocks of activation rows. The sums are the same
in the same order, so the embeddings have the bits they
had (the files of the two sizes below compared byte for byte
with those of the kernels before), and no recording or bound moved. The
encoder alone on the picture of 2 600 x 1 982 pixels, eight threads of a
Core i7-11700K, OKLM then llama.cpp (llama-mtmd-cli, no GPU
layer, the projector on the CPU, --image-max-tokens), one
after the other on 2026-11-14 with other work on the machine at times
(load averages of 4 to 12):
| Encoder on the CPU | OKLM before | OKLM | llama.cpp |
|---|---|---|---|
Qwen2-VL, 1 008 embeddings (--tiles 900) |
32.4 s, 34.7 s | 16.5 s | 16.8 s (14.4 s and 15.6 s on other passes) |
| Qwen2-VL, 4 015 | 389 s in the ops | 164 s | 136 s (135 s on another pass) |
| Qwen2.5-VL 7B, 1 008 | not measured | 13.5 s | 12.4 s (11.3 s on another pass) |
| Qwen2.5-VL 7B, 4 015 | 169 s (the table's) | 66.3 s | 83.9 s |
One block of Qwen2-VL at 4 032 patch rows: the attention 0.41 s
before and 0.14 to 0.17 s now, the six products 0.53 s and 0.36 s. OKLM
is level with llama.cpp at a megapixel and ahead on Qwen2.5-VL's large
picture, whose 28 windowed blocks attend 64 rows at a time; it is a
fifth behind on Qwen2-VL's, where the attention over 16 060 rows is 114
of the 164 s: 370 GFlop/s on eight threads against 590 at 4 032 rows,
each tile of eight rows reading its head's 10 MiB of keys and values
from the last-level cache, and tiles of more rows no faster (their
scores then leave the second-level cache). Qwen2.5-VL at a megapixel is
9 % behind in its products (10.8 of the 13.5 s), the MLP's second one
having rows of 3 420 elements that do not stay in the first-level cache
eight at a time. The CPU's language model reads the 2B's 4 042 rows at
304 rows/s and the 7B's at 82. A device's embeddings part from the CPU's
as the picture grows, a row's attention summing over more keys: the
tests bound a large picture's at 6e-4 where a small one's are bounded at
2e-4, and the answers are the CPU's (one choice of the 7B's parts at 4
015 embeddings, its two tokens 0.23 apart).
oklm describe --temp 0 on the 2B writes the same sentence
at both sizes on both backends ("The newspaper says, "Men Walk on Moon.
Astronauts land on plain; collect rocks, plant flag.""): the prompt of 2
579 tokens in 4.89 s on OpenCL and 5.61 s on Vulkan, that of 4 042
tokens in 10.80 s and 12.78 s. Qwen2.5-VL's windowed blocks make its
encoder 2.7 to 4.4 times faster than Qwen2-VL's, whose 32 blocks attend
over every row. Through oklm serve with its default context
of 4 096 the picture of 5.2 megapixels is 3 024 embeddings (a prompt of
3 051 tokens), answered in the same words, and the request sent again
finds 3 040 of its rows in the prefix cache.
Several pictures in one prompt (2026-11-14;
test_qwen2vl_several_images,
test_qwen25vl_several_images). Three pictures: llama.cpp's
test image (391 embeddings), a part of it (the headline, 364 x 196
pixels: 91) and the picture the tests draw (273); two of them and the
three in one user turn, then two and three across two turns with an
answer between them. Each image takes its own three-part positions after
the one before: an image of w x h embeddings advances the position by
max(w, h), so the three take 23 + 13 + 21 positions for their 755 rows.
The prompt is the file's template's (a marker a picture, the question
after the last); llama.cpp tokenizes it to the same rows. Its 32 greedy
tokens are then fed here on the CPU, from the same embeddings (its
language model given OKLM's) and from its whole path (its own encoder);
then the same rows on OpenCL and on Vulkan, every layer on the
device:
| Model, prompt | Rows (of the pictures) | Choices llama.cpp's, of 32: same embeddings, whole path | Logits from llama.cpp's: largest (mean) | OpenCL: choices the CPU's, logits | Vulkan |
|---|---|---|---|---|---|
| Qwen2-VL 2B, two pictures, one turn | 518 (482) | 31, 32 | 0.75, 1.19 | 31, 0.77 | 32, 0.97 |
| three pictures, one turn | 797 (755) | 32, 32 | 0.98, 1.18 | 32, 0.90 | 32, 0.97 |
| two pictures, two turns | 719 (664) | 32, 32 | 2.62, 1.99 | 31, 3.32 | 31, 3.39 |
| three pictures, two turns | 817 (755) | 31, 31 | 3.04, 2.10 | 30, 3.04 | 29, 3.17 |
| Qwen2.5-VL 7B, two pictures, one turn | 518 (482) | 28, 29 | 6.06, 5.69 | 29, 1.31 | 30, 1.28 |
| three pictures, one turn | 797 (755) | 29, 29 | 12.8, 14.3 | 32, 2.68 | 32, 1.63 |
| two pictures, two turns | 719 (664) | 28, 29 | 4.57, 5.36 | 31, 1.48 | 31, 1.41 |
| three pictures, two turns | 817 (755) | 31, 29 | 7.45, 7.34 | 29, 1.60 | 30, 1.95 |
With half the layers on the device (paged, contiguous, the CPU's half
streamed for the prompt) the 2B gives 30 to 32 of the CPU's 32 choices
on either backend (logits within 2.53) and the 7B 29 to 32 (within
1.99). The 2B is llama.cpp's to a near tie. The 7B is further
from llama.cpp with several pictures than with one (46 of 48
choices and logits within 3.2 for one picture; 2.0 on a text alone):
with three pictures in one turn the logit of llama.cpp's first token is
31.0 there and 22.9 here, and the choices part where llama.cpp's two
best are up to 1.22 apart. The rows and the positions are the 2B's,
which agrees; the CPU and the two devices, three sets of kernels, agree
with each other within 2.7 on that prompt. It is not the
quantization: the same prompts and embeddings through the 7B's
BF16 language model (unsloth's, 14.2 GiB, fetched for this
and removed), whose products are float products in both programs, give
on the CPU
| Qwen2.5-VL 7B BF16, prompt | Rows | Choices llama.cpp's | Logits from llama.cpp's: largest, mean |
|---|---|---|---|
| a text alone | 38 | 47 of 48 | 1.71, 0.16 |
| one picture | 418 | 48 of 48 | 3.70, 0.44 |
| two pictures, one turn | 518 | 31 of 32 | 6.30, 0.49 |
| three pictures, one turn | 797 | 29 of 32 | 9.43, 1.26 |
| two pictures, two turns | 719 | 30 of 32 | 5.22, 0.36 |
| three pictures, two turns | 817 | 31 of 32 | 8.49, 0.28 |
(llama.cpp given OKLM's embeddings; recordings made for the
comparison and not kept). The two programs are 0.16 apart on average on
a text of 38 tokens of this model, where Qwen2-VL 2B's Q4_K_M is 0.11
from llama.cpp's on the same text and on a picture of 4 015 embeddings;
the distance grows with the image rows, and the choices still part at
near ties only (0.53 at most on the float model). OKLM is the
one nearer the model, and the distance is llama.cpp's: the
pinned llama.cpp evaluates this architecture's RMS norms with an epsilon
of 0 where the file says 1e-6, which the 7B's rows of
<|im_start|> and <|im_end|> need
and the 2B's do not; a float64 pass of the model's definition
(tools/qwen2vl_ref.py) is within 0.15 of OKLM's logits on
the BF16 file where llama.cpp is 1.13 to 6.42 from it, and the same pass
with an epsilon of 0 is llama.cpp's within 0.23 (Qwen2.5-VL 7B and
llama.cpp's RMS norm). The tests keep llama.cpp as the reference and
bound the several-picture cases at 85 % of the choices and near ties of
1.5.
The language models of LLaVA 1.5, LLaVA 1.6 and the SmolVLM files
needed nothing to run on a device with an image's embeddings: their rows
are evaluated as tokens' are. test_mm_public runs them on
OpenCL and on Vulkan (2026-11-14, RX 7900 XT) on the same rows and
tokens as the CPU: llama.cpp's 48 greedy tokens for the LLaVA files, the
CPU's own 48 for the SmolVLM files, which have no recorded llama.cpp
answer; every layer on the device with the contiguous cache, and half of
them with a paged cache (each on a model loaded for it; the two other
layouts of half the layers, the contiguous cache and the CPU's layers
streamed for the prompt, are below the table):
| Model | Rows of prompt | Layout | Choices the CPU's, of 48 | Logits from the CPU's: largest, mean | Choices llama.cpp's | Prompt, rows/s | Tokens/s |
|---|---|---|---|---|---|---|---|
| LLaVA 1.5 7B Q4_K | 595 | CPU | 47 | 105 | 6.4 | ||
| OpenCL, every layer | 47 | 0.44, 0.053 | 48 | 2 157 | 101 | ||
| OpenCL, 16 of 32, paged | 48 | 0.35, 0.044 | 47 | 203 | 10.9 | ||
| Vulkan, every layer | 47 | 0.37, 0.049 | 48 | 2 011 | 127 | ||
| Vulkan, 16 of 32, paged | 48 | 0.29, 0.040 | 47 | 213 | 11.4 | ||
| LLaVA 1.6 Mistral 7B Q4_K_M | 2 357 | CPU | 48 | 89 | 6.1 | ||
| OpenCL, every layer | 48 | 0.39, 0.045 | 48 | 2 023 | 82.9 | ||
| OpenCL, 16 of 32, paged | 48 | 0.33, 0.042 | 48 | 166 | 10.8 | ||
| Vulkan, every layer | 48 | 0.43, 0.044 | 48 | 1 802 | 113 | ||
| Vulkan, 16 of 32, paged | 48 | 0.43, 0.040 | 48 | 173 | 11.2 | ||
| SmolVLM-256M Q8_0 | 348 | CPU | 2 561 | 170 | |||
| OpenCL, every layer | 47 | 1.89, 0.171 | 6 038 | 366 | |||
| OpenCL, 15 of 30, paged | 48 | 0.94, 0.106 | 3 040 | 191 | |||
| Vulkan, every layer | 48 | 1.84, 0.173 | 4 809 | 675 | |||
| Vulkan, 15 of 30, paged | 48 | 0.94, 0.106 | 2 825 | 241 | |||
| SmolVLM-500M Q8_0 | 348 | CPU | 1 143 | 68.8 | |||
| OpenCL, every layer | 48 | 1.74, 0.157 | 4 939 | 343 | |||
| OpenCL, 16 of 32, paged | 48 | 0.99, 0.102 | 1 616 | 105 | |||
| Vulkan, every layer | 47 | 1.77, 0.158 | 4 710 | 540 | |||
| Vulkan, 16 of 32, paged | 48 | 1.00, 0.102 | 1 585 | 120 | |||
| SmolVLM2-2.2B Q8_0 | 433 | CPU | 294 | 14.9 | |||
| OpenCL, every layer | 47 | 1.22, 0.062 | 3 457 | 219 | |||
| OpenCL, 12 of 24, paged | 47 | 0.63, 0.049 | 557 | 28.2 | |||
| Vulkan, every layer | 47 | 1.17, 0.061 | 4 003 | 282 | |||
| Vulkan, 12 of 24, paged | 47 | 0.60, 0.049 | 566 | 29.1 |
The rates are of one run each, the prompt's the rows through the
language model alone. With half the layers on the device and the
contiguous cache the logits are the paged cache's to the digits printed.
With the CPU's half run on the device for the prompt, its weights
streamed (--gpu-stream; 2026-11-14, one run), the choices
the CPU's are 48 and 47 of 48 for LLaVA 1.5 (OpenCL, Vulkan; logits
within 0.38), 48 and 48 for LLaVA 1.6 (0.39), 47 and 48 for SmolVLM-256M
(1.29), 48 and 48 for SmolVLM-500M (1.65), 47 and 47 for SmolVLM2-2.2B
(1.19), 46 and 45 for Gemma 3 4B, whose image rows see each other (3.16;
46 and 47 without the streaming). The prompt is faster streamed where
the model is large (LLaVA 1.6's 2 357 rows at 889 and 785 rows/s against
167 and 172; LLaVA 1.5's at 531 and 521 against 173 and 202; Gemma 3
4B's at 718 and 815 against 393 and 389) and slower for the small files
(SmolVLM-500M's at 1 532 and 1 150 rows/s against 1 732 and 1 673). The
server with an image runs on a device since (server.md, SmolVLM-500M among
its files). SmolVLM-256M and SmolVLM-500M on the Intel UHD 750 are with
Qwen2-VL 2B's figures there (above: 47 or 48
of the CPU's 48 choices, at a sixth to a half of the CPU's tokens a
second).
On the CPU, Qwen2-VL 2B takes 12 s for a prompt with llama.cpp's test
image (1 564 patches through 32 blocks, 418 rows of prompt) and
Qwen2.5-VL 7B 13 s for the prompt and 13 s for 64 tokens, on a machine
shared with other work. With the language model on a device the prompt
is the encoder's time, which runs on the CPU for these two models (the encoder on a device): oklm describe
with the 2B, 8.5 s on OpenCL and 8.4 s on Vulkan against 9.7 s on the
CPU (the 418 rows through the language model take 0.07 s there), then 23
tokens in 0.12 s and 0.08 s against 0.82 s; with the 7B, 7.3 s for the
prompt on either device against 10.9 s, and 48 tokens in 0.52 s and 0.41
s against 7.4 s. On the CPU, Gemma 3 4B takes 86 s for a prompt with one
image (4 096 patches through 27 blocks) and LLaVA 1.5 7B 19 s (577 rows
through 23 blocks, then 595 rows of prompt), on a machine shared with
other work. LLaVA 1.6 Mistral 7B takes 105 s for a prompt with
llama.cpp's test image (five tiles through the encoder, then 2 357 rows
of prompt) and 32 s for 96 tokens, with 5 threads on the same shared
machine (241 s with 4 threads at a busier time); 26 s for the prompt
with the overview alone (594 rows); through the server, 243 s for the
first request and 47 s for the second, its image found in the cache. On
the CPU, SmolVLM-256M reads the headline of llama.cpp's test image ("MEN
WALK ON MOON", 320 image embeddings, prompt in 6.7 s); Whisper base
transcribes jfk.wav (11 s) in 0.9 s; Ultravox with Llama-3.2-1B quotes
the speech. The large Whisper models are slow on the CPU: measured on a
machine shared with other work, a 30-second chunk took 34 to 54 s with
large-v3-turbo and 80 to 135 s with large-v3 (base: 1 to 2 s, small: 9
to 18 s); on a quiet machine 6 and 12 s, and 0.3 and 0.7 s with
--backend vulkan (Whisper on
a device).