OKLMdocumentation v1.0.3
On this page

Multimodal

Images, sounds and speech given to a language model: decoding, the vision and audio encoders whose embeddings a model reads in place of tokens, speech recognition with Whisper, and how each is checked against llama.cpp. Video containers follow adr/0007-video-containers.md.

Status: implemented: images (SigLIP encoders with the idefics3 projector: SmolVLM 256M, 500M, SmolVLM2 2.2B; with the gemma3 projector: Gemma 3 4B; the CLIP encoder with the mlp projector: LLaVA 1.5 7B, and LLaVA 1.6 Mistral 7B with its grid of tiles; the encoders of Qwen2-VL and Qwen2.5-VL at the image's own resolution: Qwen2-VL 2B, Qwen2.5-VL 7B), sounds (Ultravox), both through the server too, speech recognition (Whisper), documents (text, JSON, HTML, DOCX, PDF), videos (Ogg Theora natively, other containers through ffmpeg when allowed), oklm describe, oklm transcribe, oklm ingest, on Linux and Windows.

Using it

oklm describe -m models/smolvlm-256m/SmolVLM-256M-Instruct-Q8_0.gguf \
    --mmproj models/smolvlm-256m/mmproj-SmolVLM-256M-Instruct-f16.gguf \
    -p "What does the newspaper say?" photo.jpg
oklm describe -m models/gemma3-4b/gemma-3-4b-it-Q4_K_M.gguf \
    --mmproj models/gemma3-4b/mmproj-model-f16.gguf \
    -p "What does the newspaper say?" photo.jpg
oklm describe -m models/llava-1.5-7b/ggml-model-q4_k.gguf \
    --mmproj models/llava-1.5-7b/mmproj-model-f16.gguf \
    -p "Describe the picture in detail." photo.jpg
oklm describe -m models/llava-1.6-mistral-7b/llava-v1.6-mistral-7b.Q4_K_M.gguf \
    --mmproj models/llava-1.6-mistral-7b/mmproj-model-f16.gguf \
    -p "What does the newspaper say?" photo.jpg
oklm describe -m models/qwen2.5-vl-7b/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
    --mmproj models/qwen2.5-vl-7b/mmproj-F16.gguf \
    -p "What does the newspaper say?" photo.jpg
oklm describe -m models/ultravox-1b/Llama-3.2-1B-Instruct-Q8_0.gguf \
    --mmproj models/ultravox-1b/mmproj-ultravox-v0_5-llama-3_2-1b-f16.gguf \
    -p "What is said?" speech.wav
oklm transcribe -m models/whisper-base speech.wav
oklm transcribe -m models/whisper-small --format srt talk.ogg > talk.srt
oklm transcribe -m models/whisper-small --language fr --translate fable.ogg

oklm describe takes images, sounds and videos (by extension: .wav, .mp3, .flac, .ogg, .opus are sounds; .ogv, .mp4, .mkv, .webm, .mov, .avi, ... videos), puts each where the prompt has the marker <__media__> (or before the prompt; a sound for Ultravox after it, see the table below), and generates -n tokens (256). Without -p the question is Describe this image., Describe this video. when a video is among a vision model's files, and Describe this sound. for an audio model (it was the image's for all three until 2026-11-14). --tiles splits large images into tiles plus an overview, as SmolVLM was trained; without it the overview alone is encoded. Gemma 3 reads one image of 896 pixels and LLaVA 1.5 one of 336, whatever the option says. LLaVA 1.6 reads an overview of 336 pixels and the tiles of the image fitted in the resolution of its file's list that suits it best (two to four tiles: 577 rows for the overview, then up to 2 352 more), whatever SIDE says, except that a SIDE up to 336 gives the overview alone (577 rows: a prompt four times shorter, without the detail the model was trained to read). Qwen2-VL and Qwen2.5-VL read one image at its own size, as many embeddings as it has squares of 28 pixels, up to 4 096 (3.2 megapixels, llama.cpp's limit; a larger picture is reduced to that); --tiles SIDE caps the largest area at SIDE x SIDE pixels instead (--tiles 1002 for 1 280 embeddings, the default before 2026-11-14: the encoder's attention grows with the square of their number, 10 s for a picture of 5 megapixels on an RX 7900 XT and one to three minutes on the CPU). Without -c, oklm describe makes the context as long as the pictures' rows need. oklm transcribe writes what a sound file says (speech recognition below):

Option
--language CODE the language spoken (en, fr, de, ...: the codes of the model's language tokens; an unknown one prints the list). auto, the default: the model finds it from the first 30 seconds, and the line [language: fr (0.99)] on the error stream says which, with its probability
--translate write English whatever the language spoken (Whisper's translation task; not large-v3-turbo's strength: it was trained without it)
--format text the words (the default)
--format timed, --timestamps a line a segment: [00:00:06.000 --> 00:00:11.360] text
--format srt, --format vtt subtitles (SubRip, WebVTT)
--format json task, language, language_probability (when found), duration, text and segments (id, seek, start, end, text, avg_logprob, no_speech_prob): the route's verbose_json
--backend cpu|opencl|vulkan|auto where the model computes (cpu by default): on a device the encoder's blocks and the decoder run there (on a device)
--no-timestamps decode without timestamp tokens: a cut every 30 seconds wherever that falls, one segment a cut
--detect-language print the language and its probability (fr 0.997) and transcribe nothing: one pass of the encoder, one decoder step

The other way, a text spoken into a sound with oklm speak, is in speech.md.

oklm serve --mmproj FILE takes the same images and sounds in chat messages, as base64 image_url and input_audio parts, their rows cached by the media's content hash (server.md). A turn's text around its media is written by one function for both, oklm_mm_content(): each image or sound a marker, joined to the parts around it as the projector's model writes them, so oklm describe -p QUESTION picture and a chat request holding the picture then the question give the model the same tokens (tests/server/test_server.c, on the tiny projector of each family: as many rows, the same greedy answer, and describe's prompt sent as a raw completion finds every row but the last in the prefix cache). The retrieval's captions (rag.md) are written by it too. What follows a marker that another part follows, by projector:

Projector After a marker The reference
idefics3 (SmolVLM) nothing the template in the model's file, given content in parts: User:<image>What is this?
gemma3 nothing the template in the model's file, given content in parts: <start_of_image>What is this?
qwen2vl_merger, qwen2.5vl_merger nothing the template in the model's file: <|vision_start|><|image_pad|><|vision_end|>What is this?
mlp (LLaVA 1.5, LLaVA 1.6) a new line the model's own conversations: <image>\n before the question. The files' templates take a string, not parts (LLaVA 1.5's files have none) and say nothing of it
ultravox a new line; and a space before a sound that comes right after a text the model's own pipeline (ultravox_pipeline.py): the sound at the end of the user's turn, a space after the text, Transcribe this. <|audio|>. The language model's template takes a string and says nothing of a sound; a text after a sound, which the pipeline never writes, keeps the new line
any other a new line none: LLaVA's form

oklm describe -p QUESTION puts the pictures before the question, and for Ultravox the question before the sound, the pipeline's order. The order decides the answer: asked "Transcribe this." about jfk.wav, Ultravox v0.5 with Llama-3.2-1B (Q8_0, CPU, greedy) writes the sentence when the sound follows the question ("Here's the transcribed text: "And so, my fellow Americans ask not what your country can do for you, ask what you can do for your country.""), and quotes "My Country, 'Tis of Thee" when the question follows the sound, which is what oklm describe wrote until 2026-11-14. A chat request keeps the order of its parts: the question first, then the input_audio part, is describe's prompt (228 tokens by either way, the same answer; test_sound_describe in tests/server/test_server.c on the tiny encoder: the same rows).

Otherwise nothing is put before a marker, and nothing between two texts. llama.cpp at the pinned revision joins a marker and a text with nothing for every projector (common/chat.cpp; mtmd-cli writes the markers right before the prompt): for SmolVLM, Gemma 3 and Qwen it writes what the templates write, for LLaVA it leaves out the model's new line. Gemma 3 has a second reference, which is not followed: the Transformers processor replaces the image's place in the rendered template by two new lines, the image's tokens and two more new lines, an empty line on each side of the image that neither the template nor llama.cpp writes. OKLM writes the template's form, and llama.cpp's recorded answer for Gemma 3 4B is made from it (tools/gen_mm_ref.sh). oklm_mm_build() gives a prompt's rows without evaluating them (tokens, and each media row's content hash and index, with the embeddings; an item may be named by the hash an earlier build gave it, oklm_mm_item.hash, and is then built from the embeddings kept of it with no pixel or sample read, which is how the server skips decoding a part it has seen), oklm_mm_eval_rows() evaluates a prompt and names its rows the same way, without keeping the embeddings, and oklm_image_decode() / oklm_audio_decode() read files held in memory, oklm_video_decode() a video (written to a private temporary file under TMPDIR for the time of the call: the decoders read a path).

A conversation kept in a file

oklm describe -m MODEL --mmproj MMPROJ -p "Describe this image." --kv-save moon.kv moon.jpg
oklm describe -m MODEL --mmproj MMPROJ --kv-load moon.kv -p "What colours are in it?"
oklm describe -m MODEL --mmproj MMPROJ -p "Describe this image." --then "What colours are in it?" moon.jpg
oklm describe -m MODEL --mmproj MMPROJ --kv-load moon.kv -p "Is this a part of it?" crater.jpg
oklm describe -m MODEL --mmproj MMPROJ -p "Describe this image." --then "Is this a part of it?" --then-with crater.jpg moon.jpg

--kv-save FILE writes the conversation's cache after the answer: the rows of the prompt, the image's among them, and of the answer, as a KV file (oklm_kv_save_media(): each row with its token or its media hash, and for Qwen2-VL and Qwen2.5-VL the positions of its rows). --kv-load FILE -p QUESTION, in a later run, takes the rows back (oklm_kv_load_media()) and asks the question as the conversation's next turn: no picture is given and none is encoded, the model evaluates the question's tokens only. --then QUESTION asks a second question after the first answer in the same run. The three combine (--kv-load a.kv -p ... --kv-save a.kv goes on turn by turn). A next turn may bring media of its own (2026-11-14; it was text only): the files named with --kv-load go with its question, and --then-with FILE (repeatable) gives one to the question of --then; a picture, a sound or a video, laid out as in a first turn (the items beside the question, in the model's order; a video's frames announced for an encoder of still images). Their rows take the positions that follow the conversation's: for Qwen2-VL and Qwen2.5-VL the three-part positions from the sequence's counter, which the file restores. Without -c the context is sized for the pictures of both questions of one run; after --kv-load it is the default (4 096) or -c. The next turn's text is what the model's chat template writes after an answer for a new user message; a template that writes a longer conversation another way is refused with a message. SmolVLM's own format, which ends its prompt at Assistant: and writes a kept answer after a space, was refused so until 2026-11-14, with --then and with --kv-load alike: the space is the one the model's answer begins with, and the conversation goes on (SmolVLM-256M: a second question, with and without a picture of its own, answered from a file as in one run). Each answer is sampled from a sampler of its own, so a second answer does not depend on whether the first was generated in the same run.

Run on Qwen2-VL 2B (Q4_K_M, the CPU, --temp 0 -n 64, the picture of test_mm_public's answer case, a newspaper's front page): the first answer, saved (480 rows, 391 of them the image's, 13.8 MB), then the second question in a new process from the file, give the 64 and the 8 tokens' text of the one run with --then, byte for byte; the new process evaluated 16 tokens where the prompt had 416 rows. The file is told which rows are the media's by the evaluation itself (oklm_mm_eval_rows(): the prompt evaluated, each row named by its token or its item's hash and index as oklm_mm_build() names it), so the encoders run once: the same picture with -n 16 and --kv-save took 19.5 s before that, the encoders run a second time for the names, and takes 10.4 s, the file the same bytes (434 rows, 12.5 MB).

A next turn with a picture, the same model and settings (the front page at 448 x 336, What is the headline?, then its top left corner enlarged, Is this second picture a part of the first one? Say what it shows.): saved after the first answer (227 rows, 6.5 MB), the second question and its picture in a new process (192 embeddings, 220 rows evaluated after the 227 loaded) give the text of the one run with --then ... --then-with, byte for byte ("The headline is "Men Walk on Moon."", "Yes, this is a part of the first image."). With the clip as the next turn's media (What happens in this video?, 384 embeddings in 402 rows after the 225 loaded): the same text either way again ("In this video, we see a newspaper with the headline "Men Walk on Moon." [...]"). On the tiny qwen2vl model (test_kv_file's test_next_turn_media): a turn with a picture saved, then a text, a second picture or a video and a text after the load, contiguous and paged either way, give the logits and the position counter of the context that never left memory, bit for bit.

In the terminal chat

oklm chat -m MODEL --mmproj MMPROJ     # then: /attach moon.jpg, and a message
oklm chat -m smolvlm-256m              # a catalogue model that carries a projector: no option

oklm chat takes the same projector (--mmproj FILE|NAME|none, --tiles, --fps, --frames, --ffmpeg) since 2026-11-15, and its /attach FILE a picture, a sound or a video that goes with the next message, read by the code that reads a question's file here (a video is its sound, one item, or announced frames, by the encoder). The chat's turn is the conversation rendered whole and made rows by oklm_mm_build(); the rows are compared with the context's, a media row by its item's hash and its index within it, and what follows the common prefix is evaluated: tokens through oklm_eval(), an item's rows through oklm_eval_embd(), oklm_eval_embd_mutual(), oklm_eval_embd_grid() or oklm_eval_embd_video() as its model reads them. A media item is never cut by the prefix: it is in the cache whole, or evaluated from its first row. Each item is encoded once: its embeddings are kept by its hash for the later turns' prompts, and the pixels or samples are freed. --kv-save writes the rows as --kv-save does here (oklm_kv_save_media(), appended turn by turn) and --resume takes them back (oklm_kv_load_media()), the conversation's file naming the media files to encode again. The commands, what they do with media in the conversation and the measured runs (SmolVLM-256M, Qwen2-VL 2B, Gemma 3 4B, Ultravox) are in inference.md.

Video

A video becomes frames for the vision encoder or its sound for the audio encoder, whichever the projector file has (adr/0007-video-containers.md). For a vision encoder each frame is announced by its time (Frame at 0:03:), the question after them. For an audio encoder (Ultravox) the video is its sound track and nothing else: one sound, placed where the model's own order puts a sound, after the question and a space (the table above), so oklm describe -p "Transcribe this." talk.ogv is the prompt of the same command on the sound alone; a video without sound is refused there. Until 2026-11-14 the sound had lines of its own before the question (Video talk.ogv (11.0 s):, Sound: ), which is not the order of Ultravox's own prompt: jfk.wav as the sound track of an Ogg video, asked "Transcribe this." (Ultravox v0.5 1B, the CPU, greedy), was 243 prompt tokens and an answer that spoke of the quote before giving a version of it ("The given text appears to be a quote from John F. Kennedy's inaugural address [...]"); it is 228 tokens, the count of the sound file alone, and the sentence ("[...] "And so, my fellow Americans ask not what your country can do for you, ask what you can do for your country." This is a famous quote from John F"). No model here is given a video's frames and its sound together: a projector file holds one encoder, and Qwen2-VL and Qwen2.5-VL have no audio one. Qwen2-VL and Qwen2.5-VL read a video as their own processors and code define one, the frames two by two and no text between them (below):

oklm describe -m models/smolvlm-256m/SmolVLM-256M-Instruct-Q8_0.gguf \
    --mmproj models/smolvlm-256m/mmproj-SmolVLM-256M-Instruct-f16.gguf \
    --frames 4 -p "What happens in this video?" clip.ogv
oklm describe ... --ffmpeg -p "What does the speaker say?" talk.mp4
  • Ogg Theora with Vorbis sound (.ogv) is decoded by OKLM (libtheora, libvorbis and libogg, the libraries of Allegro's video addon), as fast as they decode, without an audio device.
  • Any other container goes through ffmpeg, only with --ffmpeg or OKLM_FFMPEG=1 (or OKLM_FFMPEG=/path/to/ffmpeg): frames as PPM images on a pipe, the sound as 16 kHz PCM, an argument vector and no shell. Without it, such a file is refused with both ways out named.
  • --fps F (1) frames a second at most, --frames N (8, at most 32) frames kept: past N, the interval doubles and every other frame goes, so memory stays bounded; the frames kept are the first, the last and evenly spaced ones between.
  • A reader may bound what it decodes (oklm_video_params.max_pixels): the frames asked for times a frame's width and height, checked when the stream's headers give the size (the first picture's, through ffmpeg), before a frame is decoded; above it the video is refused (OKLM_ERR_SHAPE) with the figures ("8 frames of 3840 x 2160 are 66.36 megapixels, above the limit of 16.00: fewer frames or a smaller video"). oklm describe sets none; the server does (server.md).
  • The lines that give the frames to an encoder of still images (Video clip.ogv (4.0 s):, then Frame at 0:03: and the frame's marker, a line each) are written by one function, oklm_mm_video_text(), for oklm describe and for the server's chat routes, which have no file name and write Video (4.0 s):.

SmolVLM-256M reads the headline of a still Theora video of llama.cpp's test image from two frames; Ultravox quotes the speech of an MP4 read through ffmpeg.

The server takes a video the same way, as a video_url part of a chat request (/v1/chat/completions; a video block in /v1/messages), base64 in a data: URL: the same decoder, the same sampling by default with fps and frames on the part, the same items and the same text for each kind of projector, so a request holding a video and a question is the prompt oklm describe writes for them, and, for Qwen2-VL and Qwen2.5-VL, the same rows. The decoded frames are bounded there (server.md).

Documents

oklm_ingest_file() turns a document into the text a model reads, by its extension, then its content:

Document Text
Plain text, Markdown, CSV, TSV, code as it is, without a byte order mark or carriage returns
JSON checked, indented two spaces a level
HTML rendered by nilorea's n_html: scripts and styles dropped, entities decoded, blocks on their lines
DOCX the paragraphs of word/document.xml (read by nilorea's n_zip): headings as # lines, list items as - lines, tables as Markdown rows, text boxes once
PDF the text of each page in order, lines and spaces from where the glyphs land, pages apart by a blank line (below)
oklm ingest report.docx
oklm run -m model.gguf --doc report.docx -p "Summarize this report."

--doc puts the text before the prompt, between <document> tags. The agent's Read tool shows a Word document's text the same way, and a PDF's pages with its pages argument (Edit and Write refuse both). A PDF's text is read (below); scanned pages, images without text, are not.

Decoding

Input Without Allegro 5 With Allegro 5
Images PNM (P5, P6) PNG, JPEG, BMP, TGA, PCX (allegro_image)
Sounds PCM WAV (8, 16, 24, 32-bit integer, 32-bit float) FLAC, Ogg Vorbis, Opus, MP3 (allegro_acodec)

Allegro decodes into memory bitmaps and samples, without a display. The Windows build gets it from the Allegro project's MinGW-w64 release: make windows-deps downloads it and its dependency pack into external/mingw (checksums checked), after which make MINGW=1 builds with the POSIX-threads MinGW-w64 compiler and Allegro (oklm-gui.exe too), and make windows-dist gathers the executables and every DLL they need in build/mingw/dist/oklm. Every multimodal test passes under Wine. Sounds are mixed to mono and resampled to 16 kHz with a Kaiser-windowed sinc (32 zero crossings, unit gain at DC). An output sample sits at one of a few places between two input samples (160 from 44.1 kHz, one from 48 kHz), and the filter's weights depend on that place alone: they are computed once per place, not once per sample (reading 52 seconds at 44.1 kHz: 17 s before, about one now; the same values within 1e-6). Images are resized with a separable Lanczos-3 filter (bilinear and bicubic also), widened when shrinking.

Vision

oklm_vision loads a llama.cpp mmproj GGUF with a vision encoder:

  • SigLIP: patches embedded as one matrix product, learned positions, pre-norm blocks with GELU (tanh), a final norm; a tile side that is not a multiple of the patch loses its last pixels, as the unpadded convolution does (SmolVLM2-2.2B: 384 pixels, 27 patches of 14);
  • the idefics3 projector: a pixel shuffle (each output token concatenates a scale x scale group of patches) and one linear layer;
  • the gemma3 projector: the mean of each 4 x 4 group of patches, an RMS norm with a weight (mm.soft_emb_norm), one linear layer (mm.input_projection, stored with its sides exchanged);
  • CLIP (LLaVA 1.5): a class token before the patches, patches without a bias, a norm before the blocks, quick-GELU (x sigmoid(1.702 x)), no norm after the blocks;
  • the mlp projector: the patches without the class token through two linear layers (mm.0, mm.2) with the exact GELU between them; in a file that lists resolutions (clip.vision.image_grid_pinpoints, LLaVA 1.6), the same for an overview and for each tile, the embeddings then joined in one grid (oklm_vision_pack(), below);
  • the encoders of Qwen2-VL and Qwen2.5-VL (qwen2vl_merger, qwen2.5vl_merger): an image of any size in units of 28 pixels, two patch kernels summed, two-dimensional rotary positions in the attention, for 2.5 RMS norms, a gated MLP and attention in windows; a merger joining each 2 x 2 group of patches (below).

Any other clip.projector_type is refused when the file is loaded, and so is an mlp file with chosen feature layers (clip.vision.feature_layer) or one whose tiles are joined otherwise than LLaVA 1.6's (clip.vision.mm_patch_merge_type other than spatial_unpad).

The tiling is SmolVLM's: the image is resized so its longest side is the largest the file allows, rounded up to whole tiles, cut into tiles, and an overview of the whole image is added. The prompt is idefics3's: each tile between <fake_token_around_image><row_R_col_C> markers, a new line per row, then <global-img> and the overview. SmolVLM's own chat template reads only structured message content, so oklm_mm_format() writes its turns (User: ... <end_of_utterance> Assistant:) itself.

The embeddings go into the language model through oklm_eval_embd(): their rows replace the token lookup, after which the model runs as usual.

The encoder on a device. An encoder computes on the CPU unless oklm_vision_use_backend() gives it a device: its weights are then copied there (one buffer, with another for a tile's activations), and oklm_vision_encode() computes each tile on it through the same ops (norms, products, bidirectional attention, the activation): the tile's patches are uploaded and the projector's rows read back. The patch embedding before the blocks and the projector after them compute there with the blocks since 2026-11-14 (below; the figures of this section are from before, the blocks alone on the device). Two things differ from the CPU's pass: CLIP's activation x sigmoid(1.702 x) is three element-wise ops there, and the MLP's inside width is rounded up to a multiple of 32 with zeros (SigLIP's 4 304 becomes 4 320: a device's matrix products take rows of a multiple of 32, and handed that product back otherwise). The encoders of Qwen2-VL and Qwen2.5-VL, which read an image at its own size, go through the same blocks (below): their rows are turned by their patch's place through the rotary op, and a block with windowed attention is one attention op, each row given its window's rows as its span of keys. oklm describe does it when the language model has layers on a device, for every encoder (SmolVLM, Gemma 3, LLaVA 1.5 and 1.6, Qwen2-VL, Qwen2.5-VL); the retrieval's encoder computes on the CPU. oklm serve does it too when its model has layers on a device: a request's encoder runs in its own thread, each tile or image as one pass on the device (oklm_backend_pass_begin()), which lets the scheduler's steps in between two of its blocks (oklm_backend_pass_yield()): a request that is decoding waits for a block, 0.04 to 0.08 s with its own step where it waited 0.25 to 0.70 s for the pass on Vulkan (server.md).

Measured on 2026-11-14 (RX 7900 XT, llama.cpp's test image, the language model's layers on the device in each case): the prompt of oklm describe, encoder included, and one tile through the encoder alone (test_mm_public, a second tile, the kernels built):

Model Prompt, encoder on the CPU: OpenCL, Vulkan Prompt, blocks on the device: OpenCL, Vulkan One tile: CPU OpenCL Vulkan
LLaVA 1.5 7B (577 rows, 23 blocks) 1.40 s, 1.50 s 0.40 s, 0.41 to 0.42 s 1.16 s 0.13 s 0.11 s
LLaVA 1.6 Mistral 7B (five tiles) 7.83 s, not run 1.85 s, 1.89 s
Gemma 3 4B (4 096 rows, 27 blocks) 21.0 s, 21.1 s 0.89 to 0.90 s, 0.84 s 22.0 s 0.77 s 0.73 s
SmolVLM2-2.2B (five tiles) 10.6 s, not run 0.72 s, 0.60 s
SmolVLM-256M not run 0.33 s, 0.29 to 0.31 s

On the CPU alone the same prompts take 7.0 s (LLaVA 1.5), 23.3 s (Gemma 3), 12.0 s (SmolVLM2-2.2B) and 3.3 s (SmolVLM-256M). With the blocks on OpenCL the embeddings are within 1.4e-5 to 4.3e-5 (relative norm) of the CPU's (LLaVA 1.5 and Gemma 3, two pictures each), on Vulkan within 1.2e-5 to 5.0e-5: its matrix units take a prompt's activations rounded to F16, and an encoder's products are computed there without that rounding since 2026-11-14, which is what Vulkan's columns above are with (below). OpenCL's column is with its bidirectional attention on tiles of its own (attention_bi64 / 80 / 128, opencl.md; the figures of the day's end, with the CPU's, are below); on the kernel of one work-group per head and token Gemma 3's 4 096 rows took 3.03 s a tile and its prompt 3.15 s, SmolVLM2-2.2B's prompt 0.98 s and SmolVLM-256M's 0.51 s. Gemma 3's row is with those tiles' keys and values in one buffer of local memory (2026-11-14): with a tile of each its prompt took 1.05 to 1.11 s and its 4 096 rows alone 0.91 s on the same day. Of the 0.82 s of kernel time of that prompt on OpenCL (OKLM_CL_PROFILE=1) the encoder's attention is 0.40 s (0.50 s before), in float arithmetic, and its matrix products 0.23 s, in F32 where Vulkan's matrix units take F16: with OKLM_CL_WMMA=1, which gives OpenCL's products those units and their rounding, the prompt takes 0.72 to 0.73 s.

The patch embedding and the projector on a device

With its blocks on a device, an encoder computed its patch embedding before them and its projector after them in host memory, the stream going up and coming back between the two. Timed on 2026-11-14 (RX 7900 XT, llama.cpp's test image, Vulkan, milliseconds of one tile or image):

Encoder Patch embedding Positions, norms Projector The encoder
Gemma 3 4B (4 096 patches) 3.4 cutting the patches, 18.1 the product 3.4 1.9 the pool and its norm, 5.0 the projection 721
Qwen2-VL 2B (1 564 patches, 391 embeddings) 0.8, 6.8 0.9 79.3 the merger 341
LLaVA 1.5 7B (576 patches) 0.5, 2.2 0.4 67.0 the two layers 108
SmolVLM-256M (a tile of 1 024 patches) 1.3, 4.3 0.6 0.2 the shuffle, 3.9 the projection 42
SmolVLM2-2.2B (a tile of 729 patches) 0.6, 3.4 0.6 0.2, 11.5 89

The two parts were 4 % of Gemma 3's encoder and 65 % of LLaVA 1.5's. Both are products, a norm and an activation, which the backends have: they moved to the device without a kernel or an op more (src/mm/vision.c, dev_stream() and dev_project()).

  • The patch embedding is a convolution whose stride is its kernel: a product over the patches' rows. The host still cuts the image into rows of 3 x patch^2 values and uploads them in place of the stream (588 values a patch of 14 pixels where SigLIP's stream has 1 152). The kernel is copied to the device with its rows widened with zeros to a multiple of 32 values, which a device's products ask for (588 becomes 608), and the product is the blocks' own, with the activations at F32's precision (oklm_matmul_wide()). SigLIP's bias, the positions and CLIP's norm are ops after it; CLIP's class token is uploaded as the stream's first row and the product is written into the rows after it. For a pair of frames (Qwen2-VL's and Qwen2.5-VL's video) each frame's patches meet their own kernel and the second product is added to the first.
  • Two layers (LLaVA, and the mergers of Qwen2-VL and Qwen2.5-VL): two products with the exact GELU between them, LLaVA's over the rows after the class token (a view), the mergers' over each group of four rows read as one (the rows of a group already follow each other there), after the last norm.
  • The pixel shuffle (SmolVLM) and the pool (Gemma 3) join the patches of a square group, which are not neighbours in reading order. The device therefore takes a tile's rows group by group (a group's rows, then its columns), the positions copied to it in that order: every row sees every row in the blocks, so the order changes the order of the sums and nothing else. The shuffle is then the stream read with each group's rows as one, and the pool fifteen additions of views of every sixteenth row, in the host's order, and a scale; Gemma 3's norm and the projections follow.

OKLM_MM_DEVICE_ENDS=0 keeps these parts in host memory, the blocks alone on the device (build.md); oklm_vision_device_parts() tells which it is.

Measured on 2026-11-14 (RX 7900 XT, one build, the blocks alone on the device -> the whole encoder; the prompt of oklm describe on llama.cpp's test image with the language model on the same device, three runs each; then the encoder alone on a second image and its embeddings against the CPU's, test_mm_public, two pictures):

Model Prompt, OpenCL Prompt, Vulkan Encoder alone, OpenCL Encoder alone, Vulkan Embeddings from the CPU's (OpenCL; Vulkan)
Gemma 3 4B 0.89 to 0.90 s -> 0.86 s 0.83 to 0.84 s -> 0.80 s 0.772 s -> 0.736 s 0.733 s -> 0.689 s 1.4e-5, 4.3e-5; 1.2e-5, 2.8e-5
LLaVA 1.5 7B 0.40 to 0.42 s -> 0.33 to 0.36 s 0.40 to 0.41 s -> 0.33 to 0.34 s 0.127 to 0.129 s -> 0.058 s 0.109 s -> 0.040 to 0.042 s 1.4e-5, 2.1e-5; 2.5e-5, 4.8e-5
LLaVA 1.6 Mistral 7B (five tiles) 1.82 to 1.88 s -> 1.49 to 1.52 s 1.87 to 1.89 s -> 1.53 to 1.54 s
Qwen2-VL 2B 0.47 s -> 0.38 s 0.44 s -> 0.35 s 0.254 s -> 0.188 s 0.234 to 0.235 s -> 0.171 to 0.172 s 6.4e-5, 5.1e-5; 2.1e-5, 3.0e-5
Qwen2.5-VL 7B 0.60 to 0.61 s -> 0.48 to 0.51 s 0.55 s -> 0.45 s 0.232 to 0.245 s -> 0.155 s 0.199 to 0.202 s -> 0.124 to 0.125 s 1.5e-5, 1.1e-5; 8.0e-6, 9.8e-6
SmolVLM-256M 0.33 s -> 0.28 s 0.29 to 0.30 s -> 0.25 s 0.044 s -> 0.035 s (a tile) 0.037 s -> 0.028 s 2.7e-6; 4.0e-6 (one tile)
SmolVLM2-2.2B (five tiles) 0.70 to 0.71 s -> 0.59 to 0.60 s 0.60 to 0.61 s -> 0.49 s

The embeddings are as far from the CPU's as with the blocks alone on the device (within 1.3e-5 of those figures in every row). SmolVLM's embeddings on a device are compared by the test since (the overview tile of SmolVLM-256M). The greedy answer about the newspaper from the device encoder's embeddings and from the CPU's agree on 46 of Qwen2-VL 2B's 48 choices with the encoder on OpenCL and 47 on Vulkan (the two tokens 0.18 apart where they part; 48 and 47 with the blocks alone there), and on 48 and 48 for Qwen2.5-VL 7B (47 and 47): near ties that fall either way with the order of a sum.

Checks: test_vision runs every tiny encoder (the generated SigLIP with a pixel shuffle, Gemma 3's, LLaVA's, Qwen2-VL's and Qwen2.5-VL's, their sized images and their pair of frames) on OpenCL and Vulkan both ways, the whole encoder on the device and its blocks alone, each within 8e-7 of the CPU's embeddings on values up to 2 (9.6e-7 on the Intel UHD 750), and back on the CPU to the CPU's bits; test_mm_public the published files, both ways too. test_encoder_types writes the generated encoder with its blocks' matrices in F16 and in BF16 (no file on disk is BF16), checks it against the reference in doubles and runs it through the device path on the tests' counting device, which computes with the CPU's kernels: the path's copies and widened rows in a type of 16 bits, not a device's kernels for it (those are the op tests'). An encoder with quantized matrices in its blocks is not put on a device at all (oklm_vision_use_backend() answers that it does not take them); no tiny file has such matrices (the encoder the test writes is 16 values wide, half a quantized block, and tools/gen_vision_fixtures.py writes floats), so that answer has no test. The forms the ops take there (a product into the rows after the first, additions of views of every n-th row) are checked as ops against the CPU's arithmetic by test_conv_views in tests/core/test_opencl.c and tests/core/test_vulkan_gemm.c, on the RX 7900 XT and the Intel UHD 750. None of them is handed back to the CPU on the published files (the log of OKLM_LOG_LEVEL=INFO names the ops a backend hands back).

Limits. The device holds the patch kernels, the positions, the norms and the projector's matrices beside the blocks, as the file holds them (oklm_vision_device_bytes() counts them and the larger activations); a device with room for the blocks and not for these keeps them in host memory, with a notice (the device has no room for the patch embedding and the projector): test_vision gives each tiny encoder a device with room for the blocks to the byte (the tests' counting device with a bound, computing with the CPU's kernels, tests/mm/count_backend.h) and finds the notice, the bytes held and the embeddings of OKLM_MM_DEVICE_ENDS=0, bit for bit, then with one byte less the refusal and the CPU's bits. So does an encoder whose projector's matrices are quantized or have rows that are not a multiple of 32 values (none on disk). The image is still cut into patches and normalized on the host. Gemma 3 gains 4 %: its 4 096 rows of attention are the encoder.

The tables measured again

At the end of the day's changes (2026-11-15, RX 7900 XT, one job at a time, load average 1.1 to 5.0): the whole encoder on the device, Vulkan's products at F32's precision, the question written as each model's template has it. The prompt of oklm describe on llama.cpp's test image, the language model on the same backend, three runs on each device and one on the CPU; then the encoder alone on a second image (test_mm_public, two pictures, the kernels built) and its embeddings against the CPU's:

Model (embeddings of the prompt) Prompt: CPU OpenCL Vulkan Encoder alone: CPU OpenCL Vulkan Embeddings from the CPU's (OpenCL; Vulkan)
LLaVA 1.5 7B (576) 7.17 s 0.33 to 0.35 s 0.34 s 1.12 to 1.61 s 0.058 s 0.040 s 1.4e-5, 2.1e-5; 2.5e-5, 4.8e-5
LLaVA 1.6 Mistral 7B (five tiles, 2 340) 30.2 s 1.48 to 1.51 s 1.53 to 1.54 s
Gemma 3 4B (256, from 4 096 rows) 12.3 s 0.85 s 0.80 s 10.3 to 10.7 s 0.737 s 0.689 s 1.4e-5, 4.3e-5; 1.2e-5, 2.8e-5
SmolVLM-256M (five tiles, 320) 2.98 s 0.28 s 0.24 to 0.26 s 0.40 s (a tile) 0.034 s 0.028 s 2.7e-6; 4.0e-6 (one tile)
SmolVLM2-2.2B (five tiles, 405) 9.64 s 0.59 s 0.49 to 0.51 s
Qwen2-VL 2B (391; the encoder alone on 273) 6.57 s 0.38 s 0.35 s 4.14 to 4.22 s 0.186 s 0.171 s 6.4e-5, 5.1e-5; 2.1e-5, 3.0e-5
Qwen2.5-VL 7B (391; 273) 8.93 s 0.48 to 0.52 s 0.45 s 3.22 to 3.64 s 0.154 s 0.125 to 0.126 s 1.5e-5, 1.1e-5; 8.0e-6, 9.8e-6

The devices' figures are those of the table of the patch embedding and the projector to within a hundredth of a second: nothing after it changed an image encoder's path. The CPU's are new: its encoder was taken on the same day for the large pictures (below), and Gemma 3's prompt, 23.3 s in the first table of this section, is 12.3 s.

A picture of 2 000 x 2 000 pixels, the two Qwen files, three runs on each device: at 1 225 embeddings (--tiles 980: the largest size until the default went to 4 096) the prompt of Qwen2-VL 2B takes 1.60 s on OpenCL and 1.67 to 1.68 s on Vulkan (1.89 to 1.91 s and 1.78 to 1.96 s when last measured, the blocks alone on the device), and Qwen2.5-VL 7B's 1.35 to 1.38 s and 1.22 s (1.72 to 1.76 s and 1.40 to 1.42 s then); at the default largest size, 4 096 embeddings and a prompt of 4 121 rows, 11.1 to 11.2 s and 13.4 s for Qwen2-VL 2B, 5.51 to 5.53 s and 5.58 s for Qwen2.5-VL 7B.

The encoders' precision on Vulkan

Vulkan's matrix units (cooperative matrices, vulkan.md) take the activations of a prompt's products rounded to F16. A language model's prompt bears it; an encoder's blocks do not: their activations reach tens (80 in Qwen2-VL's), the terms of a product cancel, and 23 to 32 blocks add the rounding up, to embeddings 2.6e-3 to 1.5e-2 from the CPU's where OpenCL's F32 products give 1.4e-5 to 5.3e-5 (3.1e-4 for Ultravox). The blocks of the vision and audio encoders therefore ask for their products with the activations at F32's precision (oklm_matmul_wide()), which Vulkan computes on the same units with each activation as two F16 parts, the rounded value and what the rounding dropped (vulkan.md: where the error enters, product by product, and the forms measured). The CPU and OpenCL compute them as before.

Measured on 2026-11-14 on the RX 7900 XT (RADV), one build, the activations rounded as before (OKLM_VK_WIDE=0) -> the default: the embeddings against the CPU's (relative norm; two pictures a vision encoder, jfk.wav for Ultravox) and the encoder alone (test_mm_public, a second image or sound), then the prompt of oklm describe on llama.cpp's test image or jfk.wav, the language model on Vulkan:

Encoder Embeddings from the CPU's Encoder alone Prompt
Qwen2-VL 2B (273 embeddings; the prompt's picture 391) 1.5e-2, 1.1e-2 -> 1.9e-5, 3.0e-5 0.18 s -> 0.23 to 0.24 s 0.37 to 0.38 s -> 0.43 to 0.45 s
Qwen2.5-VL 7B 6.3e-3, 4.8e-3 -> 1.1e-5, 9.6e-6 0.15 to 0.16 s -> 0.20 s 0.48 to 0.50 s -> 0.55 to 0.56 s
Gemma 3 4B (SigLIP, 4 096 rows) 2.6e-3, 5.1e-3 -> 1.2e-5, 2.8e-5 0.63 s -> 0.73 s 0.74 s -> 0.84 s
LLaVA 1.5 7B (CLIP, 577 rows) 5.6e-3, 7.2e-3 -> 2.3e-5, 5.0e-5 0.11 s -> 0.11 s 0.40 to 0.41 s -> 0.41 to 0.42 s
LLaVA 1.6 Mistral 7B (five tiles) 1.82 to 1.83 s -> 1.89 s
SmolVLM2-2.2B (five tiles) 0.49 to 0.50 s -> 0.60 s
SmolVLM-256M 0.26 s -> 0.29 to 0.31 s
Ultravox v0.5 (Whisper large-v3-turbo's encoder, jfk.wav) 9.6e-3 (largest difference 8.1e-2) -> 5.5e-5 (8.1e-4) 0.23 s -> 0.30 s 0.27 s -> 0.33 s
Whisper large-v3-turbo, oklm transcribe of jfk.wav 0.24 to 0.25 s -> 0.31 to 0.32 s
Whisper large-v3, the same 0.41 s -> 0.49 s

An encoder takes 1.0 to 1.3 times what it took, a prompt 1.03 to 1.22 times, a transcription of jfk.wav 1.2 to 1.3 times: 0.01 to 0.10 s an image or a 30-second window. With OKLM_VK_WIDE=f32 the same products run on the F32 tiles instead: Qwen2-VL 2B's embeddings 5.3e-5 and 4.6e-5 from the CPU's, its encoder 0.28 to 0.29 s and its prompt 0.50 s, Gemma 3's prompt 0.95 to 0.96 s, Ultravox's 0.38 to 0.39 s, jfk.wav in 0.37 s (large-v3-turbo) and 0.55 to 0.56 s (large-v3): no closer and slower, so the two parts are the default. OKLM_VK_WIDE=0 gives the earlier speed and rounding (build.md).

The answers do not move with it more than with any change of a sum's order. With the language model on Vulkan (its own prompt products keep their F16 rounding, to the bit), the newspaper answer from the Vulkan encoder's embeddings and from the CPU's agree on 47 of 48 greedy choices for Qwen2-VL 2B (47 before; the two tokens 0.24 apart where they part) and on 47 for Qwen2.5-VL 7B (48 before; 0.11 apart), as OpenCL's encoder does (48 and 47): a difference of 1e-5 in the embeddings is rounded to F16 in the language model's first products like one of 1e-2, so the logits of the two runs are as far apart as before (within 1.07 and 0.93, 0.90 and 1.02 before) and a near tie falls either way. Whisper's texts for jfk.wav are the same on both settings; large-v3-turbo's probability of English is 0.9499 where the CPU gives 0.9498 and the rounded products 0.9508.

Gemma 3

Gemma 3 (4B, 12B, 27B; the 1B and 270M read text only) was audited against llama.cpp's tools/mtmd at the pinned revision, which OKLM follows where the two differ from the model's own processor:

  • The picture. One image whatever its size, no tiles: fitted in a square of 896 pixels keeping its proportions (the sides rounded up), centred, the rest black, with a bilinear filter; then each channel to (x / 255 - 0.5) / 0.5. The resampling is Pillow's, in its fixed-point arithmetic (oklm_image_resize(), oklm_image_fit()): the bytes are llama.cpp's. The model's own processor stretches the picture to the square instead, and its "pan and scan" (crops of a large or elongated picture given as further images) is optional: neither llama.cpp nor OKLM does it.
  • The encoder. The SigLIP encoder above at 896 pixels in patches of 14: 4 096 patches through 27 blocks, every block and the final norm used, the tanh GELU.
  • The projector. The patches as a 64 x 64 grid, the mean of each 4 x 4 group (row by row: 256 embeddings), an RMS norm (epsilon 1e-6) times mm.soft_emb_norm, then mm.input_projection. The files do not say the group's side: 4, as for every size of the family.
  • The prompt. <start_of_image>, the 256 embeddings, <end_of_image>, where the marker is in the turn Gemma's template renders. llama.cpp adds no new line around them at this revision (the model's processor writes two before and two after).
  • Rows that see each other. Gemma 3 was trained with the image's embeddings attending to one another in both directions, and llama.cpp evaluates them so. oklm_eval_embd_mutual() and oklm_eval_batch_mutual() do it: each of these rows sees the rows before the image and every row of the image, the window of a windowed layer still counted from its own position; the tokens after the image see it as usual. oklm describe and the server use them when the projector asks (oklm_vision_mutual()); the server gives such an item to the model in one piece, waiting a step rather than cutting it, so that an answer does not depend on what else is being served. On a device the attention takes the last key these rows all see: OpenCL's attention kernel and Vulkan's attn_split shader, the ones of one work-group per head and token, on the contiguous and the paged cache (Verification has the runs). OpenCL's tiled prompt kernels take that last key too since 2026-11-14 (every block of queries then walks the keys up to it), where they take causal rows: heads up to 128 (Gemma 3 27B's). The 4B's and 12B's heads of 256 stay on the attention kernel, which measured faster for them: 20.0 ms of a prompt's 0.95 s of kernel time for the image's 256 rows and the text after them in the 4B's 34 layers, 25.6 to 35.5 ms on the tiles (opencl.md).

LLaVA 1.5

LLaVA 1.5 7B (Vicuna 7B, architecture llama, with CLIP ViT-L/14 at 336 pixels) runs from the files llama.cpp's documentation names (mys/ggml_llava-v1.5-7b, 2023). Audited against llama.cpp's tools/mtmd at the pinned revision and against the model's definition (CLIP's and LLaVA's own code), which differ here. OKLM computes the encoder as the model defines it and keeps doing so (which of the two is right):

  • The file. It names no clip.projector_type; a file that has clip.has_llava_projector is taken for mlp, as llama.cpp took it when the file was made. The pinned llama.cpp refuses it; the test reference reads a copy with the key added (tools/mmproj_add_type.py). The conversion left the last of CLIP's 24 blocks out, LLaVA reading the output of the one before it: the file holds 23.
  • The picture. Fitted in a square of 336 pixels keeping its proportions, centred, the rest the mean colour of CLIP's training images (122, 116, 104), with a bicubic filter; then CLIP's mean and deviation from the file. The bytes are llama.cpp's.
  • The encoder, as the model defines it, which is what OKLM computes. The class token first, with the first position; the patches after it; a norm; all 23 blocks with quick-GELU; no final norm; the class token's row dropped: 576 rows.
  • The encoder, as the pinned llama.cpp computes it. Three things differ: the class token is placed after the patches, so patch i takes the position of patch i - 1 and the class token the last; 22 blocks are run, one fewer; and the row dropped is the first, a patch, the class token's being kept as the last. A float64 pass written both ways (tools/vision_ref.py, --as-llama-cpp) shows these three are the whole difference: llama.cpp's embeddings are within 6e-3 and 1.5e-2 of the pass written its way, and 0.99 and 0.79 from the model's (mostly the rows being one patch apart). OKLM does not follow llama.cpp here.
  • The projector. mm.0, GELU, mm.2. The model's GELU is the exact one, which OKLM uses; llama.cpp's is the tanh form.
  • The prompt. The 576 embeddings where the marker is, nothing around them. The file has no chat template: the turns are Vicuna's as llama.cpp's vicuna template writes them (a system text and an empty line; USER: and ASSISTANT: lines; ASSISTANT: to answer), in oklm describe and in a server with this projector. LLaVA's own prompt joins the turns with spaces and opens with a sentence about a curious human and an assistant; give it with --system if wanted. Without the projector the same turns come from the catalogue, which names vicuna for this model: oklm chat -m llava-1.5-7b (or the file by its path in the model's folder) chats in them, and --chat-template vicuna gives them to a copy kept elsewhere (inference.md).

The CLIP encoder: the model's definition, or llama.cpp's

The pinned llama.cpp reads a LLaVA file with the class token after the patches, one block fewer, and the first patch dropped where the model drops the class token. The model's definition is the right one, on three grounds that do not depend on each other, and OKLM keeps it, for LLaVA 1.5 and 1.6 alike (the same encoder):

  • The model's code. CLIPVisionEmbeddings.forward() (transformers, modeling_clip.py) concatenates the class embedding and then the patches, and adds the position table row for row. LLaVA's CLIPVisionTower.feature_select() (llava/model/multimodal_encoder/clip_encoder.py) takes hidden_states[mm_vision_select_layer], -2 in the models' configurations, and with patch drops its first row; LlavaNextForConditionalGeneration.get_image_features() does the same (vision_feature_layer -2, [:, 1:]).
  • The file's position table. The positions of the patches were learned as a grid, so the table shows where its rows are without any code being run (tools/clip_positions.py): of the 577 rows of v.position_embd.weight, row 0 resembles none (its cosine with row 1 is 0.040) while the last two are neighbours (0.961, the median of consecutive rows being 0.980); and consecutive rows are least alike each time the first of the two is a multiple of 24 (0.92 against 0.97 to 0.99), which is where a row of 24 patches ends if the patches start at row 1. Row 0 is the class token's position and rows 1 to 576 the patches', in both files. Placed after the patches, the class token takes the last patch's position, and each patch its left neighbour's (the first of each row, the last of the row above; the very first, the class token's).
  • The blocks the file holds. CLIP ViT-L/14 has 24 blocks and a final norm; hidden_states[-2] is what 23 of them give, without that norm. The conversion kept exactly that: 23 blocks, no v.post_ln. Running 22 of them gives hidden_states[-3], which no configuration of the model asks for.

What it costs to differ from llama.cpp: the embeddings of one image are not llama.cpp's (0.99 and 0.79 apart in relative norm on the two test pictures, mostly the rows being one patch apart), so an answer may read differently in the two programs; and llama.cpp cannot be the reference of this encoder, which is held to a float64 pass of the definition instead (Verification).

LLaVA 1.6

LLaVA 1.6 ("LLaVA-NeXT") Mistral 7B (Mistral 7B Instruct v0.2, architecture llama, with the encoder of LLaVA 1.5) runs from the published files (cjpais/llava-1.6-mistral-7b-gguf, catalogue llava-1.6-mistral-7b). Its definition here is the model's own processor and code as transformers has them (image_processing_llava_next.py, modeling_llava_next.py, read at v4.49, with LLaVA's own llava/mm_utils.py and llava/model/llava_arch.py beside them), not llama.cpp: the pinned llama.cpp gives a file that lists resolutions to its own slicing, an overview and tiles of 576 embeddings each, without the cut nor the ends of rows described below.

  • The file. clip.projector_type is mlp; clip.vision.image_grid_pinpoints lists five resolutions (336 x 672, 672 x 336, 672 x 672, 1 008 x 336, 336 x 1 008), each a whole number of tiles of 336 pixels; clip.vision.mm_patch_merge_type is spatial_unpad; the tensor model.image_newline is one embedding of the language model's width. The encoder's tensors are LLaVA 1.5's (23 blocks, no final norm).
  • The resolution. The one that keeps the most of the image's pixels once the image is scaled into it without changing its proportions, and, among equals, the one with the least area left over; the first of the list at a tie (select_best_resolution()). The pairs are read as transformers reads them, a height then a width. LLaVA's own code reads the same list as widths then heights: the list being symmetric, only a tie differs, and a picture small enough to fit whole in every resolution (under 336 pixels both ways) gets the first of the list, two tiles side by side here, one above the other there.
  • The tiles. The image is resized into that resolution keeping its proportions (the shorter side rounded up, bicubic), pasted centred on black (the offset rounded down), and cut in tiles of 336 pixels row by row. The overview is the whole image stretched to 336 x 336 (bicubic), without the mean-coloured bars of LLaVA 1.5. Each is then normalized with CLIP's mean and deviation. transformers v4.49 pads both ends by the offset rounded down, which leaves the canvas a pixel short when the padding is odd; later versions put the pixel at the end, as LLaVA's own paste does and as OKLM does.
  • The encoder and the projector. LLaVA 1.5's, on the overview and on each tile: 576 embeddings each.
  • The rows (pack_image_features(), oklm_vision_pack()). The overview's 576 embeddings first. Then the tiles' embeddings as one grid of features, 24 to a tile's side (48 x 48 for four tiles), from which the part that covers the black padding is cut (unpad_image()): the image's proportions against the grid's say which side was padded; the image's size on that side, in features, is rounded down, and half the difference, rounded down, is cut from each end, so that one row or column of padding stays when the difference is odd. The grid is read row by row, and after each row comes the image_newline embedding. A picture of 640 x 488 pixels gives 576 + 36 x (48 + 1) = 2 340 rows; the most is 2 928 (a square picture of 672 pixels or more). The overview alone (--tiles 336 or less) is the model's own form for an image of one tile: its 576 embeddings and that embedding once.
  • The prompt. The rows where the marker is, nothing around them. The file carries Mistral's chat template, which is used: [INST] <image>\nQUESTION [/INST], the form LLaVA's own mistral_instruct conversation writes. That template refuses a system message, as Mistral 7B Instruct v0.2's does (LLaVA's conversation for this model has none either). A LLaVA 1.6 file without a template (the Vicuna variants) gets Vicuna's turns, as LLaVA 1.5 does; no such file was run.
  • Room. An image takes up to 2 928 rows of the context: the default of 4 096 holds one with its answer, and a second image in a conversation needs -c 8192.

Qwen2-VL and Qwen2.5-VL

Qwen2-VL (2B run here) and Qwen2.5-VL (3B, 7B, 32B and 72B run here: the other sizes) were audited against the models' own code (transformers: modeling_qwen2_vl.py, image_processing_qwen2_vl.py, modeling_qwen2_5_vl.py, read at v4.49) and against llama.cpp's tools/mtmd at the pinned revision (models/qwen2vl.cpp, clip.cpp, mtmd.cpp), on the published files (ggml-org/Qwen2-VL-2B-Instruct-GGUF, unsloth/Qwen2.5-VL-7B-Instruct-GGUF). The two agree on the picture's rows and their positions; they differ on the limits of the picture's size, in arithmetic (Verification) and, in the language model, on the epsilon of the RMS norms, which the pinned llama.cpp does not read for this architecture (Qwen2.5-VL 7B and llama.cpp's RMS norm).

  • The picture. One image at its own resolution, no tiles and no padding: each side rounded to the nearest multiple of 28 pixels (a patch of 14 times the merger's 2; Python's round(), halves to the even neighbour), then, when the area passes the maximum, both sides divided by sqrt(area / maximum) and rounded down to a multiple of 28, or, when it is under the minimum, multiplied by sqrt(minimum / area) and rounded up. The picture is stretched to that size with a bicubic filter (the proportions change by the rounding only), then CLIP's mean and deviation from the file. The bytes are llama.cpp's.
  • The limits of the area are where the references differ, the files holding neither key (clip.vision.image_min_pixels, image_max_pixels, read when present). The processor class's defaults are 3 136 and 1 003 520 pixels (4 to 1 280 embeddings). The preprocessor_config.json published with every size of the two models (read on 2026-11-14: Qwen2-VL 2B, Qwen2.5-VL 3B, 7B, 32B, 72B) keeps the minimum and raises the maximum to 12 845 056 pixels (16 384 embeddings); llama.cpp at the pinned revision takes 8 and 4 096 embeddings (6 272 and 3 211 264 pixels). OKLM takes the processor's minimum and llama.cpp's maximum, 4 to 4 096 embeddings, since 2026-11-14 (1 280 before): a picture between 6 272 and 3 211 264 pixels gets llama.cpp's size, a larger one is reduced to 3.2 megapixels as llama.cpp reduces it. The models' own 16 384 is not reachable (the owner's decision, 2026-11-14): no reference encodes it, its 65 536 patches attend each to each (sixteen times the work of 4 096 embeddings), and the rows are four times the default context. --tiles SIDE (of oklm describe and of oklm serve) caps the area under the default, SIDE x SIDE pixels. Where the rows must fit a context of a fixed length the default is bounded by it (oklm_vision_limit_tokens()): oklm serve and the captioner of oklm rag index give an image at most three quarters of a sequence's context (3 072 embeddings with -c 4096; the whole 4 096 from -c 5464 on), so a large photograph is reduced rather than its request refused for its length; --tiles replaces that bound by the area asked. A prompt's rows name a picture by its content and by the bound in force, so embeddings kept under another bound (a server whose context was resized) are not taken for it. oklm describe without -c sizes its context to the pictures instead (oklm_vision_image_tokens() gives an image's embeddings before a pixel is read): 5 120 tokens for a picture of 4 015 embeddings.
  • The patches. The model embeds two frames at once with a three-dimensional convolution without bias (kernel 2 x 14 x 14); a still image is given twice, so its embedding is the sum of the two kernels (v.patch_embd.weight, v.patch_embd.weight.1) applied to the same patch: the kernels are summed when the file is loaded. No class token, no learned positions, no norm before the blocks.
  • The order of the rows. The patches are not in reading order: the four patches of each 2 x 2 group follow each other (the merger joins consecutive rows), groups in reading order; Qwen2.5-VL orders the groups by window as well. Attention does not depend on the order, each row carrying its own position: OKLM orders the groups window by window for both (one window holding everything for Qwen2-VL), so a window is a run of rows, and puts the embeddings back in reading order at the end.
  • Positions. Rotary, two-dimensional, in every block's attention: with a head of d values, pair i (values i and i + d/2) is rotated by the patch's row times 10000^(-i / (d/4)) for i under d/4, and by its column times 10000^(-(i - d/4) / (d/4)) from d/4 on. Rows and columns are counted in patches.
  • The blocks of Qwen2-VL (32, width 1 280, 16 heads): layer norm (epsilon 1e-6), attention with biases, every row seeing every row; layer norm, a linear layer to 5 120, quick-GELU (x sigmoid(1.702 x)), a linear layer back. The file names the first of the two ffn_down, and gives 1 536, the language model's width, as clip.vision.feed_forward_length: the MLP's width is read from the tensors.
  • The blocks of Qwen2.5-VL (32, width 1 280, 16 heads): RMS norms with a weight and no bias; a gated MLP with biases, down(SiLU(gate(x)) times up(x)), 3 420 wide; and windowed attention: the merged grid is cut into windows of 4 x 4 embeddings (8 x 8 patches, 112 pixels, clip.vision.window_size when the file has it; the windows at the right and bottom edges are smaller), a row sees its window only, except in the blocks with full attention: 7, 15, 23 and 31 (fullatt_block_indexes in the model's configuration; the file says clip.vision.n_wa_pattern = 8, every eighth block, which is the same list).
  • The merger. A norm (v.post_ln: layer norm for Qwen2-VL, RMS for 2.5), the four rows of each group joined into one of 5 120, mm.0, GELU, mm.2 to the language model's width (1 536 for the 2B, 3 584 for the 7B). The model's GELU is the exact one, which OKLM uses; llama.cpp's is the tanh form. An image of w x h pixels becomes (w / 28) x (h / 28) embeddings in reading order.
  • The prompt. <|vision_start|>, the embeddings, <|vision_end|> where the marker is in the turn the file's ChatML template renders. oklm describe and the server's chat routes put the question right after the image, as the model's own template does (the table of the projectors).
  • Positions in the language model (M-RoPE). general.architecture is qwen2vl: the tensors and every other key are qwen2's, plus qwen2vl.rope.dimension_sections = [16, 24, 24, 0]. A row has three positions: time, height, width. Of the 64 rotated pairs of a head (pair i is values i and i + 64, frequency 1000000^(-i / 64) as in qwen2), the first 16 turn by the time position, the next 24 by the height, the last 24 by the width. A token has the three equal, so a text is rotated exactly as qwen2 rotates it. An image whose first embedding would take position p has, for the embedding at row r and column c of its grid, time p, height p + r, width p + c; the text after it continues from p + max(rows, columns), the largest position used plus one, not from p plus the number of embeddings: the 391 embeddings of a 644 x 476 picture take 23 positions. llama.cpp does the same at the pinned revision.

How the language side does it (src/model/mrope.c):

  • oklm_eval_embd_grid() and oklm_eval_batch_grid() take an image's rows with the columns of its grid, and the index within the image of the first row given, so an image may come in pieces (the server's); the positions follow from that and from what the sequence holds. oklm_mm_eval() and the server call them for these projectors (oklm_mm_media.grid_w). oklm_seq_n_pos() gives the position the next token takes, which is no longer the number of cached rows.
  • The context keeps two numbers per cached row, its time position and the position a token after it takes. A cut (oklm_seq_truncate()) anywhere, in the middle of an image too, leaves the rows before it with theirs, and the rest of the image can be given again; a copy of a sequence copies them; a sequence swapped out keeps them. A paged cache keeps them per block, so the prefix cache gives a sequence the rows another computed, an image's included, with their positions (equal rows after equal rows have equal positions). A KV file of a sequence that holds an image or a video keeps the two per row too (version 2 of the format, inference.md): the context that loads it has the positions of the one that saved, and a token after the image takes the position it would have taken there. Whether a sequence needs that version is told row by row: an image of one column ends at the position its row count gives, as tokens would, and still needs it.
  • A batch whose rows all have three equal positions (text, in a conversation that holds images too) goes through the usual rotary operation. A batch holding an image's rows goes through the same operation given three positions a row and the bands (oklm_rope_parts(): the first pairs of a head turned by the first position, the next by the second, the others by the third), where the block runs: the CPU kernel, OpenCL's rope kernel and Vulkan's norm_rope shader read the position of a pair's band, and compute its angle as the one-position rotation does at that position (on the CPU bit for bit, test_ops). The three positions of a batch's rows follow its one-part positions in the context's position buffer of each device. OpenCL's fused launches (the rotation with the head norms, or with the cache writes) take one position a row: such a batch is rotated by the plain kernel.
  • The labels of the template (2026-11-14). The chat template in the files of Qwen2-VL and Qwen2.5-VL counts the pictures and the videos of a conversation and, when its add_vision_id variable is set, writes Picture 1: , Picture 2: right before each picture and Video 1: before each video, each kind counted through the whole conversation. Nothing sets the variable unless the caller does: the template renders no label without it, and the pinned llama.cpp never sets it (it gives the template a string with its markers in it), so no label is the default here too, and the answers compared with llama.cpp's are unchanged. The Transformers processor passes the variable through when its caller gives it (its code is not installed here: the default was read in the template, not in the processor). The labelled form is offered: oklm describe --vision-id, and chat_template_kwargs: {"add_vision_id": true} (or add_vision_id: true beside the messages) in a chat request to oklm serve, both written by oklm_mm_content_numbered(), which oklm_mm_content() calls without a numbering. test_vision's test_vision_id renders four conversations (two pictures, a picture between two texts, two turns with a video and two pictures, two videos) with the files' template and compares them, with and without labels, with jinja2's rendering of the same conversations given in parts (tools/qwen_vision_id_ref.py, the same output for the 2B's and the 7B's file). Run on Qwen2-VL 2B (Q4_K_M, the CPU, --temp 0, two pictures of 448 x 336, a newspaper's front page and its top left corner, Describe Picture 2 in one sentence.): 425 prompt tokens with --vision-id against 415 (5 tokens a label), and an answer either way ("The New York Times newspaper from July 20, 1969, reports on the historic event of men walking on the moon." with the labels, "[...] featuring the headline "Men Walk on Moon."" without); asked which picture shows the whole page, it answers A with the labels and B without, so the labels are read but this small model does not answer by them. For the other projectors the option changes nothing (their templates have no such variable): oklm describe says so, a request is answered as without it. --vision-id is refused with --kv-load: the file does not say how many pictures and videos the conversation holds, so the count cannot go on.
  • Not covered. Several images at once in the model's own batch.

Qwen2.5-VL 7B and llama.cpp's RMS norm

The 7B's logits part from llama.cpp's as a prompt takes pictures, the same embeddings given to both (Verification), where the 2B's do not. The cause, found on 2026-11-14, is one number: the pinned llama.cpp evaluates a qwen2vl file's RMS norms with an epsilon of 0, and this model has rows that need the file's.

  • The model. Qwen2RMSNorm divides a row by sqrt(mean(x^2) + epsilon), and the decoder's norms take config.rms_norm_eps, 1e-6 in the published config.json (modeling_qwen2.py, modeling_qwen2_5_vl.py, read at v4.57). The GGUF files carry it (qwen2vl.attention.layer_norm_rms_epsilon = 1e-6, the 2B's and the 7B's), and OKLM reads it as for every architecture (src/io/hf_config.c).
  • llama.cpp (commit 8642541b, src/models/qwen2vl.cpp): load_arch_hparams() of this architecture reads the rotary sections and nothing else, so f_norm_rms_eps is never set (qwen2.cpp and qwen3vl.cpp read the key; the file is the same on its master branch on 2026-11-14). Its first norm gives every row the norm sqrt(3 584) = 59.867 before the weights, which is an epsilon of 0; with the file's an ordinary token's row has 59.77 and <|im_start|>'s 10.78.
  • Why the 7B and not the 2B. The epsilon matters for a row whose mean square is near it. The 7B's embeddings of <|im_start|> and <|im_end|> have norms 0.0110 and 0.0149 (mean squares 3.4e-8 and 6.2e-8) where an ordinary token's is near 1.05 (3.1e-4): the model's first norm leaves them 0.18 and 0.24 of what a norm without epsilon gives, and llama.cpp reads them 5.5 and 4.1 times too large. 4 248 of the vocabulary's 152 064 rows have a mean square under 1e-5. The 2B's two tokens have norms 0.50 and 0.56 (a factor 0.997), and none of its rows is under 1e-5: both programs give it the same norms to 0.3 %.
  • What it changes inside. Every chat prompt starts with <|im_start|>. With an epsilon of 0 that first row is the one whose residual stream grows to a norm of thousands from the fourth block on (2 825 after it, the others 26 to 55; Q4_K_M, the one-picture prompt); with the model's epsilon it is the third row, the new line after system (7 330), and the first stays at 41. The rows after attend to a different row for it, and the image rows are most of them.

The arbiter is tools/qwen2vl_ref.py: the language model of a qwen2vl GGUF in numpy float64, written from the transformers code (the norm above, get_rope_index()'s three positions, apply_multimodal_rotary_pos_emb(), grouped-query attention, SwiGLU), the weights through gguf-py's dequantizers, on a prompt's rows and embeddings as test_mm_public writes them; --eps 0 is the same pass as a program that ignores the key computes it. On the BF16 file (unsloth's, 14.2 GiB, fetched for this and removed), whose products are float products everywhere, the residual stream after a block of the one-picture prompt (418 rows; relative difference over all the rows, then of the last row) and the logits of the answer's first token:

Qwen2.5-VL 7B BF16, one picture OKLM from the model's pass llama.cpp from the model's pass llama.cpp from the pass with an epsilon of 0
after block 0 3.0e-4, 2.6e-3 3.7e-2, 0.10 (its first row: 0.99) 7.2e-4, 3.1e-3
after block 3 7.2e-4, 1.8e-3 1.06, 0.25 (its first row: 70) 9.2e-3, 2.9e-3
after block 10 5.6e-4, 1.2e-3 1.10, 0.31 5.4e-3, 4.6e-3
after block 26 5.4e-4, 6.0e-4 0.52, 0.11 3.2e-3, 2.3e-3
after block 27, the last row 9.8e-4 0.21 3.4e-3
logits: largest, mean 0.014, 0.0020 3.60, 0.85 0.089, 0.014

The difference is there after the first block, in the rows of the special tokens, and does not come from the positions: the two programs give every row the same three (the text before at 0 to 14, the image's 23 x 17 embeddings at time 15, rows 15 to 31, columns 15 to 37, the text after from 38; with three pictures the texts after them from 38, 53 and 76), which are get_rope_index()'s, and the same bands (16, 24 and 24 pairs of a head of 128).

The logits of the first token on the four prompts (largest difference over the vocabulary, then the mean; llama.cpp given OKLM's embeddings):

Prompt (rows) OKLM from the model's pass llama.cpp from the model's pass llama.cpp from the pass with an epsilon of 0
BF16: a text alone (38) 0.007, 0.0006 1.13, 0.27 0.055, 0.011
one picture (418) 0.014, 0.0020 3.60, 0.85 0.089, 0.014
two pictures, one turn (518) 0.149, 0.072 6.42, 2.42 0.150, 0.022
three pictures, one turn (797) 0.013, 0.0021 4.87, 2.45 0.228, 0.139
Q4_K_M: a text alone (38) 0.84, 0.13 1.32, 0.18 1.14, 0.33
one picture (418) 1.03, 0.19 2.86, 0.50 0.80, 0.12
two pictures, one turn (518) 1.25, 0.27 3.29, 0.51 1.43, 0.35
three pictures, one turn (797) 1.31, 0.45 8.53, 4.14 1.36, 0.42

OKLM is the model's on the float file (the 0.149 of the two-picture prompt is the keys kept as half floats: 2.2e-4 with an F32 cache), and llama.cpp is the model with an epsilon of 0 (OKLM with its epsilon set to 0 in a test harness, not an option of the runtime, comes within 0.05, 0.09, 0.13 and 0.17 of llama.cpp on the four BF16 prompts). On the quantized file the two programs' 8-bit activations cost up to 1.3 of a logit against a pass that rounds nothing, and llama.cpp's distance stands out of that with pictures. The best token is the same in the three computations of the eight cases; with three pictures the model gives it 21.10, OKLM 21.09 and llama.cpp 25.18 (BF16). Nothing was changed in OKLM, and the tests keep llama.cpp as their reference with the bounds they had (test_qwen25vl_several_images: 85 % of its choices, near ties of 1.5).

Whole answers, and the 3B. With --answer the pass takes the 48 tokens of a recorded answer after the prompt (teacher forcing: one causal pass gives the logits before each of them, 80 s for the 7B's 418 rows and 48 tokens, 42 s for the 3B's), and compares every position. On the Q4_K_M files, each program fed the same rows and the tokens of llama.cpp's answer (largest difference over the vocabulary and the 48 positions, the mean, and the positions whose best token is the pass's):

Q4_K_M, 48 tokens OKLM from the model's pass llama.cpp from the model's pass llama.cpp from the pass with an epsilon of 0
7B, one picture (418 rows) 1.26, 0.169, 47 2.96, 0.403, 47 1.12, 0.143, 48
7B, a text alone (38 rows) 1.03, 0.156, 46 2.30, 0.233, 47 not run
3B, one picture (418 rows) 4.58, 0.220, 46 4.21, 0.211, 47 4.26, 0.211, 47

The 7B's later tokens say what its first did: OKLM stays within 1.26 of the model at every position (0.51 to 1.26 a position) where llama.cpp is 0.73 to 2.96 from it, and llama.cpp is the pass with an epsilon of 0 (OKLM is 3.24 from that one, 0.358 on average). Where OKLM's best token is not the pass's, the pass has the two within 0.055 (the picture) and 0.133 and 0.175 (the text).

The 3B does not show the effect: its embeddings of <|im_start|> and <|im_end|> have mean squares of 1.91e-4 and 1.99e-4, a thousand times the epsilon (its first norm leaves them 0.997 and 0.998 of what a norm without epsilon gives, as the 2B's), and llama.cpp is as far from the pass with the file's epsilon as from the one with 0 (the first token: 0.970 and 0.968; OKLM 1.12). Its largest differences are not the epsilon's: over the first 37 tokens OKLM is within 1.61 of the pass and llama.cpp within 1.50, and from the 38th on single logits move by up to 4.58 (OKLM) and 4.21 (llama.cpp) at the same tokens, the mean staying at 0.22: both programs round the activations of this file's products to 8 bits, and the pass rounds nothing.

test_qwen25vl_answer and test_qwen25vl_text hold OKLM's CPU logits to that pass where its recording is present (build/ref/mm/<tag>_same_answer_f64.bin and <tag>_text_answer_f64.bin, 96 KiB each: the pass's 256 largest logits and their tokens before each of the 48, written by tools/qwen2vl_ref.py --record; tools/gen_mm_ref.sh runs it for the 3B and the 7B). This is the bound that can be tight, the weights being the file's in both: over those logits OKLM is within 0.943 (0.157 on average) on the 7B's picture, 0.689 (0.148) on its text and 3.17 (0.225) on the 3B's picture, bounded at 1.5, 1.5 and 5.0 with a mean of 0.35; llama.cpp, printed beside it and not bounded, is 2.31, 1.95 and 4.21 from the same logits. The comparison with llama.cpp keeps its bounds.

Not measured: the 32B and the 72B against the float64 pass (their files, 18.5 and 22.1 GiB, were removed and do not fit beside the other work on this disk; the pass itself would take their 64 and 80 layers in minutes), so whether their special tokens' rows are as small as the 7B's is not known; whole answers on the BF16 file (14.2 GiB, removed), where the first token alone was compared; whole answers of the prompts with several pictures.

The other sizes of Qwen2.5-VL

The 3B, the 32B and the 72B have the 7B's encoder (32 blocks of width 1 280, windowed attention) with a merger as wide as their language model (2 048, 5 120 and 8 192), and the same qwen2vl architecture (36, 64 and 80 layers). Each was fetched on 2026-11-14, run as the 7B was and removed, one file on the disk at a time (tests/public_models_large.txt has the URLs and SHA-256 sums; the catalogue has them as qwen2.5-vl-3b, qwen2.5-vl-32b and qwen2.5-vl-72b): ggml-org's Q4_K_M for the 3B (1.8 GiB) and the 32B (18.5 GiB), and for the 72B bartowski's IQ1_M (22.1 GiB, 1.75 bits a weight), since no 2-bit or 3-bit file of it (23.7 GiB and more) left room on a disk with 26.8 GiB free; the F16 projector each time. test_mm_public runs the three as it runs the 7B: two pictures through the encoder against a float64 pass of the model's definition and against llama.cpp's embeddings, then the answer about llama.cpp's test image on llama.cpp's greedy tokens (48 for the 3B, 16 for the two large files), from the same embeddings and against its whole path, then on OpenCL and on Vulkan.

3B Q4_K_M 32B Q4_K_M 72B IQ1_M
Embeddings from the float64 pass (two pictures) 3.5e-6, 4.8e-6 1.2e-5, 2.9e-6 2.7e-6, 3.8e-6
from llama.cpp's 4.2e-3, 6.8e-3 1.2e-2, 3.4e-2 4.1e-3, 9.6e-3
The encoder on OpenCL, from the CPU's; time 8.2e-6, 1.0e-5; 0.15 s 4.8e-5, 1.4e-5; 0.15 s 1.3e-5, 2.7e-5; 0.15 s
on Vulkan 1.6e-5, 1.2e-5; 0.12 s 1.7e-4, 8.5e-5; 0.12 s 8.7e-6, 2.1e-5; 0.12 s
Rows of prompt, as llama.cpp's 418 418 418
Choices llama.cpp's: same embeddings, whole path 47, 47 of 48 16, 16 of 16 15, 15 of 16
Logits from llama.cpp's: largest (mean) 4.93 (0.19), 1.96 (0.18) 3.40 (0.25), 3.39 (0.25) 1.18 (0.19), 1.29 (0.15)
CPU: prompt rows/s, tokens/s 241, 14.5 23, 1.5 8, 0.7
OpenCL, the layers that fit: choices the CPU's, logits; rows/s, tokens/s 36 of 36: 47, 4.18; 3 460, 141 61 of 64: 16, 0.72; 253, 13.7 62 of 80: 14, 0.83; 29, 2.7
OpenCL, half the layers, paged cache 47, 4.34; 493, 24.0 16, 0.73; 43, 2.8 15, 0.78; 15, 1.2
Vulkan, the layers that fit 48, 3.42; 2 458, 193 16, 0.87; 227, 14.9 14, 0.84; 29, 2.7
Vulkan, half the layers, paged cache 48, 1.49; 500, 28.5 16, 0.74; 42, 2.9 15, 0.80; 15, 1.3
oklm describe, the prompt: CPU, OpenCL, Vulkan 8.72 s, 0.37 s, 0.32 s not run, 3.50 s, 3.70 s not run, 21.0 s, 18.0 s

The 3B was also run with half its layers and the contiguous cache (47 and 48 of 48) and with the CPU's half streamed for the prompt (48 and 48, 1 163 and 1 192 rows/s). In the first of its layouts' rows the two large files have what oklm_context_params.n_gpu_layers = -1 places on the 20 GiB card, the other layers on the CPU; oklm describe with its context of 4 096 and the encoder on the device places 59 of the 32B's 64. The 72B's rates were measured while another test ran on the machine. Where the 72B's choices part from the CPU's (14 of 16) the two tokens are 0.18 apart on the CPU: the tests do not hold choices that part at such near ties against a device on these short answers. The 1.75-bit 72B reads the page ("Men Walk on Moon" / The subheadlines include: / "Astronauts Land on Plain [...]"); the 32B names the paper and its date ("The newspaper shown is the New York Times, dated Monday, July 21, 1969 [...]"). The 32B's embeddings are the furthest of the four from the CPU's on a device (1.1e-4 on OpenCL with the blocks alone there, 1.7e-4 on Vulkan, for one of the two pictures: its projector's values reach 46), which the tests' bound of 2e-4 holds. Not run: the other quantizations, a video and several pictures on these three sizes, the 32B and the 72B on the CPU through oklm describe.

A video for Qwen2-VL and Qwen2.5-VL

oklm describe gives these models a video (.ogv, or another container through --ffmpeg) as their processors and code do (video_processing_qwen2_vl.py, modeling_qwen2_vl.py and modeling_qwen2_5_vl.py of transformers), where the other projectors get the frames as still images:

oklm describe -m models/qwen2.5-vl-7b/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
    --mmproj models/qwen2.5-vl-7b/mmproj-F16.gguf \
    --fps 2 --frames 8 -p "What happens in this video?" clip.ogv
  • Frame pairs. The patch embedding is a convolution over two frames: the file's first kernel (v.patch_embd.weight) reads the first frame of a pair and the second (v.patch_embd.weight.1) the second, one grid of embeddings a pair (oklm_vision_encode_pair()). A still image is the same frame twice, which is the two kernels summed: the image path, unchanged. An odd count repeats its last frame, as the processor does: that pair is the last frame as a still image, bit for bit. Each pair then goes through the blocks and the merger on its own, as an image does: in the models' code a row sees the rows of its own grid only, in the blocks with full attention as in Qwen2.5-VL's windows.
  • The frames' size. Every frame is stretched (bicubic) to one size, that of the first within the limits of the models' video processor: 128 to 768 embeddings a frame (an image's are 4 to 4 096), sides in units of 28 pixels (oklm_vision_frame_size(), oklm_vision_frame()). --tiles SIDE sets the largest area to SIDE x SIDE pixels instead; under 128 embeddings the smallest area is an image's, so smaller frames can be asked for (a prompt that fits a small context, or the CPU's patience).
  • The prompt. <|vision_start|>, the grids of all the pairs one after the other, <|vision_end|>, right before the question as an image is: the processor writes one <|video_pad|> an embedding (pairs x rows x columns) between the two. No text gives a frame's time: Qwen2.5-VL's "absolute time" is a position, not a token.
  • The time position. A row of the video has, as height and width, the video's start p plus its row and column in its own grid, and as time p plus what its grid has advanced: its pair's index for Qwen2-VL, whatever the frames' times; for Qwen2.5-VL the index times the seconds a pair covers times two (vision_config.tokens_per_second of Qwen2.5-VL-7B-Instruct's configuration; the mmproj file has no key for it), the fraction dropped. A pair covers twice the distance between frames: at the two frames a second of the model's processor a pair is one second and two positions; at oklm describe's default of one frame a second, four. The text after the video continues from the largest of the three positions used plus one, which the time can be: three pairs of Qwen2.5-VL at one frame a second end at p + 8 when the grid is narrower than that. oklm_vision_time_step() gives the step, oklm_eval_embd_video() and oklm_eval_batch_video() take the rows with the grid's columns, rows and step (a video in pieces, a cut in its middle, a copy, the paged cache and its prefix cache as for an image: the context's two numbers a row are enough, a video's start being its last row's time less what its grid has advanced). One or two frames are one grid: an image's positions.
  • transformers after 4.49. Qwen2.5-VL's time is taken here as the model was published (4.49.0: index x seconds x 2, then truncated). transformers 4.57.1 truncates the seconds a pair first: the same positions whenever a pair covers whole seconds (its processor's default), and every pair at the first one's time when it covers less than one (four frames a second or more). OKLM keeps the published form there, by the owner's decision (2026-11-14).
  • Frames taken. --fps (1) and --frames (8) choose the frames as for the other models; the processors' own sampling (two frames a second, 4 to 768 frames) is not applied (the owner's decision, 2026-11-14: these defaults stay), and the time position follows the distance between the frames kept. Eight frames of the smallest size are 512 embeddings, of the largest 3 072: give the context room with -c.
  • Through the server. A video_url part of a chat request is this video item (server.md): the scheduler gives its rows to the model in the pieces its steps cut, each with the grid's columns, rows and time step (oklm_eval_batch_video()), and the item's hash (frames, count, spacing, --tiles) keys its embeddings in the server's cache and its rows in the prefix cache. The clip of the run below (four frames a second apart of 448 x 336) through oklm serve and through oklm describe --temp 0 on Qwen2-VL 2B (Q4_K_M, the CPU): 384 embeddings and 418 prompt tokens by either way and the same 64 tokens ("The video starts with a close-up shot of a newspaper with the headline "Men Walk on Moon" and the text "Astronauts Land on Plain, Collect Rocks, Plant Flag." It then cuts to a black and white image of a man walking on the moon. [...]"); the request sent again finds 416 of its 418 rows in the prefix cache, the video's embeddings kept; /v1/messages with the video as a video block gives the same text.
  • Not covered. The processors' rule of at least four frames (by decision, above); a video's sound (these files have no audio encoder); several videos at once in the model's own batch.

The encoder of Qwen2-VL and Qwen2.5-VL on a device

With oklm_vision_use_backend() the 32 blocks of these encoders run on the device, as the other encoders' do, and with them, since 2026-11-14, the patch embedding before them and the merger after them (above); the CPU's pass is unchanged (its rotation is still the host's table, the same bits as before).

  • Positions. The rows' two-dimensional rotation is the rotary op with three positions a row and a band of pairs each (oklm_rope_parts_factors()): a row's positions are its patch's row, its column and 0, the first quarter of a head's values turns by the first, the second quarter by the second, no pair is left to the third. The op's frequencies run on across the bands, the model's start again at each: the op is given a base of 1 (every pair's angle is then its band's position) and a factor a pair, 10000^(i / (d/4)), which divides it. The angle is the host table's to the last place (the table multiplies by the inverse).
  • Windows. A block with windowed attention (28 of Qwen2.5-VL's 32) is one attention op, oklm_attention_spans(): bidirectional attention with a table of two values a row, the first row of its window and one past the last, uploaded once a picture beside the positions (the rows are ordered window by window already, so a window is a run of rows). Both backends attend it on their bidirectional tiles in one launch a block (opencl.md, vulkan.md): a block of 32 rows walks the tiles of keys from the first row any of its rows sees to the last, each row keeping the scores of its own window, so windows of any sizes are taken, the smaller ones at the right and bottom edges included. A picture of 273 embeddings has 24 windows, one of 391 has 30, one of 1 225 has 81: that many launches a block before. Where a backend's tiles do not take the operands (OpenCL: OKLM_CL_ATT_BLOCKS=0, fewer than 16 rows; Vulkan: fewer rows than half a block of the tiles; no published file's shapes) the op returns OKLM_ERR_UNSUPPORTED and the block runs the attention op once a window, on views of the window's rows in the queries, keys, values and output, as it did. On the CPU the pass is the host loop it was. The launches were a small part of the time: of the 0.455 s of kernel time of the test image's prompt on OpenCL (OKLM_CL_PROFILE=1, 2026-11-14) the windowed blocks' attention was 0.029 s in 840 launches and is 0.006 s in 28; Vulkan recorded its dispatches in one pass already and its times did not move. Qwen2.5-VL's encoder is faster than Qwen2-VL's, whose 32 blocks attend over every row.
  • The MLP. Qwen2-VL's quick-GELU is three element-wise ops, as CLIP's; Qwen2.5-VL's gate is a product, SiLU and a multiplication, and its inside width of 3 420 is padded to 3 424 for the second product (the padded values stay zero).
  • Memory. The blocks' weights (most of the file's 1.3 GB) and a buffer of activations for a picture of 1 280 embeddings (5 120 rows: 0.49 GiB for Qwen2-VL with the patch embedding and the merger on the device, a little less for Qwen2.5-VL, the rows' positions and spans included), which oklm_vision_device_bytes() reports for what shares the device: 1.74 GiB and 1.71 GiB with the weights. A larger picture, up to the 4 096 embeddings of the default, gets a larger buffer when it comes: 0.99 GiB of activations at 2 552 embeddings and 1.55 GiB at 4 015 for Qwen2-VL, the device giving up 2.26 GiB and 2.62 to 2.80 GiB in all for the encoder (OpenCL's count of its free memory on the RX 7900 XT, two runs; 2.14 GiB and 2.65 GiB for Qwen2.5-VL). oklm_vision_device_bytes_at() counts the buffer of a picture of a given size, and a server reserves that of the largest picture it accepts before it sizes its cache (three quarters of a sequence's context: 3 072 embeddings at -c 4096, 2.43 GiB with the weights where 1.74 GiB were kept; server.md). oklm describe reserves nothing: it has no cache to size. When the device has no room for it, that picture is encoded on the CPU, with a notice ("vision: the device has no room for the 1010 MiB of activations of a picture of 10208 patches, the encoder computes it on the CPU": 140 s then for the 2B's prompt, on a busy machine, where the device takes 5 s), and the next picture asks the device again. OKLM_MM_DEVICE_ACT_BYTES=N bounds the buffer for a test of that path (test_vision: the picture's embeddings are then the CPU's to the bit, the next one's the device's).
  • Who uses it. oklm describe, oklm serve --mmproj and the captioner of oklm rag index --caption-model (with --gen-backend, rag.md) put an image encoder on the device their language model's layers are on.

Measured on 2026-11-14 (RX 7900 XT, the language model's layers on the device in each case): the prompt of oklm describe on llama.cpp's test image (391 embeddings, a prompt of 418 rows), before (the encoder on the CPU) and after; and the encoder alone on the tests' picture of 273 embeddings (test_mm_public, a second image, the kernels built):

Model Prompt, encoder on the CPU: OpenCL, Vulkan Prompt, blocks on the device: OpenCL, Vulkan One image: CPU OpenCL Vulkan
Qwen2-VL 2B 8.32 s, 8.57 s 0.48 s, 0.43 to 0.45 s 4.84 to 5.67 s 0.26 s 0.23 to 0.24 s
Qwen2.5-VL 7B 7.20 s, 8.71 s 0.58 to 0.59 s, 0.55 to 0.56 s 4.43 to 4.73 s 0.23 s 0.20 s

Vulkan's columns are with the encoder's products at F32's precision (above: 0.37 s and 0.48 s for the prompts, 0.18 s and 0.15 s for the encoders with the activations rounded to F16, OKLM_VK_WIDE=0); the Vulkan figures of the next two paragraphs were measured with that rounding, and again without it at the day's end (the tables measured again: a picture of 1 225 embeddings in 1.67 to 1.68 s for Qwen2-VL 2B and 1.22 s for Qwen2.5-VL 7B on Vulkan, 1.60 s and 1.35 to 1.38 s on OpenCL). Qwen2.5-VL's row is with its windowed blocks in one launch each; with a launch a window, on the same day and build, its prompt took 0.61 to 0.63 s on OpenCL and 0.48 to 0.49 s on Vulkan, and its encoder alone 0.25 s and 0.14 to 0.15 s. Its prompt on a picture of 2 000 x 2 000 pixels (1 225 embeddings, 81 windows a block) went from 1.80 to 1.83 s to 1.72 to 1.76 s on OpenCL and stayed at 1.40 to 1.42 s on Vulkan.

A picture of 2 000 x 2 000 pixels (1 225 embeddings at the default largest size, 4 900 rows) takes Qwen2-VL 2B's prompt from 43.4 s to 1.79 s on Vulkan and to 1.89 to 1.91 s on OpenCL (6.66 s on its kernel of one work-group per head and token, 2.06 s on the first form of its bidirectional tiles: opencl.md; Vulkan 1.78 to 1.96 s in the session of the last figure, 0.72 of OpenCL's 1.48 s of kernel time the encoder's attention and 0.42 s its products). Through oklm serve a request that is decoding waits 0.18 s (Vulkan) and 0.30 s (OpenCL) for the encoder of the test image (server.md).

With the blocks on OpenCL the embeddings are within 5.0e-5 to 5.3e-5 (relative norm) of the CPU's for Qwen2-VL 2B and 1.7e-5 for Qwen2.5-VL 7B (1.3e-5 to 1.7e-5 with a launch a window); on Vulkan within 1.9e-5 to 3.0e-5 and 9.6e-6 to 1.1e-5 (above: 1.1e-2 to 1.5e-2 and 4.8e-3 to 6.3e-3 while its matrix units took the encoder's activations rounded to F16. Qwen2-VL's activations are large: its embeddings reach 80; llama.cpp's own are 2.0e-2 to 2.4e-2 from the CPU's for that model). The tiny files give the CPU's embeddings within 5.4e-7 on OpenCL and 6.0e-7 on Vulkan (test_vision). The answer about the newspaper is the one the CPU's embeddings give, the language model on the same device both times: 48 (OpenCL) and 47 (Vulkan) of 48 greedy choices for Qwen2-VL 2B (parting where the two tokens were 0.24 apart), 47 and 47 of 48 for Qwen2.5-VL 7B (0.19 and 0.11 apart; 47 and 48 with the rounded activations, 48 and 47 with a launch a window: the near ties fall as the sums' order does; test_qwen2vl_encoder_devices, test_qwen25vl_encoder_devices). oklm describe writes the same 48 tokens before and after for the 2B on both backends; the 7B's answers part at one near tie ("and" against "and below it,"), as its two backends already did with the encoder on the CPU. Qwen2-VL 2B's wording moves at such ties with whatever changes a sum's order: of five pictures asked about at once through oklm serve, beside transcriptions, streams and speech, one (Vulkan) and three (OpenCL) answers were oklm describe's to the token, the others a reading of the same picture in other words.

Audio

The spectrogram is whisper.cpp's: a reflected half window at the start, 30 seconds of silence at the end, a 400-sample periodic Hann window every 160 samples, a Slaney mel filterbank from 0 to 8 kHz (area normalized), log10 clamped 8 below the maximum and scaled to (x + 4) / 4, cut into 30-second chunks of 3000 frames.

The Whisper encoder: two convolutions (as products over unfolded frames), GELU, positions, pre-norm blocks with bidirectional attention, a final norm.

Its products are an encoder's: 1500 positions (3000 for the first convolution) against float matrices, nearly all the time of a window with the large checkpoints. They go through the blocked kernel (oklm_matmul_dense(), core.md). The decoder's products are one token against whole matrices, so they stay oklm_matmul()'s. The spectrogram's frames are computed on the worker team, and a frame of zeros (the 30 seconds of padding) without its transform.

  • oklm_whisper loads a Hugging Face Whisper directory (config.json, model.safetensors, tokenizer.json): speech recognition below.
  • oklm_audio loads an mmproj GGUF with an audio encoder. The ultravox projector stacks 8 frames, applies an RMS norm, a SwiGLU MLP and another RMS norm. The last group of frames is padded, as the original model does (llama.cpp drops it: 188 embeddings for 30 seconds against 187).

The audio encoder on a device

With the language model's layers on a device (--backend opencl|vulkan, --gpu-layers), oklm describe and oklm serve --mmproj put the blocks of the Ultravox encoder there too (oklm_audio_use_backend()), through the path of Whisper's encoder (Whisper on a device): the same blocks, the same ops, no kernel of its own. The blocks' weights and the final norm's are copied to the device; each 30-second chunk's stream is uploaded, its blocks and the final norm are one pass there (oklm_backend_pass_begin()), and their output is read back for the projector. The spectrogram, the two convolutions before the blocks and the projector (two RMS norms, the SwiGLU MLP, a linear layer over 188 rows) stayed on the host; since 2026-11-14 the convolutions and the projector compute on the device too (the convolutions on a device), and what is read back is the projector's 188 rows. The figures below are from before; with the whole encoder on the device and Vulkan's products at F32's precision the prompt of oklm describe on jfk.wav (228 rows) takes 0.33 s on OpenCL and 0.26 s on Vulkan against 5.35 s on the CPU (2026-11-15, three runs each on the devices, the same figure each time), and the encoder alone on a second sound 0.295 s and 0.228 s against 4.68 s (test_mm_public), its embeddings 2.4e-4 and 7.2e-5 from the CPU's.

Measured on 2026-11-14 (RX 7900 XT, Ultravox v0.5 with Llama-3.2-1B Q8_0 and its F16 projector, the 32 blocks of Whisper large-v3-turbo's encoder; jfk.wav, 188 embeddings; the language model on the device in each case, the encoder on the CPU before; another job was using the CPU):

OpenCL Vulkan
the prompt of oklm describe (228 rows), before -> after 5.77 s -> 0.41 s 5.32 s -> 0.26 s (0.33 s since the encoder's products keep F32's precision)
a chat request of oklm serve with a new sound, 48 tokens, before -> after 5.80 to 5.88 s -> 0.57 to 0.60 s 5.65 to 5.79 s -> 0.42 to 0.43 s
the encoder alone, a second sound (test_mm_public; the CPU 4.8 to 5.9 s) 0.36 to 0.38 s 0.23 s (0.30 s since)
the embeddings against the CPU's, relative (largest difference) 3.1e-4 (3.1e-3) 9.6e-3 (8.1e-2); 5.5e-5 (8.1e-4) since

The answers are the ones the CPU's embeddings give: the 48 tokens of oklm describe and of three chat requests through the server (the recording, its first 8 and its first 5 seconds) are the same text before and after on each backend. Vulkan's 9.6e-3 was the rounding of its matrix units, which took the encoder's activations as F16, as for the vision encoders and as llama.cpp's own (the CPU's embeddings are 1.1e-2 from llama.cpp's); the encoder's products keep F32's precision there since 2026-11-14, for 0.07 s a sound (the encoders' precision on Vulkan; a chat request with a new sound and 8 tokens takes 0.28 s on Vulkan and 0.36 s on OpenCL alone since, 0.40 s and 0.50 s among a stream: server.md). OpenCL's is larger than a vision encoder's (1e-5): it comes with the tiled products (gm_f16; 3.1e-5 with OKLM_CL_GEMM_TILES=0), through 32 blocks. It is rounding, not a wrong term: each output of a tile is one running sum over the 1 280 or 5 120 terms of a product, and the same kernel with a partial sum every 32 terms gives 4.5e-5, every 128 terms 1.1e-5, for 7 % more time in the products (opencl.md); the single sum is kept. On the tiny file of the tests (test_audio_enc_devices in tests/mm/test_audio.c: a random encoder of 2 blocks, 32 values wide, two chunks) the embeddings are 1.9e-6 (OpenCL) and 1.7e-6 (Vulkan) from the CPU's on the RX 7900 XT, 2.0e-6 and 1.8e-6 on the Intel UHD 750.

Limits: the device takes a copy of the blocks' weights (about the file's 1.4 GB) and 0.1 GB of activations (oklm_audio_device_bytes()); a device without that room leaves the encoder on the CPU with a notice (the audio encoder runs on the CPU), and a file with quantized blocks stays there without one, as a vision encoder's does. A sound longer than 30 seconds is one pass a chunk. The spectrogram is host work (24 ms for jfk.wav). No published file was run on the UHD 750.

Speech recognition

oklm_whisper (include/oklm/mm/audio.h) loads a Hugging Face Whisper directory and decodes greedily with cross-attention and a KV cache. Everything that differs between the checkpoints is read from the files: the mel bin count (80, or 128 for the large-v3 family, the filter bank built for either), the layer counts of the encoder and of the decoder (32 and 4 for large-v3-turbo), and the ids of the special tokens, looked up by name in the tokenizer: the languages are the tokens between the start token and the tasks' (large-v3 has one more, <|yue|>, which moves the tokens after it), the timestamps those after <|notimestamps|>. The weights are F32 (base, small) or F16 (large-v3, large-v3-turbo): the matrices of an F16 checkpoint, the token embeddings among them, are read where the file is mapped, and its biases, norms and positions are widened to F32 once at load.

Whisper on a device

oklm_whisper_use_backend(w, backend) (oklm transcribe --backend opencl|vulkan) copies the weights of the encoder's blocks, of the decoder and the output matrix to a device, and a window is then computed there through the same ops as on the CPU: the blocks of the encoder (bidirectional attention over the 1 500 positions), its final norm, the keys and values of the decoder's cross-attention, and for each token the decoder's blocks (causal attention over a cache of its own in the device's memory, the keys and values written where the cache keeps them, cross-attention over the encoder's output) and the product with the token embeddings. The host keeps the spectrogram and the choice of each token with its timestamp rules: the first convolution's windows are uploaded once a window, the logits are read back once a token. The two convolutions before the blocks were host work too (0.08 to 0.11 s a full window) until 2026-11-14 (the convolutions on a device); the table below is with them there, the paragraph after it from before.

Where a 30-second window's time goes (RX 7900 XT, measured again on 2026-11-15 with the convolutions on the device and Vulkan's products at F32's precision, load average 3.7 to 5.6; one window of 69 tokens, --no-timestamps, two runs on each device and one on the CPU, seconds; the encoder is waited for before the decoder starts; the spectrogram, 0.045 s, is in the window and in neither part):

Model CPU OpenCL Vulkan
large-v3-turbo (32 encoder blocks, 4 decoder blocks) encoder 4.93 0.27 0.20 to 0.21
decoder 1.10 0.13 to 0.14 0.11
the window 6.07 0.45 0.36
large-v3 (32 and 32) encoder 5.09 0.27 0.20 to 0.21
decoder 6.25 0.60 0.46
the window 11.4 0.92 0.71
small the window (58 to 66 tokens) 2.21 0.29 0.24
base the window 0.55 0.17 0.14

jfk.wav (11 seconds, with timestamps) in the same session: 0.27 s on Vulkan and 0.35 s on OpenCL with large-v3-turbo against 5.19 s on the CPU, 0.44 s and 0.59 s with large-v3 (7.86 s), 0.13 s and 0.16 s with small (1.33 s), 0.08 s and 0.09 s with base (0.37 s). The day's first table, before the convolutions went to the device, had the encoder at 0.34 to 0.36 s on OpenCL (timed with every kernel waited for) and 0.17 s on Vulkan (its activations rounded to F16), the windows at 0.47 to 0.49 s and 0.33 s for large-v3-turbo, 0.97 s and 0.66 s for large-v3.

On the CPU the encoder is four fifths of large-v3-turbo's window and the decoder more than half of large-v3's: both are on the device. OpenCL's encoder and windows are with its bidirectional attention on tiles of its own (opencl.md); on the kernel of one work-group per head and token the encoder took 0.66 s (that kernel 69 % of a run's kernel time) and the windows 0.83 and 1.32 s. Its encoder is timed with every kernel waited for (OKLM_CL_PROFILE=1), its windows without. Of the kernel time of jfk.wav's window (0.28 s) the encoder's attention is 0.085 s and the matrix products (gm_f16) 0.14 s: these are F32 products where Vulkan's matrix units take F16, and what is left of the distance between the two (jfk.wav transcribed in 0.39 to 0.40 s on OpenCL, 0.30 to 0.32 s on Vulkan, 0.35 to 0.36 s on OpenCL with OKLM_CL_WMMA=1, its products then 0.035 s). Vulkan's column and these figures are with the encoder's activations rounded to F16 in the matrix units; since 2026-11-14 its products keep F32's precision (the encoders' precision on Vulkan), as do the keys and values of the cross-attention, and jfk.wav goes from 0.24 to 0.25 s to 0.31 to 0.32 s with large-v3-turbo and from 0.41 to 0.49 s with large-v3 (one session; OKLM_VK_WIDE=0 for the first figure of each); the table above is with them. Whisper small took 3.14, 0.43 and 0.35 s for the same 30 seconds decoded with timestamps (before the convolutions went to the device).

The text is the CPU's on the samples the tests use: the tiny checkpoints' texts, segments, times and languages (test_whisper_devices in tests/mm/test_audio.c, on the RX 7900 XT and the Intel UHD 750, both backends), and jfk.wav with base, small, large-v3-turbo and large-v3 (tests/mm/test_mm_public.c). A device's products sum in another order than the CPU's, so a token whose two best candidates are all but tied may differ on another recording; none did here.

A server's recognizer takes a device with --transcribe-backend (cpu by default): a window's encoder and each token of the decoder are passes on the device (oklm_backend_pass_begin()) that take their turn with the scheduler's steps, the encoder's between two of its blocks: the streams wait 0.015 s (Vulkan) or 0.02 s (OpenCL) for a block of large-v3-turbo's encoder, where they waited 0.20 s and 0.28 s for its pass (server.md).

Limits: widths that are not a multiple of 32 values (the tiny checkpoints' 16) have their products handed back to the CPU, the rest of the pass staying on the device; the device takes a copy of the blocks' weights (about the file's size: 1.6 GB for large-v3-turbo, 3.1 GB for large-v3) and 0.2 to 0.8 GB of activations, and a device without that room leaves the recognizer on the CPU with a notice; on the Intel UHD 750 Whisper base takes 7.4 s (OpenCL) and 2.7 s (Vulkan) for jfk.wav where the CPU takes 0.6 s.

The decoder multiplies one token by each of its matrices and by the token embeddings (the logits): products bound by the memory they read, about 550 MiB a token for Whisper small in F32. OpenAI trained and published its checkpoints in half precision, and the F32 files of base and small hold nothing but half-precision values (every one of their 245 and 479 tensors converts to F16 and back unchanged). At load, a decoder matrix stored in F32 whose values are all halves gets an F16 copy, and the products read that: the same values (the dot product widens the halves back to the same floats) from half the memory. The decoder of small then takes 1.7 to 2.2 s for the 66 tokens of a window instead of 2.5 to 3.2 s, that of base 0.5 to 0.7 s for 58 tokens instead of 0.9 to 1.5 s (eight threads, four alternations on a loaded machine). A matrix with one value that is not a half's keeps the file's floats.

The language. The logits after the start token, taken over the language tokens, are the model's opinion of the language spoken: oklm_whisper_detect_language() returns the most probable one and its probability among them, and a recognition without a language (--language auto, the default) uses the one of its first window for the whole recording. oklm_whisper_n_languages() and oklm_whisper_language() list the codes a model knows (99 for base and small, 100 for the large-v3 family), oklm_whisper_describe() reads them from a directory's tokenizer without loading the weights. Translation is the other task token: <|translate|> in place of <|transcribe|> makes the model write English (--translate).

Timestamps. oklm_whisper_run() decodes with Whisper's timestamp tokens (<|0.00|> to <|30.00|>, one every 20 ms) under the rules of the reference implementation (openai/whisper's ApplyTimestampRules):

  • a window opens with a timestamp, of at most one second;
  • timestamps come in pairs, the end of a segment and the start of the next, except a single one right before the end of text: after one timestamp that follows text, no text; after two, no timestamp;
  • timestamps never decrease, and a segment ends after it starts;
  • when the timestamps together are more probable than the most probable other token, the next token is a timestamp.

The tokens of a window are then cut at each pair of timestamps into segments with a start, an end and a text (oklm_whisper_segment), and the next window starts where the last closed segment ended, not 30 seconds further: the model closes its segments at pauses, so the cuts fall in silences and the sentence a window's end interrupts is decoded again, whole, by the next. A window that ends on a single timestamp (the speech stopped inside it), or holds no pair, is followed by the next 30 seconds. A window whose no-speech probability (the <|nospeech|> token after the start token) is above 0.6 and whose tokens have a mean log-probability under -1 writes nothing.

oklm_whisper_transcribe() and --no-timestamps are the plain form: no timestamp tokens, a window every 30 seconds wherever that falls (a recording of exactly 30 seconds is one window: nothing is decoded from the padding after it), so the sentence spoken across a cut can lose words. The chat window's dictation uses it for its short recordings.

What it does not do: beam search and the temperature fallback (the decoding is greedy: a window the reference would decode again at a higher temperature keeps its first text), a prompt from the text before the window (so a name is not spelled the same way from one window to the next by construction, and a first word that starts exactly at a window's start can be missed), word-level times, and a voice-activity detector before the model.

Speed. Seconds for one window: a recording of exactly 30 seconds, decoded without timestamps so that it is one window (58 to 69 tokens), the model's loading left out. Eight threads on an i7-11700K (8 cores, 16 threads) that five other jobs shared, so each figure is three alternations of the two binaries and the load average is said:

Model before after load
base 1.45 to 1.97 0.78 to 1.42 9 to 11
small 4.61 to 7.17 2.48 to 3.98 9 to 11
large-v3-turbo 11.48 to 17.45 7.78 to 11.94 10 to 11
large-v3 17.92 to 21.19 13.58 to 15.71 10 to 11

"Before" is the encoder's products computed one dot product at a time and the decoder reading F32 matrices; "after" the blocked product, the spectrogram on the worker team, the decoder's query, key and value as one loop and its matrices read as halves. Under a heavier load (16 to 19) the same alternation gave 20.9 to 39.2 s then 14.6 to 30.4 s for large-v3-turbo, and 55.7 to 87.2 s then 24.1 to 40.0 s for large-v3 (and once 151 s, for a cause not found: the page cache was full): what else the machine does counts more than anything here. On one thread large-v3 took 132.6 s of CPU before and 73.6 s after (load 8 to 10).

Where a window's time goes now (eight threads, load 10): for large-v3-turbo 11.4 s of 13.4 in the encoder (8 s its products, 2.7 s its attention) and 2 s in the decoder; for large-v3 13.5 s in the encoder and 12 s in the decoder, whose 32 layers read 1.5 GiB of matrices for each of the 69 tokens: that part is bound by memory, and only smaller (quantized) matrices would shorten it.

With timestamps a window starts before the end of the one it follows, so a recording can take more windows than its length in half minutes: the 127-second address takes five either way (small's start at 0, 29.1, 58.3, 85.6 and 114.7 s instead of every 30 s) and 7.7 s with base, 24 s with small (18 s without timestamps: fewer tokens) and 47 s with large-v3-turbo (44 s); a recording of exactly 30 seconds whose last sentence the window cuts takes two.

The convolutions on a device

With the blocks of Whisper's encoder on a device, the host still computed what stands before them, and for Ultravox what stands after. Timed on 2026-11-14 on jfk.wav (11 seconds: the frames after the sound are silence; RX 7900 XT, Vulkan, large-v3-turbo's encoder, milliseconds): the spectrogram 24.0; the first convolution 6.4 with 0.6 for its windows, its GELU 1.8; the second convolution 23.7 with 2.7 for its windows, its GELU 1.4; then the blocks, 200. For Ultravox the projector after them, 28.0. So 37 ms of convolutions and 28 ms of projector stood beside 0.20 s of blocks.

  • The first convolution (kernel 3, stride 1) is, as on the CPU, a product over the windows the host unfolds from the spectrogram (3 x 128 or 3 x 80 values a frame, widened with zeros to a multiple of 32: 80 bins give 240, and the device's copy of the kernel has rows of 256). The windows are uploaded in place of the stream.
  • The second (kernel 3, stride 2) reads the first one's output, which is on the device. It is not unfolded there: the kernel is copied as three matrices, one a position, and the convolution is three products, each over a view of every second row from that position on, the first written and the two others added to it (OKLM_OP_MATMUL accumulating). The first convolution writes its rows after one row of zeros, which is the padding the first position reads. The products are the blocks' (oklm_matmul_wide()), the GELUs and the positions are ops: no kernel and no op was added.
  • Ultravox's projector reads the encoder's output where it is: the 1 500 rows and four rows of zeros after them as 188 rows of eight frames (a reshape), the RMS norm, the first product, the SwiGLU of its two halves (views), the second norm and product. The 188 rows are read back, where the 1 500 of the output were.

src/mm/audio.c: wenc_convs_place() copies the kernels, wenc_device_convs() computes a chunk, ultravox_project_device() the projector. OKLM_MM_DEVICE_ENDS=0 keeps the three in host memory (build.md); oklm_whisper_device_parts() and oklm_audio_device_parts() tell which it is.

Measured on 2026-11-14 (RX 7900 XT, one build, the convolutions and the projector in host memory -> on the device, three runs each):

OpenCL Vulkan
oklm transcribe of jfk.wav, large-v3-turbo 0.38 to 0.40 s -> 0.35 s 0.31 s -> 0.27 s
large-v3 0.63 s -> 0.59 s 0.48 to 0.49 s -> 0.44 to 0.45 s
base 0.10 s -> 0.09 s 0.08 to 0.09 s -> 0.08 s
the prompt of oklm describe with Ultravox, jfk.wav 0.40 to 0.41 s -> 0.33 s 0.33 s -> 0.26 s
Ultravox's encoder alone, a second sound (test_mm_public) 0.363 s -> 0.298 s 0.298 s -> 0.230 s
its embeddings against the CPU's, relative 3.1e-4 -> 2.4e-4 5.5e-5 -> 7.2e-5

The texts of jfk.wav are the same before and after, and the CPU's, with the four checkpoints on both backends (test_mm_public), and Ultravox starts its answer with the same words. The tiny checkpoints' texts, segments, times and languages are the reference's both ways on both backends, and the tiny Ultravox encoder's embeddings are within 1.9e-6 of the CPU's both ways (test_whisper_devices and test_audio_enc_devices in tests/mm/test_audio.c; on the Intel UHD 750 too, within 2.0e-6). The second convolution's form (accumulated products over views of every second row) is checked as an op against a convolution in doubles, with Whisper's 1 280 channels among the shapes (test_conv_views, both backends, the RX 7900 XT and the Intel UHD 750: within 1.8e-7 of the sum of the terms' magnitudes).

Limits. The spectrogram stays on the host: 24 ms of jfk.wav's 0.27 s on Vulkan, in doubles on the worker team, a transform of 400 points a frame that has sound. A transform on a device was not written: it would be a product by a table of cosines at F32's precision, and a logarithm, which no backend has as an op. The device holds the two kernels (the second as its three matrices), the biases and the positions beside the blocks; a device without that room keeps the convolutions in host memory with a notice, the blocks staying on it, and likewise for Ultravox's projector (oklm_audio_device_bytes() counts both). The room is asked for in this order: the blocks' weights, the projector's, the activations, the convolutions' weights. Where the activations do not fit, the buffer is asked again without the first convolution's windows (the larger part for a narrow encoder, the blocks' for the published ones), then with the projector given back: a device that holds what OKLM_MM_DEVICE_ENDS=0 asks for holds the blocks, whatever the rest takes. test_whisper_small_device and test_audio_enc_devices run the tiny recognizer and the tiny Ultravox encoder on a device with that room to the byte (the tests' counting device with a bound, computing with the CPU's kernels, tests/mm/count_backend.h): the notices, the bytes held and the result of OKLM_MM_DEVICE_ENDS=0; with room for all but the convolutions, the projector on the device; with a byte less than the blocks need, the refusal and the CPU's result. No published file was run on the UHD 750.

PDF

src/mm/pdf.c, with nothing but zlib (nilorea's copy):

  • Objects are found by scanning the file for n g obj, a later definition replacing an earlier one (incremental updates), so a damaged or missing cross-reference table does not matter; object streams are unpacked.
  • The page tree is walked in order with inherited resources (every page object in file order when there is no tree). Each page's content runs through an interpreter of the text, font and graphics state operators; form XObjects are entered, inline images skipped.
  • Strings are decoded by the font's ToUnicode CMap (bfchar, bfrange and its array form, surrogate pairs), else its simple encoding: StandardEncoding, WinAnsi or MacRoman, the encoding built into an embedded Type 1 program, then /Differences glyph names (the Adobe Glyph List's Latin, Greek and typographic names, uniXXXX). Composite fonts with the Identity CMap need a ToUnicode map; UCS-2 CMaps are read directly. Ligatures are spelled out.
  • A space goes where the gap to the previous glyph passes 0.15 em and 0.4 of the font's own space (letter-spaced text stays one word); a line break where the baseline moves half a line, a blank line past two.
  • Streams: Flate, LZW, ASCII85, ASCIIHex, RunLength, PNG and TIFF predictors.
  • Encrypted documents open when their user password is empty, the usual case of a document encrypted only to restrict printing or editing: the standard security handler's revisions 2 to 6, RC4 40 to 128 bits, AES-128, AES-256 (src/mm/pdf_crypt.c: MD5, RC4, AES, SHA-384/512 of its own, nilorea's SHA-256). Others are refused (OKLM_ERR_UNSUPPORTED).
  • Bounded for untrusted files: nesting 64, 64 MB per decoded stream, 1 GB over a document, 100 million operators, 100 000 pages, forms 8 deep.
  • Not read: scanned pages (images, no text without OCR), Type 3 bitmap fonts without a ToUnicode map, CFF fonts' built-in encodings, reading order beyond the content's own (columns and tables come as the content draws them).

Against poppler's pdftotext (a test-only reference), tools/check_pdf.sh compares the words of both texts on every PDF it finds: on the 618 PDFs of this machine's /usr/share and /opt (TeX Live's documentation, mostly; 4.1 million words), 96.5 % of pdftotext's words are found and 97.8 % of OKLM's are in pdftotext's text, weighted by words; most of the rest is order (pdftotext reorders columns and tables), math set in TeX's own glyph names, Type 3 bitmap fonts where neither reader gets text, and code pdftotext splits into letters where OKLM keeps the words. The largest document (222 000 words) takes 0.3 s.

Verification

tests/mm/test_ingest.c checks the documents (a DOCX written by the test included). tests/mm/test_pdf.c reads the PDF fixtures of tools/gen_pdf_fixtures.py (each feature above, a damaged table, an incremental update), the encrypted copies qpdf made of one of them at every revision against its plain text, refuses one with a user password, checks the hashes and AES against published vectors, and feeds damaged and hostile inputs: every truncation of a fixture, byte flips, nesting past the limit, a 200 MB decompression bomb. fuzz/fuzz_pdf.c fuzzes the reader. tests/mm/test_video.c decodes a 2-second Theora fixture (frames, times, sampling, the 440 Hz tone) and, with ffmpeg installed, the same stream remuxed into Matroska through ffmpeg (mean difference 0.55 of 255 against the native frames); the same file held in memory gives the file's frames byte for byte, and the limit on decoded pixels refuses the frames asked for one pixel under their size and passes them at it, on both paths. tests/mm/test_vision.c and tests/mm/test_audio.c check every piece on small random encoders written by the tests against double-precision references, image operations, WAV formats, resampling and the spectrogram. tests/mm/test_vision.c also loads the tiny projector files of tools/gen_vision_fixtures.py (random weights in the layout of a published file) and requires the embeddings its numpy float64 implementation gives, and the bytes Pillow gives for the projector's resize of a test picture; for the two Qwen layouts, the embeddings of a wide and of a tall image (windows cut at the edges for 2.5) and the picture at the size the model's own rule gives it; and a video of three frames (the first two through the two kernels, the last one twice: within 6e-7 of the reference on values up to 2.1), the size of a video's frame, and the video in a prompt: the rows the processor writes placeholders for, between the two vision texts, the grid and time step on the item, one frame the rows of that frame as an image. tests/model/test_model.c runs a tiny qwen2vl model on the cases of tools/gen_mrope_fixture.py (a numpy pass with three positions a row: tokens, wide and tall images, two images, an image at either end, a column of one embedding a row, and videos: Qwen2-VL's step of one a grid, Qwen2.5-VL's of two, four, one and a half and a half, a video then an image; the layout of each is asserted there against transformers' get_rope_index() written again in numpy, as are 98 videos of the published models' grid of 23 x 17 at eight frame rates) seven ways each: one call an image, pieces of 4 rows, a batch with another sequence's tokens on a contiguous and on a paged cache, after a cut in the middle of an image, on a copy of the sequence, and from rows found in the prefix cache (a cut inside an image for the longer cases): within 3e-5 of the reference's logits, the next position the reference's; a text gives bit for bit the logits of the same tensors as qwen2. tests/core/test_attention.c, tests/model/test_model.c (cases of tools/gen_mutual_fixture.py, a numpy pass of the gemma3 architecture with a window of 4: within 8e-7 on a contiguous cache, in a batch and on a paged cache) and tests/server/test_server.c (the server's answer is oklm_mm_eval()'s, and not the one the same rows give read one after the other) check the rows that see each other. tests/mm/test_mm_public.c (part of make test-public) compares with llama.cpp, whose outputs tools/gen_mm_ref.sh records with tools/mtmd_ref.cpp from the very inputs OKLM computes. A whole answer is asked about the picture kept beside its recording (build/ref/mm/<tag>_answer.ppm), so a tree that holds the recordings without the llama.cpp build compares the same pixels:

Check Result
SmolVLM-256M, overview tile of llama.cpp's test image max difference 0.1 on values up to 109
Gemma 3 4B, two pictures (one drawn by the test, 600 x 360; llama.cpp's test image, 640 x 488): the 896 x 896 pixels after the resize llama.cpp's, byte for byte
Gemma 3 4B, the 256 embeddings of each, against a float64 pass of the encoder (tools/vision_ref.py) relative difference 8.9e-6 and 1.6e-5 (max 0.0013 on values up to 33)
the same against llama.cpp relative difference 3.2e-3 and 6.9e-3 (max 0.22 and 0.77): llama.cpp is as far from the float64 pass, 3.2e-3 and 6.9e-3
LLaVA 1.5 7B, the same two pictures: the 336 x 336 pixels after the resize llama.cpp's, byte for byte
LLaVA 1.5 7B, the 576 embeddings of each, against the float64 pass of the model's definition relative difference 5.4e-6 and 6.9e-6 (max 0.00033 on values up to 38)
LLaVA 1.5 7B Q4_K, "Describe the picture in detail." about llama.cpp's test image, llama.cpp's language model given OKLM's embeddings the prompt's 595 rows are llama.cpp's (its own tokens around 576 image rows); on its 48 greedy tokens, 47 choices equal, logits within 0.38; they part at the fifth token, where llama.cpp's two best are 0.007 apart
LLaVA 1.6 Mistral 7B, the same two pictures: the tiles of each (2 x 2 and the overview), against the model's own processor written again with Pillow (tools/llava_next_ref.py) the reference's bytes, normalized: every pixel equal
LLaVA 1.6, the 576 embeddings of each of the ten tiles, against the float64 pass of the model's definition on the reference's pixels relative difference 2.1e-6 to 7.6e-6 (max 0.0002 on values up to 37)
LLaVA 1.6, the rows of each image (1 948 and 2 340), against the model's own code written again with numpy, given the same ten tiles' embeddings the same rows: no value differs
LLaVA 1.6 Mistral 7B Q4_K_M, "What does the newspaper say?" about llama.cpp's test image, llama.cpp's language model given OKLM's rows 2 357 rows (2 340 of the image); on its 48 greedy tokens, 48 choices equal, logits within 0.32
Gemma 3 4B Q4_K_M, "What does the newspaper say?" about llama.cpp's test image, llama.cpp's language model given OKLM's embeddings the prompt's 273 rows are llama.cpp's (256 of the image); on llama.cpp's 48 greedy tokens, 47 choices equal, the first 12 in a row, logits within 3.78; where they part llama.cpp's two best are 0.84 apart
the same against llama.cpp's whole path (its own encoder) 44 of 48 choices equal, the first 10 in a row, logits within 3.85; where they part llama.cpp's two best are 0.21 apart, 1.11 at the widest of the four choices that differ. llama.cpp given OKLM's embeddings leaves its own answer at that eleventh token too. With a new line after the image, the prompt until 2026-11-14 (274 rows), the whole path gave 47 of 48, the first 23 in a row, logits within 3.64, 0.32 apart
Qwen2-VL 2B and Qwen2.5-VL 7B, two pictures (the one drawn by the test, 600 x 360, and the same turned a quarter, 360 x 600): the 588 x 364 and 364 x 588 pixels after the resize llama.cpp's, byte for byte
Qwen2-VL 2B, the 273 embeddings of each (21 x 13, 13 x 21), against the float64 pass of the model's definition (tools/vision_ref.py) relative difference 1.3e-5 and 1.3e-5 (max 0.0012 on values up to 82)
the same against llama.cpp 2.4e-2 and 2.0e-2 (max 4.7): llama.cpp is as far from the float64 pass
Qwen2.5-VL 7B, the 273 embeddings of each, against the float64 pass relative difference 2.9e-6 and 2.4e-6 (max 1.3e-4 on values up to 17)
the same against llama.cpp 3.0e-3 and 1.0e-2 (max 0.75): llama.cpp is as far from the float64 pass
Qwen2-VL 2B Q4_K_M, a question without a picture (architecture qwen2vl), llama.cpp's 48 greedy tokens fed one by one the prompt's 38 tokens are llama.cpp's; 48 choices equal, logits within 0.79
Qwen2-VL 2B, "What does the newspaper say?" about llama.cpp's test image, llama.cpp's language model given OKLM's embeddings at mtmd's positions the prompt's 418 rows are llama.cpp's (391 of the image, a grid of 23 x 17); 46 of 48 choices equal, the first 26 in a row, logits within 1.76; where they part llama.cpp's two best are 0.09 apart
the same against llama.cpp's whole path (its own encoder) 45 of 48 choices equal, the first 26 in a row, logits within 2.34; where they part llama.cpp's two best are 0.24 apart
Qwen2.5-VL 7B Q4_K_M, a question without a picture 47 of 48 choices equal, logits within 2.24; where they part llama.cpp's two best are 0.23 apart
Qwen2.5-VL 7B, the newspaper question, llama.cpp's language model given OKLM's embeddings 418 rows (391 of the image); 46 of 48 choices equal, logits within 3.16; they part at the third token, where llama.cpp's two best are 0.40 apart
the same against llama.cpp's whole path 46 of 48 choices equal, logits within 3.45; llama.cpp's two best 0.26 apart where they part
jfk.wav spectrogram (128 bins) max difference 2.6e-5
Ultravox on jfk.wav (Whisper large-v3-turbo encoder) relative difference 0.011 over the 187 common rows (llama.cpp rounds activations to F16)
the same with the encoder's blocks on a device, against the CPU's relative difference 3.1e-4 on OpenCL, 5.5e-5 on Vulkan (the audio encoder on a device)
Whisper base, small, large-v3-turbo and large-v3 on jfk.wav the exact transcript of each (large-v3-turbo writes a comma after "And so")

Speech recognition has transformers as its reference, llama.cpp having no Whisper decoder: the same model.safetensors and the same 16 kHz WAV, greedy, no timestamps, each 30 seconds on its own. jfk.wav comes out word for word with the four checkpoints, and a 127-second radio address (five chunks) word for word with base (375 words) and small (365). With large-v3-turbo OKLM writes 357 of transformers' 364 words, and with large-v3 214 of 215 on the first 75 seconds: each time a few words of the sentence a 30-second cut falls in. The chunk in question transcribed alone gives transformers' text exactly; inside a recording OKLM's spectrogram is the whole recording's (its maximum sets the floor and the frames at a cut see the samples across it, as in whisper.cpp), where transformers computes each chunk's. tests/mm/test_audio.c also decodes a 35-second test signal with two tiny random checkpoints (the layout of base, and that of large-v3-turbo: 128 bins, F16, a shorter decoder, one more language token) and requires the text a double-precision numpy implementation decodes (tools/gen_whisper_fixtures.py), without timestamps and with them: the language found and its probability, each segment's times and text and so every window's start, for a given language and for the translation task. That reference writes the timestamp rules, the cutting and the seeking from openai/whisper's decoding.py and transcribe.py, on its own.

With timestamps there is no reference implementation on the machine to compare real recordings with; what was looked at instead, on the CPU:

  • The cuts. The same 127-second address, with timestamps (the windows start where the last closed segment ended) and without (a cut every 30 seconds), words counted:

    Model with timestamps a cut every 30 s what the cuts lost
    base 372 375 nothing: the three words differ both ways (one word missed with timestamps, "Voting", at a window's start)
    small 375 365 a sentence of 11 words
    large-v3-turbo 375 357 three fragments, 18 words
    large-v3 375 364 two fragments, 11 words

    The texts of large-v3 and large-v3-turbo with timestamps are the same word for word, and small's differs from them by two places ("and we" for "Americans", and a missing "In" at a window's start).

  • The language. French, German and Spanish readings and the English address, the first 30 seconds of each: the four models name the right language, with a probability of 0.985 to 1.000.

  • The times. A French fable of 52 seconds read with pauses: base, small and large-v3 close their segments at the pauses (large-v3: 0.88 to 5.56, 6.44 to 10.92, 11.98 to 17.90 ...), and the second window starts at one (25.16 s for base). The 11-second jfk.wav is one segment from 0.00 to 11.00 for base and small, three for large-v3.

  • Translation. The fable in English: large-v3 writes it ("Master crow on a perched tree held in his beak a cheese..."), small roughly, base only English words; large-v3-turbo writes French, as its authors say it does (it was trained without the task).

  • What goes wrong. Greedy decoding without the reference's fallbacks keeps a window's first text. Seen: large-v3-turbo closing a segment at the window's end (29.98 s) and so skipping the sentence spoken from 25 to 30 seconds of the fable; large-v3 adding "Ah!" after the sentence of jfk.wav when 20 seconds of silence surround it; and, with 20 seconds of silence before the speech, every model starting its first segment at 0.00 (a window must open within its first second), so that segment's start is the window's, not the speech's.

Gemma 3's encoder is where the two references part. llama.cpp gives F16 matrices their activations rounded to F16 (27 blocks here); a numpy pass that rounds them so moves 2.6e-3 from the float64 one, the distance llama.cpp is at. OKLM keeps them in F32 and stays within 2e-5 of the float64 pass, so the test holds it to that pass (1e-4) and to llama.cpp more loosely (2e-2).

The answer was read three ways on the same picture and question, greedy: oklm describe and oklm serve --mmproj (an image_url part and the question in a chat request) write the same 64 tokens ("Here's the text from the New York Times newspaper [...] Headline: MEN WALK ON MOON [...] ASTRONAUTS LAND ON PLAIN; COLLECT ROCKS, PLANT FLAG"), and llama.cpp's own run writes the same headline and subheading, its sentence parting from OKLM's at the token counted above. Not run: a second image in one prompt, a prompt long enough for the window of 1 024 positions to cut an image's rows (the tiny model's window of 4 does, on the CPU and on the devices).

On OpenCL and on Vulkan (2026-11-14, RX 7900 XT) the same 273 rows and 48 tokens, against the CPU's logits and against llama.cpp's answer from the same embeddings (test_mm_public; each layout on a model loaded for it):

Gemma 3 4B, layout Choices the CPU's, of 48 Logits from the CPU's: largest, mean Choices llama.cpp's Logits from llama.cpp's Prompt, rows/s Tokens/s
CPU 47 3.78 184 10.7
OpenCL, every layer, contiguous cache 46 3.67, 0.353 46 3.50 2 809 104
OpenCL, 17 of 34 layers, paged cache 46 3.18, 0.298 46 4.32 391 21.8
Vulkan, every layer, contiguous cache 45 3.35, 0.323 45 3.47 2 740 146
Vulkan, 17 of 34 layers, paged cache 47 2.86, 0.288 46 4.16 389 22.8

oklm describe --temp 0 writes the same headline and subheading on the three backends (Vulkan the CPU's 48 tokens but one, OpenCL parting from them at the fourteenth). The tiny model's five cases run three ways each on OpenCL and on each Vulkan device (RX 7900 XT, UHD 750), every block there and the last one alone, within 1e-6 of the numpy pass, and no attention is handed back to the CPU. The server with an image runs on a device since (server.md, Gemma 3 4B among its files). Not run on a device: the 12B and 27B files.

LLaVA 1.5 has no whole-path comparison with llama.cpp, its encoder being read differently there: the encoder is held to the float64 pass, and the language side to llama.cpp's own model fed the embeddings OKLM computed (tools/mtmd_ref.cpp --answer-rows). oklm describe and oklm serve --mmproj write the same 64 tokens about the newspaper ("The image is a vintage newspaper clipping featuring a story about the Apollo 11 moon landing. The headline reads "Men Walk on Moon." [...]"). Asked what the newspaper says, llama.cpp with its own encoder answers "Men Walked on Moon" and with OKLM's embeddings "Men Walk on Moon", the headline. The dashboard's scripted check (tools/check_dashboard.sh) was run with LLaVA 1.5 as its vision model: a pasted picture is sent and answered. Not run: LLaVA 1.5 13B; a file made by a newer conversion (which names its projector and may hold all 24 blocks: OKLM runs every block a file holds).

LLaVA 1.6 has no reference in llama.cpp, which slices such a file its own way, and transformers could not be run here (no PyTorch on the machine). Its reference is the model's own processor and code written again from transformers' sources, function by function, in tools/llava_next_ref.py (numpy and Pillow), and what is compared with what is this, in three parts that follow each other from the picture to the rows:

  • The tiles. From the picture alone, the reference chooses the resolution, resizes, pads and cuts; OKLM's pixels are its bytes normalized. Nothing of OKLM enters the reference.
  • Each tile's embeddings. The float64 pass of the encoder (tools/vision_ref.py, the pass LLaVA 1.5 is held to) on the reference's pixels, one tile at a time; OKLM's embeddings of the ten tiles are within 7.6e-6 of it.
  • The rows. The reference places the embeddings of the tiles as pack_image_features() does; the embeddings it is given are the ones OKLM computed, so this part checks the placement alone (which feature goes where, what is cut, the ends of rows), and the test asks for equal values.

What these do not cover: transformers itself run on the same picture (the port could carry a misreading of its sources), and the choices where transformers' versions differ, which are listed in the script. The same checks run in make test on a tiny file, with pictures of five sizes that take every branch (one row of two or three tiles, a square of four; a width cut, a height cut; an odd difference): tests/mm/test_vision.c.

The language side is held to llama.cpp's own model fed the rows OKLM computed (tools/mtmd_ref.cpp --answer-rows): the 48 greedy tokens are the same. oklm describe --temp 0 and oklm serve --mmproj (an image_url part and the question, greedy) write the same 64 tokens about the newspaper ("The newspaper headline reads: "MEN WALK ON MOON ASTRONAUTS LAND ON PLAIN; COLLECT ROCKS, PLANT FLAG VOICE FROM MOON: 'EAGLE HAS LANDED' A POWDERY SURFACE IS"): the headline and the subheadings under it, as printed. With the overview alone (--tiles 336, 577 rows) the same model reads the headline and gives the last subheading as "A POWERFUL SURFACE". Asked to describe the picture drawn by the test, it names a large blue circle, a red square, a yellow circle and a checkered background. The server's second request for the same picture finds 2 352 of its 2 355 rows in the prefix cache (in that run the server put the question right after the image; it writes a new line between them since, as oklm describe and the model's own prompt do). Not run: a LLaVA 1.6 with another language model (the Vicuna and 34B variants: the same projector layout, and for Vicuna the turns LLaVA 1.5 gets); two images in one prompt on the published file (the context must then hold up to 5 856 rows); a picture that takes three tiles in a row on the published file (the tiny file has the case).

The Qwen encoders are held to the float64 pass of the model's definition (1e-4), and to llama.cpp loosely (5e-2), for what llama.cpp's distance is made of. ggml gives a product with an F16 matrix its operand rounded to F16; a float64 pass that rounds each operand so and uses the tanh GELU (tools/vision_ref.py --as-llama-cpp) comes within 2.9e-3 and 5.2e-3 of llama.cpp's embeddings for Qwen2-VL, from 2.4e-2 and 2.0e-2. The distance is not spread evenly: the median embedding of llama.cpp is 0.2 to 0.8 % from the exact one, and a few rows carry the rest, the same rows in llama.cpp and in the rounded pass (one of the 273 moves by 36 to 79 % in three of the four cases). For the second picture with Qwen2.5-VL the rounded pass is as far from llama.cpp (1.1e-2) as the exact one, for that reason: both differ from the exact pass mostly on one row, not by the same amount.

The language side was checked three ways. A text alone gives llama.cpp's choices (all 48 with the 2B; 47 with the 7B, the other at a tie of 0.23). With the same embeddings on both sides (tools/mtmd_ref.cpp --answer with the embeddings OKLM computed, which mtmd places at its three-part positions), the choices part only where llama.cpp's two best logits are close: 0.09 for the 2B, 0.40 for the 7B, whose logits are further from llama.cpp's on a text already (2.24: the two compute the quantized products differently, and llama.cpp's norms lack the file's epsilon, Qwen2.5-VL 7B and llama.cpp's RMS norm). As a control, the same rows given the positions of a text, one after the other, move the 2B's logits by 9.8 and change a choice where llama.cpp's two best are 1.8 apart: the positions are what the agreement rests on. oklm describe --temp 0 and oklm serve --mmproj (an image_url part and the question, greedy) write the same 64 tokens with both models ("The newspaper says, "Men Walk on Moon. Astronauts Land on Plain; Collect Rocks, Plant Flag.""; the 7B: "The newspaper in the image is from The New York Times, dated Monday, July 21, 1969. The headline reads: "MEN WALK ON MOON" [...]"). The server's second request for the same picture finds 416 of its 418 rows in the prefix cache, and a second turn of the conversation (the 7B's) 480 of 497. A picture of 1 920 x 1 464 pixels (the test image enlarged) is reduced to 1 240 embeddings and read the same by the 2B, its prompt taking 66 s; no reference was run at that size. Not run: several images in one prompt on the published files (the tiny model has the case), Qwen2.5-VL 3B (its checks are in the test, skipped without the files).

A video on the published files (2026-11-14, CPU, greedy): four frames a second apart of 448 x 336, the test image for two and its top left corner enlarged for two, are 2 pairs of 192 embeddings, a prompt of 423 tokens (8.7 s for the 2B, 10.8 s for the 7B). Asked what the video shows at its start and at its end, the 2B answers "The video starts with a close-up shot of a newspaper, specifically the New York Times, and then transitions to a black and white image of a man walking on the moon. The video ends with a shot of the moon."; asked to describe its two parts, the 7B reads the page ("The first part of the video shows a newspaper front page from The New York Times dated July 20, 1969. The headline reads "MEN WALK ON MOON," with a subheading that says "Astronauts Land on Plain; Collect Rocks, Plant Flag." [...]") and had not reached the second part after 110 tokens. No reference exists for a video on these files (llama.cpp has no video path for them): the embeddings, the positions and the layout are checked on the tiny models.

On OpenCL and on Vulkan (2026-11-14, RX 7900 XT) the same rows and tokens give, against the CPU's logits and against llama.cpp's (test_mm_public, the language model given OKLM's embeddings; each layout on a model loaded for it):

Model, layout Choices the CPU's, of 48 Logits from the CPU's: largest, mean Choices llama.cpp's Logits from llama.cpp's Prompt, rows/s Tokens/s
Qwen2-VL 2B, CPU 46 1.76 483 28.1
OpenCL, every layer, contiguous cache 46 2.16, 0.121 46 2.59 6 131 202
OpenCL, 14 of 28 layers, paged cache 47 1.12, 0.094 47 1.82 1 019 50.6
Vulkan, every layer, contiguous cache 47 2.47, 0.141 45 2.64 6 183 303
Vulkan, 14 of 28 layers, paged cache 48 1.38, 0.114 46 1.84 1 008 55.0
Qwen2.5-VL 7B, CPU 46 3.16 111 6.5
OpenCL, every layer, contiguous cache 47 1.27, 0.162 46 3.00 1 910 91.2
OpenCL, 14 of 28 layers, paged cache 47 1.14, 0.150 46 3.17 236 12.7
Vulkan, every layer, contiguous cache 47 1.38, 0.180 47 2.95 1 754 115
Vulkan, 14 of 28 layers, paged cache 46 1.11, 0.160 48 3.18 233 13.1

The rates are of one run each, the prompt's the 418 rows through the language model alone (Vulkan's first prompt includes its pipelines: 3 581 rows/s on another run). oklm describe --temp 0 --backend opencl and --backend vulkan write the CPU's sentence with the 2B; with the 7B OpenCL writes the CPU's 48 tokens and Vulkan parts from them at the tenth. The tiny model runs its seven cases seven ways each on OpenCL and on each Vulkan device (RX 7900 XT, UHD 750), every block on the device and the last one alone, within 6e-5 of the numpy pass (the paged cache, its prefix cache, cuts and copies among them).

Layers split between the device and the CPU (2026-11-14, RX 7900 XT, answer_devices() of test_mm_public). An image's rows then get their three-part positions on both sides: the CPU's kernel in the first layers, the device's in the last. Beside the paged cache above, the same rows and tokens with the contiguous cache, and with the CPU's layers run on the device for the prompt, their weights streamed (--gpu-stream, oklm_context_params.stream_batch: the image's rows are then rotated on the device in every layer and the tokens after them on the CPU in the first 14), each on a model loaded for it:

Model, layout (14 of 28 layers on the device) Choices the CPU's, of 48 Logits from the CPU's: largest, mean Choices llama.cpp's Logits from llama.cpp's Prompt, rows/s Tokens/s
Qwen2-VL 2B, OpenCL, contiguous cache 47 1.12, 0.094 47 1.82 989 52.4
OpenCL, the CPU's layers streamed for the prompt 45 1.71, 0.113 47 1.78 2 256 52.4
Vulkan, contiguous cache 48 1.38, 0.114 46 1.84 987 54.7
Vulkan, the CPU's layers streamed for the prompt 46 2.73, 0.130 45 2.63 2 067 55.1
Qwen2.5-VL 7B, OpenCL, contiguous cache 47 1.14, 0.150 46 3.17 225 12.8
OpenCL, the CPU's layers streamed for the prompt 47 1.20, 0.157 47 3.01 534 12.8
Vulkan, contiguous cache 46 1.11, 0.160 48 3.18 223 13.0
Vulkan, the CPU's layers streamed for the prompt 47 1.26, 0.170 47 3.03 488 13.0

The split's logits are the paged cache's to the digits printed: the cache's layout changes no sum. A video goes the same four ways (test_qwen_video_devices: four frames of 448 x 336 cut from the test image, each 32 pixels to the right of the one before and half a second after it, two grids of 192 embeddings, a prompt of 411 rows, asked what the newspaper says; the CPU's own 48 greedy tokens, no reference existing for a video):

Model Layout Choices the CPU's, of 48 Logits from the CPU's: largest, mean Prompt, rows/s Tokens/s
Qwen2-VL 2B CPU 475 27.2
OpenCL, every layer 48 1.73, 0.123 6 672 199
OpenCL, 14 of 28, paged 48 1.00, 0.100 1 010 50.9
OpenCL, 14 of 28, contiguous 48 1.00, 0.100 993 52.4
OpenCL, 14 of 28, streamed for the prompt 48 1.54, 0.115 2 296 52.7
Vulkan, every layer 48 1.81, 0.129 6 763 304
Vulkan, 14 of 28, paged 48 1.04, 0.106 914 46.3
Vulkan, 14 of 28, contiguous 48 1.04, 0.106 925 55.5
Vulkan, 14 of 28, streamed for the prompt 48 1.76, 0.124 1 917 55.0
Qwen2.5-VL 7B CPU 110 6.5
OpenCL, every layer 47 1.62, 0.163 2 157 92.2
OpenCL, 14 of 28, paged 48 1.08, 0.153 235 12.7
OpenCL, 14 of 28, contiguous 48 1.08, 0.153 221 12.8
OpenCL, 14 of 28, streamed for the prompt 48 2.25, 0.176 552 12.8
Vulkan, every layer 47 1.29, 0.159 2 011 115
Vulkan, 14 of 28, paged 48 1.08, 0.142 235 12.6
Vulkan, 14 of 28, contiguous 48 1.08, 0.142 225 12.9
Vulkan, 14 of 28, streamed for the prompt 46 1.71, 0.162 560 13.1

Where a choice parts from the CPU's the two tokens were 0.04 to 0.35 apart on the CPU. Asked "What happens in this video?" instead, the 2B parted from the CPU's answer at 2 to 5 of the 48 tokens (43 on Vulkan with every layer on the device, under the test's 90 %), its logits as far from the CPU's as in the table (1.9 at most, 0.14 on average): the 2B's wording moves at near ties, and the test asks the question whose answer is the page's text. oklm describe --backend opencl and --backend vulkan with --gpu-layers 14 --gpu-stream 16 write the sentence the CPU writes for the 2B (the encoder on the device, a prompt of 418 tokens in 0.53 s and 0.54 s).

On the Intel UHD 750 (2026-11-14; OKLM_CL_DEVICE=UHD OKLM_VK_DEVICE=Intel with test_mm_public in a directory holding these files alone). The published vision files that fit it run there on both backends, slower than the eight cores beside it: nothing is refused, nothing fails, and the answers are the CPU's to near ties.

File Encoder: CPU, OpenCL, Vulkan from the CPU's embeddings Language model, every layer: choices the CPU's of 48, logits; prompt rows/s, tokens/s (CPU; OpenCL; Vulkan) Half the layers, paged cache (OpenCL; Vulkan)
SmolVLM-256M Q8_0 (one tile) 2.3 s, 10.7 s, 3.2 s 2.6e-6, 2.3e-6 2 545, 157; 48, 1.85; 69, 28.7; 47, 1.25; 148, 42.9 48, 0.94; 130, 41.3; 47, 1.09; 397, 62.1
SmolVLM-500M Q8_0 1 145, 68.8; 47, 1.80; 25, 13.1; 47, 1.58; 100, 24.1 48, 1.00; 48, 19.5; 48, 0.88; 192, 34.6
Qwen2-VL 2B Q4_K_M (273 embeddings) 6.9 s, 104 to 122 s, 25 to 32 s 5.1e-5 to 5.5e-5, 4.9e-5 to 6.2e-5 353, 26.9; 46, 2.50; 4, 3.1; 46, 1.83; 35, 14.4 48, 1.22; 10, 5.1; 48, 1.15; 66, 18.2

Qwen2-VL 2B's answer on the UHD 750 gives 46 of llama.cpp's 48 tokens on both backends, as the CPU does, and 46 of 48 of the choices its CPU embeddings give when the encoder's blocks are there too (the two tokens 0.25 apart). With the contiguous cache and half the layers the logits are the paged cache's; with the CPU's half streamed for the prompt the choices are 46 to 48 of 48 for the three files and the prompt is slower than unstreamed (6 against 12 rows/s for the 2B on OpenCL, 36 against 66 on Vulkan): the device is the slow side. The rates above were taken while other runs used the machine; alone, oklm describe --temp 0 on the test image (391 embeddings, the encoder on the device) writes the same sentence three ways: the prompt of 418 tokens in 9.78 s on the CPU, 40.4 s on the UHD 750 through Vulkan and 196 s through OpenCL, the 23 tokens after it in 0.82 s, 1.50 s and 5.69 s. The time is mostly the encoder's (1 564 rows through 32 blocks), where Intel's OpenCL takes four times Vulkan's. Where OpenCL's 196 s go there (OKLM_CL_PROFILE=1, 2026-11-14, 195.5 s of kernel time in 3 034 launches): the matrix products, 92.5 % (the encoder's F16 products on gm_f16 115.0 s in 194 launches, the language model's prompt on gm_q4_k 55.0 s and gm_q6_k 10.7 s); the encoder's bidirectional tiles are 8.3 s (attention_bi80, 32 launches), the language model's attention 0.75 s, and no op falls back to the CPU. It is not the tiles' shape: with OKLM_CL_GEMM_TILES=0 the same products on the row kernels take as long (mm_f16 127.9 s, mm_q4_k 48.3 s, the prompt 195.6 s either way). The encoder's products are about 1 970 GFLOP, so both kernel families run at about 17 GFLOP a second on this device, whatever their work-groups; nothing cheap changes that, and the CPU beside it is twenty times faster. Not run there: Qwen2.5-VL (its 7B is 4.4 GiB of weights on shared memory), Gemma 3, LLaVA, SmolVLM2-2.2B, a picture above 273 to 391 embeddings, several pictures, a video.

Pictures above a megapixel (2026-11-14; test_qwen_large_pictures). llama.cpp's test image enlarged (bicubic) to 1 624 x 1 238 pixels, 2.0 megapixels, is 2 552 embeddings (58 x 44); enlarged to 2 600 x 1 982, 5.2 megapixels, it is reduced to 2 044 x 1 540 and 4 015 embeddings (73 x 55), the most its proportions give under 4 096. llama.cpp gives both the same sizes and its prompts the same rows (2 579 and 4 042). Its 16 greedy tokens fed here on the CPU, from the same embeddings and from its whole path; then the encoder on each device against the CPU's embeddings, and the language model with every layer there:

Model, embeddings Choices llama.cpp's, of 16: same embeddings, whole path Logits from llama.cpp's: largest (mean) Encoder, CPU OpenCL: time, from the CPU's Vulkan Language model on OpenCL: choices the CPU's, logits; prompt rows/s on Vulkan
Qwen2-VL 2B, 2 552 16, 16 0.94 (0.11), 0.80 (0.11) 215 s 4.36 s, 1.3e-4 5.03 s, 7.7e-5 16, 1.13; 6 469 16, 0.97; 5 290
Qwen2-VL 2B, 4 015 16, 16 0.93 (0.12), 1.03 (0.12) 467 s 9.87 s, 2.9e-4 11.6 s, 9.8e-5 16, 0.80; 5 366 16, 0.88; 3 952
Qwen2.5-VL 7B, 2 552 15, 15 3.92 (0.49), 3.69 (0.42) 90 s 1.63 s, 1.1e-4 1.39 s, 7.8e-5 16, 1.44; 2 156 16, 1.29; 1 933
Qwen2.5-VL 7B, 4 015 15, 15 3.60 (0.33), 3.67 (0.32) 169 s 2.92 s, 2.1e-4 2.63 s, 1.7e-4 15, 1.15; 1 858 15, 1.34; 1 582

The devices' columns were measured again on 2026-11-15 by the same test on the last code and are the table's: the encoder at 2 552 embeddings in 4.36 s (OpenCL) and 5.04 s (Vulkan) for Qwen2-VL 2B and 1.62 s and 1.39 s for Qwen2.5-VL 7B, at 4 015 in 9.82 s and 11.6 s and in 2.92 s and 2.63 s, 7.7e-5 to 2.9e-4 from the CPU's embeddings.

The CPU's encoder times of the table are of one pass each while other work ran on the machine, before its kernels were looked into (llama.cpp's whole path, its own encoder on the CPU with the prompt and the 16 tokens, took 70 s and 159 s for the 2B and 76 s and 206 s for the 7B on the same day, busy as well). Where the time went, at 4 015 embeddings of Qwen2-VL (16 060 patch rows through 32 blocks, 389 s in the ops): 290 s in the attention, every block's rows seeing all of them, each tile of 32 rows gathering its head's 16 060 keys and values from the tensors again, and 95 s in the products, whose F16 weights had a one-row dot product. Since then (core.md): the attention of such rows runs head by head on copies of the head's keys and values, the products of many rows over F16 weights take eight rows a weight row (AVX-512; four on AVX2) and are cut by blocks of activation rows. The sums are the same in the same order, so the embeddings have the bits they had (the files of the two sizes below compared byte for byte with those of the kernels before), and no recording or bound moved. The encoder alone on the picture of 2 600 x 1 982 pixels, eight threads of a Core i7-11700K, OKLM then llama.cpp (llama-mtmd-cli, no GPU layer, the projector on the CPU, --image-max-tokens), one after the other on 2026-11-14 with other work on the machine at times (load averages of 4 to 12):

Encoder on the CPU OKLM before OKLM llama.cpp
Qwen2-VL, 1 008 embeddings (--tiles 900) 32.4 s, 34.7 s 16.5 s 16.8 s (14.4 s and 15.6 s on other passes)
Qwen2-VL, 4 015 389 s in the ops 164 s 136 s (135 s on another pass)
Qwen2.5-VL 7B, 1 008 not measured 13.5 s 12.4 s (11.3 s on another pass)
Qwen2.5-VL 7B, 4 015 169 s (the table's) 66.3 s 83.9 s

One block of Qwen2-VL at 4 032 patch rows: the attention 0.41 s before and 0.14 to 0.17 s now, the six products 0.53 s and 0.36 s. OKLM is level with llama.cpp at a megapixel and ahead on Qwen2.5-VL's large picture, whose 28 windowed blocks attend 64 rows at a time; it is a fifth behind on Qwen2-VL's, where the attention over 16 060 rows is 114 of the 164 s: 370 GFlop/s on eight threads against 590 at 4 032 rows, each tile of eight rows reading its head's 10 MiB of keys and values from the last-level cache, and tiles of more rows no faster (their scores then leave the second-level cache). Qwen2.5-VL at a megapixel is 9 % behind in its products (10.8 of the 13.5 s), the MLP's second one having rows of 3 420 elements that do not stay in the first-level cache eight at a time. The CPU's language model reads the 2B's 4 042 rows at 304 rows/s and the 7B's at 82. A device's embeddings part from the CPU's as the picture grows, a row's attention summing over more keys: the tests bound a large picture's at 6e-4 where a small one's are bounded at 2e-4, and the answers are the CPU's (one choice of the 7B's parts at 4 015 embeddings, its two tokens 0.23 apart). oklm describe --temp 0 on the 2B writes the same sentence at both sizes on both backends ("The newspaper says, "Men Walk on Moon. Astronauts land on plain; collect rocks, plant flag.""): the prompt of 2 579 tokens in 4.89 s on OpenCL and 5.61 s on Vulkan, that of 4 042 tokens in 10.80 s and 12.78 s. Qwen2.5-VL's windowed blocks make its encoder 2.7 to 4.4 times faster than Qwen2-VL's, whose 32 blocks attend over every row. Through oklm serve with its default context of 4 096 the picture of 5.2 megapixels is 3 024 embeddings (a prompt of 3 051 tokens), answered in the same words, and the request sent again finds 3 040 of its rows in the prefix cache.

Several pictures in one prompt (2026-11-14; test_qwen2vl_several_images, test_qwen25vl_several_images). Three pictures: llama.cpp's test image (391 embeddings), a part of it (the headline, 364 x 196 pixels: 91) and the picture the tests draw (273); two of them and the three in one user turn, then two and three across two turns with an answer between them. Each image takes its own three-part positions after the one before: an image of w x h embeddings advances the position by max(w, h), so the three take 23 + 13 + 21 positions for their 755 rows. The prompt is the file's template's (a marker a picture, the question after the last); llama.cpp tokenizes it to the same rows. Its 32 greedy tokens are then fed here on the CPU, from the same embeddings (its language model given OKLM's) and from its whole path (its own encoder); then the same rows on OpenCL and on Vulkan, every layer on the device:

Model, prompt Rows (of the pictures) Choices llama.cpp's, of 32: same embeddings, whole path Logits from llama.cpp's: largest (mean) OpenCL: choices the CPU's, logits Vulkan
Qwen2-VL 2B, two pictures, one turn 518 (482) 31, 32 0.75, 1.19 31, 0.77 32, 0.97
three pictures, one turn 797 (755) 32, 32 0.98, 1.18 32, 0.90 32, 0.97
two pictures, two turns 719 (664) 32, 32 2.62, 1.99 31, 3.32 31, 3.39
three pictures, two turns 817 (755) 31, 31 3.04, 2.10 30, 3.04 29, 3.17
Qwen2.5-VL 7B, two pictures, one turn 518 (482) 28, 29 6.06, 5.69 29, 1.31 30, 1.28
three pictures, one turn 797 (755) 29, 29 12.8, 14.3 32, 2.68 32, 1.63
two pictures, two turns 719 (664) 28, 29 4.57, 5.36 31, 1.48 31, 1.41
three pictures, two turns 817 (755) 31, 29 7.45, 7.34 29, 1.60 30, 1.95

With half the layers on the device (paged, contiguous, the CPU's half streamed for the prompt) the 2B gives 30 to 32 of the CPU's 32 choices on either backend (logits within 2.53) and the 7B 29 to 32 (within 1.99). The 2B is llama.cpp's to a near tie. The 7B is further from llama.cpp with several pictures than with one (46 of 48 choices and logits within 3.2 for one picture; 2.0 on a text alone): with three pictures in one turn the logit of llama.cpp's first token is 31.0 there and 22.9 here, and the choices part where llama.cpp's two best are up to 1.22 apart. The rows and the positions are the 2B's, which agrees; the CPU and the two devices, three sets of kernels, agree with each other within 2.7 on that prompt. It is not the quantization: the same prompts and embeddings through the 7B's BF16 language model (unsloth's, 14.2 GiB, fetched for this and removed), whose products are float products in both programs, give on the CPU

Qwen2.5-VL 7B BF16, prompt Rows Choices llama.cpp's Logits from llama.cpp's: largest, mean
a text alone 38 47 of 48 1.71, 0.16
one picture 418 48 of 48 3.70, 0.44
two pictures, one turn 518 31 of 32 6.30, 0.49
three pictures, one turn 797 29 of 32 9.43, 1.26
two pictures, two turns 719 30 of 32 5.22, 0.36
three pictures, two turns 817 31 of 32 8.49, 0.28

(llama.cpp given OKLM's embeddings; recordings made for the comparison and not kept). The two programs are 0.16 apart on average on a text of 38 tokens of this model, where Qwen2-VL 2B's Q4_K_M is 0.11 from llama.cpp's on the same text and on a picture of 4 015 embeddings; the distance grows with the image rows, and the choices still part at near ties only (0.53 at most on the float model). OKLM is the one nearer the model, and the distance is llama.cpp's: the pinned llama.cpp evaluates this architecture's RMS norms with an epsilon of 0 where the file says 1e-6, which the 7B's rows of <|im_start|> and <|im_end|> need and the 2B's do not; a float64 pass of the model's definition (tools/qwen2vl_ref.py) is within 0.15 of OKLM's logits on the BF16 file where llama.cpp is 1.13 to 6.42 from it, and the same pass with an epsilon of 0 is llama.cpp's within 0.23 (Qwen2.5-VL 7B and llama.cpp's RMS norm). The tests keep llama.cpp as the reference and bound the several-picture cases at 85 % of the choices and near ties of 1.5.

The language models of LLaVA 1.5, LLaVA 1.6 and the SmolVLM files needed nothing to run on a device with an image's embeddings: their rows are evaluated as tokens' are. test_mm_public runs them on OpenCL and on Vulkan (2026-11-14, RX 7900 XT) on the same rows and tokens as the CPU: llama.cpp's 48 greedy tokens for the LLaVA files, the CPU's own 48 for the SmolVLM files, which have no recorded llama.cpp answer; every layer on the device with the contiguous cache, and half of them with a paged cache (each on a model loaded for it; the two other layouts of half the layers, the contiguous cache and the CPU's layers streamed for the prompt, are below the table):

Model Rows of prompt Layout Choices the CPU's, of 48 Logits from the CPU's: largest, mean Choices llama.cpp's Prompt, rows/s Tokens/s
LLaVA 1.5 7B Q4_K 595 CPU 47 105 6.4
OpenCL, every layer 47 0.44, 0.053 48 2 157 101
OpenCL, 16 of 32, paged 48 0.35, 0.044 47 203 10.9
Vulkan, every layer 47 0.37, 0.049 48 2 011 127
Vulkan, 16 of 32, paged 48 0.29, 0.040 47 213 11.4
LLaVA 1.6 Mistral 7B Q4_K_M 2 357 CPU 48 89 6.1
OpenCL, every layer 48 0.39, 0.045 48 2 023 82.9
OpenCL, 16 of 32, paged 48 0.33, 0.042 48 166 10.8
Vulkan, every layer 48 0.43, 0.044 48 1 802 113
Vulkan, 16 of 32, paged 48 0.43, 0.040 48 173 11.2
SmolVLM-256M Q8_0 348 CPU 2 561 170
OpenCL, every layer 47 1.89, 0.171 6 038 366
OpenCL, 15 of 30, paged 48 0.94, 0.106 3 040 191
Vulkan, every layer 48 1.84, 0.173 4 809 675
Vulkan, 15 of 30, paged 48 0.94, 0.106 2 825 241
SmolVLM-500M Q8_0 348 CPU 1 143 68.8
OpenCL, every layer 48 1.74, 0.157 4 939 343
OpenCL, 16 of 32, paged 48 0.99, 0.102 1 616 105
Vulkan, every layer 47 1.77, 0.158 4 710 540
Vulkan, 16 of 32, paged 48 1.00, 0.102 1 585 120
SmolVLM2-2.2B Q8_0 433 CPU 294 14.9
OpenCL, every layer 47 1.22, 0.062 3 457 219
OpenCL, 12 of 24, paged 47 0.63, 0.049 557 28.2
Vulkan, every layer 47 1.17, 0.061 4 003 282
Vulkan, 12 of 24, paged 47 0.60, 0.049 566 29.1

The rates are of one run each, the prompt's the rows through the language model alone. With half the layers on the device and the contiguous cache the logits are the paged cache's to the digits printed. With the CPU's half run on the device for the prompt, its weights streamed (--gpu-stream; 2026-11-14, one run), the choices the CPU's are 48 and 47 of 48 for LLaVA 1.5 (OpenCL, Vulkan; logits within 0.38), 48 and 48 for LLaVA 1.6 (0.39), 47 and 48 for SmolVLM-256M (1.29), 48 and 48 for SmolVLM-500M (1.65), 47 and 47 for SmolVLM2-2.2B (1.19), 46 and 45 for Gemma 3 4B, whose image rows see each other (3.16; 46 and 47 without the streaming). The prompt is faster streamed where the model is large (LLaVA 1.6's 2 357 rows at 889 and 785 rows/s against 167 and 172; LLaVA 1.5's at 531 and 521 against 173 and 202; Gemma 3 4B's at 718 and 815 against 393 and 389) and slower for the small files (SmolVLM-500M's at 1 532 and 1 150 rows/s against 1 732 and 1 673). The server with an image runs on a device since (server.md, SmolVLM-500M among its files). SmolVLM-256M and SmolVLM-500M on the Intel UHD 750 are with Qwen2-VL 2B's figures there (above: 47 or 48 of the CPU's 48 choices, at a sixth to a half of the CPU's tokens a second).

On the CPU, Qwen2-VL 2B takes 12 s for a prompt with llama.cpp's test image (1 564 patches through 32 blocks, 418 rows of prompt) and Qwen2.5-VL 7B 13 s for the prompt and 13 s for 64 tokens, on a machine shared with other work. With the language model on a device the prompt is the encoder's time, which runs on the CPU for these two models (the encoder on a device): oklm describe with the 2B, 8.5 s on OpenCL and 8.4 s on Vulkan against 9.7 s on the CPU (the 418 rows through the language model take 0.07 s there), then 23 tokens in 0.12 s and 0.08 s against 0.82 s; with the 7B, 7.3 s for the prompt on either device against 10.9 s, and 48 tokens in 0.52 s and 0.41 s against 7.4 s. On the CPU, Gemma 3 4B takes 86 s for a prompt with one image (4 096 patches through 27 blocks) and LLaVA 1.5 7B 19 s (577 rows through 23 blocks, then 595 rows of prompt), on a machine shared with other work. LLaVA 1.6 Mistral 7B takes 105 s for a prompt with llama.cpp's test image (five tiles through the encoder, then 2 357 rows of prompt) and 32 s for 96 tokens, with 5 threads on the same shared machine (241 s with 4 threads at a busier time); 26 s for the prompt with the overview alone (594 rows); through the server, 243 s for the first request and 47 s for the second, its image found in the cache. On the CPU, SmolVLM-256M reads the headline of llama.cpp's test image ("MEN WALK ON MOON", 320 image embeddings, prompt in 6.7 s); Whisper base transcribes jfk.wav (11 s) in 0.9 s; Ultravox with Llama-3.2-1B quotes the speech. The large Whisper models are slow on the CPU: measured on a machine shared with other work, a 30-second chunk took 34 to 54 s with large-v3-turbo and 80 to 135 s with large-v3 (base: 1 to 2 s, small: 9 to 18 s); on a quiet machine 6 and 12 s, and 0.3 and 0.7 s with --backend vulkan (Whisper on a device).