Vision model benchmarks
Overview
—Capability & performance
Compare speed, memory and task scoresTask measurements & size
Time, tokens and model parametersBenchmark breakdown
Instruction following, tool calling and long promptsVision overview
Answering questions about imagesQuality & performance
Image performance
Detailed measurements
Run progress
Video understanding
Questions about actions and eventsThe question test uses 32 short clips, shown as eight ordered frames without sound or subtitles. These are small public test selections, not full benchmark scores.
Quality & size
Video performance
Three fixed clips, each repeated three timesDo more frames help?
The same 32 questions, with different visual inputQuestion-only checks reveal what can be guessed without seeing the clip. One frame adds a still image; eight frames add visual coverage and action order. This comparison alone does not prove temporal reasoning.
Scores, failed answers & tested setups
Video model progress
Speech & audio input
Transcription and spoken instructionsSpeech performance
Accuracy & performance
Compare speech models
— = unavailable or outside the model’s tested tasks. Formats show language-model / audio-encoder precision. Speed and memory refer to the Raspberry Pi 5.
Audio run progress
Results table · 17 configurations
VQAv2 tests answers to image questions. POPE checks whether named objects are present. Each uses 96 fixed questions; higher is better. Speed and memory are measured on Pi 5, including the image encoder. A dash means no matching measurement is available.
| Model | VQAv2 / 100 ↑ | POPE / 100 ↑ | Pi 5 first token · s ↓ | Pi 5 task time · s ↓ | Peak memory · MiB ↓ |
|---|---|---|---|---|---|
| Qwen3.5-0.8B | 80 | 91 | 4.1 | 4.3 | 1,591 |
| Qwen3.5-2B | 85 | 93 | 10.8 | 11.2 | 3,498 |
| Qwen3.5-4B | 88 | 88 | 17.4 | 18.2 | 6,526 |
| Qwen3.5-9B | 86 | 91 | 29.8 | 30.7 | 11,051 |
| Gemma 4 E2B IT | 54 | 69 | 6.8 | 7.3 | 5,545 |
| Gemma 4 E4B IT | 54 | 80 | 10.7 | 11.6 | 8,796 |
| Ministral 3 3B Instruct | 78 | 79 | 212.9 | 227.7 | 6,049 |
| Ministral 3 8B Instruct | 73 | 77 | 263.7 | 280.4 | 11,435 |
| Qwen3-VL 4B Instruct | 82 | 91 | 16.4 | 17.1 | 7,169 |
| Qwen3-VL 8B Instruct | 86 | 94 | 28.6 | 29.6 | 11,254 |
| MiniCPM-V 4.7 1B | 78 | 89 | 111.7 | 138.3 | 2,455 |
| LFM2.5-VL-1.6B | 88 | 89 | 11 | 11.3 | 2,505 |
| LFM2.5-VL-3B | 89 | 93 | 15 | 15.5 | 4,369 |
| SmolVLM2 256M | 62 | 69 | 6.8 | 6.9 | 688 |
| SmolVLM2 2.2B | 79 | 86 | 21.9 | 22.1 | 4,484 |
| LFM3-VL-2B | 88 | 86 | 6.8 | 7 | 2,865 |
| LFM3-Omni-2B | 78 | 90 | 6.7 | 7 | 3,429 |
How to read these results
Image questions
VQAv2 compares answers to 96 image questions with human answers. POPE checks whether named objects are present, using another 96 questions. Higher scores are better.
Speed and memory
Nine image questions are repeated three times on the Pi 5. Images are resized to at most 512 pixels on the longest side. Timing and memory include the image encoder.
Comparing models
First-token time is the wait until an answer starts. Task time ends when the answer finishes. A dash means no matching measurement is available.