Video model benchmarks
Overview
—Capability & performance
Compare speed, memory and task scoresTask measurements & size
Time, tokens and model parametersBenchmark breakdown
Instruction following, tool calling and long promptsVision overview
Answering questions about imagesQuality & performance
Image performance
Detailed measurements
Run progress
Video understanding
Questions about actions and eventsThe question test uses 32 short clips, shown as eight ordered frames without sound or subtitles. These are small public test selections, not full benchmark scores.
Quality & size
Video performance
Three fixed clips, each repeated three timesDo more frames help?
The same 32 questions, with different visual inputQuestion-only checks reveal what can be guessed without seeing the clip. One frame adds a still image; eight frames add visual coverage and action order. This comparison alone does not prove temporal reasoning.
Scores, failed answers & tested setups
Video model progress
Speech & audio input
Transcription and spoken instructionsSpeech performance
Accuracy & performance
Compare speech models
— = unavailable or outside the model’s tested tasks. Formats show language-model / audio-encoder precision. Speed and memory refer to the Raspberry Pi 5.
Audio run progress
Results table · 16 configurations
MVBench uses 32 questions about short clips, shown as eight ordered frames. Event overlap tests when an action happens. Higher is better for both; these small selections do not measure live camera streaming. A dash means no matching measurement is available.
| Model | Video questions · % ↑ | Event overlap · % ↑ | Pi 5 task time · s ↓ | Model files · MiB |
|---|---|---|---|---|
| Qwen3.5-0.8B | 16 | — | — | 703 |
| Qwen3.5-2B | 81 | — | — | 1,859 |
| Qwen3.5-4B | 78 | 21 | — | 3,255 |
| Qwen3.5-9B | 72 | — | — | 6,293 |
| Gemma 4 E2B IT | 41 | — | — | 3,903 |
| Gemma 4 E4B IT | 41 | — | — | 5,691 |
| Ministral 3 3B Instruct | 59 | — | — | 2,850 |
| Ministral 3 8B Instruct | 38 | — | — | 5,777 |
| Qwen3-VL 4B Instruct | 72 | — | — | 3,179 |
| Qwen3-VL 8B Instruct | 72 | — | — | 5,900 |
| MiniCPM-V 4.7 1B | 78 | 5.8 | — | 1,700 |
| LFM2.5-VL-1.6B | 66 | — | — | 1,511 |
| LFM2.5-VL-3B | 66 | — | — | 2,411 |
| SmolVLM2 256M | 41 | — | — | 306 |
| SmolVLM2 2.2B | 63 | — | — | 1,893 |
| VideoJev 4B | 72 | 49 | — | 8,547 |
How to read these results
Video questions
Models answer 32 MVBench questions about actions, using eight ordered frames per clip. Wrong, unfinished and unreadable answers score zero.
Event timing
Models locate a described action in 32 Charades-STA clips. The score measures overlap with the annotated time interval. Higher is better; the whole-clip line is a simple reference.
Test scope
The tests use sampled frames without sound or subtitles. They do not measure live camera streaming. A dash means no matching measurement is available.