Audio model benchmarks
Overview
—Capability & performance
Compare speed, memory and task scoresTask measurements & size
Time, tokens and model parametersBenchmark breakdown
Instruction following, tool calling and long promptsVision overview
Answering questions about imagesQuality & performance
Image performance
Detailed measurements
Run progress
Video understanding
Questions about actions and eventsThe question test uses 32 short clips, shown as eight ordered frames without sound or subtitles. These are small public test selections, not full benchmark scores.
Quality & size
Video performance
Three fixed clips, each repeated three timesDo more frames help?
The same 32 questions, with different visual inputQuestion-only checks reveal what can be guessed without seeing the clip. One frame adds a still image; eight frames add visual coverage and action order. This comparison alone does not prove temporal reasoning.
Scores, failed answers & tested setups
Video model progress
Speech & audio input
Transcription and spoken instructionsSpeech performance
Accuracy & performance
Compare speech models
— = unavailable or outside the model’s tested tasks. Formats show language-model / audio-encoder precision. Speed and memory refer to the Raspberry Pi 5.
Audio run progress
Results table · 11 configurations
Word error rate (WER) counts missed, replaced and extra words per 100 reference words; lower is better and values can exceed 100%. Spoken IFEval tests following instructions read aloud; higher is better. A dash means no matching measurement is available.
| Model | Clean speech WER · % ↓ | Harder speech WER · % ↓ | English sentences WER · % ↓ | German sentences WER · % ↓ | Spoken instructions · % ↑ |
|---|---|---|---|---|---|
| Gemma 4 E2B IT | 4.4 | 9.8 | 12 | 9.6 | 55 |
| Gemma 4 E4B IT | 4.1 | 9.5 | 9.4 | 9.9 | 67 |
| Qwen3-ASR 0.6B | 2.6 | 4.3 | 6 | 8.8 | — |
| Qwen3-ASR 1.7B | 1.8 | 3.3 | 4.3 | 5.2 | — |
| Voxtral Mini 3B | 48 | 81 | 43 | 24 | 31 |
| LFM2.5-Audio-1.5B | 2.3 | 4.4 | 6.6 | — | — |
| Qwen2.5 Omni 3B | 2.5 | 25 | 9.1 | 13 | 38 |
| Ultravox 0.5 1B | 247 | 286 | 220 | 93 | 41 |
| Qwen2.5 Omni 7B | 20 | 68 | 87 | 135 | 26 |
| LFM3-Audio-2B | 2.9 | 5.5 | 6.1 | 6.6 | 32 |
| LFM3-Omni-2B | 2.8 | 5.4 | 6.2 | 7.1 | 66 |
How to read these results
Transcription
Word error rate counts missed, changed and extra words per 100 reference words. Lower is better. Extra or repeated words can push it above 100%.
Spoken instructions
Spoken IFEval checks whether models follow requests read aloud, including answer formats. Higher is better. Transcription-only models do not receive an instruction score.
Speed and memory
Processing time counts seconds of work per second of audio; below 1 is faster than playback. Pi 5 measurements repeat the same recordings three times and include the audio encoder.