Benchmark methodology
Quality scores show how well a model handles each kind of question. Speed, response time and memory show how it runs on the Raspberry Pi 5.
Text capability
| Benchmark | What it measures | Questions tested |
|---|---|---|
| IFEval | Following instructions such as word limits, required phrases and answer formats. | 64 |
| BFCL offline | Choosing the correct tool and filling in its arguments, without running the tool. | 96 |
| RULER | Finding and using information in a long prompt. | 48 |
Each test scores from 0 to 100; higher is better. The overall capability score averages all three equally. Every component must be complete before an overall score appears. We use the same fixed selection of public questions for every model, rather than the full official benchmarks. Small score differences should be treated cautiously.
Speed, response time and memory
Output speed is how quickly the model writes an answer, in tokens per second. Prompt processing is how quickly it reads the input. Higher is faster for both. First-token time is the wait until the first text appears; task time is the wait until the answer finishes. Lower is faster for these times.
Peak memory is the most RAM used by the model during the test, including the loaded model files. It is not the memory used by the entire device, and longer prompts may need more. Model file size is storage on disk. Parameter count describes the size of the model; it does not directly measure answer quality.
The standard Pi text setup uses llama.cpp, four threads and one request at a time. Speed tests read 256 input tokens and generate 128 output tokens, repeated ten times. Task time and memory use six fixed prompts with up to 128 output tokens, repeated three times. Exact model files, software versions and settings are recorded in the data download.
Missing results and comparisons
A dash means a measurement is not available yet. Before combining results, we check that they refer to matching model files, settings and questions. Incomplete tests stay unavailable. A quality score does not tell you whether a model will finish your own task within a particular deadline.
Reading charts
Scores of 10 and above are displayed as whole numbers; lower scores can show one decimal. Downloads retain the original precision. The default model selection includes popular models and every model on the tradeoff frontier: models where improving one plotted measure requires giving up something on the other. Each dot is one model. The chart explains whether better values are higher, lower, left or right. Axis ranges and frontier calculations include all eligible measured models, so showing another model does not redefine the boundary.
Image understanding
VQAv2 tests answers to questions about an image, with credit based on agreement with human answers. POPE tests whether named objects are actually present, including avoiding invented objects. Higher is better for both. Each benchmark uses 96 fixed questions. Images have a longest edge of 512 pixels, without cropping or enlarging smaller images. Image response times include reading the image; image memory includes its encoder. These scores are separate from text capability.
Speech recognition and spoken instructions
LibriSpeech tests transcription of clear and more difficult English audiobook recordings. FLEURS tests transcription of spoken sentences in English and German. Word error rate counts missed, changed and added words per 100 reference words; lower is better. It can exceed 100% if a model adds or repeats many words. The overall English recognition result averages the three English tests equally; German currently uses FLEURS alone.
Spoken IFEval tests whether a model follows instructions spoken aloud, including the requested response format. Higher is better. Transcription models are not ranked as spoken assistants unless they support that task. Audio processing factor is processing time divided by recording duration: below 1 means faster than playback. These tests submit complete recordings; they do not measure a live conversation.
Text to speech
Each model reads the same 48 public sentences per tested language and preset voice. Qwen ASR and Whisper independently transcribe the generated audio. Their word error rates show whether the words can be recovered accurately; the overall result averages the two equally. Lower is better. This measures intelligibility, not naturalness, emotion, voice similarity or voice-cloning quality. Listening samples let you hear the original output.
Synthesis speed compares the time needed to generate speech with the length of the resulting recording. A factor below 1 is faster than playback. First audio chunk is the wait for the first piece of speech, where the model provides it. Offline models that return a complete recording do not receive a streaming-latency value.
Video understanding
Models answer 32 questions about action order and what happens next in short MVBench clips. They receive eight ordered frames without sound or subtitles. Separate event-location tests check when an event happens. Question-only and single-frame comparisons show how much the visual input helps. These are small public test selections; they do not measure live camera streaming.
OCR
Models read 32 English document pages. Seventy-two text checks test whether selected passages are recovered and read in the right order. The overall score averages four document categories equally. This tests text recovery, not table structure, equations or complete page layout. Pi timing includes image loading, preprocessing, text detection and recognition, with the model already loaded.
Sources and reproduction
We use fixed selections from public datasets. The data download records test coverage, model files, software versions, generation settings and measurement dates. Original answers and logs are kept with the benchmark records. Advertised input support alone does not count as a measured result.