Choosing a local assistant on Raspberry Pi
Start with the measurements for your device, then try the models on your own tasks.
1. Check memory and speed
Compare peak memory, the wait for the first text, time to finish an answer and output speed. Leave room in RAM for the operating system and your application. Model size on disk and parameter count are useful, but neither tells you how much memory a running model needs.
2. Choose the relevant test
IFEval tests instructions such as answer formats and word limits. BFCL tests tool choice and arguments. RULER tests finding information in long prompts. Use the individual scores to match your task, alongside the overall capability score.
3. Try real commands
For a home assistant, try commands you actually use: turning lights on, changing a temperature or asking about a sensor. Check the chosen device, action arguments and response time. Include unclear requests and actions that should be refused. These benchmarks test individual skills, so a strong score still needs testing with your integrations.
4. Compare images and speech separately
Use the Vision tab for image questions, STT for transcription, and TTS for generated speech and listening samples. The Audio tab also tests spoken instructions. The Video tab tests questions about actions across sampled frames and finding when an event happens. Continuous camera streaming is not covered.