Llama.cpp tester

[fuutott]

I want to share a little experiment I'm doing. I've got a script running on a change frozen vm that

1. Every two hours downloads and compiles cuda backend llama.cpp
2. runs a set of prompts against a selction of models at temp 0 seed 42
3. logs the output and does md5/vector(nomic 1.5)/llama-perplexity ppl nll on each model/response
4. uploads results to:
   https://llama-tester-results.surge.sh/#/models

llama-completion -hf "[model]" -p "[prompt]" -s 42 --temp 0 -n 512 -st --jinja --log-disable --no-display-prompt
llama-perplexity -hf "[model]" --ctx-size 512 --seed 42 --file "eval_corpus.txt"

Models were picked based on ability to fit in vram, but open to suggestions.

    ggml-org/gpt-oss-20b-GGUF
    unsloth/LFM2-8B-A1B-GGUF:Q8_0
    unsloth/Ministral-3-14B-Instruct-2512-GGUF:Q4_0
    unsloth/Qwen3-4B-Instruct-2507-GGUF:Q8_0
    unsloth/Qwen3-VL-30B-A3B-Instruct-GGUF:Q2_K
    unsloth/gemma-3-12b-it-qat-int4-GGUF:Q4_0
    unsloth/granite-4.0-h-tiny-GGUF:q8_0
    unsloth/phi-4-GGUF:Q4_K_M
    

Been running it for 24h and highlights so far:

1. LFM2 missing tensor:
   https://llama-tester-results.surge.sh/#/model/unsloth-LFM2-8B-A1B-GGUF-Q8_0/prompt/p0
2. gpt-oss changing ever so slightly:
   https://llama-tester-results.surge.sh/#/model/ggml-org-gpt-oss-20b-GGUF/prompt/p0
3. granite being random sometimes:
   https://llama-tester-results.surge.sh/#/model/unsloth-granite-4.0-h-tiny-GGUF-q8_0/prompt/p0
4. and sometimes not:
   https://llama-tester-results.surge.sh/#/model/unsloth-granite-4.0-h-tiny-GGUF-q8_0/prompt/p1

[am17an]

I get a 504 while seeing the results. But for sure this is useful in catching regressions and it would be nice if something like this is part of the CI. Even say if could run nightly and take a batch of commits from master to verify rather than run on every commit. Currently the paths that are not tested AFAIK

• Multi-GPU (-sm row, -sm layer)
• Any -ot configuration
• MoE models (AFAIK only Qwen 0.6B is tested)
• Hybrid models (now that Qwen3-next is popular)
• AMD GPUs
• Performance (harder to do)

All these have a good user base (judging from the issues created when one of these breaks) and would it be nice to have a test suite which covers at least these. I realize this has a monetary cost though, but considering the user/developer time saved in catching regressions early the benefit would be significant IMO. I think the entirety of this suite can be run in <1 GPU hour, which should cost less than 1$.