Models and performance | ToshLLM Documentation

Model selection

Start with the workload, then the size.

Model size alone does not describe how a model will feel. Architecture, quantization, context length, prompt size and the split between GPU and CPU all affect speed and memory use. ToshLLM recommendations combine the detected hardware with measured estimates, but they remain estimates rather than guarantees.

Dense modelsEvery parameter participates in each token. They are predictable to place in memory and useful as a general baseline.

MoE modelsOnly selected experts activate for each token. They may offer more capability than their active footprint suggests, but expert placement matters.

Quantization

Trade memory for fidelity deliberately.

GGUF quantization reduces model size and memory pressure. Lower-bit variants usually fit more easily and load faster, while larger variants preserve more precision. Names such as Q4_K_M, Q6_K and Q8_0 identify different storage formats, not different parameter counts.

Compare results only when model, quantization and relevant runtime settings match. Two files with the same display name can still have different artifact hashes or internal metadata.

Memory controls

Leave room for the runtime.

GPU layers: the default value of 99 requests full GPU offload when the model permits it.
CPU MoE experts: moving experts to the CPU can make a large MoE model fit, with a generation-speed tradeoff.
VRAM reserve: ToshLLM reserves memory for macOS and runtime allocations instead of filling the GPU to the final megabyte.
Context size: larger windows consume more KV-cache memory. Reduce context before reducing model quality when an oversized window is unnecessary.
KV cache types: quantized cache formats reduce memory use, and the app applies the compatible attention route when needed.

If loading fails, change one variable at a time.

Lower context, increase the VRAM reserve or move more MoE experts to CPU. This keeps the cause visible and makes the next benchmark meaningful.

Multiple GPUs

Use every card with context.

ToshLLM can pin a model to one GPU or split it across selected AMD GPUs. Tensor splitting and direct peer transfers are available where the hardware supports them. Duo cards are still composed of multiple GPU devices, so the benchmark records the complete detected configuration.

More GPUs can improve prompt processing while cross-device synchronization can limit token generation. Measure both prompt and generation speed before deciding that a wider split is better.