PERFORMANCE TROUBLESHOOTING

Why is local AI so slow?

The usual causes are simple: the model is too large for fast loading on your hardware, too much work is falling onto the CPU, the context is long, or the runtime is not using the acceleration you expected.

Check whether the model fits cleanly in GPU memory

If only part of the model is on the GPU, the runtime may move data between GPU and system memory or perform more work on the CPU. That can turn a technically successful load into a frustrating chat experience.

Separate prompt processing from generation speed

Reading a long prompt and generating new tokens are different workloads. A system can process input quickly but generate slowly, or the reverse. Our benchmark page keeps those measurements separate for exactly this reason.

Context length can quietly increase work

Longer conversations require more cache memory and more prompt processing. If performance worsens as a chat grows, test the same model in a fresh short conversation.

Confirm the GPU is actually being used

With Ollama, ollama ps shows whether a loaded model is on GPU, CPU or a mixture. With other runtimes, use their hardware status or logs rather than assuming that installing a GPU automatically means the workload uses it.

Model size matters more than brand

A smaller model that stays entirely on the GPU can feel much faster than a larger model that spills into CPU processing. If response quality is good enough for your task, the smaller model may be the better practical choice.

Do not confuse loading time with token speed

A model may take time to load from storage and then generate quickly. Keeping frequently used models loaded can improve repeated use, but consumes memory while they remain resident.

If the runtime reports an actual memory error rather than simple slowness, use the out-of-memory troubleshooting guide.