Local AI Assistants: Setting Realistic Expectations for LLM Performance

When it comes to using local Large Language Models (LLMs), many users have high expectations, often mirroring the performance of cloud-based models like ChatGPT. However, these expectations can be unrealistic, especially when running on modest hardware. In this article, we’ll explore why local LLMs struggle with broad questions and what tasks they excel at, helping you set realistic expectations for their capabilities.

The largest models that can practically run on a mini PC are 8B models, which is significantly smaller than the leading cloud-based models like DeepSeek-V3, published in 2024, boasting an impressive 671 billion total parameters. It’s likely that current frontier models are at least as large or even larger. This disparity highlights the limitations of local LLMs when compared to their cloud-based counterparts.

When I first installed a local LLM on my mini PC, I expected it to work similarly to ChatGPT. However, this wasn’t the case. The model took an age to respond to broad questions and even then produced disappointing results. It soon became apparent that using a local LLM required a different approach.

Leading cloud-based models excel at answering complex, open-ended questions by combining technical knowledge with historical context, economics, and more into coherent arguments. For instance, when I asked ChatGPT why IPv6 adoption has been slow despite being available for decades, it generated an instant response covering multiple ideas and forming a convincing argument.

I posed the same question to Llama 3.1 8B running in Ollama on my mini PC. The response took over 11 minutes to generate, and even then missed key factors such as how Network Address Translation (NAT) delayed the impact of IPv4 running out of addresses by allowing multiple devices to share a public address.

The results were disappointing because local LLMs struggle with broad questions that require lengthy answers. This doesn’t mean they’re useless; it simply means we need to use them in more targeted ways. By giving my LLM small, simple tasks, I’ve found it can be much more useful than initially thought.

Being specific makes a big difference when working with local LLMs. Unlike cloud-based models that often infer intent from vague prompts, local LLMs require clear instructions to produce accurate results. For example, I use my local LLM to take news from RSS feeds and summarize it into a concise report. This is a narrow task: take this information and condense it in natural language.

Another way I utilize my local LLM is by turning data pulled from Home Assistant, including calendar events and current weather, into a spoken morning briefing. The model has a clearly defined task: transform this collection of data into a coherent narrative. Tasks like these have clear boundaries, limited context, and predictable outputs, allowing the local LLM to excel.

The key is not having highly detailed prompts but rather narrowly defined tasks that don’t require extensive processing or inference. By setting realistic expectations for what our local LLMs can accomplish, we can unlock their full potential in specific areas.

When deciding which tasks to assign to a local LLM, it’s essential to consider whether the task is too broad or complex. If so, it may be better suited for cloud-based models like ChatGPT or Claude. However, by understanding what our local LLMs can handle effectively, we can leverage their strengths and make them an integral part of our workflow.

Local AI assistants have many benefits, including enhanced privacy, but they’re not a replacement for all tasks. By recognizing the limitations of local LLMs and using them in targeted ways, we can maximize their utility without overextending their capabilities.