Exploring Local AI: A Journey of Discovery and Frustration

Local artificial intelligence (AI) is an emerging trend that allows users to run powerful models on their own devices, rather than relying on cloud services. This approach offers improved privacy and control over personal data, which can be a major draw for those hesitant to entrust sensitive information to the cloud.

The concept of local AI has gained significant attention in recent times, with tech giants like Apple highlighting its benefits for Mac desktops. The new RTX Spark Windows machines are also designed to leverage agentic AI capabilities, featuring up to 128GB of RAM and a range of specialized models.

Running local AI on personal devices can be both exciting and overwhelming due to the sheer number of available models and their varying parameters. Models with billions of parameters offer more advanced capabilities but require significant computational resources and storage space.

The author has embarked on an experiment using the M5 Ultra Mac Studio, which boasts 256GB of unified memory. This allows for running complex AI models locally without worrying about token costs associated with cloud services.

One such model is Hermes Agent, a self-hosted AI agent desktop app compatible with macOS, Windows, and Linux. It’s free to use when working with local LLMs (large language models), making it an attractive option for those seeking a more private approach to AI.

The author chose the Qwen 3.8 Flash Next model, which boasts 125 billion parameters and requires around 105GB of storage space. This decision was made possible by the M5 Ultra Mac Studio’s ample memory capacity.

Using Hermes’ straightforward onboarding interface and model picker, the author quickly got up and running with the chosen model. Controlling it via a Telegram bot also proved to be relatively simple.

One of the first tasks assigned to the local AI was generating a daily morning briefing using email and calendar data. While this initial attempt failed due to macOS being asleep during the scheduled job time, subsequent attempts were successful once the issue was addressed.

The author then used Hermes to reorganize their Steam library by genre, leveraging its ability to access installed games through the Steam client. This process required registering a Steam web API key and granting permissions for the AI to take actions on behalf of the user.

Data analysis is another area where local AI has proven useful, with the author using it to crunch financial records and create a spec comparison spreadsheet for a new laptop. These tasks are particularly appealing due to their sensitivity and the need for control over personal data.

The author’s current focus lies in automating benchmark tests for laptops, which can be an incredibly time-consuming process when done manually. While progress has been slow, the potential benefits of automation make this effort worthwhile.

Throughout this journey, the author remains cautious and vigilant about the capabilities and limitations of local AI. They emphasize that these models are not magical assistants but rather tools to augment human productivity and decision-making.

The author’s goal is to explore the full range of possibilities offered by local AI without becoming overly reliant on it or expecting too much from its capabilities. By doing so, they aim to provide a balanced perspective on this emerging technology.

Despite initial frustrations with model selection and task execution, the author remains optimistic about the potential benefits of local AI for personal productivity and data management.