AI Tools for Businesses Get a Boost from New LFM2.5 Q4_0 Checkpoints

In recent years, researchers have been working to improve the performance of AI tools on edge devices. They’ve made significant strides by developing techniques like quantization-aware distillation.

A breakthrough in this area involves creating high-precision teacher models that can be distilled into more efficient student models with minimal loss of accuracy. This is a crucial step towards deploying reliable and accurate language models in real-world scenarios.

The LFM2.5 Q4_0 checkpoints were trained using quantization-aware distillation, which has yielded impressive results in various benchmark tests. When compared to their native counterparts, these checkpoints retain 97% of their baseline performance for BF16 average accuracy lost due to quantization.

Several benchmarks were used to test the LFM2.5 Q4_0 checkpoints, including GPQA Diamond, MMLU-Pro, IFEval, IFBench, Multi-IF, and BFCLv4. The results showed that these QAD checkpoints outperformed their native counterparts in all four models tested.

To further evaluate the performance of these checkpoints on real hardware, researchers measured decode throughput for each model across devices like MacBook Pro, NucBox EVO-X2, Samsung Galaxy S26 Ultra, and Raspberry Pi 5. The results indicated that the QAD checkpoints maintained accuracy while achieving higher decode throughputs compared to their native counterparts.

To use the LFM2.5 Q4_0 checkpoints, simply load them into llama.cpp or any runtime supporting GGUF Q4_0 artifacts. For example, users can run a command using llama-cli: ‘llama-cli -hf LiquidAI/LFM2.5-350M —hf-file LFM2.5-350M-QAD-Q4_0.gguf -p What is C. elegans?’

The QAD GGUFs are now available on Hugging Face for use in various AI applications, including those requiring high-performance language models on edge devices. This move enables developers to tap into these powerful tools and enhance their projects.