One of the biggest challenges of AI remains its dependence on specialized hardware, especially the expensive graphics processing units (GPUs), needed to run complex models. But a recent advance from Microsoft could change this reality.
Thanks to a new architecture called BitNet b1.58 2B4T, it is now possible to run high-performance AI models directly on CPUs, the most common and cheapest type of processor in personal computers and devices.
Let’s analyze how this model powered by Microsoft Copilot marks a before and after in the accessibility of AI, the techniques that make it possible and what we can expect for the future.
The Microsoft Model: BitNet b1.58 2B4T
On April 13, 2025, Microsoft released BitNet b1.58 2B4T, a next-generation AI model optimized to run on CPUs. With 2 billion parameters and trained with 4 billion tokens, BitNet uses an extreme quantization technique that reduces all its weights to just three values: -1, 0 and 1.
This radical simplification allows the model to operate with exceptional efficiency both in terms of memory and computing power. Surprisingly, it can run even on hardware like Apple’s M2 chip, without requiring a GPU.
Despite its lightness, BitNet does not sacrifice performance. In benchmarks like GSM8K and PIQA, it has outperformed much larger models, challenging the traditional notion that “bigger is better.”
Additionally, it is publicly available on Hugging Face under the MIT license**, making it easy for developers, researchers, and enthusiasts to access.
Not everything is perfect, of course: BitNet is not compatible with GPUs and depends on the custom bitnet.cpp environment developed by Microsoft. Still, it represents a giant step toward the democratization of AI, opening up new possibilities for those without specialized hardware.
Key techniques for running AIs on CPUs
Getting complex models like BitNet to work on CPUs requires more than good intention: it relies on a series of advanced techniques that optimize their efficiency. Here we explain the three main ones:
Pruning
Pruning removes model components (such as neurons or connections) that are not essential to its performance.
This reduces its size and speeds up inference time, something vital when working with CPUs that are not designed for heavy loads. Of course, its effectiveness depends on the hardware being able to handle these dispersed structures well.
Distillation
Distillation trains a smaller model, called the “student”, to imitate a larger one, the “teacher”.
In this way, a lighter model is obtained that maintains good performance, ideal for tasks such as classification or text analysis. Additionally, it can be combined with other techniques, such as quantization, to further improve efficiency.
Quantization
Quantization reduces the precision of the values used by the model. For example, you can go from using 32 bits to only 8, or even 1 bit as BitNet does.
This dramatically decreases model size and speeds up calculations, especially on CPUs that process low-precision operations well. It is one of the most effective strategies to bring advanced AI to common devices.
Beyond accessibility, these techniques also promote sustainability by reducing energy consumption and operating costs in the face of the demands of GPUs.
More sustainable and accessible AI
The development of BitNet is not an isolated case, but part of a broader trend in the technology industry. Companies like Ampere and NVIDIA are also working to optimize AI models for CPUs.
For example, Ampere, together with Wallaroo.AI, has managed to run Whisper, OpenAI’s speech recognition model, on CPUs with energy efficiency up to 3.6 times higher than that obtained on GPUs.
For its part, NVIDIA has incorporated techniques such as pruning and distillation into its NeMo platform, with the aim of creating lighter and more accessible models.
The research community is not far behind either. New methods such as AQLM and EfficientQAT are being explored, allowing model quantization to be performed with up to 2 bits of precision without compromising performance too much.
Another example is exo, a project that seeks to run large language models (LLMs) on CPU-based server clusters, which could significantly make access to high-level AI cheaper.
All of this points toward a future where AI models will not be limited to data centers with expensive racks of GPUs.
Instead, they could run directly on personal computers, laptops or even smartphones, thanks to improvements in both hardware and optimization techniques.
Decentralized AI, just around the corner
The release of BitNet b1.58 2B4T marks a turning point in the development of artificial intelligence models.
Its ability to run on CPUs without sacrificing performance is the result of innovations in techniques such as pruning, distillation and quantization.
This advance not only democratizes access to AI, but also promotes a more sustainable and efficient approach, with lower energy consumption and operating costs.
And if we add the efforts of other companies and the scientific community, the outlook becomes even more promising. We are entering a new era where AI will be truly ubiquitous, accessible and efficient. BitNet is just the beginning.
This post is also available in: