Running LLM models in artificial intelligence, even the simplest ones, is inherently computationally demanding. Doing it locally on home hardware requires real beasts of power.
This is not an exaggeration. Running a modest model, such as the Llama 3.1 70B, with acceptable performance, requires a high-end GPU with at least 24GB of VRAM, such as the NVIDIA RTX 3090, and at least 64GB of RAM.
Not all developers and enthusiasts have access to this type of hardware. Therefore, the preferred option is to use one of the many ways to run them in the cloud. Will there be a cheap and accessible way to run an LLM locally?
This is where Exo presents itself as an innovative solution: it allows the workload to be distributed across multiple devices, maximizing the use of available resources and making AI more accessible to everyone.
What is Exo?
Exo is experimental software developed by Exo Labs that allows users to leverage the ability of multiple networked devices to run complex AI models.
Exo’s premise is simple but revolutionary: transform your regular devices, such as an iPhone, an iPad, a MacBook or even a Linux computer, into a powerful AI processing cluster without the need to invest in expensive graphics cards or AI accelerators.
This approach makes AI technology more accessible, especially for startups and small and medium-sized businesses (SMBs) looking to immerse themselves in the world of AI without a significant initial outlay on hardware.
The development of Exo is recent, and it is considered an experimental software in constant change. You can find more up-to-date information on their Github page (github.com/exo-explore/exo).
##How does Exo work?
Exo’s operation is based on the ability to split and distribute a large language model (LLM) among all available devices on the home network. Exo uses a memory scheme that assigns tasks based on the amount of memory each device has.
In this way, an AI model that would normally require a powerful GPU can be run efficiently using the sum of the capabilities of several less powerful devices.
The process begins by installing the Exo software on supported devices, which must have Python 3.12.0 or later.
Once installed and running, Exo automatically detects devices connected to the network and adds them to the cluster.
This is done through peer-to-peer connections, ensuring that all devices have equal resources and can participate in the AI processing task.
One of the most notable features of Exo is its dynamic model partitioning. This technology allows models to be optimally partitioned based on network topology and resources available on connected devices.
Instead of relying on a single device to run a large model, Exo allows multiple devices to collaborate to perform the task, maximizing efficiency and utilization of available hardware.
AI models supported by Exo
Exo supports a wide range of popular AI models, including Meta’s Llama 3. This is a large open source language model that has gained popularity for its performance and flexibility.
Additionally, with small changes, Exo allows you to integrate a ChatGPT-compatible API, making it easier to run AI models on personal hardware.
This broad support for different models makes Exo a versatile option for developers and businesses looking to experiment locally with AI without being limited by their lack of infrastructure.
Can GPT-4 be used with Exo?
GPT-4 is a proprietary model developed by OpenAI and is not available as open source. This means it runs on OpenAI servers and cannot be downloaded or run locally on personal devices.
Therefore, to use Exo and similar softwares, we are limited to language models that are open source, such as Llama 3 mentioned above.
Advantages of using a device cluster for AI
Using Exo to create an AI cluster with home devices offers several significant advantages:
The main advantage is cost reduction. Instead of investing in specialized hardware such as GPUs, Exo allows you to use devices you already own, such as smartphones, tablets and computers, to run complex AI models.
This democratizes access to AI, allowing more people and companies to participate in innovation.
By supporting multiple device types, Exo offers great flexibility in cluster configuration.
Users can take advantage of a variety of devices, allowing for greater customization based on specific project needs.
Thanks to its peer-to-peer architecture, Exo allows you to scale the cluster easily by adding more devices to the network. Each new device brings additional resources to the cluster, improving overall performance without the need to reconfigure the system.
Dynamic model partitioning ensures that the resources of each device are used optimally.
This means that even devices with limited capabilities can contribute significantly to the processing of large AI models, optimizing overall performance.
Limitations and final considerations
Being experimental software, Exo may still present bugs and stability issues. For example, there are limitations on iOS devices (iPhone and iPad) where support is limited and requires more development for widespread use.
On the other hand, the efficiency of the cluster will largely depend on the home network and the quality of the connected devices.
Finally, although Exo optimizes task distribution, many devices with very limited hardware may not offer the same performance as a cluster based on high-end GPUs.
However, its ability to transform everyday devices into a powerful AI cluster offers developers, startups and SMEs a powerful tool to explore and harness the potential of artificial intelligence.
Exo promises to be a key tool in the democratization of AI technology, allowing more people to participate in creating and experimenting with advanced AI models.
This post is also available in: