EXO

Exo Cluster is an open-source software framework that turns multiple everyday devices into a unified, distributed AI supercluster for running large language models (LLMs) locally.

Core Concept

Exo automatically discovers and connects devices on your local network - like laptops, smartphones, tablets, Raspberry Pis, or Macs and NVIDIA GPUs - pooling their memory and compute power. It shards heavy AI models (e.g., LLaMA or Mistral) across these devices using techniques like tensor parallelism, where model components are split for parallel inference, achieving speedups like 1.8x on two devices or 3.2x on four.

How It Works

Exo maps your hardware topology, assessing each device’s resources (memory, compute, network speed), then intelligently distributes model layers or tensors to avoid bottlenecks. No manual setup needed: launch the software, and devices auto-join the cluster for seamless local AI inference, keeping data private without cloud reliance.

Key Features

Device Agnostic: Supports iPhone, Android, Mac, Linux, Windows, Raspberry Pi—anything with compatible runtimes like tinygrad or MLX.
Auto-Parallelism: Topology-aware splitting optimizes for your setup, scaling with more devices.
Easy Onboarding: Simple install; Exo handles discovery and load balancing.
Privacy-Focused: All processing stays on-device, ideal for cost-free, high-capacity AI.