Distributed local inference

Every machine you own.
One endpoint.

shoal puts the GPUs and Macs you already have behind a single OpenAI- and Anthropic-compatible endpoint. Workers run beside your model servers and dial out to it, so nothing needs a port opened.

Self-host shoal View on GitHub

$ curl -fsSL https://shoal.school/install.sh | sh

Apache-2.0

Change one line.
Keep your code.

Anything that speaks the OpenAI or Anthropic API already speaks shoal: SDKs, chat apps, coding agents, your own scripts. Point the base URL at your server and use a model any worker has.

app.py

client = OpenAI(-    base_url="https://api.openai.com/v1",+    base_url="https://shoal.example.com/api/v1",     api_key=os.environ["SHOAL_API_KEY"],)client.chat.completions.create(model="qwen3-32b", ...)

How it works

Your apps

openai & anthropic sdkschat apps, coding agentscurl, scripts, cron

shoal server

Knows which worker has which model loaded, and how many slots are free.Routes each request to a worker that can take it now.Holds your org, members and keys.

Workers

desktop · ollama● 3/4laptop · lm studio● 1/2homelab · vllm○ 0/4

Outbound only

Workers open a WebSocket to the server. Laptops behind NAT, boxes on another network, a GPU at a friend’s place: if it can reach the server, it can join.

Routes to what’s warm

Requests go to a worker that already has the model loaded and a free slot, so you aren’t paying a cold load for every call.

Bring any backend

Presets for Ollama, LM Studio, llama.cpp, llama-swap, vLLM and Lemonade. Anything else, point --backend at it.

More than chat

Chat, vision, embeddings, image generation and text-to-speech sit behind the same API, on whichever machine has the hardware for them.

Slots, not guesswork

Each worker declares how many jobs it can run at once. The server respects it, so a laptop never gets the load meant for a workstation.

Share with your team

Organizations, members and API keys. Everyone adds machines to one pool and calls it with their own key.

Works with what you runOllamaLM Studiollama.cppllama-swapvLLMLemonadeor any URL with --backend

Your whole fleet at a glance

The dashboard shows what's connected, what's loaded and what's busy, and warns you when a kind of model only lives on one machine.

Get started

01Run the server

$ curl -fsSL https://shoal.school/install.sh | sh

Needs Docker. Run it anywhere your apps and workers can both reach: it's the only piece that needs an address. On Windows: irm https://shoal.school/install.ps1 | iex

02Add workers

$ npx @shoal.school/cli worker start \
    --server wss://shoal.example.com/ws/workers \
    --token sk_…

On each machine with a model server. Connect worker in the dashboard fills in the server and token for you.

Star on GitHub

Every machine you own.One endpoint.

Change one line.Keep your code.