Install the engine

One server drives the nodes over SSH. The nodes hold the models and run the inference. Nothing runs where you did not put it.

This page installs OdyssAI-X, the engine. It is a condensed version of the README and of AGENTS.md, the step-by-step checklist an AI agent (or you) can follow; when the two disagree, the repository wins. For the client and the router, see Nemo and the CoeOS box.

macOS on Apple Silicon only. Two roles:

Role	What it does	Machine
Server	Runs the orchestrator container and the dashboard on `:8000`. Routes requests, never runs inference itself.	Any Apple Silicon Mac with Docker Desktop; a Mac mini is enough. Can also be a node.
Node	Holds the model files and runs the MLX runner the server starts over SSH.	Each Mac that serves models.

Prerequisites

Nodes: Python 3.11 (brew install python@3.11); Python 3.12 as well if the node should serve vision models. Remote Login enabled (System Settings → General → Sharing) and a passwordless SSH key from the server (ssh-copy-id).
Server: Docker Desktop.
Memory: enough unified memory for the model you intend to serve — weights about the file size, plus around 20 % for the KV cache and scratch.
Disk: models live per node under <models_dir>/<org>/<name>/. Volumes are not shared between nodes.

1. Bootstrap each node

One command per node, idempotent:

scripts/bootstrap-node.sh user@node.lan                  # default models dir: ~/mlx-models
scripts/bootstrap-node.sh user@node.lan <models_dir>

It checks SSH and Python, creates ~/mlx-cluster with a pinned virtual environment, copies the runner and the patches, installs the vendored model modules and the patched JACCL library, runs a smoke import, and sets up the optional vision environment. Re-run it after any pip upgrade on the node: upgrading mlx-lm silently removes the vendored modules, upgrading mlx puts the stock JACCL back.

2. Pin the GPU memory budget

A persistent wired-memory limit on each node. It needs the node’s password, so it is a deliberate step:

scripts/wired-limit/install.sh user@node.lan 250880      # e.g. 245 GB on a 256 GB Mac

3. Describe the cluster, start the orchestrator

On the server:

mkdir -p ~/.odysseus && cp config/topology.example.yaml ~/.odysseus/topology.yaml
$EDITOR ~/.odysseus/topology.yaml      # ssh targets, models_dir, backend (ring | jaccl)
docker compose up -d
curl -s http://localhost:8000/health   # {"status":"idle","version":"…"}

Single Mac? Use host.docker.internal as the SSH target — the container cannot reach the Mac through localhost. The example topology shows it.

4. First model

Download from the dashboard (http://localhost:8000/, Models → Download; org/name or org/name/<quant-subfolder>), load it on a cluster, and chat:

curl -s http://localhost:8000/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"<alias from /v1/models>","messages":[{"role":"user","content":"hi"}]}'

A streaming answer ending in data: [DONE] means the engine is installed.

5. Choose how the Macs work together

You want	Create the cluster as	Read
Serve many users on a model that fits one Mac	replica — one full copy per node, continuous batching optional	Multi-user serving
Serve a model too big for one Mac	distributed — split across nodes over TCP (`ring`) or RDMA over Thunderbolt 5 (`jaccl`)	The cluster, Inference modes
Serve vision models	VLM — a dedicated `mlx-vlm` environment, fronted by the same API	HTTP API

For throughput, create the cluster with Kind = replica in the dashboard and load it with batching:

curl -s -X POST http://localhost:8000/admin/clusters/<id>/load -H 'content-type: application/json' \
  -d '{"model":"<org>/<name>","nodes":4,"batch":true}'

Optional — RDMA over Thunderbolt 5

For distributed pools that need the bandwidth: cable the nodes in a full mesh, run scripts/rdma-onboard.sh on each node (it provisions the Thunderbolt network), then declare backend: jaccl and the port wiring in topology.yaml. Every edge is checked before a load. TCP ring needs none of this and always works.

Admin access

/admin/* is open on a default LAN install: whoever can reach :8000 is the operator. To require a Bearer token on a deployment reachable from outside, set the admin token in the environment (see the repository’s docker-compose.yml). /v1/* stays public.

When it does not work

Symptom	Fix
`docker compose` exits immediately	A bad `~/.odysseus/topology.yaml` or an unreachable SSH target; the boot log names the key.
Engine up, `/v1/models` empty	No model loaded yet.
A node loses its model modules after a `pip` upgrade	Re-run `scripts/bootstrap-node.sh` for that node.
RDMA edges fail their check	A cable is unplugged, a node is off, or the node was never onboarded with `scripts/rdma-onboard.sh`.
`errno 16 / 96 / 2` after a while on a JACCL cluster	Queue-pair degradation, a known MLX behaviour; reboot the nodes (the dashboard has Reboot all).

More in Troubleshooting and Cluster health.