Skip to content
OdyssAI

Architecture overview

The stack stays replaceable because the protocols stay standard. Every layer speaks a documented surface; you can swap one without disturbing the others.

Three layers

Section titled “Three layers”
Layer 01 — Clients     Nemo · CodeOS · IDE agents · OpenAI / Anthropic SDKs · curl
                                  ↓  OpenAI or Anthropic wire format
Layer 02 — Router      the CoeOS box     one address, one competence axis per request
                                  ↓
Layer 03 — Engine      OdyssAI-X         your Apple Silicon cluster (this section)
                                  ↓
Ground                 Apple Silicon · cloud providers with your keys

Layer 01 — Clients. Anything that speaks OpenAI or Anthropic HTTP. Nemo, the desktop client; CodeOS; an IDE agent; the OpenAI Python SDK pointed at an address. This layer is not part of the engine: it is what the engine is built to serve.

Layer 02 — Router. The CoeOS box. It classifies each request on a competence axis and relays it to the model bound to that axis — on this engine, or at a cloud provider. The box never runs inference. A client may also skip it and call the engine directly.

Layer 03 — Engine. OdyssAI-X. One control plane (:8000, a Docker container) that SSHes into the nodes to start long-lived MLX runners, and exposes everything they serve behind one /v1. It never runs inference itself either: a Mac mini can drive a rack of Mac Studios.

Ground. Apple Silicon, the hardware MLX was written for. Cloud providers sit at the same level, behind the box, with your keys.

Three ways to put Macs to work

Section titled “Three ways to put Macs to work”
ModeWhat it doesWhen
DistributedOne model split across N nodes — pipeline or tensor parallel, over RDMA on Thunderbolt 5 (jaccl) or a TCP ring (ring).The model does not fit one Mac.
ReplicaOne full copy of the model per node, least-busy dispatch, session affinity, optional continuous batching, self-healing replicas. No collective between nodes, so no node cap.The model fits one Mac and you need users, not size.
VLMVision models served through a dedicated mlx-vlm environment, fronted by the same API.Images in the prompt.

Any number of clusters can coexist in topology.yaml, with different models loaded side by side, all behind one address. Details: Inference modes, Multi-user serving, The cluster.

Boundaries

Section titled “Boundaries”
  • OdyssAI-X is not an exo fork. It is a control plane on Apple’s mlx and mlx-lm, with vendored model modules and a patched JACCL for RDMA. The collective communication comes from MLX.
  • The engine owns no memory and no routing policy. Memory lives in the client (Nemo keeps a local Markdown vault); routing lives in the box. The engine serves models and reports what it can do.
  • The orchestrator never runs a model. Nodes do. The server can be a node, but the roles stay distinct.

The capability contract

Section titled “The capability contract”

Every engine advertises what it can do at /.well-known/inference-engine.json, and every model carries an x_odyssai block in /v1/models (context length, maximum output, tools, vision, kind). A client such as Nemo, or the CoeOS box, reads them during pairing instead of hard-coding what is behind the address. Full contract →

The HTTP surface

Section titled “The HTTP surface”

POST /v1/chat/completions and POST /v1/messages are the two endpoints that matter; /v1/responses, /v1/models, /v1/messages/count_tokens and /health complete the public surface. /admin/* manages clusters, loads and models; it is open on a default LAN install and can be locked with a Bearer token.

Three extensions on top of the standard schemas:

  • session_id — optional string. Scopes the KV prefix cache to a conversation; on a replica pool it also keeps the conversation on the replica that holds its cache.
  • enable_thinking — boolean. The only thinking flag the engine reads; the box translates cloud-style thinking and reasoning fields into it.
  • reasoning_effort — sets the reasoning budget on models that expose one.

Reasoning is split from the answer per model (reasoning_content in the stream), and the usage block reports prompt_tokens_details.cached_tokens, the hit count on the prefix cache. Cloud aliases such as or:<org/model> pass through to a provider declared in the dashboard. Full reference →

What runs where, in one screen

Section titled “What runs where, in one screen”
Nemo · CodeOS · SDKs
   │  HTTP (OpenAI / Anthropic)
   ▼
the CoeOS box                     :4600   routes per axis; "CoeOS" is its model id
   │
   ▼
OdyssAI-X orchestrator            :8000   FastAPI in Docker, dashboard included
   ├── cluster A   distributed · ring    → runner × N nodes over TCP
   ├── cluster B   distributed · jaccl   → runner × N nodes over Thunderbolt 5 RDMA
   ├── cluster C   replica · batch       → one full copy per node, several conversations each
   ├── cluster D   vlm                   → mlx-vlm, proxied
   └── alias or:*  cloud passthrough     → a provider declared in the dashboard

The box sees one catalogue. The orchestrator hides the topology.

Section titled “Read next”