Architecture overview

OdyssAI-X is our own inference engine for Apple Silicon, built on MLX and mlx-distributed. It stands on its own: any client that speaks the OpenAI or Anthropic API can use it, with or without CoeOS. With CoeOS, it is the engine that serves the Hybrid and Local levels. The two are independent and complementary.

Where it fits

any OpenAI / Anthropic client          CoeOS (web app · CoeOS Desktop)
  (SDK, IDE agent, curl…)                         │
              │                            the CoeOS box  (router)
              │                                   │  a `local` provider, marked safe
              ▼                                   ▼
                    OdyssAI-X   (control plane :8000, one /v1)
                                  │
                       your Apple Silicon nodes

The engine. One control plane (:8000, a Docker container) that SSHes into the nodes to start long-lived MLX runners, and exposes everything they serve behind one /v1. It never runs inference itself: a Mac mini can drive a rack of Mac Studios.

Directly. Point any OpenAI- or Anthropic-compatible client at the engine and pick a model by its alias.

With CoeOS. The CoeOS box declares the engine as a local provider, marked safe, and routes confidential work to it (Hybrid), or all work (Local). See OdyssAI-X and CoeOS.

Ways to put Macs to work

Mode	What it does	When
Single node	One Mac, one model, the whole unified memory of the machine.	The model fits one Mac.
Distributed	One model split across N nodes — pipeline or tensor parallel, over RDMA on Thunderbolt 5 at 120 Gb/s (`jaccl`) or a TCP ring (`ring`).	The model does not fit one Mac.
Replica	One full copy of the model per node, least-busy dispatch, session affinity, optional continuous batching, self-healing replicas. No collective between nodes, so no node cap.	The model fits one Mac and you need users, not size.
VLM	Vision models served through a dedicated `mlx-vlm` environment, fronted by the same API.	Images in the prompt.

Any number of clusters can coexist in topology.yaml, with different models loaded side by side, all behind one address. Details: Inference modes, Multi-user serving, The cluster.

What the models get

The latest open models, supported as they come out, in their best quantization (never below Q6, or Q4 when a model was trained for it).
Multimodal input on vision models.
Speculative decoding: DSpark and native MTP, plus drafter models.
Decision models, served alongside chat models.

Boundaries

Built on Apple’s mlx and mlx-lm, with vendored model modules and a patched JACCL for RDMA. The collective communication comes from MLX.
The engine owns no memory and no routing policy. Memory lives in the client; routing, when you want it, lives in the CoeOS box. The engine serves models and reports what it can do.
The orchestrator never runs a model. Nodes do. The server can be a node, but the roles stay distinct.

The capability contract

Every engine advertises what it can do at /.well-known/inference-engine.json, and every model carries an x_odyssai block in /v1/models (context length, maximum output, tools, vision, kind). A client such as the CoeOS box reads them during pairing instead of hard-coding what is behind the address. Full contract →

The HTTP surface

POST /v1/chat/completions and POST /v1/messages are the two endpoints that matter; /v1/responses, /v1/models, /v1/messages/count_tokens and /health complete the public surface. /admin/* manages clusters, loads and models; it is open on a default LAN install and can be locked with a Bearer token.

Three extensions on top of the standard schemas:

session_id — optional string. Scopes the KV prefix cache to a conversation; on a replica pool it also keeps the conversation on the replica that holds its cache.
enable_thinking — boolean. The only thinking flag the engine reads; the CoeOS box translates cloud-style thinking and reasoning fields into it.
reasoning_effort — sets the reasoning budget on models that expose one.

Reasoning is split from the answer per model (reasoning_content in the stream), and the usage block reports prompt_tokens_details.cached_tokens, the hit count on the prefix cache. Cloud aliases such as or:<org/model> pass through to a provider declared in the dashboard. Full reference →

What runs where, in one screen

clients (SDKs, agents, the CoeOS box)
   │  HTTP (OpenAI / Anthropic)
   ▼
OdyssAI-X orchestrator            :8000   FastAPI in Docker, dashboard included
   ├── cluster A   single node            → one runner on one Mac
   ├── cluster B   distributed · jaccl    → runner × N nodes over Thunderbolt 5 RDMA
   ├── cluster C   replica · batch        → one full copy per node, several conversations each
   ├── cluster D   vlm                    → mlx-vlm, proxied
   └── alias or:*  cloud passthrough      → a provider declared in the dashboard

Clients see one catalogue. The orchestrator hides the topology.