OdyssAI-X and CoeOS
In one sentence: OdyssAI-X is an inference engine; CoeOS is a product with its own router, the CoeOS box. Each works without the other. Together, the engine serves the levels of CoeOS that stay on your premises: confidential work in Hybrid, everything in Local.
How they connect
Section titled “How they connect”The CoeOS box declares the engine as a provider of type local, marked safe. For Hybrid and Local, the engine is linked to your CoeOS instance by an encrypted tunnel. The box then:
- routes confidential conversations to the engine’s models through its Local table, composed from safe models only;
- in Local, routes everything there;
- only binds an axis to a model the engine reports as loaded.
In a managed CoeOS instance, we do this wiring for you.
What the engine exposes to the box
Section titled “What the engine exposes to the box”| Surface | Used for |
|---|---|
POST /v1/chat/completions, POST /v1/messages | the relay itself (OpenAI and Anthropic wire formats; the box forwards the client’s format) |
GET /v1/models with x_odyssai blocks | which models are ready, their capabilities (tools, vision, context, kind) |
GET /.well-known/inference-engine.json | the capability contract |
enable_thinking | the only thinking flag this engine reads. The box translates thinking/reasoning from cloud-style clients into it |
reasoning_content in deltas | reasoning is split from the answer per model; the box passes both through |
Decision models served by the engine can also act as the box’s decider for confidential work, so that even the classification of a request stays on site.
The natural backend: a replica pool
Section titled “The natural backend: a replica pool”For a team, the box’s axes are best served by a replica pool: one full copy of the model per Mac, continuous batching inside each, session affinity so a conversation keeps its KV cache. Measured in Replica mode on four Mac Studio M3 Ultra with 256 GB each: 518 tok/s aggregate for MiMo V2.6 Flash at 96 concurrent requests, 985 tok/s for Qwen3 Next 80B-A3B 8-bit at 256. On three 96 GB nodes, Qwen3 Coder Next 6-bit reaches 650 tok/s at 96 (full measurement protocol).
When to call the engine directly
Section titled “When to call the engine directly”- You want a specific model, not “the best per skill” — call the engine’s
/v1with the model alias. - You are benchmarking a model (routing would hide which one you measured).
- You do not use CoeOS: the engine is a complete OpenAI- and Anthropic-compatible server on its own.
Read next
Section titled “Read next”- Architecture overview — the engine on its own.
- Multi-user serving — replica pools and continuous batching.
- What CoeOS is · The router · Confidentiality and Guardian.