---
title: "@octabits-io/models"
summary: "The model layer: a provider registry with residency, keys turned into AI SDK models that explain a vendor's refusal, lenient JSON from a model's answer, a per-run circuit breaker and a connection test, providers an administrator manages, a model served by a process that connects out, prices and quotas — and, on the octaflow engine, AI steps, agents and case loops."
install: "pnpm add @octabits-io/models"
availability: "Available from November 2026"
version: "0.1.0"
license: "MIT"
related:
  - name: "@octabits-io/postgres"
    url: "https://octabits.io/platform/postgres.md"
  - name: "@octabits-io/result"
    url: "https://octabits.io/platform/result.md"
  - name: "octaflow"
    url: "https://octabits.io/platform/octaflow.md"
canonical_url: "https://octabits.io/platform/models"
---

# `@octabits-io/models`

The model layer: **which models an app may call, how it reaches them, and what
it does with what they return**. A provider registry that carries each vendor's
region and legal entity, so residency is data a picker and the disclosure gate
read; a key turned into an AI SDK model that explains a vendor's refusal by
key and hint; lenient JSON from a model's answer; a circuit breaker per run and
a connection test; providers an administrator manages as rows, with their
models' declared capabilities; a model served by a process that connects
*out*, for a machine the app cannot reach; prices and quotas. And, on the
[octaflow](https://octabits.io/platform/octaflow.md) engine, AI steps, the model decider, agents
that end by choosing an exit, and case loops in which a manager agent chooses
every next move.

```bash
npm i @octabits-io/models zod     # + ai @ai-sdk/provider for ./ai-sdk, octaflow for ./octaflow
```

```ts
import { AI_PROVIDERS, createModelProviders } from '@octabits-io/models';
import { createModelResolver, withCircuitBreaker } from '@octabits-io/models/ai-sdk';
import { createPgModelProviderStore } from '@octabits-io/models/postgres';

// The providers an administrator manages: rows, a key that crosses a cipher seam.
const providers = createModelProviders({ store: createPgModelProviderStore({ executor: pool }), cipher });
// A row becomes an AI SDK model, each call shaped to what the model declares.
const resolver = createModelResolver({ providers });
const resolved = await resolver.languageModel({ provider: 'mistral-eu', require: { vision: true } });
if (resolved.ok) {
  const model = withCircuitBreaker(resolved.value.model);   // a rejected key fails the run once, not every step
  // resolved.value.destination — the provider's id and region — is what the disclosure gate reads before the call
}
```

| Entry | Peer | What it is |
|---|---|---|
| `.` | `zod` | the contracts, no SDK behind them: the provider registry (`AI_PROVIDERS`, `EMBEDDING_PROVIDERS`, `resolveEndpointResidency`), model capabilities, prices (`DEFAULT_MODEL_PRICING`, `createCostEstimator`), a vendor's refusal explained (`explainProviderError`), lenient JSON (`parseModelJson`), the managed providers (`createModelProviders`, `createInMemoryModelProviderStore`) |
| `./ai-sdk` | `ai`, `@ai-sdk/provider`, the vendors (all optional) | a key becomes a model: the vendor factories (async — each vendor is loaded on first use), OpenAI-compatible endpoints and gateways, Ollama's native API (`createOllama`), the resolver over the managed providers (`createModelResolver`), the refusal-aware and instrumented wrappers, `withJsonRepair`, `withCircuitBreaker`, `testProvider` |
| `./runner` | `@ai-sdk/provider` (optional) | a model served by a process that connects out: the hub (`createModelRunnerHub`), the app-side models, the runner loop and its HTTP client, the store contract |
| `./octaflow` | `octaflow` (optional) | the model layer on the engine: `createAiWorkflowHooks` (an instrumented model per step, cost, quota, a `usage` sink), `defineAiStep`, `createModelDecider`, agents (`defineAgentStep`, `runAgentToExit`), case loops (`defineCaseLoop`), the quota and usage-rollup services |
| `./postgres` | `@octabits-io/postgres` (optional) | the provider store and the runners' three tables as plain SQL over the structural executor, with their DDL |
| `./drizzle` | `drizzle-orm` (optional) | the same four tables as spreadable column sets |

The root imports nothing but `zod` and `@octabits-io/result`, so a browser
bundle — a provider picker, a residency badge — imports it alone; the
registry file itself imports nothing at all. The boundary lint holds both.

## When not to use it

- **You call one vendor with one key and never change it.** The AI SDK's own
  provider is enough; this package earns its place when an administrator
  picks the provider, when residency is a question someone asks, or when a
  model lives somewhere the app cannot reach.
- **You want a gateway in front of every call.** That is LiteLLM's job (with
  `@octabits-io/disclosure`'s guardrail in front of it); the
  OpenAI-compatible gateway provider here talks *to* one.
- **You want billing.** Prices here are an estimate per step for quotas and
  the run view; what a customer is charged is `@octabits-io/metering`'s.

## The provider registry and residency

`AI_PROVIDERS` and `EMBEDDING_PROVIDERS` list the vendors with a fixed
residency — the processing region, the legal entity and its country — that a settings
page iterates and the subprocessor annex (`infra/SUBPROCESSORS.md`) is
generated from. An endpoint you run yourself (vLLM, llama.cpp, mlx_lm.server)
is not in the registry: its residency is the deployment's declaration, checked
by `resolveEndpointResidency`. A new provider states its residency here or it
does not exist.

## Managed providers

`createModelProviders({ store, cipher })`: providers as rows, each with its
kind, server, region, a key that crosses a cipher seam (a read carries a masked hint of it, never the key; `reveal` decrypts it for the call that needs it), tags, and its models with declared capabilities —
temperature, JSON mode, vision, video, tools, embedding. `createModelResolver`
(`./ai-sdk`) builds the AI SDK model from a row (`languageModel(ref)`,
`embeddingModel(ref)`), shapes each call to what the model declares, hands
the host its `destination` for the disclosure guard, and rebuilds after a
write. The table is `model_provider`; it carries no scope
column, because the providers are the deployment's — a host that lets each
customer bring their own keys declares the table with a scope column beside
these columns.

## What a model returns

- **A vendor's error, explained.** `explainProviderError(error)` reads any
  error a vendor or the network produced as a key — `provider_auth`,
  `provider_billing`, `provider_model_not_found`, `provider_rate_limited`,
  `provider_unavailable`, `provider_unreachable`, `budget_exceeded` — with a
  sentence, a hint a person can act on, and whether it is fatal: whether
  further calls in the same run are pointless. Keys are masked in what it
  keeps.
- **A declined generation says so.** A model can decline with HTTP 200 and an
  empty answer; `createRefusalAwareModel`, applied by every factory, turns
  that into an error instead of an empty string.
- **A circuit breaker per run.** `withCircuitBreaker(model)`: the first fatal
  error trips it, `onFatal` is told once, and every further call fails at
  once with the same error and no request. A rate limit or an outage does not
  trip it — the step's retry policy is the tool for those.
- **Lenient JSON.** A local or smaller model that fences its JSON, wraps it in
  prose or answers one object per image is read as the JSON it meant:
  `withJsonRepair(model, { merge: true })`, or `parseModelJson(text)` on a
  string. `retryWithoutJsonMode` and `convertWith` buy one more request each
  when the first answer cannot be read.
- **A connection test.** `testProvider(modelOrProvider)` sends one prompt and
  reports the model and the time it took, or the explanation; it never throws
  for anything the vendor or the network did. `testOllamaModel` asks an
  Ollama server whether the model is installed.

## Model runners — a model that connects out

A model the app cannot reach: a laptop with the model loaded for a demo, a GPU box behind
someone's firewall, a machine that is only on when it is needed. The runner never opens a
port. It connects *out* to the app with a token, subscribes with its tags, claims the model
calls that carry those tags and posts the answers back — the shape of a self-hosted CI
runner. The mailbox is Postgres, because the call runs in the worker and the door the runner
knocks on runs in the API; a request row lives only while the call is in flight.

```ts
// the app — once, beside the providers
import { createModelRunnerHub } from '@octabits-io/models/runner';
import { createPgModelRunnerStore } from '@octabits-io/models/postgres';

const runners = createModelRunnerHub({ store: createPgModelRunnerStore({ executor }) });
const resolver = createModelResolver({ providers, runners, wrap });

// an administrator: the provider row carries tags, the runner gets its token once
await providers.save({ id: 'mac', kind: 'runner', label: 'The Mac', region: 'self-hosted', tags: ['qwen'], models: [{ id: 'qwen3:32b', settings: { numCtx: 16_384 } }] });
const { token } = (await runners.registerRunner({ id: 'gpu-box', label: 'The GPU box', tags: ['qwen'] })).value;

// the API: the door, behind the host's rate limit
import { createModelRunnerHandler } from '@octabits-io/server/fetch/models';
export const POST = compose(withErrorMapping())(createModelRunnerHandler({ prefix: '/api/model-runner', hub: runners }));

// the runner, on the machine with the model — the bin, or the loop in your own process
//   npx -p @octabits-io/models model-runner --url https://app.example/api/model-runner --token omr_… --ollama http://localhost:11434
import { createHttpRunnerClient, runModelRunner } from '@octabits-io/models/runner';
import { createOllama } from '@octabits-io/models/ai-sdk';
await runModelRunner({
  client: createHttpRunnerClient({ url: 'https://app.example/api/model-runner', token }),
  backend: createOllama({ baseUrl: 'http://localhost:11434' }),
});
```

**`model-runner`**, the bin, is the runner as a process: `--url` and `--token` (or
`MODEL_RUNNER_URL` / `MODEL_RUNNER_TOKEN`), one backend — `--ollama <url>` for Ollama's
native API, `--openai-compatible <url>` (with `--api-key`) for mlx_lm.server, vLLM or llama.cpp
through the optional `@ai-sdk/openai-compatible` peer — and `--concurrency`. At start it says
who the token belongs to, which models the app's provider rows declare for its tags (the door
tells it) and which of them the local server has, with the `ollama pull` for a missing one;
`--check` does that and exits. A door it cannot reach is retried with a backoff, forever.
The skill `serve-a-model-from-a-runner` walks through the four pieces.

A runner may claim a request when it has **every tag** of the request — the provider row's
tags — as a CI job's labels must all be on the runner. A call with nobody online for its
tags fails at once with `provider_unreachable`, so `withCircuitBreaker` trips and
`testProvider` reports it in milliseconds instead of a timeout; whether another provider is
tried is the host's resolver's choice, through the same disclosure gate. A hub made with
`waitForRunnerMs` gives a runner that long to come (back) — a laptop waking up, a runner
restarting — before the call fails the same way; a call can set its own (0 for a probe). A
claim lasts 30 s without a heartbeat; an expired claim is requeued once and failed the second
time. The waiting caller expires its own request's lease, so a request held by a runner that
died is requeued even when no other runner comes to claim, and fails with `no_runner` once
nobody is online to take it over, instead of sitting until the call's deadline. The
region is the administrator's declaration about where runners with these tags are, as for
every provider, and the disclosure gate reads it the same way.

The wire is the AI SDK's own call and answer as JSON — binary file data as base64, a URL
restored on the other side — so the runner's backend is any `LanguageModelV4`: `createOllama`
from this package, or `@ai-sdk/openai-compatible` for mlx_lm.server, vLLM and llama.cpp.
`streamText` streams: the runner runs `doStream` and posts the parts in chunks as they come
(every 100 ms, or 64 parts), a third table carries them, and the app emits each chunk as it
lands; a chunk the door refuses tells the runner the caller gave up, and it cancels the
stream. A runner on the older wire answers a stream whole, and the app emits that as parts
too. Files reach the runner as bytes, since the runner model declares no URL support and the
SDK downloads on the app's side. Embeddings go the same way.

## On the octaflow engine — `./octaflow`

`createAiWorkflowHooks({ modelResolver, usageRecorder, quotaPolicy?, usage?, costEstimator? })`
is the engine's `hooks`: each AI step gets an instrumented model
(`ctx.context.model`), the tokens it used are captured per step and priced,
a `usage` sink books each step into `@octabits-io/metering` (by shape — this
package never imports it), and a quota policy refuses a run before it starts. `defineAiStep` and
`buildAiWorkflow` type the steps over that context.

- **Decisions with a model.** `createModelDecider()` is a decider for
  octaflow's decision steps: the model answers with a choice and a confidence,
  and below the threshold a person settles it.
- **Agents.** `defineAgentStep` / `runAgentToExit`: an agent's tool loop that
  ends by choosing one of the step's exits, each with its own schema, so the
  policy that refuses a choice is a schema, not a prompt.
- **Case loops.** `defineCaseLoop`: a manager agent chooses every next move of
  a case, each move its own workflow ending in the same tail (result → decide
  → next); policy refuses a choice with its reason inside the exit's schema, a
  move with a step that could fail is refused at definition time, and a
  manager that cannot decide closes the case for a person.
- **Quotas and rollups.** `createAiQuotaService` and
  `createAiUsageAggregationService` over structural `AiQuotaStore` /
  `AiUsageStore` seams — the host keeps the SQL.

The engine's own guide to this layer is
[the AI add-on](https://octaflow.octabits.io/extending/ai/).

## The tables

`./postgres` owns four tables and ships their DDL: `modelProviderDdl()`
(`model_provider`) and `modelRunnerDdl()` (`model_runner`,
`model_runner_request`, `model_runner_chunk`). Each is idempotent
`CREATE TABLE IF NOT EXISTS`, takes a `schema`, and is meant to be pasted into
your own migration system. The stores reach the database only through
`@octabits-io/postgres`'s `SqlExecutor` — `pg`, PGlite and postgres.js all
fit — and every write is one statement.

`./drizzle` ships the same tables as `modelProviderColumns`,
`modelRunnerColumns`, `modelRunnerRequestColumns` and
`modelRunnerChunkColumns`, for a host that lets drizzle-kit own its
migrations. The unique index on a runner's token hash and the chunk table's
composite key are yours to declare on the `pgTable`; each set's doc shows how.
A test holds every column set and its DDL equal, column for column.

```ts
import { index, pgTable, primaryKey, uniqueIndex } from 'drizzle-orm/pg-core';
import { modelProviderColumns, modelRunnerChunkColumns, modelRunnerColumns, modelRunnerRequestColumns } from '@octabits-io/models/drizzle';

export const modelProvider = pgTable('model_provider', modelProviderColumns);
export const modelRunner = pgTable('model_runner', modelRunnerColumns, (t) => [uniqueIndex('model_runner_token_hash_idx').on(t.tokenHash)]);
export const modelRunnerRequest = pgTable('model_runner_request', modelRunnerRequestColumns, (t) => [index('model_runner_request_pending_idx').on(t.status, t.createdAt)]);
export const modelRunnerChunk = pgTable('model_runner_chunk', modelRunnerChunkColumns, (t) => [primaryKey({ columns: [t.requestId, t.seq] })]);
```

Until octaflow 0.36 these tables were `octaflow/postgres`'s and
`octaflow/drizzle`'s; the names and columns are unchanged, so a host moves
its imports and needs no migration.

## License

MIT