`@octabits-io/models`

The model layer: which models an app may call, how it reaches them, and what it does with what they return. A provider registry that carries each vendor’s region and legal entity, so residency is data a picker and the disclosure gate read; a key turned into an AI SDK model that explains a vendor’s refusal by key and hint; lenient JSON from a model’s answer; a circuit breaker per run and a connection test; providers an administrator manages as rows, with their models’ declared capabilities; a model served by a process that connects out, for a machine the app cannot reach; prices and quotas. And, on the octaflow engine, AI steps, the model decider, agents that end by choosing an exit, and case loops in which a manager agent chooses every next move.

npm i @octabits-io/models zod     # + ai @ai-sdk/provider for ./ai-sdk, octaflow for ./octaflow

import { AI_PROVIDERS, createModelProviders } from '@octabits-io/models';
import { createModelResolver, withCircuitBreaker } from '@octabits-io/models/ai-sdk';
import { createPgModelProviderStore } from '@octabits-io/models/postgres';

// The providers an administrator manages: rows, a key that crosses a cipher seam.
const providers = createModelProviders({ store: createPgModelProviderStore({ executor: pool }), cipher });
// A row becomes an AI SDK model, each call shaped to what the model declares.
const resolver = createModelResolver({ providers });
const resolved = await resolver.languageModel({ provider: 'mistral-eu', require: { vision: true } });
if (resolved.ok) {
  const model = withCircuitBreaker(resolved.value.model);   // a rejected key fails the run once, not every step
  // resolved.value.destination — the provider's id and region — is what the disclosure gate reads before the call
}

Entry	Peer	What it is
`.`	`zod`	the contracts, no SDK behind them: the provider registry (`AI_PROVIDERS`, `EMBEDDING_PROVIDERS`, `resolveEndpointResidency`), model capabilities, prices (`DEFAULT_MODEL_PRICING`, `createCostEstimator`), a vendor’s refusal explained (`explainProviderError`), lenient JSON (`parseModelJson`), the managed providers (`createModelProviders`, `createInMemoryModelProviderStore`)
`./ai-sdk`	`ai`, `@ai-sdk/provider`, the vendors (all optional)	a key becomes a model: the vendor factories (async — each vendor is loaded on first use), OpenAI-compatible endpoints and gateways, Ollama’s native API (`createOllama`), the resolver over the managed providers (`createModelResolver`), the refusal-aware and instrumented wrappers, `withJsonRepair`, `withCircuitBreaker`, `testProvider`
`./runner`	`@ai-sdk/provider` (optional)	a model served by a process that connects out: the hub (`createModelRunnerHub`), the app-side models, the runner loop and its HTTP client, the store contract
`./octaflow`	`octaflow` (optional)	the model layer on the engine: `createAiWorkflowHooks` (an instrumented model per step, cost, quota, a `usage` sink), `defineAiStep`, `createModelDecider`, agents (`defineAgentStep`, `runAgentToExit`), case loops (`defineCaseLoop`), the quota and usage-rollup services
`./postgres`	`@octabits-io/postgres` (optional)	the provider store and the runners’ three tables as plain SQL over the structural executor, with their DDL
`./drizzle`	`drizzle-orm` (optional)	the same four tables as spreadable column sets

The root imports nothing but zod and @octabits-io/result, so a browser bundle — a provider picker, a residency badge — imports it alone; the registry file itself imports nothing at all. The boundary lint holds both.

When not to use it

You call one vendor with one key and never change it. The AI SDK’s own provider is enough; this package earns its place when an administrator picks the provider, when residency is a question someone asks, or when a model lives somewhere the app cannot reach.
You want a gateway in front of every call. That is LiteLLM’s job (with @octabits-io/disclosure’s guardrail in front of it); the OpenAI-compatible gateway provider here talks to one.
You want billing. Prices here are an estimate per step for quotas and the run view; what a customer is charged is @octabits-io/metering’s.

The provider registry and residency

AI_PROVIDERS and EMBEDDING_PROVIDERS list the vendors with a fixed residency — the processing region, the legal entity and its country — that a settings page iterates and the subprocessor annex (infra/SUBPROCESSORS.md) is generated from. An endpoint you run yourself (vLLM, llama.cpp, mlx_lm.server) is not in the registry: its residency is the deployment’s declaration, checked by resolveEndpointResidency. A new provider states its residency here or it does not exist.

Managed providers

createModelProviders({ store, cipher }): providers as rows, each with its kind, server, region, a key that crosses a cipher seam (a read carries a masked hint of it, never the key; reveal decrypts it for the call that needs it), tags, and its models with declared capabilities — temperature, JSON mode, vision, video, tools, embedding. createModelResolver (./ai-sdk) builds the AI SDK model from a row (languageModel(ref), embeddingModel(ref)), shapes each call to what the model declares, hands the host its destination for the disclosure guard, and rebuilds after a write. The table is model_provider; it carries no scope column, because the providers are the deployment’s — a host that lets each customer bring their own keys declares the table with a scope column beside these columns.

What a model returns

A vendor’s error, explained. explainProviderError(error) reads any error a vendor or the network produced as a key — provider_auth, provider_billing, provider_model_not_found, provider_rate_limited, provider_unavailable, provider_unreachable, budget_exceeded — with a sentence, a hint a person can act on, and whether it is fatal: whether further calls in the same run are pointless. Keys are masked in what it keeps.
A declined generation says so. A model can decline with HTTP 200 and an empty answer; createRefusalAwareModel, applied by every factory, turns that into an error instead of an empty string.
A circuit breaker per run. withCircuitBreaker(model): the first fatal error trips it, onFatal is told once, and every further call fails at once with the same error and no request. A rate limit or an outage does not trip it — the step’s retry policy is the tool for those.
Lenient JSON. A local or smaller model that fences its JSON, wraps it in prose or answers one object per image is read as the JSON it meant: withJsonRepair(model, { merge: true }), or parseModelJson(text) on a string. retryWithoutJsonMode and convertWith buy one more request each when the first answer cannot be read.
A connection test. testProvider(modelOrProvider) sends one prompt and reports the model and the time it took, or the explanation; it never throws for anything the vendor or the network did. testOllamaModel asks an Ollama server whether the model is installed.

Model runners — a model that connects out

A model the app cannot reach: a laptop with the model loaded for a demo, a GPU box behind someone’s firewall, a machine that is only on when it is needed. The runner never opens a port. It connects out to the app with a token, subscribes with its tags, claims the model calls that carry those tags and posts the answers back — the shape of a self-hosted CI runner. The mailbox is Postgres, because the call runs in the worker and the door the runner knocks on runs in the API; a request row lives only while the call is in flight.

// the app — once, beside the providers
import { createModelRunnerHub } from '@octabits-io/models/runner';
import { createPgModelRunnerStore } from '@octabits-io/models/postgres';

const runners = createModelRunnerHub({ store: createPgModelRunnerStore({ executor }) });
const resolver = createModelResolver({ providers, runners, wrap });

// an administrator: the provider row carries tags, the runner gets its token once
await providers.save({ id: 'mac', kind: 'runner', label: 'The Mac', region: 'self-hosted', tags: ['qwen'], models: [{ id: 'qwen3:32b', settings: { numCtx: 16_384 } }] });
const { token } = (await runners.registerRunner({ id: 'gpu-box', label: 'The GPU box', tags: ['qwen'] })).value;

// the API: the door, behind the host's rate limit
import { createModelRunnerHandler } from '@octabits-io/server/fetch/models';
export const POST = compose(withErrorMapping())(createModelRunnerHandler({ prefix: '/api/model-runner', hub: runners }));

// the runner, on the machine with the model — the bin, or the loop in your own process
//   npx -p @octabits-io/models model-runner --url https://app.example/api/model-runner --token omr_… --ollama http://localhost:11434
import { createHttpRunnerClient, runModelRunner } from '@octabits-io/models/runner';
import { createOllama } from '@octabits-io/models/ai-sdk';
await runModelRunner({
  client: createHttpRunnerClient({ url: 'https://app.example/api/model-runner', token }),
  backend: createOllama({ baseUrl: 'http://localhost:11434' }),
});

model-runner, the bin, is the runner as a process: --url and --token (or MODEL_RUNNER_URL / MODEL_RUNNER_TOKEN), one backend — --ollama <url> for Ollama’s native API, --openai-compatible <url> (with --api-key) for mlx_lm.server, vLLM or llama.cpp through the optional @ai-sdk/openai-compatible peer — and --concurrency. At start it says who the token belongs to, which models the app’s provider rows declare for its tags (the door tells it) and which of them the local server has, with the ollama pull for a missing one; --check does that and exits. A door it cannot reach is retried with a backoff, forever. The skill serve-a-model-from-a-runner walks through the four pieces.

A runner may claim a request when it has every tag of the request — the provider row’s tags — as a CI job’s labels must all be on the runner. A call with nobody online for its tags fails at once with provider_unreachable, so withCircuitBreaker trips and testProvider reports it in milliseconds instead of a timeout; whether another provider is tried is the host’s resolver’s choice, through the same disclosure gate. A hub made with waitForRunnerMs gives a runner that long to come (back) — a laptop waking up, a runner restarting — before the call fails the same way; a call can set its own (0 for a probe). A claim lasts 30 s without a heartbeat; an expired claim is requeued once and failed the second time. The waiting caller expires its own request’s lease, so a request held by a runner that died is requeued even when no other runner comes to claim, and fails with no_runner once nobody is online to take it over, instead of sitting until the call’s deadline. The region is the administrator’s declaration about where runners with these tags are, as for every provider, and the disclosure gate reads it the same way.

The wire is the AI SDK’s own call and answer as JSON — binary file data as base64, a URL restored on the other side — so the runner’s backend is any LanguageModelV4: createOllama from this package, or @ai-sdk/openai-compatible for mlx_lm.server, vLLM and llama.cpp. streamText streams: the runner runs doStream and posts the parts in chunks as they come (every 100 ms, or 64 parts), a third table carries them, and the app emits each chunk as it lands; a chunk the door refuses tells the runner the caller gave up, and it cancels the stream. A runner on the older wire answers a stream whole, and the app emits that as parts too. Files reach the runner as bytes, since the runner model declares no URL support and the SDK downloads on the app’s side. Embeddings go the same way.

On the octaflow engine — `./octaflow`

createAiWorkflowHooks({ modelResolver, usageRecorder, quotaPolicy?, usage?, costEstimator? }) is the engine’s hooks: each AI step gets an instrumented model (ctx.context.model), the tokens it used are captured per step and priced, a usage sink books each step into @octabits-io/metering (by shape — this package never imports it), and a quota policy refuses a run before it starts. defineAiStep and buildAiWorkflow type the steps over that context.

Decisions with a model. createModelDecider() is a decider for octaflow’s decision steps: the model answers with a choice and a confidence, and below the threshold a person settles it.
Agents. defineAgentStep / runAgentToExit: an agent’s tool loop that ends by choosing one of the step’s exits, each with its own schema, so the policy that refuses a choice is a schema, not a prompt.
Case loops. defineCaseLoop: a manager agent chooses every next move of a case, each move its own workflow ending in the same tail (result → decide → next); policy refuses a choice with its reason inside the exit’s schema, a move with a step that could fail is refused at definition time, and a manager that cannot decide closes the case for a person.
Quotas and rollups. createAiQuotaService and createAiUsageAggregationService over structural AiQuotaStore / AiUsageStore seams — the host keeps the SQL.

The engine’s own guide to this layer is the AI add-on.

The tables

./postgres owns four tables and ships their DDL: modelProviderDdl() (model_provider) and modelRunnerDdl() (model_runner, model_runner_request, model_runner_chunk). Each is idempotent CREATE TABLE IF NOT EXISTS, takes a schema, and is meant to be pasted into your own migration system. The stores reach the database only through @octabits-io/postgres’s SqlExecutor — pg, PGlite and postgres.js all fit — and every write is one statement.

./drizzle ships the same tables as modelProviderColumns, modelRunnerColumns, modelRunnerRequestColumns and modelRunnerChunkColumns, for a host that lets drizzle-kit own its migrations. The unique index on a runner’s token hash and the chunk table’s composite key are yours to declare on the pgTable; each set’s doc shows how. A test holds every column set and its DDL equal, column for column.

import { index, pgTable, primaryKey, uniqueIndex } from 'drizzle-orm/pg-core';
import { modelProviderColumns, modelRunnerChunkColumns, modelRunnerColumns, modelRunnerRequestColumns } from '@octabits-io/models/drizzle';

export const modelProvider = pgTable('model_provider', modelProviderColumns);
export const modelRunner = pgTable('model_runner', modelRunnerColumns, (t) => [uniqueIndex('model_runner_token_hash_idx').on(t.tokenHash)]);
export const modelRunnerRequest = pgTable('model_runner_request', modelRunnerRequestColumns, (t) => [index('model_runner_request_pending_idx').on(t.status, t.createdAt)]);
export const modelRunnerChunk = pgTable('model_runner_chunk', modelRunnerChunkColumns, (t) => [primaryKey({ columns: [t.requestId, t.seq] })]);

Until octaflow 0.36 these tables were octaflow/postgres’s and octaflow/drizzle’s; the names and columns are unchanged, so a host moves its imports and needs no migration.

License

MIT

@octabits-io/models