octabits

@octabits-io/knowledge

Company knowledge on your own Postgres, with answers that cite their source.

Documents are read into blocks that keep their page and their section. The blocks are cut into chunks that never cross either. A question is answered from the chunks a person may read, and a model’s answer counts only where its quote is found, verbatim, in the passage it cites. Every citation comes back with the document, the version, the digest of the bytes, the page and the character span, in the same shape @octabits-io/proposal uses. A retrieval log records which passages each question reached, beside the disclosure log that records which model they went to.

  • Full text first. Retrieval works with Postgres full-text search alone. That needs no extension and no embedding model, so a deployment without a GPU still has zero subprocessors.
  • Vectors when you want them. With pgvector and an embedding model, the search is hybrid: the full-text ranking and the vector ranking are fused by reciprocal rank.
  • Access in SQL. Scope and audience filter every query before anything is ranked. A passage a person may not read is never a candidate, so it can never reach a prompt.
  • Checked, not trusted. A citation exists only where its quote stands in the passage it names. An invented quote is rejected and reported, and a statement left without a citation is marked unsupported.
import { createExtractor, ingestDocument, retrieve } from '@octabits-io/knowledge';
import { createPostgresKnowledgeStore, createPostgresRetrievalLogStore } from '@octabits-io/knowledge/postgres';
import { extractPdf } from '@octabits-io/knowledge/pdf';
import { answerWithModel, embedderFromModel } from '@octabits-io/knowledge/ai-sdk';

const store = createPostgresKnowledgeStore({ db: pool, scope: { column: 'org_id' }, languages: ['german', 'english'], embeddings: true });
const log = createPostgresRetrievalLogStore({ db: pool, scope: { column: 'org_id' } });
const extract = createExtractor({ extractors: { 'application/pdf': extractPdf } });
const embedder = embedderFromModel(guardedEmbeddingModel, { dimensions: 1024 });

await ingestDocument({ store, extract, embedder, scopeKey: orgId, document: { id, bytes, mediaType: 'application/pdf', language: 'german', audiences: ['hr'] } });

const found = await retrieve({ store, embedder, log, scopeKey: orgId, question, audiences: user.audiences, correlationId: runId });
if (!found.ok) return found;
const answer = await answerWithModel({ model: guardedLanguageModel, question, hits: found.value.hits });
// answer.value.statements[i] = { text, citations: Citation[], supported }; answer.value.rejected = the citations that did not stand

Install

pnpm add @octabits-io/knowledge @octabits-io/result zod

Each optional peer is needed only by its subpath:

Subpath Peer What it does
. — the shapes, the chunker, citations and their check, the store contracts with in-memory references, rank fusion, the hashing embedder, the text and Markdown extractors, the pipeline, the answer step
./postgres — (a structural executor: pg, PGlite, postgres.js) the store and the retrieval log over plain SQL, with their DDL
./drizzle drizzle-orm the four column sets, so drizzle-kit owns the tables
./ai-sdk ai, @ai-sdk/provider an embedder from any AI SDK embedding model; the answer step over any language model
./pdf unpdf PDF: a block per paragraph with its page and box, headings from font size
./html node-html-parser HTML: a block per paragraph under its heading path, table rows as blocks
./docx mammoth Word, through its HTML into the HTML extractor
./docling — (fetch) a self-hosted docling-serve: scans with OCR, tables, layout, page and box per block

The pieces

Blocks, chunks and spans

An extractor returns Blocks: { text, page, locator, headings?, bbox? }.

  • text is as the document says it, with whitespace collapsed and nothing rewritten.
  • page is 1-based for a paged document and null otherwise.
  • locator is the block’s place in the document’s own terms: § 4 Ort der Verarbeitung, Page 3.

chunkDocument packs consecutive blocks of one section up to maxTokens (default 400, estimated at four characters a token):

  • It never crosses a page or a change of locator.
  • It splits an over-long block at sentence ends, and at spaces when one sentence is still too long.
  • It does no overlap. Chroma’s evaluation found overlap cost precision and bought no recall.

Each chunk carries a context: by default the title and heading path, which are indexed with it but never cited. It also carries its span in the page text, which is the page’s blocks joined by a blank line (pageText(blocks, page)). A citation’s span means the same thing, so it can be checked against the page later.

The store

KnowledgeStore is the contract. It has two implementations: createInMemoryKnowledgeStore (the reference) and createPostgresKnowledgeStore. One conformance suite runs both, unscoped and scoped.

  • put indexes a version in one statement, so there is never half a version: the document row, every chunk replaced, and the vectors of unchanged passages (same content hash) carried over. A new version therefore re-embeds only what changed.
  • remove deletes the document with every chunk and vector, also in one statement. It is the erasure path. Embeddings can reproduce the text they were made from, so they are personal data whenever the text is.
  • setAudiences changes who may retrieve a document without re-indexing it.
  • search takes the caller’s audiences (or 'all' for a system read) and, optionally, a query vector. It returns SearchHits with the fused score and both ranks.
  • pendingEmbeddings / putEmbeddings are the work list of an embedding run. embedPending drives them in batches, for a background job or a change of model.

Scope. Pass scope: { column } and every call must name its scopeKey; with { column, value } every call is pinned to value. A call without a scope is refused, not widened.

  • Every key leads with the scope column: (scope, id) for documents and chunks, (scope, chunk_id, model_id) for vectors. The same document id in two scopes is two documents, so neither can overwrite the other or learn that the other exists.
  • Chunk ids are <document>@<version>#<ordinal> with %, @ and # escaped, and a chunk whose id is not its document’s is refused. No spelling of an id or a version reaches another document’s chunk.
  • Put row-level security on all four tables as the second wall. Explicit filtering is also what lets a vector index serve a small scope.

Concurrent writes. Each chunk row carries its document’s version, and every read joins on the version the document row holds now. Two puts of one document racing under READ COMMITTED can leave the loser’s chunks behind until the next put, but they are never searched, cited or embedded.

Audiences. A document with no audiences is open to its whole scope. Otherwise a caller must hold one of its audiences. The names are yours: a role, a department, a group from your identity provider. The filter runs in the query, before ranking, on the chunk and vector rows themselves.

Full text and vectors on Postgres

  • Full text needs nothing installed.
    • Each chunk’s tsvector is built with its document’s text-search configuration (languages: ['german', 'english', 'simple']; a document picks one with language).
    • A question matches a passage holding any of its words under every configuration the store accepts, and ts_rank_cd ranks the passages with more of them, closer together, first.
    • Postgres’s german configuration stems words but does not split compounds: Auftragsverarbeitungsvertrag stays one word. Vectors close that gap.
  • Vectors need pgvector 0.8 or later and embeddings: true.
    • Vectors live in knowledge_embeddings, one row per chunk and model, in an untyped vector column. A second model is a second set of rows; switching to it is an embedPending run, then a change of the embedder you search with.
    • Scope, document and audiences are copied onto every vector row, so the filter is on the indexed table.
    • Every vector search switches on pgvector’s iterative index scan for itself, transaction-local, before the index is read. Without it, an HNSW search filtered to a scope that holds 1% of the rows can return nothing; the test suite proves both halves of that.
    • A vector search always has nearest neighbours, even for a question nothing answers. maxDistance (cosine distance, on a search or on retrieve, defaulting to the embedder’s own) keeps the vector side to passages that are actually near. Measure it for your model: the hashing embedder declares 0.8, where related German and English questions land at 0.55–0.7 and unrelated ones at 0.85 and beyond.
    • Add an HNSW index per model with knowledgeVectorIndexDdl({ modelId, dimensions }) once an exact scan gets slow. It uses halfvec above 2,000 dimensions; pgvector indexes up to 4,000.
import { knowledgeDdl, knowledgeEmbeddingsDdl, knowledgeVectorIndexDdl, retrievalLogDdl } from '@octabits-io/knowledge/postgres';

await pool.query('CREATE EXTENSION IF NOT EXISTS vector'); // a privileged step; not in the DDL helpers
await pool.query(knowledgeDdl({ scope: { column: 'org_id' } }) + knowledgeEmbeddingsDdl({ scope: { column: 'org_id' } }) + retrievalLogDdl({ scope: { column: 'org_id' } }));
await pool.query(knowledgeVectorIndexDdl({ modelId: embedder.modelId, dimensions: 1024 }));

A Drizzle host declares the tables from knowledgeDocumentColumns, knowledgeChunkColumns, knowledgeEmbeddingColumns and retrievalLogColumns. It adds its scope column and declares every primary key with the scope first: (org_id, id), (org_id, id), (org_id, chunk_id, model_id). The HNSW index goes in a custom migration. On PGlite, load the extension with @electric-sql/pglite-pgvector.

Embeddings

An Embedder is { modelId, dimensions, embed(values, 'document' | 'query') }. Instruction-tuned models want a different prefix for passages and for questions.

  • embedderFromModel(model, { dimensions, documentPrefix?, queryPrefix? }) wraps any AI SDK embedding model. QWEN3_QUERY_PREFIX and E5_PREFIXES are provided.
  • Self-hosted models keep documents in the house. Qwen3-Embedding-0.6B (Apache-2.0, 100+ languages, 1,024 dimensions) and BGE-M3 (MIT) both run on Ollama, vLLM or text-embeddings-inference behind createOpenAiCompatibleEmbeddingProvider from octaflow/ai/providers.
  • createHashingEmbedder() needs no model: it hashes words and character trigrams into a fixed-width vector. It is deterministic, which is what tests and demos need, and it catches shared word parts. It is not a semantic model.

Put the disclosure gate around the model first: guardEmbeddingModel from @octabits-io/disclosure/ai-sdk. Every passage indexed and every question embedded is then decided by content class and destination, and recorded as a digest. Document text is confidential unless you declare otherwise, and confidential text goes only to a self-hosted model.

Answers and citations

  • The prompt. answerPrompt numbers the passages and puts them in one delimited block, as data the model must not take instructions from. It asks for JSON: statements, each with the passages it rests on and a verbatim quote from each.
  • Grounding. groundAnswer looks every quote up in the passage it names. Matching compares per user-perceived character (NFKC, so a decomposed accent equals a precomposed one), folds typographic quotes and dashes, drops soft hyphens and zero-width characters, and collapses whitespace. It stays case-sensitive.
    • A quote shorter than minQuoteLength (default 12 non-space characters) supports nothing, unless it is the whole passage: a quote of e is found in every passage.
    • A quote that is found becomes a Citation whose quote is the document’s own text, never the model’s spelling of it.
    • A quote that is not found is in rejected.
  • answerQuestion runs the step over any generate(prompt) → text, and answerWithModel does the same over an AI SDK language model, at temperature 0. With no hits, no model is called.

What to do with an unsupported statement is the host’s decision: show it flagged, or not at all. checkCitation(citation, { version, pageText }) re-checks a stored citation later and returns stands, superseded, moved or missing. That is what tells a fact its source has changed.

The retrieval log

retrieve({ …, log, correlationId }) writes one row per question:

  • the digest of the question, never its text;
  • the mode and the embedding model;
  • for every hit, the document, version, digest, chunk and rank.

Give it the same correlationId as the disclosure log. Like the store, a log scoped by { column } refuses a read that names no scopeKey. An audit can then go from an answer to the passages behind it and to the model call that carried them. A failed log write fails the retrieval: an answer whose sources cannot be accounted for is not handed out.

Extractors

createExtractor({ extractors, maxBytes }) routes by media type and refuses a file over the limit (default 50 MiB) before any parser sees it. It also refuses a file with no text, and reports what an extractor skipped as warnings. Plain text and Markdown are built in; register the rest:

import { extractPdf } from '@octabits-io/knowledge/pdf';
import { extractHtml } from '@octabits-io/knowledge/html';
import { extractDocx } from '@octabits-io/knowledge/docx';
import { createDoclingExtractor } from '@octabits-io/knowledge/docling';

const extract = createExtractor({
  extractors: {
    'application/pdf': extractPdf,
    'text/html': extractHtml,
    'application/vnd.openxmlformats-officedocument.wordprocessingml.document': extractDocx,
    'image/png': createDoclingExtractor({ url: 'http://docling:5001', ocrLanguages: ['de', 'en'] }),
  },
});

A .docx is a zip: createDocxExtractor({ maxUnzippedBytes, maxEntries }) (defaults 100 MiB, 10,000 entries) reads the zip’s directory and refuses a file that would unpack past the limit before mammoth inflates anything. The check trusts the sizes the zip declares, so extract untrusted uploads where memory is bounded as well.

A scanned PDF has no text layer, and the PDF extractor says so in a warning. Route scans to Docling, which runs OCR on your own host; the MIT-licensed quay.io/docling-project/docling-serve image works air-gapped.

What it does not do

  • It does not decide who holds which audience. That is your identity and role model.
  • It does not keep the original files. Use @octabits-io/storage, and index what you stored.
  • It does not rerank with a model, or write a context line per chunk with one. Both measurably help retrieval, and both send every passage to a model. They are hooks you can add (chunking.context, a reorder of hits), not defaults.
  • It does not judge whether a statement is fully supported by its citation. The check is that the quote exists where it is said to. Whether it says what the statement claims is a reviewer’s call, or a model-graded evaluation you run deliberately.

From @octabits-io/knowledge@0.2.2 in the platform repository.