---
title: "@octabits-io/knowledge"
summary: "Company knowledge on your own Postgres, with answers that cite their source."
install: "pnpm add @octabits-io/knowledge"
availability: "Available from November 2026"
version: "0.2.2"
license: "MIT"
related:
  - name: "@octabits-io/result"
    url: "https://octabits.io/platform/result.md"
canonical_url: "https://octabits.io/platform/knowledge"
---

# @octabits-io/knowledge

Company knowledge on your own Postgres, with answers that cite their source.

Documents are read into blocks that keep their page and their section. The blocks are cut into chunks that never cross either. A question is answered from the chunks a person may read, and a model's answer counts only where its quote is found, verbatim, in the passage it cites. Every citation comes back with the document, the version, the digest of the bytes, the page and the character span, in the same shape `@octabits-io/proposal` uses. A retrieval log records which passages each question reached, beside the disclosure log that records which model they went to.

- **Full text first.** Retrieval works with Postgres full-text search alone. That needs no extension and no embedding model, so a deployment without a GPU still has zero subprocessors.
- **Vectors when you want them.** With pgvector and an embedding model, the search is hybrid: the full-text ranking and the vector ranking are fused by reciprocal rank.
- **Access in SQL.** Scope and audience filter every query before anything is ranked. A passage a person may not read is never a candidate, so it can never reach a prompt.
- **Checked, not trusted.** A citation exists only where its quote stands in the passage it names. An invented quote is rejected and reported, and a statement left without a citation is marked unsupported.

```ts
import { createExtractor, ingestDocument, retrieve } from '@octabits-io/knowledge';
import { createPostgresKnowledgeStore, createPostgresRetrievalLogStore } from '@octabits-io/knowledge/postgres';
import { extractPdf } from '@octabits-io/knowledge/pdf';
import { answerWithModel, embedderFromModel } from '@octabits-io/knowledge/ai-sdk';

const store = createPostgresKnowledgeStore({ db: pool, scope: { column: 'org_id' }, languages: ['german', 'english'], embeddings: true });
const log = createPostgresRetrievalLogStore({ db: pool, scope: { column: 'org_id' } });
const extract = createExtractor({ extractors: { 'application/pdf': extractPdf } });
const embedder = embedderFromModel(guardedEmbeddingModel, { dimensions: 1024 });

await ingestDocument({ store, extract, embedder, scopeKey: orgId, document: { id, bytes, mediaType: 'application/pdf', language: 'german', audiences: ['hr'] } });

const found = await retrieve({ store, embedder, log, scopeKey: orgId, question, audiences: user.audiences, correlationId: runId });
if (!found.ok) return found;
const answer = await answerWithModel({ model: guardedLanguageModel, question, hits: found.value.hits });
// answer.value.statements[i] = { text, citations: Citation[], supported }; answer.value.rejected = the citations that did not stand
```

## Install

```bash
pnpm add @octabits-io/knowledge @octabits-io/result zod
```

Each optional peer is needed only by its subpath:

| Subpath | Peer | What it does |
|---|---|---|
| `.` | — | the shapes, the chunker, citations and their check, the store contracts with in-memory references, rank fusion, the hashing embedder, the text and Markdown extractors, the pipeline, the answer step |
| `./postgres` | — (a structural executor: `pg`, PGlite, postgres.js) | the store and the retrieval log over plain SQL, with their DDL |
| `./drizzle` | `drizzle-orm` | the four column sets, so drizzle-kit owns the tables |
| `./ai-sdk` | `ai`, `@ai-sdk/provider` | an embedder from any AI SDK embedding model; the answer step over any language model |
| `./pdf` | `unpdf` | PDF: a block per paragraph with its page and box, headings from font size |
| `./html` | `node-html-parser` | HTML: a block per paragraph under its heading path, table rows as blocks |
| `./docx` | `mammoth` | Word, through its HTML into the HTML extractor |
| `./docling` | — (fetch) | a self-hosted [docling-serve](https://github.com/docling-project/docling-serve): scans with OCR, tables, layout, page and box per block |

## The pieces

### Blocks, chunks and spans

An extractor returns `Block`s: `{ text, page, locator, headings?, bbox? }`.

- `text` is as the document says it, with whitespace collapsed and nothing rewritten.
- `page` is 1-based for a paged document and `null` otherwise.
- `locator` is the block's place in the document's own terms: `§ 4 Ort der Verarbeitung`, `Page 3`.

`chunkDocument` packs consecutive blocks of one section up to `maxTokens` (default 400, estimated at four characters a token):

- It never crosses a page or a change of locator.
- It splits an over-long block at sentence ends, and at spaces when one sentence is still too long.
- It does no overlap. [Chroma's evaluation](https://www.trychroma.com/research/evaluating-chunking) found overlap cost precision and bought no recall.

Each chunk carries a `context`: by default the title and heading path, which are indexed with it but never cited. It also carries its `span` in the **page text**, which is the page's blocks joined by a blank line (`pageText(blocks, page)`). A citation's span means the same thing, so it can be checked against the page later.

### The store

`KnowledgeStore` is the contract. It has two implementations: `createInMemoryKnowledgeStore` (the reference) and `createPostgresKnowledgeStore`. One conformance suite runs both, unscoped and scoped.

- **`put`** indexes a version in one statement, so there is never half a version: the document row, every chunk replaced, and the vectors of unchanged passages (same content hash) carried over. A new version therefore re-embeds only what changed.
- **`remove`** deletes the document with every chunk and vector, also in one statement. It is the erasure path. Embeddings can reproduce the text they were made from, so they are personal data whenever the text is.
- **`setAudiences`** changes who may retrieve a document without re-indexing it.
- **`search`** takes the caller's `audiences` (or `'all'` for a system read) and, optionally, a query vector. It returns `SearchHit`s with the fused score and both ranks.
- **`pendingEmbeddings` / `putEmbeddings`** are the work list of an embedding run. `embedPending` drives them in batches, for a background job or a change of model.

**Scope.** Pass `scope: { column }` and every call must name its `scopeKey`; with `{ column, value }` every call is pinned to `value`. A call without a scope is refused, not widened.

- Every key leads with the scope column: `(scope, id)` for documents and chunks, `(scope, chunk_id, model_id)` for vectors. The same document id in two scopes is two documents, so neither can overwrite the other or learn that the other exists.
- Chunk ids are `<document>@<version>#<ordinal>` with `%`, `@` and `#` escaped, and a chunk whose id is not its document's is refused. No spelling of an id or a version reaches another document's chunk.
- Put [row-level security](https://octabits.io/platform/rls.md) on all four tables as the second wall. Explicit filtering is also what lets a vector index serve a small scope.

**Concurrent writes.** Each chunk row carries its document's version, and every read joins on the version the document row holds now. Two `put`s of one document racing under READ COMMITTED can leave the loser's chunks behind until the next `put`, but they are never searched, cited or embedded.

**Audiences.** A document with no audiences is open to its whole scope. Otherwise a caller must hold one of its audiences. The names are yours: a role, a department, a group from your identity provider. The filter runs in the query, before ranking, on the chunk and vector rows themselves.

### Full text and vectors on Postgres

- **Full text needs nothing installed.**
  - Each chunk's `tsvector` is built with its document's text-search configuration (`languages: ['german', 'english', 'simple']`; a document picks one with `language`).
  - A question matches a passage holding any of its words under every configuration the store accepts, and `ts_rank_cd` ranks the passages with more of them, closer together, first.
  - Postgres's `german` configuration stems words but does not split compounds: *Auftragsverarbeitungsvertrag* stays one word. Vectors close that gap.
- **Vectors need [pgvector](https://github.com/pgvector/pgvector) 0.8 or later** and `embeddings: true`.
  - Vectors live in `knowledge_embeddings`, one row per chunk and model, in an untyped `vector` column. A second model is a second set of rows; switching to it is an `embedPending` run, then a change of the embedder you search with.
  - Scope, document and audiences are copied onto every vector row, so the filter is on the indexed table.
  - Every vector search switches on pgvector's iterative index scan for itself, transaction-local, before the index is read. Without it, an HNSW search filtered to a scope that holds 1% of the rows can return nothing; the test suite proves both halves of that.
  - A vector search always has nearest neighbours, even for a question nothing answers. `maxDistance` (cosine distance, on a search or on `retrieve`, defaulting to the embedder's own) keeps the vector side to passages that are actually near. Measure it for your model: the hashing embedder declares 0.8, where related German and English questions land at 0.55–0.7 and unrelated ones at 0.85 and beyond.
  - Add an HNSW index per model with `knowledgeVectorIndexDdl({ modelId, dimensions })` once an exact scan gets slow. It uses `halfvec` above 2,000 dimensions; pgvector indexes up to 4,000.

```ts
import { knowledgeDdl, knowledgeEmbeddingsDdl, knowledgeVectorIndexDdl, retrievalLogDdl } from '@octabits-io/knowledge/postgres';

await pool.query('CREATE EXTENSION IF NOT EXISTS vector'); // a privileged step; not in the DDL helpers
await pool.query(knowledgeDdl({ scope: { column: 'org_id' } }) + knowledgeEmbeddingsDdl({ scope: { column: 'org_id' } }) + retrievalLogDdl({ scope: { column: 'org_id' } }));
await pool.query(knowledgeVectorIndexDdl({ modelId: embedder.modelId, dimensions: 1024 }));
```

A Drizzle host declares the tables from `knowledgeDocumentColumns`, `knowledgeChunkColumns`, `knowledgeEmbeddingColumns` and `retrievalLogColumns`. It adds its scope column and declares every primary key with the scope first: `(org_id, id)`, `(org_id, id)`, `(org_id, chunk_id, model_id)`. The HNSW index goes in a custom migration. On PGlite, load the extension with `@electric-sql/pglite-pgvector`.

### Embeddings

An `Embedder` is `{ modelId, dimensions, embed(values, 'document' | 'query') }`. Instruction-tuned models want a different prefix for passages and for questions.

- **`embedderFromModel(model, { dimensions, documentPrefix?, queryPrefix? })`** wraps any AI SDK embedding model. `QWEN3_QUERY_PREFIX` and `E5_PREFIXES` are provided.
- **Self-hosted models** keep documents in the house. Qwen3-Embedding-0.6B (Apache-2.0, 100+ languages, 1,024 dimensions) and BGE-M3 (MIT) both run on Ollama, vLLM or text-embeddings-inference behind `createOpenAiCompatibleEmbeddingProvider` from `octaflow/ai/providers`.
- **`createHashingEmbedder()`** needs no model: it hashes words and character trigrams into a fixed-width vector. It is deterministic, which is what tests and demos need, and it catches shared word parts. It is not a semantic model.

**Put the disclosure gate around the model first:** `guardEmbeddingModel` from `@octabits-io/disclosure/ai-sdk`. Every passage indexed and every question embedded is then decided by content class and destination, and recorded as a digest. Document text is `confidential` unless you declare otherwise, and confidential text goes only to a self-hosted model.

### Answers and citations

- **The prompt.** `answerPrompt` numbers the passages and puts them in one delimited block, as data the model must not take instructions from. It asks for JSON: statements, each with the passages it rests on and a verbatim quote from each.
- **Grounding.** `groundAnswer` looks every quote up in the passage it names. Matching compares per user-perceived character (NFKC, so a decomposed accent equals a precomposed one), folds typographic quotes and dashes, drops soft hyphens and zero-width characters, and collapses whitespace. It stays case-sensitive.
  - A quote shorter than `minQuoteLength` (default 12 non-space characters) supports nothing, unless it is the whole passage: a quote of `e` is found in every passage.
  - A quote that is found becomes a `Citation` whose `quote` is the document's own text, never the model's spelling of it.
  - A quote that is not found is in `rejected`.
- **`answerQuestion`** runs the step over any `generate(prompt) → text`, and `answerWithModel` does the same over an AI SDK language model, at temperature 0. With no hits, no model is called.

What to do with an unsupported statement is the host's decision: show it flagged, or not at all. `checkCitation(citation, { version, pageText })` re-checks a stored citation later and returns `stands`, `superseded`, `moved` or `missing`. That is what tells a fact its source has changed.

### The retrieval log

`retrieve({ …, log, correlationId })` writes one row per question:

- the digest of the question, never its text;
- the mode and the embedding model;
- for every hit, the document, version, digest, chunk and rank.

Give it the same `correlationId` as the disclosure log. Like the store, a log scoped by `{ column }` refuses a read that names no `scopeKey`. An audit can then go from an answer to the passages behind it and to the model call that carried them. A failed log write fails the retrieval: an answer whose sources cannot be accounted for is not handed out.

### Extractors

`createExtractor({ extractors, maxBytes })` routes by media type and refuses a file over the limit (default 50 MiB) before any parser sees it. It also refuses a file with no text, and reports what an extractor skipped as `warnings`. Plain text and Markdown are built in; register the rest:

```ts
import { extractPdf } from '@octabits-io/knowledge/pdf';
import { extractHtml } from '@octabits-io/knowledge/html';
import { extractDocx } from '@octabits-io/knowledge/docx';
import { createDoclingExtractor } from '@octabits-io/knowledge/docling';

const extract = createExtractor({
  extractors: {
    'application/pdf': extractPdf,
    'text/html': extractHtml,
    'application/vnd.openxmlformats-officedocument.wordprocessingml.document': extractDocx,
    'image/png': createDoclingExtractor({ url: 'http://docling:5001', ocrLanguages: ['de', 'en'] }),
  },
});
```

A `.docx` is a zip: `createDocxExtractor({ maxUnzippedBytes, maxEntries })` (defaults 100 MiB, 10,000 entries) reads the zip's directory and refuses a file that would unpack past the limit before `mammoth` inflates anything. The check trusts the sizes the zip declares, so extract untrusted uploads where memory is bounded as well.

A scanned PDF has no text layer, and the PDF extractor says so in a warning. Route scans to Docling, which runs OCR on your own host; the MIT-licensed `quay.io/docling-project/docling-serve` image works air-gapped.

## What it does not do

- **It does not decide who holds which audience.** That is your identity and role model.
- **It does not keep the original files.** Use `@octabits-io/storage`, and index what you stored.
- **It does not rerank with a model, or write a context line per chunk with one.** Both measurably help retrieval, and both send every passage to a model. They are hooks you can add (`chunking.context`, a reorder of `hits`), not defaults.
- **It does not judge whether a statement is fully supported by its citation.** The check is that the quote exists where it is said to. Whether it says what the statement claims is a reviewer's call, or a model-graded evaluation you run deliberately.