Datasets

A dataset is a named pile of your documents that the agentic runtime can search. You put text or files in; you ask a question in plain language; you get back the passages that answer it, with a pointer to which document each one came from.

This is what people usually mean by "make it answer from my documents". No model is retrained. The documents go into the gateway you are talking to and stay in its own store — for a gateway on your own machine, that means they never leave it.

cargo install --path crates/kx-cli --features serve-engine,hnsw

Put documents in

Start a gateway, then ingest. Each --text or --file you pass becomes one document.

kx serve --dev-allow-local

kx datasets ingest handbook --file ./handbook.md

kx datasets ingest handbook --file ./policy.md --file ./faq.md --text "office hours are 9 to 5"

kx datasets list

The store is append-only and content-addressed: ingesting the same bytes twice is a no-op, and there is no delete subcommand. To retire a corpus, build a new one under a new name.

Ask it something

kx datasets query handbook --text "what is the refund window?" --k 5

--k is how many results you want back; it defaults to 10, and the server clamps it to its own maximum of 64. Each hit carries its text, its parent document, and a similarity score.

What "hybrid" retrieval actually does

Two different searches run over your corpus, and their results are merged.

• Dense (vector) search compares the meaning of your question to the meaning of each passage. It finds a paraphrase — you ask about "refund window", the passage says "money back within 30 days".
• Sparse (BM25) search compares words. It finds the exact term a meaning-based search fumbles: a product code, a surname, a rare acronym.

Neither is reliably better, and their scores are not on the same scale, so they are not averaged. Instead the runtime uses Reciprocal Rank Fusion (RRF): it throws away both scores and keeps only each document's position in each list. A document ranked n-th in a list contributes 1 / (60 + n) to its fused total; the totals are summed and re-sorted. Something both searches liked beats something only one of them liked.

Then an MMR pass (Maximal Marginal Relevance) walks the fused list and picks results one at a time, each time trading relevance against how similar the candidate already is to what has been picked. The effect is that three near-identical paragraphs do not eat your top three slots — the second and third get demoted and something else surfaces.

Both steps are pure arithmetic over ranks and vectors. Ties break on ascending content reference, so for a fixed index state, query and configuration the same order comes back on any machine.

Choose the mode per query:

kx datasets query handbook --text "what is the refund window?" --mode hybrid
kx datasets query handbook --text "error code KX-4021" --mode dense

The MMR diversity pass follows the operator's default. Turn it off for one query when you want the raw fused ranking:

kx datasets query handbook --text "what is the refund window?" --mode hybrid --rerank off

--rerank accepts on or off. Omit both flags and you get whatever the server is configured for.

kx datasets ingest handbook --file ./handbook.md
kx datasets query handbook --text "what is the refund window?" --k 5 --mode hybrid

from kortecx import KxClient, RetrievalMode

client = KxClient("http://127.0.0.1:50151")

# Raw bytes use the server-embed path (the gateway needs a served embedding model).
client.ingest_documents("handbook", [open("handbook.md", "rb").read()])

hits = client.query_dataset(
    "handbook",
    text="what is the refund window?",
    k=5,
    mode=RetrievalMode.HYBRID,
)

import { KxClient, RetrievalMode } from "@kortecx/sdk";

const client = new KxClient("http://127.0.0.1:50151");

const hits = await client.queryDataset("handbook", {
  text: "what is the refund window?",
  k: 5,
  mode: RetrievalMode.HYBRID,
});

Chunking, and knowing where an answer came from

A long document is not embedded whole — a single vector for forty pages means nothing. Before embedding, each document is split into overlapping passages, and a search hit is a passage, not the whole file.

The splitter is deterministic. It prefers to break at a paragraph, then a line, then a sentence, then a word, and only cuts mid-word when nothing else fits. Two defaults control it, both operator-set:

The overlap matters: a sentence that straddles a boundary is still wholly present in at least one passage, so it can still be found.

Every passage keeps its provenance — the parent document it came from, its position within that document (shown as [chunk i/N]), and its exact character range in the parent. That is what lets a result say "this came from section 4 of the handbook" rather than handing you an anonymous fragment. kx datasets list shows the passage count alongside the parent-document count.

The stale-index refusal

Search compares your question's vector to the passage vectors already stored. Those vectors only mean anything if they were produced by the same embedding model with the same chunk settings. Change the embedding model, or the chunk size or overlap, and the stored vectors now live in a different space — comparing against them produces results that look fine and are quietly wrong.

The runtime refuses instead. At first ingest it stamps the corpus with a fingerprint of everything that makes an index incompatible: the embedding model id, its pooling and dimension, the chunker version, the chunk size and overlap, the word-tokenizer version and its stopword setting. On a later ingest or query it recomputes that fingerprint and compares.

If it differs, you get FAILED_PRECONDITION with a message saying the dataset was indexed under a different embed model or chunk config, and to create a new dataset or re-ingest to rebuild. You do not get a silently mis-ranked answer.

Note what is not in the fingerprint: the RRF constant, the MMR trade-off, and the BM25 scoring constants. Those change the display order only, never what was indexed, so an operator can retune them without a rebuild.

Embedding quality

Retrieval is only as good as its embeddings, and the runtime is candid about this: a general chat model produces weak sentence embeddings. It will mis-rank paraphrases — the thing dense search exists to catch. Use a dedicated embedding model instead and point KX_SERVE_EMBED_MODEL at it.

The runtime tells you when you have this wrong rather than letting you discover it through bad answers. When the configured embedder is a chat model, kx datasets ingest prints a one-line advisory on stderr recommending a dedicated embedder, and kx info flags it. Retrieval is never blocked — hybrid search and chunking lift quality either way — but the ceiling is lower.

If there is no embedder at all, text ingest and text query answer FAILED_PRECONDITION with actionable guidance. The SDKs also accept a pre-computed vector per document and per query, which needs no server-side model at all.

Let an agent search it

You do not have to run the queries yourself. Hand a dataset to an agent and it searches on its own, reading passages and re-querying across turns:

kx agent run --goal "summarise our refund policy" --dataset handbook

That is the same retrieval pipeline described above, reached through a read-only search tool inside a bounded loop. On a gateway without the dataset feature the run degrades to a plain agent with no corpus. See Agentic RAG for how the loop decides when it has enough.