Export formats

What is inside each kind of export, field by field. How to choose an export and set its options is in Export.

File names

An export is named from the File name option (agnos-dataset unless you change it), followed by the date and time it was built:

agnos-dataset_20260923_141502.jsonl
agnos-dataset_20260923_141502.zip

Characters other than letters, digits, dots, dashes and underscores become a dash. If the name changes, the field shows the final one after Saved as.

RAG corpus

One JSONL file: one JSON record per line, one piece per record, in document order. This line is from a Word document cleaned with the RAG / Retrieval Corpus purpose:

{"text": "## 1. Working hours\n\nThe workshop is staffed from 08:00 to 18:00. Anyone working outside those hours tells their team lead first, in writing.", "chunk_id": "9f2c41d7a0b34e6c8d15f0a2b7c93e61_000001", "source": "staff-handbook.docx", "file_id": 12, "uploaded_at": "2026-09-23T09:41:07.218532Z", "quality_score": 0.84, "language_confidence": 0.0, "content_type": "prose", "chunk_index": 1, "heading_path": ["Northwind Tools Staff Handbook", "1. Working hours"]}

Which fields appear depends on the document and the project's Output settings. A field with nothing to say is left out: a Word document has no pages, so its pieces have no page numbers.

Field	What it holds
`text`	The cleaned piece.
`chunk_id`	An id for the piece, unique in the project.
`source`	The name of the file you added.
`file_id`	That file's number in Agnos.
`uploaded_at`	When the file was added, in UTC. With Source file details on.
`quality_score`	From 0 to 1; higher is cleaner. With Quality score on.
`token_count`	The piece's length in tokens. With Token count on; RAG / Retrieval Corpus leaves it off.
`token_ids`	The token numbers, when Token ids is ticked on Export.
`language_confidence`	How sure language detection was; 0 when it did not run.
`content_type`	`prose`, `structured`, `code` or `mixed`.
`training_quality`	A combined score from 0 to 1, from purposes other than RAG / Retrieval Corpus.
`record_fields`	Fields you chose to keep beside the text, from a JSON record or a spreadsheet row.
`chunk_index`	The piece's position in its document.
`page_start`, `page_end`	The PDF pages it came from.
`slide_start`, `slide_end`	The slides it came from.
`sheet`	The worksheet it came from.
`record`, `record_id`	The record or row it came from, and that record's own id if it has one.
`heading_path`	The headings the piece sits under, outermost first.
`restored`, `restored_from`	On a piece you kept with Keep this anyway: `true`, and the reasons it was removed for. Its `token_count` is an estimate, marked by `token_count_estimated`.

Cleaned documents

A zip with one plain text file per document, named after the original, and a manifest:

agnos-dataset_20260923_141502.zip
  faq.txt
  returns-policy.txt
  staff-handbook.txt
  manifest.json

If two documents share a name, the second becomes staff-handbook_2.txt. A document you changed in Review is rebuilt from its pieces, so it holds only what you kept.

Training data

A zip of records that hold only the text, one per line:

{"text": "## 1. Working hours\n\nThe workshop is staffed from 08:00 to 18:00. Anyone working outside those hours tells their team lead first, in writing."}

Without a validation split, the zip holds data.jsonl and manifest.json. With one, it holds train.jsonl, val.jsonl and manifest.json.

How the split is made is in Export. The manifest's leak_check counts any text that still ended up on both sides.

Everything

agnos-dataset_20260923_141502.zip
  data.jsonl               (train.jsonl and val.jsonl with a split)
  rejected.jsonl
  duplicates.jsonl
  near_duplicates.jsonl
  settings/<hash>.json
  manifest.json
  dataset_card.md

data.jsonl: The dataset, in the same record shape as the RAG corpus.
rejected.jsonl: Every piece not in the dataset, with its reasons and the stage that removed it: cleaning, quality, or review for pieces you left out (reason removed_by_reviewer). Cleaning settings matches each reason to the label Review shows.
duplicates.jsonl: Every piece left out because the same text was already there from another file, with the file it was kept in and the file it was dropped from.
near_duplicates.jsonl: Up to 5,000 pairs of nearly identical pieces, with how alike they are and whether the repeat was removed. It is written either way, so you can see what Keep one copy, counting near matches would remove before you choose it.
settings/<hash>.json: The settings each file was cleaned with, one file per distinct set. The manifest says which file used which.

A line from each list: two from rejected.jsonl, then one from duplicates.jsonl and one from near_duplicates.jsonl (texts shortened):

{"chunk_index": 5, "text": "Questions? Write to support.", "reasons": ["exact_duplicate"], "stage": "quality", "heading_path": ["Returns"], "source": "faq.md", "file_id": 13}
{"text": "We use cookies to improve your experience. Accept all", "reasons": ["web_boilerplate"], "stage": "cleaning", "source": "support-article.html", "file_id": 15}
{"text": "Returns are accepted within 30 days of delivery when the item is unused and in its original packaging.", "kept_in": "returns-policy.md", "kept_file_id": 11, "dropped_from": "faq.md", "dropped_file_id": 13, "chunk_index": 4}
{"similarity": 0.9412, "same_file": false, "removed": false, "first": {"file_id": 11, "file": "returns-policy.md", "chunk_index": 2, "page_start": null, "page_end": null, "heading_path": ["Returns"], "text": "..."}, "repeat": {"file_id": 13, "file": "faq.md", "chunk_index": 6, "page_start": null, "page_end": null, "heading_path": ["Returns"], "text": "..."}}

The manifest

manifest.json is in every zip. It records what the dataset is, what was taken out, and which settings made it, so anyone handed the dataset can answer those questions later.

generated_at, agnos_version, profile, dataset_name, options: When and how it was built. profile is rag, documents, training or bundle (Everything).
total_files, total_chunks, total_tokens, tokens_estimated: The totals. tokens_estimated is true when tokens were not counted and the total is estimated at about four characters per token.
quality: The lowest and the mean quality score.
files: One entry per file: pieces kept and removed, repeats found, tokens, the settings used and whether they are out of date, your Review decisions, and the side of the split.
duplicates, near_duplicates: How repeated text was handled, how much was found and removed, and from which files.
split: Whether and how the files were split, what is on each side, leak_check, and what the split cannot detect.
reviewer_decisions: How many pieces a person kept or left out by hand, counted apart from the rules.
stale_files, notes: Files cleaned with settings the project has since changed, and plain sentences about anything worth knowing.

The Cleaned documents manifest also lists each text file under documents.

The dataset card

dataset_card.md comes with Everything. It is the manifest written for a person, in Markdown. It starts like this:

# agnos-dataset

Generated: 2026-09-23T14:15:02.114873Z
Agnos version: 2.3.0
Export profile: bundle

## Summary
- **Files**: 5
- **Chunks**: 19
- **Tokens**: about 1,078
- **Quality** (mean / min): 0.78 / 0.49
- **Languages**: unknown: 19

It then lists the options used and notes on anything worth knowing. When they apply, it also covers repeats removed, decisions made in Review, and the split with its leak_check.