> ## Documentation Index
> Fetch the complete documentation index at: https://docs.omnia-voice.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets & formats

> Prepare, upload, and map training data for fine-tuning.

Fine-tuning trains on your data. You provide it either as a **training file**
(JSONL) or a **managed dataset**, mapped into one of the supported training
formats. Files and datasets are managed via the API at
`https://gateway.omnia-voice.com/v1` or the dashboard's fine-tuning wizard.

You can also curate a managed dataset straight from your
[logged traffic](/reference/request-logging#browsing-and-exporting);
filtered by task, model, or auto-detected [segment](/concepts/segments), with
a disjoint eval holdout. A dataset is a **frozen snapshot**: logs age out on
your retention window (7/30/90 days), but a dataset curated from them does
not: curate what matters before it expires.

<Tip>
  **Scope a dataset to one task.** A dataset built from mixed traffic teaches
  a muddle of behaviours; one scoped to a single objective teaches that
  objective well. Send a task label on every call with the `X-Omnia-Tag`
  header (e.g. `document-summary`), then filter by it when you curate. Because
  the task label is independent of the model, the *same task* spans every
  model you route it to; swap models or rewrite prompts, and the task
  identity (and its dataset) persists. Calls with no tag fall under untagged
  traffic.

  ```bash theme={null}
  curl https://gateway.omnia-voice.com/v1/chat/completions \
    -H "X-Omnia-Tag: document-summary" \
    -H "Authorization: Bearer $OMNIA_API_KEY" \
    -d '{ "model": "...", "messages": [...] }'
  ```
</Tip>

Curation isn't just a filter; it **cleans** as it builds. Only successful
responses are included; byte-identical duplicates (cache-replay collapse),
truncated answers (`finish_reason=length`), and empty replies are dropped, and
multi-turn conversations are reconstructed and folded so early turns aren't
over-trained. Every build reports exactly what it kept and why
("kept 4,812 of 6,003: 891 duplicates, 300 truncated…").

## Upload methods

<CardGroup cols={3}>
  <Card title="Direct file" icon="file-arrow-up">
    Upload a JSONL file directly as a training file.
  </Card>

  <Card title="Object storage (S3)" icon="bucket">
    Use an S3-compatible bucket as a dataset source with your credentials.
  </Card>

  <Card title="Chunked upload" icon="layer-group">
    Large files upload in parts (chunked/multipart) so big datasets upload
    reliably.
  </Card>
</CardGroup>

## Uploading a training file

Upload a JSONL training file with `POST /v1/fine_tuning/files`, and list your
files with `GET /v1/fine_tuning/files`.

<CodeGroup>
  ```bash cURL theme={null}
  # Upload
  curl https://gateway.omnia-voice.com/v1/fine_tuning/files \
    -H "Authorization: Bearer $OMNIA_API_KEY" \
    -F "file=@training.jsonl"

  # List
  curl https://gateway.omnia-voice.com/v1/fine_tuning/files \
    -H "Authorization: Bearer $OMNIA_API_KEY"
  ```

  ```python Python theme={null}
  import requests

  BASE = "https://gateway.omnia-voice.com/v1"
  headers = {"Authorization": f"Bearer {OMNIA_API_KEY}"}

  # Upload
  with open("training.jsonl", "rb") as f:
      file = requests.post(
          f"{BASE}/fine_tuning/files",
          headers=headers,
          files={"file": f},
      ).json()

  # List
  files = requests.get(f"{BASE}/fine_tuning/files", headers=headers).json()
  ```
</CodeGroup>

The response includes the file's id, which you reference as `trainingFileId` (or
`validationFileId`) when [creating a job](/fine-tuning/create-job).

## Training formats

When you use a managed dataset, you map its columns into **one of four** training
formats. Pick the format that matches your task:

<AccordionGroup>
  <Accordion title="Text — raw text completion" icon="align-left">
    Plain text. Map a single column to the text field. Use for continued
    pretraining or simple completion.

    ```json theme={null}
    { "type": "text", "text": "<column>" }
    ```
  </Accordion>

  <Accordion title="Prompts — prompt / completion pairs" icon="arrow-right">
    Prompt/completion pairs. Map one column to the prompt and one to the
    completion. Use for instruction tuning.

    ```json theme={null}
    { "type": "prompts", "prompt": "<column>", "completion": "<column>" }
    ```
  </Accordion>

  <Accordion title="Messages — chat format (role / content)" icon="comments">
    OpenAI chat format: role/content conversations. Map the column that holds
    the conversation. Use for chat/assistant fine-tuning.

    ```json theme={null}
    { "type": "messages", "messages": "<column>" }
    ```
  </Accordion>

  <Accordion title="Pretokenized — token-id arrays" icon="hashtag">
    Already-tokenized data as token-id arrays. Map `input_ids` (and optionally
    `labels`, `attention_mask`). Use when you control tokenization.

    ```json theme={null}
    { "type": "pretokenized", "input_ids": "<column>", "labels": "<column>" }
    ```
  </Accordion>
</AccordionGroup>

## Validation

JSONL is validated **before a job starts**. Line-level JSONL errors and format
problems are surfaced with the exact reason, so you fix them before spending on a
job rather than after. You can also preview dataset rows in the dashboard before
training.

## Using a dataset in a job

Once your file or dataset is ready, reference it when
[creating a fine-tuning job](/fine-tuning/create-job). A managed dataset is
converted to a **training\_file** automatically, using the column-mapping format
you chose; you then pass the resulting file id as `trainingFileId`.

<CardGroup cols={2}>
  <Card title="Create a job" icon="play" href="/fine-tuning/create-job">
    Start training with your prepared data.
  </Card>

  <Card title="Deploy the result" icon="server" href="/fine-tuning/deploy-model">
    Serve a trained model on a dedicated endpoint.
  </Card>
</CardGroup>