great-ai/docs/examples/simple/data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0",
   "metadata": {},
   "source": [
    "# Simple example: data engineering\n",
    "\n",
    "Train a scientific domain classifier with an explainable Naive Bayes model.\n",
    "Run `data.ipynb`, `train.ipynb`, then `deploy.ipynb` in that order.\n",
    "\n",
    "The original example downloaded large Semantic Scholar corpus shards.\n",
    "For a reproducible local run, we use the bundled MAG sentences instead,\n",
    "wrapping each domain in a list to keep the same multilabel data interface.\n",
    "By default we take 60 examples per class; `GREAT_AI_NOTEBOOK_FULL=1` uses\n",
    "the entire bundled training split. These results are not the original\n",
    "Semantic Scholar experiment's metrics.\n",
    "\n",
    "### Extract"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0df47c23",
   "metadata": {},
   "source": [
    "## Local setup\n",
    "\n",
    "Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n",
    "See **Run notebooks locally** in the repository README for the kernel, execution order,\n",
    "and full-data options. Run related notebooks from the same working directory\n",
    "to share the default `.cache/` model store and `tracing_database.json` database.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a800dc4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from docs.notebook_support import DATA_DIR, FULL_RUN, mag_lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cell-3",
   "metadata": {},
   "outputs": [],
   "source": [
    "PER_LABEL = None if FULL_RUN else 60"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cell-4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 420 bundled MAG sentences\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "records = [json.loads(line) for line in mag_lines(DATA_DIR / \"train.txt\", PER_LABEL)]\n",
    "print(f\"Loaded {len(records)} bundled MAG sentences\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-5",
   "metadata": {},
   "source": [
    "### Transform\n",
    "\n",
    "Clean the text with `great_ai.utilities.clean` and keep a list of accepted\n",
    "labels for each publication. Duplicate texts are combined before splitting\n",
    "to keep every copy of a paper in the same split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cell-6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from great_ai.utilities import clean\n",
    "\n",
    "labels_by_text = {}\n",
    "for record in records:\n",
    "    text = clean(record[\"text\"], convert_to_ascii=True)\n",
    "    labels_by_text.setdefault(text, set()).add(record[\"label\"])\n",
    "preprocessed_data = [(text, sorted(labels)) for text, labels in labels_by_text.items()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cell-7",
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = zip(*preprocessed_data)  # X is the input, y is the expected output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-8",
   "metadata": {},
   "source": [
    "### Load\n",
    "\n",
    "Store the dataset in the default local TinyDB tracing database,\n",
    "`tracing_database.json`. Run the data, training and deployment notebooks\n",
    "from the same working directory to share it. A dataset tag makes rerunning\n",
    "this cell replace just this example's ground truth instead of duplicating it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cell-9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from great_ai import add_ground_truth, delete_ground_truth\n",
    "\n",
    "random.seed(42)\n",
    "delete_ground_truth(\"local-domain-example\")\n",
    "add_ground_truth(\n",
    "    X, y, tags=\"local-domain-example\", train_split_ratio=0.8, test_split_ratio=0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-10",
   "metadata": {},
   "source": [
    "### Next: [Part 2](https://great-ai.schmelczer.dev/examples/simple/train)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv (3.13.5)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}