Andras Schmelczer
cf61b38b45
All checks were successful
Publish documentation to great-ai.schmelczer.dev / publish (push) Successful in 1m10s
Check / Lint, format & type checks (push) Successful in 1m55s
Check / Test on Python 3.12 (push) Successful in 3m8s
Check / Test on Python 3.13 (push) Successful in 2m32s
Check / Notebooks on Python 3.12 (push) Successful in 8m49s
Check / Notebooks on Python 3.13 (push) Successful in 7m53s
309 lines
8.4 KiB
Text
309 lines
8.4 KiB
Text
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-0",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Simple example: data engineering\n",
|
|
"\n",
|
|
"Train a scientific domain classifier with an explainable Naive Bayes model.\n",
|
|
"Run `data.ipynb`, `train.ipynb`, then `deploy.ipynb` in that order.\n",
|
|
"\n",
|
|
"The original example downloaded large Semantic Scholar corpus shards.\n",
|
|
"For a reproducible local run, we use the bundled MAG sentences instead,\n",
|
|
"wrapping each domain in a list to keep the same multilabel data interface.\n",
|
|
"By default we take 60 examples per class; `GREAT_AI_NOTEBOOK_FULL=1` uses\n",
|
|
"the entire bundled training split. These results are not the original\n",
|
|
"Semantic Scholar experiment's metrics.\n",
|
|
"\n",
|
|
"### Extract"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0df47c23",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Local setup\n",
|
|
"\n",
|
|
"Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n",
|
|
"See **Run notebooks locally** in the repository README for the kernel, execution order,\n",
|
|
"and full-data options. Run related notebooks from the same working directory\n",
|
|
"to share the default `.cache/` model store and `tracing_database.json` database.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "a800dc4d",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-10-28T22:47:31.221616Z",
|
|
"iopub.status.busy": "2026-10-28T22:47:31.221419Z",
|
|
"iopub.status.idle": "2026-10-28T22:47:31.227624Z",
|
|
"shell.execute_reply": "2026-10-28T22:47:31.226922Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from docs.notebook_support import DATA_DIR, FULL_RUN, mag_lines"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "cell-3",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-10-28T22:47:31.229286Z",
|
|
"iopub.status.busy": "2026-10-28T22:47:31.229129Z",
|
|
"iopub.status.idle": "2026-10-28T22:47:31.231400Z",
|
|
"shell.execute_reply": "2026-10-28T22:47:31.230864Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"PER_LABEL = None if FULL_RUN else 60"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "cell-4",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-10-28T22:47:31.232556Z",
|
|
"iopub.status.busy": "2026-10-28T22:47:31.232454Z",
|
|
"iopub.status.idle": "2026-10-28T22:47:31.318875Z",
|
|
"shell.execute_reply": "2026-10-28T22:47:31.318472Z"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Loaded 420 bundled MAG sentences\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import json\n",
|
|
"\n",
|
|
"records = [json.loads(line) for line in mag_lines(DATA_DIR / \"train.txt\", PER_LABEL)]\n",
|
|
"print(f\"Loaded {len(records)} bundled MAG sentences\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-5",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Transform\n",
|
|
"\n",
|
|
"Clean the text with `great_ai.utilities.clean` and keep a list of accepted\n",
|
|
"labels for each publication. Duplicate texts are combined before splitting\n",
|
|
"to keep every copy of a paper in the same split."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "cell-6",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-10-28T22:47:31.319705Z",
|
|
"iopub.status.busy": "2026-10-28T22:47:31.319649Z",
|
|
"iopub.status.idle": "2026-10-28T22:47:32.655835Z",
|
|
"shell.execute_reply": "2026-10-28T22:47:32.655342Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from great_ai.utilities import clean\n",
|
|
"\n",
|
|
"labels_by_text = {}\n",
|
|
"for record in records:\n",
|
|
" text = clean(record[\"text\"], convert_to_ascii=True)\n",
|
|
" labels_by_text.setdefault(text, set()).add(record[\"label\"])\n",
|
|
"preprocessed_data = [(text, sorted(labels)) for text, labels in labels_by_text.items()]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "cell-7",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-10-28T22:47:32.656875Z",
|
|
"iopub.status.busy": "2026-10-28T22:47:32.656706Z",
|
|
"iopub.status.idle": "2026-10-28T22:47:32.658407Z",
|
|
"shell.execute_reply": "2026-10-28T22:47:32.657982Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"X, y = zip(*preprocessed_data) # X is the input, y is the expected output"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-8",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Load\n",
|
|
"\n",
|
|
"Store the dataset in the default local TinyDB tracing database,\n",
|
|
"`tracing_database.json`. Run the data, training and deployment notebooks\n",
|
|
"from the same working directory to share it. A dataset tag makes rerunning\n",
|
|
"this cell replace just this example's ground truth instead of duplicating it."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "cell-9",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-10-28T22:47:32.659171Z",
|
|
"iopub.status.busy": "2026-10-28T22:47:32.659099Z",
|
|
"iopub.status.idle": "2026-10-28T22:47:32.670028Z",
|
|
"shell.execute_reply": "2026-10-28T22:47:32.669511Z"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226mCannot find credentials files, defaulting to using ParallelTinyDbDriver\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226mCannot find credentials files, defaulting to using LargeFileLocal\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;39mGreatAI (v0.1.11): configured ✅\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226mYou still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226m> Find out more at https://se-ml.github.io/practices\u001b[0m\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import random\n",
|
|
"from great_ai import add_ground_truth, delete_ground_truth\n",
|
|
"\n",
|
|
"random.seed(42)\n",
|
|
"delete_ground_truth(\"local-domain-example\")\n",
|
|
"add_ground_truth(\n",
|
|
" X, y, tags=\"local-domain-example\", train_split_ratio=0.8, test_split_ratio=0.2\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-10",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Next: [Part 2](https://great-ai.schmelczer.dev/examples/simple/train)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": ".venv (3.13.5)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.13.5"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|