andras/great-ai
andras/great-ai
Watch
1
0
Fork
You've already forked great-ai
 0
Code Issues Pull requests 2 Projects Releases 4 Packages 1 Wiki Activity Actions

Modernise docs

Browse source
This commit is contained in:
Andras Schmelczer 2026-10-28 12:50:43 +00:00
commit 37b4c17f59
1 changed files with 8196 additions and 3218 deletions
Download patch file Download diff file Expand all files Collapse all files

280
docs/examples/simple/data.ipynb
View file

@ -2,141 +2,148 @@
"cells": [
{
"cell_type": "markdown",
"id": "cell-0",
"metadata": {},
"source": [
"# Simple example: data engineering\n",
"\n",
"Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best practices. In short, we train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus) by taking full advantage of `great-ai`. Subsequently, we create a production-ready deployment.\n",
"Train a scientific domain classifier with an explainable Naive Bayes model.\n",
"Run `data.ipynb`, `train.ipynb`, then `deploy.ipynb` in that order.\n",
"\n",
"![position of this step in the lifecycle](https://great-ai.scoutinscience.com/media/scope-data.svg)\n",
"> The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.\n",
"The original example downloaded large Semantic Scholar corpus shards.\n",
"For a reproducible local run, we use the bundled MAG sentences instead,\n",
"wrapping each domain in a list to keep the same multilabel data interface.\n",
"By default we take 60 examples per class; `GREAT_AI_NOTEBOOK_FULL=1` uses\n",
"the entire bundled training split. These results are not the original\n",
"Semantic Scholar experiment's metrics.\n",
"\n",
"Since the true scope of `great-ai` is the phase between proof-of-concept code and production-ready service, it is predominantly used in the [deployment notebook](https://great-ai.scoutinscience.com/examples/simple/deploy). Feel free to skip there, or continue reading if you'd like to see the full picture.\n",
"### Extract"
]
},
{
"cell_type": "markdown",
"id": "0df47c23",
"metadata": {},
"source": [
"## Local setup\n",
"\n",
"### Extract\n",
"\n",
"This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n",
"\n",
"In this example, we download the semantic scholar dataset from a public S3 bucket."
"Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n",
"See **Run notebooks locally** in the repository README for the kernel, execution order,\n",
"and full-data options. Generated artifacts stay under `.cache/notebooks/`.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"MAX_CHUNK_COUNT = 4"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Processing 4 out of the 6002 available chunks'"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import urllib.request\n",
"from random import shuffle\n",
"\n",
"manifest = (\n",
" urllib.request.urlopen(\n",
" \"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/\"\n",
" \"open-corpus/2022-02-01/manifest.txt\"\n",
" )\n",
" .read()\n",
" .decode()\n",
") # a list of available chunks separated by '\\n' characters\n",
"\n",
"lines = manifest.split()\n",
"shuffle(lines)\n",
"chunks = lines[:MAX_CHUNK_COUNT]\n",
"\n",
"f\"\"\"Processing {len(chunks)} out of the {\n",
" len(manifest.split())\n",
"} available chunks\"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Transform\n",
"\n",
"- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n",
"- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n",
"- We will speed up processing using `great_ai.utilities.simple_parallel_map`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "a800dc4d",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 4/4 [04:22<00:00, 65.51s/it] \n"
"\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
"\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
"\u001b[38;5;39mGreatAI (v0.1.11): configured ✅\u001b[0m\n",
"\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
"\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n",
"\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n",
"\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n",
"\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Notebook artifacts: /Users/schmelczer/Projects/great-ai/.cache/notebooks/simple\n"
]
}
],
"source": [
"from typing import List, Tuple\n",
"import json\n",
"import gzip\n",
"from great_ai.utilities import (\n",
" simple_parallel_map,\n",
" clean,\n",
" is_english,\n",
" predict_language,\n",
" unchunk,\n",
"import os\n",
"import sys\n",
"from pathlib import Path\n",
"\n",
"# Works when opened from the repository root or any notebook directory.\n",
"_start = Path(globals().get(\"__file__\", Path.cwd())).resolve()\n",
"ROOT = next(\n",
" p\n",
" for p in [_start, *_start.parents]\n",
" if (p / \"pyproject.toml\").is_file() and (p / \"great_ai\").is_dir()\n",
")\n",
"sys.path.insert(0, str(ROOT / \"docs\"))\n",
"from notebook_support import setup_notebook, mag_lines\n",
"\n",
"FULL_RUN = os.environ.get(\"GREAT_AI_NOTEBOOK_FULL\") == \"1\"\n",
"WORK_DIR = setup_notebook(ROOT, \"simple\")\n",
"DATA_DIR = ROOT / \"docs/tutorial/data\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "cell-3",
"metadata": {},
"outputs": [],
"source": [
"PER_LABEL = None if FULL_RUN else 60"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "cell-4",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded 420 bundled MAG sentences\n"
]
}
],
"source": [
"import json\n",
"\n",
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
" response = urllib.request.urlopen(\n",
" f\"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/\"\n",
" f\"open-corpus/2022-02-01/{chunk_key}\"\n",
" ) # a gzipped JSON Lines file\n",
"records = [json.loads(line) for line in mag_lines(DATA_DIR / \"train.txt\", PER_LABEL)]\n",
"print(f\"Loaded {len(records)} bundled MAG sentences\")"
]
},
{
"cell_type": "markdown",
"id": "cell-5",
"metadata": {},
"source": [
"### Transform\n",
"\n",
" decompressed = gzip.decompress(response.read())\n",
" decoded = decompressed.decode()\n",
" chunk = [json.loads(line) for line in decoded.split(\"\\n\") if line]\n",
"\n",
" # Transform\n",
" return [\n",
" (\n",
" clean(\n",
" f'{c[\"title\"]} {c[\"paperAbstract\"]} '\n",
" f'{c[\"journalName\"]} {c[\"venue\"]}',\n",
" convert_to_ascii=True,\n",
" ), # The text is cleaned to remove common artifacts\n",
" c[\"fieldsOfStudy\"],\n",
" ) # Create pairs of `(text, [...domains])`\n",
" for c in chunk\n",
" if (c[\"fieldsOfStudy\"] and is_english(predict_language(c[\"paperAbstract\"])))\n",
" ]\n",
"\n",
"\n",
"preprocessed_data = unchunk(\n",
" simple_parallel_map(preprocess_chunk, chunks, concurrency=4)\n",
")"
"Clean the text with `great_ai.utilities.clean` and keep a list of accepted\n",
"labels for each publication. Duplicate texts are combined before splitting\n",
"to keep every copy of a paper in the same split."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "cell-6",
"metadata": {},
"outputs": [],
"source": [
"from great_ai.utilities import clean\n",
"\n",
"labels_by_text = {}\n",
"for record in records:\n",
" text = clean(record[\"text\"], convert_to_ascii=True)\n",
" labels_by_text.setdefault(text, set()).add(record[\"label\"])\n",
"preprocessed_data = [(text, sorted(labels)) for text, labels in labels_by_text.items()]"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "cell-7",
"metadata": {},
"outputs": [],
"source": [
@ -145,58 +152,37 @@
},
{
"cell_type": "markdown",
"id": "cell-8",
"metadata": {},
"source": [
"### Load\n",
"\n",
"Upload the dataset (or a part of it) to a central repository using `great_ai.add_ground_truth`. This step automatically tags each data-point with a split label according to the ratios we set. Additional tags can be also given.\n",
"\n",
"#### Production-ready backend\n",
"\n",
"The MongoDB driver is automatically configured if `mongo.ini` exists with the following scheme:\n",
"\n",
"```ini\n",
"mongo_connection_string=mongodb://localhost:27017/\n",
"mongo_database=my_great_ai_db\n",
"```\n",
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n",
"\n",
"Otherwise, TinyDB is used which is just a local JSON file."
"Store the dataset in the local TinyDB tracing database. All notebooks in\n",
"this example share `.cache/notebooks/simple/` (or `simple-full/`). No S3 or\n",
"MongoDB credentials are needed. A dataset tag makes rerunning this cell\n",
"replace just this example's ground truth instead of duplicating it."
]
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 6,
"id": "cell-9",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
"\u001b[38;5;226mCannot find credentials files, defaulting to using ParallelTinyDbDriver\u001b[0m\n",
"\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
"\u001b[38;5;226mCannot find credentials files, defaulting to using LargeFileLocal\u001b[0m\n",
"\u001b[38;5;39mGreatAI (v0.1.6): configured ✅\u001b[0m\n",
"\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
"\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n",
"\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n",
"\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n",
"\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n",
"\u001b[38;5;226mYou still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
"\u001b[38;5;226m> Find out more at https://se-ml.github.io/practices\u001b[0m\n"
]
}
],
"outputs": [],
"source": [
"from great_ai import add_ground_truth\n",
"import random\n",
"from great_ai import add_ground_truth, delete_ground_truth\n",
"\n",
"add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)"
"random.seed(42)\n",
"delete_ground_truth(\"local-domain-example\")\n",
"add_ground_truth(\n",
" X, y, tags=\"local-domain-example\", train_split_ratio=0.8, test_split_ratio=0.2\n",
")"
]
},
{
"cell_type": "markdown",
"id": "cell-10",
"metadata": {},
"source": [
"### Next: [Part 2](https://great-ai.scoutinscience.com/examples/simple/train)"
@ -205,7 +191,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.10.4 ('.env': venv)",
"display_name": ".venv (3.13.5)",
"language": "python",
"name": "python3"
},
@ -219,15 +205,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
}
"version": "3.13.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 5
}