andras/great-ai
andras/great-ai
Watch
1
0
Fork
You've already forked great-ai
 0
Code Issues Pull requests 2 Projects Releases 4 Packages 1 Wiki Activity Actions

Modernise docs

Browse source
This commit is contained in:
Andras Schmelczer 2026-10-28 12:50:43 +00:00
commit 37b4c17f59
5 changed files with 8196 additions and 3218 deletions
Download patch file Download diff file Expand all files Collapse all files

43
docs/examples/scibert/additional-files.md
View file

@ -37,15 +37,19 @@ numpy==1.23.0
## Dockerfile
```Dockerfile title="Dockerfile"
# syntax=docker/dockerfile:1
FROM schmelczera/great-ai:v0.1.6
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
COPY requirements.txt ./
RUN pip install --no-cache-dir --requirement requirements.txt
RUN uv pip install --system --no-cache --requirement requirements.txt
RUN --mount=type=secret,id=config,target=/run/secrets/config.ini,required=true \
large-file --backend s3 --secrets /run/secrets/config.ini --cache scibert-highlights
COPY . ./
RUN large-file --backend s3 --secrets s3.ini --cache scibert-highlights
CMD ["deploy.ipynb"]
```
@ -59,6 +63,9 @@ CMD ["deploy.ipynb"]
data
.gitignore
.env
config.ini
s3.ini
mongo.ini
.vscode
.dockerignore
Dockerfile
@ -73,3 +80,33 @@ README.md
!!! Note ".gitignore"
A very similar looking `.gitignore` file should also be present.
## Build and run with model caching
Use [Docker BuildKit secrets](https://docs.docker.com/build/building/secrets/) to make the build configuration available to the model-cache command without copying credentials into the image. The application's `config.ini` above also refers to runtime environment variables such as `MONGO_CONNECTION_STRING`. For the build, prepare a separate `config.ini` outside the project, for example at `/secure/build/config.ini`, containing only these four S3 settings with your actual values:
```ini title="/secure/build/config.ini"
AWS_REGION_NAME = eu-west-2
AWS_ACCESS_KEY_ID = YOUR_BUILD_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY = YOUR_BUILD_SECRET_ACCESS_KEY
LARGE_FILES_BUCKET_NAME = my-orgs-large-files
```
Use concrete values here: the build container does not inherit your shell's environment for resolving `ENV:` entries. After uploading `scibert-highlights` in the training notebook, build from the project's top-level directory:
```sh
DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain \
--secret id=config,src=/secure/build/config.ini --tag scibert-highlights .
```
`--no-cache` makes Docker run the model-cache step even if a previous image layer exists. The model stays in the image, while the secret mount is removed after that step. The `.dockerignore` above also prevents the application's `config.ini` from being included by `COPY . ./`.
At runtime, mount the application's `config.ini` and pass the environment variables it references from your configured shell:
```sh
docker run --rm -p 6060:6060 \
--mount type=bind,source="$(pwd)/config.ini",target=/app/config.ini,readonly \
--env MONGO_CONNECTION_STRING --env AWS_REGION_NAME \
--env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY \
--env LARGE_FILES_BUCKET_NAME scibert-highlights
```

2185
docs/examples/scibert/data.ipynb
View file

File diff suppressed because one or more lines are too long

296
docs/examples/scibert/deploy.ipynb
View file

@ -2,58 +2,93 @@
"cells": [
{
"cell_type": "markdown",
"id": "cell-0",
"metadata": {},
"source": [
"# Create an inference function\n",
"\n",
"Everything is ready to wrap the previously trained model and deploy it. \n",
"Run `data.ipynb` and `train.ipynb` first in the same local/full mode.\n",
"This notebook loads their local model and tokenizer; no S3 credentials,\n",
"MongoDB server, or model downloads are needed for the default demo.\n",
"The tiny demo model's scores have no scientific interpretation."
]
},
{
"cell_type": "markdown",
"id": "85c2ee1d",
"metadata": {},
"source": [
"## Local setup\n",
"\n",
"First, we need to configure the LargeFileBackend, the TracingDatabase and GreatAI."
"Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n",
"See **Run notebooks locally** in the repository README for the kernel, execution order,\n",
"and full-data options. Generated artifacts stay under `.cache/notebooks/`.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 9,
"id": "3775e11d",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226mThe value of `ENVIRONMENT` contains the \"ENV` prefix but `ENVIRONMENT` is not defined as an environment variable, using the default value defined above (`DEVELOPMENT`)\u001b[0m\n",
"\u001b[38;5;196mConfiguration has been already initialised, overwriting.\n",
"Make sure to call `configure()` before importing your application code.\u001b[0m\n",
"\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
"\u001b[38;5;39mMongoDbDriver has been already configured: skipping initialisation\u001b[0m\n",
"\u001b[38;5;39mLargeFileS3 has been already configured: skipping initialisation\u001b[0m\n",
"\u001b[38;5;39mGreatAI (v0.1.6): configured ✅\u001b[0m\n",
"\u001b[38;5;39m 🔩 tracing_database: MongoDbDriver\u001b[0m\n",
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileS3\u001b[0m\n",
"\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
"\u001b[38;5;39mGreatAI (v0.1.11): configured ✅\u001b[0m\n",
"\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
"\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n",
"\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n",
"\u001b[38;5;39m 🔩 prediction_cache_size: 4096\u001b[0m\n",
"\u001b[38;5;39m 🔩 dashboard_table_size: 100\u001b[0m\n",
"\u001b[38;5;226mYou still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
"\u001b[38;5;226m> Find out more at https://se-ml.github.io/practices\u001b[0m\n"
"\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n",
"\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Notebook artifacts: /Users/schmelczer/Projects/great-ai/.cache/notebooks/scibert\n"
]
}
],
"source": [
"from great_ai.utilities import ConfigFile\n",
"from great_ai.large_file import LargeFileS3\n",
"from great_ai import configure, MongoDbDriver\n",
"import os\n",
"import sys\n",
"from pathlib import Path\n",
"\n",
"configuration = ConfigFile(\"config.ini\")\n",
"# Works when opened from the repository root or any notebook directory.\n",
"_start = Path(globals().get(\"__file__\", Path.cwd())).resolve()\n",
"ROOT = next(\n",
" p\n",
" for p in [_start, *_start.parents]\n",
" if (p / \"pyproject.toml\").is_file() and (p / \"great_ai\").is_dir()\n",
")\n",
"sys.path.insert(0, str(ROOT / \"docs\"))\n",
"from notebook_support import setup_notebook, mag_lines\n",
"\n",
"LargeFileS3.configure_credentials_from_file(configuration)\n",
"MongoDbDriver.configure_credentials_from_file(configuration)\n",
"\n",
"configure(\n",
" dashboard_table_size=100, # traces are small, we can show many\n",
" prediction_cache_size=4096, # predictions are expensive, cache them\n",
")"
"FULL_RUN = os.environ.get(\"GREAT_AI_NOTEBOOK_FULL\") == \"1\"\n",
"WORK_DIR = setup_notebook(ROOT, \"scibert\")\n",
"DATA_DIR = ROOT / \"docs/tutorial/data\""
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "cell-3",
"metadata": {},
"outputs": [],
"source": [
"# The setup cell selects local model storage and a local TinyDB tracing database."
]
},
{
"cell_type": "markdown",
"id": "cell-4",
"metadata": {},
"source": [
"For a pleasant developer experience, we create some typed models that will show up in the automatically generated OpenAPI schema specification and will also provide runtime type validation."
@ -61,7 +96,8 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 11,
"id": "cell-5",
"metadata": {},
"outputs": [],
"source": [
@ -82,6 +118,7 @@
},
{
"cell_type": "markdown",
"id": "cell-6",
"metadata": {},
"source": [
"Even though `@use_model` caches the remote files locally and it also handles deserialising objects, we only use it to store a directory. In this case, it gives back a path, the path to that directory. So, we need to load the files from that folder ourselves. In order to only load it once per process, we create a small model loader helper function.\n",
@ -91,58 +128,44 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 12,
"id": "cell-7",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39mFetching cached versions of scibert-highlights\u001b[0m\n",
"\u001b[38;5;39mLatest version of scibert-highlights is 0 (from versions: 0)\u001b[0m\n",
"\u001b[38;5;39mFile scibert-highlights-0 found in cache\u001b[0m\n"
]
}
],
"source": [
"from functools import lru_cache\n",
"from great_ai import use_model\n",
"from pathlib import Path\n",
"from typing import Tuple\n",
"from transformers import (\n",
" PreTrainedModel,\n",
" PreTrainedTokenizer,\n",
")\n",
"from transformers import (\n",
" AutoConfig,\n",
" AutoModelForSequenceClassification,\n",
" AutoTokenizer,\n",
")\n",
"from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
"\n",
"_tokenizer: PreTrainedTokenizer = None\n",
"_loaded_model: PreTrainedModel = None\n",
"\n",
"@lru_cache(maxsize=1)\n",
"def load_tokenizer_and_model(model_path):\n",
" tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)\n",
" model = AutoModelForSequenceClassification.from_pretrained(\n",
" model_path, local_files_only=True, attn_implementation=\"eager\"\n",
" )\n",
" model.eval()\n",
" return tokenizer, model\n",
"\n",
"\n",
"@use_model(\"scibert-highlights\", version=\"latest\", model_kwarg_name=\"model_path\")\n",
"def get_tokenizer_and_model(\n",
" model_path: Path, original_model: str = \"allenai/scibert_scivocab_uncased\"\n",
") -> Tuple[PreTrainedTokenizer, PreTrainedModel]:\n",
" global _tokenizer, _loaded_model\n",
"\n",
" if _tokenizer is None:\n",
" _tokenizer = AutoTokenizer.from_pretrained(original_model)\n",
"\n",
" if _loaded_model is None:\n",
" config = AutoConfig.from_pretrained(\n",
" model_path, output_hidden_states=True, output_attentions=True\n",
" )\n",
" _loaded_model = AutoModelForSequenceClassification.from_pretrained(\n",
" model_path, config=config\n",
" )\n",
"\n",
" return _tokenizer, _loaded_model"
"def get_tokenizer_and_model(model_path: Path):\n",
" return load_tokenizer_and_model(model_path)"
]
},
{
"cell_type": "markdown",
"id": "cell-8",
"metadata": {},
"source": [
"Finally, implement the inference function."
@ -150,76 +173,40 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 13,
"id": "cell-9",
"metadata": {},
"outputs": [],
"source": [
"from great_ai import GreatAI\n",
"from great_ai.utilities import clean\n",
"\n",
"import re\n",
"import numpy as np\n",
"import torch\n",
"from transformers.modeling_outputs import SequenceClassifierOutput\n",
"\n",
"\n",
"@GreatAI.create\n",
"def find_highlights(sentence: str) -> EvaluatedSentence:\n",
" \"\"\"Get the interestingness prediction of the input sentence using SciBERT.\n",
" \"\"\"Score a sentence and expose the final layer's CLS attention per token.\n",
"\n",
" Run the SciBERT model in inference mode and evaluate the sentence.\n",
" Additionally, provide explanation in the form of the last layer's sum attention\n",
" between `[CLS]` and the other tokens.\n",
" Attention weights are a model diagnostic, not a causal explanation.\n",
" \"\"\"\n",
"\n",
" tokenizer, loaded_model = get_tokenizer_and_model()\n",
" tokenizer, model = get_tokenizer_and_model()\n",
" sentence = clean(sentence, convert_to_ascii=True, remove_brackets=True)\n",
"\n",
" tensors = tokenizer(sentence, return_tensors=\"pt\", truncation=True, max_length=512)\n",
"\n",
" with torch.inference_mode():\n",
" result: SequenceClassifierOutput = loaded_model(**tensors)\n",
" positive_likelihood = torch.nn.Softmax(dim=1)(result.logits)[0][1]\n",
" tokens = tensors[\"input_ids\"][0]\n",
"\n",
" attentions = np.sum(result.attentions[-1].numpy()[0], axis=0)[0][1:-1]\n",
" # Tuple of `torch.FloatTensor` (one for each layer) of shape\n",
" # `(batch_size, num_heads, sequence_length, sequence_length)`.\n",
"\n",
" explanation = []\n",
"\n",
" token_attentions = list(zip(attentions, tokens[1:-1]))\n",
" for token in re.split(r\"([ .,])\", sentence):\n",
" token = token.strip()\n",
" if not token:\n",
" continue\n",
" bert_tokens = tokenizer(\n",
" token, return_tensors=\"pt\", truncation=True, max_length=512\n",
" )[\"input_ids\"][0][\n",
" 1:-1\n",
" ] # truncation=True needed to fix `RuntimeError: Already borrowed`\n",
" weight = 0\n",
" for t1 in bert_tokens:\n",
" if not token_attentions:\n",
" break\n",
" a, t2 = token_attentions.pop(0)\n",
" assert t1 == t2, sentence\n",
" weight += a\n",
" explanation.append(\n",
" Attention(\n",
" token=token if token in \".,\" else \" \" + token, weight=round(weight, 4)\n",
" )\n",
" )\n",
" if not token_attentions:\n",
" break\n",
"\n",
" return EvaluatedSentence(\n",
" score=positive_likelihood, text=sentence, explanation=explanation\n",
" )"
" result = model(**tensors, output_attentions=True)\n",
" score = torch.softmax(result.logits, dim=1)[0, 1].item()\n",
" weights = result.attentions[-1][0, :, 0, :].sum(dim=0).cpu().tolist()\n",
" tokens = tokenizer.convert_ids_to_tokens(tensors[\"input_ids\"][0].tolist())\n",
" explanation = [\n",
" Attention(token=token, weight=round(weight, 4))\n",
" for token, weight in zip(tokens[1:-1], weights[1:-1])\n",
" ]\n",
" return EvaluatedSentence(score=score, text=sentence, explanation=explanation)"
]
},
{
"cell_type": "markdown",
"id": "cell-10",
"metadata": {},
"source": [
"A simple test to see everything works. Note that the models list is filled by the `@use_model` call even though it's not on the main inference function."
@ -227,66 +214,61 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 14,
"id": "cell-11",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"W1028 12:49:11.667000 52910 torch/distributed/elastic/multiprocessing/redirects.py:35] NOTE: Redirects are currently not supported in MacOs.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6ef4b00220304248a94035042cb5fa15",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"(Trace[EvaluatedSentence]({'created': '2022-07-16T18:47:29.581701',\n",
" 'exception': None,\n",
" 'feedback': None,\n",
" 'logged_values': { 'arg:sentence:length': 51,\n",
" 'arg:sentence:value': 'Our solution has outperformed the '\n",
" 'state-of-the-art.'},\n",
" 'models': [{'key': 'scibert-highlights', 'version': 0}],\n",
" 'original_execution_time_ms': 7127.2063,\n",
" 'output': { 'explanation': [ {'token': ' Our', 'weight': 0.3993},\n",
" {'token': ' solution', 'weight': 0.3481},\n",
" {'token': ' has', 'weight': 0.2945},\n",
" {'token': ' outperformed', 'weight': 0.4011},\n",
" {'token': ' the', 'weight': 0.1484},\n",
" {'token': ' state-of-the-art', 'weight': 0.5727},\n",
" {'token': '.', 'weight': 7.775}],\n",
" 'score': 0.9991180300712585,\n",
" 'text': 'Our solution has outperformed the state-of-the-art.'},\n",
" 'tags': ['find_highlights', 'online', 'development'],\n",
" 'trace_id': '56e20e94-79df-4793-ae61-d20820ebe2d3'}),\n",
" Trace[EvaluatedSentence]({'created': '2022-07-16T18:47:37.020275',\n",
" 'exception': None,\n",
" 'feedback': None,\n",
" 'logged_values': { 'arg:sentence:length': 36,\n",
" 'arg:sentence:value': 'Their solution did not perform '\n",
" 'well.'},\n",
" 'models': [{'key': 'scibert-highlights', 'version': 0}],\n",
" 'original_execution_time_ms': 170.7057,\n",
" 'output': { 'explanation': [ {'token': ' Their', 'weight': 1.1475},\n",
" {'token': ' solution', 'weight': 0.8205},\n",
" {'token': ' did', 'weight': 0.3254},\n",
" {'token': ' not', 'weight': 0.2921},\n",
" {'token': ' perform', 'weight': 0.4293},\n",
" {'token': ' well', 'weight': 0.2772},\n",
" {'token': '.', 'weight': 4.4723}],\n",
" 'score': 0.12305451184511185,\n",
" 'text': 'Their solution did not perform well.'},\n",
" 'tags': ['find_highlights', 'online', 'development'],\n",
" 'trace_id': '7fcf8271-1738-4025-8305-d5a1e5100aea'}))"
"Loading weights: 0%| | 0/25 [00:00<?, ?it/s]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"score=0.4977932274341583 text='Our solution has outperformed the state-of-the-art.' explanation=[Attention(weight=0.1332, token='our'), Attention(weight=0.133, token='[UNK]'), Attention(weight=0.1332, token='[UNK]'), Attention(weight=0.1329, token='[UNK]'), Attention(weight=0.134, token='the'), Attention(weight=0.1342, token='[UNK]'), Attention(weight=0.1333, token='[UNK]'), Attention(weight=0.1332, token='[UNK]'), Attention(weight=0.1327, token='[UNK]'), Attention(weight=0.1316, token='the'), Attention(weight=0.1339, token='[UNK]'), Attention(weight=0.1325, token='[UNK]'), Attention(weight=0.1347, token='.')]\n",
"score=0.49779173731803894 text='Their solution did not perform well.' explanation=[Attention(weight=0.2217, token='[UNK]'), Attention(weight=0.2211, token='[UNK]'), Attention(weight=0.2215, token='[UNK]'), Attention(weight=0.221, token='[UNK]'), Attention(weight=0.2239, token='[UNK]'), Attention(weight=0.223, token='[UNK]'), Attention(weight=0.2243, token='.')]\n"
]
}
],
"source": [
"if __name__ == \"__main__\":\n",
" find_highlights(\n",
" \"Our solution has outperformed the state-of-the-art.\"\n",
" ), find_highlights(\"Their solution did not perform well.\")"
" from fastapi.testclient import TestClient\n",
"\n",
" first = find_highlights(\"Our solution has outperformed the state-of-the-art.\")\n",
" second = find_highlights(\"Their solution did not perform well.\")\n",
" assert 0 <= first.output.score <= 1\n",
" assert first.output.explanation\n",
" print(first.output)\n",
" print(second.output)\n",
" with TestClient(find_highlights.app) as client:\n",
" response = client.post(\n",
" \"/predict\", json={\"sentence\": \"Our method improves accuracy.\"}\n",
" )\n",
" response.raise_for_status()\n",
" assert 0 <= response.json()[\"output\"][\"score\"] <= 1"
]
},
{
"cell_type": "markdown",
"id": "cell-12",
"metadata": {},
"source": [
"In this case, the service is built as a docker image, pushed to our image registry and subsequent rolling update is performed in the production cluster.\n",
@ -296,7 +278,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.10.4 ('.env': venv)",
"display_name": ".venv (3.13.5)",
"language": "python",
"name": "python3"
},
@ -310,15 +292,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
}
"version": "3.13.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 5
}

4
docs/examples/scibert/index.md
View file

@ -2,8 +2,8 @@
This is a simplified example illustrating how `great-ai` is used in practice at [ScoutinScience](https://www.scoutinscience.com/){ target=_blank }. The subpages show `great-ai` in action by going over the lifecycle of fine-tuning and deploying a BERT-based software service.
??? note "Propriety data"
The purpose of this example is to show you different ways in which `great-ai` can assist you. The exact NLP task being solved is not central. Stemming from this and from the difficult nature of obtaining appropriate training data, the propriety dataset used for the experiments is not shared.
??? note "Local demo and proprietary data"
The original dataset is not shared. The notebooks now default to synthetic annotations and a tiny randomly initialised BERT so the full workflow runs offline on a CPU. These demo scores do not reproduce the historical SciBERT experiment. See the repository README's **Run notebooks locally** section for setup, full pretrained SciBERT, and supplying your own annotations.
## Objectives

485
docs/examples/scibert/train.ipynb
View file

@ -2,187 +2,180 @@
"cells": [
{
"cell_type": "markdown",
"id": "cell-0",
"metadata": {},
"source": [
"# Fine-tune SciBERT\n",
"# Fine-tune a BERT sentence classifier\n",
"\n",
"We are planning to do a simple classification task on scientific text. For that, [SciBERT](https://github.com/allenai/scibert) is an ideal model to fine-tune since it has been pretrained of academic publications.\n",
"Install the checkout's `[notebooks]` extra and run `data.ipynb` first.\n",
"The default is an offline CPU demonstration: a tiny randomly initialised\n",
"BERT with a local vocabulary, trained for one epoch. It verifies the\n",
"pipeline and does **not** reproduce SciBERT or its historical scores.\n",
"\n",
"This notebook was updated so that it can run in [Google Colab](https://colab.research.google.com/).\n",
"Set `GREAT_AI_NOTEBOOK_FULL=1` before starting Jupyter to fine-tune\n",
"`allenai/scibert_scivocab_uncased` instead. This downloads pretrained weights\n",
"and takes substantially more memory and time; a GPU is recommended.\n",
"Run the data and deployment notebooks in the same mode."
]
},
{
"cell_type": "markdown",
"id": "2346798f",
"metadata": {},
"source": [
"## Local setup\n",
"\n",
"First, we need to install the dependencies."
"Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n",
"See **Run notebooks locally** in the repository README for the kernel, execution order,\n",
"and full-data options. Generated artifacts stay under `.cache/notebooks/`.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"executionInfo": {
"elapsed": 2529,
"status": "ok",
"timestamp": 1656596749103,
"user_tz": -120
},
"id": "j7l0nD9hDQbB",
"outputId": "88a9931b-396a-4cf1-c659-8a7b098b3cdd"
},
"outputs": [],
"source": [
"!pip install transformers datasets great-ai > /dev/null"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load the training data from S3. (We have uploaded this to S3 in the `data` notebook.)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "824837fa",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
"\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
"\u001b[38;5;39mGreatAI (v0.1.11): configured ✅\u001b[0m\n",
"\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
"\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n",
"\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n",
"\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n",
"\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Notebook artifacts: /Users/schmelczer/Projects/great-ai/.cache/notebooks/scibert\n"
]
}
],
"source": [
"import os\n",
"import sys\n",
"from pathlib import Path\n",
"\n",
"# Works when opened from the repository root or any notebook directory.\n",
"_start = Path(globals().get(\"__file__\", Path.cwd())).resolve()\n",
"ROOT = next(\n",
" p\n",
" for p in [_start, *_start.parents]\n",
" if (p / \"pyproject.toml\").is_file() and (p / \"great_ai\").is_dir()\n",
")\n",
"sys.path.insert(0, str(ROOT / \"docs\"))\n",
"from notebook_support import setup_notebook, mag_lines\n",
"\n",
"FULL_RUN = os.environ.get(\"GREAT_AI_NOTEBOOK_FULL\") == \"1\"\n",
"WORK_DIR = setup_notebook(ROOT, \"scibert\")\n",
"DATA_DIR = ROOT / \"docs/tutorial/data\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "cell-3",
"metadata": {},
"outputs": [],
"source": [
"# Dependencies are installed once with uv pip install --editable \".[notebooks]\"."
]
},
{
"cell_type": "markdown",
"id": "cell-4",
"metadata": {},
"source": [
"Load the dataset saved locally by `data.ipynb`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "cell-5",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39mFetching cached versions of summary-train-dataset-small\u001b[0m\n",
"\u001b[38;5;39mLatest version of summary-train-dataset-small is 0 (from versions: 0)\u001b[0m\n",
"\u001b[38;5;39mFile summary-train-dataset-small-0 found in cache\u001b[0m\n"
]
}
],
"source": [
"from great_ai.large_file import LargeFileS3\n",
"from great_ai.large_file import LargeFileLocal\n",
"import json\n",
"\n",
"LargeFileS3.configure_credentials_from_file(\"config.ini\")\n",
"\n",
"with LargeFileS3(\"summary-train-dataset-small\", encoding=\"utf-8\") as f:\n",
" # splitting training and test data is done later by `datasets`\n",
"with LargeFileLocal(\"summary-train-dataset-small\", encoding=\"utf-8\") as f:\n",
" X, y = json.load(f)"
]
},
{
"cell_type": "markdown",
"id": "cell-6",
"metadata": {},
"source": [
"Finetune SciBERT, for more info about this step, check out [HuggingFace](https://huggingface.co/docs/transformers/training).\n",
"If you're only here for `great-ai`, feel free to skip the next cell."
"If you're only here for `great-ai`, feel free to skip the next cell.\n",
"\n",
"We use an 80/20 training/validation split. Early stopping and checkpoint selection both use validation binary F1 for positive label `1`. There is no independent test split in this example. The split and training seed is fixed for future runs, and the selected checkpoint's evaluation and run metadata are saved under `models/`.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"executionInfo": {
"elapsed": 118131,
"status": "ok",
"timestamp": 1656593941974,
"user_tz": -120
},
"id": "AL3etUQ3LtKN",
"outputId": "fe00589f-64dd-4b70-e612-3873b504c00a"
},
"execution_count": 4,
"id": "cell-7",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"W1028 12:48:58.270000 53390 torch/distributed/elastic/multiprocessing/redirects.py:35] NOTE: Redirects are currently not supported in MacOs.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "9c57de70e68a41ecbde5093bd671715a",
"model_id": "2c74823d6364447a96609229fbbc5512",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/1 [00:00<?, ?ba/s]"
"Map: 0%| | 0/60 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'loss': '0.6939', 'grad_norm': '0.5046', 'learning_rate': '1.667e-05', 'epoch': '1'}\n",
"{'eval_loss': '0.6939', 'eval_f1': '0', 'eval_runtime': '0.0123', 'eval_samples_per_second': '974.9', 'eval_steps_per_second': '81.24', 'epoch': '1'}\n"
]
},
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='130' max='650' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [130/650 01:43 < 07:01, 1.23 it/s, Epoch 10/50]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Epoch</th>\n",
" <th>Training Loss</th>\n",
" <th>Validation Loss</th>\n",
" <th>F1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>0.586800</td>\n",
" <td>0.512138</td>\n",
" <td>0.719101</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.411600</td>\n",
" <td>0.416675</td>\n",
" <td>0.849057</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.245600</td>\n",
" <td>0.417070</td>\n",
" <td>0.864000</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>0.147800</td>\n",
" <td>0.575878</td>\n",
" <td>0.852459</td>\n",
" </tr>\n",
" <tr>\n",
" <td>5</td>\n",
" <td>0.056800</td>\n",
" <td>0.474259</td>\n",
" <td>0.896552</td>\n",
" </tr>\n",
" <tr>\n",
" <td>6</td>\n",
" <td>0.022500</td>\n",
" <td>0.754236</td>\n",
" <td>0.843137</td>\n",
" </tr>\n",
" <tr>\n",
" <td>7</td>\n",
" <td>0.001000</td>\n",
" <td>0.857636</td>\n",
" <td>0.834783</td>\n",
" </tr>\n",
" <tr>\n",
" <td>8</td>\n",
" <td>0.000500</td>\n",
" <td>0.920232</td>\n",
" <td>0.869565</td>\n",
" </tr>\n",
" <tr>\n",
" <td>9</td>\n",
" <td>0.000300</td>\n",
" <td>0.970790</td>\n",
" <td>0.877193</td>\n",
" </tr>\n",
" <tr>\n",
" <td>10</td>\n",
" <td>0.000300</td>\n",
" <td>0.948689</td>\n",
" <td>0.862385</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"application/vnd.jupyter.widget-view+json": {
"model_id": "23492278f66e46f09e9eee0680c21b92",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"<IPython.core.display.HTML object>"
"Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
@ -192,29 +185,31 @@
"name": "stderr",
"output_type": "stream",
"text": [
"...\n",
"Deleting older checkpoint [models/checkpoint-39] due to args.save_total_limit\n",
"***** Running Evaluation *****\n",
" Num examples = 100\n",
" Batch size = 32\n",
"Saving model checkpoint to models/checkpoint-117\n",
"Configuration saved in models/checkpoint-117/config.json\n",
"Model weights saved in models/checkpoint-117/pytorch_model.bin\n",
"Deleting older checkpoint [models/checkpoint-52] due to args.save_total_limit\n",
"***** Running Evaluation *****\n",
" Num examples = 100\n",
" Batch size = 32\n",
"Saving model checkpoint to models/checkpoint-130\n",
"Configuration saved in models/checkpoint-130/config.json\n",
"Model weights saved in models/checkpoint-130/pytorch_model.bin\n",
"Deleting older checkpoint [models/checkpoint-78] due to args.save_total_limit\n",
"\n",
"\n",
"Training completed. Do not forget to share your model on huggingface.co/models =)\n",
"\n",
"\n",
"Loading best model from models/checkpoint-65 (score: 0.896551724137931).\n"
"[transformers] early stopping required metric_for_best_model, but did not find eval_f1 so early stopping is disabled\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'train_runtime': '0.0882', 'train_samples_per_second': '544.2', 'train_steps_per_second': '34.01', 'train_loss': '0.6939', 'epoch': '1'}\n",
"{'validation_loss': '0.6939', 'validation_f1': '0', 'validation_runtime': '0.0025', 'validation_samples_per_second': '4804', 'validation_steps_per_second': '400.3', 'epoch': '1'}\n"
]
},
{
"data": {
"text/plain": [
"{'validation_loss': 0.693895161151886,\n",
" 'validation_f1': 0.0,\n",
" 'validation_runtime': 0.0025,\n",
" 'validation_samples_per_second': 4804.013,\n",
" 'validation_steps_per_second': 400.334,\n",
" 'epoch': 1.0}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
@ -225,120 +220,193 @@
" Trainer,\n",
" TrainingArguments,\n",
" EarlyStoppingCallback,\n",
" set_seed,\n",
")\n",
"from pathlib import Path\n",
"import numpy as np\n",
"from datasets import Dataset, load_metric\n",
"from datasets import Dataset\n",
"from sklearn.metrics import f1_score\n",
"import torch\n",
"\n",
"MODEL = \"allenai/scibert_scivocab_uncased\"\n",
"BATCH_SIZE = 32\n",
"BATCH_SIZE = 8 if FULL_RUN else 16\n",
"SPLIT_SEED = 42\n",
"\n",
"set_seed(SPLIT_SEED)\n",
"\n",
"if FULL_RUN:\n",
" tokenizer = AutoTokenizer.from_pretrained(MODEL)\n",
" model = AutoModelForSequenceClassification.from_pretrained(MODEL, num_labels=2)\n",
"else:\n",
" import re\n",
" from transformers import BertConfig, BertForSequenceClassification, BertTokenizer\n",
"\n",
" torch.set_num_threads(2)\n",
" vocabulary = [\"[PAD]\", \"[UNK]\", \"[CLS]\", \"[SEP]\", \"[MASK]\"]\n",
" vocabulary += sorted(set(re.findall(r\"[a-z]+|[0-9]+|[^\\w\\s]\", \" \".join(X).lower())))\n",
" vocab_path = WORK_DIR / \"vocab.txt\"\n",
" vocab_path.write_text(\"\\n\".join(vocabulary) + \"\\n\", encoding=\"utf-8\")\n",
" tokenizer = BertTokenizer(vocab=str(vocab_path), do_lower_case=True)\n",
" model = BertForSequenceClassification(\n",
" BertConfig(\n",
" vocab_size=len(tokenizer),\n",
" hidden_size=32,\n",
" num_hidden_layers=1,\n",
" num_attention_heads=2,\n",
" intermediate_size=64,\n",
" max_position_embeddings=512,\n",
" num_labels=2,\n",
" )\n",
" )\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(MODEL)\n",
"model = AutoModelForSequenceClassification.from_pretrained(MODEL, num_labels=2)\n",
"data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n",
"\n",
"\n",
"def tokenize_function(v):\n",
" return tokenizer(v[\"text\"])\n",
"\n",
"\n",
"dataset = (\n",
" Dataset.from_dict({\"text\": X, \"label\": y})\n",
" .map(lambda v: tokenizer(v[\"text\"], truncation=True), batched=True)\n",
" .map(lambda v: tokenizer(v[\"text\"], truncation=True, max_length=128), batched=True)\n",
" .remove_columns(\"text\")\n",
" .train_test_split(test_size=0.2, shuffle=True) # test is actually validation\n",
" .train_test_split(test_size=0.2, shuffle=True, seed=SPLIT_SEED)\n",
")\n",
"\n",
"f1_score = load_metric(\"f1\")\n",
"# The split API calls the held-out partition \"test\"; we use it for validation.\n",
"dataset[\"validation\"] = dataset.pop(\"test\")\n",
"\n",
"\n",
"def compute_metrics(p):\n",
" pred, labels = p\n",
" pred = np.argmax(pred, axis=1)\n",
" return f1_score.compute(predictions=pred, references=labels)\n",
" predictions = (\n",
" p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions\n",
" )\n",
" return {\n",
" \"f1\": f1_score(\n",
" p.label_ids,\n",
" np.argmax(predictions, axis=1),\n",
" average=\"binary\",\n",
" pos_label=1,\n",
" zero_division=0,\n",
" )\n",
" }\n",
"\n",
"\n",
"training_args = TrainingArguments(\n",
" output_dir=Path(\"models\"),\n",
" output_dir=WORK_DIR / \"models\",\n",
" per_device_train_batch_size=BATCH_SIZE,\n",
" per_device_eval_batch_size=BATCH_SIZE,\n",
" save_total_limit=5,\n",
" num_train_epochs=50,\n",
" num_train_epochs=50 if FULL_RUN else 1,\n",
" save_strategy=\"epoch\",\n",
" evaluation_strategy=\"epoch\",\n",
" eval_strategy=\"epoch\",\n",
" logging_strategy=\"epoch\",\n",
" weight_decay=0.01,\n",
" metric_for_best_model=\"f1\",\n",
" load_best_model_at_end=True,\n",
" seed=SPLIT_SEED,\n",
" use_cpu=not FULL_RUN,\n",
" report_to=\"none\",\n",
" dataloader_pin_memory=FULL_RUN,\n",
" disable_tqdm=not FULL_RUN,\n",
")\n",
"\n",
"result = Trainer(\n",
"trainer = Trainer(\n",
" model=model,\n",
" processing_class=tokenizer,\n",
" args=training_args,\n",
" train_dataset=dataset[\"train\"],\n",
" eval_dataset=dataset[\"test\"],\n",
" eval_dataset=dataset[\"validation\"],\n",
" data_collator=data_collator,\n",
" compute_metrics=compute_metrics,\n",
" callbacks=[EarlyStoppingCallback(early_stopping_patience=5)],\n",
").train()"
")\n",
"result = trainer.train()\n",
"\n",
"# This evaluates the selected checkpoint on validation, not an untouched test set.\n",
"validation_metrics = trainer.evaluate(metric_key_prefix=\"validation\")\n",
"trainer.save_metrics(\"validation\", validation_metrics)\n",
"(WORK_DIR / \"models/validation-run.json\").write_text(\n",
" json.dumps(\n",
" {\n",
" \"mode\": \"scibert\" if FULL_RUN else \"tiny-random-bert-demo\",\n",
" \"split_seed\": SPLIT_SEED,\n",
" \"training_seed\": SPLIT_SEED,\n",
" \"validation_fraction\": 0.2,\n",
" \"train_samples\": len(dataset[\"train\"]),\n",
" \"validation_samples\": len(dataset[\"validation\"]),\n",
" \"metric\": \"binary_f1\",\n",
" \"positive_label\": 1,\n",
" \"best_checkpoint\": trainer.state.best_model_checkpoint,\n",
" \"best_validation_f1\": trainer.state.best_metric,\n",
" },\n",
" indent=2,\n",
" )\n",
" + \"\\n\",\n",
" encoding=\"utf-8\",\n",
")\n",
"validation_metrics"
]
},
{
"cell_type": "markdown",
"id": "cell-8",
"metadata": {},
"source": [
"The best macro F1-score on the test set is **0.89** which is (not surprisingly) substantially more than the SVM achieved. We have a great model, it's time to deploy it. But first, we have to store it in a secure place."
"The [saved historical run](https://great-ai.scoutinscience.com/examples/scibert/validation-results.json) selected a checkpoint with **validation binary F1 of 0.8966** (recorded value: 0.896551724137931). This validation split was also used for early stopping and checkpoint selection, so the number is not an independent test result or macro F1. The historical split seed was not recorded; the seeded code above defines future runs and does not reproduce that split. Its results are saved to `models/validation_results.json` and `models/validation-run.json`.\n",
"\n",
"To estimate performance after model selection, evaluate once on a separate untouched test split. We make no test-set performance claim here. Next, store the selected model for deployment."
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"executionInfo": {
"elapsed": 25368,
"status": "ok",
"timestamp": 1656594537509,
"user": {
"displayName": "Schmelczer András",
"userId": "08401926777942666437"
},
"user_tz": -120
},
"id": "fyNKltdquZSP",
"outputId": "e8c2cbb1-78e1-41a3-b7cf-b0cd573bc45d"
},
"execution_count": 5,
"id": "cell-9",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "634b947ab9ee4f20bef78f780d5441ba",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Configuration saved in pretrained/config.json\n",
"Model weights saved in pretrained/pytorch_model.bin\n"
"\u001b[38;5;39mFetching cached versions of scibert-highlights\u001b[0m\n",
"\u001b[38;5;39mFetching cached versions of scibert-highlights\u001b[0m\n",
"\u001b[38;5;39mFetching cached versions of scibert-highlights\u001b[0m\n",
"\u001b[38;5;39mModel scibert-highlights uploaded with version 0\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" adding: pretrained/ (stored 0%)\n",
" adding: pretrained/config.json (deflated 49%)\n",
" adding: pretrained/pytorch_model.bin (deflated 7%)\n"
]
"data": {
"text/plain": [
"'scibert-highlights:0'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from great_ai import save_model\n",
"\n",
"# save Torch model to local disk\n",
"model.save_pretrained(\"pretrained\")\n",
"\n",
"# upload model from local disk to S3\n",
"# (because the S3 credentials have been already set, `save_model` will use LargeFileS3)\n",
"save_model(\"pretrained\", key=\"scibert-highlights\")"
"# Trainer has loaded the selected validation checkpoint. Save its tokenizer\n",
"# too, so deployment never needs to download a different vocabulary.\n",
"trainer.save_model(str(WORK_DIR / \"pretrained\"))\n",
"tokenizer.save_pretrained(WORK_DIR / \"pretrained\")\n",
"save_model(WORK_DIR / \"pretrained\", key=\"scibert-highlights\", keep_last_n=2)"
]
},
{
"cell_type": "markdown",
"id": "cell-10",
"metadata": {},
"source": [
"Next: [Part 3](https://great-ai.scoutinscience.com/examples/scibert/deploy)"
@ -347,7 +415,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.10.4 ('.env': venv)",
"display_name": ".venv (3.13.5)",
"language": "python",
"name": "python3"
},
@ -361,14 +429,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
},
"vscode": {
"interpreter": {
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
}
"version": "3.13.5"
}
},
"nbformat": 4,
"nbformat_minor": 0
"nbformat_minor": 5
}