andras/great-ai
andras/great-ai
Watch
1
0
Fork
You've already forked great-ai
 0
Code Issues Pull requests 2 Projects Releases 4 Packages 1 Wiki Activity Actions

Modernise docs

Browse source
This commit is contained in:
Andras Schmelczer 2026-10-28 12:50:43 +00:00
commit 37b4c17f59
1 changed files with 8196 additions and 3218 deletions
Download patch file Download diff file Expand all files Collapse all files

296
docs/examples/scibert/deploy.ipynb
View file

@ -2,58 +2,93 @@
"cells": [
{
"cell_type": "markdown",
"id": "cell-0",
"metadata": {},
"source": [
"# Create an inference function\n",
"\n",
"Everything is ready to wrap the previously trained model and deploy it. \n",
"Run `data.ipynb` and `train.ipynb` first in the same local/full mode.\n",
"This notebook loads their local model and tokenizer; no S3 credentials,\n",
"MongoDB server, or model downloads are needed for the default demo.\n",
"The tiny demo model's scores have no scientific interpretation."
]
},
{
"cell_type": "markdown",
"id": "85c2ee1d",
"metadata": {},
"source": [
"## Local setup\n",
"\n",
"First, we need to configure the LargeFileBackend, the TracingDatabase and GreatAI."
"Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n",
"See **Run notebooks locally** in the repository README for the kernel, execution order,\n",
"and full-data options. Generated artifacts stay under `.cache/notebooks/`.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 9,
"id": "3775e11d",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226mThe value of `ENVIRONMENT` contains the \"ENV` prefix but `ENVIRONMENT` is not defined as an environment variable, using the default value defined above (`DEVELOPMENT`)\u001b[0m\n",
"\u001b[38;5;196mConfiguration has been already initialised, overwriting.\n",
"Make sure to call `configure()` before importing your application code.\u001b[0m\n",
"\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
"\u001b[38;5;39mMongoDbDriver has been already configured: skipping initialisation\u001b[0m\n",
"\u001b[38;5;39mLargeFileS3 has been already configured: skipping initialisation\u001b[0m\n",
"\u001b[38;5;39mGreatAI (v0.1.6): configured ✅\u001b[0m\n",
"\u001b[38;5;39m 🔩 tracing_database: MongoDbDriver\u001b[0m\n",
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileS3\u001b[0m\n",
"\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
"\u001b[38;5;39mGreatAI (v0.1.11): configured ✅\u001b[0m\n",
"\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
"\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n",
"\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n",
"\u001b[38;5;39m 🔩 prediction_cache_size: 4096\u001b[0m\n",
"\u001b[38;5;39m 🔩 dashboard_table_size: 100\u001b[0m\n",
"\u001b[38;5;226mYou still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
"\u001b[38;5;226m> Find out more at https://se-ml.github.io/practices\u001b[0m\n"
"\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n",
"\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Notebook artifacts: /Users/schmelczer/Projects/great-ai/.cache/notebooks/scibert\n"
]
}
],
"source": [
"from great_ai.utilities import ConfigFile\n",
"from great_ai.large_file import LargeFileS3\n",
"from great_ai import configure, MongoDbDriver\n",
"import os\n",
"import sys\n",
"from pathlib import Path\n",
"\n",
"configuration = ConfigFile(\"config.ini\")\n",
"# Works when opened from the repository root or any notebook directory.\n",
"_start = Path(globals().get(\"__file__\", Path.cwd())).resolve()\n",
"ROOT = next(\n",
" p\n",
" for p in [_start, *_start.parents]\n",
" if (p / \"pyproject.toml\").is_file() and (p / \"great_ai\").is_dir()\n",
")\n",
"sys.path.insert(0, str(ROOT / \"docs\"))\n",
"from notebook_support import setup_notebook, mag_lines\n",
"\n",
"LargeFileS3.configure_credentials_from_file(configuration)\n",
"MongoDbDriver.configure_credentials_from_file(configuration)\n",
"\n",
"configure(\n",
" dashboard_table_size=100, # traces are small, we can show many\n",
" prediction_cache_size=4096, # predictions are expensive, cache them\n",
")"
"FULL_RUN = os.environ.get(\"GREAT_AI_NOTEBOOK_FULL\") == \"1\"\n",
"WORK_DIR = setup_notebook(ROOT, \"scibert\")\n",
"DATA_DIR = ROOT / \"docs/tutorial/data\""
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "cell-3",
"metadata": {},
"outputs": [],
"source": [
"# The setup cell selects local model storage and a local TinyDB tracing database."
]
},
{
"cell_type": "markdown",
"id": "cell-4",
"metadata": {},
"source": [
"For a pleasant developer experience, we create some typed models that will show up in the automatically generated OpenAPI schema specification and will also provide runtime type validation."
@ -61,7 +96,8 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 11,
"id": "cell-5",
"metadata": {},
"outputs": [],
"source": [
@ -82,6 +118,7 @@
},
{
"cell_type": "markdown",
"id": "cell-6",
"metadata": {},
"source": [
"Even though `@use_model` caches the remote files locally and it also handles deserialising objects, we only use it to store a directory. In this case, it gives back a path, the path to that directory. So, we need to load the files from that folder ourselves. In order to only load it once per process, we create a small model loader helper function.\n",
@ -91,58 +128,44 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 12,
"id": "cell-7",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39mFetching cached versions of scibert-highlights\u001b[0m\n",
"\u001b[38;5;39mLatest version of scibert-highlights is 0 (from versions: 0)\u001b[0m\n",
"\u001b[38;5;39mFile scibert-highlights-0 found in cache\u001b[0m\n"
]
}
],
"source": [
"from functools import lru_cache\n",
"from great_ai import use_model\n",
"from pathlib import Path\n",
"from typing import Tuple\n",
"from transformers import (\n",
" PreTrainedModel,\n",
" PreTrainedTokenizer,\n",
")\n",
"from transformers import (\n",
" AutoConfig,\n",
" AutoModelForSequenceClassification,\n",
" AutoTokenizer,\n",
")\n",
"from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
"\n",
"_tokenizer: PreTrainedTokenizer = None\n",
"_loaded_model: PreTrainedModel = None\n",
"\n",
"@lru_cache(maxsize=1)\n",
"def load_tokenizer_and_model(model_path):\n",
" tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)\n",
" model = AutoModelForSequenceClassification.from_pretrained(\n",
" model_path, local_files_only=True, attn_implementation=\"eager\"\n",
" )\n",
" model.eval()\n",
" return tokenizer, model\n",
"\n",
"\n",
"@use_model(\"scibert-highlights\", version=\"latest\", model_kwarg_name=\"model_path\")\n",
"def get_tokenizer_and_model(\n",
" model_path: Path, original_model: str = \"allenai/scibert_scivocab_uncased\"\n",
") -> Tuple[PreTrainedTokenizer, PreTrainedModel]:\n",
" global _tokenizer, _loaded_model\n",
"\n",
" if _tokenizer is None:\n",
" _tokenizer = AutoTokenizer.from_pretrained(original_model)\n",
"\n",
" if _loaded_model is None:\n",
" config = AutoConfig.from_pretrained(\n",
" model_path, output_hidden_states=True, output_attentions=True\n",
" )\n",
" _loaded_model = AutoModelForSequenceClassification.from_pretrained(\n",
" model_path, config=config\n",
" )\n",
"\n",
" return _tokenizer, _loaded_model"
"def get_tokenizer_and_model(model_path: Path):\n",
" return load_tokenizer_and_model(model_path)"
]
},
{
"cell_type": "markdown",
"id": "cell-8",
"metadata": {},
"source": [
"Finally, implement the inference function."
@ -150,76 +173,40 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 13,
"id": "cell-9",
"metadata": {},
"outputs": [],
"source": [
"from great_ai import GreatAI\n",
"from great_ai.utilities import clean\n",
"\n",
"import re\n",
"import numpy as np\n",
"import torch\n",
"from transformers.modeling_outputs import SequenceClassifierOutput\n",
"\n",
"\n",
"@GreatAI.create\n",
"def find_highlights(sentence: str) -> EvaluatedSentence:\n",
" \"\"\"Get the interestingness prediction of the input sentence using SciBERT.\n",
" \"\"\"Score a sentence and expose the final layer's CLS attention per token.\n",
"\n",
" Run the SciBERT model in inference mode and evaluate the sentence.\n",
" Additionally, provide explanation in the form of the last layer's sum attention\n",
" between `[CLS]` and the other tokens.\n",
" Attention weights are a model diagnostic, not a causal explanation.\n",
" \"\"\"\n",
"\n",
" tokenizer, loaded_model = get_tokenizer_and_model()\n",
" tokenizer, model = get_tokenizer_and_model()\n",
" sentence = clean(sentence, convert_to_ascii=True, remove_brackets=True)\n",
"\n",
" tensors = tokenizer(sentence, return_tensors=\"pt\", truncation=True, max_length=512)\n",
"\n",
" with torch.inference_mode():\n",
" result: SequenceClassifierOutput = loaded_model(**tensors)\n",
" positive_likelihood = torch.nn.Softmax(dim=1)(result.logits)[0][1]\n",
" tokens = tensors[\"input_ids\"][0]\n",
"\n",
" attentions = np.sum(result.attentions[-1].numpy()[0], axis=0)[0][1:-1]\n",
" # Tuple of `torch.FloatTensor` (one for each layer) of shape\n",
" # `(batch_size, num_heads, sequence_length, sequence_length)`.\n",
"\n",
" explanation = []\n",
"\n",
" token_attentions = list(zip(attentions, tokens[1:-1]))\n",
" for token in re.split(r\"([ .,])\", sentence):\n",
" token = token.strip()\n",
" if not token:\n",
" continue\n",
" bert_tokens = tokenizer(\n",
" token, return_tensors=\"pt\", truncation=True, max_length=512\n",
" )[\"input_ids\"][0][\n",
" 1:-1\n",
" ] # truncation=True needed to fix `RuntimeError: Already borrowed`\n",
" weight = 0\n",
" for t1 in bert_tokens:\n",
" if not token_attentions:\n",
" break\n",
" a, t2 = token_attentions.pop(0)\n",
" assert t1 == t2, sentence\n",
" weight += a\n",
" explanation.append(\n",
" Attention(\n",
" token=token if token in \".,\" else \" \" + token, weight=round(weight, 4)\n",
" )\n",
" )\n",
" if not token_attentions:\n",
" break\n",
"\n",
" return EvaluatedSentence(\n",
" score=positive_likelihood, text=sentence, explanation=explanation\n",
" )"
" result = model(**tensors, output_attentions=True)\n",
" score = torch.softmax(result.logits, dim=1)[0, 1].item()\n",
" weights = result.attentions[-1][0, :, 0, :].sum(dim=0).cpu().tolist()\n",
" tokens = tokenizer.convert_ids_to_tokens(tensors[\"input_ids\"][0].tolist())\n",
" explanation = [\n",
" Attention(token=token, weight=round(weight, 4))\n",
" for token, weight in zip(tokens[1:-1], weights[1:-1])\n",
" ]\n",
" return EvaluatedSentence(score=score, text=sentence, explanation=explanation)"
]
},
{
"cell_type": "markdown",
"id": "cell-10",
"metadata": {},
"source": [
"A simple test to see everything works. Note that the models list is filled by the `@use_model` call even though it's not on the main inference function."
@ -227,66 +214,61 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 14,
"id": "cell-11",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"W1028 12:49:11.667000 52910 torch/distributed/elastic/multiprocessing/redirects.py:35] NOTE: Redirects are currently not supported in MacOs.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6ef4b00220304248a94035042cb5fa15",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"(Trace[EvaluatedSentence]({'created': '2022-07-16T18:47:29.581701',\n",
" 'exception': None,\n",
" 'feedback': None,\n",
" 'logged_values': { 'arg:sentence:length': 51,\n",
" 'arg:sentence:value': 'Our solution has outperformed the '\n",
" 'state-of-the-art.'},\n",
" 'models': [{'key': 'scibert-highlights', 'version': 0}],\n",
" 'original_execution_time_ms': 7127.2063,\n",
" 'output': { 'explanation': [ {'token': ' Our', 'weight': 0.3993},\n",
" {'token': ' solution', 'weight': 0.3481},\n",
" {'token': ' has', 'weight': 0.2945},\n",
" {'token': ' outperformed', 'weight': 0.4011},\n",
" {'token': ' the', 'weight': 0.1484},\n",
" {'token': ' state-of-the-art', 'weight': 0.5727},\n",
" {'token': '.', 'weight': 7.775}],\n",
" 'score': 0.9991180300712585,\n",
" 'text': 'Our solution has outperformed the state-of-the-art.'},\n",
" 'tags': ['find_highlights', 'online', 'development'],\n",
" 'trace_id': '56e20e94-79df-4793-ae61-d20820ebe2d3'}),\n",
" Trace[EvaluatedSentence]({'created': '2022-07-16T18:47:37.020275',\n",
" 'exception': None,\n",
" 'feedback': None,\n",
" 'logged_values': { 'arg:sentence:length': 36,\n",
" 'arg:sentence:value': 'Their solution did not perform '\n",
" 'well.'},\n",
" 'models': [{'key': 'scibert-highlights', 'version': 0}],\n",
" 'original_execution_time_ms': 170.7057,\n",
" 'output': { 'explanation': [ {'token': ' Their', 'weight': 1.1475},\n",
" {'token': ' solution', 'weight': 0.8205},\n",
" {'token': ' did', 'weight': 0.3254},\n",
" {'token': ' not', 'weight': 0.2921},\n",
" {'token': ' perform', 'weight': 0.4293},\n",
" {'token': ' well', 'weight': 0.2772},\n",
" {'token': '.', 'weight': 4.4723}],\n",
" 'score': 0.12305451184511185,\n",
" 'text': 'Their solution did not perform well.'},\n",
" 'tags': ['find_highlights', 'online', 'development'],\n",
" 'trace_id': '7fcf8271-1738-4025-8305-d5a1e5100aea'}))"
"Loading weights: 0%| | 0/25 [00:00<?, ?it/s]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"score=0.4977932274341583 text='Our solution has outperformed the state-of-the-art.' explanation=[Attention(weight=0.1332, token='our'), Attention(weight=0.133, token='[UNK]'), Attention(weight=0.1332, token='[UNK]'), Attention(weight=0.1329, token='[UNK]'), Attention(weight=0.134, token='the'), Attention(weight=0.1342, token='[UNK]'), Attention(weight=0.1333, token='[UNK]'), Attention(weight=0.1332, token='[UNK]'), Attention(weight=0.1327, token='[UNK]'), Attention(weight=0.1316, token='the'), Attention(weight=0.1339, token='[UNK]'), Attention(weight=0.1325, token='[UNK]'), Attention(weight=0.1347, token='.')]\n",
"score=0.49779173731803894 text='Their solution did not perform well.' explanation=[Attention(weight=0.2217, token='[UNK]'), Attention(weight=0.2211, token='[UNK]'), Attention(weight=0.2215, token='[UNK]'), Attention(weight=0.221, token='[UNK]'), Attention(weight=0.2239, token='[UNK]'), Attention(weight=0.223, token='[UNK]'), Attention(weight=0.2243, token='.')]\n"
]
}
],
"source": [
"if __name__ == \"__main__\":\n",
" find_highlights(\n",
" \"Our solution has outperformed the state-of-the-art.\"\n",
" ), find_highlights(\"Their solution did not perform well.\")"
" from fastapi.testclient import TestClient\n",
"\n",
" first = find_highlights(\"Our solution has outperformed the state-of-the-art.\")\n",
" second = find_highlights(\"Their solution did not perform well.\")\n",
" assert 0 <= first.output.score <= 1\n",
" assert first.output.explanation\n",
" print(first.output)\n",
" print(second.output)\n",
" with TestClient(find_highlights.app) as client:\n",
" response = client.post(\n",
" \"/predict\", json={\"sentence\": \"Our method improves accuracy.\"}\n",
" )\n",
" response.raise_for_status()\n",
" assert 0 <= response.json()[\"output\"][\"score\"] <= 1"
]
},
{
"cell_type": "markdown",
"id": "cell-12",
"metadata": {},
"source": [
"In this case, the service is built as a docker image, pushed to our image registry and subsequent rolling update is performed in the production cluster.\n",
@ -296,7 +278,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.10.4 ('.env': venv)",
"display_name": ".venv (3.13.5)",
"language": "python",
"name": "python3"
},
@ -310,15 +292,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
}
"version": "3.13.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 5
}