great-ai/docs/examples/scibert/deploy.ipynb
Andras Schmelczer cf61b38b45
All checks were successful
Publish documentation to great-ai.schmelczer.dev / publish (push) Successful in 1m10s
Check / Lint, format & type checks (push) Successful in 1m55s
Check / Test on Python 3.12 (push) Successful in 3m8s
Check / Test on Python 3.13 (push) Successful in 2m32s
Check / Notebooks on Python 3.12 (push) Successful in 8m49s
Check / Notebooks on Python 3.13 (push) Successful in 7m53s
Rerender
2026-10-28 22:53:24 +00:00

740 lines
23 KiB
Text
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"cell_type": "markdown",
"id": "cell-0",
"metadata": {},
"source": [
"# Create an inference function\n",
"\n",
"Run `data.ipynb` and `train.ipynb` first in the same local/full mode.\n",
"This notebook loads their local model and tokenizer; no S3 credentials,\n",
"MongoDB server, or model downloads are needed for the default demo.\n",
"The tiny demo model's scores have no scientific interpretation."
]
},
{
"cell_type": "markdown",
"id": "85c2ee1d",
"metadata": {},
"source": [
"## Local setup\n",
"\n",
"Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n",
"See **Run notebooks locally** in the repository README for the kernel, execution order,\n",
"and full-data options. Run related notebooks from the same working directory\n",
"to share the default `.cache/` model store and `tracing_database.json` database.\n"
]
},
{
"cell_type": "markdown",
"id": "cell-4",
"metadata": {},
"source": [
"For a pleasant developer experience, we create some typed models that will show up in the automatically generated OpenAPI schema specification and will also provide runtime type validation."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "cell-5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-28T22:48:09.038647Z",
"iopub.status.busy": "2026-10-28T22:48:09.038493Z",
"iopub.status.idle": "2026-10-28T22:48:09.083813Z",
"shell.execute_reply": "2026-10-28T22:48:09.083283Z"
}
},
"outputs": [],
"source": [
"from typing import List\n",
"from pydantic import BaseModel\n",
"\n",
"\n",
"class Attention(BaseModel):\n",
" weight: float\n",
" token: str\n",
"\n",
"\n",
"class EvaluatedSentence(BaseModel):\n",
" score: float\n",
" text: str\n",
" explanation: List[Attention]"
]
},
{
"cell_type": "markdown",
"id": "cell-6",
"metadata": {},
"source": [
"Even though `@use_model` caches the remote files locally and it also handles deserialising objects, we only use it to store a directory. In this case, it gives back a path, the path to that directory. So, we need to load the files from that folder ourselves. In order to only load it once per process, we create a small model loader helper function.\n",
"\n",
"> This is usually not needed, however, when we can outsmart `dill` so for optimisation purposes, we do it."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "cell-7",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-28T22:48:09.084713Z",
"iopub.status.busy": "2026-10-28T22:48:09.084644Z",
"iopub.status.idle": "2026-10-28T22:48:11.256390Z",
"shell.execute_reply": "2026-10-28T22:48:11.256030Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226mCannot find credentials files, defaulting to using ParallelTinyDbDriver\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226mCannot find credentials files, defaulting to using LargeFileLocal\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39mGreatAI (v0.1.11): configured ✅\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226mYou still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226m> Find out more at https://se-ml.github.io/practices\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39mFetching cached versions of scibert-highlights\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39mLatest version of scibert-highlights is 0 (from versions: 0)\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39mFile scibert-highlights-0 found in cache\u001b[0m\n"
]
}
],
"source": [
"from functools import lru_cache\n",
"from pathlib import Path\n",
"from great_ai import use_model\n",
"from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
"\n",
"\n",
"@lru_cache(maxsize=1)\n",
"def load_tokenizer_and_model(model_path):\n",
" tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)\n",
" model = AutoModelForSequenceClassification.from_pretrained(\n",
" model_path, local_files_only=True, attn_implementation=\"eager\"\n",
" )\n",
" model.eval()\n",
" return tokenizer, model\n",
"\n",
"\n",
"@use_model(\"scibert-highlights\", version=\"latest\", model_kwarg_name=\"model_path\")\n",
"def get_tokenizer_and_model(model_path: Path):\n",
" return load_tokenizer_and_model(model_path)"
]
},
{
"cell_type": "markdown",
"id": "cell-8",
"metadata": {},
"source": [
"Finally, implement the inference function."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "cell-9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-28T22:48:11.257418Z",
"iopub.status.busy": "2026-10-28T22:48:11.257269Z",
"iopub.status.idle": "2026-10-28T22:48:11.289789Z",
"shell.execute_reply": "2026-10-28T22:48:11.289400Z"
}
},
"outputs": [],
"source": [
"from great_ai import GreatAI\n",
"from great_ai.utilities import clean\n",
"import torch\n",
"\n",
"\n",
"@GreatAI.create\n",
"def find_highlights(sentence: str) -> EvaluatedSentence:\n",
" \"\"\"Score a sentence and expose the final layer's CLS attention per token.\n",
"\n",
" Attention weights are a model diagnostic, not a causal explanation.\n",
" \"\"\"\n",
" tokenizer, model = get_tokenizer_and_model()\n",
" sentence = clean(sentence, convert_to_ascii=True, remove_brackets=True)\n",
" tensors = tokenizer(sentence, return_tensors=\"pt\", truncation=True, max_length=512)\n",
" with torch.inference_mode():\n",
" result = model(**tensors, output_attentions=True)\n",
" score = torch.softmax(result.logits, dim=1)[0, 1].item()\n",
" weights = result.attentions[-1][0, :, 0, :].sum(dim=0).cpu().tolist()\n",
" tokens = tokenizer.convert_ids_to_tokens(tensors[\"input_ids\"][0].tolist())\n",
" explanation = [\n",
" Attention(token=token, weight=round(weight, 4))\n",
" for token, weight in zip(tokens[1:-1], weights[1:-1])\n",
" ]\n",
" return EvaluatedSentence(score=score, text=sentence, explanation=explanation)"
]
},
{
"cell_type": "markdown",
"id": "cell-10",
"metadata": {},
"source": [
"A simple test to see everything works. Note that the models list is filled by the `@use_model` call even though it's not on the main inference function."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "cell-11",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-28T22:48:11.290551Z",
"iopub.status.busy": "2026-10-28T22:48:11.290495Z",
"iopub.status.idle": "2026-10-28T22:48:11.803477Z",
"shell.execute_reply": "2026-10-28T22:48:11.803126Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"W1028 22:48:11.714000 21400 .venv/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/redirects.py:35] NOTE: Redirects are currently not supported in MacOs.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "494840f6abbc4b7aaad2ce762d1e32e8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Loading weights: 0%| | 0/25 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"score=0.4977932274341583 text='Our solution has outperformed the state-of-the-art.' explanation=[Attention(weight=0.1332, token='our'), Attention(weight=0.133, token='[UNK]'), Attention(weight=0.1332, token='[UNK]'), Attention(weight=0.1329, token='[UNK]'), Attention(weight=0.134, token='the'), Attention(weight=0.1342, token='[UNK]'), Attention(weight=0.1333, token='[UNK]'), Attention(weight=0.1332, token='[UNK]'), Attention(weight=0.1327, token='[UNK]'), Attention(weight=0.1316, token='the'), Attention(weight=0.1339, token='[UNK]'), Attention(weight=0.1325, token='[UNK]'), Attention(weight=0.1347, token='.')]\n",
"score=0.49779173731803894 text='Their solution did not perform well.' explanation=[Attention(weight=0.2217, token='[UNK]'), Attention(weight=0.2211, token='[UNK]'), Attention(weight=0.2215, token='[UNK]'), Attention(weight=0.221, token='[UNK]'), Attention(weight=0.2239, token='[UNK]'), Attention(weight=0.223, token='[UNK]'), Attention(weight=0.2243, token='.')]\n"
]
}
],
"source": [
"if __name__ == \"__main__\":\n",
" from fastapi.testclient import TestClient\n",
"\n",
" first = find_highlights(\"Our solution has outperformed the state-of-the-art.\")\n",
" second = find_highlights(\"Their solution did not perform well.\")\n",
" assert 0 <= first.output.score <= 1\n",
" assert first.output.explanation\n",
" print(first.output)\n",
" print(second.output)\n",
" with TestClient(find_highlights.app) as client:\n",
" response = client.post(\n",
" \"/predict\", json={\"sentence\": \"Our method improves accuracy.\"}\n",
" )\n",
" response.raise_for_status()\n",
" assert 0 <= response.json()[\"output\"][\"score\"] <= 1"
]
},
{
"cell_type": "markdown",
"id": "cell-12",
"metadata": {},
"source": [
"In this case, the service is built as a docker image, pushed to our image registry and subsequent rolling update is performed in the production cluster.\n",
"To check out the Dockerimage, go to [the additional files page](https://great-ai.schmelczer.dev/examples/scibert/additional-files)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv (3.13.5)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.5"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {
"3db5ee6325b74e25903d2d083b4a79ac": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HTMLStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HTMLStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "StyleView",
"background": null,
"description_width": "",
"font_size": null,
"text_color": null
}
},
"494840f6abbc4b7aaad2ce762d1e32e8": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "2.0.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_a60ffd19ff674f3087de262cca2823ea",
"IPY_MODEL_5dc69cd57cac4608a2b0d4b4a9a2150c",
"IPY_MODEL_4cd08817c5684abeb1146d2c8e2a4f95"
],
"layout": "IPY_MODEL_d6a5b48086a84a2291dcfc6991c24af1",
"tabbable": null,
"tooltip": null
}
},
"4cd08817c5684abeb1146d2c8e2a4f95": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "2.0.0",
"_view_name": "HTMLView",
"description": "",
"description_allow_html": false,
"layout": "IPY_MODEL_bd3bce93f04348f29314e8437dae3c49",
"placeholder": "​",
"style": "IPY_MODEL_94030007cfed4d96b4bb93184555c8d3",
"tabbable": null,
"tooltip": null,
"value": " 25/25 [00:00&lt;00:00, 7301.55it/s]"
}
},
"5dc69cd57cac4608a2b0d4b4a9a2150c": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "2.0.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_allow_html": false,
"layout": "IPY_MODEL_6c5077713e5e47dea8ff3f067b042f73",
"max": 25.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_c232499e62a342ac8bc98cf0a598d75e",
"tabbable": null,
"tooltip": null,
"value": 25.0
}
},
"6c5077713e5e47dea8ff3f067b042f73": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"94030007cfed4d96b4bb93184555c8d3": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HTMLStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HTMLStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "StyleView",
"background": null,
"description_width": "",
"font_size": null,
"text_color": null
}
},
"a60ffd19ff674f3087de262cca2823ea": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "2.0.0",
"_view_name": "HTMLView",
"description": "",
"description_allow_html": false,
"layout": "IPY_MODEL_b8cda8463d0d41269292fab71cd1333f",
"placeholder": "​",
"style": "IPY_MODEL_3db5ee6325b74e25903d2d083b4a79ac",
"tabbable": null,
"tooltip": null,
"value": "Loading weights: 100%"
}
},
"b8cda8463d0d41269292fab71cd1333f": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"bd3bce93f04348f29314e8437dae3c49": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"c232499e62a342ac8bc98cf0a598d75e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"d6a5b48086a84a2291dcfc6991c24af1": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
}
},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}