{ "cells": [ { "cell_type": "markdown", "id": "cell-0", "metadata": {}, "source": [ "# Hardening and deployment\n", "\n", "> Part 3: Create production inference function\n", "\n", "![position of this step in the lifecycle](https://great-ai.schmelczer.dev/media/scope-deploy.svg)\n", "> The blue boxes show the steps implemented in this notebook.\n", "\n", "In [Part 2](https://great-ai.schmelczer.dev/examples/simple/train), we trained our AI model. Now, it's time to create **G**eneral **R**obust **E**nd-to-end **A**utomated **T**rustworthy deployment.\n", "\n", "## Create the inference function\n", "\n", "Next to the prediction, we also return the top-n most influential words based on their weights." ] }, { "cell_type": "markdown", "id": "e8e78891", "metadata": {}, "source": [ "## Local setup\n", "\n", "Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n", "See **Run notebooks locally** in the repository README for the kernel, execution order,\n", "and full-data options. Run related notebooks from the same working directory\n", "to share the default `.cache/` model store and `tracing_database.json` database.\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "cell-3", "metadata": { "execution": { "iopub.execute_input": "2026-10-28T22:47:38.748279Z", "iopub.status.busy": "2026-10-28T22:47:38.748156Z", "iopub.status.idle": "2026-10-28T22:47:40.219787Z", "shell.execute_reply": "2026-10-28T22:47:40.219011Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226mCannot find credentials files, defaulting to using ParallelTinyDbDriver\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226mCannot find credentials files, defaulting to using LargeFileLocal\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39mGreatAI (v0.1.11): configured ✅\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226mYou still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226m> Find out more at https://se-ml.github.io/practices\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39mFetching cached versions of small-domain-prediction\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39mLatest version of small-domain-prediction is 0 (from versions: 0)\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39mFile small-domain-prediction-0 found in cache\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39mFile small-domain-prediction-0 found in cache\u001b[0m\n" ] } ], "source": [ "import re\n", "import numpy as np\n", "from sklearn.pipeline import Pipeline\n", "from great_ai.utilities import clean\n", "from great_ai import (\n", " MultiLabelClassificationOutput,\n", " ClassificationOutput,\n", " GreatAI,\n", " use_model,\n", " parameter,\n", ")\n", "\n", "\n", "@GreatAI.create\n", "@use_model(\"small-domain-prediction\", version=\"latest\")\n", "@parameter(\"target_confidence\", validate=lambda c: 0 <= c <= 100)\n", "def predict_domain(\n", " text: str, model: Pipeline, target_confidence: int = 50\n", ") -> MultiLabelClassificationOutput:\n", " \"\"\"\n", " Predict the scientific domain of the input text.\n", " Return labels until their sum likelihood is larger than `target_confidence`.\n", " \"\"\"\n", "\n", " preprocessed = re.sub(r\"[^a-zA-Z\\s]\", \"\", clean(text, convert_to_ascii=True))\n", " features = model.named_steps[\"vectorizer\"].transform([preprocessed])\n", " prediction = model.named_steps[\"classifier\"].predict_proba(features)[0]\n", "\n", " best_classes = sorted(enumerate(prediction), key=lambda v: v[1], reverse=True)\n", "\n", " results = MultiLabelClassificationOutput()\n", " for class_index, probability in best_classes:\n", " results.labels.append(\n", " get_label(\n", " model=model,\n", " features=features,\n", " class_index=class_index,\n", " probability=probability,\n", " )\n", " )\n", "\n", " if sum(r.confidence for r in results.labels) >= target_confidence:\n", " break\n", "\n", " return results\n", "\n", "\n", "def get_label(\n", " model: Pipeline, features: np.ndarray, class_index: int, probability: float\n", ") -> ClassificationOutput:\n", " return ClassificationOutput(\n", " label=model.named_steps[\"classifier\"].classes_[class_index],\n", " confidence=round(probability * 100),\n", " explanation=[\n", " word\n", " for _, word in sorted(\n", " (\n", " (weight, word)\n", " for weight, word, count in zip(\n", " model.named_steps[\"classifier\"].feature_log_prob_[class_index],\n", " model.named_steps[\"vectorizer\"].get_feature_names_out(),\n", " features.toarray()[0],\n", " )\n", " if count > 0\n", " ),\n", " reverse=True,\n", " )\n", " ][:5],\n", " )" ] }, { "cell_type": "markdown", "id": "cell-4", "metadata": {}, "source": [ "## Check accuracy on the test split\n", "\n", "Anything under `if __name__ == \"__main__\":` will not be run when the script is executed by the `great-ai` CLI app. This, combined with `query_ground_truth` and the `/traces/{trace_id}/feedback` endpoint are ideal for creating a continuous-integration job for checking the quality of the model before deployment.\n", "\n", "We keep each paper's full set of accepted labels fixed. **Top-1 hit rate** is the fraction of papers whose highest-ranked returned label belongs to that set (an empty prediction is a miss). It is an application-specific hit criterion, not ordinary multiclass accuracy.\n", "\n", "We also report per-class and micro, macro, weighted and sample-averaged multilabel precision, recall and F1 for **all returned labels**, using the default `target_confidence=50`. The label vocabulary comes from the train/test annotations, independently of predictions. Each class's support is the number of test papers annotated with that class. A lower confidence target can return fewer labels and trade recall for precision. `zero_division=0` assigns zero to undefined metrics.\n", "\n", "The earlier report and confusion matrix used prediction-dependent targets. Those outputs have been removed; rerun this cell to generate the corrected report for your model.\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "cell-5", "metadata": { "execution": { "iopub.execute_input": "2026-10-28T22:47:40.220877Z", "iopub.status.busy": "2026-10-28T22:47:40.220772Z", "iopub.status.idle": "2026-10-28T22:47:41.416823Z", "shell.execute_reply": "2026-10-28T22:47:41.416155Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\r", " 0%| | 0/84 [00:00