{ "cells": [ { "cell_type": "markdown", "id": "cell-0", "metadata": {}, "source": [ "# Explore data and feasibility of approach\n", "\n", "The original annotations are proprietary. This notebook defaults to 60\n", "clearly synthetic sentences with two simulated annotators so the entire\n", "training/deployment workflow can run locally without credentials.\n", "These data and their scores demonstrate execution only.\n", "\n", "To use your own JSON annotation files (each maps sentences to binary\n", "labels), set `GREAT_AI_ANNOTATIONS_DIR` to their directory before starting\n", "Jupyter. No private data or cloud configuration is included in this repo." ] }, { "cell_type": "markdown", "id": "6da6bb8a", "metadata": {}, "source": [ "## Local setup\n", "\n", "Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n", "See **Run notebooks locally** in the repository README for the kernel, execution order,\n", "and full-data options. Run related notebooks from the same working directory\n", "to share the default `.cache/` model store and `tracing_database.json` database.\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "cell-3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using synthetic demo annotations; scores are not research results.\n" ] } ], "source": [ "import json\n", "import os\n", "from pathlib import Path\n", "\n", "from docs.notebook_support import demo_annotations\n", "\n", "annotation_dir = os.environ.get(\"GREAT_AI_ANNOTATIONS_DIR\")\n", "if annotation_dir:\n", " annotations = [\n", " json.loads(p.read_text(encoding=\"utf-8\"))\n", " for p in sorted(Path(annotation_dir).expanduser().glob(\"*.json\"))\n", " ]\n", " if not annotations:\n", " raise ValueError(f\"No annotation JSON files found in {annotation_dir}\")\n", "else:\n", " annotations = demo_annotations()\n", " print(\"Using synthetic demo annotations; scores are not research results.\")\n", "\n", "evaluations = {\n", " sentence: [\n", " annotation[sentence] for annotation in annotations if sentence in annotation\n", " ]\n", " for sentence in sorted(\n", " {sentence for annotation in annotations for sentence in annotation}\n", " )\n", "}\n", "X = list(evaluations)\n", "y = [int(sum(e) > 0) for e in evaluations.values()]" ] }, { "cell_type": "markdown", "id": "cell-4", "metadata": {}, "source": [ "Save the processed dataset using local versioned storage for the training notebook." ] }, { "cell_type": "code", "execution_count": 3, "id": "cell-5", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39mFetching cached versions of summary-train-dataset-small\u001b[0m\n", "\u001b[38;5;39mFetching cached versions of summary-train-dataset-small\u001b[0m\n", "\u001b[38;5;39mFetching cached versions of summary-train-dataset-small\u001b[0m\n" ] } ], "source": [ "from great_ai.large_file import LargeFileLocal\n", "\n", "with LargeFileLocal(\"summary-train-dataset-small\", \"w\", encoding=\"utf-8\") as f:\n", " json.dump((X, y), f)" ] }, { "cell_type": "markdown", "id": "cell-6", "metadata": {}, "source": [ "Filter out sentences which don't have enough annotations." ] }, { "cell_type": "code", "execution_count": 4, "id": "cell-7", "metadata": {}, "outputs": [], "source": [ "y1 = [e[0] for e in evaluations.values() if len(e) == 2]\n", "y2 = [e[1] for e in evaluations.values() if len(e) == 2]" ] }, { "cell_type": "markdown", "id": "cell-8", "metadata": {}, "source": [ "Calculate [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa).\n", "\n", "It's a bit low but the task itself is pretty subjective so it's not all that surprising." ] }, { "cell_type": "code", "execution_count": 5, "id": "cell-9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.8\n" ] } ], "source": [ "import sklearn.metrics\n", "\n", "if y1:\n", " print(sklearn.metrics.cohen_kappa_score(y1, y2))\n", "else:\n", " print(\"No sentences have exactly two annotations; kappa is unavailable.\")" ] }, { "cell_type": "markdown", "id": "cell-10", "metadata": {}, "source": [ "Can we train anything on this data?\n", "\n", "Let's try with a trivial SVM." ] }, { "cell_type": "code", "execution_count": 6, "id": "cell-11", "metadata": {}, "outputs": [], "source": [ "X = [s for s in evaluations.keys()]\n", "y = [int(sum(e) > 0) for e in evaluations.values()]" ] }, { "cell_type": "code", "execution_count": 7, "id": "cell-12", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0.41666667, 0.25 , 0.16666667, 0.41666667, 0.5 ])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.model_selection import train_test_split\n", "from sklearn.svm import LinearSVC\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.model_selection import cross_val_score\n", "\n", "model = Pipeline(\n", " steps=[\n", " (\"vectorizer\", TfidfVectorizer(sublinear_tf=True, min_df=3, max_df=0.3)),\n", " (\"classifier\", LinearSVC(random_state=42)),\n", " ]\n", ") # baseline model\n", "\n", "cross_val_score(model, X, y, cv=5)" ] }, { "cell_type": "markdown", "id": "cell-13", "metadata": {}, "source": [ "The cross-validation shows promising accuracies. But accuracy isn't everything, therefore, we should investigate the accuracy metrics." ] }, { "cell_type": "code", "execution_count": 8, "id": "cell-14", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('vectorizer',\n",
" TfidfVectorizer(max_df=0.3, min_df=3, sublinear_tf=True)),\n",
" ('classifier', LinearSVC(random_state=42))])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. | Name | \n", "Type | \n", "Value | \n", "
|---|---|---|
| \n",
" \n",
" classes_\n",
" \n",
" classes_: ndarray of shape (n_classes,) The classes labels. Only exist if the last step of the pipeline is a classifier.\n", " \n", " | \n",
" ndarray[int64](2,) | \n", "[0,1] | \n", "\n", "\n", "
| Name | \n", "Type | \n", "Value | \n", "
|---|---|---|
| \n",
" \n",
" fixed_vocabulary_\n",
" \n",
" fixed_vocabulary_: bool True if a fixed vocabulary of term to indices mapping is provided by the user.\n", " \n", " | \n",
" bool | \n", "False | \n", "\n", "\n", "
| \n",
" \n",
" idf_\n",
" \n",
" idf_: array of shape (n_features,) The inverse document frequency (IDF) vector; only defined if ``use_idf`` is True.\n", " \n", " | \n",
" ndarray[float64](6,) | \n", "[2.49,2.81,2.69,2.81,2.69,2.69] | \n", "\n", "\n", "
| \n",
" \n",
" vocabulary_\n",
" \n",
" vocabulary_: dict A mapping of terms to feature indices.\n", " \n", " | \n",
" dict | \n", "{'cancer': 0, 'genome': 1, 'neural': 2, 'protein': 3, ...} | \n", "\n", "\n", "
| cancer | \n", "
| genome | \n", "
| neural | \n", "
| protein | \n", "
| quantum | \n", "
| solar | \n", "