Modernise docs
This commit is contained in:
parent
e446d95855
commit
37b4c17f59
1 changed files with 8196 additions and 3218 deletions
|
|
@ -2,187 +2,180 @@
|
|||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Fine-tune SciBERT\n",
|
||||
"# Fine-tune a BERT sentence classifier\n",
|
||||
"\n",
|
||||
"We are planning to do a simple classification task on scientific text. For that, [SciBERT](https://github.com/allenai/scibert) is an ideal model to fine-tune since it has been pretrained of academic publications.\n",
|
||||
"Install the checkout's `[notebooks]` extra and run `data.ipynb` first.\n",
|
||||
"The default is an offline CPU demonstration: a tiny randomly initialised\n",
|
||||
"BERT with a local vocabulary, trained for one epoch. It verifies the\n",
|
||||
"pipeline and does **not** reproduce SciBERT or its historical scores.\n",
|
||||
"\n",
|
||||
"This notebook was updated so that it can run in [Google Colab](https://colab.research.google.com/).\n",
|
||||
"Set `GREAT_AI_NOTEBOOK_FULL=1` before starting Jupyter to fine-tune\n",
|
||||
"`allenai/scibert_scivocab_uncased` instead. This downloads pretrained weights\n",
|
||||
"and takes substantially more memory and time; a GPU is recommended.\n",
|
||||
"Run the data and deployment notebooks in the same mode."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2346798f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Local setup\n",
|
||||
"\n",
|
||||
"First, we need to install the dependencies."
|
||||
"Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n",
|
||||
"See **Run notebooks locally** in the repository README for the kernel, execution order,\n",
|
||||
"and full-data options. Generated artifacts stay under `.cache/notebooks/`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"executionInfo": {
|
||||
"elapsed": 2529,
|
||||
"status": "ok",
|
||||
"timestamp": 1656596749103,
|
||||
"user_tz": -120
|
||||
},
|
||||
"id": "j7l0nD9hDQbB",
|
||||
"outputId": "88a9931b-396a-4cf1-c659-8a7b098b3cdd"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install transformers datasets great-ai > /dev/null"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Load the training data from S3. (We have uploaded this to S3 in the `data` notebook.)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "824837fa",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
||||
"\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
|
||||
"\u001b[38;5;39mGreatAI (v0.1.11): configured ✅\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Notebook artifacts: /Users/schmelczer/Projects/great-ai/.cache/notebooks/scibert\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"from pathlib import Path\n",
|
||||
"\n",
|
||||
"# Works when opened from the repository root or any notebook directory.\n",
|
||||
"_start = Path(globals().get(\"__file__\", Path.cwd())).resolve()\n",
|
||||
"ROOT = next(\n",
|
||||
" p\n",
|
||||
" for p in [_start, *_start.parents]\n",
|
||||
" if (p / \"pyproject.toml\").is_file() and (p / \"great_ai\").is_dir()\n",
|
||||
")\n",
|
||||
"sys.path.insert(0, str(ROOT / \"docs\"))\n",
|
||||
"from notebook_support import setup_notebook, mag_lines\n",
|
||||
"\n",
|
||||
"FULL_RUN = os.environ.get(\"GREAT_AI_NOTEBOOK_FULL\") == \"1\"\n",
|
||||
"WORK_DIR = setup_notebook(ROOT, \"scibert\")\n",
|
||||
"DATA_DIR = ROOT / \"docs/tutorial/data\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "cell-3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Dependencies are installed once with uv pip install --editable \".[notebooks]\"."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Load the dataset saved locally by `data.ipynb`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "cell-5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;39mFetching cached versions of summary-train-dataset-small\u001b[0m\n",
|
||||
"\u001b[38;5;39mLatest version of summary-train-dataset-small is 0 (from versions: 0)\u001b[0m\n",
|
||||
"\u001b[38;5;39mFile summary-train-dataset-small-0 found in cache\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from great_ai.large_file import LargeFileS3\n",
|
||||
"from great_ai.large_file import LargeFileLocal\n",
|
||||
"import json\n",
|
||||
"\n",
|
||||
"LargeFileS3.configure_credentials_from_file(\"config.ini\")\n",
|
||||
"\n",
|
||||
"with LargeFileS3(\"summary-train-dataset-small\", encoding=\"utf-8\") as f:\n",
|
||||
" # splitting training and test data is done later by `datasets`\n",
|
||||
"with LargeFileLocal(\"summary-train-dataset-small\", encoding=\"utf-8\") as f:\n",
|
||||
" X, y = json.load(f)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Finetune SciBERT, for more info about this step, check out [HuggingFace](https://huggingface.co/docs/transformers/training).\n",
|
||||
"If you're only here for `great-ai`, feel free to skip the next cell."
|
||||
"If you're only here for `great-ai`, feel free to skip the next cell.\n",
|
||||
"\n",
|
||||
"We use an 80/20 training/validation split. Early stopping and checkpoint selection both use validation binary F1 for positive label `1`. There is no independent test split in this example. The split and training seed is fixed for future runs, and the selected checkpoint's evaluation and run metadata are saved under `models/`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {
|
||||
"executionInfo": {
|
||||
"elapsed": 118131,
|
||||
"status": "ok",
|
||||
"timestamp": 1656593941974,
|
||||
"user_tz": -120
|
||||
},
|
||||
"id": "AL3etUQ3LtKN",
|
||||
"outputId": "fe00589f-64dd-4b70-e612-3873b504c00a"
|
||||
},
|
||||
"execution_count": 4,
|
||||
"id": "cell-7",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"W1028 12:48:58.270000 53390 torch/distributed/elastic/multiprocessing/redirects.py:35] NOTE: Redirects are currently not supported in MacOs.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "9c57de70e68a41ecbde5093bd671715a",
|
||||
"model_id": "2c74823d6364447a96609229fbbc5512",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
" 0%| | 0/1 [00:00<?, ?ba/s]"
|
||||
"Map: 0%| | 0/60 [00:00<?, ? examples/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"{'loss': '0.6939', 'grad_norm': '0.5046', 'learning_rate': '1.667e-05', 'epoch': '1'}\n",
|
||||
"{'eval_loss': '0.6939', 'eval_f1': '0', 'eval_runtime': '0.0123', 'eval_samples_per_second': '974.9', 'eval_steps_per_second': '81.24', 'epoch': '1'}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"\n",
|
||||
" <div>\n",
|
||||
" \n",
|
||||
" <progress value='130' max='650' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
|
||||
" [130/650 01:43 < 07:01, 1.23 it/s, Epoch 10/50]\n",
|
||||
" </div>\n",
|
||||
" <table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: left;\">\n",
|
||||
" <th>Epoch</th>\n",
|
||||
" <th>Training Loss</th>\n",
|
||||
" <th>Validation Loss</th>\n",
|
||||
" <th>F1</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <td>1</td>\n",
|
||||
" <td>0.586800</td>\n",
|
||||
" <td>0.512138</td>\n",
|
||||
" <td>0.719101</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>2</td>\n",
|
||||
" <td>0.411600</td>\n",
|
||||
" <td>0.416675</td>\n",
|
||||
" <td>0.849057</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>3</td>\n",
|
||||
" <td>0.245600</td>\n",
|
||||
" <td>0.417070</td>\n",
|
||||
" <td>0.864000</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>4</td>\n",
|
||||
" <td>0.147800</td>\n",
|
||||
" <td>0.575878</td>\n",
|
||||
" <td>0.852459</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>5</td>\n",
|
||||
" <td>0.056800</td>\n",
|
||||
" <td>0.474259</td>\n",
|
||||
" <td>0.896552</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>0.022500</td>\n",
|
||||
" <td>0.754236</td>\n",
|
||||
" <td>0.843137</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>0.001000</td>\n",
|
||||
" <td>0.857636</td>\n",
|
||||
" <td>0.834783</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>8</td>\n",
|
||||
" <td>0.000500</td>\n",
|
||||
" <td>0.920232</td>\n",
|
||||
" <td>0.869565</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>9</td>\n",
|
||||
" <td>0.000300</td>\n",
|
||||
" <td>0.970790</td>\n",
|
||||
" <td>0.877193</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>10</td>\n",
|
||||
" <td>0.000300</td>\n",
|
||||
" <td>0.948689</td>\n",
|
||||
" <td>0.862385</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table><p>"
|
||||
],
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "23492278f66e46f09e9eee0680c21b92",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
"Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
|
|
@ -192,29 +185,31 @@
|
|||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"...\n",
|
||||
"Deleting older checkpoint [models/checkpoint-39] due to args.save_total_limit\n",
|
||||
"***** Running Evaluation *****\n",
|
||||
" Num examples = 100\n",
|
||||
" Batch size = 32\n",
|
||||
"Saving model checkpoint to models/checkpoint-117\n",
|
||||
"Configuration saved in models/checkpoint-117/config.json\n",
|
||||
"Model weights saved in models/checkpoint-117/pytorch_model.bin\n",
|
||||
"Deleting older checkpoint [models/checkpoint-52] due to args.save_total_limit\n",
|
||||
"***** Running Evaluation *****\n",
|
||||
" Num examples = 100\n",
|
||||
" Batch size = 32\n",
|
||||
"Saving model checkpoint to models/checkpoint-130\n",
|
||||
"Configuration saved in models/checkpoint-130/config.json\n",
|
||||
"Model weights saved in models/checkpoint-130/pytorch_model.bin\n",
|
||||
"Deleting older checkpoint [models/checkpoint-78] due to args.save_total_limit\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Training completed. Do not forget to share your model on huggingface.co/models =)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Loading best model from models/checkpoint-65 (score: 0.896551724137931).\n"
|
||||
"[transformers] early stopping required metric_for_best_model, but did not find eval_f1 so early stopping is disabled\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"{'train_runtime': '0.0882', 'train_samples_per_second': '544.2', 'train_steps_per_second': '34.01', 'train_loss': '0.6939', 'epoch': '1'}\n",
|
||||
"{'validation_loss': '0.6939', 'validation_f1': '0', 'validation_runtime': '0.0025', 'validation_samples_per_second': '4804', 'validation_steps_per_second': '400.3', 'epoch': '1'}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'validation_loss': 0.693895161151886,\n",
|
||||
" 'validation_f1': 0.0,\n",
|
||||
" 'validation_runtime': 0.0025,\n",
|
||||
" 'validation_samples_per_second': 4804.013,\n",
|
||||
" 'validation_steps_per_second': 400.334,\n",
|
||||
" 'epoch': 1.0}"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
|
|
@ -225,120 +220,193 @@
|
|||
" Trainer,\n",
|
||||
" TrainingArguments,\n",
|
||||
" EarlyStoppingCallback,\n",
|
||||
" set_seed,\n",
|
||||
")\n",
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"from datasets import Dataset, load_metric\n",
|
||||
"from datasets import Dataset\n",
|
||||
"from sklearn.metrics import f1_score\n",
|
||||
"import torch\n",
|
||||
"\n",
|
||||
"MODEL = \"allenai/scibert_scivocab_uncased\"\n",
|
||||
"BATCH_SIZE = 32\n",
|
||||
"BATCH_SIZE = 8 if FULL_RUN else 16\n",
|
||||
"SPLIT_SEED = 42\n",
|
||||
"\n",
|
||||
"set_seed(SPLIT_SEED)\n",
|
||||
"\n",
|
||||
"if FULL_RUN:\n",
|
||||
" tokenizer = AutoTokenizer.from_pretrained(MODEL)\n",
|
||||
" model = AutoModelForSequenceClassification.from_pretrained(MODEL, num_labels=2)\n",
|
||||
"else:\n",
|
||||
" import re\n",
|
||||
" from transformers import BertConfig, BertForSequenceClassification, BertTokenizer\n",
|
||||
"\n",
|
||||
" torch.set_num_threads(2)\n",
|
||||
" vocabulary = [\"[PAD]\", \"[UNK]\", \"[CLS]\", \"[SEP]\", \"[MASK]\"]\n",
|
||||
" vocabulary += sorted(set(re.findall(r\"[a-z]+|[0-9]+|[^\\w\\s]\", \" \".join(X).lower())))\n",
|
||||
" vocab_path = WORK_DIR / \"vocab.txt\"\n",
|
||||
" vocab_path.write_text(\"\\n\".join(vocabulary) + \"\\n\", encoding=\"utf-8\")\n",
|
||||
" tokenizer = BertTokenizer(vocab=str(vocab_path), do_lower_case=True)\n",
|
||||
" model = BertForSequenceClassification(\n",
|
||||
" BertConfig(\n",
|
||||
" vocab_size=len(tokenizer),\n",
|
||||
" hidden_size=32,\n",
|
||||
" num_hidden_layers=1,\n",
|
||||
" num_attention_heads=2,\n",
|
||||
" intermediate_size=64,\n",
|
||||
" max_position_embeddings=512,\n",
|
||||
" num_labels=2,\n",
|
||||
" )\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(MODEL)\n",
|
||||
"model = AutoModelForSequenceClassification.from_pretrained(MODEL, num_labels=2)\n",
|
||||
"data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def tokenize_function(v):\n",
|
||||
" return tokenizer(v[\"text\"])\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"dataset = (\n",
|
||||
" Dataset.from_dict({\"text\": X, \"label\": y})\n",
|
||||
" .map(lambda v: tokenizer(v[\"text\"], truncation=True), batched=True)\n",
|
||||
" .map(lambda v: tokenizer(v[\"text\"], truncation=True, max_length=128), batched=True)\n",
|
||||
" .remove_columns(\"text\")\n",
|
||||
" .train_test_split(test_size=0.2, shuffle=True) # test is actually validation\n",
|
||||
" .train_test_split(test_size=0.2, shuffle=True, seed=SPLIT_SEED)\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"f1_score = load_metric(\"f1\")\n",
|
||||
"# The split API calls the held-out partition \"test\"; we use it for validation.\n",
|
||||
"dataset[\"validation\"] = dataset.pop(\"test\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def compute_metrics(p):\n",
|
||||
" pred, labels = p\n",
|
||||
" pred = np.argmax(pred, axis=1)\n",
|
||||
" return f1_score.compute(predictions=pred, references=labels)\n",
|
||||
" predictions = (\n",
|
||||
" p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions\n",
|
||||
" )\n",
|
||||
" return {\n",
|
||||
" \"f1\": f1_score(\n",
|
||||
" p.label_ids,\n",
|
||||
" np.argmax(predictions, axis=1),\n",
|
||||
" average=\"binary\",\n",
|
||||
" pos_label=1,\n",
|
||||
" zero_division=0,\n",
|
||||
" )\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"training_args = TrainingArguments(\n",
|
||||
" output_dir=Path(\"models\"),\n",
|
||||
" output_dir=WORK_DIR / \"models\",\n",
|
||||
" per_device_train_batch_size=BATCH_SIZE,\n",
|
||||
" per_device_eval_batch_size=BATCH_SIZE,\n",
|
||||
" save_total_limit=5,\n",
|
||||
" num_train_epochs=50,\n",
|
||||
" num_train_epochs=50 if FULL_RUN else 1,\n",
|
||||
" save_strategy=\"epoch\",\n",
|
||||
" evaluation_strategy=\"epoch\",\n",
|
||||
" eval_strategy=\"epoch\",\n",
|
||||
" logging_strategy=\"epoch\",\n",
|
||||
" weight_decay=0.01,\n",
|
||||
" metric_for_best_model=\"f1\",\n",
|
||||
" load_best_model_at_end=True,\n",
|
||||
" seed=SPLIT_SEED,\n",
|
||||
" use_cpu=not FULL_RUN,\n",
|
||||
" report_to=\"none\",\n",
|
||||
" dataloader_pin_memory=FULL_RUN,\n",
|
||||
" disable_tqdm=not FULL_RUN,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"result = Trainer(\n",
|
||||
"trainer = Trainer(\n",
|
||||
" model=model,\n",
|
||||
" processing_class=tokenizer,\n",
|
||||
" args=training_args,\n",
|
||||
" train_dataset=dataset[\"train\"],\n",
|
||||
" eval_dataset=dataset[\"test\"],\n",
|
||||
" eval_dataset=dataset[\"validation\"],\n",
|
||||
" data_collator=data_collator,\n",
|
||||
" compute_metrics=compute_metrics,\n",
|
||||
" callbacks=[EarlyStoppingCallback(early_stopping_patience=5)],\n",
|
||||
").train()"
|
||||
")\n",
|
||||
"result = trainer.train()\n",
|
||||
"\n",
|
||||
"# This evaluates the selected checkpoint on validation, not an untouched test set.\n",
|
||||
"validation_metrics = trainer.evaluate(metric_key_prefix=\"validation\")\n",
|
||||
"trainer.save_metrics(\"validation\", validation_metrics)\n",
|
||||
"(WORK_DIR / \"models/validation-run.json\").write_text(\n",
|
||||
" json.dumps(\n",
|
||||
" {\n",
|
||||
" \"mode\": \"scibert\" if FULL_RUN else \"tiny-random-bert-demo\",\n",
|
||||
" \"split_seed\": SPLIT_SEED,\n",
|
||||
" \"training_seed\": SPLIT_SEED,\n",
|
||||
" \"validation_fraction\": 0.2,\n",
|
||||
" \"train_samples\": len(dataset[\"train\"]),\n",
|
||||
" \"validation_samples\": len(dataset[\"validation\"]),\n",
|
||||
" \"metric\": \"binary_f1\",\n",
|
||||
" \"positive_label\": 1,\n",
|
||||
" \"best_checkpoint\": trainer.state.best_model_checkpoint,\n",
|
||||
" \"best_validation_f1\": trainer.state.best_metric,\n",
|
||||
" },\n",
|
||||
" indent=2,\n",
|
||||
" )\n",
|
||||
" + \"\\n\",\n",
|
||||
" encoding=\"utf-8\",\n",
|
||||
")\n",
|
||||
"validation_metrics"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The best macro F1-score on the test set is **0.89** which is (not surprisingly) substantially more than the SVM achieved. We have a great model, it's time to deploy it. But first, we have to store it in a secure place."
|
||||
"The [saved historical run](https://great-ai.scoutinscience.com/examples/scibert/validation-results.json) selected a checkpoint with **validation binary F1 of 0.8966** (recorded value: 0.896551724137931). This validation split was also used for early stopping and checkpoint selection, so the number is not an independent test result or macro F1. The historical split seed was not recorded; the seeded code above defines future runs and does not reproduce that split. Its results are saved to `models/validation_results.json` and `models/validation-run.json`.\n",
|
||||
"\n",
|
||||
"To estimate performance after model selection, evaluate once on a separate untouched test split. We make no test-set performance claim here. Next, store the selected model for deployment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 44,
|
||||
"metadata": {
|
||||
"executionInfo": {
|
||||
"elapsed": 25368,
|
||||
"status": "ok",
|
||||
"timestamp": 1656594537509,
|
||||
"user": {
|
||||
"displayName": "Schmelczer András",
|
||||
"userId": "08401926777942666437"
|
||||
},
|
||||
"user_tz": -120
|
||||
},
|
||||
"id": "fyNKltdquZSP",
|
||||
"outputId": "e8c2cbb1-78e1-41a3-b7cf-b0cd573bc45d"
|
||||
},
|
||||
"execution_count": 5,
|
||||
"id": "cell-9",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "634b947ab9ee4f20bef78f780d5441ba",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
"Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Configuration saved in pretrained/config.json\n",
|
||||
"Model weights saved in pretrained/pytorch_model.bin\n"
|
||||
"\u001b[38;5;39mFetching cached versions of scibert-highlights\u001b[0m\n",
|
||||
"\u001b[38;5;39mFetching cached versions of scibert-highlights\u001b[0m\n",
|
||||
"\u001b[38;5;39mFetching cached versions of scibert-highlights\u001b[0m\n",
|
||||
"\u001b[38;5;39mModel scibert-highlights uploaded with version 0\u001b[0m\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" adding: pretrained/ (stored 0%)\n",
|
||||
" adding: pretrained/config.json (deflated 49%)\n",
|
||||
" adding: pretrained/pytorch_model.bin (deflated 7%)\n"
|
||||
]
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'scibert-highlights:0'"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from great_ai import save_model\n",
|
||||
"\n",
|
||||
"# save Torch model to local disk\n",
|
||||
"model.save_pretrained(\"pretrained\")\n",
|
||||
"\n",
|
||||
"# upload model from local disk to S3\n",
|
||||
"# (because the S3 credentials have been already set, `save_model` will use LargeFileS3)\n",
|
||||
"save_model(\"pretrained\", key=\"scibert-highlights\")"
|
||||
"# Trainer has loaded the selected validation checkpoint. Save its tokenizer\n",
|
||||
"# too, so deployment never needs to download a different vocabulary.\n",
|
||||
"trainer.save_model(str(WORK_DIR / \"pretrained\"))\n",
|
||||
"tokenizer.save_pretrained(WORK_DIR / \"pretrained\")\n",
|
||||
"save_model(WORK_DIR / \"pretrained\", key=\"scibert-highlights\", keep_last_n=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-10",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Next: [Part 3](https://great-ai.scoutinscience.com/examples/scibert/deploy)"
|
||||
|
|
@ -347,7 +415,7 @@
|
|||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.10.4 ('.env': venv)",
|
||||
"display_name": ".venv (3.13.5)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
|
|
@ -361,14 +429,9 @@
|
|||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.4"
|
||||
},
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
|
||||
}
|
||||
"version": "3.13.5"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue