From 6bbfeadf9f14366dcf0a1c1f25b64fd078872815 Mon Sep 17 00:00:00 2001 From: Andras Schmelczer Date: Tue, 27 Oct 2026 21:59:13 +0000 Subject: [PATCH] Describe the tutorial's historical SciBERT comparison --- docs/tutorial/index.md | 6 ++++-- docs/tutorial/train.ipynb | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/tutorial/index.md b/docs/tutorial/index.md index c300b19..b75e95b 100644 --- a/docs/tutorial/index.md +++ b/docs/tutorial/index.md @@ -1,4 +1,4 @@ -# Train and deploy a SOTA model +# Train and deploy a sentence classifier Let's see `great-ai` in action by going over the lifecycle of a simple service. @@ -10,7 +10,9 @@ Let's see `great-ai` in action by going over the lifecycle of a simple service. ## Overview -You will train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the [SciBERT paper](https://arxiv.org/abs/1903.10676){ target=_blank } in which SciBERT [achieved an F1-score of 0.6571](https://paperswithcode.com/sota/sentence-classification-on-paper-field){ target=_blank }. We are going to outperform it using a trivial text classification model: a [Linear SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html){ target=_blank }. +You will train a field of study (domain) classifier for scientific sentences using a [Linear SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html){ target=_blank }. The training notebook's recorded run achieves a test macro F1-score of about 0.69 on the MAG sentence-classification dataset, compared with the historical SciBERT baseline of 0.6571 associated with the [2019 SciBERT work](https://arxiv.org/abs/1903.10676){ target=_blank }. + +This is a comparison with one historical baseline. A current state-of-the-art claim would require an up-to-date benchmark against comparable systems, using the same data splits, preprocessing, and evaluation metric. We use the same synthetic dataset derived from the [Microsoft Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/){ target=_blank }. The dataset is [available here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag){ target=_blank }. diff --git a/docs/tutorial/train.ipynb b/docs/tutorial/train.ipynb index 61b8e1e..26c0b21 100644 --- a/docs/tutorial/train.ipynb +++ b/docs/tutorial/train.ipynb @@ -234,7 +234,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Great work, we can be rightfully satisfied with our model. Seeing the results, we achieved an F1-score of 0.69 which is about **5% better than SciBERT's** 0.6571!\n", + "The recorded run achieves a test macro F1-score of about **0.69** on the MAG sentence-classification dataset, above the historical SciBERT baseline of **0.6571** associated with the [2019 SciBERT work](https://arxiv.org/abs/1903.10676). This compares a linear SVM with one historical baseline; it does not establish current state-of-the-art performance. A broader claim would require a current benchmark with comparable data splits, preprocessing, and metrics.\n", "\n", "You might wonder that *\"this is great, but besides some utility functions (`clean`, `simple_parallel_map`, ...) what more value does GreatAI add?\"*. This would be a valid argument because the scope of GreatAI actually only starts here.\n", "\n",