Describe the tutorial's historical SciBERT comparison
This commit is contained in:
parent
d2616cf997
commit
6bbfeadf9f
2 changed files with 5 additions and 3 deletions
|
|
@ -1,4 +1,4 @@
|
||||||
# Train and deploy a SOTA model
|
# Train and deploy a sentence classifier
|
||||||
|
|
||||||
Let's see `great-ai` in action by going over the lifecycle of a simple service.
|
Let's see `great-ai` in action by going over the lifecycle of a simple service.
|
||||||
|
|
||||||
|
|
@ -10,7 +10,9 @@ Let's see `great-ai` in action by going over the lifecycle of a simple service.
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
You will train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the [SciBERT paper](https://arxiv.org/abs/1903.10676){ target=_blank } in which SciBERT [achieved an F1-score of 0.6571](https://paperswithcode.com/sota/sentence-classification-on-paper-field){ target=_blank }. We are going to outperform it using a trivial text classification model: a [Linear SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html){ target=_blank }.
|
You will train a field of study (domain) classifier for scientific sentences using a [Linear SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html){ target=_blank }. The training notebook's recorded run achieves a test macro F1-score of about 0.69 on the MAG sentence-classification dataset, compared with the historical SciBERT baseline of 0.6571 associated with the [2019 SciBERT work](https://arxiv.org/abs/1903.10676){ target=_blank }.
|
||||||
|
|
||||||
|
This is a comparison with one historical baseline. A current state-of-the-art claim would require an up-to-date benchmark against comparable systems, using the same data splits, preprocessing, and evaluation metric.
|
||||||
|
|
||||||
We use the same synthetic dataset derived from the [Microsoft Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/){ target=_blank }. The dataset is [available here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag){ target=_blank }.
|
We use the same synthetic dataset derived from the [Microsoft Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/){ target=_blank }. The dataset is [available here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag){ target=_blank }.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -234,7 +234,7 @@
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Great work, we can be rightfully satisfied with our model. Seeing the results, we achieved an F1-score of 0.69 which is about **5% better than SciBERT's** 0.6571!\n",
|
"The recorded run achieves a test macro F1-score of about **0.69** on the MAG sentence-classification dataset, above the historical SciBERT baseline of **0.6571** associated with the [2019 SciBERT work](https://arxiv.org/abs/1903.10676). This compares a linear SVM with one historical baseline; it does not establish current state-of-the-art performance. A broader claim would require a current benchmark with comparable data splits, preprocessing, and metrics.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You might wonder that *\"this is great, but besides some utility functions (`clean`, `simple_parallel_map`, ...) what more value does GreatAI add?\"*. This would be a valid argument because the scope of GreatAI actually only starts here.\n",
|
"You might wonder that *\"this is great, but besides some utility functions (`clean`, `simple_parallel_map`, ...) what more value does GreatAI add?\"*. This would be a valid argument because the scope of GreatAI actually only starts here.\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue