Deployed b1a66cb with MkDocs version: 1.3.0
This commit is contained in:
parent
0683061983
commit
7305d2ead3
38 changed files with 3499 additions and 2753 deletions
33
404.html
33
404.html
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -346,6 +359,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="/how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="/how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -596,7 +623,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="/examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -610,7 +637,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="/examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -624,7 +651,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="/examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -601,7 +628,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -615,7 +642,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -629,7 +656,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -694,6 +721,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="train.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -4,17 +4,20 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n",
|
||||
"> Part 1: obtain and clean data\n",
|
||||
"# Simple example: data engineering\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> The blue boxes show the steps implemented in this notebook.\n",
|
||||
"Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus) by taking full advantage of `great-ai`. Subsequently, we create a production-ready deployment.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.\n",
|
||||
"\n",
|
||||
"Since the true scope of `great-ai` is the phase between proof-of-concept code and production-ready service, it is predominantly used in the [deployment notebook](/examples/simple/deploy). Feel free to skip there, or continue reading if you'd like to see the full picture.\n",
|
||||
"\n",
|
||||
"### Extract\n",
|
||||
"\n",
|
||||
"This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n",
|
||||
"\n",
|
||||
"In this case, we download the semantic scholar dataset from a public S3 bucket."
|
||||
"In this example, we download the semantic scholar dataset from a public S3 bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -23,7 +26,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MAX_CHUNK_COUNT = 1"
|
||||
"MAX_CHUNK_COUNT = 4"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -34,7 +37,7 @@
|
|||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'Processing 1 out of the 6002 available chunks'"
|
||||
"'Processing 4 out of the 6002 available chunks'"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
|
|
@ -69,7 +72,7 @@
|
|||
"\n",
|
||||
"- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n",
|
||||
"- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n",
|
||||
"- We will speed up processing using `great_ai.utilities.parallel_map`."
|
||||
"- We will speed up processing using `great_ai.utilities.simple_parallel_map`."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -77,56 +80,11 @@
|
|||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Spacy model en_core_web_sm not found locally, downloading...\n",
|
||||
"Collecting en-core-web-sm==3.3.0\n",
|
||||
" Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl (12.8 MB)\n",
|
||||
" ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 3.6 MB/s eta 0:00:00\n",
|
||||
"Requirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in ./.env/lib/python3.10/site-packages (from en-core-web-sm==3.3.0) (3.3.1)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.64.0)\n",
|
||||
"Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.9.1)\n",
|
||||
"Requirement already satisfied: srsly<3.0.0,>=2.4.3 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.4.3)\n",
|
||||
"Requirement already satisfied: typer<0.5.0,>=0.3.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.4.1)\n",
|
||||
"Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (59.6.0)\n",
|
||||
"Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.7)\n",
|
||||
"Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.2)\n",
|
||||
"Requirement already satisfied: blis<0.8.0,>=0.4.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.7.8)\n",
|
||||
"Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.7)\n",
|
||||
"Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)\n",
|
||||
"Requirement already satisfied: requests<3.0.0,>=2.13.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.28.0)\n",
|
||||
"Requirement already satisfied: numpy>=1.15.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.23.0)\n",
|
||||
"Requirement already satisfied: cymem<2.1.0,>=2.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.6)\n",
|
||||
"Requirement already satisfied: pathy>=0.3.5 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.6.1)\n",
|
||||
"Requirement already satisfied: jinja2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.1.2)\n",
|
||||
"Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3.0)\n",
|
||||
"Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.8.2)\n",
|
||||
"Requirement already satisfied: preshed<3.1.0,>=3.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.6)\n",
|
||||
"Requirement already satisfied: thinc<8.1.0,>=8.0.14 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.0.17)\n",
|
||||
"Requirement already satisfied: packaging>=20.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (21.3)\n",
|
||||
"Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.env/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)\n",
|
||||
"Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in ./.env/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (5.2.1)\n",
|
||||
"Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.env/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.2.0)\n",
|
||||
"Requirement already satisfied: charset-normalizer~=2.0.0 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.12)\n",
|
||||
"Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2022.6.15)\n",
|
||||
"Requirement already satisfied: idna<4,>=2.5 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3)\n",
|
||||
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.26.9)\n",
|
||||
"Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.env/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.1.3)\n",
|
||||
"Requirement already satisfied: MarkupSafe>=2.0 in ./.env/lib/python3.10/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.1.1)\n",
|
||||
"Installing collected packages: en-core-web-sm\n",
|
||||
"Successfully installed en-core-web-sm-3.3.0\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226m2022-06-25 14:21:57,983 | WARNING | Limiting concurrency to 1 because there are only 1 chunks\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:21:57,984 | INFO | Starting parallel map (concurrency: 1, chunk size: 1)\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:21:57,984 | WARNING | Running in series, there is no reason for parallelism\u001b[0m\n",
|
||||
"100%|██████████| 1/1 [03:26<00:00, 206.86s/it]\n"
|
||||
"100%|██████████| 4/4 [04:42<00:00, 70.62s/it] \n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -134,7 +92,13 @@
|
|||
"from typing import List, Tuple\n",
|
||||
"import json\n",
|
||||
"import gzip\n",
|
||||
"from great_ai import parallel_map, clean, is_english, predict_language\n",
|
||||
"from great_ai.utilities import (\n",
|
||||
" simple_parallel_map,\n",
|
||||
" clean,\n",
|
||||
" is_english,\n",
|
||||
" predict_language,\n",
|
||||
" unchunk,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
|
||||
|
|
@ -161,7 +125,9 @@
|
|||
" ]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"preprocessed_chunks = parallel_map(preprocess_chunk, chunks)"
|
||||
"preprocessed_data = unchunk(\n",
|
||||
" simple_parallel_map(preprocess_chunk, chunks, concurrency=4)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -170,12 +136,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from itertools import chain\n",
|
||||
"\n",
|
||||
"preprocessed_data = list(chain(*preprocessed_chunks))\n",
|
||||
"X, y = zip(\n",
|
||||
" *preprocessed_data\n",
|
||||
") # X is the input, y is the expected (ground truth) output"
|
||||
"X, y = zip(*preprocessed_data) # X is the input, y is the expected output"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -194,33 +155,16 @@
|
|||
"mongo_connection_string=mongodb://localhost:27017/\n",
|
||||
"mongo_database=my_great_ai_db\n",
|
||||
"```\n",
|
||||
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n"
|
||||
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n",
|
||||
"\n",
|
||||
"Otherwise, TinyDB is used which is just a local JSON file."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,989 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,990 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongoDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,991 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,992 | INFO | Settings: configured ✅\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,993 | INFO | 🔩 tracing_database: MongoDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,994 | INFO | 🔩 large_file_implementation: LargeFileMongo\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,994 | INFO | 🔩 is_production: False\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,995 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,996 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,997 | INFO | 🔩 dashboard_table_size: 50\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,998 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,998 | WARNING | > Find out more at https://se-ml.github.io/practices/\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from great_ai import add_ground_truth\n",
|
||||
"\n",
|
||||
|
|
@ -231,7 +175,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Next: [Part 2](train.ipynb)"
|
||||
"### Next: [Part 2](/examples/simple/train)"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -256,7 +200,7 @@
|
|||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "c1f394f9662881005685eeb18d8f9f77079b1b8b9a5ece1f825bfa01fcb7f52f"
|
||||
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@
|
|||
|
||||
|
||||
|
||||
<title>Train a domain classifier on the semantic scholar dataset - GreatAI documentation</title>
|
||||
<title>Simple example: data engineering - GreatAI documentation</title>
|
||||
|
||||
|
||||
|
||||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -70,7 +83,7 @@
|
|||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#train-a-domain-classifier-on-the-semantic-scholar-dataset" class="md-skip">
|
||||
<a href="#simple-example-data-engineering" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
|
|
@ -103,7 +116,7 @@
|
|||
<div class="md-header__topic" data-md-component="header-topic">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
|
||||
</span>
|
||||
</div>
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -614,12 +641,12 @@
|
|||
|
||||
|
||||
<label class="md-nav__link md-nav__link--active" for="__toc">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<a href="./" class="md-nav__link md-nav__link--active">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
|
||||
|
||||
|
|
@ -694,7 +721,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -708,7 +735,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -822,6 +849,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="data.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -1871,13 +1899,13 @@ Licensed under the Apache License, Version 2.0.
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h1 id="train-a-domain-classifier-on-the-semantic-scholar-dataset">Train a domain classifier on the <a href="https://api.semanticscholar.org/corpus">semantic scholar dataset</a><a class="anchor-link" href="#train-a-domain-classifier-on-the-semantic-scholar-dataset">¶</a></h1><blockquote><p>Part 1: obtain and clean data</p>
|
||||
</blockquote>
|
||||
<p><img src="../diagrams/scope-data.svg" alt="position of this step in the lifecycle"></p>
|
||||
<blockquote><p>The blue boxes show the steps implemented in this notebook.</p>
|
||||
<h1 id="simple-example-data-engineering">Simple example: data engineering<a class="anchor-link" href="#simple-example-data-engineering">¶</a></h1><p>Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the <a href="https://api.semanticscholar.org/corpus">semantic scholar dataset</a> by taking full advantage of <code>great-ai</code>. Subsequently, we create a production-ready deployment.</p>
|
||||
<p><img src="/media/scope-data.svg" alt="position of this step in the lifecycle"></p>
|
||||
<blockquote><p>The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.</p>
|
||||
</blockquote>
|
||||
<p>Since the true scope of <code>great-ai</code> is the phase between proof-of-concept code and production-ready service, it is predominantly used in the <a href="/examples/simple/deploy">deployment notebook</a>. Feel free to skip there, or continue reading if you'd like to see the full picture.</p>
|
||||
<h3 id="extract">Extract<a class="anchor-link" href="#extract">¶</a></h3><p>This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.</p>
|
||||
<p>In this case, we download the semantic scholar dataset from a public S3 bucket.</p>
|
||||
<p>In this example, we download the semantic scholar dataset from a public S3 bucket.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1903,9 +1931,9 @@ Licensed under the Apache License, Version 2.0.
|
|||
</div>
|
||||
</clipboard-copy>
|
||||
</div>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">MAX_CHUNK_COUNT</span> <span class="o">=</span> <span class="mi">1</span>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">MAX_CHUNK_COUNT</span> <span class="o">=</span> <span class="mi">4</span>
|
||||
</pre></div>
|
||||
<div id="cell-1" class="clipboard-copy-txt">MAX_CHUNK_COUNT = 1</div>
|
||||
<div id="cell-1" class="clipboard-copy-txt">MAX_CHUNK_COUNT = 4</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1991,7 +2019,7 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output jp-OutputArea-executeResult" data-mime-type="text/plain">
|
||||
<pre>'Processing 1 out of the 6002 available chunks'</pre>
|
||||
<pre>'Processing 4 out of the 6002 available chunks'</pre>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
|
@ -2011,7 +2039,7 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<h3 id="transform">Transform<a class="anchor-link" href="#transform">¶</a></h3><ul>
|
||||
<li>Filter out non-English abstracts using <code>great_ai.utilities.predict_language</code></li>
|
||||
<li>Project it to only keep the necessary components (text and labels), clean the textual content using <code>great_ai.utilities.clean</code></li>
|
||||
<li>We will speed up processing using <code>great_ai.utilities.parallel_map</code>.</li>
|
||||
<li>We will speed up processing using <code>great_ai.utilities.simple_parallel_map</code>.</li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
|
@ -2041,7 +2069,13 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span><span class="p">,</span> <span class="n">Tuple</span>
|
||||
<span class="kn">import</span> <span class="nn">json</span>
|
||||
<span class="kn">import</span> <span class="nn">gzip</span>
|
||||
<span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">parallel_map</span><span class="p">,</span> <span class="n">clean</span><span class="p">,</span> <span class="n">is_english</span><span class="p">,</span> <span class="n">predict_language</span>
|
||||
<span class="kn">from</span> <span class="nn">great_ai.utilities</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">simple_parallel_map</span><span class="p">,</span>
|
||||
<span class="n">clean</span><span class="p">,</span>
|
||||
<span class="n">is_english</span><span class="p">,</span>
|
||||
<span class="n">predict_language</span><span class="p">,</span>
|
||||
<span class="n">unchunk</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">preprocess_chunk</span><span class="p">(</span><span class="n">chunk_key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]]:</span>
|
||||
|
|
@ -2068,12 +2102,20 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<span class="p">]</span>
|
||||
|
||||
|
||||
<span class="n">preprocessed_chunks</span> <span class="o">=</span> <span class="n">parallel_map</span><span class="p">(</span><span class="n">preprocess_chunk</span><span class="p">,</span> <span class="n">chunks</span><span class="p">)</span>
|
||||
<span class="n">preprocessed_data</span> <span class="o">=</span> <span class="n">unchunk</span><span class="p">(</span>
|
||||
<span class="n">simple_parallel_map</span><span class="p">(</span><span class="n">preprocess_chunk</span><span class="p">,</span> <span class="n">chunks</span><span class="p">,</span> <span class="n">concurrency</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
|
||||
<span class="p">)</span>
|
||||
</pre></div>
|
||||
<div id="cell-3" class="clipboard-copy-txt">from typing import List, Tuple
|
||||
import json
|
||||
import gzip
|
||||
from great_ai import parallel_map, clean, is_english, predict_language
|
||||
from great_ai.utilities import (
|
||||
simple_parallel_map,
|
||||
clean,
|
||||
is_english,
|
||||
predict_language,
|
||||
unchunk,
|
||||
)
|
||||
|
||||
|
||||
def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
|
||||
|
|
@ -2100,7 +2142,9 @@ def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
|
|||
]
|
||||
|
||||
|
||||
preprocessed_chunks = parallel_map(preprocess_chunk, chunks)</div>
|
||||
preprocessed_data = unchunk(
|
||||
simple_parallel_map(preprocess_chunk, chunks, concurrency=4)
|
||||
)</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2121,58 +2165,8 @@ preprocessed_chunks = parallel_map(preprocess_chunk, chunks)</div>
|
|||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="text/plain">
|
||||
<pre>Spacy model en_core_web_sm not found locally, downloading...
|
||||
Collecting en-core-web-sm==3.3.0
|
||||
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl (12.8 MB)
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 3.6 MB/s eta 0:00:00
|
||||
Requirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in ./.env/lib/python3.10/site-packages (from en-core-web-sm==3.3.0) (3.3.1)
|
||||
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.64.0)
|
||||
Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.9.1)
|
||||
Requirement already satisfied: srsly<3.0.0,>=2.4.3 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.4.3)
|
||||
Requirement already satisfied: typer<0.5.0,>=0.3.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.4.1)
|
||||
Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (59.6.0)
|
||||
Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.7)
|
||||
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.2)
|
||||
Requirement already satisfied: blis<0.8.0,>=0.4.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.7.8)
|
||||
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.7)
|
||||
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
|
||||
Requirement already satisfied: requests<3.0.0,>=2.13.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.28.0)
|
||||
Requirement already satisfied: numpy>=1.15.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.23.0)
|
||||
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.6)
|
||||
Requirement already satisfied: pathy>=0.3.5 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.6.1)
|
||||
Requirement already satisfied: jinja2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.1.2)
|
||||
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3.0)
|
||||
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.8.2)
|
||||
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.6)
|
||||
Requirement already satisfied: thinc<8.1.0,>=8.0.14 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.0.17)
|
||||
Requirement already satisfied: packaging>=20.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (21.3)
|
||||
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.env/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
|
||||
Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in ./.env/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (5.2.1)
|
||||
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.env/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.2.0)
|
||||
Requirement already satisfied: charset-normalizer~=2.0.0 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.12)
|
||||
Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2022.6.15)
|
||||
Requirement already satisfied: idna<4,>=2.5 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3)
|
||||
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.26.9)
|
||||
Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.env/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.1.3)
|
||||
Requirement already satisfied: MarkupSafe>=2.0 in ./.env/lib/python3.10/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.1.1)
|
||||
Installing collected packages: en-core-web-sm
|
||||
Successfully installed en-core-web-sm-3.3.0
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="jp-OutputArea-child">
|
||||
|
||||
|
||||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr">
|
||||
<pre><span style="color: rgb(255,255,0)">2022-06-25 14:21:57,983 | WARNING | Limiting concurrency to 1 because there are only 1 chunks</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:21:57,984 | INFO | Starting parallel map (concurrency: 1, chunk size: 1)</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:21:57,984 | WARNING | Running in series, there is no reason for parallelism</span>
|
||||
100%|██████████| 1/1 [03:26<00:00, 206.86s/it]
|
||||
<pre>100%|██████████| 4/4 [04:42<00:00, 70.62s/it]
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2203,19 +2197,9 @@ Successfully installed en-core-web-sm-3.3.0
|
|||
</div>
|
||||
</clipboard-copy>
|
||||
</div>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="kn">from</span> <span class="nn">itertools</span> <span class="kn">import</span> <span class="n">chain</span>
|
||||
|
||||
<span class="n">preprocessed_data</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">chain</span><span class="p">(</span><span class="o">*</span><span class="n">preprocessed_chunks</span><span class="p">))</span>
|
||||
<span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span>
|
||||
<span class="o">*</span><span class="n">preprocessed_data</span>
|
||||
<span class="p">)</span> <span class="c1"># X is the input, y is the expected (ground truth) output</span>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span><span class="o">*</span><span class="n">preprocessed_data</span><span class="p">)</span> <span class="c1"># X is the input, y is the expected output</span>
|
||||
</pre></div>
|
||||
<div id="cell-4" class="clipboard-copy-txt">from itertools import chain
|
||||
|
||||
preprocessed_data = list(chain(*preprocessed_chunks))
|
||||
X, y = zip(
|
||||
*preprocessed_data
|
||||
) # X is the input, y is the expected (ground truth) output</div>
|
||||
<div id="cell-4" class="clipboard-copy-txt">X, y = zip(*preprocessed_data) # X is the input, y is the expected output</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2238,17 +2222,18 @@ X, y = zip(
|
|||
</pre></div>
|
||||
<blockquote><p>You can install MongoDB from <a href="https://www.mongodb.com/docs/manual/installation">here</a> or <a href="https://www.mongodb.com/cloud/atlas/register">use it as a service</a></p>
|
||||
</blockquote>
|
||||
<p>Otherwise, TinyDB is used which is just a local JSON file.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
<div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
</div><div class="jp-Cell jp-CodeCell jp-Notebook-cell jp-mod-noOutputs ">
|
||||
<div class="jp-Cell jp-CodeCell jp-Notebook-cell jp-mod-noOutputs ">
|
||||
<div class="jp-Cell-inputWrapper">
|
||||
<div class="jp-Collapser jp-InputCollapser jp-Cell-inputCollapser">
|
||||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea">
|
||||
<div class="jp-InputPrompt jp-InputArea-prompt">In [5]:</div><div class="jp-CodeMirrorEditor jp-Editor jp-InputArea-editor" data-type="inline">
|
||||
<div class="jp-InputPrompt jp-InputArea-prompt">In [8]:</div><div class="jp-CodeMirrorEditor jp-Editor jp-InputArea-editor" data-type="inline">
|
||||
|
||||
|
||||
<div class="CodeMirror cm-s-jupyter">
|
||||
|
|
@ -2277,40 +2262,6 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<div class="jp-Cell-outputWrapper">
|
||||
<div class="jp-Collapser jp-OutputCollapser jp-Cell-outputCollapser">
|
||||
</div>
|
||||
|
||||
|
||||
<div class="jp-OutputArea jp-Cell-outputArea">
|
||||
|
||||
<div class="jp-OutputArea-child">
|
||||
|
||||
|
||||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr">
|
||||
<pre><span style="color: rgb(255,255,0)">2022-06-25 14:25:24,989 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,990 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongoDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,991 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,992 | INFO | Settings: configured ✅</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,993 | INFO | 🔩 tracing_database: MongoDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,994 | INFO | 🔩 large_file_implementation: LargeFileMongo</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,994 | INFO | 🔩 is_production: False</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,995 | INFO | 🔩 should_log_exception_stack: True</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,996 | INFO | 🔩 prediction_cache_size: 512</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,997 | INFO | 🔩 dashboard_table_size: 50</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:25:24,998 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:25:24,998 | WARNING | > Find out more at https://se-ml.github.io/practices/</span>
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="jp-Cell jp-MarkdownCell jp-Notebook-cell">
|
||||
|
|
@ -2319,7 +2270,7 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h3 id="next-part-2">Next: <a href="train.ipynb">Part 2</a><a class="anchor-link" href="#next-part-2">¶</a></h3>
|
||||
<h3 id="next-part-2">Next: <a href="/examples/simple/train">Part 2</a><a class="anchor-link" href="#next-part-2">¶</a></h3>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2333,7 +2284,7 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
@ -2370,13 +2321,13 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
|
||||
|
||||
|
||||
<a href="../train/" class="md-footer__link md-footer__link--next" aria-label="Next: Train a domain classifier on the semantic scholar dataset" rel="next">
|
||||
<a href="../train/" class="md-footer__link md-footer__link--next" aria-label="Next: Optimise and train a model" rel="next">
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Next
|
||||
</span>
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</div>
|
||||
</div>
|
||||
<div class="md-footer__button md-icon">
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -601,7 +628,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -615,7 +642,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -629,7 +656,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -706,6 +733,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/explanation.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
|
|
@ -745,7 +773,7 @@
|
|||
<nav class="md-footer__inner md-grid" aria-label="Footer" >
|
||||
|
||||
|
||||
<a href="../examples/simple/deploy/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Train a domain classifier on the semantic scholar dataset" rel="prev">
|
||||
<a href="../examples/simple/deploy/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Hardening and deployment" rel="prev">
|
||||
<div class="md-footer__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
|
||||
</div>
|
||||
|
|
@ -754,7 +782,7 @@
|
|||
<span class="md-footer__direction">
|
||||
Previous
|
||||
</span>
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</div>
|
||||
</div>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -601,7 +628,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -615,7 +642,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -629,7 +656,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -696,6 +723,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/call-remote.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
|
|
|
|||
|
|
@ -35,9 +35,9 @@ configure(
|
|||
|
||||
The only aspect that cannot be automated is choosing the backing storage for the database and file storage.
|
||||
|
||||
Right now, you have 3 options for storing the models and large datasets: [great_ai.large_file.LargeFileLocal][], [great_ai.large_file.LargeFileMongo][], and [great_ai.large_file.LargeFileS3][].
|
||||
Right now, you have 3 options for storing the models and large datasets: [LargeFileLocal][great_ai.large_file.LargeFileLocal], [LargeFileMongo][great_ai.large_file.LargeFileMongo], and [LargeFileS3][great_ai.large_file.LargeFileS3].
|
||||
|
||||
Without explicit configuration, [great_ai.large_file.LargeFileLocal][] is selected by default. This one still version-controls your files but it only stores them in a local path.
|
||||
Without explicit configuration, [LargeFileLocal][great_ai.large_file.LargeFileLocal] is selected by default. This one still version-controls your files but it only stores them in a local path.
|
||||
|
||||
!!! important
|
||||
If your working directory contains a `mongo.ini` or `s3.ini` file, an attempt is made to auto-configure [LargeFileMongo][great_ai.large_file.LargeFileMongo] or [LargeFileS3][great_ai.large_file.LargeFileS3] respectively.
|
||||
|
|
@ -76,6 +76,7 @@ save_model(model, 'my-model')
|
|||
MONGO_CONNECTION_STRING=mongodb://localhost:27017 # this is the default value
|
||||
# if `MONGO_CONNECTION_STRING` is specified, this default is overridden
|
||||
MONGO_CONNECTION_STRING=ENV:MONGO_CONNECTION_STRING
|
||||
|
||||
MONGO_DATABASE=my-database # it is automatically created if doesn't exist
|
||||
```
|
||||
|
||||
|
|
@ -98,4 +99,4 @@ By default, a thread-safe version of [TinyDB](https://tinydb.readthedocs.io/en/l
|
|||
|
||||
### MongoDB
|
||||
|
||||
At the moment, only MongoDB is supported as a production-ready `TracingDatabase`. In order to use it, you have to either place a file named `mongo.ini` in your working directory, or explicitly call [MongoDbDriver.configure_credentials_from_file][great_ai.MongoDbDriver.configure_credentials_from_file] or [MongoDbDriver.configure_credentials][great_ai.MongoDbDriver.configure_credentials].
|
||||
At the moment, only MongoDB is supported as a production-ready `TracingDatabase`. In order to use it, you have to either place a file named `mongo.ini` in your working directory, or explicitly call [MongoDbDriver.configure_credentials_from_file][great_ai.MongoDbDriver] or [MongoDbDriver.configure_credentials][great_ai.MongoDbDriver.configure_credentials].
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -353,6 +366,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -691,7 +718,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -705,7 +732,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -719,7 +746,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -848,6 +875,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/configure-service.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
|
|
@ -886,8 +914,8 @@
|
|||
</ol>
|
||||
<h2 id="using-remote-storage">Using remote storage<a class="headerlink" href="#using-remote-storage" title="Permanent link">#</a></h2>
|
||||
<p>The only aspect that cannot be automated is choosing the backing storage for the database and file storage.</p>
|
||||
<p>Right now, you have 3 options for storing the models and large datasets: <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileLocal">great_ai.large_file.LargeFileLocal</a>, <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileMongo">great_ai.large_file.LargeFileMongo</a>, and <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileS3">great_ai.large_file.LargeFileS3</a>.</p>
|
||||
<p>Without explicit configuration, <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileLocal">great_ai.large_file.LargeFileLocal</a> is selected by default. This one still version-controls your files but it only stores them in a local path.</p>
|
||||
<p>Right now, you have 3 options for storing the models and large datasets: <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileLocal">LargeFileLocal</a>, <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileMongo">LargeFileMongo</a>, and <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileS3">LargeFileS3</a>.</p>
|
||||
<p>Without explicit configuration, <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileLocal">LargeFileLocal</a> is selected by default. This one still version-controls your files but it only stores them in a local path.</p>
|
||||
<div class="admonition important">
|
||||
<p class="admonition-title">Important</p>
|
||||
<p>If your working directory contains a <code>mongo.ini</code> or <code>s3.ini</code> file, an attempt is made to auto-configure <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileMongo">LargeFileMongo</a> or <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileS3">LargeFileS3</a> respectively.</p>
|
||||
|
|
@ -919,7 +947,8 @@
|
|||
<div class="highlight"><span class="filename">mongo.ini</span><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a><span class="n">MONGO_CONNECTION_STRING</span><span class="o">=</span><span class="n">mongodb</span><span class="p">:</span><span class="err">//</span><span class="n">localhost</span><span class="p">:</span><span class="mi">27017</span><span class="w"> </span><span class="c1"># this is the default value</span><span class="w"></span>
|
||||
<a id="__codelineno-3-2" name="__codelineno-3-2" href="#__codelineno-3-2"></a><span class="c1"># if `MONGO_CONNECTION_STRING` is specified, this default is overridden</span><span class="w"></span>
|
||||
<a id="__codelineno-3-3" name="__codelineno-3-3" href="#__codelineno-3-3"></a><span class="n">MONGO_CONNECTION_STRING</span><span class="o">=</span><span class="n">ENV</span><span class="p">:</span><span class="n">MONGO_CONNECTION_STRING</span><span class="w"></span>
|
||||
<a id="__codelineno-3-4" name="__codelineno-3-4" href="#__codelineno-3-4"></a><span class="n">MONGO_DATABASE</span><span class="o">=</span><span class="n">my-database</span><span class="w"> </span><span class="c1"># it is automatically created if doesn't exist</span><span class="w"></span>
|
||||
<a id="__codelineno-3-4" name="__codelineno-3-4" href="#__codelineno-3-4"></a>
|
||||
<a id="__codelineno-3-5" name="__codelineno-3-5" href="#__codelineno-3-5"></a><span class="n">MONGO_DATABASE</span><span class="o">=</span><span class="n">my-database</span><span class="w"> </span><span class="c1"># it is automatically created if doesn't exist</span><span class="w"></span>
|
||||
</code></pre></div>
|
||||
<div class="highlight"><span class="filename">use-mongo.py</span><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a><span class="kn">from</span> <span class="nn">great_ai.large_file</span> <span class="kn">import</span> <span class="n">LargeFileMongo</span>
|
||||
<a id="__codelineno-4-2" name="__codelineno-4-2" href="#__codelineno-4-2"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">save_model</span>
|
||||
|
|
@ -936,14 +965,14 @@
|
|||
<h2 id="using-a-database">Using a database<a class="headerlink" href="#using-a-database" title="Permanent link">#</a></h2>
|
||||
<p>By default, a thread-safe version of <a href="https://tinydb.readthedocs.io/en/latest/" target="_blank">TinyDB</a> is utilised for saving the prediction traces into a local file. Unfortunately, for most production needs, this method is not suitable.</p>
|
||||
<h3 id="mongodb">MongoDB<a class="headerlink" href="#mongodb" title="Permanent link">#</a></h3>
|
||||
<p>At the moment, only MongoDB is supported as a production-ready <code>TracingDatabase</code>. In order to use it, you have to either place a file named <code>mongo.ini</code> in your working directory, or explicitly call [MongoDbDriver.configure_credentials_from_file][great_ai.MongoDbDriver.configure_credentials_from_file] or <a class="autorefs autorefs-internal" href="../../reference/#great_ai.persistence.mongodb_driver.MongoDbDriver.configure_credentials">MongoDbDriver.configure_credentials</a>.</p>
|
||||
<p>At the moment, only MongoDB is supported as a production-ready <code>TracingDatabase</code>. In order to use it, you have to either place a file named <code>mongo.ini</code> in your working directory, or explicitly call <a class="autorefs autorefs-internal" href="../../reference/#great_ai.MongoDbDriver">MongoDbDriver.configure_credentials_from_file</a> or <a class="autorefs autorefs-internal" href="../../reference/#great_ai.persistence.mongodb_driver.MongoDbDriver.configure_credentials">MongoDbDriver.configure_credentials</a>.</p>
|
||||
|
||||
<hr>
|
||||
<div class="md-source-file">
|
||||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ def greeter(your_name):
|
|||
|
||||
## With types
|
||||
|
||||
[Type annotating your codebase](https://realpython.com/python-type-checking/){ target=_blank } can save you from lots of trivial mistakes, that's why it's highly advised. Simply add the expected types to your function's signature.
|
||||
Even though it's not required by GreatAI, [type annotating your codebase](https://realpython.com/python-type-checking/){ target=_blank } can save you from lots of trivial mistakes, that's why it's highly advised. Simply add the expected types to your function's signature.
|
||||
|
||||
```python title="type_safe_greeter.py"
|
||||
from great_ai import GreatAI
|
||||
|
|
@ -100,7 +100,7 @@ assert type_safe_greeter('Andras').output == 'Hi Andras'
|
|||
Refer to the following example summarising the options you have when instantiating a GreatAI service.
|
||||
|
||||
```python title="complex.py"
|
||||
from great_ai import save_model, GreatAI, parameter, use_model
|
||||
from great_ai import save_model, GreatAI, parameter, use_model, log_metric
|
||||
|
||||
save_model(4, 'secret-number') #(1)
|
||||
|
||||
|
|
@ -108,6 +108,10 @@ save_model(4, 'secret-number') #(1)
|
|||
@parameter('positive_number', validator=lambda n: n > 0, disable_logging=True)
|
||||
@use_model('secret-number', version='latest', model_kwarg_name='secret')
|
||||
def add_number(positive_number: int, secret: int) -> int:
|
||||
log_metric(
|
||||
'log directly into the returned Trace',
|
||||
positive_number * 2
|
||||
)
|
||||
return positive_number + secret
|
||||
|
||||
assert add_number(1).output == 5
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -352,6 +365,20 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -685,7 +712,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -699,7 +726,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -713,7 +740,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -836,6 +863,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/create-service.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
|
|
@ -861,7 +889,7 @@
|
|||
<p>The purpose of the <a class="autorefs autorefs-internal" href="../../reference/#great_ai.deploy.great_ai.GreatAI.create">@GreatAI.create</a> is simply to provide you with type-checking through MyPy, Pylance, and similar libraries. However, the overloading support for <code>__new__</code> is lacking in MyPy, thus, a static factory method is used instead.</p>
|
||||
</details>
|
||||
<h2 id="with-types">With types<a class="headerlink" href="#with-types" title="Permanent link">#</a></h2>
|
||||
<p><a href="https://realpython.com/python-type-checking/" target="_blank">Type annotating your codebase</a> can save you from lots of trivial mistakes, that's why it's highly advised. Simply add the expected types to your function's signature.</p>
|
||||
<p>Even though it's not required by GreatAI, <a href="https://realpython.com/python-type-checking/" target="_blank">type annotating your codebase</a> can save you from lots of trivial mistakes, that's why it's highly advised. Simply add the expected types to your function's signature.</p>
|
||||
<div class="highlight"><span class="filename">type_safe_greeter.py</span><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">GreatAI</span>
|
||||
<a id="__codelineno-2-2" name="__codelineno-2-2" href="#__codelineno-2-2"></a>
|
||||
<a id="__codelineno-2-3" name="__codelineno-2-3" href="#__codelineno-2-3"></a><span class="nd">@GreatAI</span><span class="o">.</span><span class="n">create</span>
|
||||
|
|
@ -920,7 +948,7 @@
|
|||
</div>
|
||||
<h2 id="complex-example">Complex example<a class="headerlink" href="#complex-example" title="Permanent link">#</a></h2>
|
||||
<p>Refer to the following example summarising the options you have when instantiating a GreatAI service.</p>
|
||||
<div class="highlight"><span class="filename">complex.py</span><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">save_model</span><span class="p">,</span> <span class="n">GreatAI</span><span class="p">,</span> <span class="n">parameter</span><span class="p">,</span> <span class="n">use_model</span>
|
||||
<div class="highlight"><span class="filename">complex.py</span><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">save_model</span><span class="p">,</span> <span class="n">GreatAI</span><span class="p">,</span> <span class="n">parameter</span><span class="p">,</span> <span class="n">use_model</span><span class="p">,</span> <span class="n">log_metric</span>
|
||||
<a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a>
|
||||
<a id="__codelineno-6-3" name="__codelineno-6-3" href="#__codelineno-6-3"></a><span class="n">save_model</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="s1">'secret-number'</span><span class="p">)</span> <span class="c1">#(1)</span>
|
||||
<a id="__codelineno-6-4" name="__codelineno-6-4" href="#__codelineno-6-4"></a>
|
||||
|
|
@ -928,9 +956,13 @@
|
|||
<a id="__codelineno-6-6" name="__codelineno-6-6" href="#__codelineno-6-6"></a><span class="nd">@parameter</span><span class="p">(</span><span class="s1">'positive_number'</span><span class="p">,</span> <span class="n">validator</span><span class="o">=</span><span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span> <span class="o">></span> <span class="mi">0</span><span class="p">,</span> <span class="n">disable_logging</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
|
||||
<a id="__codelineno-6-7" name="__codelineno-6-7" href="#__codelineno-6-7"></a><span class="nd">@use_model</span><span class="p">(</span><span class="s1">'secret-number'</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s1">'latest'</span><span class="p">,</span> <span class="n">model_kwarg_name</span><span class="o">=</span><span class="s1">'secret'</span><span class="p">)</span>
|
||||
<a id="__codelineno-6-8" name="__codelineno-6-8" href="#__codelineno-6-8"></a><span class="k">def</span> <span class="nf">add_number</span><span class="p">(</span><span class="n">positive_number</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">secret</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
|
||||
<a id="__codelineno-6-9" name="__codelineno-6-9" href="#__codelineno-6-9"></a> <span class="k">return</span> <span class="n">positive_number</span> <span class="o">+</span> <span class="n">secret</span>
|
||||
<a id="__codelineno-6-10" name="__codelineno-6-10" href="#__codelineno-6-10"></a>
|
||||
<a id="__codelineno-6-11" name="__codelineno-6-11" href="#__codelineno-6-11"></a><span class="k">assert</span> <span class="n">add_number</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">output</span> <span class="o">==</span> <span class="mi">5</span>
|
||||
<a id="__codelineno-6-9" name="__codelineno-6-9" href="#__codelineno-6-9"></a> <span class="n">log_metric</span><span class="p">(</span>
|
||||
<a id="__codelineno-6-10" name="__codelineno-6-10" href="#__codelineno-6-10"></a> <span class="s1">'log directly into the returned Trace'</span><span class="p">,</span>
|
||||
<a id="__codelineno-6-11" name="__codelineno-6-11" href="#__codelineno-6-11"></a> <span class="n">positive_number</span> <span class="o">*</span> <span class="mi">2</span>
|
||||
<a id="__codelineno-6-12" name="__codelineno-6-12" href="#__codelineno-6-12"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-6-13" name="__codelineno-6-13" href="#__codelineno-6-13"></a> <span class="k">return</span> <span class="n">positive_number</span> <span class="o">+</span> <span class="n">secret</span>
|
||||
<a id="__codelineno-6-14" name="__codelineno-6-14" href="#__codelineno-6-14"></a>
|
||||
<a id="__codelineno-6-15" name="__codelineno-6-15" href="#__codelineno-6-15"></a><span class="k">assert</span> <span class="n">add_number</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">output</span> <span class="o">==</span> <span class="mi">5</span>
|
||||
</code></pre></div>
|
||||
<ol>
|
||||
<li>Refer to <a href="/how-to-guides/store-models">storing models</a> for specifying where to store your models. </li>
|
||||
|
|
@ -941,7 +973,7 @@
|
|||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
@ -962,7 +994,7 @@
|
|||
<nav class="md-footer__inner md-grid" aria-label="Footer" >
|
||||
|
||||
|
||||
<a href="../../tutorial/deploy/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Harden and deploy your app" rel="prev">
|
||||
<a href="../install/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Installation guide" rel="prev">
|
||||
<div class="md-footer__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
|
||||
</div>
|
||||
|
|
@ -971,7 +1003,7 @@
|
|||
<span class="md-footer__direction">
|
||||
Previous
|
||||
</span>
|
||||
Harden and deploy your app
|
||||
Installation guide
|
||||
</div>
|
||||
</div>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -353,6 +366,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -651,7 +678,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -665,7 +692,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -679,7 +706,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -768,6 +795,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/handle-training-data.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
|
|
|
|||
924
how-to-guides/install/index.html
Normal file
924
how-to-guides/install/index.html
Normal file
|
|
@ -0,0 +1,924 @@
|
|||
|
||||
<!doctype html>
|
||||
<html lang="en" class="no-js">
|
||||
<head>
|
||||
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
|
||||
<meta name="description" content="GreatAI helps you easily transform your prototype AI code into production-ready software.">
|
||||
|
||||
|
||||
<meta name="author" content="András Schmelczer">
|
||||
|
||||
|
||||
<link rel="icon" href="../../media/favicon.ico">
|
||||
<meta name="generator" content="mkdocs-1.3.0, mkdocs-material-8.3.9">
|
||||
|
||||
|
||||
|
||||
<title>Installation guide - GreatAI documentation</title>
|
||||
|
||||
|
||||
|
||||
<link rel="stylesheet" href="../../assets/stylesheets/main.1d29e8d0.min.css">
|
||||
|
||||
|
||||
<link rel="stylesheet" href="../../assets/stylesheets/palette.cbb835fc.min.css">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback">
|
||||
<style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style>
|
||||
|
||||
|
||||
|
||||
<link rel="stylesheet" href="../../assets/_mkdocstrings.css">
|
||||
|
||||
<script>__md_scope=new URL("../..",location),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<body dir="ltr" data-md-color-scheme="default" data-md-color-primary="light-blue" data-md-color-accent="">
|
||||
|
||||
|
||||
|
||||
<script>var palette=__md_get("__palette");if(palette&&"object"==typeof palette.color)for(var key of Object.keys(palette.color))document.body.setAttribute("data-md-color-"+key,palette.color[key])</script>
|
||||
|
||||
<input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
|
||||
<input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
|
||||
<label class="md-overlay" for="__drawer"></label>
|
||||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#installation-guide" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
</div>
|
||||
<div data-md-component="announce">
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
<header class="md-header" data-md-component="header">
|
||||
<nav class="md-header__inner md-grid" aria-label="Header">
|
||||
<a href="../.." title="GreatAI documentation" class="md-header__button md-logo" aria-label="GreatAI documentation" data-md-component="logo">
|
||||
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54Z"/></svg>
|
||||
|
||||
</a>
|
||||
<label class="md-header__button md-icon" for="__drawer">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 6h18v2H3V6m0 5h18v2H3v-2m0 5h18v2H3v-2Z"/></svg>
|
||||
</label>
|
||||
<div class="md-header__title" data-md-component="header-title">
|
||||
<div class="md-header__ellipsis">
|
||||
<div class="md-header__topic">
|
||||
<span class="md-ellipsis">
|
||||
GreatAI documentation
|
||||
</span>
|
||||
</div>
|
||||
<div class="md-header__topic" data-md-component="header-topic">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Installation guide
|
||||
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<form class="md-header__option" data-md-component="palette">
|
||||
|
||||
|
||||
|
||||
<input class="md-option" data-md-color-media="(prefers-color-scheme: light)" data-md-color-scheme="default" data-md-color-primary="light-blue" data-md-color-accent="" aria-label="Switch to dark mode" type="radio" name="__palette" id="__palette_1">
|
||||
|
||||
<label class="md-header__button md-icon" title="Switch to dark mode" for="__palette_2" hidden>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 2a7 7 0 0 0-7 7c0 2.38 1.19 4.47 3 5.74V17a1 1 0 0 0 1 1h6a1 1 0 0 0 1-1v-2.26c1.81-1.27 3-3.36 3-5.74a7 7 0 0 0-7-7M9 21a1 1 0 0 0 1 1h4a1 1 0 0 0 1-1v-1H9v1Z"/></svg>
|
||||
</label>
|
||||
|
||||
|
||||
|
||||
|
||||
<input class="md-option" data-md-color-media="(prefers-color-scheme: dark)" data-md-color-scheme="slate" data-md-color-primary="light-blue" data-md-color-accent="" aria-label="Switch to light mode" type="radio" name="__palette" id="__palette_2">
|
||||
|
||||
<label class="md-header__button md-icon" title="Switch to light mode" for="__palette_1" hidden>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 2a7 7 0 0 1 7 7c0 2.38-1.19 4.47-3 5.74V17a1 1 0 0 1-1 1H9a1 1 0 0 1-1-1v-2.26C6.19 13.47 5 11.38 5 9a7 7 0 0 1 7-7M9 21v-1h6v1a1 1 0 0 1-1 1h-4a1 1 0 0 1-1-1m3-17a5 5 0 0 0-5 5c0 2.05 1.23 3.81 3 4.58V16h4v-2.42c1.77-.77 3-2.53 3-4.58a5 5 0 0 0-5-5Z"/></svg>
|
||||
</label>
|
||||
|
||||
|
||||
</form>
|
||||
|
||||
|
||||
|
||||
<label class="md-header__button md-icon" for="__search">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5Z"/></svg>
|
||||
</label>
|
||||
<div class="md-search" data-md-component="search" role="dialog">
|
||||
<label class="md-search__overlay" for="__search"></label>
|
||||
<div class="md-search__inner" role="search">
|
||||
<form class="md-search__form" name="search">
|
||||
<input type="text" class="md-search__input" name="query" aria-label="Search" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="search-query" required>
|
||||
<label class="md-search__icon md-icon" for="__search">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5Z"/></svg>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
|
||||
</label>
|
||||
<nav class="md-search__options" aria-label="Search">
|
||||
|
||||
<button type="reset" class="md-search__icon md-icon" aria-label="Clear" tabindex="-1">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M19 6.41 17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12 19 6.41Z"/></svg>
|
||||
</button>
|
||||
</nav>
|
||||
|
||||
</form>
|
||||
<div class="md-search__output">
|
||||
<div class="md-search__scrollwrap" data-md-scrollfix>
|
||||
<div class="md-search-result" data-md-component="search-result">
|
||||
<div class="md-search-result__meta">
|
||||
Initializing search
|
||||
</div>
|
||||
<ol class="md-search-result__list"></ol>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="md-header__source">
|
||||
<a href="https://github.com/schmelczer/great-ai" title="Go to repository" class="md-source" data-md-component="source">
|
||||
<div class="md-source__icon md-icon">
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M439.55 236.05 244 40.45a28.87 28.87 0 0 0-40.81 0l-40.66 40.63 51.52 51.52c27.06-9.14 52.68 16.77 43.39 43.68l49.66 49.66c34.23-11.8 61.18 31 35.47 56.69-26.49 26.49-70.21-2.87-56-37.34L240.22 199v121.85c25.3 12.54 22.26 41.85 9.08 55a34.34 34.34 0 0 1-48.55 0c-17.57-17.6-11.07-46.91 11.25-56v-123c-20.8-8.51-24.6-30.74-18.64-45L142.57 101 8.45 235.14a28.86 28.86 0 0 0 0 40.81l195.61 195.6a28.86 28.86 0 0 0 40.8 0l194.69-194.69a28.86 28.86 0 0 0 0-40.81z"/></svg>
|
||||
</div>
|
||||
<div class="md-source__repository">
|
||||
schmelczer/great-ai
|
||||
</div>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</nav>
|
||||
|
||||
</header>
|
||||
|
||||
<div class="md-container" data-md-component="container">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<main class="md-main" data-md-component="main">
|
||||
<div class="md-main__inner md-grid">
|
||||
|
||||
|
||||
|
||||
<div class="md-sidebar md-sidebar--primary" data-md-component="sidebar" data-md-type="navigation" >
|
||||
<div class="md-sidebar__scrollwrap">
|
||||
<div class="md-sidebar__inner">
|
||||
|
||||
|
||||
|
||||
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
||||
<label class="md-nav__title" for="__drawer">
|
||||
<a href="../.." title="GreatAI documentation" class="md-nav__button md-logo" aria-label="GreatAI documentation" data-md-component="logo">
|
||||
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54Z"/></svg>
|
||||
|
||||
</a>
|
||||
GreatAI documentation
|
||||
</label>
|
||||
|
||||
<div class="md-nav__source">
|
||||
<a href="https://github.com/schmelczer/great-ai" title="Go to repository" class="md-source" data-md-component="source">
|
||||
<div class="md-source__icon md-icon">
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M439.55 236.05 244 40.45a28.87 28.87 0 0 0-40.81 0l-40.66 40.63 51.52 51.52c27.06-9.14 52.68 16.77 43.39 43.68l49.66 49.66c34.23-11.8 61.18 31 35.47 56.69-26.49 26.49-70.21-2.87-56-37.34L240.22 199v121.85c25.3 12.54 22.26 41.85 9.08 55a34.34 34.34 0 0 1-48.55 0c-17.57-17.6-11.07-46.91 11.25-56v-123c-20.8-8.51-24.6-30.74-18.64-45L142.57 101 8.45 235.14a28.86 28.86 0 0 0 0 40.81l195.61 195.6a28.86 28.86 0 0 0 40.8 0l194.69-194.69a28.86 28.86 0 0 0 0-40.81z"/></svg>
|
||||
</div>
|
||||
<div class="md-source__repository">
|
||||
schmelczer/great-ai
|
||||
</div>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../.." class="md-nav__link">
|
||||
Overview
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_2" type="checkbox" id="__nav_2" >
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_2">
|
||||
Tutorial
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="Tutorial" data-md-level="1">
|
||||
<label class="md-nav__title" for="__nav_2">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Tutorial
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../tutorial/" class="md-nav__link">
|
||||
Tutorial overview
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../tutorial/train/" class="md-nav__link">
|
||||
Train your model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../tutorial/deploy/" class="md-nav__link">
|
||||
Harden and deploy your app
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_3" type="checkbox" id="__nav_3" checked>
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3">
|
||||
User Guides
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="User Guides" data-md-level="1">
|
||||
<label class="md-nav__title" for="__nav_3">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
User Guides
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--active">
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="toc" type="checkbox" id="__toc">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link md-nav__link--active" for="__toc">
|
||||
Installation guide
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<a href="./" class="md-nav__link md-nav__link--active">
|
||||
Installation guide
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__title" for="__toc">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Table of contents
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#command-line-tools" class="md-nav__link">
|
||||
Command-line tools
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../configure-service/" class="md-nav__link">
|
||||
How to configure GreatAI
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../use-service/" class="md-nav__link">
|
||||
How to use a GreatAI service
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../handle-training-data/" class="md-nav__link">
|
||||
How to handle training data
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../large_file/" class="md-nav__link">
|
||||
How to use LargeFile
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../call_remote.md" class="md-nav__link">
|
||||
None
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../scraping/" class="md-nav__link">
|
||||
Scraping
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_4" type="checkbox" id="__nav_4" >
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_4">
|
||||
Reference
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="Reference" data-md-level="1">
|
||||
<label class="md-nav__title" for="__nav_4">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Reference
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../reference/" class="md-nav__link">
|
||||
GreatAI reference
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../reference/utilities/" class="md-nav__link">
|
||||
Utilities
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../reference/large-file/" class="md-nav__link">
|
||||
LargeFile
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../reference/views/" class="md-nav__link">
|
||||
View models
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_5" type="checkbox" id="__nav_5" >
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_5">
|
||||
Examples
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="Examples" data-md-level="1">
|
||||
<label class="md-nav__title" for="__nav_5">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Examples
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_5_1" type="checkbox" id="__nav_5_1" >
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_5_1">
|
||||
Explainable Naive Bayes
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="Explainable Naive Bayes" data-md-level="2">
|
||||
<label class="md-nav__title" for="__nav_5_1">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Explainable Naive Bayes
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../explanation/" class="md-nav__link">
|
||||
Explanation
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<div class="md-sidebar md-sidebar--secondary" data-md-component="sidebar" data-md-type="toc" >
|
||||
<div class="md-sidebar__scrollwrap">
|
||||
<div class="md-sidebar__inner">
|
||||
|
||||
|
||||
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__title" for="__toc">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Table of contents
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#command-line-tools" class="md-nav__link">
|
||||
Command-line tools
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="md-content" data-md-component="content">
|
||||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/install.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<h1 id="installation-guide">Installation guide<a class="headerlink" href="#installation-guide" title="Permanent link">#</a></h1>
|
||||
<p>Provided you already have <a href="https://www.python.org/downloads/" target="_blank">Python3</a> (and pip) installed, simply execute:</p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a>pip install great-ai
|
||||
</code></pre></div>
|
||||
<blockquote>
|
||||
<p>Python 3.8 or later is required.</p>
|
||||
</blockquote>
|
||||
<p>This will work on all major operating systems.</p>
|
||||
<h2 id="command-line-tools">Command-line tools<a class="headerlink" href="#command-line-tools" title="Permanent link">#</a></h2>
|
||||
<p>After installation, <code>great-ai</code> and <code>large-file</code> are available as commands. The former is required for deploying your application while the latter lets you manage models and datasets from your terminal.</p>
|
||||
<div class="admonition warning">
|
||||
<p class="admonition-title">Windows</p>
|
||||
<p>On Windows, you might encounter a similar <mark>warning</mark> from <code>pip</code>:</p>
|
||||
<blockquote>
|
||||
<p>WARNING: The scripts great-ai.exe, great_ai.exe, large-file.exe and large_file.exe are installed in 'C:\Users...\Scripts' which is not on PATH.
|
||||
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.</p>
|
||||
</blockquote>
|
||||
<p>This means that <code>great-ai.exe</code> and <code>large-file.exe</code> are not in your <code>PATH</code>. Either add their containing directory ('C:\Users...\Scripts' in this case) to your <code>PATH</code> or use <code>python3 -m great_ai</code> and <code>python3 -m great_ai.large_file</code> instead of the exe-s.</p>
|
||||
</div>
|
||||
|
||||
<hr>
|
||||
<div class="md-source-file">
|
||||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</article>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</main>
|
||||
|
||||
<footer class="md-footer">
|
||||
|
||||
|
||||
<nav class="md-footer__inner md-grid" aria-label="Footer" >
|
||||
|
||||
|
||||
<a href="../../tutorial/deploy/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Harden and deploy your app" rel="prev">
|
||||
<div class="md-footer__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
|
||||
</div>
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Previous
|
||||
</span>
|
||||
Harden and deploy your app
|
||||
</div>
|
||||
</div>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<a href="../create-service/" class="md-footer__link md-footer__link--next" aria-label="Next: How to create a GreatAI service" rel="next">
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Next
|
||||
</span>
|
||||
How to create a GreatAI service
|
||||
</div>
|
||||
</div>
|
||||
<div class="md-footer__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11H4Z"/></svg>
|
||||
</div>
|
||||
</a>
|
||||
|
||||
</nav>
|
||||
|
||||
<div class="md-footer-meta md-typeset">
|
||||
<div class="md-footer-meta__inner md-grid">
|
||||
<div class="md-copyright">
|
||||
|
||||
<div class="md-copyright__highlight">
|
||||
GNU General Public License v3
|
||||
</div>
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
<div class="md-social">
|
||||
|
||||
|
||||
|
||||
<a href="https://schmelczer.dev" target="_blank" rel="noopener" title="about the author" class="md-social__link">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="m414.8 40.79-128 448.01c-4.9 17-22.6 26.8-39.6 22-17-4.9-26.8-22.6-22-39.6l128-447.99c4.9-16.994 22.6-26.834 39.6-21.978 17 4.855 26.8 22.568 22 39.558zm103.8 80.61 112 112c12.5 12.5 12.5 32.7 0 45.2l-112 112c-12.5 12.5-32.7 12.5-45.2 0s-12.5-32.7 0-45.2l89.3-89.4-89.3-89.4c-12.5-12.5-12.5-32.7 0-45.2s32.7-12.5 45.2 0zm-352 45.2L77.25 256l89.35 89.4c12.5 12.5 12.5 32.7 0 45.2s-32.7 12.5-45.2 0L9.372 278.6c-12.496-12.5-12.496-32.7 0-45.2l112.028-112c12.5-12.5 32.7-12.5 45.2 0s12.5 32.7 0 45.2z"/></svg>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai" target="_blank" rel="noopener" title="great-ai on Github" class="md-social__link">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 496 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<a href="https://hub.docker.com/repository/docker/schmelczera/great-ai" target="_blank" rel="noopener" title="great-ai on DockerHub" class="md-social__link">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M349.9 236.3h-66.1v-59.4h66.1v59.4zm0-204.3h-66.1v60.7h66.1V32zm78.2 144.8H362v59.4h66.1v-59.4zm-156.3-72.1h-66.1v60.1h66.1v-60.1zm78.1 0h-66.1v60.1h66.1v-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7H2.4c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4.4 67.6.1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zm-511.1-27.9h-66v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm-78.1-72.1h-66.1v60.1h66.1v-60.1z"/></svg>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<a href="https://pypi.org/project/great-ai" target="_blank" rel="noopener" title="great-ai on PyPI" class="md-social__link">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M439.8 200.5c-7.7-30.9-22.3-54.2-53.4-54.2h-40.1v47.4c0 36.8-31.2 67.8-66.8 67.8H172.7c-29.2 0-53.4 25-53.4 54.3v101.8c0 29 25.2 46 53.4 54.3 33.8 9.9 66.3 11.7 106.8 0 26.9-7.8 53.4-23.5 53.4-54.3v-40.7H226.2v-13.6h160.2c31.1 0 42.6-21.7 53.4-54.2 11.2-33.5 10.7-65.7 0-108.6zM286.2 404c11.1 0 20.1 9.1 20.1 20.3 0 11.3-9 20.4-20.1 20.4-11 0-20.1-9.2-20.1-20.4.1-11.3 9.1-20.3 20.1-20.3zM167.8 248.1h106.8c29.7 0 53.4-24.5 53.4-54.3V91.9c0-29-24.4-50.7-53.4-55.6-35.8-5.9-74.7-5.6-106.8.1-45.2 8-53.4 24.7-53.4 55.6v40.7h106.9v13.6h-147c-31.1 0-58.3 18.7-66.8 54.2-9.8 40.7-10.2 66.1 0 108.6 7.6 31.6 25.7 54.2 56.8 54.2H101v-48.8c0-35.3 30.5-66.4 66.8-66.4zm-6.7-142.6c-11.1 0-20.1-9.1-20.1-20.3.1-11.3 9-20.4 20.1-20.4 11 0 20.1 9.2 20.1 20.4s-9 20.3-20.1 20.3z"/></svg>
|
||||
</a>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</footer>
|
||||
|
||||
</div>
|
||||
<div class="md-dialog" data-md-component="dialog">
|
||||
<div class="md-dialog__inner md-typeset"></div>
|
||||
</div>
|
||||
<script id="__config" type="application/json">{"base": "../..", "features": ["content.code.annotate", "content.tooltips"], "search": "../../assets/javascripts/workers/search.b97dbffb.min.js", "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.config.lang": "en", "search.config.pipeline": "trimmer, stopWordFilter", "search.config.separator": "[\\s\\-]+", "search.placeholder": "Search", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version.title": "Select version"}}</script>
|
||||
|
||||
|
||||
<script src="../../assets/javascripts/bundle.6c7ad80a.min.js"></script>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
21
how-to-guides/install/install.md
Normal file
21
how-to-guides/install/install.md
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
# Installation guide
|
||||
|
||||
Provided you already have [Python3](https://www.python.org/downloads/){ target=_blank } (and pip) installed, simply execute:
|
||||
|
||||
```sh
|
||||
pip install great-ai
|
||||
```
|
||||
> Python 3.8 or later is required.
|
||||
|
||||
This will work on all major operating systems.
|
||||
|
||||
## Command-line tools
|
||||
|
||||
After installation, `great-ai` and `large-file` are available as commands. The former is required for deploying your application while the latter lets you manage models and datasets from your terminal.
|
||||
|
||||
!!! warning "Windows"
|
||||
On Windows, you might encounter a similar ==warning== from `pip`:
|
||||
> WARNING: The scripts great-ai.exe, great_ai.exe, large-file.exe and large_file.exe are installed in 'C:\Users\...\Scripts' which is not on PATH.
|
||||
> Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
|
||||
|
||||
This means that `great-ai.exe` and `large-file.exe` are not in your `PATH`. Either add their containing directory ('C:\Users\...\Scripts' in this case) to your `PATH` or use `python3 -m great_ai` and `python3 -m great_ai.large_file` instead of the exe-s.
|
||||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -353,6 +366,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -732,7 +759,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -746,7 +773,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -760,7 +787,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -930,6 +957,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/large_file.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -348,6 +361,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -606,7 +633,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -620,7 +647,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -634,7 +661,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -697,6 +724,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="scraping.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -353,6 +366,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -705,7 +732,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -719,7 +746,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -733,7 +760,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -876,6 +903,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/use-service.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
|
|
@ -919,9 +947,10 @@
|
|||
<p>Some configuration options are also supported.</p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a>great-ai greeter.py --port <span class="m">8000</span> --host <span class="m">127</span>.0.0.1 --timeout_keep_alive <span class="m">10</span>
|
||||
</code></pre></div>
|
||||
<blockquote>
|
||||
<p>For more options (but no Notebook support, use <a href="https://www.uvicorn.org/" target="_blank">uvicorn</a>)</p>
|
||||
</blockquote>
|
||||
<details class="note">
|
||||
<summary>More options</summary>
|
||||
<p>For more options (but no Notebook support), simply use <a href="https://www.uvicorn.org/" target="_blank">uvicorn</a> for starting your app (available at <code>greeter.app</code>).</p>
|
||||
</details>
|
||||
<h3 id="in-production">In production<a class="headerlink" href="#in-production" title="Permanent link">#</a></h3>
|
||||
<p>There are three main approaches for deploying a GreatAI service.</p>
|
||||
<h4 id="manual-deployment">Manual deployment<a class="headerlink" href="#manual-deployment" title="Permanent link">#</a></h4>
|
||||
|
|
@ -941,7 +970,7 @@
|
|||
<p>You can replace <code>pwd</code> with the path to your code's folder.</p>
|
||||
</blockquote>
|
||||
<h4 id="use-a-platform-as-a-service">Use a Platform-as-a-Service<a class="headerlink" href="#use-a-platform-as-a-service" title="Permanent link">#</a></h4>
|
||||
<p>Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as <a href="https://www.digitalocean.com/products/app-platform" target="_blank">DO App platform</a> or <a href="https://mlem.ai/" target="_blank">MLEM</a>) that take a Docker image as a source and handle the rest of the deployment.</p>
|
||||
<p>Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as <a href="https://aws.amazon.com/ecs/" target="_blank">AWS ECS</a>, <a href="https://www.digitalocean.com/products/app-platform" target="_blank">DO App platform</a>, <a href="https://mlem.ai/" target="_blank">MLEM</a>) that take a Docker image as a source and handle the rest of the deployment.</p>
|
||||
<p>To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as <a href="https://pytorch.org/" target="_blank">PyTorch</a> or <a href="https://www.tensorflow.org/" target="_blank">TensorFlow</a>.</p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="k">FROM</span><span class="w"> </span><span class="s">schmelczera/great-ai:latest</span>
|
||||
<a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a>
|
||||
|
|
@ -988,7 +1017,7 @@
|
|||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue