Minor docs fixes

This commit is contained in:
Andras Schmelczer 2022-07-16 15:35:46 +02:00
commit adcd300fa4
No known key found for this signature in database
GPG key ID: 39260B5B0614A13E
5 changed files with 29 additions and 28 deletions

View file

@ -0,0 +1,114 @@
# How to use LargeFiles
The functions [save_model][great_ai.use_model] and [@use_model][great_ai.use_model] wrap LargeFile instances. Hence, besides configuring [LargeFile](/reference/large-file), users have few reasons to use LargeFiles directly.
## Motivation
Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation [doesn't support deleting](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository) large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.
[DVC](https://dvc.org/) is a viable alternative, however, it requires users to learn to use one more CLI tool.
??? note "Using LargeFile-s directly (usually not needed)"
LargeFile doesn't require users to learn too much new. It is a nearly exact copy of the built-in `open()` function of Python with which users are certainly already familiar.
## Simple example
```python
from great_ai.large_file import LargeFileS3
LargeFileS3.configure_credentials({
"aws_region_name": "your_region_like_eu-west-2",
"aws_access_key_id": "YOUR_ACCESS_KEY_ID",
"aws_secret_access_key": "YOUR_VERY_SECRET_ACCESS_KEY",
"large_files_bucket_name": "create_a_bucket_and_put_its_name_here",
})
# Creates a new version and deletes the older version
# leaving the 3 most recently used intact
with LargeFileS3("test.txt", "w", keep_last_n=3) as f:
for i in range(100000):
f.write('test\n')
# The latest version is returned by default
# but an optional `version` keyword argument can be provided as well
with LargeFileS3("test.txt", "r") as f: #(1)
print(f.readlines()[0])
```
1. In this case, the latest version is already in the local cache, no download is required.
### More details
`LargeFile` behaves like an opened file (in the background it is a temp file after all). Binary reads and writes are supported along with the [different keywords `open()` accepts](https://docs.python.org/3/library/functions.html#open){ target=_blank }.
The local cache can be configured with these properties:
```python
LargeFileS3.cache_path = Path('.cache')
LargeFileS3.max_cache_size = "30 GB"
```
#### I only need a path
In case you only need a path to the (proxy of the) remote file, this pattern can be applied:
```python
path_to_model = LargeFileS3("folder-of-my-bert-model", version=31).get()
```
> This will first download the file/folder into your local cache folder. Then, it returns a `Path` object to the local version. Which can be turned into a string with `str(path_to_model)`.
The same approach works for uploads:
```python
LargeFileS3("folder-of-my-bert-model").push('path_to_local/folder_or_file')
```
> This way both regular files and folders can be handled. The uploaded file is called **folder-of-my-bert-model**, the local name is ignored.
Lastly, all version of the remote object can be deleted by calling `LargeFileS3("my-file").delete()`. It will still reside in your local cache afterwards; its deletion will happen next time your local cache has to be pruned.
## From the command-line
The main reason for using the `large-file` or `python3 -m great_ai.large_file` commands is to upload or download models from the terminal. For example, when building a docker image, it is best-practice to cache the referred models.
### Setup
Create an .ini file (or use *~/.aws/credentials*). It may look like this:
```ini
aws_region_name = your_region_like_eu-west-2
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_VERY_SECRET_ACCESS_KEY
large_files_bucket_name = my_large_files
```
### Upload some files
```sh
large-file --backend s3 --secrets secrets.ini \
--push my_first_file.json folder/my_second_file my_folder
```
> Only the filename is used as the S3 name, the rest of the path is ignored.
!!! important "Using MongoDB"
The possible values for `--backend` are `s3`, `mongo`, and `local`. The latter doesn't need credentials, it only versions and stores your files in a local folder. MongoDB on the other hand requires a `mongo_connection_string` and a `mongo_database` to be specified. For storing large files, it uses the [GridFS](https://www.mongodb.com/docs/manual/core/gridfs){ target=_blank } specification.
### Download some files to the local cache
This can be useful when building a Docker image for example. This way, the files can already reside inside the container and need not be downloaded later.
```sh
large-file --backend s3 --secrets ~/.aws/credentials \
--cache my_first_file.json:3 my_second_file my_folder:0
```
> Versions may be specified by using `:`-s.
### Delete remote files
```sh
large-file --backend s3 --secrets ~/.aws/credentials \
--delete my_first_file.json
```

View file

@ -1,4 +1,5 @@
"""GreatAI.""" """Transform your prototype AI code into production-ready software."""
__version__ = "0.1.6" __version__ = "0.1.6"

View file

@ -34,7 +34,7 @@ def evaluate_ranking(
`expected`) `expected`)
title: Title of the plot. title: Title of the plot.
disable_interpolation: Do not interpolate. disable_interpolation: Do not interpolate.
axes: Matplotlib axes for ploting inside a subplot. axes: Matplotlib axes for plotting inside a subplot.
output_svg: If specified, save the chart as an svg to the given Path. output_svg: If specified, save the chart as an svg to the given Path.
reverse_order: Reverse the ranking specified by `expected`. reverse_order: Reverse the ranking specified by `expected`.
plot: Display a plot on the screen. plot: Display a plot on the screen.

View file

@ -91,7 +91,7 @@ def parallel_map(
or ignored. or ignored.
The new processes are forked if the OS allows it, otherwise, new Python processes The new processes are forked if the OS allows it, otherwise, new Python processes
are bootstrapped which can incur some start-sup cost. Each process processes a single are bootstrapped which can incur some start-up cost. Each process processes a single
chunk at once. chunk at once.
Examples: Examples:

View file

@ -6,7 +6,7 @@ repo_url: https://github.com/schmelczer/great-ai
repo_name: schmelczer/great-ai repo_name: schmelczer/great-ai
edit_uri: edit/main/docs/ edit_uri: edit/main/docs/
copyright: GNU General Public License v3 copyright: GNU General Public License v3
extra: extra:
generator: false generator: false
@ -22,7 +22,7 @@ extra:
name: great-ai on DockerHub name: great-ai on DockerHub
- icon: fontawesome/brands/python - icon: fontawesome/brands/python
link: https://pypi.org/project/great-ai link: https://pypi.org/project/great-ai
name: great-ai on PyPI name: great-ai on PyPI
theme: theme:
name: "material" name: "material"
@ -48,8 +48,8 @@ theme:
name: Switch to light mode name: Switch to light mode
watch: watch:
- docs - docs
- mkdocs.yaml - mkdocs.yaml
plugins: plugins:
- git-revision-date-localized: - git-revision-date-localized:
@ -89,27 +89,27 @@ markdown_extensions:
nav: nav:
- Overview: index.md - Overview: index.md
- Tutorial: - Tutorial:
- Tutorial overview: tutorial/index.md - Tutorial overview: tutorial/index.md
- tutorial/train.ipynb - tutorial/train.ipynb
- tutorial/deploy.ipynb - tutorial/deploy.ipynb
- User Guides: - User Guides:
- how-to-guides/install.md - how-to-guides/install.md
- how-to-guides/create-service.md - how-to-guides/create-service.md
- how-to-guides/configure-service.md - how-to-guides/configure-service.md
- how-to-guides/use-service.md - how-to-guides/use-service.md
- how-to-guides/handle-training-data.md - how-to-guides/handle-training-data.md
- how-to-guides/large_file.md - how-to-guides/large-file.md
- how-to-guides/call-remote.md - how-to-guides/call-remote.md
- Reference: - Reference:
- reference/index.md - reference/index.md
- reference/utilities.md - reference/utilities.md
- reference/large-file.md - reference/large-file.md
- reference/views.md - reference/views.md
- Examples: - Examples:
- Explainable Naive Bayes: - Explainable Naive Bayes:
- examples/simple/data.ipynb - examples/simple/data.ipynb
- examples/simple/train.ipynb - examples/simple/train.ipynb
- examples/simple/deploy.ipynb - examples/simple/deploy.ipynb
- Explainable SciBERT: examples/scibert.md - Explainable SciBERT: examples/scibert.md
- explanation.md - explanation.md