Add files

Signed-off-by: András Schmelczer <andras@schmelczer.dev>

good_ai/.gitignore

@@ -0,0 +1 @@
+build

good_ai/README.md

@@ -0,0 +1,33 @@
+# **S**coutinScience **U**tilitie**S** for text processing [![Lint and test ScoutinScience utilities](https://github.com/ScoutinScience/platform/actions/workflows/sus-general.yaml/badge.svg)](https://github.com/ScoutinScience/platform/actions/workflows/sus-general.yaml)
+
+> amogus
+
+## Exports
+
+- [clean](src/sus/clean.py)
+- [unique](src/sus/unique.py)
+- [parallel_map](src/sus/parallel_map.py)
+- [match_names](src/sus/match_names/match_names.py)
+- [evaluate_ranking](src/sus/evaluate_ranking/evaluate_ranking.py)
+
+### Requires loading spacy model
+
+> This is automatic but will require some time.
+
+> Add this to the Dockerfile for caching the spaCy model:
+>
+> ```docker
+> RUN pip install --no-cache-dir en-core-web-lg@https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.2.0/en_core_web_lg-3.2.0-py3-none-any.whl
+> ```
+
+- [spacy model (nlp)](src/sus/nlp.py)
+- [get_sentences](src/sus/get_sentences.py)
+- [lemmatize_text](src/sus/lemmatize_text.py)
+- [lemmatize_token](src/sus/lemmatize_token.py)
+- [publication TEI](src/sus/publication_tei/publication_tei.py)
+
+## Development
+
+- Optional booleans must have a default value of `False`.
+- No imports in top-level `__init__.py`, in order to not load anything unnecessary automatically
+- Should only be updated through a PR

good_ai/example_secrets.ini

@@ -0,0 +1,6 @@
+[DEFAULT]
+aws_region_name = your_region_like_eu-west-2
+aws_access_key_id = YOUR_ACCESS_KEY_ID
+aws_secret_access_key = YOUR_VERY_SECRET_ACCESS_KEY
+large_files_bucket_name = create_a_bucket_and_put_its_name_here
+endpoint_url = this is optional, for backblaze, use this: https://s3.us-west-002.backblazeb2.com

good_ai/open_s3.md

@@ -0,0 +1,119 @@
+# [open(S3)](https://pypi.org/project/open-large/)
+
+Storing, versioning, and downloading files from S3 made as easy as using `open()` in Python. Caching included.
+
+## Motivation
+
+Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation [doesn't support the deletion](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository) of large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.
+
+## Solution
+
+```
+pip install open-large
+```
+
+### Simple example
+
+```python
+from open_s3 import LargeFile
+
+LargeFile.configure_credentials({
+    "aws_region_name": "your_region_like_eu-west-2",
+    "aws_access_key_id": "YOUR_ACCESS_KEY_ID",
+    "aws_secret_access_key": "YOUR_VERY_SECRET_ACCESS_KEY",
+    "large_files_bucket_name": "create_a_bucket_and_put_its_name_here",
+})
+
+# Creates a new version and deletes the older version leaving the 3 most recently used intact
+with LargeFile("test.txt", "w", keep_last_n=3) as f:
+    for i in range(100000):
+        f.write('test\n')
+
+# By default the latest version is returned
+# but an optional `version` keyword argument can be provided as well
+with LargeFile("test.txt", "r") as f:
+    print(f.readlines()[0])
+```
+
+> Automatically creates a file, writes to it, uploads it to S3, and then queries the most recent version of it.
+> In this case, the latest version is already in the local cache, no download is required.
+
+### More details
+
+`LargeFile` behaves like an opened file (in the background it is a temp file after all). Binary reading and writing is supported along with the [different keywords](https://docs.python.org/3/library/functions.html#open) `open()` accepts.
+
+The local cache can be configured with these properties:
+
+```python
+LargeFile.cache_path = Path('.cache')
+LargeFile.max_cache_size = "30 GB"
+```
+
+#### I only need a path
+
+In case you only need a path to the "remote" file, this pattern can be applied:
+
+```python
+path_to_model = LargeFile("folder-of-my-bert-model", version=31).get()
+```
+
+> This will first download the file/folder into your local cache folder. Then, it returns a `Path` object to the local version. Which can be turned into a string with `str(path_to_model)`.
+
+The same approach works for uploads:
+
+```python
+LargeFile("folder-of-my-bert-model").push('path_to_local/folder_or_file')
+```
+
+> This way, both regular files and folders can be handled. The uploaded file is called **folder-of-my-bert-model**, the local name is ignored.
+
+Lastly, all version of the remote object can be deleted by calling `LargeFile("my-file").delete()`. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.
+
+### Command-line example
+
+The package can be used as a module from the command-line to give you more flexibility.
+
+#### Setup
+
+Create an .ini file (or use _~/.aws/credentials_). It may look like this:
+
+```ini
+[DEFAULT]
+aws_region_name = your_region_like_eu-west-2
+aws_access_key_id = YOUR_ACCESS_KEY_ID
+aws_secret_access_key = YOUR_VERY_SECRET_ACCESS_KEY
+large_files_bucket_name = my_large_files
+endpoint_url = this is optional, for backblaze, use this: https://s3.us-west-002.backblazeb2.com
+```
+
+> Just like in [example secrets](example_secrets.ini).
+
+#### Print the expected options
+
+```sh
+python3 -m open_s3 --help
+```
+
+#### Upload some files
+
+```sh
+python3 -m open_s3 --secrets secrets.ini --push my_first_file.json folder/my_second_file my_folder
+```
+
+> Only the filename is used as the S3 name, the rest of the path is ignored.
+
+#### Download some files to the local cache
+
+This can be useful when building a Docker image for example. This way, the files can already reside inside the container and need not be downloaded later.
+
+```sh
+python3 -m open_s3 --secrets ~/.aws/credentials --cache my_first_file.json:3 my_second_file my_folder:0
+```
+
+> Versions may be specified by using `:`-s.
+
+#### Delete remote files
+
+```sh
+python3 -m open_s3 --secrets ~/.aws/credentials --delete my_first_file.json
+```

good_ai/pyproject.toml

@@ -0,0 +1,7 @@
+[build-system]
+requires = [
+    "setuptools>=42",
+    "setuptools-git",
+    "wheel"
+]
+build-backend = "setuptools.build_meta"

good_ai/requirements.txt

@@ -0,0 +1,11 @@
+unidecode >= 1.3.0
+multiprocess >= 0.70.0.0
+tqdm >= 4.0.0
+psutil >= 5.9.0
+beautifulsoup4 >= 4.10.0
+lxml >= 4.6.0
+spacy >= 3.2.0
+pydantic >= 1.8.0
+scikit-learn >= 1.0.0
+matplotlib >= 3.5.0
+numpy >= 1.22.0

good_ai/setup.cfg

@@ -0,0 +1,39 @@
+[metadata]
+name = sus
+version = 0.0.3
+author = ScoutinScience B.V.
+author_email = andras@scoutinscience.com
+description = [S]coutinScience [u]tilitie[s]: reusable utilities for text processing
+long_description = file: README.md
+long_description_content_type = text/markdown
+url = https://github.com/ScoutinScience/platform
+project_urls =
+    Bug Tracker = https://github.com/ScoutinScience/platform/issues
+classifiers =
+    Programming Language :: Python :: 3
+    Operating System :: OS Independent
+
+[options]
+package_dir =
+    = src
+packages = find:
+include_package_data = True
+python_requires = >=3.8
+install_requires =
+    click < 8.1.0
+    unidecode >= 1.3.0
+    multiprocess >= 0.70.0.0
+    tqdm >= 4.0.0
+    psutil >= 5.9.0
+    beautifulsoup4 >= 4.10.0
+    lxml >= 4.6.0
+    spacy >= 3.2.0
+    pydantic >= 1.8.0
+    scikit-learn >= 1.0.0
+    matplotlib >= 3.5.0
+    numpy >= 1.22.0
+    langcodes[data] >= 3.3.0
+    langdetect >= 1.0.9
+
+[options.packages.find]
+where = src

good_ai/src/__init__.py

@@ -0,0 +1 @@
+

good_ai/src/open_s3/__init__.py

@@ -0,0 +1 @@
+from .large_file import LargeFile

good_ai/src/open_s3/__main__.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+
+import logging
+from pathlib import Path
+
+from large_file import LargeFile
+from parse_arguments import parse_arguments
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO)
+    parser, args = parse_arguments()
+
+    LargeFile.configure_credentials_from_file(args.secrets)
+
+    if not args.cache and not args.push and not args.delete:
+        logging.warn("No action required.")
+        parser.print_help()
+    try:
+        if args.cache:
+            for c in args.cache:
+                split = c.split(":")
+                file_name = split[0]
+                version = None if len(split) == 1 else int(split[1])
+                LargeFile(file_name, "r", version=version).get()
+
+        if args.push:
+            for p in args.push:
+                path = Path(p)
+                LargeFile(path.name, "w").push(path)
+
+        if args.delete:
+            for f in args.delete:
+                LargeFile(f).delete()
+    except Exception as e:
+        logging.error(e)

good_ai/src/open_s3/helper/__init__.py

@@ -0,0 +1,2 @@
+from .progress_bar import DownloadProgressBar, UploadProgressBar
+from .human_readable_to_byte import human_readable_to_byte

good_ai/src/open_s3/helper/human_readable_to_byte.py

@@ -0,0 +1,31 @@
+import re
+
+
+def human_readable_to_byte(size: str) -> int:
+    """Case is ignored, kb, kB, Kb, and KB are all treated as kilobyte."""
+
+    if size.strip() == "0":
+        return 0
+
+    possible_units = ("B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB")
+    units_re = "|".join(possible_units)
+    regex = re.compile(
+        rf"""
+        \s*                         # trim
+        (?P<scalar>\d+(.\d+)?)      # get scalar, it might be a float
+        \s*                         # ignore optional whitespace
+        (?P<unit>{units_re})        # capture the unit
+    """,
+        flags=re.VERBOSE | re.IGNORECASE,
+    )
+
+    match = regex.match(size)
+    if not match:
+        raise ValueError(f'Could not find values in "{size}"')
+
+    results = match.groupdict()
+
+    scalar = float(results["scalar"])
+    idx = possible_units.index(results["unit"].upper())
+    factor = 1024 ** idx
+    return round(scalar * factor)

good_ai/src/open_s3/helper/progress_bar.py

@@ -0,0 +1,40 @@
+import os
+import threading
+from logging import Logger
+from typing import IO, Any, Optional
+
+from tqdm.auto import tqdm
+from pathlib import Path
+
+import os
+import sys
+import threading
+
+
+class ProgressBar:
+    def __init__(self, file_size: int, logger: Logger, prefix: str):
+        self._file_size = file_size
+        self._logger = logger
+        self._prefix=prefix
+
+        self._seen_so_far = 0
+        self._lock = threading.Lock()
+
+    def __call__(self, bytes_amount: int):
+        with self._lock:
+            self._seen_so_far += bytes_amount
+            percentage = (self._seen_so_far / float(self._file_size)) * 100
+            size_length = len(str(self._file_size))
+            progress = str(self._seen_so_far).rjust(size_length)
+            self._logger.info(f"{self._prefix} {progress}/{self._file_size} bytes ({percentage:.1f}%)")
+
+
+class DownloadProgressBar(ProgressBar):
+    def __init__(self, name: str, size: int, logger: Logger):
+        super().__init__(file_size=size, logger=logger, prefix=f'Downloading {name}')
+
+
+class UploadProgressBar(ProgressBar):
+    def __init__(self, path: Path, logger: Logger):
+        size = os.path.getsize(path)
+        super().__init__(file_size=size, logger=logger, prefix=f'Uploading {path.name}')

good_ai/src/open_s3/large_file.py

@@ -0,0 +1,346 @@
+import configparser
+import logging
+import os
+import shutil
+import tempfile
+from pathlib import Path
+from types import TracebackType
+from typing import IO, Any, Dict, List, Optional, Type, Union
+import boto3
+from botocore.exceptions import ClientError
+
+from helper import DownloadProgressBar, UploadProgressBar, human_readable_to_byte
+
+logger = logging.getLogger("open_s3")
+
+
+class LargeFile:
+    """
+    Store large files in S3. Use local cache for speed up.
+
+    Examples:
+
+    ```
+    with LargeFile("test.txt", "w", keep_last_n=3) as f:
+        for i in range(1000000):
+            f.write('test\n')
+
+    with LargeFile("test.txt", "r") as f:
+        print(f.readlines()[0])
+
+    path_to_cached_text_file = LargeFile("test.txt", version=0).get()
+    ```
+
+    By default, files are stored in the ".cache" folder and the
+    least recently use is deleted after the overall size reaches 30 GBs.
+
+    Change it with the following properties.
+
+    ```
+    LargeFile.cache_path = Path(".cache")
+    LargeFile.max_cache_size = "30GB"
+    ```
+    """
+
+    region_name = None
+    access_key_id = None
+    secret_access_key = None
+    bucket_name = None
+    endpoint_url = None
+
+    cache_path = Path(".cache")
+    max_cache_size: Optional[str] = "30GB"
+
+    def __init__(
+        self,
+        name: str,
+        mode: str = "r",
+        *,
+        buffering: int = -1,
+        encoding: Optional[str] = None,
+        errors: Optional[str] = None,
+        newline: Optional[str] = None,
+        version: Optional[int] = None,
+        keep_last_n: Optional[int] = None,
+        offline_mode: bool = False
+    ):
+        self._name: str = name
+        self._version = version
+        self._mode: str = mode
+        self._keep_last_n = keep_last_n
+        self._offline_mode = offline_mode
+
+        self._buffering = buffering
+        self._encoding = encoding
+        self._errors = errors
+        self._newline = newline
+
+        LargeFile.cache_path.mkdir(parents=True, exist_ok=True)
+
+        self._find_versions()
+        self._check_mode_and_set_version()
+        
+    @classmethod
+    def configure_credentials(
+        cls,
+        *,
+        aws_region_name: str,
+        aws_access_key_id: str,
+        aws_secret_access_key: str,
+        large_files_bucket_name: str,
+        endpoint_url: Optional[str] = None,
+        **_: Dict[str, Any],
+    ) -> None:
+        cls.region_name = aws_region_name
+        cls.access_key_id = aws_access_key_id
+        cls.secret_access_key = aws_secret_access_key
+        cls.bucket_name = large_files_bucket_name
+        cls.endpoint_url = endpoint_url
+
+    @classmethod
+    def configure_credentials_from_file(
+        cls,
+        secrets_path: Union[Path, str],
+    ) -> None:
+
+        if isinstance(secrets_path, str):
+            secrets_path = Path(secrets_path)
+
+        if not secrets_path.exists():
+            raise FileNotFoundError(secrets_path.resolve())
+
+        credentials = configparser.ConfigParser()
+        credentials.read(secrets_path)
+        credentials.default_section
+        cls.configure_credentials(**credentials[credentials.default_section])
+
+    def __enter__(self) -> IO:
+        self._file: IO[Any] = (
+            tempfile.NamedTemporaryFile(
+                mode=self._mode,
+                buffering=self._buffering,
+                encoding=self._encoding,
+                newline=self._newline,
+                errors=self._errors,
+                delete=False,
+                prefix="large-file-",
+            )
+            if "w" in self._mode
+            else open(
+                self.get(),
+                mode=self._mode,
+                buffering=self._buffering,
+                encoding=self._encoding,
+                newline=self._newline,
+                errors=self._errors,
+            )
+        )
+
+        return self._file
+
+    def __exit__(
+        self,
+        type: Optional[Type[BaseException]],
+        exc: Optional[BaseException],
+        traceback: Optional[TracebackType],
+    ) -> bool:
+        self._file.close()
+
+        if type is None:
+            if "w" in self._mode:
+                self.push(Path(self._file.name))
+                os.unlink(self._file.name)
+        else:
+            logger.exception("Could not finish operation.")
+
+        return True
+    
+    @property
+    def version_ids(self) -> List[int]:
+        return [self._get_version_from_key(key) for key in self._versions]
+
+    def get(self, hide_progress: bool = False) -> Path:
+        key = next(
+            key
+            for key in self._versions
+            if self._get_version_from_key(key) == self._version
+        )
+
+        destination = self.cache_path / self._local_name
+        if not destination.exists():
+            logger.info(
+                f"File {self._local_name} does not exist locally, starting download from S3"
+            )
+
+            with tempfile.TemporaryDirectory() as tmp:
+                tmp_file_archive = Path(tmp) / f"{self._local_name}.tar.gz"
+
+                size = self._client.head_object(Bucket=self.bucket_name, Key=key)['ContentLength']
+                self._client.download_file(Bucket=self.bucket_name, Key=key, Filename=str(tmp_file_archive), Callback=None if hide_progress else DownloadProgressBar(size=size, name=key, logger=logger))
+                logger.info(f"Decompressing {self._local_name}")
+                shutil.unpack_archive(str(tmp_file_archive), tmp, "gztar")
+                tmp_file = Path(tmp) / self._local_name
+                shutil.move(str(tmp_file), str(destination))
+        else:
+            logger.info(f"File {self._local_name} found in cache")
+
+        return destination
+
+    def push(self, path: Union[Path, str], hide_progress: bool = False) -> None:
+        if isinstance(path, str):
+            path = Path(path)
+
+        with tempfile.TemporaryDirectory() as tmp:
+            if path.is_file():
+                logger.info(f"Copying file for {self._local_name}")
+                copy = shutil.copy 
+            else:
+                logger.info(f"Copying directory for {self._local_name}")
+                copy = shutil.copytree
+
+            try:
+                # Make local copy in the cache
+                copy(str(path), str(self.cache_path / self._local_name))
+            except shutil.SameFileError:
+                pass  # No worries
+
+            copy(str(path), str(Path(tmp) / self._local_name))
+
+            logger.info(f"Compressing {self._local_name}")
+            shutil.make_archive(
+                str(Path(tmp) / self._local_name),
+                "gztar",
+                tmp,
+            )
+
+            logger.info(f"Uploading {self._local_name} to S3 from {path}")
+
+            file_to_be_uploaded = Path(tmp) / f"{self._local_name}.tar.gz"
+            self._client.upload_file(Filename=str(file_to_be_uploaded), Bucket=self.bucket_name, Key=self._s3_name, Callback=None if hide_progress else UploadProgressBar(file_to_be_uploaded, logger=logger))
+
+        self.clean_up()
+
+    def delete(self) -> None:
+        self._keep_last_n = 0
+        self._delete_old_versions_from_s3()
+
+    def clean_up(self) -> None:
+        self._delete_old_versions_from_s3()
+        self._delete_old_versions_from_disk()
+
+    def _create_client(self) -> None:
+        if (
+            self.region_name is None
+            or self.access_key_id is None
+            or self.secret_access_key is None
+            or self.bucket_name is None
+        ):
+            raise ValueError(
+                "Please configure the S3 access options by calling LargeFile.configure_credentials or set offline_mode=True in the constructor."
+            )
+
+        self._client = boto3.client('s3', aws_access_key_id=self.access_key_id, aws_secret_access_key=self.secret_access_key, region_name=self.region_name, endpoint_url=self.endpoint_url)
+        
+    def _find_versions(self) -> None:
+        if self._offline_mode:
+            self._fetch_versions_from_cache()
+        else:
+            self._create_client()
+            self._fetch_versions_from_s3()
+        
+        if self._versions:
+            logger.info(f"Found versions: {self.version_ids}")
+        else:
+            logger.info("No versions found")
+
+    def _fetch_versions_from_cache(self) -> None:
+        logger.info(f"Fetching offline versions of {self._name}")
+
+        self._versions = [
+            path
+            for path in self.cache_path.glob(f'{self._local_name}-*')
+        ]
+
+    def _fetch_versions_from_s3(self) -> None:
+        logger.info(f"Fetching online versions of {self._name}")
+        found_objects = self._client.list_objects_v2(
+            Bucket=self.bucket_name, Prefix=self._name
+        )
+        self._versions = (
+            sorted(
+                o["Key"]
+                for o in found_objects["Contents"]
+            )
+            if "Contents" in found_objects
+            else []
+        )
+
+    def _check_mode_and_set_version(self) -> None:
+        if "+" in self._mode:
+            raise ValueError("Read-write mode is not allowed.")
+
+        if "w" in self._mode:
+            if self._version is not None:
+                raise ValueError("Providing a version is not allowed in write mode.")
+
+            self._version = self.version_ids[-1] + 1 if self.version_ids else 0
+
+        elif "r" in self._mode:
+            if not self.version_ids:
+                raise FileNotFoundError(
+                    f"File {self._name} not found. No versions are available."
+                )
+
+            if self._version is None:
+                self._version = self.version_ids[-1]
+                logger.info(f"Lastest version of {self._local_name} is {self._version}")
+
+            elif self._version not in self.version_ids:
+                raise FileNotFoundError(
+                    f"File {self._name} not found with version {self._version}. Available versions: {self.version_ids}"
+                )
+        else:
+            raise ValueError("Unsupported file mode.")
+
+    @property
+    def _local_name(self) -> str:
+        return f"{self._name}-{self._version}"
+
+    @property
+    def _s3_name(self) -> str:
+        return f"{self._name}/{self._version}"
+        
+    @staticmethod
+    def _get_version_from_key(key: Union[str, Path]) -> int:
+        if isinstance(key, Path):
+            return int(key.name.split("-")[-1])
+        return int(key.split("/")[-1])
+
+
+    def _delete_old_versions_from_s3(self) -> None:
+        if self._keep_last_n is not None:
+            for key in (self._versions[: -self._keep_last_n] if self._keep_last_n > 0 else self._versions):
+                logger.info(
+                    f"Removing old version (keep_last_n={self._keep_last_n}): {key}"
+                )
+                self._client.delete_object(Bucket=self.bucket_name, Key=key)
+                
+    def _delete_old_versions_from_disk(self) -> None:
+        self.cache_path.mkdir(parents=True, exist_ok=True)
+
+        if self.max_cache_size is None:
+            return
+
+        allowed_size = human_readable_to_byte(self.max_cache_size)
+        assert allowed_size >= 0
+
+        least_recently_read = sorted(
+            [f for f in self.cache_path.glob("*")], key=lambda f: f.stat().st_atime
+        )
+
+        while sum(os.path.getsize(f) for f in least_recently_read) > allowed_size:
+            file = least_recently_read.pop(0)
+            logger.info(
+                f"Deleting file from cache to meet quota (max_cache_size={self.max_cache_size}): {file}"
+            )
+            os.unlink(file)

good_ai/src/open_s3/parse_arguments.py

@@ -0,0 +1,48 @@
+from argparse import ArgumentParser, Namespace
+from typing import Tuple
+
+
+def parse_arguments() -> Tuple[ArgumentParser, Namespace]:
+    parser = ArgumentParser(
+        description="Store and version large files in S3; open them like regular files. Caching included.",
+    )
+
+    parser.add_argument(
+        "-s",
+        "--secrets",
+        type=str,
+        help="path to an .ini configration file with your S3 credentials",
+        required=True,
+    )
+
+    parser.add_argument(
+        "-c",
+        "--cache",
+        nargs="+",
+        type=str,
+        help="download file into local cache, example: file_name:version",
+        required=False,
+    )
+
+    parser.add_argument(
+        "-p",
+        "--push",
+        nargs="+",
+        type=str,
+        help="push a local file into S3 and set it as the most recent version",
+        required=False,
+    )
+
+    parser.add_argument(
+        "-d",
+        "--delete",
+        nargs="+",
+        type=str,
+        help="delete every version of file from S3",
+        required=False,
+    )
+
+    parser.print_usage = parser.print_help  # type: ignore
+    args = parser.parse_args()
+
+    return parser, args

good_ai/src/sus.egg-info/PKG-INFO

@@ -0,0 +1,50 @@
+Metadata-Version: 2.1
+Name: sus
+Version: 0.0.3
+Summary: [S]coutinScience [u]tilitie[s]: reusable utilities for text processing
+Home-page: https://github.com/ScoutinScience/platform
+Author: ScoutinScience B.V.
+Author-email: andras@scoutinscience.com
+License: UNKNOWN
+Project-URL: Bug Tracker, https://github.com/ScoutinScience/platform/issues
+Platform: UNKNOWN
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+
+# **S**coutinScience **U**tilitie**S** for text processing [![Lint and test ScoutinScience utilities](https://github.com/ScoutinScience/platform/actions/workflows/sus-general.yaml/badge.svg)](https://github.com/ScoutinScience/platform/actions/workflows/sus-general.yaml)
+
+> amogus
+
+## Exports
+
+- [clean](src/sus/clean.py)
+- [unique](src/sus/unique.py)
+- [parallel_map](src/sus/parallel_map.py)
+- [match_names](src/sus/match_names/match_names.py)
+- [evaluate_ranking](src/sus/evaluate_ranking/evaluate_ranking.py)
+
+### Requires loading spacy model
+
+> This is automatic but will require some time.
+
+> Add this to the Dockerfile for caching the spaCy model:
+>
+> ```docker
+> RUN pip install --no-cache-dir en-core-web-lg@https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.2.0/en_core_web_lg-3.2.0-py3-none-any.whl
+> ```
+
+- [spacy model (nlp)](src/sus/nlp.py)
+- [get_sentences](src/sus/get_sentences.py)
+- [lemmatize_text](src/sus/lemmatize_text.py)
+- [lemmatize_token](src/sus/lemmatize_token.py)
+- [publication TEI](src/sus/publication_tei/publication_tei.py)
+
+## Development
+
+- Optional booleans must have a default value of `False`.
+- No imports in top-level `__init__.py`, in order to not load anything unnecessary automatically
+- Should only be updated through a PR
+
+

good_ai/src/sus.egg-info/SOURCES.txt

@@ -0,0 +1,78 @@
+.gitignore
+README.md
+pyproject.toml
+requirements.txt
+setup.cfg
+src/__init__.py
+src/sus/__init__.py
+src/sus/clean.py
+src/sus/get_sentences.py
+src/sus/lemmatize_text.py
+src/sus/lemmatize_token.py
+src/sus/nlp.py
+src/sus/parallel_map.py
+src/sus/unique.py
+src/sus.egg-info/PKG-INFO
+src/sus.egg-info/SOURCES.txt
+src/sus.egg-info/dependency_links.txt
+src/sus.egg-info/requires.txt
+src/sus.egg-info/top_level.txt
+src/sus/data/__init__.py
+src/sus/data/american_spellings.py
+src/sus/data/punctuations.py
+src/sus/evaluate_ranking/__init__.py
+src/sus/evaluate_ranking/draw_f1_iso_lines.py
+src/sus/evaluate_ranking/evaluate_ranking.py
+src/sus/external/__init__.py
+src/sus/external/negspacy/README.md
+src/sus/external/negspacy/__init__.py
+src/sus/external/negspacy/negation.py
+src/sus/external/negspacy/termsets.py
+src/sus/external/pylatexenc/README.md
+src/sus/external/pylatexenc/__init__.py
+src/sus/external/pylatexenc/_util.py
+src/sus/external/pylatexenc/version.py
+src/sus/external/pylatexenc/latex2text/__init__.py
+src/sus/external/pylatexenc/latex2text/__main__.py
+src/sus/external/pylatexenc/latex2text/_defaultspecs.py
+src/sus/external/pylatexenc/latexencode/__init__.py
+src/sus/external/pylatexenc/latexencode/__main__.py
+src/sus/external/pylatexenc/latexencode/_partial_latex_encoder.py
+src/sus/external/pylatexenc/latexencode/_uni2latexmap.py
+src/sus/external/pylatexenc/latexencode/_uni2latexmap_xml.py
+src/sus/external/pylatexenc/latexencode/_unicode_to_latex_encoder.py
+src/sus/external/pylatexenc/latexwalker/__init__.py
+src/sus/external/pylatexenc/latexwalker/__main__.py
+src/sus/external/pylatexenc/latexwalker/_defaultspecs.py
+src/sus/external/pylatexenc/macrospec/__init__.py
+src/sus/external/pylatexenc/macrospec/_argparsers.py
+src/sus/language/__init__.py
+src/sus/language/english_name_of_language.py
+src/sus/language/is_english.py
+src/sus/language/predict_language.py
+src/sus/match_names/__init__.py
+src/sus/match_names/config.py
+src/sus/match_names/match_names.py
+src/sus/match_names/name_parts.py
+src/sus/publication_tei/__init__.py
+src/sus/publication_tei/publication_tei.py
+src/sus/publication_tei/models/__init__.py
+src/sus/publication_tei/models/affiliation.py
+src/sus/publication_tei/models/author.py
+src/sus/publication_tei/models/element.py
+src/sus/publication_tei/models/publication_metadata.py
+src/sus/publication_tei/models/text.py
+tests/__init__.py
+tests/test_clean.py
+tests/test_evaluate_ranking.py
+tests/test_get_sentences.py
+tests/test_language.py
+tests/test_lemmatize_text.py
+tests/test_lemmatize_token.py
+tests/test_match_names.py
+tests/test_parallel_map.py
+tests/test_publication_tei.py
+tests/test_unique.py
+tests/data/10.1136_bmjspcare-2021-003026.pdf.tei.xml
+tests/data/bad.tei.xml
+tests/data/parsed.py

good_ai/src/sus.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

good_ai/src/sus.egg-info/requires.txt

@@ -0,0 +1,14 @@
+click<8.1.0
+unidecode>=1.3.0
+multiprocess>=0.70.0.0
+tqdm>=4.0.0
+psutil>=5.9.0
+beautifulsoup4>=4.10.0
+lxml>=4.6.0
+spacy>=3.2.0
+pydantic>=1.8.0
+scikit-learn>=1.0.0
+matplotlib>=3.5.0
+numpy>=1.22.0
+langcodes[data]>=3.3.0
+langdetect>=1.0.9

good_ai/src/sus.egg-info/top_level.txt

@@ -0,0 +1 @@
+sus