How to perform prediction with GreatAI

After creating a GreatAI service by wrapping your prediction function, and optionally configuring it, it's time to do some prediction.

Let's take the following example:

from great_ai import GreatAI

@GreatAI.create
def greeter(your_name: str) -> str:
    return f'Hi {your_name}!'

One-off prediction

Even though greeter is now an instance of [GreatAI][great_ai.GreatAI], you can continue using it as a regular function.

>>> greeter('Bob')
Trace[str]({'created': '2022-07-11T14:31:46.183764',
  'exception': None,
  'feedback': None,
  'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
  'models': [],
  'original_execution_time_ms': 0.0381,
  'output': 'Hi Bob!',
  'tags': ['greeter', 'online', 'development'],
  'trace_id': '7c284fd7-7f0d-4464-b5f8-3ef126df34af'})

As you can see, the original return value is wrapped in a [Trace][great_ai.Trace] object (which is also persisted in your database of choice). You can access the original value under the output property.

Online prediction

Likely, the main way you would like to expose your model is through an HTTP API. [@GreatAI.create][great_ai.GreatAI.create] scaffolds many REST API endpoints for your model and creates a FastAPI{ target=_blank } app available under [GreatAI.app][great_ai.GreatAI]. This can be served using uvicorn{ target=_blank } or any other ASGI server{ target=_blank }.

Since most ML code lives in Jupyter{ target=_blank } notebooks, therefore, deploying a notebook containing the inference function is supported. To achieve this, uvicorn is wrapped by the great-ai command-line utility, which — among others — takes care of feeding a notebook into uvicorn. It also supports auto-reloading.

In development

great-ai greeter.py

!!! success Your model is accessible at localhost:6060{ target=_blank }.

Some configuration options are also supported.

great-ai greeter.py --port 8000 --host 127.0.0.1 --timeout_keep_alive 10

??? note "More options" For more options (but no Notebook support), simply use uvicorn{ target=_blank } for starting your app (available at greeter.app).

In production

There are three main approaches for deploying a GreatAI service.

Manual deployment

The app is run in production-mode if the value of the ENVIRONMENT environment variable is set to production.

ENVIRONMENT=production great-ai greeter.py

Simply run ENVIRONMENT=production great-ai deploy.ipynb in the command-line of a production machine.

This is the crudest approach; however, it might be fitting for some contexts.

Containerised deployment

Run the notebook directly in a container or create a service for it using your favourite container orchestrator.

docker run -p 6060:6060 --volume `pwd`:/app --rm \
  schmelczera/great-ai deploy.ipynb

You can replace pwd with the path to your code's folder.

Use a Platform-as-a-Service

Similar to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as AWS ECS{ target=_blank }, DO App platform{ target=_blank }, MLEM{ target=_blank }, Streamlit{ target=_blank }) that take a Docker image as a source and handle the rest of the deployment.

To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as PyTorch{ target=_blank } or TensorFlow{ target=_blank }.

# syntax=docker/dockerfile:1
FROM schmelczera/great-ai:latest

COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

# Remove this block if you don't have a requirements.txt
COPY requirements.txt ./   
RUN uv pip install --system --no-cache --requirement requirements.txt

# Cache an S3 model using credentials available only during this RUN.
# Remove this step if your application does not use a remote model.
RUN --mount=type=secret,id=s3,target=/run/secrets/s3.ini,required=true \
    large-file --backend s3 --secrets /run/secrets/s3.ini --cache my-domain-predictor

# Add your application code to the image
COPY . .

# The default ENTRYPOINT is great-ai; specify its argument using CMD
CMD ["deploy.ipynb"]

Create an s3.ini with the S3 keys described in the configuration guide. Use concrete values in this build configuration: ENV: references are resolved inside the build container, which does not inherit your shell's environment. Replace my-domain-predictor with the key used when saving your model.

Keep credentials and any local model cache out of COPY . . by adding these entries to your project's .dockerignore:

s3.ini
mongo.ini
.env
.cache

Build with Docker BuildKit secrets. This example disables Docker's layer cache so that the large-file --cache step runs and downloads the model into the image:

DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain \
  --secret id=s3,src=s3.ini --tag my-great-ai-service .

The secret mount is removed after the RUN instruction; the downloaded model cache remains in the image. Supply any configuration needed by the application separately at runtime. For example, an application that auto-configures S3 from s3.ini can use a read-only bind mount:

docker run --rm -p 6060:6060 \
  --mount type=bind,source="$(pwd)/s3.ini",target=/app/s3.ini,readonly \
  my-great-ai-service

Batch prediction

Processing larger amounts of data on a single machine is made easy by the [GreatAI][great_ai.GreatAI]'s [process_batch][great_ai.GreatAI.process_batch] method. This relies on multiprocessing ([parallel_map][great_ai.utilities.parallel_map.parallel_map.parallel_map]) to take full advantage of all available CPU-cores.

>>> greeter.process_batch(['Alice', 'Bob'])
[Trace[str]({'created': '2022-07-11T14:36:37.119183',
   'exception': None,
   'feedback': None,
   'logged_values': {'arg:your_name:length': 5, 'arg:your_name:value': 'Alice'},
   'models': [],
   'original_execution_time_ms':  0.1251,
   'output': 'Hi Alice!',
   'tags': ['greeter', 'online', 'development'],
   'trace_id': '90ffa15f-e839-41c4-8e7a-3211168bc138'}),
 Trace[str]({'created': '2022-07-11T14:36:37.166659',
   'exception': None,
   'feedback': None,
   'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
   'models': [],
   'original_execution_time_ms':  0.0571,
   'output': 'Hi Bob!',
   'tags': ['greeter', 'online', 'development'],
   'trace_id': 'f48e94c7-0815-48b3-a864-41349d3dae84'})]

7.1 KiB Raw Blame History