great-ai/docs/how-to-guides/use-service.md

# How to perform prediction with GreatAI

After [creating a GreatAI service](https://great-ai.schmelczer.dev/how-to-guides/create-service) by wrapping your prediction function, and optionally [configuring it](https://great-ai.schmelczer.dev/how-to-guides/configure-service), it's time to do some prediction.

Let's take the following example:

```python title="greeter.py"
from great_ai import GreatAI

@GreatAI.create
def greeter(your_name: str) -> str:
    return f'Hi {your_name}!'
```

## One-off prediction

Even though `greeter` is now an instance of [GreatAI][great_ai.GreatAI], you can continue using it as a regular function.

```python
>>> greeter('Bob')
Trace[str]({'created': '2022-07-11T14:31:46.183764',
  'exception': None,
  'feedback': None,
  'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
  'models': [],
  'original_execution_time_ms': 0.0381,
  'output': 'Hi Bob!',
  'tags': ['greeter', 'online', 'development'],
  'trace_id': '7c284fd7-7f0d-4464-b5f8-3ef126df34af'})
```

As you can see, the original return value is wrapped in a [Trace][great_ai.Trace] object (which is also persisted in your database of choice). You can access the original value under the `output` property.

Repeated inputs are cached by default and reuse their trace ID. Each call returns
an independent copy, so changing a returned output does not change later results.
Feedback updates are reflected in subsequent calls; deleting a persisted trace
causes the next prediction for that input to run again and receive a new trace ID.
Set `configure(prediction_cache_size=0)` before creating the service to disable caching.

### NumPy values

The Python API accepts NumPy arrays and scalars. Array cache keys include their
contents, shape, and dtype. Outputs retain their NumPy types in Python; traces
stored in the database and HTTP responses serialize them as JSON arrays and scalars.
Unsupported objects must be converted to JSON-compatible values before logging or
returning them.

An `np.ndarray` annotation exposes a JSON array in the HTTP API and converts the
incoming list to an array before calling your function:

```python
import numpy as np
from great_ai import GreatAI

@GreatAI.create
def double(values: np.ndarray) -> np.ndarray:
    return values * 2

assert double(np.array([1, 2])).output.tolist() == [2, 4]
```

For this service, send `{"values": [1, 2]}` to `/predict`. Return annotations,
including Pydantic models, are included in the prediction response's OpenAPI schema.
Custom validation failures from `@parameter` return HTTP 422.

## Online prediction

Likely, the main way you would like to expose your model is through an HTTP API. [@GreatAI.create][great_ai.GreatAI.create] scaffolds many REST API endpoints for your model and creates a [FastAPI](https://fastapi.tiangolo.com/){ target=_blank } app available under [GreatAI.app][great_ai.GreatAI]. This can be served using [uvicorn](https://www.uvicorn.org/){ target=_blank } or any other [ASGI server](https://asgi.readthedocs.io/en/latest/){ target=_blank }.

Since most ML code lives in [Jupyter](https://jupyter.org/){ target=_blank } notebooks, therefore, deploying a notebook containing the inference function is supported. To achieve this, `uvicorn` is wrapped by the `great-ai` command-line utility, which — among others — takes care of feeding a notebook into `uvicorn`. It also supports auto-reloading.

### In development

```sh
great-ai greeter.py
```

!!! success
    Your model is accessible at [localhost:6060](http://127.0.0.1:6060){ target=_blank }.

Some configuration options are also supported.

```sh
great-ai greeter.py --port 8000 --host 127.0.0.1 --timeout_keep_alive 10
```
??? note "More options"
    For more options (but no Notebook support), simply use [uvicorn](https://www.uvicorn.org/){ target=_blank } for starting your app (available at `greeter.app`).

### In production

There are three main approaches for deploying a GreatAI service.

#### Manual deployment

The app is run in *production-mode* if the value of the `ENVIRONMENT` environment variable is set to `production`.

```sh
ENVIRONMENT=production great-ai greeter.py
```

Simply run `ENVIRONMENT=production great-ai deploy.ipynb` in the command-line of a production machine.
> This is the crudest approach; however, it might be fitting for some contexts.

#### Containerised deployment

Run the notebook directly in a container or create a service for it using your favourite container orchestrator.

```sh
docker run -p 6060:6060 --volume `pwd`:/app --rm \
  git.schmelczer.dev/andras/great-ai:latest deploy.ipynb
```
> You can replace ``pwd`` with the path to your code's folder.

#### Use a Platform-as-a-Service

Similar to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as [AWS ECS](https://aws.amazon.com/ecs/){ target=_blank }, [DO App platform](https://www.digitalocean.com/products/app-platform){ target=_blank }, [MLEM](https://mlem.ai/){ target=_blank }, [Streamlit](https://streamlit.io/){ target=_blank }) that take a Docker image as a source and handle the rest of the deployment.

To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as [PyTorch](https://pytorch.org/){ target=_blank } or [TensorFlow](https://www.tensorflow.org/){ target=_blank }.

```Dockerfile
# syntax=docker/dockerfile:1
FROM git.schmelczer.dev/andras/great-ai:latest

COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

# Remove this block if you don't have a requirements.txt
COPY requirements.txt ./   
RUN uv pip install --system --no-cache --requirement requirements.txt

# Cache an S3 model using credentials available only during this RUN.
# Remove this step if your application does not use a remote model.
RUN --mount=type=secret,id=s3,target=/run/secrets/s3.ini,required=true \
    large-file --backend s3 --secrets /run/secrets/s3.ini --cache my-domain-predictor

# Add your application code to the image
COPY . .

# The default ENTRYPOINT is great-ai; specify its argument using CMD
CMD ["deploy.ipynb"]
```

Create an `s3.ini` with the S3 keys described in the [configuration guide](https://great-ai.schmelczer.dev/how-to-guides/configure-service/#s3-compatible). Use concrete values in this build configuration: `ENV:` references are resolved inside the build container, which does not inherit your shell's environment. Replace `my-domain-predictor` with the key used when saving your model.

Keep credentials and any local model cache out of `COPY . .` by adding these entries to your project's `.dockerignore`:

```dockerignore
s3.ini
mongo.ini
.env
.cache
```

Build with [Docker BuildKit secrets](https://docs.docker.com/build/building/secrets/). This example disables Docker's layer cache so that the `large-file --cache` step runs and downloads the model into the image:

```sh
DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain \
  --secret id=s3,src=s3.ini --tag my-great-ai-service .
```

The secret mount is removed after the `RUN` instruction; the downloaded model cache remains in the image. Supply any configuration needed by the application separately at runtime. For example, an application that auto-configures S3 from `s3.ini` can use a read-only bind mount:

```sh
docker run --rm -p 6060:6060 \
  --mount type=bind,source="$(pwd)/s3.ini",target=/app/s3.ini,readonly \
  my-great-ai-service
```

## Batch prediction

Processing larger amounts of data on a single machine is made easy by the [GreatAI][great_ai.GreatAI]'s [process_batch][great_ai.GreatAI.process_batch] method. This relies on multiprocessing ([parallel_map][great_ai.utilities.parallel_map.parallel_map.parallel_map]) to take full advantage of all available CPU-cores.

```python
>>> greeter.process_batch(['Alice', 'Bob'])
[Trace[str]({'created': '2022-07-11T14:36:37.119183',
   'exception': None,
   'feedback': None,
   'logged_values': {'arg:your_name:length': 5, 'arg:your_name:value': 'Alice'},
   'models': [],
   'original_execution_time_ms':  0.1251,
   'output': 'Hi Alice!',
   'tags': ['greeter', 'online', 'development'],
   'trace_id': '90ffa15f-e839-41c4-8e7a-3211168bc138'}),
 Trace[str]({'created': '2022-07-11T14:36:37.166659',
   'exception': None,
   'feedback': None,
   'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
   'models': [],
   'original_execution_time_ms':  0.0571,
   'output': 'Hi Bob!',
   'tags': ['greeter', 'online', 'development'],
   'trace_id': 'f48e94c7-0815-48b3-a864-41349d3dae84'})]
```