Model Data Logging and Querying

Please contact Verta at help@verta.ai to set up model data logging in your system.

Overview

In this guide we will focus on Model Data Logging functionality and how to log model inference data and intermediary data for downstream monitoring, querying, and debugging.

Verta’s Model Data Logging capability allows users to log arbitrary key-value pairs, with the key being a string and the value being any JSON-compatible object, during model predictions and have those logs stored in a data-lake compatible format.

Quick Start Guide

The benefits of having a data logging and query workflow include:

Production debugging of prediction requests and any intermediate data with robust search capabilities.
Model pipeline visibility - Log model inference, as well as data from pre and post-processing steps to gain visibility across your model inference pipeline.
Monitoring - Feed the data back to a monitoring system or a data-pipeline solution as needed.

How It Works

The verta.runtime.log() Python client API allows users to log model input, features, intermediate data, prediction information, or any complex data that is JSON compatible as key-value pairs.

Note that runtime.log() must always be called within the scope of a model's required predict() method.

Each time a prediction request is sent to the server, the model's predict() function is wrapped inside an instance of a context manager class. Once the predict() function has completed, any logs collected therein are written to storage. (Currently AWS S3 ) Writing of the logs to storage occurs after prediction results are returned to minimize added latency.

See the section below on Local Testing and Debugging for instructions on imitating the above behavior for local development.

The following data points are captured and included by default, and thus it is not necessary to add them into your logs:

prediction_id: This is a unique identifier for the prediction request. This can be a custom value provided via the prediction_id argument passed to the predict() or predict_with_id() functions, or a random UUID by default.
endpoint_id: This is the unique ID number associated with the endpoint called.
time of request: Year, month, and day the prediction request occurred.

Examples

Given below are some examples of model classes that log data during predictions.

Example 1. Basic Model

In this example, we demonstrate basic calls to runtime.log() within the predict() function of a model that simply alters and returns a string. Note that the call to runtime.log() inside the make_loud() function is valid only because make_loud() is called within the scope of predict().

from verta.registry import VertaModelBase, verify_io
from verta import runtime

class LoudEcho(VertaModelBase):
    """ Takes a string and makes it LOUD!!! """
    def __init__(self, artifacts):
        pass
            
    @verify_io
    def predict(self, input: str) -> str:
        runtime.log('model_input', input)  # log pre-processing data
        echo: str = self.make_loud(input)
        return echo

    def make_loud(self, input: str):
        loud_echo: str = input.upper() + '!!!'
        runtime.log('model_output', loud_echo)  # log post-processing data
        return loud_echo

Assuming your application makes a prediction against the above model like so:

client = verta.Client()                                               # create client connection
endpoint = client.get_or_create_endpoint('loud_echo_endpoint_name')   # fetch relevant endpoint
deployed_model = endpoint.get_deployed_model()                        # fetch model
deployed_model.predict('roar')                                        # make a prediction

Output:

"ROAR!!!"

The expected logs for this prediction would be:

{
  "model_input": "roar",
  "model_output": "ROAR!!!"
}

Example 2: Basic Model With Batch Input

Note that because logs are collected within the context of a single prediction request, repeated calls to runtime.log(key, value) using the same key within the scope of predict(), will raise an error to prevent overwriting existing data. In this example, logs for the whole batch are aggregated first then logged once under a single key.

See Example 3 for an example that uses unique key values per prediction input for a flatter data structure.

from verta.registry import VertaModelBase, verify_io
from verta import runtime
from typing import Any, Dict, List

class LoudEcho(VertaModelBase):
    """ Takes a list of strings and makes each LOUD!!! """
    def __init__(self, artifacts):
        pass
            
    @verify_io
    def predict(self, inputs: List[str]) -> List[str]:
        outputs: List[str] = []
        output_logs: List[Dict[str, Any]] = []
        for input in inputs:
            echo: str = input.upper() + '!!!'
            outputs.append(echo)
            output_logs.append({'input': input, 'output': echo})
        runtime.log('model_outputs', output_logs)  # write aggregated logs once under a single key
        return outputs

Assuming your application makes a prediction against the above model like so:

deployed_model.predict(['roar', 'howl'])  # abbreviated process from Example 1.

Output:

["ROAR!!!", "HOWL!!!"]

The expected logs for this prediction would be:

{
  "model_outputs": [
    {
      "input": "roar",
      "output": "ROAR!!!"
    },
    {
      "input": "howl",
      "output": "HOWL!!!"
    }
  ]
}

Example 3. Basic Model With Batch Input and Unique Keys

In this example, we unpack our batch of prediction inputs and use a unique identifier to ensure that our key passed to runtime.log() is unique. The advantage of this approach is that key itself becomes a custom dimension that can be used in queries of the logging data downstream.

from verta.registry import VertaModelBase, verify_io
from verta import runtime
from typing import Any, Dict, List

class LoudEcho(VertaModelBase):
    """ Takes a list of strings and makes each LOUD!!! """
    def __init__(self, artifacts):
        pass
            
    @verify_io
    def predict(self, inputs: List[Dict[str, Any]]) -> List[str]:
        outputs: List[str] = []
        output_logs: List[Dict[str, Any]] = []
        for input in inputs:
            echo: str = input['value'].upper() + '!!!'
            outputs.append(echo)
            runtime.log(
                f"prediction_{input['id']}",  # unique key per prediction value unpacked
                {'input': input, 'output': echo} 
            ) 
        return outputs

Assuming your application makes a prediction against the above model like so:

deployed_model.predict(  # abbreviated process from Example 1.
    [
        {
            'id': '0001',
            'value': 'roar',
        },
        {
            'id': '0002',
            'value': 'howl'
        }
    ]
 )

Output:

["ROAR!!!", "HOWL!!!"]

The expected logs for this prediction would be:

{
  "prediction_0001": {
    "input": "roar",
    "output": "ROAR!!!"
  },
  "prediction_0002": {
    "input": "howl",
    "output": "HOWL!!!"
  }
}

Custom Prediction IDs

When making predictions against a deployed model in Verta via the predict() or predict_with_id() functions, it is possible to set a custom value for the prediction ID via the prediction_id argument. This value is included by default in any logging data stored, making it another dimension by which you can query log data later.

For more info the default S3 paths and partitioning, see the Schema and Table Generation section below.

Local Testing and Debugging

When the predict() function of a model is called, it is wrapped inside a context manager class by default. It is this context manager that collects all the logs created when calls to runtime.log() are made. When the prediction request is complete, this context manager is closed and the aggregated logs are stored in an instance attribute. The value of that attribute gets written to storage.

While adding logging to your models you can easily imitate this behavior in order to inspect your logs during development by simply wrapping your calls to predict() or predict_with_id() inside an instance of verta.runtime.context() as in the example below.

This example demonstrates how to view and debug model logs locally.

from verta.registry import VertaModelBase, verify_io
from verta import runtime

# Given a model class like this one:
class LoudEcho(VertaModelBase):
    """ Takes a string and makes it LOUD!!! """
    def __init__(self, artifacts):
        pass
            
    @verify_io
    def predict(self, input: str) -> str:
        runtime.log('model_input', input)
        echo: str = input.upper() + '!!!'
        runtime.log('model_output', echo)
        return echo

# In order to inspect the logs generated by a prediction, nest 
# a direct call to predict() inside a verta.runtime.context() to 
# mimic the expected behavior of the model once deployed.
with runtime.context() as ctx:
    result = LoudEcho().predict('yell')
print(ctx.logs())

Output:

{'model_input': 'yell', 'model_output': 'YELL!!!'}

Querying Results

This section describes the process for configuring Athena tables to enable complex querying of your model logging data.

Requirements

AWS Permisions:
- S3
- Athena
- GLUE
Model logs in S3 (Data exists to be queried.)

Useful Documentation

Verta Python Client:

AWS Docs:

Setup Process

Amazon Athena Database

You will need an Athena database to hold your new tables. If you already have one configured, skip this step.

Access the AWS Glue console, and select the “Databases” option from the left-hand navigation pane. Click on “Add Database” in the top right corner.
Give your DB a name and optional description, then click “Create Database”.

Schema and Table Generation

Use AWS Glue crawlers to auto-generate schemas for your existing logs and create new tables.

When model logs are uploaded to S3, two copies of the same log file are saved in separate folders (paths) in order to enable the following access patterns:

Search by prediction_id:

s3://BUCKET/PREFIX/by_endpoint/endpoint_id=NNN/prediction_id=XXX/key=YYY/value.json

Search by time of prediction request:

s3://BUCKET/PREFIX/by_time/key=YYY/endpoint_id=NNN/year=YYYY/month=MM/day=DD/prediction_id=XXX/value.json

Enabling both use cases requires running through the setup process below twice to create a unique table for each. Unless you wish to segregate data from specific models or groups of models, these two tables should suffice for all you models. Depending on your anticipated use pattern, you may only need one. Also, consider your S3 retention policies and how far back in time your logging data should be searchable.

Access the AWS Athena console, and open the “Query Editor”
Select the appropriate database from the “Database” drop-down on the left-hand navigation pane.
In the “Tables and views” section click on the “Create” button and choose “AWS Glue Crawler” under the section header “Create a table from data source”.

Give the crawler a name and add any optional tags or descriptions. We suggest including “by_endpoint” or “by_time” in the name for clarity.
Use the default settings for crawler source type (“Data stores” and “Crawl all folders”) then click “Next”.

Leave the default settings for data store (S3), and you should not need to set up a connection (Logs should be in this account).
Under “Include path”, click on the folder icon to search for the existing S3 path to your Verta model logs. There you should see two options, “by_endpoint” and “by_time”.

Select only the option you are currently configuring, and click “Select”. After returning to the “Add a data store” menu, click “Next”.
For “Add another data store”, make sure “no” is selected, and click “Next”.
In the “Choose an IAM role” menu, you will need to select an existing role or create a new one based on your organization’s requirements. The default option of “Create an IAM role” will ensure that the crawler has the appropriate permissions for the S3 bucket chosen.
On the “Create a schedule for this crawler” menu, it is recommended you run the crawler periodically to adjust to any changes in logging schema resulting from users logging arbitrary key-value pairs. See AWS docs on crawler configuration for more info. Click "Next".
On the “Configure the crawler’s output” menu, select the appropriate database option from the “Database” drop-down. In the “Prefix added to tables” section put “verta_model_logs”, or your preferred prefix. Make any desired changes to the “Configuration options”, but otherwise leave defaults in place and click ”Next”.
Finally, review all the configuration options and click “Finish”.
Once the crawler has been created, you should wind up on the general “Crawlers” menu, where you should see the crawler you just created in the table.
Select the box next to the new crawler, and click the “Run crawler” button. The process may take a few minutes. Monitor the status of the crawler in the “State” column. When the crawler is run for the first time, the column for “Tables added” should read “1”.
Repeat this process for the other access pattern, if desired.

Athena Search

Query model logs via Athena.

If prompted, You may need to set up an S3 bucket for query results if one is not already configured.
In the Query Editor, select the Database you configured for Verta model logs.
You should now see two new tables under the “Tables” heading.

Perform a generic search in each table to validate your ability to query data and explore available columns:

SELECT *
FROM verta_model_logs_by_endpoint
LIMIT 10

SELECT *
FROM verta_model_logs_by_time
LIMIT 10

PreviousSecuring endpoints NextModel Data Logging - Quick User Guide

Last updated 1 year ago