Model Data Logging - Quick User Guide

Please contact Verta at [e-mail] to set up model data logging in your system.

Quick guide to logging data during model predictions.

Verta’s Model Data Logging capability allows users to log arbitrary key-value pairs, with the key being a string and the value being any JSON-compatible object, during model predictions and have those logs stored in a data-lake compatible format.

See the complete, comprehensive guide HERE

Basics

• The verta.runtime.log() python client API is inserted within the scope of a model's predict() method with the format verta.runtime.log('alphanumeric_string', <Any JSON Compatible object>)

• Logs are collected during the prediction and written to S3 after results are returned to the requestor.

• prediction_id is logged by default.

Best Practices

• Keep your logs as simple as possible with individual key-value pairs.

• Don't share keys across models unless they are the same data type.

---

The Code

from verta.registry import VertaModelBase, verify_io
from verta import runtime

class LoudEcho(VertaModelBase):
    """ Takes a string and makes it LOUD!!! """
    def __init__(self, artifacts=None):
        pass
            
    @verify_io
    def predict(self, input: str) -> str:
        runtime.log('model_input', input)  # log pre-processing data
        echo: str = input.upper() + '!!!'
        runtime.log('model_output', echo)  # log post-processing data
        return echo
    # Logs are finalized on exiting the predict function

Given a prediction:

client = verta.Client()                                               # create client connection
endpoint = client.get_or_create_endpoint('loud_echo_endpoint_name')   # fetch relevant endpoint
deployed_model = endpoint.get_deployed_model()                        # fetch model
deployed_model.predict('roar')                                        # make a prediction

Output:

"ROAR!!!"

Resulting log file will contain:

{
  "verta_prediction_id": "3c609a11-c7cd-45b0-8ba7-8789b17153f0",
  "model_input": "roar",
  "model_output": "ROAR!!!"
}

Locally inspect logs produced by your model by wrapping calls to predict() inside an instance of verta.runtime.context() and printing the logs.

with runtime.context() as ctx:
    result = LoudEcho().predict('yell')
print(ctx.logs())  # print the logs to see how they would be stored

Output:

{'model_input': 'yell', 'model_output': 'YELL!!!'}

Note: verta_prediction_id is not included in the sample output as it is only logged when an actual prediction is made against a live model.

---

Viewing the Logs

Logs are stored as JSON objects in S3 and should be immediately available for access at the following location:

s3://vertaai-model-data-logging/logs/date=YYYY-MM-DD/endpoint_id=XXX/<prediction_id>.json

Setup AWS Athena for querying results.