Model Data Logging - Quick User Guide

Please contact Verta at [e-mail] to set up model data logging in your system.

Quick guide to logging data during model predictions.

Verta’s Model Data Logging capability allows users to log arbitrary key-value pairs, with the key being a string and the value being any JSON-compatible object, during model predictions and have those logs stored in a data-lake compatible format.

See the complete, comprehensive guide HERE

Basics

• The verta.runtime.log() python client API is inserted within the scope of a model's predict() method with the format verta.runtime.log('alphanumeric_string', <Any JSON Compatible object>)

• Logs are collected during the prediction and written to S3 after results are returned to the requestor.

---

The Code

from verta.registry import VertaModelBase, verify_io
from verta import runtime

class LoudEcho(VertaModelBase):
    """ Takes a string and makes it LOUD!!! """
    def __init__(self, artifacts=None):
        pass
            
    @verify_io
    def predict(self, input: str) -> str:
        runtime.log('model_input', input)  # log pre-processing data
        echo: str = input.upper() + '!!!'
        runtime.log('model_output', echo)  # log post-processing data
        return echo
    # Logs are finalized on exiting the predict function

Given a prediction:

client = verta.Client()                                               # create client connection
endpoint = client.get_or_create_endpoint('loud_echo_endpoint_name')   # fetch relevant endpoint
deployed_model = endpoint.get_deployed_model()                        # fetch model
deployed_model.predict('roar')                                        # make a prediction

Output:

"ROAR!!!"

Logs:

{
  "model_input": "roar",
  "model_output": "ROAR!!!"
}

Locally inspect logs produced by your model by wrapping calls to predict() inside an instance of verta.runtime.context() and printing the logs.

deployed_model = endpoint.get_deployed_model()                        # fetch model
with runtime.context() as ctx:
    result = LoudEcho().predict('yell')
print(ctx.logs())  # print the logs to see how they would be stored

Output:

{'model_input': 'yell', 'model_output': 'YELL!!!'}

---

Viewing the Logs

Logs are stored as JSON objects in S3 and should be immediately available for access.

Each log file is written twice:

• s3://BUCKET/PREFIX/by_endpoint/endpoint_id=NNN/prediction_id=XXX/key=YYY/value.json

• s3://BUCKET/PREFIX/by_time/key=YYY/endpoint_id=NNN/year=YYYY/month=MM/day=DD/prediction_id=XXX/value.json

Setup AWS Athena for querying results.