Verta
Search…
Concepts

Overview

Verta Model Monitoring allows you to monitor any model, data, or ML pipeline. You can use out-of-box monitors or define and use your custom functions to monitor your models and close the loop with alerts, notifications, and automated actions.
The platform offers 2 types of model monitors:

Custom

Fully flexible and customizable monitoring capability. Supports batch, live and model pipeline monitoring running on any platform. In order to use the custom monitoring capabilities you need to follow these steps:
    Ingest I/P & O/P data
    Configure data profiles / statistical summaries
    Query, aggregate and visualize time series summaries
    Define alerts, get alerted and perform root cause analysis

Automated

For live endpoints deployed on Verta inference service, model monitoring is fully automated (currently available for tabular data). This capability deeply integrates our deployment and registry modules. In order to use the automated monitoring capabilities you need to follow these steps:
    Log reference data when you are registering a model version in registry
    When you deploy an endpoint statistical summaries (histograms, missing values etc.) are automatically logged
    Access aggregates and time series views in dashboard
    Alerts and automatically created with system defined thresholds for drift detection

Concepts

Monitored Entity

A monitored entity is a reference name of the entity that you are monitoring. It could be a dataset, a model, a live end-point, or a pipeline. Monitored entities are created within a given workspace/organization. The data summaries are produced and analyzed in context to a monitored entity.

Profiler

A profiler is a function that computes statistics about your data. There are two types of profilers: pre-built profilers and custom profilers. Verta provides a set of pre-built profiler functions like MissingValuesProfiler, BinaryHistogramProfiler, ContinuousHistogramProfiler, etc that you can import and run directly. If you want to build custom profilers that are unique to your use case, we work with you to create the appropriate visualization module.

Summary

A summary is a piece of information about your data that is generated as the output of a profiler. For example, accuracy, a mean squared error, or a histogram generated from your data table column values. A summary logged over time can provide a time series view.

Summary Sample

A summary sample is an instance of a summary, which might be logged from a training epoch or a batch of inputs and outputs for a deployed model.

Summary Labels

Summary labels are key-values metadata attached to a summary sample. One summary sample can have many labels. Labels enable you to categorize your saved summaries, easily query, filter, and aggregate summaries with shared labels. For example, you can filter, aggregate, and visualize summaries based on user segments, release versions, batch pipeline job-id, data sources (e.g. training set, reference set etc.),and more. In a nutshell, labels provide the user with a lot of flexibility to perform rich filtering and aggregation.

Alerter

Alerter allows you to define rules to detect unexpected model behavior, data quality, and distribution and trigger actions when those conditions are met. Alerter is triggered periodically and talks with the Verta API to fetch information about summaries and identify if they look wrong.

Alerter Types

You can define different alerter types based on the type of data and evaluation criteria.
    A fixed alerter compares summary samples against a fixed numerical threshold
    A range alerter compared summary samples against a range with upper and lower bounds
    A reference alerter compares a sample and a reference value against a defined threshold

Notification Channel

The notification channel lets you define the channel(s) for sending notifications when the conditions of an alerting rule are met. An example of a notification channel is a slack webhook URL that sends alerts to a Slack channel. You can attach one or more notification channels to a given alerter.
Last modified 2mo ago