Concepts

Verta Model Monitoring allows you to monitor any model, data, or ML pipeline. You can use out-of-box monitors or define and use your custom functions to monitor your models and close the loop with alerts, notifications, and automated actions.

Monitored Entity

A monitored entity is a reference name of the entity that you are monitoring. It could be a dataset, a model, a live end-point, or a pipeline. Monitored entities are created within a given workspace/organization. The data summaries are produced and analyzed in context to a monitored entity.

Profiler

A profiler is a function that computes statistics about your data. There are two types of profilers: pre-built profilers and custom profilers. Verta provides a set of pre-built profiler functions like MissingValuesProfiler, BinaryHistogramProfiler, ContinuousHistogramProfiler, etc that you can import and run directly. If you want to build custom profilers that are unique to your use case, we work with you to create the appropriate visualization module.

Summary

A summary is a piece of information about your data that is generated as the output of a profiler. For example, accuracy, a mean squared error, or a histogram generated from your data table column values. A summary logged over time can provide a time series view.

Summary Sample

A summary sample is an instance of a summary, which might be logged from a training epoch or a batch of inputs and outputs for a deployed model.

Summary Labels

Summary labels are key-values metadata attached to a summary sample. One summary sample can have many labels. Labels enable you to categorize your saved summaries, easily query, filter, and aggregate summaries with shared labels. For example, you can filter, aggregate, and visualize summaries based on user segments, release versions, batch pipeline job-id, data sources (e.g. training set, reference set etc.),and more. In a nutshell, labels provide the user with a lot of flexibility to perform rich filtering and aggregation.

Alerter

Alerter allows you to define rules to detect unexpected model behavior, data quality, and distribution and trigger actions when those conditions are met. Alerter is triggered periodically and talks with the Verta API to fetch information about summaries and identify if they look wrong.

Alerter Types

You can define different alerter types based on the type of data and evaluation criteria.

  • A fixed alerter compares summary samples against a fixed numerical threshold

  • A range alerter compared summary samples against a range with upper and lower bounds

  • A reference alerter compares a sample and a reference value against a defined threshold

Notification Channel

The notification channel lets you define the channel(s) for sending notifications when the conditions of an alerting rule are met. An example of a notification channel is a slack webhook URL that sends alerts to a Slack channel. You can attach one or more notification channels to a given alerter.