Verta Model Monitoring lets you monitor drift, outlier and model performance metrics.
Get started with monitoring:
Create a Registered Model and Registered Model Versions (RMVs) to it.
registered_model = client.get_or_create_registered_model(name="census-model")
model_version = registered_model.create_standard_model(
name = "v1",
model_cls = CensusIncomeClassifier,
model_api = ModelAPI(X_train, Y_train_with_confidence),
environment = Python(requirements=["scikit-learn"]),
artifacts = artifacts_dict
Upload your reference data as an artifact in your Registered Model Version. This helps facilitate downstream drift monitoring against this reference set.
Deploy an endpoint with the model version. When an endpoint is deployed, the monitored model automatically appears in the Monitoring list view in webapp.
endpoint = client.get_or_create_endpoint("Census")
Start sending input data for prediction. Once the data has been sent to the system, you can navigate to the webapp to view dashboards.
deployed_model = endpoint.get_deployed_model()
id,_ = deployed_model.predict_with_id(input_feature)
Drift dashboard in webapp.
Ingest ground truth and the system will start computing performance metrics like accuracy, precision, confusion matrix etc.
endpoint.log_ground_truth(id, label, "output-class") # id, gt, prediction_col_name
Performance dashboard in webapp.