Ingest ground truth

You can ingest ground truth and monitor performance metrics like accuracy, precision, confusion matrix etc.

The system supports the ability to send delayed ground truth.

Verta uses a unique UUID for each prediction in order to map a prediction with the corresponding ground truth.

A typical flow would be the following:

• Caller sends input data for prediction on a Verta endpoint

• The model makes a prediction and assigns a unique UUID for the prediction

• Ground truth can then b e registered with the system using the above UUID

• UUID and the actual label are required for the log_ground_truth function along with the prediction column name.

Here is a code sample:

def simulate_predictions(endpoint, deployed_model, input_data, ground_truth, col_name, ground_truth_delay): 
    # ground_truth_delay is delay in seconds between prediction & GT becoming available
    import time
    
    ids = []
    for i, row in input_data.iterrows():
        _id, _ = deployed_model.predict_with_id([row.tolist()])
        ids.append(_id)

    time.sleep(ground_truth_delay)
    
    id_and_gt = zip(ids, ground_truth)
    
    for t in id_and_gt:
        endpoint.log_ground_truth(t[0], [t[1]], col_name) # id, gt, prediction_col_name

deployed_model = endpoint.get_deployed_model()

simulate_predictions(endpoint, deployed_model, df_test.iloc[:100, :-1], df_test.iloc[:100, -1], ">50k", 10)

Note: If you are sending ground truth along with the prediction data, we recommend adding some 15 sec sleep time between the two so the system can process prediction data first.

For delayed ground truth the data appears immediately in the system and you can start seeing updated performance charts.