Ingest ground truth

Search…
Ingest ground truth
You can ingest ground truth and monitor performance metrics like accuracy, precision, confusion matrix etc.
The system supports the ability to send delayed ground truth.
Verta uses a unique UUID for each prediction in order to map a prediction with the corresponding ground truth.
A typical flow would be the following:
Caller sends input data for prediction on a Verta endpoint
The model makes a prediction and assigns a unique UUID for the prediction
Ground truth can then b e registered with the system using the above UUID
UUID and the actual label are required for the log_ground_truth function along with the prediction column name.
Here is a code sample:
def simulate_predictions(endpoint, deployed_model, input_data, ground_truth, col_name, ground_truth_delay): 
    # ground_truth_delay is delay in seconds between prediction & GT becoming available
    import time
    
    ids = []
    for i, row in input_data.iterrows():
        _id, _ = deployed_model.predict_with_id([row.tolist()])
        ids.append(_id)
​
    time.sleep(ground_truth_delay)
    
    id_and_gt = zip(ids, ground_truth)
    
    for t in id_and_gt:
        endpoint.log_ground_truth(t[0], [t[1]], col_name) # id, gt, prediction_col_name
deployed_model = endpoint.get_deployed_model()
simulate_predictions(endpoint, deployed_model, df_test.iloc[:100, :-1], df_test.iloc[:100, -1], ">50k", 10)
Note: If you are sending ground truth along with the prediction data, we recommend adding some 15 sec sleep time between the two so the system can process prediction data first.
For delayed ground truth the data appears immediately in the system and you can start seeing updated performance charts.
Log reference data
Model performance
Last modified 2mo ago
Copy link