How to capture training loss over time?

Observations are recurring metadata that are repeatedly measured over time, such as batch losses over an epoch or memory usage. Observation charts are generated for each experiment run.

Here is the example with a Random Forest with Grid Search (XGBoost) that showcases how to capture observations:

#Prepare Data

data = datasets.load_wine()

X = data['data']
y = data['target']

dtrain = xgb.DMatrix(X, label=y)

df = pd.DataFrame(np.hstack((X, y.reshape(-1, 1))),
                  columns=data['feature_names'] + ['species'])

df.head()

#Prepare Hyperparameters

grid = model_selection.ParameterGrid({
    'eta': [0.5, 0.7],
    'max_depth': [1, 2, 3],
    'num_class': [10],
})

#Log experiment runs and observations

def run_experiment(hyperparams):
    run = client.set_experiment_run()
    
    # log hyperparameters
    run.log_hyperparameters(hyperparams)
    
    # run cross validation on hyperparameters
    cv_history = xgb.cv(hyperparams, dtrain,
                        nfold=5,
                        metrics=("merror", "mlogloss"))

    # log observations from each iteration
    for _, iteration in cv_history.iterrows():
        for obs, val in iteration.iteritems():
            run.log_observation(obs, val)
            
    # log error from final iteration
    final_val_error = iteration['test-merror-mean']
    run.log_metric("val_error", final_val_error)
    print("{} Mean error: {:.4f}".format(hyperparams, final_val_error))
    
# NOTE: run_experiment() could also be defined in a module, and executed in parallel
for hyperparams in grid:
    run_experiment(hyperparams)

Observations can be visualized in the web UI in experiment run detail view.