How to capture training loss over time?
Observations are recurring metadata that are repeatedly measured over time, such as batch losses over an epoch or memory usage. Observation charts are generated for each experiment run.
Here is the example with a Random Forest with Grid Search (XGBoost) that showcases how to capture observations:
#Prepare Data
data = datasets.load_wine()
X = data['data']
y = data['target']
dtrain = xgb.DMatrix(X, label=y)
df = pd.DataFrame(np.hstack((X, y.reshape(-1, 1))),
columns=data['feature_names'] + ['species'])
df.head()
#Prepare Hyperparameters
grid = model_selection.ParameterGrid({
'eta': [0.5, 0.7],
'max_depth': [1, 2, 3],
'num_class': [10],
})
#Log experiment runs and observations
def run_experiment(hyperparams):
run = client.set_experiment_run()
# log hyperparameters
run.log_hyperparameters(hyperparams)
# run cross validation on hyperparameters
cv_history = xgb.cv(hyperparams, dtrain,
nfold=5,
metrics=("merror", "mlogloss"))
# log observations from each iteration
for _, iteration in cv_history.iterrows():
for obs, val in iteration.iteritems():
run.log_observation(obs, val)
# log error from final iteration
final_val_error = iteration['test-merror-mean']
run.log_metric("val_error", final_val_error)
print("{} Mean error: {:.4f}".format(hyperparams, final_val_error))
# NOTE: run_experiment() could also be defined in a module, and executed in parallel
for hyperparams in grid:
run_experiment(hyperparams)
Observations can be visualized in the web UI in experiment run detail view.
Last modified 1yr ago