Querying model metadata

For revisiting past experiment runs, the Verta API offers the ability to find runs based on their properties and logged values.

To start, you can obtain a collection of experiment runs under a project or an experiment using their expt_runs attribute:

proj = client.set_project("Project Banana")
proj.expt_runs
# <ExperimentRuns containing 24 runs>
expt = client.set_project("Experiment Coconut")
expt.expt_runs
# <ExperimentRuns containing 8 runs>

These are ExperimentRuns objects, which have a find() method for filtering their contents:

expt.expt_runs
# <ExperimentRuns containing 8 runs>
expt.expt_runs.find("metrics.acc > .95")
# <ExperimentRuns containing 3 runs>
expt.expt_runs.find("hyperparameters.dropout < .8")
# <ExperimentRuns containing 4 runs>
expt.expt_runs.find("hyperparameters.optimizer == 'adam'")
# <ExperimentRuns containing 6 runs>

Multiple queries can be specified together, which returns the intersection of their results.

expt.expt_runs.find([
    "metrics.acc > .95",
    "hyperparameters.dropout < .8",
])
# <ExperimentRuns containing 2 runs>

Query syntax

Queries are Python strings that follow this syntax:

"<field>.<key> <operator> <value>"

For example, lets say I have several runs, and I've been logging each multi-layer perceptron's hidden layer size as a hyperparameter with the key "hidden_size".

To find all runs with a hidden size greater than 512, the query would look like this:

"hyperparameters.hidden_size > 512"

If the value is a string, it should be surrounded by quotes. As examples:

"hyperparameters.optimizer == 'adam'"
"attributes.team == \"sales\""

Fields

The fields currently supported by the client are:

• id

• project_id

• experiment_id

• name

• date_created

• attributes

• hyperparameters

• metrics

date_created is represented as a Unix timestamp, in milliseconds.

Some fields inherently do not have keys, such as id and name, in which case their queries are even more straightforward:

"<field> <operator> <value>"

For example:

"name == 'Run Dragonfruit'"

Operators

The operators currently supported by the client are:

• ==

• !=

• >

• >=

• <

• <=

For string values, it is recommended to only use == and !=.