Using Release Checklists

For every model version that will be released, whether to create a new service or replace another version already in production, the release carries some level of risk. To minimize any risk during deployment and speed up a well-governed process, we have created Release Checklists (also known as Governance Checklists) within the Verta platform.

Platform admins can customize the Verta Release Checklist template, create their own templates, and set them up for models. This makes it easier for everyone involved in launching and releasing a model, from data scientists and engineers to governance team members.

Here is a step-by-step guide on how to create a release readiness checklist:

Note: This version of release checklist is only available for customers using our new permissions system.

Step 1: Define the Release Checklist

To create a release readiness checklist, the first step is to define the release criteria. These should include the minimum requirements that must be met before any model can be released. Examples of release criteria include:

• Conducting a vulnerability scan and ensuring the model has no critical vulnerabilities.

• Creating model documentation and updating the README.

• Filing any engineering tickets and recording the JIRA Ticket ID in Verta.

• Coordinating a release timeline with Product Managers.

• And so on

Keep in mind that these checklists apply to all model versions, so consider the repeat activities that must be confirmed for each release.

Step 2: Admins, build the Release Checklist into Verta

On the Organization Administration page, you will find a new tab called “Governance”. The ability to edit the default checklist or create your own is available on this page.

To create a new custom checklist, select the “Create Checklist” button to bring up this configuration:

Each checklist must include:

• A visible name for all organization team members

• At least one item to check off or include in the checklist

• For now, all items are checkboxes. Other types of automations and governance rules are on our roadmap. Reach out if you would like early access.

Using the UI, give each checklist item a name that is visible to everyone in your organization.

Optionally, you may also:

1. Provide a description of what must be completed for your organization members to reference.

2. Mark the item as a hard requirement, which will enforce a more stringent progress bar on the checklist for each RMV.

3. Ask your team members to provide evidence, which gives them a text field to record any documents or notes that need to be referenced. For example, a Service Now ticket ID that corresponds to a request.

Create a checklist with 5 to 8 items to make it easy for your team to complete governance tasks. Add new items to the checklist by clicking "Add Checklist Item".

Step 3: Team members complete checklists to release models

You can view your organization's release checklists on the "Govern" tab. To locate them, navigate to a model version and click on the "Govern" tab. This tab will display your readiness for release as a percentage of progress on the checklists. If you have multiple checklists for various model stages or model risk levels, you can filter them based on the stage or custom attributes. Any checklist that is ready to be actioned will be displayed in an active/editable state. If the model has not progressed to a specific stage, the associated checklist will be in a disabled state.

You can also see a version of release checklist readiness by stage on the "Release Checklist" widget in a model version's "Release" tab.

The way we calculate the progress bar will vary depending on how many checklists and hard requirements you have. But no matter what, if your team checks off everything on the list, it means the model is ready for release.

Step 4: Reviewing and Auditing Release Checklists

At any time, you might have to get ready for an audit or give an account of how closely you're following your model governance processes. The Verta platform will keep track of all the items on your checklist that you've checked off, and it will show you who did it and when in the user interface.

For a more detailed record, platform logs are also available. If you need assistance accessing this information, please contact us.

Additional Checklist Controls

You may choose to have a checklist specific to a stage change. This allows you to have lenient guidelines for moving to Dev, and get more strict as models advance to production. Configure this per-checklist underneath the checklist name. Each checklist may only affect one stage change.

Set controls on who may complete each items on your checklists. Using user groups, you may restrict access to certain items based on the role of the person who is looking at the model version. For example, if you have an item saying "Complete Legal Review" you could restrict only users in group "Lawyers" to complete it. You can set one approved user group per checklist item, and use different user groups for other items in the checklist if needed.

Additionally, you have the option to create checklists specific to custom attribute values. For instance, you can establish a custom attribute field named "Model Risk Level" with values such as High, Medium, and Low, and then create distinct policies or checklists for models associated with different risk levels.

We want to hear from you

We're happy to let you know that Verta platform now has governance/release checklists. We're also planning to add more features to these checklists like automation, user access controls, setting model risk levels, and creating different checklists based on specific RMV needs.

For a sneak peek of these upcoming features, early access, and a chance to give us feedback, please contact your Verta point of contact.