AI Assisted Model Documentation

Introduction to AI Assisted Model Documentation

Model cards (a key component of a Model Catalog) rely on having linked or accessible detailed model documentation to aid all the users who may need to self-serve a question about the model. This could be customer success team members investigating a user complaint, engineers working on a model integration or leadership looking for proof that bias was mitigated to the best of everyone’s ability.

In addition to the programmatic model metadata collected when a model is registered into Catalog, most of our customers prioritize adding a human element to their model documentation - context, model use cases, lineage information and so on.

AI Assisted Documentation is intended to:

1. Reduce the burden on data scientists to write mountains of documentation

2. Provide a first draft for a human to edit and ensure the documentation is correct

By using AI Assisted Documentation data scientists will be able to write more detailed and accurate documentation in less time.

This feature is in BETA meaning we would love for you to try it but caution that it may sometimes behave oddly. Please provide any feedback you have to [e-mail] and keep an eye out for an email from our product team after using this feature.

How to use AI-Assisted Documentation

Editing Documentation

Documentation can be found for models and versions on the “Documentation” page of Catalog. When looking at a model, navigation has been provided to let you quickly access and edit the documentation for all versions so you can quickly compare and update everything. When editing the documentation for a single version you’ll only have access to that version.

🔆 The download button in the toolbar generates a markdown file for your documentation which you can ingest directly into Github or another tool you have.

Edit documentation by clicking the EDIT button.

Our new documentation editor allows for every team member to help maintain documentation as long as they are familiar with text editors. A toolbar with standard formatting widgets is available.

You may also format any selected text with the hovering toolbar.

Links, formatted text and code blocks can be embedded and edited easily:

Using the AI Prompts

AI Assisted Documentation works by leveraging a large-language-model (LLM) provided by OpenAI (the people behind ChatGPT). Verta has fine tuned a selection of “prompts” that will return an ideal response from the LLM.

Our philosophy for creating these prompts was:

• We will not allow the 3rd-party language model to invent information about your model

  • Instead, we do a large amount of rewriting and formatting to let you write quickly, briefly and simple and result in legible and clear documentation

• We do not apply unnecessary AI

  • For example, in the case of the model API prompt, we simply use an LLM to read the model API you already registered and format it for your use; no additional copy is added.

• Human review is mandatory

  • In all cases, for all prompts, whoever is editing the documentation must copy the result of the model into the documentation and hit save. This prevents accidental overwriting of human context with possible inaccurate AI predictions

With that said, using the prompts is a quick and easy way to build out comprehensive documentation. Each prompt will show instructions for what it expects as well as a brief example:

Let’s try it out - notice how short and misspelled the human entry is?

Despite the rushed nature of the human input, the model produced a useful and accurate few sentences about this model's purpose. If I don’t like the result I can edit my input and rewrite again - a stack of responses will accumulate that you can then flip through. When you’re happy enough with a response, hover over the results to see the copy icon:

Paste the result into the text editor and give it a final review - of course, you may edit the text any way you see fit. When you’re happy with your documentation hit save. Your docs will be written and the history of the AI prompts will be cleared.