Using Multiple Namespaces
Using Multiple Namespaces with Verta
What are Namespaces
Namespaces are a way to organize Kubernetes clusters into virtual sub-clusters. This is part of the functionality offered by Kubernetes and their documentation can be found here. This functionality allows system admins to isolate teams/projects from each other.
Any number of namespaces are supported within a cluster, each logically separated from others but with the ability to communicate with each other. Namespaces cannot be nested within each other. Any resource that exists within Kubernetes exists either in the default namespace or a namespace that is created by the cluster operator. Only nodes and persistent storage volumes exist outside of the namespace; these low-level resources are always visible to every namespace in the cluster. Verta supports using Kubernetes with the single default namespace or with custom multiple namespaces. Your specific configuration depends on the size of your team and needs - please work with your Verta team to determine if using Multiple Namespaces is appropriate. We will enable this functionality on your account by request.
Benefits and Risks of using Multiple Namespaces with Verta
Namespaces can be helpful when different teams or projects share a Kubernetes cluster.
Namespaces are a way to divide cluster resources between multiple users (via resource quota). If different teams or projects have isolated budgets (that is, reason to measure resource usage by team) then multiple namespaces enables that functionally. If different teams have differing service levels (some working on models that need ultra high availability and scaling, others not) then limited resources can be assigned correctly using multiple namespaces.
Namespaces also further isolate and de-risk sensitive projects sharing a single Kubernetes cluster and Verta instance.
Multiple namespaces are an advanced functionality of Kubenetes, and the Kubernetes documentation advises to only use them when necessary. Your ML Ops team and Dev Ops team (if separate) will need to work closely together to ensure the entire service stack is correctly configured for each new namespace. Common errors and how Verta handles them are listed below.
How Multiple Namespaces work with Verta
Verta allows for grouping and isolation of entities (models, endpoints, experiment tracking projects) into Workspaces. Workspaces function much like a shared folder in a file-sharing system, where the Organization Admin can grant custom permissions to each workspace by user group and role.
On workspace creation, the Organization Admin configures a namespace to correlate with the workspace. Each workspace maps to a single namespace and today that namespace cannot be modified after workspace creation. Workspaces and Namespaces are always 1:1.
How to Configure Multiple Namespaces
When using Verta with Multiple Namespaces, the Organization Admin can configure which namespaces each workspace’s entities should operate in. These can be configured when creating or editing a workspace within an organization:
The recommended order of operations for configuring Multiple Namespaces is to :
Determine you want to use Multiple Namespaces
Decide how many namespaces are required, and create them within your Kubernetes cluster. Your Kubernetes admin should be able to help with this.
Create the workspaces within Verta, and map which workspaces should use which Namespaces.
Ensure every workspace is mapped to a namespace, otherwise it will fall back to the default namespace
Note, prior to release of new permission system, this will be on the Organization config
Assign correct access to each Workspace/Org using Verta’s RBAC system.
From there, deployed models in workspaces will map to the correct namespace in your Kubernetes cluster.
In early 2023 we released a new hierarchy of how organizations self-organize within Verta. Each of our customers has a single “Organization” and can make any number of Workspaces. The organization maps to the Kubernetes cluster. Customers running their own dedicated infrastructure are able to create multiple organizations that map to multiple Kubernetes clusters - on a 1:1 basis. Prior to the new permissions system, please note that the “Organizations” you'd see in the Verta UI function essentially as workspaces.
Frequent Issues and Questions
Multiple Namespaces is Enabled, but a Verta Workspace does not have a configured Namespace
For Workspaces created before enabling Multiple Namespaces, they will use the default primary namespace. You may assign a namespace later.
New workspaces will not function (models will not be deployable) and a namespace must be assigned when the workspace is created. The workspace creation UI will enforce this.
Personal Workspaces in Verta are decommissioned. You cannot use a personal workspace with multiple namespaces. If, for any reason, a personal workspace exists and Multiple Namespaces are enabled, the workspace will use the default primary namespace.
If, for any reason, while an endpoint is being deployed and there is no namespace specified, it will deploy to the primary namespace
If the endpoint is being deployed to a namespace Verta is expecting but that does not exist, it will fail to deploy. This might happen if the namespace was never created by your Kubenetes cluster admin or was deleted. In this case, any endpoints will fail to deploy with an unspecified error.
To resolve, begin by confirming all expected namespaces exist in the cluster.
Check that Verta has been deployed and is running in all namespaces.
Do end users (data scientists) know which namespace their endpoint belongs to?
No. Data Scientists do not need to understand Kubernetes, namespaces or how the workspaces they are using map to them in order to use Verta.
Last updated
