Skip to content

Day 2

Certain day 2 operations are needed in order to keep the environment up to date. The below information outlines how to update specific areas of the environment.

Check Web App Certificate Expiration

Certificates for the web app will have an expiration date. This should be monitored for expiration so that they can be updated in order to not impact user access.

  • In a Chrome web browser, navigate to your MLOps URL
  • Click the View Site Information icon in the address bar
  • Expand Connection is Secure
  • Click to expand the certificate details
  • You will then see the certificate details and expiration date

Certificate

Update Certificate

Users are recommended to refresh their certificates before expiration.

First, the config context containing the certificate will need to be updated.

  • Navigate to Environments -> Config Contexts
  • Click on the existing dns-certs context name
  • Select Input Variables
  • Populate the domain certificate and key with the updated, base64 encoded values
  • Ensure that you select the sensitive checkbox for both fields
  • Click Save

Next, the environment will need to be updated to use the new certificate values within the config context.

  • Navigate to Environments -> Environments
  • Click on the name of the environment under My Environments
  • Click Deploy

Redeploying the environment with the certificate change will only update the certificates in the environment. Users will not be impacted.

Once the deployment is finished, the user can optionally restart the ingress gateway deployment on the cluster for the certificates to take effect immediately.

  • Navigate to Infrastructure -> Clusters
  • Click Resources on the MLOps cluster
  • Click Deployments
  • Select istio-system from the namespace dropdown
  • Click the actions button on the istio-ingressgateway deployment and select Restart

Add GPU Nodes

If you are looking to deploy GPU nodes on day 2, follow the below steps.

  • Navigate to Environments -> Environments
  • Click on the name of the environment under My Environments
  • Click Edit
  • Click Show optional parameters
  • Click Add Optional Parameter
  • Select gpu_nodepools
  • Enter one or more node pools into the variable value

Below is an example with multiple node pools:

[{
  name = "gpu-nodepool1"
  size = 1
  machine_config = {
    boot_disk_size = 100
    boot_disk_type = "pd-standard"
    machine_type   = "n1-standard-4"
  }
  gpu_config = {
    type           = "nvidia-tesla-t4"
    count          = 1
    driver_version = "LATEST"
  }
},
{
  name = "gpu-nodepool2"
  size = 2
  machine_config = {
    boot_disk_size = 100
    boot_disk_type = "pd-standard"
    machine_type   = "n1-standard-4"
  }
  gpu_config = {
    type           = "nvidia-tesla-t4"
    count          = 1
    driver_version = "LATEST"
  }
}]
  • Click Save & Deploy