Skip to content

Configure

The NIM Microservices framework enables self-service deployment and management of microservices within an existing shared host Kubernetes cluster. In this guide, we will use a curated template for NIM Microservices available from the Template Catalog. The Org Admin in the user’s Rafay Org has the necessary privileges to share system templates from Rafay’s Catalog to specific or all projects, enabling teams to quickly deploy NIM-compliant microservices with minimal setup.

Note that administrators can also create and configure Custom environment templates for use cases outside the list supported out of the box in Rafay's Template Catalog.

Info

Please check Rafay's Public Roadmap or contact support for details on additional templates for the Template Catalog.

Prerequisites

A shared host Kubernetes cluster with GPUs is required to use this template. The host cluster must have the NIM Operator (v2.0.2) preinstalled onto the cluster. The cluster should also have a one or more storageclasses that will be used for the NIM Services and NIM Cache storage. The cluster should have an ingress controller installed with a wildcard DNS record pointing to the public IPs of the cluster. Along with the above infrastructure requirements, a valid NGC API key is required to pull models from.

Configure Template

When the template is executed by the Rafay agent operating behind the firewall, it will receive the configured template, the associated Infra as Code (IaC), credentials and policies from the Rafay Platform. It will then execute this code on the behalf of the user.

1. Specify Agent

Let's configure the template to be received and executed by the Rafay Agent we created in a prior step.

  • Under the "Agents" tab, click on "Add Agent"
  • Select the name of the agent you configured in the prior step
  • Ensure that the override configuration is specified to "Not Allowed" because we do not want the downstream users to be able to change this
  • Save your changes

Select Agent

2. Config Context

The config context will typically encapsulate credentials and environment variables required for the agent to perform its job. In this case, we will configure the Rafay Agent with credentials so that it can make programmatic (API) calls to the Rafay controller.

To get the Rafay API Key for the administrator user,

  • Navigate to "My Tools -> Manage Keys" and click on "New API Key".
  • Copy the API Key + Secret combination.

Info

Click here to learn more about API Key & Secret for programmatic access.

Now, we are ready to configure our agent's config context.

  • Under the "Config Contexts" tab in the environment template, edit "rafay-auth-config"
  • Expand "Environment Variables" and you should see three entries: API Key, Host Cluster API Key & Controller Endpoint
  • Click on Edit for Host Cluster API Key
  • Paste the API Key/Secret string from the above step into the value section
  • Select Override to "Not Allowed" to ensure none of the downstream users have visibility or access to the config context
  • Save & Continue

Config Context Step 1

Controller API Endpoint

For self hosted Rafay Controller deployments, the agent will need to be configured to point to its URL. In our Get Started Guide, we will be using the URL for Rafay's SaaS option.

  • Click on Edit for Controller Endpoint
  • Note that Rafay's SaaS Endpoint URL is already configured and can be updated if required
  • Select Override to "Not Allowed" to ensure none of the downstream users have visibility or access to the config context
  • Save & Continue

Controller Endpoint

  • Click Save

Config Context Step 3

Configure PaaS

Next, we will configure a custom PaaS service profile to allow self service users to deploy the template.

  • Navigate to PaaS Studio
  • Select "Service Profiles"
  • Select the project where the template was previously created
  • Click "New Service Profile"
  • Enter a name for the profile
  • Select the previously created template and version
  • Select "Custom" for the service type
  • Select "Yes" for "Will compute be auto-created"
  • Click Save & Continue

Create Profile

On the following screen, navigate to "Input Settings"

  • Deselect "Allow Override For All" and then click "Deselect All". This will prevent the user from changing these settings.
  • Select "Override" for the following variables:
    • cache_engine
    • enable_cache
    • env_vars
  • Update the input variable value for the following variables:
    • cache_storage_class_name
    • cluster_name
    • domain
    • model_info
    • model_name
    • ngc_api_key
    • node_selector
    • storage_class_name
  • Navigate to "Output Settings"
  • Click "Add Output"
  • Enter the name "Curl_Command" and label "Curl Command"
  • Enter the name "URL" and label "URL"
  • Click Save Changes

Create Profile

Next Step

At this time, the administrator is ready to open up this profile (SKU) for end users. Learn how end users can launch and use their NIM Microservices on Kubernetes.