Skip to content

Overview

This is a reference design and implementation of an environment template to provision self service developer environments for Generative AI running on Amazon EKS with a LLM from Amazon Bedrock. This reference design is based on Rafay's Kubernetes Manager and Environment Manager.

Source Code for Template

The Terraform source code and related assets are available in a public Git repo as shown below

Git Repo

Design & Architecture

This design assumes a single, shared EKS cluster with Kubernetes namespace(s) that will be created and assigned to the developer for their Generative AI applications. The available options are:

  • Namespace(s) per developer
  • Workspace
  • Project per application team

In this design, we use a "namespace per Gen AI developer" on the shared EKS Cluster.

Architecture

Initial Setup

The platform team is expected to provision an EKS cluster in a Rafay project e.g. "genai". The EKS cluster will be shared with developers either via "Kubernetes Namespaces" or "Rafay Workspaces". 

sequenceDiagram 
    participant plat as Platform Team 
    participant rafay as Environment Manager 
    participant csp as AWS 
    participant idp as Identity Provider 

    plat->>csp: Enable Amazon Bedrock in AWS Account 
    plat->>rafay: Setup Gen AI Template for EKS 
    rafay->>rafay: Create "genai" Project 
    rafay->>csp: Provision EKS Cluster 
    rafay->>idp: Integrate Env Manager <br> with Corporate IdP  
    rafay-->>plat: Setup Complete

Request for Environment

When a developer requests for a Gen AI environment (based on Kubernetes), a series of automated workflows need to be performed to fulfil this request. The sequence diagram below describes the high level steps.

Important

Although the recommended workflow assumes and recommends using an Integration with an Identity Provider (IdP) to provide a Single Sign On (SSO) experience, organizations can also use locally managed users. 

sequenceDiagram
    participant plat as Platform Engineer
    participant rafay as Rafay
    participant csp as EKS Cluster
    participant idp as Identity Provider 
    participant dev as Developer

    plat->>rafay: New Environment <br>for Developer
    rect rgba(0, 0, 255, .1)
    rafay-->>csp: Create k8s namespace w/ Resource Quota  
    rafay-->>csp: Create IRSA <br>(API Access to Bedrock)
    opt SSO via Identity Provider
        rafay-->>idp: Create Group for k8s namespace 
        rafay-->>idp: Add developer to Group 
    end
    rafay->>rafay: Create Group in Rafay
    rafay->>rafay: Add NS Admin <br>RBAC to Group
    rafay-->>csp: Deploy GenAI App-1 <br> via Kubectl 
    rafay-->>csp: Deploy GenAI App-2 <br> as Workload 
    rafay-->>csp: Deploy GenAI RAG App-3
    end
    rafay->>plat: Environment Ready
    plat->>dev: Environment Ready for Use by Developer
    dev-->>rafay: Access/Use Environment

Assumptions

  • You have access to a AWS Account
  • You have access to a Mac or Linux machine
  • You have a Git client on your machine that is setup for push/pull
  • You have Docker installed on your machine