This is a reference design and implementation of an environment template to provision self service developer environments for Generative AI running on Amazon EKS with a LLM from Amazon Bedrock. This reference design is based on Rafay's Kubernetes Manager and Environment Manager.
Design & Architecture¶
This design assumes a single, shared EKS cluster with Kubernetes namespace(s) that will be created and assigned to the developer for their Generative AI applications. The available options are:
- Namespace(s) per developer
- Project per application team
In this design, we use a "namespace per Gen AI developer" on the shared EKS Cluster.
The platform team is expected to provision an EKS cluster in a Rafay project e.g. "genai". The EKS cluster will be shared with developers either via "Kubernetes Namespaces" or "Rafay Workspaces".
sequenceDiagram participant plat as Platform Team participant rafay as Environment Manager participant csp as AWS participant idp as Identity Provider plat->>csp: Enable Amazon Bedrock in AWS Account plat->>rafay: Setup Gen AI Template for EKS rafay->>rafay: Create "genai" Project rafay->>csp: Provision EKS Cluster rafay->>idp: Integrate Env Manager <br> with Corporate IdP rafay-->>plat: Setup Complete
Request for Environment¶
When a developer requests for a Gen AI environment (based on Kubernetes), a series of automated workflows need to be performed to fulfil this request. The sequence diagram below describes the high level steps.
Although the recommended workflow assumes and recommends using an Integration with an Identity Provider (IdP) to provide a Single Sign On (SSO) experience, organizations can also use locally managed users.
sequenceDiagram participant plat as Platform Engineer participant rafay as Rafay participant csp as EKS Cluster participant idp as Identity Provider participant dev as Developer plat->>rafay: New Environment <br>for Developer rect rgba(0, 0, 255, .1) rafay-->>csp: Create k8s namespace w/ Resource Quota rafay-->>csp: Create IRSA <br>(API Access to Bedrock) opt SSO via Identity Provider rafay-->>idp: Create Group for k8s namespace rafay-->>idp: Add developer to Group end rafay->>rafay: Create Group in Rafay rafay->>rafay: Add NS Admin <br>RBAC to Group rafay-->>csp: Deploy GenAI App-1 <br> via Kubectl rafay-->>csp: Deploy GenAI App-2 <br> as Workload end rafay->>plat: Environment Ready plat->>dev: Environment Ready for Use by Developer dev-->>rafay: Access/Use Environment
- You have access to a AWS Account
- You have access to a Mac or Linux machine
- You have a Git client on your machine that is setup for push/pull
- You have Docker installed on your machine