Installation

Note

See the Self Hosted Controller v1.24 for the latest process.

Here are the detailed instructions for installation of Self Hosted Controller in EKS environments.

Before Installation¶

Create an instance/node with the specifications described in the requirements.
Create wildcard DNS entries for the controller domains mentioned in the requirements, and point their A record to node/load balancer IP addresses.
(Optional) Generate a wildcard certificate for the FQDN which is signed by a certificate authority. Alternatively, configure the controller to use self-signed certificates.

Download Controller Package¶

Download and install the controller installation package.

Click here to login and download the controller installation package to the instance.

Optionally, verify the package with MD5SUM or SHA256SUM.

https://rafay-airgap-controller.s3.us-west-2.amazonaws.com/1.5.w2/MD5SUM 
https://rafay-airgap-controller.s3.us-west-2.amazonaws.com/1.5.w2/SHA256SUM

From your home directory, untar the package using the command below
```
tar -xf rafay-controller-*
```

Example:

tar -xf rafay-controller-1.13-20-eks.tar.gz

Controller Config File¶

Update the controller config.yaml file to configure your deployment.

Copy the Config File¶

sudo mv ./radm/usr/bin/
cp -rp config.yaml-eks-tmpl config.yaml
vi config.yaml

Update the Config File¶

Customize the config.yaml. The following settings should be updated. Some settings do not apply to Amazon EKS environments.

Database

The following configuration settings are for the external-database feature.

Setting	Description
external-database	Enables using an external database.
host	The endpoint for the RDS database.
port	The port for the RDS database.

Example:

database:
    external-database: true
    sslMode: ""
    host: "controller-postgres.abc123.us-west-2.rds.amazonaws.com"
    port: "5432"

DNS

The following configuration settings are for the Amazon EKS dns feature.

Setting	Description
enabled	Enables using a DNS.
role_arn	The role ARN created for the external DNS.
hosted_zone_id	The hosted zone ID for the Route 53 domain.

Example:

dns:
    enabled: true
    eks:
        role_arn: "arn:aws:iam::123456789012:role/controller-DNS-role"
        hosted_zone_id: "A12345678BC12DE123FGH"

Monitoring

The following configuration settings are for the monitoring feature using AMP.

Setting	Description
enabled	Enables AMP monitoring.
ingest_role_arn	The ingest ARN created for the Prometheus ingest role.
query_role_arn	The query IAM role ARN created for the Prometheus query role.
region	The AWS region.
host	The Amazon Prometheus endpoint.
workspace_id	The Prometheus (AMP) workspace ID.

Example:

monitoring:
    integrations:
        amp:
            enabled: true
            ingest_role_arn: "arn:aws:iam::123456789012:role/controller-PrometheusIngest-role"
            query_role_arn: "arn:aws:iam::123456789012:role/controller-PrometheusQuery-role"
            region: "us-west-2"
            host: "aps-workspace.us-west-2.amazonaws.com"
            workspace_id: "ws-12345678-1234-5678-ab12-c123defhij4k"

Backup and Restore

The following configuration settings are for the Amazon EKS backup_restore feature.

Setting	Description
enabled	Enables the Backup feature.
restore	Enables the Restore feature.
schedule	Set the time interval for the backup.
bucketName	The name of the S3 bucket for storing all backup files.
retentionPeriod	The number of days to retain a backup file.
backupFolderName	The S3 bucket backup folder name. This setting is used when `restore` is set to true.
role_arn	Set the IAM role ARN that was created for the backup.

Example - Backup:

backup_restore:
    enabled: true
    restore: false
    schedule: "*0 0 * * *"
    bucketName: "controller-backup"
    retentionPeriod: "168h0m0s"
    backupFolderName: ""
    eks:
        role_arn: "arn:aws:iam::123456789012:role/controller-BackupRestore-role"

Repo

For the repo settings, modify the path to match the host operating system.

Example - Linux:

repo:
    infra-archive:
        path: /home/username/rafay-infra-centos.tar.gz
        dir:/tmp
    dep-archive:
    -   name: istio
        path: /home/username/istio-1.8.1-blueprint.tar.gz
    -   name: rafay-dep
        path: /home/username/rafay-dep.tar.gz
    app-archive:
    -   name: rafay-core
        path: /home/username/rafay-core.tar.gz
    cluster-archive:
    -   name: rafay-cluster
        path: /home/username/rafay-cluster.tar.gz

App-Config

The following configuration settings are for the app-config feature.

Setting	Description
generate-self-signed-certs	When enabled, the controller creates certificates for the deployment. When disabled, the `console-certificates` will be used.
console-certificates	Add the certificate and key for the domain. Use when `generate-self-signed-certs` is disabled.
super-user	The username and password for the OPS console login.
partner	Provide partner-related information, such as a company name, logo, and support email addresses.

Example:

app-config:
    generate-self-signed-certs: true
    console-certificates:
        certificate: ""
        key: ""
    super-user:
        user: admin@company.com
        password: change123
    partner:
        star-domain: example.company.com
        name: Company Cloud
        #logo: /logo.png
        product-name: Company Name

Override-Config

The following configuration settings are for the override-config features.

Setting	Description
global.external_lb	Enables using an external load balancer (lb).
global.lb_scheme	When `global.external_lb` is enabled, set this value to `internal`. When `global.external_lb` is disabled, set this value to `internet-facing`.
global.lb_subnet_ids	Add the subnet IDs.
global.minReplicaCount	The number of replicas for each resource.
global.acm_arn	Add the ACM certificate ARN for the domain.
global.cluster_name	Add the EKS cluster (controller) name.
global.use_instance_role	Enable when the controller uses its own IAM role for provisioning clusters.
global.edge.irsa_role_enabled	Enables IRSA.
global.edge.irsa_role_arn	Add the IRSA role ARN.
global.secrets.aws_account_id	Add for AWS IAM role based cluster provisioning. Also add `global.secrets.aws_access_key_id` and `global.secrets.aws_secret_access_key`.
global.secrets.aws_access_key_id	Secret must be encoded with Base64. Add if using AWS IAM role based cluster provisioning.
global.secrets.aws_secret_access_key	Secret must be encoded with Base64. Add if using AWS IAM role based cluster provisioning.
controller.serviceAccount.annotations	Add the EFS IAM role ARN.
storageClass.efsFilesystemId	Add the Filestore (EFS) ID.
karpenter.enabled	Enables Karpenter.
karpenter.serviceAccount.annotations	Add the Karpenter IAM role ARN.
karpenter.clusterName	Add the EKS cluster name.
karpenter.clusterEndpoint	Add the EKS cluster endpoint.
karpenter.aws.defaultInstanceProfile	Add the name of the Karpenter instance profile.
global.opensearch.enabled	Enables external OpenSearch.
global.opensearch.endpoint	Add the OpenSearch endpoint.
global.opensearch.user_name	Add the username for OpenSearch.
global.opensearch.user_password	Add the password for OpenSearch.
global.opensearch.region	Add the region where OpenSearch was created.
global.kinesis_firehose.delivery_stream	Add the Kinesis Firehose name.
global.kinesis_firehose.role_arn	Add the Kinesis IAM role ARN.
global.kinesis_firehose.ctrl_logs_ds	Add the controller logs Kinesis delivery stream name.
global.kinesis_firehose.ctrl_logs_ds_role_arn	Add the controller logs Kinesis IAM role ARN.

Example:

override-config:
    global.enable_hosted_dns_server: "false"
    global.external_lb: "false"
    global.lb_scheme: "internet-facing"
    global.lb_subnet_ids: "subnet-0abcd1ef2gh345i67 subnet-987z654yx3w210v9u subnet-6mn7o8pq9012r3s4t"
    global.minReplicaCount: "3"
    global.acm_arn: "arn:aws:acm:us-west-2:123456789012:certificate/123abcd4-5678-90e1-f2g3-h4i567jklmn8"
    global.cluster_name: "controller"
    global.use_instance_role: "true"
    global.edge.irsa_role_enabled: "true"
    global.edge.irsa_role_arn: "arn:aws:iam::123456789012:role/controller-cluster-irsa-role"
    global.secrets.aws_account_id: "0123456789"
    global.secrets.aws_access_key_id: "ZWRnZWRidXNlcg=="
    global.secrets.aws_secret_access_key: "ZWRnZWRidXNlcg=="
    controller.serviceAccount.annotations: "arn:aws:iam::123456789012:role/controller-efs-ROLE"
    storageClass.efsFilesystemId: "fs-1a23bc456d7efg890"
    karpenter.enabled: true
    karpenter.serviceAccount.annotations: "arn:aws:iam::123456789012:role/controller-KarepnterRole
    karpenter.clusterName: "eks-cluster"
    karpenter.clusterEndpoint: "https://1234567890.ab1.us-west-2.eks.amazonaws.com"
    karpenter.aws.defaultInstanceProfile: "KarpenterNodeInstanceProfile-controller"
    global.opensearch.enabled: "true"
    global.opensearch.endpoint: "search-companyopensearch-a1bcdef2ghijk34l5mn6opq7rs.us-west-2.es.amazonaws.com"
    global.opensearch.user_name: "username"
    global.opensearch.user_password: "change123"
    global.opensearch.region: "us-west-2"
    global.kinesis_firehose.delivery_stream: "controller-kinesis"
    global.kinesis_firehose.role_arn: "arn:aws:iam::123456789012:role/controller-KinesisRole"
    global.kinesis_firehose.ctrl_logs_ds: "controller-kinesislogs"
    global.kinesis_firehose.ctrl_logs_ds_role_arn: "arn:aws:iam::123456789012:role/controller-kinesislogs-role"

Initialize the Controller¶

Initialize the controller using the following command.
```
sudo radm init --config config.yaml
```

After initialization is complete, copy the admin config file to the home directory to access the kube controller API and CLI.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) -R $HOME/.kube

Install Controller Dependencies¶

Install the controller dependencies.

sudo radm dependency --config config.yaml

Installing Controller Apps¶

Install the controller application.

sudo radm application --config config.yaml

This will bring up all of the controller services. This process can take 20-30 minutes for all pods to be up and ready.

To confirm that all pods are in a running state, use the following command.

kubectl get pods -A

Accessing the Web Console¶

Try accessing the self hosted controller https://console. to verify that the installation was successful.

A screen appears similar to the image below when accessing the UI

EKS

Click the Sign Up link to create the first Organization of the self hosted controller
Register a new account for the organization as below screenshot

EKS

Try to login to this Organization with the newly registered account on the login screen

Upload Cluster Dependencies¶

Run the below command to enable support for Kubernetes cluster provisioning from the self hosted controller and upload dependencies for Kubernetes cluster provisioning to the controller.

sudo ./radm cluster --config config.yaml --kubeconfig <eks cluster config file>

Example:

sudo ./radm cluster --config config.yaml --kubeconfig eks-config.yaml

Setup EKS Cluster Provisioning¶

With the self hosted controller accessible through the console URL, use the following for setting up the controller for EKS cluster provisioning.

Provision EKS Clusters Using Controller Instance IAM¶

Use the controller instance IAM role to provision EKS clusters in the same AWS account as the controller.

The following parameter must be enabled in the config.yaml file to use this feature.

global.use_instance_role: true
If this feature is enabled after deploying the controller, rerun the RADM application command.

sudo radm application --config config.yaml

Create the following IAM policy for the controller EC2 instance to allow STS, PassRole, and CloudFormation access.

{
"Version": "2012-10-17",
"Statement": [
    {
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": "sts:*",
    "Resource": "*"
    },
    {
    "Sid": "cloudformation",
    "Effect": "Allow",
    "Action": [
        "cloudformation:*"
    ],
    "Resource": "*"
    },
    {
    "Sid": "iam",
    "Effect": "Allow",
    "Action": [
        "iam:PassRole"
    ],
    "Resource": "*"
    }
]

}

Create an IAM policy to use the provisioning EKS clusters. See IAM Policy for more information about the IAM policy.
After creating two policies, create a new IAM role for the controller EC2 instance to use for the EKS cluster provisioning. Choose use case EC2.

EKS

Under policies, choose the above created STS and EKS policies, and attach them to the role.

EKS

Provide the role name and create the IAM role.

EKS

After the role is created, edit the trust relationship of the IAM role for trusting the controller EC2 instances.

EKS

Edit the trust relationship of the IAM role by replacing the Principal with the controller EC2 instances Account ID and Instance ID.

Example: arn:aws:sts::<accountid>:assumed-role/aws:ec2-instance/<instance id>

EKS

Attach the IAM role to the EC2 instance of the controller. Now the controller is trusted using the IAM instance role to provision EKS clusters in the same AWS account as the controller EC2 instance.
See Create Cloud Credentials for creating the cloud credentials in the console to use for EKS cluster provisioning.
See Create Cluster for EKS cluster provisioning in the same AWS account with the controller using the above cloud credentials.

Use AWS Account ID, Access Key, and Secret to Provision EKS Clusters¶

Use the controller AWS Account ID, Access Key, and Secret to provision EKS clusters in the same AWS account as the controller EC2 instances or in different AWS accounts.

Create the STS policy and name it rafay-sts-policy.

{
"Version": "2012-10-17",
"Statement": [
    {
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": "sts:*",
    "Resource": "*"
    }
]
}

Assign the following policies to the IAM user used for delegate accounts.
rafay-sts-policy
AmazonEC2ReadOnlyAccess (AWS Managed Policy)
The following parameter must be disabled in the config.yaml file to use this feature.

global.use_instance_role: false

Optionally, you can provide details for the IAM user account, AWS secret, and key in the config.yaml file. If you do not want to provide these keys, see Adding AWS Account, Secret, and Key using Curl.

global.secrets.aws_account_id: "< AWS Account ID where controller is running>"
global.secrets.aws_access_key_id: <Base64 encoded string of access key>
global.secrets.aws_secret_access_key: <Base64 encoded string of aws secret>

If this feature is disabled after deploying the controller, rerun the RADM application command.

sudo radm application --config config.yaml

Adding AWS Account, Secret, and Key Using Curl¶

Access the operations console for the controller using the following URL. Use the super-user credentials to login. The super-user credentials were set in the config.yaml file.

https://ops-console.<company.example.com>

After login, run the following command from your local system, using curl to update the AWS key and secret to the controller.

Note

CSRF tokens and RSA ID can be obtained from the inspect screen in the browser after login. In the following example, replace the bold content.

curl -X PUT 'https://ops-console.<rafay.example.com>/edge/v1/providers/rx28oml/?partner_id=rx28oml&organization_id=rx28oml' \
-H 'authority: ops-console.<rafay.example.com>' \
-H 'x-rafay-partner: rx28oml' \
-H 'accept: application/json, text/plain, */*' \
-H 'x-csrftoken: BuSAE3rVCGCwO45N8ne2nKyXiiR53ZL2xPNi6qk2MuVvKHytdH4nKGCtkZZHajN3' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36' \
-H 'content-type: application/json;charset=UTF-8' \
-H 'origin: https://ops-console.<rafay.example.com>' \
-H 'sec-fetch-site: same-origin' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-dest: empty' \
-H 'referer: https://ops-console.<rafay.example.com>/' \
-H 'accept-language: en' \
-H 'cookie: logo_link=/logo.png; support_email=support@rafay.co; csrftoken=BuSAE3rVCGCwO45N8ne2nKyXiiR53ZL2xPNi6qk2MuVvKHytdH4nKGCtkZZHajN3; rsid=9zmsei147cok9qjqq2mkyxwqpps4ixr0' \
--data-raw '{"id":"rx28oml","partner_id":"rx28oml","credentials":"{\"account_id\":\"790000000230\",\"access_id\":\"YOUR_AWS_ACCESS_KEY_ID\",\"secret_key\":\"YOUR_AWS_ACCESS_SECRET_KEY\"}","provider":1,"credential_type":0,"name": "default-partner-credentials-1","delegate_account":true, "created_at": "2021-09-25T15:08:03.356164Z"}' \
--compressed \
--insecure

See Credentials for EKS and follow the steps to create credentials to provision EKS clusters through the console user-interface.
See Create Cluster and follow the steps for EKS cluster provisioning.