AWS Integration

Overview¶

Cost Management retrieves asset prices from the public AWS pricing API by default. You can leverage Cost Profile to interface directly with your account to receive accurate price data from AWS. This integration will properly account for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, spot usage, and more.

Necessary permissions are required to create the Cost and Usage Report (CUR), and to add IAM credentials for Athena and S3. The ability to add and use CloudFormation templates is an optional permission.

Users must perform the below steps and complete a few setups to create a Cost profile in the controller.

Step 1: CUR Setup¶

Refer CUR Creation for creating Cost and Usage Reports. It is mandatory to enable Resource IDs and Athena integration when creating the CUR. Note down the bucket name that you create for CUR data
If the Billing and Cost Management page cannot be accessed, the owner of the organization’s root account must follow the instructions provided here

Step 2: Athena Setup¶

In addition, as a part of CUR creation, Amazon creates a CloudFormation template that is used to create the Athena integration. This template is created in the CUR S3 bucket under user-billing-prefix/curname and typically has the filename crawler-cfn.yml. Deploy this CloudFormation template to complete the CUR Athena integration. Refer [here]https://docs.aws.amazon.com/cur/latest/userguide/use-athena-cf.html to know more details on CloudFormation templates.

Once the Athena setup is complete with the CUR, create a new S3 bucket for Athena query results.

Navigate to AWS S3 and select Create Bucket
Choose a name that adheres to the format aws-athena-query-results-* and make sure to use the same region as the one that was used for the CUR bucket
Select Create Bucket
Navigate to AWS Athena and select Settings
Set Query result location to the S3 bucket that got created now

Step 3: Required Policies for Role/User Integration¶

Attach the below two policies to the same role or user. Pick a user to integrate via servicekey (or) a role to integrate via IAM annotation. The SpotDataAccess policy statement is optional if the spot data feed is configured

Policy 1

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "AthenaAccess",
         "Effect": "Allow",
         "Action": [
            "athena:*"
         ],
         "Resource": [
            "*"
         ]
      },
      {
         "Sid": "ReadAccessToAthenaCurDataViaGlue",
         "Effect": "Allow",
         "Action": [
            "glue:GetDatabase*",
            "glue:GetTable*",
            "glue:GetPartition*",
            "glue:GetUserDefinedFunction",
            "glue:BatchGetPartition"
         ],
         "Resource": [
            "arn:aws:glue:*:*:catalog",
            "arn:aws:glue:*:*:database/athenacurcfn*",
            "arn:aws:glue:*:*:table/athenacurcfn*/*"
         ]
      },
      {
         "Sid": "AthenaQueryResultsOutput",
         "Effect": "Allow",
         "Action": [
            "s3:GetBucketLocation",
            "s3:GetObject",
            "s3:ListBucket",
            "s3:ListBucketMultipartUploads",
            "s3:ListMultipartUploadParts",
            "s3:AbortMultipartUpload",
            "s3:CreateBucket",
            "s3:PutObject"
         ],
         "Resource": [
            "arn:aws:s3:::aws-athena-query-results-*"
         ]
      },


      {
         "Sid": "S3ReadAccessToAwsBillingData",
         "Effect": "Allow",
         "Action": [
            "s3:Get*",
            "s3:List*"
         ],
         "Resource": [
            "arn:aws:s3:::${AthenaCURBucket}*"
         ]
      }
   ]
}

Policy 2

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "SpotDataAccess",
         "Effect": "Allow",
         "Action": [
            "s3:ListAllMyBuckets",
            "s3:ListBucket",
            "s3:HeadBucket",
            "s3:HeadObject",
            "s3:List*",
            "s3:Get*"
         ],
         "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
      }
   ]
}

Kubernetes clusters run in different accounts

Perform the following steps:

On each sub account that runs Cost Management, attach the below 2 sub account policies to the same role or user. If selecting "Use Cluster Credentials", update the two sub account policies below for the role or user used in Cluster Provisioning and skip 'Step 4'. The SpotDataAccess policy statement is optional if the spot data feed is configured

Sub account policy for CUR Integration

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "AssumeRoleInMasterPayer",
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": "arn:aws:iam::${MasterPayerAccountID}:role/costmanagementRole-${This-account’s-id}"
      }
   ]
}

Sub account policy for Spot Integration (optional)

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "SpotDataAccess",
         "Effect": "Allow",
         "Action": [
            "s3:ListAllMyBuckets",
            "s3:ListBucket",
            "s3:HeadBucket",
            "s3:HeadObject",
            "s3:List*",
            "s3:Get*"
         ],
         "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
      }
   ]
}

Master Account Policy

Attach this policy to a role on the masterpayer account (replace ${AthenaCURBucket} variable)

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "AthenaAccess",
         "Effect": "Allow",
         "Action": [
            "athena:*"
         ],
         "Resource": [
            "*"
         ]
},
{
         "Sid": "ReadAccessToAthenaCurDataViaGlue",
         "Effect": "Allow",
         "Action": [
            "glue:GetDatabase*",
            "glue:GetTable*",
            "glue:GetPartition*",
            "glue:GetUserDefinedFunction",
            "glue:BatchGetPartition"
         ],
         "Resource": [
            "arn:aws:glue:*:*:catalog",
            "arn:aws:glue:*:*:database/athenacurcfn*",
            "arn:aws:glue:*:*:table/athenacurcfn*/*"
         ]
      },
      {
         "Sid": "AthenaQueryResultsOutput",
         "Effect": "Allow",
         "Action": [
            "s3:GetBucketLocation",
            "s3:GetObject",
            "s3:ListBucket",
            "s3:ListBucketMultipartUploads",
            "s3:ListMultipartUploadParts",
            "s3:AbortMultipartUpload",
            "s3:CreateBucket",
            "s3:PutObject"
         ],
         "Resource": [
            "arn:aws:s3:::aws-athena-query-results-*"
         ]
      },
      {
         "Sid": "S3ReadAccessToAwsBillingData",
         "Effect": "Allow",
         "Action": [
            "s3:Get*",
            "s3:List*"
         ],
         "Resource": [
            "arn:aws:s3:::${AthenaCURBucket}*"
         ]
      }
   ]
}

Now, you must add the following trust statement to the role the policy is attached to (replace ${costmanagementClusterID} variable):

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::${AWSAccountID}:root"
         },
         "Action": [
            "sts:AssumeRole"
         ]
      }
   ]
}

Step 4: Attach IAM permissions to Cost Management¶

Important

If Use Cluster Credential is selected during the creation of the Cost Profile, this Step 4 does not need to be performed

Once the policies are created, users can attach those policies to Cost Management. The following methods are supported:

Via Access Key¶

Login to AWS IAM Access page
Navigate to Access Management -> Users
Find the Cost Management User and select Security Credentials -> Create Access Key. Note the Access key ID and Secret access key. You will use it to either Create a secret from helm values or Create and use an existing secret
Attach this Access ID and Secret key, and create a Cost Management AWS Cloud Credential

Create Cloud Credential

Via Role ARN¶

Create an OIDC provider for your cluster
Create a Role
Attach the policies created in Step 3
Attach this Role ARN when creating a cloud credentials of type Cost Management and provider AWS

Create Cloud Credential

Step 5: Configure CUR config values to Cost Management¶

Athena Bucket Name: An S3 bucket to store Athena query results to which Cost Management has permission to access
The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined in Step 3 will automatically allow access to it
The bucket can have a Canned ACL of Private or other permissions
Athena Region: The AWS region Athena is running in
Athena Database: The name of the database created by the Athena setup
The Athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created in Step 2
Athena Table: The name of the table created by the Athena setup
The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource)
Master Payer Account ARN: If using a multi-account setup, it is mandatory to set to the Amazon Resource Number (ARN) of the role in the masterpayer account. For example, arn:aws:iam::530337586275:role/costmanagementRole

Spot Data feed integration¶

Cost Management reconciles the users' spot prices with CUR billing reports as they become available (usually 1-2 days), but pricing data can be retrieved hourly by integrating directly with the AWS spot feed. To enable, follow the steps provided here