Providers
Providers are tools integrated into the Rafay platform to manage and deploy infrastructure resources efficiently. They allow users to choose between different methods, like HCP Terraform, OpenTofu, Rafay's System Provider, or Custom Provider for creating and managing resources. Each provider offers specific benefits, such as centralized management with HCP Terraform, open-source flexibility with OpenTofu, or streamlined native integration with Rafay's System Provider. These providers help users simplify complex infrastructure tasks, reduce dependency on external tools, and improve overall resource management within the platform.
HCP Terraform¶
HCP Terraform is a managed service that allows teams to run Terraform together. It manages Terraform runs in a consistent and reliable environment, and includes easy access to shared state and secret data
The HCP Terraform provider option is intended for licensed HashiCorp customers. With this option, users are required to provide a custom driver to manage Terraform binaries in compliance with the BSL License. The backend type is always set to HCP Terraform, enabling centralized management of state store files etc.
Note: On selecting HCP Terraform, Custom Driver is MANDATORY
- To enable the Terraform driver to work with the Terraform Cloud organization, users need to provide a token that allows the driver to authenticate with Terraform Cloud. This token should be stored in a specific file:
File Name: .terraform.d/credentials.tfrc.json
{
"credentials": {
"app.terraform.io": {
"token": "your-token-here"
}
}
}
- Next step is to configure the driver to read the token from this file using an environment variable:
Environment Variable:
- Key: TF_CLI_CONFIG_FILE
- Value: .terraform.d/credentials.tfrc.json
Specify this file and environment variable in either of the following ways: - In the custom driver configuration used for Terraform - Using a configuration context which is then attached to the Resource Template
- Additionally, in the Container Driver, the "working directory path" value should either be set to
/home/terraform
or left empty (in which case, the default value will be /home/terraform).
- Add the Cloud block. You can define it's arguments in the configuration file or supply them as environment variables
OpenTofu¶
OpenTofu is an open-source infrastructure IaC (infrastructure as code) tool. Rafay offers first class integration for OpenTofu that allows customers to leverage OpenTofu as the IaC provisioning tool with the Environment Manager framework.
Migrating from Terraform to OpenTofu
For users looking to migrate from the deprecated Terraform Provider option to OpenTofu Provider option, the process involves the following steps.
- Create a new version of the resource template with the provider selected as OpenTofu ensuring that all other configurations remain consistent with the older template version
- Additionally, create a new version of the Environment Template that includes the updated Resource Template. This new Environment Template version can then be used to create new environments
Important
Updating an existing environment that uses Terraform provider to the new OpenTofu provider is possible by republishing the existing environment using an updated version of Environment Template (that includes Resource Template(s) with the OpenTofu provider). While this approach may work in most cases, it's important to proceed with caution as there may be risks involved. Rafay recommends creating a new environment using OpenTofu, verifying its functionality, and then deleting the older environment.
By default, a few GitHub APIs need to be accessed to download the OpenTofu binary. If the infrastructure where the agent is running does not have access to GitHub APIs, or if users prefer to download the Tofu binary from a different endpoint for operational or security reasons, this can be accommodated using the OVERRIDE_OPENTOFU_DOWNLOAD_ENDPOINT
environment variable. This variable should point to any URL that hosts the Tofu binary in a zip file, and the agent will download the Tofu binary from this endpoint. Alternatively, customers can build a custom driver (based on OpenTofu) and reference that as part of the resource template configuration.
Example
OVERRIDE_OPENTOFU_DOWNLOAD_ENDPOINT=https://github.com/opentofu/opentofu/releases/download/v1.6.2/tofu_1.6.2_linux_amd64.zip
System Provider - Coming Soon
System Provider¶
The new System Provider feature allows users to natively create and manage Rafay first-class objects, such as clusters and cloud credentials, directly within the Rafay platform. This enhancement eliminates the need for third-party providers like Terraform, simplifying resource management and reducing complexities related to external tools and licensing. Users can now handle the creation of both credentials and clusters together with Environment Manager, avoiding multiple processes and streamlining their workflow.
This feature is especially useful for users facing limitations with certain Terraform versions that restrict cluster creation. By using the System Provider, users can leverage Environment Manager to manage Rafay resources effectively, without relying on external providers.
Additionally, users can add hooks before and after the deployment of clusters and credentials, offering greater flexibility and control over the deployment process. This dynamic capability facilitates a more streamlined and integrated approach to managing Rafay resources.
On selecting System Provider, configure Hooks, Agents, Input Variables, and Config Contexts.
Custom Provider - Coming Soon
Custom Provider¶
The new Custom Provider in the Environment Manager expands the platform’s capabilities by allowing users to create custom workflows tailored to their specific needs, regardless of complexity. This provider empowers users to design workflows that generate activities aligned with their unique requirements. It supports parallel and sequential execution of tasks, along with the ability to combine both in a single workflow. Tasks can be arranged with dependencies, ensuring they execute in the correct order. Built on a Directed Acyclic Graph (DAG) structure, the Custom Provider facilitates the management of complex workflows, making it suitable for use cases such as CI/CD deployments, storage scans, and batch jobs. This flexibility allows users to integrate the Custom Provider seamlessly into existing environment templates, enhancing the overall functionality of the platform.
- Select Custom Provider and click Add Task. The task page appears, allowing you to define and organize the individual steps required to execute a workflow within the Custom Provider
Task Configuration
- Provide a unique name for the task and Task type Driver is selected by default
- Select the Driver Configuration: either Select Existing Driver (referring to an inline driver) or Create New Driver
- If selecting Select Existing Driver, choose the desired driver from the drop-down menu and attach to the task. The same driver can be used for multiple tasks if needed
- If selecting Create New Driver, choose the Driver Type (either Container or HTTP) and provide the necessary details
Advanced Task Configuration
- Select an Agent to ensure that tasks are executed in the appropriate environment with the necessary resources for successful workflow completion
- Enable Continue on Failure if required, and set a Timeout
- Add dependencies using the Depends On field to specify the order of task execution and establish how tasks are connected within the workflow
Note: The default timeout for a custom provider task is 5 minutes if not specified
- Click Save
The newly added task(s) is listed as shown below:
On selecting Custom Provider, configure Hooks, Agents, Input Variables, and Config Contexts.
Task Result Sharing¶
-
Container tasks share their results by uploading a file called
output.json
. Only tasks that follow the one sharing results (and depend on it) can access the data. The drivers of these dependent tasks can use the information inoutput.json
to complete their execution. -
HTTP tasks also share their results through
output.json
. Theoutput.json
for HTTP tasks typically contains the following:- statusCode: The HTTP status code
- body: The HTTP response payload
- headers: The HTTP response headers
Below is an example on how to upload the output.json
file
import requests
def upload():
return requests.post(
os.getenv('DRIVER_UPLOAD_URL'),
headers={"X-Engine-Helper-Token": os.getenv('DRIVER_UPLOAD_TOKEN')},
files={'content': open('/path/to/output.json', 'rb')}
)
package main
import (
"bytes"
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
)
func upload() {
// Path to the file you want to upload
filePath := "/path/to/output.json"
// URL to upload the file
uploadURL := os.Getenv("DRIVER_UPLOAD_URL")
if uploadURL == "" {
fmt.Println("Error: DRIVER_UPLOAD_URL environment variable is not set")
return
}
// Authentication token
authToken := os.Getenv("DRIVER_UPLOAD_TOKEN")
if authToken == "" {
fmt.Println("Error: DRIVER_UPLOAD_TOKEN environment variable is not set")
return
}
// Open the file
file, err := os.Open(filePath)
if err != nil {
fmt.Println("Error opening file:", err)
return
}
defer file.Close()
// Create a new buffer to store the multipart form data
var body bytes.Buffer
writer := multipart.NewWriter(&body)
// Add the file to the form data
part, err := writer.CreateFormFile("content", filePath)
if err != nil {
fmt.Println("Error creating form file:", err)
return
}
_, err = io.Copy(part, file)
if err != nil {
fmt.Println("Error copying file:", err)
return
}
// Close the multipart writer
writer.Close()
// Create a new HTTP request
req, err := http.NewRequest("POST", uploadURL, &body)
if err != nil {
fmt.Println("Error creating request:", err)
return
}
// Set the helper token header
req.Header.Set("X-Engine-Helper-Token", authToken)
// Set the content type header
req.Header.Set("Content-Type", writer.FormDataContentType())
// Send the request
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request:", err)
return
}
defer resp.Body.Close()
// Check the response
fmt.Println("Response status:", resp.Status)
}
Refer to this page to learn more about Custom Provider Expressions.