This section explains how to deploy Cloud Native Qumulo (CNQ) on AWS by creating the persistent storage and the cluster compute and cache resources with Terraform. It also provides recommendations for Terraform deployments and information about post-deployment actions and optimization.
For an overview of CNQ on AWS, its prerequisites, and limits, see How Cloud Native Qumulo Works.
The aws-terraform-cnq-<x.y>.zip
file (the version in the file name corresponds to the provisioning scripts, not the version of Qumulo Core) contains comprehensive Terraform configurations that let you deploy S3 buckets and then create a CNQ cluster with 4 to 24 instances that adhere to the AWS Well-Architected Framework and have fully elastic compute and capacity.
Prerequisites
This section explains the prerequisites to deploying CNQ on AWS.
-
To allow your Qumulo instance to report metrics to Qumulo, your AWS VPC must have outbound Internet connectivity through a NAT gateway or a firewall. Your instance shares no file data during this process.
Important
Connectivity to the following endpoints is required for a successful deployment of a Qumulo instance and quorum formation:api.missionq.qumulo.com
api.nexus.qumulo.com
-
The following features require specific versions of Qumulo Core:
Feature Minimum Qumulo Core Version - Adding S3 buckets to increase persistent storage capacity
- Increasing the soft capacity limit for an existing CNQ cluster
7.2.1.1 7.2.0.2 Creating persistent storage Important
You must create persistent storage by using a separate Terraform deployment before you deploy the compute and cache resources for your cluster.7.1.3 with version 4.0 of the deployment scripts -
Before you configure your Terraform environment, you must sign in to the AWS CLI.
A custom IAM role or user must include the following AWS services:
cloudformation:*
ec2:*
elasticloadbalancing:*
iam:*
kms:*
lambda:*
logs:*
resource-groups:*
route53:*
s3:*
secretsmanager:*
sns:*
ssm:*
sts:*
Note
Although theAdministratorAccess
managed IAM policy provides sufficient permissions, your organization might use a custom policy with more restrictions.
How the CNQ Provisioner Works
The CNQ Provisioner is an m5.large EC2 instance that configures your Qumulo cluster and any additional AWS environment requirements.
The Provisioner stores all necessary state information in AWS Systems Manager (Application Management > Parameter Store) and shuts down automatically when it completes its tasks.
Step 1: Deploying Cluster Persistent Storage
This section explains how to deploy the S3 buckets that act as persistent storage for your Qumulo cluster.
-
Log in to Nexus and click Downloads > Cloud Native Qumulo Downloads.
-
On the AWS tab and, in the Download the required files section, select the Qumulo Core version that you want to deploy and then download the corresponding Terraform configuration, Debian package, and host configuration file.
-
Create a new S3 bucket and, within your S3 bucket prefix, create the
qumulo-core-install
directory. -
Within this directory, create another directory with the Qumulo Core version as its name. For example:
my-s3-bucket-name/my-s3-bucket-prefix/qumulo-core-install/7.2.3.2
Tip
Make a new subdirectory for every new release of Qumulo Core. -
Copy
qumulo-core.deb
andhost_configuration.tar.gz
into the directory named after the Qumulo Core version (in this example, it is7.2.3.2
). -
Copy
aws-terraform-cnq-<x.y>.zip
to your Terraform environment and decompress. -
Navigate to the
persistent-storage
directory and take the following steps:-
Run the
terraform init
command.Terraform prepares the environment and displays the message
Terraform has been successfully initialized!
-
Review the
terraform.tfvars
file.-
Specify the
deployment_name
and the correctaws_region
for your cluster’s persistent storage. -
Leave the
soft_capacity_limit
at500
.
-
-
Use the
aws
CLI to authenticate to your AWS account. -
Run the
terraform apply
command.Terraform displays the execution plan.
-
Review the Terraform execution plan and then enter
yes
.Terraform creates resources according the execution plan and displays:
-
The
Apply complete!
message with a count of added resources -
The names of the created S3 buckets
-
Your deployment’s unique name
For example:
Apply complete! Resources: 15 added, 0 changed, 0 destroyed. Outputs: bucket_names = [ "ab5cdefghij-my-deployment-klmnopqr9st-qps-1", "ab4cdefghij-my-deployment-klmnopqr8st-qps-2", "ab3cdefghij-my-deployment-klmnopqr7st-qps-3", "ab2cdefghij-my-deployment-klmnopqr6st-qps-4", ] deployment_unique_name = "my-deployment-ABCDEFGH1IJ" ...
Tip
You will need thedeployment_unique_name
value to deploy your cluster. -
-
Step 2: Deploying Cluster Compute and Cache Resources
This section explains how to deploy compute and cache resources for a Qumulo cluster by using a Ubuntu AMI and the Qumulo Core .deb
installer.
- Provisioning completes successfully when the provisioning instance shuts down automatically. If the provisioning instance doesn't shut down, the provisioning cycle has failed and you must troubleshoot it. To monitor the provisioner's status, you can watch the Terraform status posts in your terminal or in AWS Systems Manager (Application Management > Parameter Store).
- The first variable in the example configuration files in the
aws-terraform-cnq
repository isdeployment_name
. To help avoid conflicts between Network Load Balancers (NLBs), resource groups, cross-region CloudWatch views, and other deployment components, Terraform ignores thedeployment_name
value and any changes to it. Terraform generates the additionaldeployment_unique_name
variable; appends a random, 11-digit alphanumeric value to it; and then tags all future resources with this variable, which never changes during subsequent Terraform deployments. - If you plan to deploy multiple Qumulo clusters, give the
q_cluster_name
variable a unique name for each cluster. - (Optional) If you use Amazon Route 53 private hosted zones, give the
q_fqdn_name
variable a unique name for each cluster
-
Configure your VPC to use the gateway VPC endpoint for S3.
Important
It isn’t possible to deploy your cluster without a gateway. -
Navigate to the
aws-terraform-cnq-<x.y>
directory and then run theterraform init
command.Terraform prepares the environment and displays the message
Terraform has been successfully initialized!
-
Choose
config-standard.tfvars
orconfig-advanced.tfvars
and fill in the values for all variables.For more information, see
README.pdf
inaws-terraform-cnq-<x.y>.zip
. -
Run the
terraform apply -var-file config-standard.tfvars
command.Terraform displays the execution plan.
-
Review the Terraform execution plan and then enter
yes
.Terraform creates resources according the execution plan and displays:
-
The
Apply complete!
message with a count of added resources -
Your deployment’s unique name
-
The names of the created S3 buckets
-
The floating IP addresses for your Qumulo cluster
-
The primary (static) IP addresses for your Qumulo cluster
-
The Qumulo Core Web UI endpoint
For example:
Apply complete! Resources: 62 added, 0 changed, 0 destroyed. Outputs: cluster_provisioned = "Success" deployment_unique_name = "my-deployment-ABCDEFGH1IJ" ... persistent_storage_bucket_names = tolist([ "ab5cdefghij-my-deployment-klmnopqr9st-qps-1", "ab4cdefghij-my-deployment-klmnopqr8st-qps-2", "ab3cdefghij-my-deployment-klmnopqr7st-qps-3", "ab3cdefghij-my-deployment-klmnopqr7st-qps-3", ]) qumulo_floating_ips = [ "203.0.113.42", "203.0.113.84", ... ] ... qumulo_primary_ips = [ "203.0.113.0", "203.0.113.1", "203.0.113.2", "203.0.113.3" ] ... qumulo_private_url_node1 = "https://203.0.113.10"
-
-
To log in to your cluster’s Web UI, use the endpoint from the Terraform output and the username and password that you have configured.
Important
If you change the administrative password for your cluster by using the Qumulo Core Web UI,qq
CLI, or REST API after deployment, you must add your new password to AWS Secrets Manager.You can use the Web UI to create and manage NFS exports, SMB shares, snapshots, and continuous replication relationships You can also join your cluster to Active Directory, configure LDAP, and perform many other operations.
-
Mount your Qumulo file system by using NFS or SMB and your cluster’s DNS name or IP address.
Step 3: Performing Post-Deployment Actions
This section describes the common actions you can perform on a CNQ cluster after deploying it.
Adding a Node to an Existing Cluster
To add a node to an existing cluster, the total node count must be greater than that of the current deployment.
- Edit
config-standard.tfvars
orconfig-advanced.tfvars
and change the value ofq_node_count
to a new value. - Run the
terraform apply -var-file config-standard.tfvars
command. -
Terraform displays the execution plan.
Review the Terraform execution plan and then enter
yes
.Terraform changes resources according the execution plan and displays an additional primary (static) IP for the new node. For example:
qumulo_primary_ips = [ "203.0.113.0", "203.0.113.1", "203.0.113.2", "203.0.113.3", "203.0.113.4" ]
- To ensure that the Provisioner shut downs automatically, review the
/qumulo/my-deployment-name/last-run-status
parameter in AWS Systems Manager (Application Management > Parameter Store). - To check that the cluster is healthy, log in to the Web UI.
Removing a Node from an Existing Cluster
Removing a node from an existing cluster is a two-step process:
- Remove the node from your cluster’s quorum.
- Tidy up your AWS resources.
Step 1: Remove the Node from the Cluster’s Quorum
You must perform this step while the cluster is running.
After you remove nodes from your cluster, you must clean up these nodes’ cloud infrastructure by using CloudFormation or Terraform.
-
Copy the
remove-nodes.sh
script from theaws-terraform-cnq-<x.y>/utilities
directory to a machine running in your VPC that has the AWS CLI tools installed (for example, an AWS Linux 2 AMI).Tip
- To make the script executable, run the
chmod +x remove-nodes.sh
command. - To see a list of required parameters, run
remove-nodes.sh
- To make the script executable, run the
-
Run the
remove-nodes.sh
script and specify the AWS region, the unique deployment name, the current node count, and the final node count.In the following example, we reduce a cluster from 6 to 4 nodes.
./remove-nodes.sh \ --region us-west-2 \ --qstackname my-unique-deployment-name \ --currentnodecount 6 \ --finalnodecount 4
-
Review the nodes to be removed and then enter
y
. -
Enter the administrator password for your cluster.
The script removes the nodes and displays:
-
Confirmation that your cluster formed a new quorum
-
Confirmation that the new quorum is active
-
The new total number of nodes in the quorum
-
The EC2 identifiers for the removed nodes
-
The endpoint for your cluster’s Web UI
{"monitor_uri": "/v1/node/state"} --Waiting for new quorum --New quorum formed --Quorum is ACTIVE --Validating quorum --4 Nodes in Quorum --REMOVED: EC2 ID=i-0ab12345678c9012d >> Qumulo node_id=5 --REMOVED: EC2 ID=i-9dc87654321b0987a >> Qumulo node_id=6 **Verify the cluster is healthy in the Qumulo UI at https://203.0.113.10 ...
-
-
To check that the cluster is healthy, log in to the Web UI.
Step 2: Tidy Up Your AWS Resources
- Navigate to the
aws-terraform-cnq-<x.y>
directory. - Edit
config-standard.tfvars
orconfig-advanced.tfvars
and change the value ofq_node_count
to a lower value (for example,4
). -
Run the
terraform apply -var-file config-standard.tfvars
command.Review the Terraform execution plan and then enter
yes
.Terraform removes the resources for the removed nodes according the execution plan and displays the primary (static) IPs for the remaining nodes. For example:
qumulo_primary_ips = [ "203.0.113.0", "203.0.113.1", "203.0.113.2", "203.0.113.3", "203.0.113.4" ]
The node and the infrastructure associated with the node are removed.
- To check that the cluster is healthy, log in to the Web UI.
Increasing the Soft Capacity Limit for an Existing Cluster
Increasing the soft capacity limit for an existing cluster is a two-step process:
- Configure new persistent storage parameters.
- Configure new compute and cache deployment parameters.
Step 1: Set New Persistent Storage Parameters
- Edit the
terraform.tfvars
file in thepersistent-storage
directory and set thesoft_capacity_limit
variable to a higher value. -
Run the
terraform apply
command.Review the Terraform execution plan and then enter
yes
.Terraform creates new S3 buckets as necessary and displays:
-
The
Apply complete!
message with a count of changed resources -
The names of the created S3 buckets
-
Your deployment’s unique name
-
The new soft capacity limit
Apply complete! Resources: 0 added, 1 changed, 0 destroyed. Outputs: bucket_names = [ "ab5cdefghij-my-deployment-klmnopqr9st-qps-1", "ab4cdefghij-my-deployment-klmnopqr8st-qps-2", "ab3cdefghij-my-deployment-klmnopqr7st-qps-3", "ab3cdefghij-my-deployment-klmnopqr7st-qps-3", ] deployment_unique_name = "lucia-deployment-GKMVD58UF2F" ... soft_capacity_limit = "1000 TB"
-
Step 2: Update Existing Compute and Cache Resource Deployment
- Navigate to the root directory of the
aws-terraform-cnq-<x.y>
repository. -
Run the
terraform apply -var-file config-standard.tfvars
command.Review the Terraform execution plan and then enter
yes
.Terraform updates the necessary IAM roles and S3 bucket policies, adds S3 buckets to the persistent storage list for the cluster, increases the soft capacity limit, and displays the
Apply complete!
message.When the Provisioner shuts down automatically, this process is complete.
Scaling Your Existing CNQ on AWS Cluster
To minimize potential availability interruptions, you must perform this cluster replacement procedure as a two-quorum event. For example, if you stop the existing EC2 instances by using the AWS Management Console and change the EC2 instance types, two quorum events occur for each node and the read and write cache isn’t optimized for the EC2 instance type.
You can scale an existing CNQ on AWS cluster by changing the EC2 instance type. This is a three-step process:
- Create a new deployment in a new Terraform workspace and join the new EC2 instances to a quorum.
- Remove the existing EC2 instances.
- Clean up your S3 bucket policies.
Step 1: Create a New Deployment in a New Terraform Workspace
- To create a new Terraform workspace, run the
terraform workspace new my-new-workspace-name
command. - To initialize the workspace, run the
terraform init
command. -
Edit
config-standard.tfvars
orconfig-advanced.tfvars
:- Specify the value for the
q_instance_type
variable. - Set the value of the
q_replacement_cluster
variable totrue
. - Set the value of the
q_existing_deployment_unique_name
variable to the current deployment’s name. - (Optional) To change the number of nodes, specify the value for the
q_node_count
variable.
- Specify the value for the
-
Run the
terraform apply -var-file config-standard.tfvars
command.Review the Terraform execution plan and then enter
yes
.Terraform creates resources according the execution plan and displays:
-
The
Apply complete!
message with a count of added resources -
Your deployment’s unique name
-
The names of the created S3 buckets
-
The same floating IP addresses for your Qumulo cluster
-
New primary (static) IP addresses for your Qumulo cluster
-
The Qumulo Core Web UI endpoint
Apply complete! Resources: 66 added, 0 changed, 0 destroyed. Outputs: cluster_provisioned = "Success" deployment_unique_name = "my-deployment-ABCDEFGH1IJ" ... persistent_storage_bucket_names = tolist([ "ab5cdefghij-my-deployment-klmnopqr9st-qps-1", "ab4cdefghij-my-deployment-klmnopqr8st-qps-2", "ab3cdefghij-my-deployment-klmnopqr7st-qps-3", "ab3cdefghij-my-deployment-klmnopqr7st-qps-3", ]) qumulo_floating_ips = [ "203.0.113.42", "203.0.113.84", ... ] ... qumulo_primary_ips = [ "203.0.113.4", "203.0.113.5", "203.0.113.6", "203.0.113.7" ] ... qumulo_private_url_node1 = "https://203.0.113.10"
-
- To ensure that the Provisioner shut downs automatically, review the
/qumulo/my-deployment-name/last-run-status
parameter in AWS Systems Manager (Application Management > Parameter Store). - To check that the cluster is healthy, log in to the Web UI.
To perform future node addition or removal operations, edit the
config-standard.tfvars
or config-advanced.tfvars
file and set the q_replacement_cluster
variable to false
.Step 2: Remove the Existing EC2 Instances
- To select the previous Terraform workspace (for example,
default
), run theterraform workspace select default
command. - To ensure that the correct workspace is selected, run the
terraform workspace show
command. -
Run the
terraform destroy -var-file config-standard.tfvars
command.Review the Terraform execution plan and then enter
yes
.Terraform deletes resources according to the execution plan and displays the
Destroy complete!
message with a count of destroyed resources.The previous EC2 instances are deleted.
The persistent storage deployment remains in its original Terraform workspace. You can perform the next cluster replacement procedure in the
default
workspace.Step 3: Clean Up S3 Bucket Policies
- To list your Terraform workspaces, run the
terraform workspace list
command. - To select your new Terraform workspace, run the
terraform workspace select <my-new-workspace-name>
command. - Edit the
config-standard.tfvars
orconfig-advanced.tfvars
file and set theq_replacement_cluster
variable tofalse
. -
Run the
terraform apply -var-file config-standard.tfvars
command. This ensures that the S3 bucket policies have least privilege.Review the Terraform execution plan and then enter
yes
.Terraform deletes resources according to the execution plan and displays the
Apply complete!
message with a count of destroyed resources.
Deleting an Existing Cluster
Deleting a cluster is a two-step process:
- Delete your Cloud Native Qumulo resources.
- Delete your persistent storage.
- When you no longer need your cluster, you must back up all important data on the cluster safely before deleting the cluster.
- When you delete your cluster's cache and computer resources, it isn't possible to access your persistent storage anymore.
Step 1: To Delete Your Cluster’s Cloud Native Qumulo Resources
- After you back up your data safely, edit your
config-standard.tfvars
orconfig-advanced.tfvars
file and set theterm_protection
variable tofalse
. -
Run the
terraform apply -var-file config-standard.tfvars
command.Review the Terraform execution plan and then enter
yes
.Terraform marks resources for deletion according to the execution plan and displays the
Apply complete!
message with a count of changed resources. -
Run the
terraform destroy -var-file config-standard.tfvars
command.Review the Terraform execution plan and then enter
yes
.Terraform deletes all of your cluster’s CNQ resources and displays the
Destroy complete!
message and a count of destroyed resources.
Step 2: To Delete Your Cluster’s Persistent Storage
- Navigate to the
persistent-storage
directory. - Edit your
terraform.tfvars
file and set theprevent_destroy
parameter tofalse
. -
Run the
terraform apply
command.Review the Terraform execution plan and then enter
yes
.Terraform marks resources for deletion according to the execution plan and displays the
Apply complete!
message with a count of changed resources. -
Run the
terraform destroy
command.Review the Terraform execution plan and then enter
yes
.Terraform deletes all of your cluster’s persistent storage and displays the
Destroy complete!
message and a count of destroyed resources.