This section explains what happens when a Qumulo Core node goes offline and how you can check the remaining free space.
What Happens When a Qumulo Core Node Goes Offline
Qumulo Core uses erasure coding to let multiple drives or nodes to go offline but continue to serve data. For more information, see Qumulo Drive Failure Protection on Qumulo Care.
On hybrid Qumulo nodes (that have HDDs and SSDs), Qumulo Core attempts to maintain cluster functionality for as long as possible. When a node goes offline, the cluster evicts existing data promoted to SSDs on the remaining nodes and makes all writes to the free space on the SSDs. When the node comes online, Qumulo Core begins to push writes to the HDDs that back the SSDs.
Writes, deletes, and changes count towards SSD space. For more information, see Checking Remaining Free Space on Your Cluster.
A cluster can operate with an offline node until its SSD space fills up. When a cluster has no more free space, the cluster goes into read-only (ENOSPC) state until the node comes back online and all nodes rejoin the quorum.
The amount of time that users have before the cluster enters the
ENOSPC state depends on the rate of change in new rates to the cluster.Checking Remaining Free Space on Your Cluster
You can estimate the amount of time before your cluster enters the ENOSPC state by running multiple iterations of the debug_metrics_get command (metrics_get in Qumulo Core 4.2.0 and lower) on every node in your cluster.
The output of the command shows the number of valid tokens remaining for writes to the cluster. At 50,000 (or fewer) tokens, an ENOSPC event is imminent.
Because SSD space on the cluster is limited, new writes, deletes, and changes consume any reclaimable tokens.
| Qumulo Core Version | Command |
|---|---|
| 4.2.1 (and higher) | sudo qsh -c /opt/qumulo/qq debug_metrics_get \ --measurement space_agent | grep reclaimable; sleep 60; \ sudo qsh -c /opt/qumulo/qq debug_metrics_get \ --measurement space_agent | grep reclaimable |
| 3.3.2 - 4.2.0 | sudo qsh -c /opt/qumulo/qq metrics_get \ --measurement space_agent | grep reclaimable; sleep 60; \ sudo qsh -c /opt/qumulo/qq metrics_get \ --measurement space_agent | grep reclaimable |
| 3.1 - 3.3.1 | sudo /opt/qumulo/qq metrics_get \ --measurement space_agent | grep reclaimable; sleep 60; \ sudo /opt/qumulo/qq metrics_get \ --measurement space_agent | grep reclaimable |
You can also contact the Qumulo Care team for a time estimate.