This section describes common troubleshooting procedures for configuring NFSv4.1 to work with Kerberos.
Following General Debugging Techniques
This section lists common debugging techniques.
To Turn Up Logging Levels for Client-Side Tools
- In the
debug_level = 9.
- In the
Verbosity = 9.
[gssd]section of the
In versions of Ubuntu lower than 22.04 (and possibly on other Linux distributions), you can't use the
/etc/nfs.conffile to configure
gssd. If this is the case for your system, we recommend starting the
rpc.gssdservice by using the
rpcdebug, for example:
rpcdebug -m nfs -s all && rpcdebug -m rpc -s all
Taking a Client-Side Packet Capture
Normally, there should be:
- Kerberos and LDAP traffic between the client and the domain controller
- DNS traffic between the client and DNS server
- RPC or NFS traffic between the client and the Qumulo cluster
Because a Kerberos mount requires the client to perform a series of steps, in most cases, the last traffic that the client issues indicates the source of failure. To view encrypted Kerberos traffic, use Wireshark with a Kerberos keytab file. For more information, see Kerberos in the Wireshark documentation.
For help with interpreting logging and metrics from your Qumulo cluster and for insights from the telemetry of our Kerberos implementation, contact Qumulo Care.
Resolving Incorrect Display of Users or Groups
Under certain conditions, users or groups display as
nobody when you run the
ls -l or
Differentiating Client and Cluster Issues
To resolve this issue, determine whether it is with the client or with the cluster by running the
nfs4_getfacl command on a file. If the presentation in the ACL editor appears correct, the issue is with the client. Otherwise, the issue is with the cluster.
The ACL editor doesn’t perform any ID mapping. It only passes ACE trustees through, in plaintext.
Resolving Client-Side Issues
If the issue is with the client, it is most often an ID mapping issue. Confirm that your mappings are configured correctly. For more information, see User-Defined Identity Mappings on Qumulo Care.
If the issue persists, investigate logging and packet captures.
Resolving Cluster-Side Issues
If the issue is with the cluster, confirm that your cluster’s Active Directory settings include the Base DNs that contain the expected users. For more information, see Prerequisites for Joining a Qumulo Cluster to Active Directory.
Diagnosing Mount-Failed Errors
Under certain conditions, you might receive mount-failed errors from
mount.nfs. To diagnose this type of error, you can try the following procedures.
Confirm that the
rpc.gssdservice is running.
Confirm that the cluster and client both resolve from the client. It should be possible to reach the cluster and client through a fully qualified domain name (FQDN), such as
Confirm that reverse DNS works for the IP addresses on both the client and the cluster.
Confirm that the client has a
hostservice principal name (SPN) and that the cluster has an
nfsSPN that matches the DNS records.
Do one of the following:
If you use a machine account or
kinitauthentication, confirm that the credentials are correct. You can use the keytab
ktutilcommand or the credential cache
klistcommand to list the encryption methods.
Confirm that Kerberos tickets use AES-128 or AES-256 for service encryption by examining a packet capture or your Active Directory Kerberos settings.
If you use domain trusts, confirm that trust has AES-128 or AES-256 enabled.
Confirm that the clocks on the client, cluster, and domain controller are synchronized to the same time.
Confirm that the mount uses the
Qumulo Core doesn’t support the
Inspect logs and packet captures.