Fixing 'NFS Permission Denied' and 'Connection Refused' Errors in Linux

Understanding the Error

When working with Network File System (NFS) in Linux environments, encountering a Permission denied or Connection refused error can halt critical infrastructure. These errors often stem from a misalignment in how NFS handles user identities, network security, and file permissions. NFS relies on Remote Procedure Calls (RPC) and assumes a level of trust between the client and the server. When that trust is broken—whether by a mismatched User ID (UID), an overly restrictive export option, or a firewall blocking port 2049—the system throws these cryptic errors.

A Permission denied error usually occurs after the mount is successful, when a user tries to read or write a file. Conversely, access denied by server while mounting or Connection refused happens during the initial mount attempt. Understanding the difference is crucial for rapid troubleshooting.

Common Symptoms

• NFS Permission Denied: You can mount the share, but running touch file.txt yields touch: cannot touch 'file.txt': Permission denied.
• NFS Connection Refused: Running mount -t nfs server:/share /mnt returns mount.nfs: Connection refused.
• NFS Not Working / Hanging: Commands like ls or df -h hang indefinitely inside the mounted directory.
• NFS Slow: File transfers take exponentially longer than the network bandwidth should allow, or processes enter the D (uninterruptible sleep) state.

Step 1: Diagnose the Root Cause

Before making changes, you must identify where the breakdown is occurring. Is it a network issue, an authentication issue, or a filesystem permission issue?

1. Check the NFS Server Exports

The /etc/exports file dictates which clients can access which directories and with what privileges. Run cat /etc/exports on the server. Look for the target directory. A typical entry looks like: /data/share 192.168.1.0/24(rw,sync,no_subtree_check)

If your client IP is not in the allowed list, or if the share is exported as ro (read-only), you will get permission errors when writing.

2. Verify UID and GID Alignment

NFS version 3 and version 4 (without Kerberos) rely on the client and server agreeing on what a UID means. If User A has UID 1000 on the client, but UID 1000 is User B on the server, User A will have User B's permissions on the NFS share. On both client and server, run id <username> and compare the output.

3. Inspect Root Squashing

By default, NFS implements root_squash. This means if the root user on the client tries to write to the NFS share, the server maps those requests to the nfsnobody (or nobody) user. If the directory is owned by root and has 755 permissions, the client's root user will get a Permission denied error when trying to write.

4. Test Network Connectivity and RPC

If you are getting Connection refused, test the network. Use rpcinfo -p <server_ip> from the client to see if the NFS and mountd ports are open and responding. Also, try nc -zv <server_ip> 2049 to verify the main NFS port is reachable.

Step 2: Fix 'Permission Denied' Errors

Solution A: Fix UID/GID Mismatches

If the UIDs don't match, the cleanest solution is to align them.

1. Identify the correct UID/GID on the server.
2. On the client, modify the user's UID to match: usermod -u <new_uid> <username>
3. Update group ownership: groupmod -g <new_gid> <groupname>
4. Recursively update files owned by the user on the client's local filesystem: find / -user <old_uid> -exec chown -h <new_uid> {} + (Use with extreme caution).

Alternatively, if using NFSv4, ensure idmapd is configured correctly. The /etc/idmapd.conf file on both client and server must have the same Domain = <your_domain> setting. Restart the nfs-idmapd service after making changes.

Solution B: Address Root Squashing

If your application runs as root on the client and needs root permissions on the server (e.g., for container storage or VM images), you need to disable root squashing.

1. On the server, edit /etc/exports.
2. Change the options to include no_root_squash: /data/share 192.168.1.100(rw,sync,no_root_squash)
3. Re-export the shares: exportfs -arv Warning: no_root_squash is a significant security risk if the client network is untrusted, as a compromised client immediately gains root access to the exported directory on the server.

Solution C: Correct Filesystem Permissions

The underlying Linux filesystem on the server still enforces permissions. Even if the NFS export allows read/write, the actual directory must be writable by the mapped user.

1. On the server, check the directory permissions: ls -ld /data/share
2. Change ownership or permissions as needed. For example, to give the specific user ownership: chown -R 1000:1000 /data/share
3. Or, for a shared group directory, use the SGID bit: chmod 2775 /data/share

Step 3: Fix 'Connection Refused' and Networking Issues

Solution D: Configure the Firewall

A Connection refused error almost always points to a firewall blocking traffic to port 2049 (NFS) or 111 (rpcbind). On the server (if using UFW/Ubuntu): sudo ufw allow from <client_ip> to any port nfs sudo ufw allow from <client_ip> to any port 111

If using Firewalld (CentOS/RHEL): sudo firewall-cmd --permanent --add-service=nfs sudo firewall-cmd --permanent --add-service=rpc-bind sudo firewall-cmd --permanent --add-service=mountd sudo firewall-cmd --reload

Solution E: Verify NFS Services are Running

Ensure the NFS server daemon is actually active. On the server: systemctl status nfs-server If it is stopped or failed, check the logs (journalctl -xeu nfs-server) and restart it: systemctl restart nfs-server.

Step 4: Fix 'NFS Slow' and Performance Drops

If NFS is mounted but operations are painfully slow, or if the client crashes/hangs under heavy load, the issue is likely packet fragmentation or poorly tuned mount options.

Solution F: Tune rsize and wsize

The rsize (read size) and wsize (write size) define the maximum chunk of data sent over the network in one RPC request. Modern networks handle 1MB (1048576 bytes) easily. On the client, unmount and remount with tuned options: mount -t nfs -o rw,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 server:/share /mnt

Solution G: Hard vs. Soft Mounts

Always use hard mounts for read/write data. A soft mount will eventually time out and return an I/O error to the application if the server becomes unreachable. This can cause severe data corruption. A hard mount will pause the application until the server returns, which prevents data loss, though it may cause a temporary process hang.

Solution H: Asynchronous Exports

On the server, exporting a share with the async option can massively improve write performance. /data/share 192.168.1.0/24(rw,async,no_subtree_check) With async, the server replies to the client as soon as the data is in memory, before it hits the disk. Warning: If the NFS server loses power before writing the memory buffer to disk, data will be lost. Use sync if data integrity is paramount (e.g., databases).

Advanced Troubleshooting: NFSv3 vs NFSv4

A common source of confusion and connection issues arises from the differences between NFSv3 and NFSv4. NFSv3 is stateless and relies heavily on an auxiliary protocol called rpcbind (port 111) to inform the client which random, high-numbered ports the mountd, statd, and nlockmgr services are listening on. This makes firewalling NFSv3 a nightmare, as you must explicitly pin these services to static ports in configuration files like /etc/sysconfig/nfs or /etc/nfs.conf before you can open them in your firewall.

NFSv4, on the other hand, is stateful and operates entirely over a single well-known TCP port: 2049. It integrates mounting and locking into the core protocol, eliminating the need for rpcbind and auxiliary ports. If you are setting up a new environment, forcefully using NFSv4 will save you countless hours of network troubleshooting. To force an NFSv4 mount on the client: mount -t nfs4 server:/share /mnt To configure the server to only offer NFSv4 (disabling v3), edit /etc/nfs.conf:

[nfsd]
 vers3=n
 vers4=y

Advanced Security: Kerberos Authentication

While traditional NFS uses AUTH_SYS (trusting the client-provided UIDs), enterprise environments often require cryptographic authentication. This is where Kerberos (sec=krb5, sec=krb5i for integrity, sec=krb5p for privacy/encryption) comes into play. If you receive a Permission denied in a Kerberos-secured environment, the standard UID checks won't help. You must investigate:

1. Keytabs: Does the client have a valid machine keytab (/etc/krb5.keytab) with an nfs/client.domain.com principal?
2. Tickets: Does the user attempting the access have a valid Kerberos ticket? Run klist to check.
3. Time Sync: Kerberos strictly requires time synchronization. If the client and server clocks drift by more than 5 minutes, authentication will fail with cryptic errors. Ensure chronyd or ntpd is running and synced on both nodes.

Packet-Level Diagnostics with tcpdump

When logs provide no clear answers, inspecting the wire traffic is the ultimate source of truth. If a client hangs during a mount, run tcpdump on the server to see if packets are arriving and how the server responds. tcpdump -i eth0 -n -s0 host <client_ip> and port 2049 If you see SYN packets arriving from the client but no SYN-ACK leaving the server, the server's firewall is dropping the packets. If the TCP handshake completes, but the server immediately sends an RPC reply with PROG_MISMATCH or AUTH_ERROR, you know the issue lies in the NFS/RPC layer configuration, not the network routing.

Resolving "Stale File Handle" Errors

While distinct from Permission denied, the Stale file handle error (ESTALE) often accompanies NFS crashes or server reboots. This occurs when a file or directory that the client was accessing has been deleted or moved on the server, or if the server's filesystem was unmounted and remounted with a different inode structure. The only reliable fix for a stale file handle is to unmount the share on the client. If the filesystem is busy, you may need to force a lazy unmount: umount -l /mnt After a lazy unmount, wait a few moments and remount the share.

Conclusion and Best Practices

To minimize NFS issues in production environments, adhere to these best practices:

1. Standardize Identity Management: Use LDAP, FreeIPA, or Active Directory to ensure UIDs and GIDs are identical across all servers and clients. Never rely on manual /etc/passwd synchronization.
2. Embrace NFSv4: Move away from NFSv3 to simplify firewall rules and improve performance over WAN links.
3. Monitor Performance: Keep an eye on RPC retransmissions and timeouts using nfsstat -c. A high retransmission rate indicates network packet loss or an overloaded server.
4. Automate Mounts: Use autofs instead of static /etc/fstab entries. autofs mounts the share only when accessed and automatically unmounts it after a period of inactivity, reducing the impact of a server outage on client stability.

By methodically verifying network access, aligning identities, and optimizing mount parameters, you can permanently resolve Permission denied and Connection refused errors, ensuring your NFS infrastructure operates seamlessly.