Troubleshooting 'SELinux Permission Denied' Errors (avc: denied)

Understanding the Error

One of the most frustrating experiences for a Linux system administrator or developer is encountering a Permission denied error even when the process is running as root, or when standard standard POSIX permissions (chmod/chown) appear perfectly open. If standard permissions look correct, the culprit is almost certainly SELinux (Security-Enhanced Linux).

SELinux implements Mandatory Access Control (MAC). Unlike traditional Discretionary Access Control (DAC) where users can grant access to their own files, MAC enforces system-wide security policies defined by administrators. Every process, file, directory, and port has a specific SELinux 'context' (or label). If a process with the context httpd_t tries to access a file labeled user_home_t, SELinux will block the action and log an avc: denied message, resulting in a Permission denied error at the application level.

Typical symptoms include:

• Nginx or Apache returning 403 Forbidden despite directories having 755 permissions.
• SSH keys being ignored, prompting for passwords instead.
• Database servers (MySQL/PostgreSQL) failing to start when data directories are moved to a non-standard location.
• Web applications failing to connect to external APIs or databases (13: Permission denied).

Step 1: Diagnose and Confirm SELinux Interference

Before changing system configurations, confirm that SELinux is actually blocking the action. Do not guess; the audit logs contain all the answers.

First, verify SELinux is currently enforcing policies:

getenforce
# Should output: Enforcing

Next, search the audit logs for Access Vector Cache (AVC) denials. The primary log file is /var/log/audit/audit.log.

You can search for recent denials using ausearch:

ausearch -m AVC,USER_AVC,SELINUX_ERR -ts recent

A typical AVC denial looks like this:

type=AVC msg=audit(1694523123.456:789): avc:  denied  { read } for  pid=1234 comm="nginx" name="index.html" dev="sda1" ino=56789 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0

Reading the Log:

• { read }: The action denied.
• comm="nginx": The process attempting the action.
• scontext=...:httpd_t:...: The source context (the process label).
• tcontext=...:user_home_t:...: The target context (the file/resource label).
• tclass=file: The type of resource.

In this example, the web server (httpd_t) is trying to read a file located in a user's home directory (user_home_t), which violates the default strict isolation policy.

To get human-readable explanations and suggested fixes, install and use setroubleshoot:

yum install setroubleshoot-server
sealert -a /var/log/audit/audit.log

Step 2: Fix File Context Issues

The most frequent cause of an avc: denied error is an incorrect file context. This commonly happens when files are created in one location (like /tmp or ~) and then moved (mv) to a destination like /var/www/html. Moving a file preserves its original SELinux context, while copying (cp) a file applies the default context of the destination directory.

Check the current context of a file or directory using the -Z flag:

ls -lZ /var/www/html/index.html
# Output might show: unconfined_u:object_r:user_home_t:s0

The Fix: Restore the default file contexts for that directory.

restorecon -Rv /var/www/html/

If you have configured a custom directory path (e.g., serving web files from /data/www), restorecon won't know the correct context unless you define it in the SELinux policy using semanage.

# Tell SELinux that /data/www and everything inside it should be labeled as web content
semanage fcontext -a -t httpd_sys_content_t "/data/www(/.*)?"

# Apply the new rules
restorecon -Rv /data/www

Step 3: Fix Policy Issues with Booleans

Sometimes the file contexts are perfectly correct, but an application is trying to do something the default policy forbids. For example, a web server might try to connect to a remote database, but SELinux explicitly prevents web servers from initiating outbound network connections to prevent them from becoming part of a botnet if compromised.

SELinux provides toggles called 'booleans' to enable these behaviors without writing custom policies.

To list all available booleans and their current states:

getsebool -a

Or search for specific booleans:

getsebool -a | grep httpd

If your web application needs to connect to a database or external API, you need to enable the httpd_can_network_connect boolean.

# The -P flag makes the change Persistent across reboots
setsebool -P httpd_can_network_connect 1

Commonly used booleans include:

• httpd_can_network_connect 1 (Allows web servers to make external network connections)
• httpd_enable_homedirs 1 (Allows web servers to read ~/.html user directories)
• ftp_home_dir 1 (Allows FTP daemons to read/write in user home directories)

Step 4: Create Custom Policy Modules (audit2allow)

If you are running a custom daemon or third-party application that does not have built-in SELinux policies or booleans, you will need to generate a custom rule. audit2allow reads the denial logs and translates them into an SELinux policy module that permits the exact actions that were blocked.

Warning: Only do this if you completely trust the application and understand why it is requesting access. Never blindly run audit2allow on a compromised system, or you will bake attacker persistence into your security policy.

1. Clear the audit log (optional, but helps isolate the issue):

   > /var/log/audit/audit.log
   

2. Put SELinux in Permissive mode temporarily so the application can run fully and log all required actions without failing midway:

   setenforce 0
   

3. Start the application and reproduce the intended workflow.
4. Generate the policy module from the logged denials:

   grep myapp /var/log/audit/audit.log | audit2allow -M myapp_custom_policy
   

5. Install the new policy module:

   semodule -i myapp_custom_policy.pp
   

6. Re-enable enforcing mode:

   setenforce 1
   

By following these diagnostic and remediation steps, you can maintain the robust security posture provided by SELinux without compromising system functionality.