How to Fix 'systemctl failed' and 'systemctl permission denied' Errors in Linux

Understanding the Error

When managing Linux servers, encountering a systemctl failed error is a rite of passage for any DevOps engineer or Sysadmin. systemctl is the central management command for systemd, the init system that bootstraps user space and manages system processes on most modern Linux distributions, including Ubuntu, Debian, RHEL, CentOS, and Arch Linux.

When a service fails to start, restart, or crashes unexpectedly during runtime, systemd catches the failure, terminates any hanging subprocesses, and flags the service state as failed. If you try to interact with it, you will likely see the infamous message:

Job for <service>.service failed because the control process exited with error code. See "systemctl status <service>.service" and "journalctl -xe" for details.

This guide provides a comprehensive, step-by-step methodology to diagnose systemctl not working, analyze the systemctl failed log, and resolve restrictive systemctl permission denied errors.

Step 1: Diagnose the Failure

Before you apply fixes, you must identify exactly what failed and why. Blindly restarting services rarely solves the root cause and can exacerbate data corruption issues.

1. Identify Failed Services

To get a high-level overview of every service, socket, mount, or device that has failed on your Linux system, run:

systemctl --failed

This command isolates units in the failed state, filtering out the noise of the hundreds of healthy systemd units running in the background. It provides the UNIT name, LOAD state, ACTIVE state, and SUB state.

2. Inspect the Service Status

Once you have identified the culprit (for example, nginx.service, postgresql.service, or kubelet.service), inspect its current status:

systemctl status <service_name>.service

Look closely at the Active: line. It will likely say failed (Result: exit-code). More importantly, pay attention to the Main PID exit code. For instance, code=exited, status=1/FAILURE indicates a general application error, while status=210/CHROOT or status=203/EXEC can immediately point you to specific systemd execution failures.

3. Analyze the systemctl failed log

The status command only truncates and displays the last ten lines of the log. To get the full context of why the service crashed, you must query the systemd journal:

journalctl -xeu <service_name>.service

The -x flag appends explanatory text from the journal catalog (if available), -e jumps to the end of the pager (displaying the most recent events), and -u filters the logs by the specific unit. Scan these logs for application-level syntax errors, missing dependencies, or port binding conflicts (e.g., Address already in use).

Step 2: Fix Common Root Causes

Scenario A: Configuration Errors

The most common reason a service fails to start is a typo, syntax error, or misconfiguration in the application's configuration file (e.g., /etc/nginx/nginx.conf or /etc/ssh/sshd_config).

• Fix: Do not rely on systemd to tell you what is wrong with the application's config. Run the application's built-in syntax checker. For Nginx, use sudo nginx -t. For Apache, use sudo apachectl configtest. For SSH, use sudo sshd -t. Fix the reported syntax error, and then execute sudo systemctl restart <service_name>.

Scenario B: systemctl permission denied

You might encounter a Failed to restart service: Access denied or Interactive authentication required error when running systemctl commands directly.

• Root Cause 1: Missing Privileges. Modifying system state requires root privileges. Always prefix commands that change state (start, stop, restart, enable, disable) with sudo.
• Root Cause 2: Polkit Issues. systemd uses Polkit (PolicyKit) to manage privileges. If Polkit is crashed, missing, or misconfigured, non-root users (even those in the sudo/wheel group) might receive permission denied errors. Check the Polkit status: systemctl status polkit.service.
• Root Cause 3: File Permissions (Execution Phase). If the error is found within the journal logs stating the service itself encountered a "Permission denied" (often exiting with code 210, 213, or 214), the service user (e.g., www-data, postgres) lacks read/execute access to its required binaries, certificates, or directories. Use chown and chmod to correct ownership of the paths mentioned in the logs.
• Root Cause 4: Mandatory Access Control. Do not forget to check SELinux (sestatus, audit2allow -a) or AppArmor (dmesg | grep apparmor). These security modules enforce MAC and can silently block processes from accessing files or ports, even if standard POSIX file permissions are correct.

Scenario C: systemctl not working or hanging

If running systemctl commands hangs indefinitely or returns Failed to connect to bus: No such file or directory:

• Fix 1: WSL or Docker environments. Standard Docker containers and older Windows Subsystem for Linux (WSL1) environments do not run systemd as PID 1 by default. If you are in a standard container, you cannot use systemctl; you must run the daemon directly. If you require systemd inside Docker, you must run an image explicitly configured for it, utilizing --privileged mode and specific volume mounts for cgroups.
• Fix 2: D-Bus is down. systemctl communicates with systemd via D-Bus. If dbus.service has crashed or is masked, systemctl commands will hang. You may need to restart the D-Bus socket or reboot the server.

Step 3: Advanced Troubleshooting (cgroups and strace)

When journalctl does not provide enough context, and the service repeatedly and silently crashes, you must dive deeper into system internals.

1. Inspecting cgroups (Control Groups) and OOM

systemd utilizes cgroups to organize processes and manage resource limits (CPU, memory, I/O). A service may fail because it hits an Out-Of-Memory (OOM) limit defined in its unit file (MemoryLimit=) or by the kernel.

Run systemctl status <service_name> and look for the Memory: metric. If it is hitting a limit, check the kernel logs for OOM killer invocations using dmesg -T | grep -i oom. If the kernel is killing the process, you will see Killed process <PID> (<process_name>) in the output. You must either increase the memory limit in the service file, optimize the application's memory usage, or add swap space to the server.

2. Tracing the Execution with strace

If a service dies immediately upon startup and the systemd logs are entirely empty, the process might be failing before it even attaches to standard output or the journal.

To debug this, bypass systemctl temporarily and run the service binary manually using the exact same user and arguments defined in the unit file, prefixing the command with strace to intercept system calls.

First, extract the execution command and user from the unit file: cat /lib/systemd/system/<service_name>.service | grep -E '^ExecStart|^User'

Next, execute it manually (assuming User=www-data and ExecStart=/usr/sbin/nginx -g 'daemon off;'): sudo -u www-data strace -f /usr/sbin/nginx -g 'daemon off;'

The strace output will flood your terminal, but the final lines right before the process exits will reveal the exact system call that failed. You will typically see an EACCES (Permission denied) when the process tries to open a file, or ENOENT (No such file or directory) when a critical configuration file or shared library is missing.

3. Handling Timeout Errors

Sometimes, systemctl start hangs and eventually fails with a Timeout setting up VIP or start operation timed out error. This occurs when systemd expects a specific notification from the process (if Type=notify) or expects it to fork (Type=forking), but the application takes too long or fails to signal readiness entirely.

• Fix: Evaluate if the application genuinely needs more time to initialize (for example, a massive database restoring data from disk). You can increase the timeout threshold by adding TimeoutStartSec=600 to the [Service] block in the unit file.

Step 4: Cleanup and Recovery

Once you have resolved the underlying issue (fixed the configuration syntax, corrected file permissions, or disabled SELinux blocks), you must instruct systemd to attempt starting the service again.

1. If you modified the .service unit file itself (e.g., adding a timeout in /etc/systemd/system/), you MUST reload the systemd manager configuration so it reads the new file: sudo systemctl daemon-reload
2. Restart your service: sudo systemctl restart <service_name>.service
3. Clear the failure state. If the service is running successfully but systemd still shows a residual failed flag from a previous crash when running systemctl --failed, reset the counters: sudo systemctl reset-failed

By systematically checking systemctl status, digging into the journalctl logs, verifying permissions, and using strace when necessary, you can reliably resolve almost any systemd service failure and ensure high availability for your Linux infrastructure.