## Introduction

Logrotate is a valuable utility used to manage log files on Linux systems. By rotating logs regularly, it helps to save disk space and avoid storage issues. However, even a simple typo in the logrotate configuration file can have significant consequences, leading to disk space depletion and potential system failures. In this blog, I will explore a real-world scenario where a typo in logrotate caused the /storage/log directory to run out of disk space, triggering a vCenter root partition storage health check failure. I will then demonstrate how to identify and rectify the issue to ensure the proper functioning of log rotation.

 

## The Problem

Upon encountering the “vCenter root partition storage health check failed” error, I immediately investigated the root cause. After running the `df -Th` command, I identified that the /storage/log directory was utilizing 100% of the available disk space.

 

## The Investigation

To investigate further, I checked the logrotate configuration file responsible for managing the VMware vCenter Server Appliance log files. The configuration file, `/etc/logrotate.d/vmware-vmafd.lr`, was found to contain the following settings:

/var/log/vmware/vmafdd/vmafdd.log
{
size 10M
nodateext
rotate 10
missingok
notifempty
compress
create 644 vmafdd-user lwis
postrotate
/usr/sbin/killproc -L -HUP /usr/lib/vmware-vmafd/sbin/vmafdd
endscript
}

 

## The Misconfiguration

After carefully reviewing the configuration, I identified a crucial typo in the log file path. Instead of `/var/log/vmware/vmafd/vmafdd.log`, the path was incorrectly set to `/var/log/vmware/vmafdd/vmafdd.log`. As a result, logrotate was unable to locate the log file and couldn’t rotate or compress it, leading to log files continuously growing in size and filling up the disk space.

 

## The Solution

To rectify the issue, I corrected the typo in the logrotate configuration file by changing the log file path to the correct one:

/var/log/vmware/vmafd/vmafdd.log
{
size 10M
nodateext
rotate 10
missingok
notifempty
compress
create 644 vmafdd-user lwis
postrotate
/usr/sbin/killproc -L -HUP /usr/lib/vmware-vmafd/sbin/vmafdd
endscript
}

 

## Testing the Configuration

Before applying the new configuration, it is essential to test it using the `-d` flag to enable debugging mode. This mode allows us to see how logrotate will operate without making any actual changes:

logrotate -d /etc/logrotate.d/vmware-vmafd.lr

If the debug output confirms that the configuration is now correctly pointing to the log file, I can proceed with applying the changes:

logrotate /etc/logrotate.d/vmware-vmafd.lr

After executing the logrotate command, the log files should be rotated, compressed, and the disk space issue should be resolved. To verify this, run `df -Th` to check the disk space usage again.

 

## Conclusion

A simple typo in logrotate configuration can have severe consequences, leading to critical errors like the “vCenter root partition storage health check failed.” It is crucial for administrators to be meticulous when configuring logrotate settings to prevent such issues. Regularly reviewing and testing logrotate configurations can help avoid storage-related problems, maintain system health, and ensure smooth operations of log management.

Remember, a small typo can cause big problems. So, always double-check your configurations and maintain a proactive approach to log management to keep your systems running smoothly.