Log Rotation Setup: How Not to Drown in Terabytes of Data

Every day, your servers generate a huge amount of logs. If you don't control this process, disk space can run out faster than you can say "infrastructure". In this article, we'll talk about how to set up log rotation to avoid drowning in terabytes of data.

What is Log Rotation?

Log rotation is the process of automatic management of log files: archiving, compressing, and deleting old entries. The goal is to free up disk space and maintain readability of log files.

Why Rotation is Important?

Imagine you have a web server serving a million requests a day. Each request creates a log entry, and if nothing is done, the file could grow to several gigabytes in a week. This would lead to:

  • Slower system performance
  • Disk space overflow
  • Difficulties analyzing logs
  • Possible data loss during failures

Rotation in Linux: logrotate

One of the most popular tools for log rotation in Linux is logrotate.

Installing logrotate

In most distributions, logrotate is already installed. If not:

```bash

Ubuntu/Debian

sudo apt-get install logrotate

CentOS/RHEL

sudo yum install logrotate ```

Main Configuration File

The logrotate configuration is located in /etc/logrotate.conf:

```bash

Example configuration file

compress # compress old logs delaycompress # delay compression until next rotation copytruncate # copy and truncate original file rotate 5 # keep 5 archives weekly # perform rotation weekly ```

Configuration for Specific Services

Often configurations for specific services are located in /etc/logrotate.d/. For example, Apache configuration:

/var/log/apache2/*.log { daily missingok rotate 14 compress delaycompress notifempty create 640 root adm sharedscripts postrotate if systemctl is-active apache2 > /dev/null; then \ systemctl reload apache2; \ fi; endscript }

Rotation Parameters

Here are the main parameters you can use:

  • daily/weekly/monthly — frequency of rotation
  • size N — rotation when size N is reached
  • rotate N — number of archives to keep
  • compress — compress archives
  • delaycompress — delay compression by one cycle
  • missingok — don't error if file doesn't exist
  • notifempty — don't rotate empty files
  • copytruncate — copy and truncate original file

Testing Configuration

Before applying changes, check the syntax:

bash sudo logrotate -d /etc/logrotate.conf

For forced rotation:

bash sudo logrotate -f /etc/logrotate.conf

Rotation in Windows

Windows doesn't have a built-in equivalent to logrotate, but there are several ways to implement log rotation:

PowerShell Scripts

Create a PowerShell script to archive and delete old logs:

```powershell

Archive logs older than 7 days

$OldLogs = Get-ChildItem "C:\Logs" -Recurse | Where-Object {$_.CreationTime -lt (Get-Date).AddDays(-7)} foreach ($Log in $OldLogs) { Compress-Archive -Path $Log.FullName -DestinationPath "$($Log.Directory)\$($Log.BaseName).zip" Remove-Item $Log.FullName } ```

Built-in Application Features

Many Windows applications support built-in rotation:

  • IIS: In site settings, you can set maximum log size and rotation frequency
  • SQL Server: Transaction log rotation settings via SQL Agent
  • Exchange: Built-in log archiving policies

Using Third-Party Tools

  • LogRotate for Windows — port of the original logrotate
  • Robocopy — for copying and archiving files
  • Task Scheduler — for running rotation scripts on schedule

Practical Recommendations

Determine Log Importance

Not all logs are equally important:

  • Critical logs (authentication, security) — store longer (1-2 years)
  • System logs — medium term (3-6 months)
  • Application logs — shorter (1-3 months)
  • Debug logs — minimum time (1-2 weeks)

Monitor Free Space

Set up disk space monitoring:

```bash

Example script to check space

df -h | awk '$5 > 80 {print $1 " is full at " $5}' ```

Archiving Important Logs

For long-term storage, use external solutions:

  • Cloud storage (AWS S3, Azure Blob Storage)
  • Archive to separate server
  • Use WORM media for legal requirements

Conclusion

Proper log rotation setup is not just about saving disk space. It's an important part of infrastructure management strategy, ensuring both system efficiency and the possibility of analyzing historical data when needed.

Remember that there's no universal recipe — settings should depend on system load, data importance, and business requirements. Regularly review rotation policy and adapt it to current needs.