Ubuntu in production: Logrotate

This is the first post of a series of useful server configurations. We use Ubuntu in many production environments, and so these posts will be geared towards Ubuntu Server (12.04).

Some frameworks have their own logging in place (CodeIgniter). Some custom applications may have custom loggers (Monolog!). In cases such as this, you should set up rotating logs and a log backup. Logrotate is there to do this for you.

What does Logrotate do?

Logrotate periodically reads your log files, backs up older ones and creates a new log. This helps prevent log files from getting unwieldy in size. It also will delete old log files so as not to fill your server memory.

Many apps setup logrotate for you. For instance, installing tasksel's lamp-server in Ubuntu adds the file /etc/logrotate.d/apache2, which is a configuration for Logrotate to rotate all apache access and error logs.

Configuring Logrotate

In stock Ubuntu, any config file you put into /etc/logrotate.d is going to run once per day. Logs are typically rotates once per day or less (Apache default in Ubuntu is in fact weekly).

Default Apache

Let's look over Apache's default in Ubuntu - /etc/logrotate.d/apache2.

/var/log/apache2/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
            /etc/init.d/apache2 reload > /dev/null
        endscript
        prerotate
            if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                    run-parts /etc/logrotate.d/httpd-prerotate; \
            fi; \
        endscript
}

This will rotate any files in /var/log/apache2 that end in ".log". Let's go through the options.

weekly: Rotate logs once per week

missingok: If no *.log files are found, don't freak out

rotate 52: Keep 52 files before deleting old log files (Thats a default of 52 weeks, or one years worth of logs!)

compress: Compress (gzip) log files

delaycompress: Delays compression until 2nd time around rotating. As a result, you'll have one current log, one older log which remains uncompressed, and then a series of compressed logs. More info on its specifics.

notifempty: Don't rotate empty files

create 640 root adm: Create new log files with set permissions/owner/group

sharedscripts: Run postrotate script after all logs are rotated versus once per each log.

postrotate: Scripts to run after rotating is done. In this case, Apache is reloaded so it writes to the newly created log files.

prerotate: Run before log rotating begins

Here's what I have for a CodeIgniter app in production, which uses Mongolog rather than the default logger.

/var/www/my-app/application/logs/*.log {
daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 660 appuser www-data
    sharedscripts
    dateext
    dateformat -web01-%Y-%m-%d-%s
    postrotate
        /etc/init.d/apache2 reload > /dev/null
        /usr/bin/s3cmd sync /var/www/my-app/application/logs/*.gz s3://app_logs
    endscript
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi; \
    endscript
}

The differences:

daily: This writes daily, since we expect a decent amount of traffic.

rotate 7: Keep only the last 7 days of logs in the server.

create 660 appuser www-data: New log files are owned by appuser, which is the user used to deploy files to the server. The log files are in group www-data, the same group that Apache runs as. The permissions (660) allow both owner and users of the same group to write to the file (appuser and apache users can then edit these files). This let's the CI app write to the log files! (Apache can write to the file as its in the same group, www-data. User appuser, as owner of the log files, can edit file if needed during or post-deployment).

dateext: Logs by default get a number appended to their filename. This option appends a date instead.

dateformat: The format of the date appended to the log filename you want. These logs also add web01, web02 (and so on) to the log file name so we know which webserver the log came from. This is needed if you are logging on multiple web servers behind a load balancer and will combine the logs at a later date.

postrotate: Note that I'm saving log files to Amazon S3 using the s3cmd CLI tool. Using 'sync' is similar to rsync - It will overwrite files in S3 with newer files. This is especially why differentiating log file names between web01, web02, etc is necessary. This is some magic right here - Both giving you off-site backups to your log files, and not taking up unnecessary server space.

The two most useful things here: Creating the log files with the correct permissions and saving log files to Amazon S3 for off-site backup.