If you’ve ever run a fairly busy webserver on FreeBSD -or you just happen to have particularly badly written code spamming the error log, such as some wordpress based plugin spaghetti- you probably found yourself into the situation where your logs grow out of proportion. Or maybe you ran software that went into some sort of endless restart loop until the disk got full (happened to me!).
It could be simply that your nginx or mysql logs grew so big that it takes 10 seconds just to open them in putty. Of course you could just wipe them out and start from scratch, but a cleaner solution is to do with them what FreeBSD does with its own system logs: rotation.
Rotation involves deleting the log file based on a certain condition being met (size, time or both), optionally storing it for reference, and replacing it with a new blank file; and most important, telling the process about this new file. And why would we do that?
Well, once a file is opened for I/O by a process, it is referenced by a file descriptor, not by its name. This means that if we delete a log file, the process may simply stop logging, unless it includes some specific code to deal with impolite admins who do this. This is why using newsyslog is important.
Newsyslog -which, as it tends to happen in the Unix world, is not really that new- is the daemon responsible for the logging feature. It can rotate the current log to a backup file (such as myserver.log.1) once it reaches a certain size. We can truncate or compress it, and we can do it every day or every week. The configuration file storing the system’s defaults is /etc/newsyslog.conf. Entries are structured as follows:
- logfilename, owner:group and mode are self explanatory
- count: the number of rotated log files to keep.
- size: how big a log must become before it gets rotated. If empty, logs won’t be rotated based on size.
- when: how often to rotate a log. If empty, logs won’t be rotated based on this criteria. See the accepted values here.
- flags: some of the important ones. For a full list see here.
B: log is not text, do not append rotation message.
C: create log file if it doesn’t exist.
E: do not rotate if empty.
G: filename is a shell pattern, such as php*
J: compress with bzip
Z: compress with gzip
-: a placeholder in case we are not using any of the flags
- the last two are optional, but path_to_pid_cmd_file is fundamental if we want to add our own entries, as here we will tell syslog the path to the pid file of the process we are logging.
I suggest to practice changing the rotation settings here, reducing the number of copies to keep (we don’t know what these logs are for anyway) or disabling bzip compression (we are not THAT short in disk space). Running service newsyslog restart will apply your changes. After getting the gist of it, we can create our own configuration files in /etc/newsyslog.conf.d/<processname>.conf
As an example, here is how we rotate NGINX logs every day at midnight, and do not keep any old log:
We will add these lines; remember to separate with tabs for clarity:
/var/log/nginx-access.log www:www 644 1 * @T00 C /var/run/nginx.pid
/var/log/nginx-error.log www:www 644 1 * @T00 C /var/run/nginx.pid
Second example we rotate the MySQL logs every time they hit a megabyte of size, keeping 3 logs compressed with gzip:
/var/log/mysql/error.log mysql:mysql 644 3 1024 * CEZ /var/run/mysql/mysqld.pid
(Before using these examples, do check that the logs and the pid file are actually there – these may change for different versions of the software in question)
And now, how do we know if we did it properly without staring at the log for hours on end?
Well, ever seen that weird message that sounds like a prank from some 80s movie, “You have new mail”? The system will send us a message if something is wrong, but only if we set a mail address for the root mail to be forwarded to.
By default FreeBSD sends error messages to the root account itself which we can read in the console using the “mail” command but since this is quite primitive and unconvenient let’s have all mail sent to our e-mail:
Below this text:
#Pretty much everything else in this file points to "root", so #you would do well in either reading root's mailbox or forwarding #root's email from here.
Add this line:
root: <your email address>
Save and exit to the shell to apply the new settings:
Ready! Now you should receive all the system email in your own mailbox… assuming mail is properly set up in your box. But that’s a topic for another article.