I’ve recently released a new website on AWS Elastic Beanstalk (EB) and thought I would document how I’ve set up centralised logging using papertrailapp.
The following steps use Amazon’s configuration files to customise the software stack running on an EC2 instance.
- Get syslog pushing entries into papertrailapp.
- Get other log files pushing to papertrailapp.
- Encrypt the logs during transport
Step 1: Get syslog pushing entries into papertrailapp
Knowing that the EC2 operating system is the Amazon Linux AMI we know that we are using rsyslog as our system logger. With this knowledge we can move straight into editing it’s config file.
The next step is to append a papertrailapp line along the lines of
*.* @logs.papertrailapp.com:12345 to the bottom of the /etc/rsyslog.conf file, changing your port number as appropriate. Since it would be pointless doing this directly due to the very nature of an elastic provisioned architecture we are going to use the EB config files so we only have to do this once and once only.
Copy the contents from /etc/rsyslog.conf into a new file and append the papertraillapp log line to it. Now push this up to some place publicly accessible for download, such as Amazon’s S3 service.
Now within the root directory of your application, create a config file called .ebextensions/papertraillapp.config and paste into it the following content:
Obviously, change the source URL to point to the rsyslog file you uploaded earlier.
With the change to the config file we just need to restart the rsyslog service to apply our changes. To do this, update the .ebextensions/papertraillapp.config file by appending:
This basically watches for changes to the /etc/rsyslog.conf file and will automatically restart rsyslog when the file is modified. This will also ensure the service is always running after a system boot as well as any deployments.
Now commit this file and push to EB with
git aws.push. Shortly after you should receive the usual confirmation email from EB that your deployment is successful. Within your papertrailapp account you should now see newly appended content to your syslog file.
Step 2: Get other log files pushing to papertrailapp
With papertrailapp now receiving our syslog entries, we need to expand this to other log files such as Apache errors and access logs, application logs and any others of interest.
remote_syslog is a service that will monitor for changes in the log files we tell it about and will send any of those changes to a destination we also tell it about, in this case, papertraillapp. To install this, add the following lines to your .ebextensions/papertraillapp.config file:
Basically, we are setting up instructions to install the remote_syslog service using RubyGems. Empty brackets denote the latest release.
Log file setup with remote_syslog
With remote_syslog installed, we now need to tell it which log files to watch and send over to papertrailapp. To do that create a new file called log_files.yml. For the purpose of this guide, we will set up entries for Apache and PHP, so please craft as needed. The contents of this file are:
This should be self-explanatory but the files block simply lists the absolute paths to the logs you wish to watch and the destination block points to where you want to send any new entries to, in this case to papertrailapp. Now you just need to upload this to a reachable Internet address such as Amazon’s S3 for reference later.
By default, the Apache logs aren’t readable by the remote_syslog service and PHP needs the /var/log/php directory created to write to, to correct this, add the following to your .ebextensions/papertraillapp.config file:
Here we are creating a couple commands to achieve our goal. We give them suitably descriptive names and assign the command we want invoked at the shell to the command keyword. To help make this indempotent and ensure it runs correctly we assign a test command to the test keyword.
Now to get our /etc/log_files.yml config file on the server we add a new entry to our .ebextensions/papertraillapp.config within the files block as shown below:
Given the source value, change as needed, our log_files.yml file will be downloaded and written to /etc/log_files.yml. Now we just need to watch for changes to this file and restart the remote_syslog service when this happens. To do so simply add to the services block within the .ebextensions/papertraillapp.config file:
Making remote_syslog bootable
The one final task to do here before we commit and deploy is to ensure the remote_syslog service runs at boot. To do this, update the commands block with the following:
Again, we give a meaningful name to our action, ensure a test is run before that action is invoked, in this case if the init.d script is not already present within the init.d directory then copy across the remote_syslog init.d script. The regular expressions in the above example take into account version changes to the 1.x branch.
Now we can commit and deploy with
git aws.push and see the fruits of our labour within papertrailapp.
Step 3: Encrypt the logs during transport
Finally, we want to ensure that our logs are encrypted during transport to papertraillapp. To do this we want to make sure the rsyslog-gnutls package is installed. To do so add the following to your packages block within .ebextensions/papertrailapp.config:
This will install the latest release from the yum repository.
Next we need to ensure we have a root certificate saved locally on the server. To get this in place add the following to the files block of the .ebextensions/papertraillapp.config file:
The source referenced here could easily be downloaded and saved into the same place as the other files used so far in this guide. It’s your call.
Next we need to replace the line we appended at the bottom of the /etc/rsyslog config file with the following:
Remember to change the port as needed.
Finally, commit and push to EB.
Verify the traffic is encrypted
To ensure our log data is encrypted during transport to papertrailapp. Run sudo tcpdump -s 1500 -X src or dst logs.papertrailapp.com and trigger a log message through both syslog and remote_syslog. Once the necessary handshaking is complete all data should be encrypted with no clear text present. If this is not the case restart the relevant service(s).
Everything should now be in place so all of the log messages generated on the EB application will appear within papertrailapp. The next sensible step to take would be to setup some alerts on papertrailapp so notifications are received when any pertinent events occur or possibly to send any relevant events to a graphing solution such as Librato.
For reference the full paperrtraillapp.config file can be seen below:
Hopefully, that demonstrates how to customise an AWS EB software install without having to manually customise the server through the command line. Once you get your head around it and get your first one or two customisations done this way it’s quite easy. However, I would limit the amount of customisation you need to do as you will soon encounter limitations. If you know ahead of time that you will need to customise the server quite a bit then check out AWS OpsWorks.