AWS ECS logging with Papertrail

I've been spending a large amount of time with setting up a modern infra stack with lately, which pretty much involves Github + Docker + CircleCI + AWS ecosystem. What?!? No Kubernetes?! Iono...I just didn't feel like setting up Kubernetes since "container orchestration" is not really an issue for me right now. As long as I containerize my apps, orchestration will be a scaling problem for future Kelvin~~ (and if scaling is really the only issue I have to worry about, then that's a good problem to have).

So instead of Kubernetes, I'm leveraging Amazon's Elastic Container Service (ECS) to manage, deploy, and orchestrate my Dockerized apps. When setting up your container, you have the ability to configure the logging mechanism you use when your container is running. In ECS, if you don't define a different logging mechanism, it'll default to AWS Cloudwatch through Docker's awslogs driver. However, like most AWS products, UI is not necessarily their forte. So Cloudwatch isn't super easy to navigate, set up rules, or even use without your eyes bleeding. I wanted to use a different logging tool that was easier to navigate, provided color-coding, more intuitive search/filter options, and could easily set up alerts.

Here comes Papertrail

Papertrail is a very popular log management tool and is one of my favorites (especially since it's free-tier is very generous). Papertrail already provides instructions on how to set up logging with Amazon ECS, but that requires spinning up another Docker container that runs logspout - a log router.

Because I'm a cheapass (as mentioned many a times before), I didn't want to use up server resources just to route logs. I'm sure this is a more ideal solution, but my cheapness will always win out. Instead, there is a more direct way to immediately log to Papertrail with zero fuss and minimal setup.

Docker has another logging driver called syslog. You can read more about that here, but in layman's terms, as long as you provide a syslog-address with the address of the external syslog server, any logs you output to your console will be sent over there. Papertrail supports this. So for this post, here are simple instructions on how to log to Papertrail through your Amazon ECS containers.

Instructions

Papertrail

1. Create an account on Papertrail
2. Keep note of the log destination URL, which should look something like: logs2.papertrailapp.99999. If you can't find it, try going to Settings > Log Destinations

AWS ECS

1. Open your AWS console and navigate to Elastic Container Service
2. Click Task Definitions and then click Create new task definition (or modify an existing one) and select a launch type
3. Click Add Container
4. Scroll until you find STORAGE AND LOGGING
5. For log configuration, change it to syslog
6. For log options, you'll want to add for the key: syslog-address, and for the value: udp://logs2.papertrailapp:99999 (which is your log destination URL from Papertrail prefixed with the udp transport)
7. I'd also recommend adding a tag, which will be the visible name for any log entries from this container (read up on the acceptable tag options).

At the end, it should look something like this:

And that's it! Your logs should now stream from your Docker container to Papertrail. I'd still recommend enforcing good logging practicing (i.e. creating log files and putting them on S3), but that's a post for another day...

This was a simple, straightforward post, but one that has a surprising lack of documentation online. Actually, there's a lot of lacking documentation for Amazon ECS and integrating it with your other apps/services. So you'll probably be seeing a lot of specific posts that involve this ecosystem in the future - simply because I had to go through a bit of hurdles to get all of these things to work well together. But for now, happy logging!