< index

GCE logs via PubSub to Loggly

· 6 minute read

I use Google Container Engine (GCE) to deploy Docker containers to a managed kubernetes cluster, building my containers to log to stdout. To access the logs of a container I can use kubectl logs <pod_name>:

$ kubectl logs gauge-bigquery-0n7ul
{"event":"runtime.memory","heap":4.5684e+06,"timestamp":"2016-06-02T00:17:28.580723809Z"}
...

Manual and not that useful. Like many I’m a fan of aggregating structured machine-readable logs to achieve useful insight through dashboards and search. Google Cloud Platform at time of writing had Stackdriver Logging Service in beta in this space, but when I experiemented it had too limited a dashboarding interface. I use Splunk at work but for my log volume (~1Gb/day) it’s too expensive for a self-funded hobby system; setting up my own ELK stack was an option but I didn’t want the maintenance overhead; so I settled on Loggly as best value for what I needed.

How to get logs from GCE containers to Loggly?

A GCE Kuberenetes cluster already has Fluentd running on each node, packaged with various specific Google config and add-ons as the google-fluentd package. I could hook into that, adding some config that would forward logs to Loggly. But… that would break my use of GCE as-a-service as I’d need to worry about setup per-node: scaling the cluster up and down, replacing failed nodes and so on all becomes harder.

While I wasn’t completely happy with StackDriver as the whole solution, it does offer the ability to export logs from an GCE cluster to Google Cloud PubSub (a publish/subscribe message queue service). In 15 mins I had all my cluster logs available as a stream of messages on a PubSub topic and a possible solution was coming together:

The last piece in this chain I had to manage myself: a PubSub subscriber that uploaded to Loggly. I tackled this in Golang and Dockerized it so I could deploy it as a container on the GCE cluster itself:

PubSub to Loggly Golang (github)

Circle CI

Docker Repository on Quay

Gotcha: Re-inforcing log loop

My uploader not only takes a stream of cluster log events as an input from PubSub, it also needs to log it’s own operational health (a log event that will come back to it via PubSub to be uploaded to Loggly). Over-zealous logging from this application itself could cause a re-inforcing run-away logging loop. For example if every message received from PubSub resulted in an error that was logged into 2 log lines:

I did consider having this application simply to log directly to Loggly rather than stdout to prevent this, but that renders other normal log viewing use-cases awkward (e.g. what happens when the application can’t connect to Loggly?). To mitigate I made sure this app logged just the essentials, logged errors per-batch rather than per-message, and added defense-in-depth sleep clauses on error to prevent a run-away log explosion. Some sort of exponential backoff on error would probably be best, but I figured “don’t let perfect be the enemy of good”.

Gotcha: GCE cluster must have PubSub permission

Consuming PubSub in Golang was the biggest challenge as there wasn’t much information or examples of how to approach this. My pubsub.go file shows the basic Go steps to subscribe to Google PubSub. In addition, your Google compute instances must have the

The Google Container Engine cluster must also have the required permissions () to access the PubSub service. A caveat is that the permissions of a Google Container Engine kubernetes cluster can’t be changed after it’s creation so make sure to include your likely service usages up front: --scopes if using gcloud container clusters create or via checkboxes options in the console UI. You can check your current scopes via the console UI or checking your oauthScopes:

$ gcloud container clusters describe CLUSTER_NAME --zone=ZONE_NAME
... <snip> ...
nodeConfig:
  diskSizeGb: 100
  machineType: n1-standard-1
  oauthScopes:
  - https://www.googleapis.com/auth/compute
  - https://www.googleapis.com/auth/devstorage.read_write
  - https://www.googleapis.com/auth/logging.write
  - https://www.googleapis.com/auth/pubsub
  serviceAccount: default
... <snip> ...

References

StackDriver Log Export to PubSub

PubSub to Loggly Golang (github)

Circle CI

Docker Repository on Quay

Related...