cloud-test: Analyzing your Google App Engine Logs with Google BigQuery

Google Cloud Platform Blog

Analyzing your Google App Engine Logs with Google BigQuery

Monday, July 23, 2012

Developers know that logging and logs analysis can often mean the difference between delighting and disappointing users. With the Google App Engine LogService API, it’s easy to add logging to your App Engine App with just a few lines of code. But of course, logging events is only the beginning, and today we’re particularly excited to highlight using Google BigQuery to analyze your App Engine logs.

Google BigQuery is an externalized version of Google’s own logs analysis framework that allows developers to run queries across billions of rows of data in seconds via a RESTful API. BigQuery uses a familiar SQL-like query language and is able to scale to datasets that are terabytes in size and beyond. We see BigQuery as a natural fit for logs analysis, and at I/O this year, our developer relations team led a codelab demonstrating how to import and analyze App Engine logs with BigQuery.

Our customers have also had success with this technique, and App Engine developers at Streak.com posted their own walkthrough and Java framework, called Mache, for automatically exporting App Engine logs into BigQuery. Mache provides a simple interface for scheduling cron jobs that parse and ingest log file data into BigQuery at user-defined intervals.

If you’re interested in trying out Google BigQuery with App Engine, check out the getting started guide and the sample code from our I/O codelab. Happy logging!

- Posted by the Google App Engine Team

Also, if you’re interested in analyzing Datastore data in BigQuery, check out our article that shows how to use App Engine MapReduce to manage the transformation and export of Datastore entities.