cloud-test: Leverage the power of log data to drive operational and business decisions through Google Cloud Logging

Google Cloud Platform Blog

Leverage the power of log data to drive operational and business decisions through Google Cloud Logging

Thursday, March 19, 2015

Google Cloud LoggingGoogle Compute EngineGoogle App EngineGoogle Cloud Monitoring

Ingest and view the log data, so that you can see all your logs in one place

Search the log data in real-time, so that you can resolve operational issues

Analyze the log data in real-time, so that you can glean actionable insights

Archive logs data for longer periods, to meet backup and compliance requirements

At Wix we use BigQuery to analyze logs of Compute Engine auto-scaled deployments. We get a large volume of syslog data that we send to BigQuery to get insights on system health state and error rates. We generate time series data and integrate it with Google Cloud Monitoring to monitor system performance and business metrics. This provides us with essential insight for the running of our operations. - Dmitry Shestak, Engineer@Infrastructure team, Wix
Ingest and view the log data

Compute Engine VM logs can be automatically collected for about two dozen log types through the Google packaged fluentd agent, with additional logs possible through custom configuration.

Compute Engine Activity logs record all system actions and API calls are enabled by default, with no agent installation required.

App Engine logs that include syslog, request logs and application logs are automatically enabled for all App Engine projects, including applications using Managed VM runtimes.

Google Developers Console

When viewing logs in the Logs Viewer, you can filter results using filter text or drop-downs

Search the log data in real-time

A filtered view of logs data using the Logs Viewer

Analyze the log data in real-timeGoogle BigQueryExportsdetailed documentation

Log data in the BigQuery tables

Monitoring Code Performance: There are situations when something unexpected happened or is indicative of an imminent problem e.g. “disk space low.” With Compute Engine log data in BigQuery, you can generate a time series and monitor logs with a particular severity. It’s simple, you just query metadata.severity = “WARNING” in the relevant tables. E.g.

SELECT Count (*) AS total, Date(metadata.timestamp) AS time FROM   (Table_date_range(TABLE ID,  Timestamp('2015-03-01'), Timestamp('2015-03-12')))  WHERE  metadata.severity = "warning"  GROUP  BY time ORDER  BY total;

Monitoring Request Latency: High latency leads to poor user experience and failed requests, which can lead to frustrated users and lost revenue. With App Engine log data in BigQuery, you can create time series of latency data by aggregating and charting the “protoPayload.latency” field. You can see unusual latencies in real-time and take steps to resolve the issue.

Archive logs data for longer period Google Cloud StorageGoogle Cloud DataflowHadoop Google Cloud Storage Nearlinemore affordable
Getting StartedCloud Logging documentation pagefeedback