cloud-test: Near Real-Time Log Streaming and Analysis with Google Cloud Platform & Logentries

Google Cloud Platform Blog

Near Real-Time Log Streaming and Analysis with Google Cloud Platform & Logentries

Monday, June 22, 2015

At Google we spend a lot of time thinking about how we can make DevOps easy for Google Cloud Platform customers. Whether you are using Google App Engine, Google Compute Engine, or any other service, you want access to logs produced by your system and applications.

Google Cloud Platform delivers support for centralized logging via Google Cloud Logging which provides you with the ability to view, search, and analyze log data. Cloud Logging includes the capability for log archival in Google Cloud Storage and the ability to send logs to Google BigQuery. In addition, Cloud Logging also allows you to forward these logs to any custom endpoint including third party log management services for advanced and tailored log analytics via the near real-time streaming Google Cloud Pub/Sub API.

We are happy to announce a real-time integration of Logentries, a third party log analytics service, with the Google Cloud Platform. Log Management and Analytics is a critical customer need and we are excited to offer Google customers a choice to easily send logs to a key provider like Logentries. This integration offers Google Cloud Platform customers an easily configurable choice for log management and advanced analytics that includes anomaly detection. Customers can now use Logentries for Google App Engine and services like Cloud Dataflow as well and makes it even easier to get started. At Google, we are committed to creating an open ecosystem with easy path of integration for partners, and Logentries provides a great example of a leading partner.

"Thanks to Google Cloud Logging export feature to Pub/Sub, it was easy to build a direct integration between Logentries and Google Cloud," explained Marc Concannon, VP of Product at Logentries. "The Pub/Sub API was well documented and the Google's commitment to developing an open collaboration made the integration smooth”.

Overview of the Google Cloud Pub/Sub API
Cloud Pub/Sub is a powerful messaging service responsible for routing data between applications at scale that delivers notifications within milliseconds, even when handling more than 1 million messages per second. In essence it is a near real time many-to-many, asynchronous messaging service that helps to create simple, reliable, and flexible applications by decoupling senders and receivers. It allows for secure and highly available communication between independently written applications.

Cloud Pub/Sub is thus an ideal service for transporting your logs and it allows you to either push your log events, or pull them as they happen.

Figure one: Google Cloud Pub/Sub Data Flow Schema

Logentries, a near real-time log analytics service, is the first third party service to integrate with Google Cloud Pub/Sub near real-time log streaming, allowing users to configure alerts, perform anomaly detection as well as advanced analytics.

How To Configure Logentries with Google Cloud Logging
Streaming Google Cloud Platform logs to Logentries can be configured as follows:

Enable the Cloud Pub/Sub API

Add the Logentries Service Account to your project

Configure Export to Cloud Pub/Sub

Add a Log in Logentries

Step by Step instructions are available to get configured quickly.

Near Real-Time Log Analytics
Logentries uses a unique pre-processing engine to perform advanced analysis on Google logs in near real-time such that data is pre-analysed, thus reducing the requirement for complex search queries on your logs to identify important system or user activity.

Figure two: Logentries Log Management and Analysis Flow

Logentries’ integration with Google Cloud Platform enables you to pinpoint issues quickly as well as look at long term trends across your log data. Some of the most useful capabilities of Logentries for Google Cloud Platform customers include:

Live Tail with Event Tagging: The Logentries pre-processing engine automatically tags important events such as exceptions, warnings, or errors allowing users to easily spot issues in a live view of your log data.

Figure three: Live Tail of Logs in Logentries

Near Real-Time Notifications and Inactivity Alerts: Get notified about important events within seconds of them occurring. Notifications can be configured to be sent to email, or can be integrated with other third party APIs and tools (e.g. Slack, HipChat, PagerDuty…).

Use your Logs as Data: Logs contain lots of very useful information beyond stack traces and error codes. Field level log analytics allows you to extract key metrics (e.g. server resource usage, or API response time) from your logs and roll these metrics up into interesting charts and graphs.

Figure four: Live Charts in Logentries

Google Cloud Logging supports a long list of known log formats via the google-fluentd collector - e.g. Apache, Chef, MongoDB, NginX and several others. Logentries also provides out-of-the-box intelligence (tags, alerts and dashboards) for these log formats via the Logentries community packs such that you do not need to spend time configuring rules or queries. Get started with Logentries, now paired with Google Cloud Logging service, today.

We are excited by this collaboration between Google Cloud Platform and Logentries, and we welcome your feedback. You can find more on the Logentries forum as well as send us feedback at cloud-logging-feedback@google.com.

- Posted by Deepak Tiwari (Product Manager, Google Cloud Platform) and Trevor Parsons (Co-founder and Chief Scientist, Logentries)