Google Cloud Platform Blog
Six things Stackdriver brings to the DevOps table
Thursday, June 9, 2016
Posted by Aja Hammerly, Developer Advocate
As someone for whom DevOps and sysadmin tasks are only part of my job, having all the tools I commonly need in one place is a huge advantage.
Stackdriver
gives me exactly that. Monitoring, logging, debugging and error reporting are all integrated and provide the essential tools I need to keep my websites up and healthy. I also like that Stackdriver doesn’t require me to have deep system administration knowledge to set up basic monitoring. With minimal effort, I’m confident that I'll be notified if my application has an issue.
I gave a talk at Google I/O 2016 titled "Just Enough Stackdriver to Sleep At Night" that gives an overview of what I like about Stackdriver. You can watch
the whole thing
, but this post covers some of the highlights.
Monitoring and uptime monitoring
Setting up basic monitoring is one of the most common DevOps tasks. Stackdriver offers uptime monitoring for URLs, App Engine applications and modules, load balancers or specific instances. Uptime checks can run over HTTP, HTTPS, UDP or TCP and you can customize how often the check runs. Most of the time, I use a URL check against the root of my application or another vital endpoint, and once you've set up the check you can configure how you want to be notified. In addition to common notification methods like email and SMS, Stackdriver supports notification via messaging platforms like Hipchat, Slack, and Campfire, as well as PagerDuty and the Google Cloud Console mobile app. And if none of these options works for your team, there's a configurable webhook.
Application-level monitoring
Another thing DevOps teams want is application-level monitoring. Stackdriver can monitor many common tools/frameworks including nginx, Apache, Memecached, MongoDB, MySQL, PostgreSQL and RabbitMQ. To begin monitoring these applications, all you need to do is add a config file to your system and restart the monitoring agent. Of course Stackdriver supports custom monitoring if your particular stack isn't already supported.
If your application's running on
Google Cloud Platform
, Stackdriver automatically looks at open ports, running services and instance names to determine if you're running any common tools, and if so, it makes metrics for those tools available for monitoring. For example, if you're running a MySQL server on
Google Compute Engine
with an instance called "MySQL" and the mysql process is running, Stackdriver will detect that and add the MySQL metrics to the monitoring options.
And if you're using
Google App Engine
, Stackdriver supports request-level latency monitoring. You can look at latency for a particular class of responses, say 5xx errors or 2xx successful responses. You can also look at the overall average or the 95% or 5% case. This is particularly helpful when your request latency occasionally has outliers.
(click to enlarge)
System-level monitoring
Stackdriver also supports system-level monitoring. You can monitor disk usage and I/O, memory usage and swap, CPU usage and
steal
, processes (running, sleeping, zombies), network traffic and open TCP connections. System-level monitoring can alert you if disks are filling up too quickly or if the CPU is spiking outside of the acceptable range.
Monitoring some parts of the system requires installing the Stackdriver monitoring agent on the machine. Installing the agent only takes a few minutes and there's a cookbook for Chef, a module for Puppet and a role for Ansible as well.
Logging
Much like Stackdriver Monitoring,
Stackdriver Logging
works on both Cloud Platform and Amazon Web Services. It's set up by default for App Engine, and also captures some
Google Container Engine
events. Installing the Logging agent on your Compute Engine VMs is simple. Additionally, there are packages available for many web frameworks to integrate Stackdriver Logging with your application.
If your framework isn't supported or you need custom events, you can use the Stackdriver Logging API to send events directly to Stackdriver. The API also supports viewing entries and managing logging for your project.
I like that the Stackdriver Logging UI supports searching by time interval, response code, log level, log source and other things that I find helpful. In the past, I've had to write code to do this level of filtering. And if search capabilities of the Logging UI aren't sufficient, you can export your logs to
Google BigQuery
, which can quickly query, aggregate or filter several terabytes of data. You can also save your queries with BigQuery to repeat them later and to share results with others.
(click to enlarge)
Error reporting
One of the problems I've often run into is the idea of a "normal error." Most applications seem to have an edge case or other error condition that causes an error, but that isn't a priority to fix. This is why I like
Stackdriver Error Reporting
. Stackdriver Error Reporting monitors your application errors, aggregates them, and then alerts you to new errors that arise.
You can use the Error Reporting console to see how many of each error have occured, what versions of your application the error occurred in, and when it was first or last seen. Error Reporting saves a few representative stack traces from the error to help you debug your application. You can also link a specific error to a bug in your bug tracker.
Error Reporting is automatically set up for App Engine applications. It currently supports Java, Python, Javascript, PHP and Go. To use Error Reporting in other environments you can call an API from your application or you can send error events to Stackdriver Logging in a specific format. To receive alerts about new errors you can opt-in from
Google Cloud Console
.
(click to enlarge)
Debugging
Once you've noticed an error in your application with Error Reporting or Stackdriver Logging, you may need to debug your application to prevent the error from happening again.
Stackdriver Debugger
can help you here. Instead of hooking up a debugger to the production website (something many of us have done and very few will recommend), Stackdriver Debugger takes a snapshot of the application state at a specified point. The snapshot shows you the call stack and variable values without the need to push instrumented code to production.
To take a snapshot, all you need to do is supply a filename and line number. If you have access to the source code for your application you can upload it to Stackdriver Debugger. You can also point Debugger at a cloud repository or load the source code into the browser locally. When the source code is available you can set snapshot points in Debugger much like you set breakpoints in an IDE. This allows you to see the captured values in the context of the code.
Stackdriver Debugger is automatically enabled for all App Engine applications. Better yet, it doesn't add a large amount of latency to captured requests so your users will likely not notice a performance hit.
Conclusion
You may've been running applications in the cloud for years, but keeping tabs on your application and dealing with errors has usually involved multiple tools from multiple vendors that may or may not share data with each other. Stackdriver provides the tools you need in one place, with one login, and they all integrate together. While looking at an error in Error Reporting you can seamlessly see the related logs in Cloud Logging. You can set up monitoring and alerting on events in Cloud Logging. And once you find problems, debugging them in production is straightforward. Check out Stackdriver when you get a chance and let me know what you think
@the_thagomizer
on Twitter.
Your Google Cloud Platform compute options, explained
Tuesday, June 7, 2016
Posted by Alex Barrett, Editor, Google Cloud Platform Blog
When choosing to host an application on
Google Cloud Platform
, one of the first decisions organizations make is which compute offering to choose from:
Google Compute Engine
(GCE)?
Google App Engine
(GAE)?
Google Container Engine
(GKE)? There’s no right answer — it all depends on your developers’ preferences, what kind of functionality the application requires, and the use case.
Our new “
Choosing a Computing Option
” guide is a convenient way to visualize all these options at a glance, to help you make the best choice for your application. Then again, you may not want to choose. There’s nothing stopping you from choosing multiple compute options, across different application tiers.
And going a step further, if you’re comparing compute options across cloud providers, these resources will be helpful:
Learn how
Google Cloud Platform services map to Amazon Web Services (AWS)
Check out
Google Cloud Platform for AWS professionals
: A guide designed to equip users familiar with AWS with the key concepts required to get started with Google Cloud Platform
Learn how
Google Cloud Platform services map to Microsoft Azure services
If you’ve built an application that spans GCE, GAE and GKE or across other clouds, send us a note on Twitter
@googlecloud
. We’d love to hear more!
Best practices for Tableau Server on Google Compute Engine
Monday, June 6, 2016
Posted by Danny Bain, Manager of Cloud Server Offerings, Tableau
Most Tableau users storing and working with data on
Google Cloud Platform
have probably heard of
Tableau Desktop
, which helps you connect to data in
Google BigQuery,
Cloud SQL
and other databases to quickly create visualizations and dashboards for better insight.
Exploring and analyzing data is often only the first step in the analytics journey; at some point everyone wants to share what they’ve created. That’s where
Tableau Server
comes in. You can create a visualization with Tableau Desktop and then publish it to Tableau Server where it can be shared, edited and interacted with, using any browser.
Installation guidelines
We recently announced support for Tableau Server running on Google Cloud Platform (GCP). For new users, we've created a walkthrough in the Tableau Knowledge Base article “
Tableau Server and Cloud Platform Installation Walkthrough
” that shows you how to get Tableau Server running on
Google Compute Engine
.
If you already have a
GCP account
, here’s a brief overview of what you’ll need to do to get started. You can also read a much more detailed explanation on the
Tableau Knowledgebase
.
Set up a new project and create a new Compute Engine instance
Minimum requirements are 8x vCPUs, running Windows Server 2012 R2 Datacenter Edition with 128 GB persistent SSD. You’ll also want to leave API access at its default setting and allow both HTTP and HTTPS traffic.
Download the RDP and connect as normal
Download a Windows-compatible 14-day free trial version from the
Tableau Server Trial Download
page
Install Tableau Server, following the prompts on the screen
Create your administrator account in Tableau Server (this should be prompted automatically after installation)
Make sure to
download Tableau Desktop
as well, so you can connect to data and create visualizations to publish to Tableau Server
Best practices
If you follow these steps, Tableau Server should work out of the gate, but here are some tips to make sure Tableau Server runs as well as it possibly can.
While 8 vCPUs and 30GB of memory is the minimum, you’ll see a 75% improvement in performance with 16x vCPUs and 60GB of memory.
When scaling instances, it’s preferable to add vCPUs in smaller increments.
8 vCPUs delivers very poor performance for hundreds of users. The image below shows load testing results for Tableau Server in more detail.
Performance testing of Tableau Server running on GCE (click to enlarge)
Doubling your instance
disk size
will double performance of the overall system (1:1 win).
For even more performance, include multiple workers in a cluster to linearly scale performance as the number of users grows. You’ll need to
install a domain controller
in GCP to use multiple workers in your cluster.
Select HTTPs for the best security.
Read here
about how to get an SSL certificate for Tableau Server.
Free certificates are available from
letsencrypt.org
.
If you want to use unencrypted connections, configure the Tableau Server admin password BEFORE enabling public HTTP access to your server.
Once you’ve installed and configured Tableau Server, be sure to watch
our free online training videos
to learn everything you need to know about installing, administering, using and expanding Tableau Server within your organization.
What’s new in the Kubernetes community: this week on Google Cloud Platform
Friday, June 3, 2016
Posted Alex Barrett, Editor, Google Cloud Platform Blog
Judging by the
spate of Kubernetes-related activity these days
, it seems like the community's fired up and putting the
open-source cluster management system
to work in all sorts of interesting ways.
In recent days, we’ve seen proposals for new Kubernetes-compliant storage systems (
Torus
, by CoreOS, and
KubeFuse
by OpenCredo); new approaches to Kubernetes security (
Hypernetes
from HyperHQ); and even an enhanced version of Kubernetes inspired by its maker’s experience in the supercomputing market (
NavOps Command
, by Univa). And Google’s own
Brendan Burns
introduced
ksql
, which allows users to query Kubernetes objects using SQL.
What’s all the excitement about? In a nutshell,
Chris Kleban
argues that what Kubernetes (and Docker) bring to the table are “
anywhere cloud services
.” “Docker enables us to easily build, ship and run software by packaging it up in a way that will run on a wide range of systems,” he writes. “But, that isn't enough. We need a way to get that software installed, working and highly available. We need something like Kubernetes.”
Enterprises seem to be getting the memo too. Samsung SDS Research America told Timothy Prickett Morgan at the TheNextPlatform about how it’s
putting Kubernetes through the paces
, while Beth Pariseau of SearchITOperations.com details how
Concur and Barkly Protects Inc.
are using Kubernetes to manage applications running in Amazon Web Services and on-premises.
Of course, we recommend you run on Google Cloud Platform with
Google Container Engine
(GKE). As of last week, GKE now supports
node pools
, which make it possible to run nodes of different configurations within the same cluster.
If that’s still not enough Kubernetes for you, Google developer advocate
Sandeep Dinesh
writes about
using kube-ui and Weave Scope with Google Container Engine
, while
Ian Lewis
tackles the topic of
using Kubernetes Health Checks
.
Be sure to stop by soon for a piece of Kubernetes cake when the project turns two in July!
The five phases of migrating to Google Cloud Platform
Thursday, June 2, 2016
Posted by Alex Barrett, Editor, Google Cloud Platform Blog
Migrating to
Google Cloud Platform
isn’t hard per se, but there’s a lot to think about. And thankfully, it doesn’t need to happen all at once.
In his presentation, “Getting to the Cloud with Google Cloud Platform: A Sequential Approach” Google Cloud Solutions Architect
Peter-Mark Verwoerd
lays out a five-step plan to migrate workloads from an on-premises environment to GCP. It’s worth watching
the webcast
in its entirety, but until then, here are the high notes.
Phase One: Assess
Before you move a single bit, take stock of your applications and how suitable they are for the cloud. Things to think about include (but are not limited to) hardware and performance requirements, users, licensing, compliance needs and application dependencies.
Generally speaking, apps fall into one of three buckets: Easy to Move, Hard to Move, and Can’t Move. In our experience, applications that most often fall into the Easy to Move bucket are greenfield apps (no surprise there), test and dev and Q&A. Internal web apps and batch processing applications are also good cloud candidates, because they can scale horizontally rather than vertically.
Phase Two: Pilot
This is the point where you take one or two applications, and try moving them. Learn about Cloud Platform and its design patterns, take the time to validate performance, consider your licensing options and establish how to perform a rollback. Don’t skip this step, and don’t be tempted to try and migrate too many apps all at once!
Phase Three: Move Data
Some people will tell you to move your applications first, then move your data, but we beg to differ. Most applications have a lot of data with a lot of dependencies. Properly moving data to the cloud sets the stage for a successful application migration later on.
This is also the time to consider your various cloud storage options — regular
Google Cloud Storage
or
Nearline
?
Local SSDs or persistent disks
?
Google Cloud SQL
,
Datastore
or
Bigtable
? You should also think about how you’re going to move all that data — via batch data transfers, offline disk imports, with database dumps, or streaming to persistent disks? There are lots of things to consider here.
Phase Four: Move Applications
Now that your data is in the cloud, you’re ready to move the actual apps. Here too, there are decisions to make. We recommend keeping things simple, and doing the minimum necessary to get the application up and running in the cloud, for example doing a straight lift-and-shift. Or perhaps you can get an app into the cloud by way of backing it up there? That way, in the event of an outage, there’s a full copy of your environment in GCP waiting to take over.
Phase Five: Optimize
This is where the fun begins. Once an application and its data have been migrated to GCP, you can start thinking about all the cool ways to make it better. For example, this might be a time to add redundancy in the form of availability zones, elasticity with autoscaling groups, or enhanced monitoring with
Stackdriver
. You might want to offload static assets off of your application tier into Cloud Storage, or decouple tiers by using
Pub/Sub
. Google’s
Deployment Manager
can make it easier to launch and scale new instances, and copying your configuration into a second region can insulate you from a regional outage.
See? That wasn’t so bad. In the meantime, if you’d like to turbocharge your VM migration process, we also have systems integration partners that are cloud migration and GCP experts and would be happy to assist you. You can find more resources on migrating to GCP, including a list of certifed partners, at
https://cloud.google.com/migrate/
. Here’s the
webinar
once again, for a more detailed step-by-step guide.
Top 10 GCP sessions from Google I/O 2016
Tuesday, May 31, 2016
Posted by Jo Maitland, Managing Editor, Google Cloud Platform Blog
In-between
daydreaming about virtual reality
and saying
Allo to Duo
, there were some fantastic
Google Cloud Platform
sessions and demos at
Google I/O
this year. In case you weren’t able to attend the show, here are the recordings.
In
One Lap Around Google Cloud Platform
, developer advocate
Mandy Waite
, and
Brad Abrams
walk through the process of building a Node.js backend for an iOS and Android based game. Easily the best bit of the session is a demo that uses
Kubernetes
and GCP
Container Registry
to deploy a Docker container to
App Engine Flexible Environment
,
Google Container Engine
and an
AWS EC2 instance
. It’s a simple demo of portability across clouds, a key differentiator for GCP.
Speaking of multi-cloud environments, developer advocate
Aja Hammerly
presented a great session on
Stackdriver
, called
Just Enough Stackdriver To Sleep Well At Night
, which will be music to the ears for any of you who have to carry a pager to support your site or application. In a nutshell, Aja shows how Stackdriver unifies a bunch of different monitoring and management tools into a single solution.
If you’ve made the leap to containers or are thinking about it, you’ll want to check
Carter Morgan's
session:
Best practices for orchestrating the cloud with Kubernetes
. This session includes the basics of modern day applications and how containers work. It also covers packaging and distributing apps using Docker and how to up your game by running applications on Kubernetes.
Did you know Kubernetes has seen 5,000 commits and over 50% from unique contributors since January 2015?
Next, IoT ideas are a dime a dozen, but bringing them to life is another story. In
Making sense of IoT data with the cloud
, developer advocate
Ian Lewis
shows how you can manage a large number of devices on GCP and how to ingest, store and analyze the data from those devices.
In
Supercharging Firebase with Google Cloud Platform
developer advocates
Sandeep Dinesh
and
Bret McGowen
use
Firebase
to build a real-time game that interacts with virtual machines, big data and machine learning APIs on GCP. The coolest part of the demo involves the audience in the room and on the livestream interacting with the game via Speech API, all yelling instructions at the same time to move a dot through a maze. The hallmark of Firebase
—
real-time data synchronization across connected devices in milliseconds
—
is on display here and fun to see. For more Firebase tips and tricks check out
Creating interactive multiplayer experiences with Firebase
, from developer advocate
Mark Mandel
.
Switching gears to big data and the upcoming U.S. presidential election, developer advocate
Felipe Hoffa
and software engineer
Jordan Tigani
, demo the power of
Google BigQuery
to uncover some intriguing campaign insights in
Election 2016: The Big Data Showdown
. You'll learn which candidate is spending the most money and how efficient that spending is relative to their mentions on TV, by mashing together various different public datasets in BigQuery. Felipe and Jordan do a nice job showing us how BigQuery can separate the signal from the noise to figure out what it all means.
Figuring out the right storage for each application in your business can be a daunting task on the cloud.
Dominic Preuss
, group product manager, explains how in
Scaling your data from concept to petabytes
.
And of course, no event that includes Cloud Platform is complete without demos from developer advocates
Kaz Sato
on
How to build a smart RasPi bot with Cloud Vision and Speech API
, and another crowd-pleaser,
Google Cloud Spin: Stopping time with the power of Cloud
, from
Francesc Campoy Flores
.
To find more tutorials, talks and demos on GCP beyond the sessions at I/O this year, check out our
GCP YouTube channel
and weekly
podcast
, and follow
@GoogleCloud
on Twitter for all the latest news and product announcements from the Cloud Platform team.
Machine learning at the museum: this week on Google Cloud Platform
Friday, May 27, 2016
Posted by Alex Barrett, Editor, Google Cloud Platform Blog
Is there a limit to what you can do with machine learning? Doesn’t seem like it.
At
Moogfest
last week, Google researchers presented about
Project Magenta
, which uses TensorFlow, the open-source machine learning library that we developed, to see if computers can create original pieces of art and music.
Researchers have also shown that they can use TensorFlow to train systems to imitate the grand masters. With
this implementation
of
neural style
in TensorFlow, it becomes easy to render an image that looks like it was created by Vincent Van Gogh or Pablo Picasso — or a combination thereof. Take this image of Frank Gehry’s Stata Center in winter,
add style inputs from Van Gogh’s Starry Night and a Picasso Dora Maar:
and end up with:
Voila! A Picasso-esque Van Gogh for 21st century!
(The code for neural style was first posted to GitHub last December, but the author continues to update it and
welcomes pull requests
.)
Or maybe fine art isn’t your thing. This week, we also saw how to use TensorFlow to
solve trivial programming problems
,
forecast demand
— even
predict the elections
.
Because TensorFlow is open source, anyone can use it, on the platform of their choice. But it’s worth mentioning that running machine learning on
Google Cloud Platform
works especially well. You can learn more about GCP’s machine learning capabilities
here
. And if you’re doing something awesome with machine learning on GCP, we’d love to hear about it — just tweet us at
@googlecloud
.
Don't Miss Next '17
Use promo code NEXT1720 to save $300 off general admission
REGISTER NOW
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Labels
Announcements
56
Big Data & Machine Learning
91
Compute
156
Containers & Kubernetes
36
CRE
7
Customers
90
Developer Tools & Insights
80
Events
34
Infrastructure
24
Management Tools
39
Networking
18
Open Source
105
Partners
63
Pricing
24
Security & Identity
23
Solutions
16
Stackdriver
19
Storage & Databases
111
Weekly Roundups
16
Archive
2017
Feb
Jan
2016
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Feed
Subscribe by email
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow
Follow