Google Cloud Platform Blog
Lessons learned from a year of using live migration in production on Google Cloud
Monday, April 25, 2016
Posted by Scott Van Woudenberg, Product Manager, Google Cloud Platform
It's been a little over a year (and millions of migrations) since the last time we talked about our live migration technology and how we use it to keep your virtual machines humming along while we patch, repair and update the software and hardware infrastructure that powers
Google Compute Engine
. It’s also an important differentiator for our platform compared to other cloud providers.
Our customer base has grown exponentially in the past year, and brought with it a lot of new and interesting workloads to test the mettle of our live migration technology. The vast majority of customers and workloads have been able to go about their business without noticing our maintenance events, with a few exceptions.
Down and to the right
A picture is worth 1,000 words and the following graph shows the improvements we've made to live migration blackout times (the amount of time your VM is paused) over the last year (
note the log scale
):
We've done millions of live migrations in the last year and as you can see from the graph, we've made significant improvements to median blackout duration and variance. The 99th percentile blackout graph is too noisy to display nicely, but we've improved that by a factor of six in the last year as well.
Lessons learned
The graph also shows that we didn't always get it right, and we've learned a lot from working closely with the handful of customers whose applications just don't play well with live migration.
The most important thing we learned is that the current
60 second pre-migration signal
is overkill for the vast majority of customers. At the same time, the 60 second signal is too short for the handful of customers that needed to perform some sort of automated drain or failover action.
We also learned that the older in-place upgrade maintenance mechanism (not captured in the above graph) we use to update our hypervisor is problematic for customers whose applications are sensitive to live migration blackouts.
Finally, we learned that surfacing VM migrations as system events in our
ZoneOperations
list led to a lot of confusion for little incremental value, since the events were also logged, in greater detail, in Compute Engine’s
Cloud Logs
. In many cases, customers noticed an issue with their service, saw the migration system event in the ZoneOperations list, and spent a long time investigating that as the cause, only to find that it was a red herring.
What's next?
Aside from continuing to measure and improve the impact of our maintenance events, the first thing we're going to do is drop the 60 second notice for VMs that don't care about it. If a VM is actively monitoring the
metadata maintenance event URI
, we'll continue to give 60 seconds notice before the migration. If a VM is not monitoring the URI, however, we'll start the migration immediately. This will not change the behavior of VMs that are configured to terminate instead of migrate. We expect to roll this out by mid-May.
In the coming quarters we’ll also begin providing a longer, more usable advance notice for VMs that are configured to terminate instead of migrate. This advance notice signal will be available via the metadata server as well as via the Compute Engine API on the
Instance resource
.
We'll also add a new API method to the Instance resource to allow customers to trigger maintenance events on a VM. This will give customers a means to determine if a VM is impacted by maintenance events and if so, to validate that its drain/failover automation works correctly.
The second major change we'll make is to use live migration for all instance virtualization software stack updates, replacing in-place upgrades. This will make these maintenance events visible, actionable and less disruptive while allowing us to focus our improvements on one maintenance mechanism rather than two. Finally, we'll remove VM migration events from our Operations collections and expand on the details of the events we log in Cloud Logs.
We strongly believe that fast, frequent, reliable, and above all, transparent infrastructure maintenance is essential to keeping our systems secure and to delivering the new features and services that customers want. We're pleased with the results we've seen so far, and we're excited to continue making it better. Follow us on
G+
and/or
Twitter
to stay informed as we start rolling out these improvements. If you have any questions or concerns, please
reach out
—
we'd love to hear from you.
Top 5 GCP NEXT breakout sessions on YouTube (so far)
Thursday, April 21, 2016
Posted by Alex Barrett, Editor, Google Cloud Platform Blog
Couldn’t make it to
GCP NEXT
last month? Lucky for you, recordings of over
thirty sessions
are up on YouTube to watch and learn from.
But with so much quality content to watch, where do you start? Leaving aside the wildly popular
Data Center 360 video
that tours the Dalles, Oregon facility (one million views and counting), and must-see
Day One keynotes
and
Day 2 keynotes
, here are the five most popular GCP NEXT breakout sessions on YouTube.
What’s the secret behind Google’s legendary uptime? Three words: site reliability engineers. In “
How Google Does Planet-Scale Engineering for Planet-Scale Infrastructure
” Google engineering director Melissa Bindle lays out the relationship between SREs and developer teams, and how they ensure uptime for Google.com and
Google Cloud Platform
users. Even if your organization isn’t operating at quite planetary scale, lessons abound for anyone that cares about building and operating resilient environments.
What exactly is machine learning, and what can it do for you? In “
Build smart applications with your new superpower: cloud machine learning
” Google Developer Advocate
Julia Ferraioli
walks through the range of Google’s machine learning options, from
TensorFlow
, to
Google Cloud Machine Learning
to the Machine Learning APIs. She’s also joined by David Zuckerman, head of developer experience at Wix.com, who describes how it used Google machine learning to mine customer images with the new Vision API.
Cloud Platform users do the darndest things. In
Analyzing 25 billion stock market events in an hour with NoOps
on GCP, Fidelity Information Systems CTO Neil Palmer and developer Todd Ricker describe how they “ingest, process and analyze the entire U.S. options and equities market in 4 hours using
Google Cloud Dataflow
,
Google Coud Pub/Sub
and
Google Cloud Bigtable.
” That’s 15 TB of data. On a daily basis. That it then makes available for market reanalysis to the regulators.
Cloud Platform isn’t all about crunching large amounts of data for ginormous financial services firms. It’s also about democratizing access to Google’s amazing capabilities to the world at large, wherever they may be. In “
Building iOS Apps with Firebase
,” Google Cloud Developer Advocate Sara Robinson walks users through how to develop a native iOS app using Firebase, a mobile back-end as a service, and then ties that app into
Google Cloud Vision API
to provide face and object recognition.
Rounding out the list of most-watched breakout sessions is
Painless container management with GKE & Kubernetes
. Brendan Burns, Google’s lead engineer on the Kubernetes project, provides context and backstory; Rich Steenberg, principal engineer at WePay, a Kubernetes shop, describes how they’re using Kubernetes today; and Tim Hockin, a Google software engineer, describes what’s new in Kubernetes 1.2, and what’s on tap.
Did your favorite session not make the list? Share it in the comments so the rest of the world can benefit from what you learned.
Iron Mountain increases bandwidth to Google Cloud by 10x for faster data ingestion
Wednesday, April 20, 2016
Posted by Ben Chong, Product Manager
Iron Mountain has expanded the bandwidth of its network link to
Google Cloud Platform
by 10x. This enables faster transmission of your data to GCP, which is super helpful for meeting critical project deadlines. As an illustration of transmission speed, moving 50TB of data over the expanded link takes less than a day, compared to more than five days with the old connection.
Iron Mountain has been providing cloud seeding services for our customers in North America since 2015.
Enterprises are eager to run applications on public clouds to benefit from the security, scalability and reduced management burden. However, moving data to the public cloud as part of a migration or hybrid process is slow and troublesome. Uploading one terabyte of data using a typical small business DSL connection may take up to 100 days!
As a result, enterprises are frequently turning to cloud seeding services, also referred to as offline ingestion or
offline data import
, from providers like Iron Mountain.
Instead of trying to push data to a public cloud over a limited bandwidth, customers copy data to storage devices and ship them to a third party who can then securely upload the data to the customers’ data buckets in cloud storage. Cloud seeding providers include, as part of their service portfolio, chain-of-custody verification and security options like encryption to help ensure that customer data is handled securely.
Iron Mountain has also introduced support for LTO (Linear Tape-Open) tapes, which is a good transport medium for large amounts of data. Customers with data already on LTO media can immediately start moving that data into low cost
Google Cloud Storage Nearline
.
You could be eligible for up to $500 of credit on Cloud Platform if you ingest with Iron Mountain! More information can be found
here
. If you’re ready to start a project on Cloud Platform and are not sure how to start moving your data to the cloud, we’ve got you covered with
offline data import
.
A simple approach to PCI DSS compliance for Google Cloud Platform
Tuesday, April 19, 2016
Posted by Peter-Mark Verwoerd, Cloud Solutions Architect
Customers often ask us for guidance about how to build PCI DSS compliant environments on top of
Google Cloud Platform
. From our work in the field, we recently put together a handy-dandy tutorial to help them get started.
This is no small thing. Many businesses today have online storefronts, and the vast majority of those take credit cards. When you accept credit cards for your business, you have to make sure you do that securely
—
to ensure customer trust and security, to get paid and to meet the necessary regulations, namely PCI DSS.
The PCI DSS, created by the
PCI Security Standards Council
, is an information security standard created by the major credit card companies; as such, any business that takes Visa, MasterCard, Discover, American Express or JCB is expected to be PCI DSS compliant, and can be fined or penalized if it is not.
Creating and managing a compliant PCI DSS environment can be a non-trivial task. Thankfully, if you’re on Cloud Platform, managed services such as
Stackdriver Monitoring
,
Stackdriver Logging
, and
Google BigQuery
can help. Our solution, for example, includes these basic components:
A lightweight
Google Compute Engine
front-end application that accepts credit card information and sends it to an external payment processor. Importantly, that information is never recorded, it's only transmitted.
An external payment processor that charges the credit card if it's accepted or rejects it if it’s not, and notifies your application of the result. Since this is just a notification to your application, no credit card data is transmitted or recorded from the payment processor.
Stackdriver Logging, which logs the actions of every application and server via Squid Proxy which restricts the event traffic and sends them to Stackdriver Monitoring, which monitors the events
BigQuery, which can be used to analyze the logs, run ad-hoc audit queries and create reports.
(click to enlarge)
For further details, check out the full
solution
for this design. We hope you'll find it useful, and we welcome and encourage your feedback. Comment here or reach out to
@petermark
on Twitter.
Enhanced persistent disks for Google Compute Engine = better Kubernetes and Docker support
Monday, April 18, 2016
Posted by Martin Buhr, Product Manager
The infrastructure underpinning
Google Cloud Platform
has always been good, and it keeps getting better. We recently increased the limit on the
number of attached persistent disk storage volumes
per Google Compute Engine instance from 16 to as many as 128 volumes on our largest standard instance sizes (giving us three times the capacity of the competition). We also recently increased the
total quantity of attached persistent disk storage
per instances from 10TB to as much as 64TB on our largest instance sizes, and introduced the ability to
resize persistent disk volumes
with no downtime.
(click to enlarge)
These changes were enabled by Google’s continued innovations in data center infrastructure, in this case
networking
. Combined with
Colossus
, Google’s high-performance global storage fabric, we have greatly increased the size, quantity, and flexibility of network attached storage per instance without sacrificing persistent disk’s legendary rock solid
performance and reliability
.
This ties back to one of Google Cloud Platform’s core strengths: hosting Linux containers. We are committed to making GCP the best place on the web to run Docker-based workloads, building on over a decade of experience running all of Google on Linux containers. To help you realize the same benefits that we did, we created and open-sourced
Kubernetes
, a tool for creating and managing clusters of Docker containers, and launched Google Container Engine to provide a managed, hosted, Kubernetes-based service for Docker applications.
Red Hat is a large Kubernetes user and has adopted it across its product portfolio, including making it a core component of
OpenShift
, its Platform as a Service offering. Working with Red Hat late last year in preparation for
offering OpenShift Dedicated on Google Cloud Platform
, it became clear that we needed to increase the number of attached persistent disk volumes per instance to help Red Hat efficiently bin pack Kubernetes
Pods
(each of which may have one or more attached volumes) across their Google Cloud Platform infrastructure.
Being able to attach more persistent disk volumes per instance isn’t just useful to Red Hat. In addition to better support for Kubernetes and Docker, this feature can also be used in various scenarios that require a large number of disks. For example, you keep data from different web servers on separate volumes to insure data isolation when an instance is running multiple web servers.
To take advantage of this feature for your own Kubernetes clusters running on Google Compute Engine, simply set the "KUBE_MAX_PD_VOLS" environment variable to whatever you want the limit to be, based on the maximum number of volumes the nodes in your cluster support.
Google Cloud and Autodesk enable 10x improvement in media rendering efficiency
Monday, April 18, 2016
Posted by Todd Prives, Product Manager, Google Cloud Platform
We're at
NAB
this week and are thrilled to announce new
Google Cloud Platform
partners and product updates that underscore the growing role of cloud computing
in the media and entertainment industry
.
We’re collaborating with Autodesk to launch a new cloud-optimized rendering service called
Autodesk Maya
for Cloud Platform ZYNC Render. Autodesk software has been behind the past 21 Academy Award winners for Best Visual Effects and we’re bringing this capability to Google Cloud Platform.
The combined offering captures the aim of both companies to enable individual artists and developers to focus on content, abstracting away the nuances of managing infrastructure. Small teams of artists can tap world-class infrastructure to realize the creative vision of what was once limited to much larger studios.
Here’s how it works:
Product teams at Google and Autodesk have developed a cloud-optimized version of Autodesk® Maya®, the popular tool for animation, modeling and rendering. Via a Maya plugin, 3D scenes are transferred to
Google Compute Engine
in the background, while the artist is working. Because Maya is capable of running on the artist’s workstation and also in the cloud on GCP, this allows artists to run massively parallel rendering far more efficiently than before
—
taking advantage of the scalability, performance, and
price benefits of GCP
Compared with the non-optimized version, Maya® customers see up to 10x improvement in file upload efficiency. This allows many rendering jobs to start instantaneously, cutting wait time and accelerating time to finished shot
We’re also excited to announce ZYNC Render support for Pixar’s historic
RenderMan
, with licensing fully built-in to the product. ZYNC users are now able to spin up 500 machines per account, scaling to 32,000 rendering cores, with new support for 64-core machines
—
making short work of ambitious rendering jobs.
In addition to our collaboration with Autodesk, we've made some big strides in our offerings for the media and entertainment industry. Here are some recent updates:
Cloud Vision API graduates to General Availability
The goal of
Cloud Vision API
is to provide vision capability to your applications in the same way that
Google Photos
does. It's a powerful tool for media and entertainment companies enabling you to classify images and analyze emotional facial attributes. To further improve our customer experience, Cloud Vision API is going into general availability today with new features:
Logo Detection expanded to cover millions of logos
Updated OCR for understanding more granular text like receipts
Cloud CDN graduates to Beta
We’re launching
Cloud CDN Beta
, allowing your media content to be pushed out to Google’s network edge and cached close to users. As always, data travels via Google’s network and reaches users who expect instantaneous access to images and live-stream video experiences. Cloud CDN is also fully integrated with Google’s
global load balancing
and enterprise-grade security to distribute media workloads anywhere they originate, so jobs never get bogged down.
Lytro chooses Cloud Platform and The Foundry
One question we see today’s top innovators ask continuously is: with massive cloud infrastructure at our fingertips, how can we make a major leap forward in the way things are done today?
Lytro
is one example. Its technology seems to defy the traditional physics of photography, capturing massive volumes of visual data with such fidelity that infinite creative choices abound: unprecedented control over focus, perspective, aperture and shutter angle, all in post-production. Lytro selected GCP and
The Foundry
to help power their amazing invention. Learn more by
watching their video
.
For more on what's possible with cloud-enabled media, visit
cloud.google.com/solutions/media
and
zyncrender.com
, or contact us to discuss how cloud can enable your creative workflows.
Introducing Google Consumer Surveys API: Let users of your app create surveys
Thursday, April 14, 2016
Posted by Dylan Lorimer, Product Manager
Editor's note: Updated April 19, 2016
Google Consumer Surveys
has launched an API, built on
Google Cloud Platform,
that lets your app users create and integrate surveys.
We’ve spoken to research and non-research companies who are really interested in bringing the power of our Consumer Surveys tool into their own applications, and with this launch it’s finally possible. We imagine many different use-cases to tap into the millions of respondents our platform can connect you with, across a dozen or more markets around the world.
Leading up to our launch we worked closely with a handful of trusted testers who provided valuable feedback. These have included proprietary solutions to manage studies conducted on the Google Consumer Surveys platform (
Kantar
), predictive analytics solutions (
Predictvia
), and solutions that help customers visualize their Google Consumer Surveys data (
MarketSight
). We look forward to working with developers to build unique solutions that empower individuals and businesses to make better data driven decisions.
If you’re an existing Google Consumer Surveys Enterprise customer on an invoicing contract, you can start using the API immediately. If you’re not an Enterprise customer but are interested in accessing the API, you can get in touch with us at
gcs-api@google.com
. You can read more about our API on our
API documentation site
.
Don't Miss Next '17
Use promo code NEXT1720 to save $300 off general admission
REGISTER NOW
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Labels
Announcements
56
Big Data & Machine Learning
91
Compute
156
Containers & Kubernetes
36
CRE
7
Customers
90
Developer Tools & Insights
80
Events
34
Infrastructure
24
Management Tools
39
Networking
18
Open Source
105
Partners
63
Pricing
24
Security & Identity
23
Solutions
16
Stackdriver
19
Storage & Databases
111
Weekly Roundups
16
Archive
2017
Feb
Jan
2016
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Feed
Subscribe by email
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow
Follow