Introduction
What’s remarkable about April 7th, 2014 isn’t what happened that day. It’s what didn’t.
That was the day the Heartbleed bug was revealed, and people around the globe scrambled to patch their systems against this zero-day issue, which came with already-proven exploits. In other public cloud platforms, customers were impacted by rolling restarts due to a requirement to reboot VMs. At Google, we quickly rolled out the fix to all our servers, including those that host ...
Read More
Introduction
What’s remarkable about April 7th, 2014 isn’t what happened that day. It’s what didn’t.
That was the day the Heartbleed bug was revealed, and people around the globe scrambled to patch their systems against this zero-day issue, which came with already-proven exploits. In other public cloud platforms, customers were impacted by rolling restarts due to a requirement to reboot VMs. At Google, we quickly rolled out the fix to all our servers, including those that host
Google Compute Engine. And none of you, our customers, noticed. Here’s why.
We introduced
transparent maintenance for Google Compute Engine in December 2013, and since then we’ve kept customer VMs up and running as we rolled out software updates, fixed hardware problems, and recovered from some unexpected issues that have arisen. Through a combination of datacenter topology innovations and live migration technology, we now move our customers running VMs out of the way of planned hardware and software maintenance events, so we can keep the infrastructure protected and reliable—without your VMs, applications or workloads noticing that anything happened.
The Benefits of Transparent Maintenance
Our goal for live migration is to keep hardware and software updated across all our datacenters without restarting customers' VMs. Many of these maintenance events are disruptive. They require us to reboot the host machine, which, in the absence of transparent maintenance, would mean impacting customers’ VMs.
Here are a few of the issues we expected to address with live migration, and we have encountered all of these:
- Regular infrastructure maintenance and upgrades
- Network and power grid maintenance in the data centers
- Bricked memory, disk drives, and machines
- Host OS and BIOS upgrades
- Security-related updates, with the need to respond quickly
- System configuration changes, including changing the size of the host root partition, for storage of the host image and packages
We were pleasantly surprised to discover that live migration helped us deliver a better customer experience in the face of a much broader array of issues. In fact, our Site Reliability Engineers started using migration as a tool even before it was generally enabled; they found they could easily work around or mitigate potential breakages occurring in production.
Here are some of the
unexpected issues that we encountered and worked around with live migration without impacting the running guests:
- Flapping network cards — Network cards were intermittently failing. We were able to repeatedly try the VM migrations and successfully migrate them. This even worked with partially-failing NICs.
- Cascading battery/power supply issues — Some overheating batteries were overheating the neighboring machines. We were able to migrate the VMs away before bringing down the machines to swap out their batteries.
- A buggy update was pushed to production — We halted the rollout, but not before it reached some of our production machines (it didn't manifest in our canary environment). The buggy software would’ve crashed VMs within a week. Instead, we migrated the VMs on the affected machines to other hosts that didn’t have the buggy software.
- Unexpected host memory consumption — One of our backend components consumed more memory than we had allocated and threatened to OOM (out of memory) the VMs. We migrated some VMs away from the over-loaded machines and avoided the OOM failures while patching the backend system to ensure it could not overrun its allocation.
Transparent Maintenance in Action
We’ve done hundreds of thousands of migrations since introducing this functionality. Many VMs have been up since migration was introduced and all of them have been migrated multiple times.
The response from our customers has been very positive. During the early testing for migration, we engaged with Rightscale to see the impact of migrations. After we migrated all their VMs twice, they reported:
“We took a look at our log files and all the data in the database and we saw…nothing unusual. In other words, if Google hadn’t told us that our instances had been migrated, we would have never known. All our logs and data looked normal, and we saw no changes in the RightScale Cloud Management dashboard to any of our resources, including the zone, instance sizes, and IP addresses.”
We worked with David Mytton at ServerDensity to live migrate a replicated MongoDB deployment. When the migration was done, David tweeted:
“Just tested @googlecloud live migration of a @MongoDB replica set - no impact. None of the nodes noticed the primary was moved!”
In fact, Google has performed host kernel upgrades and security patches across its entire fleet without losing a single VM. This is quite a feat, given the number of components involved and factoring in that any one of them or their dependencies can fail or disappear at any point. During the migration, many of the components that comprise the VM (the disks, network, management software and so on) are duplicated on the source and target host machines. If any one of them fail at any point in the migration, either actively (e.g. by crashing) or passively (e.g. by disappearing), we back out of the migration cleanly without affecting the running VM.
How it works
When migrating a running VM from one host to another, you need to move all the state from the source to the destination in a way that is transparent to the guest VM and anyone communicating with it. There are many components involved in making this work seamlessly, but the high-level steps are illustrated here:
The process begins with a notification that VMs need to be evicted from their current host machine. The notification might start with a file change (e.g. a release engineer indicating that a new BIOS is available), Hardware Operations scheduling maintenance, an automatic signal from an impending hardware failure etc.
Our cluster management software constantly watches for such events and schedules them based on policies controlling the data centers (e.g. capacity utilization rates) and jobs (e.g. number of VMs for a single customer that could be migrated at once).
Once a VM is selected for migration, we provide a notification to the guest that a migration is
imminent. After a waiting period, a target host is selected and the host is asked to set up a new, empty “target” VM to receive the migrating “source” VM. Authentication is used to establish a connection between the source and target.
There are three stages involved in the VM’s migration:
- During pre-migration brownout, the VM is still executing on the source, while most state is sent from the source to the target. For instance, we copy all the guest memory to the target, while tracking the pages that have been re-dirtied on the source. The time spent in pre-migration brownout is a function of the size of the guest memory and the rate at which pages are being dirtied.
- During blackout, which is a very brief moment when the VM is not running anywhere, it is paused, and all the remaining state required to begin running the VM on the target is sent. We go into blackout when sending state during pre-migration brownout reaches a point of diminishing returns. We use an algorithm that balances numbers of bytes of memory being sent against the rate at which the guest VM is dirtying pages, amongst other things.
- During post-migration brownout, the VM executes on the target. The source VM is present, and may be providing supporting functionality for the target. For instance, until the network fabric has caught up with the new location of the VM, the source VM provides forwarding services for packets to and from the target VM.
Finally, the migration is complete, and the system deletes the source VM. Customers can see that the migration took place in their
logs.
Our goal for every transparent maintenance event is that not a single VM is killed. In order to meet that bar, we test live migration with a very high level of rigor. We’re using fault-injection to trigger failures at all the interesting points in the migration algorithm. We generate both active and passive failures for each component. At the peak of development testing (for months) we were doing tens of thousands of migrations every day.
Achieving this complex, multi-faceted process requires deep integration throughout the infrastructure and a powerful set of scheduling, orchestration and automation processes.
Conclusion
Live migration technology lets us maintain our infrastructure in top shape without impacting our guest VMs. One of our reviewers even claimed we’ve granted VMs
immortality. We’re able to keep our VMs running for long periods of time in the face of regular and unplanned maintenance requirements and in spite of the many issues that arise requiring the reboot of physical machines.
We’re fortunate that some of the recent security issues that have affected other cloud providers haven’t affected us, but if and when a new vulnerability affects our stack, we’ll be able to help keep Compute Engine protected without affecting our customers’ VMs.
-Posted by Miche Baker-Harvey, Tech Lead/Manager, VM Migration
.jpg)
.png)
Follow