Google Cloud Platform Blog
5 must-see security sessions at Google Cloud Next '17
Friday, February 10, 2017
By David B. Cross, Cloud Security Engineering Director
So many sessions, so little time.
Google Cloud Next '17
, taking place next month, features over 200 breakout sessions; many geared directly at security professionals. If you only have time on your schedule for a few security breakouts, here are the ones you can’t afford to miss.
For a foundation in how we help secure
Google Cloud Platform
(GCP)
—
and for a peek at the various security threats Google grapples with day in and day out
—
check out “
Lessons learned from securing both Google and Google Cloud customers
.” Here, Andy Chang, Google Senior Product Manager, will discuss the various layers of Google security, its security team and what it’s learned from preventing, detecting and responding to cyber attacks over the years.
Now that you better understand our security features, learn how to attach your on-prem environment to Google Cloud via Virtual Private Cloud. In “
How to create a secure, private environment in the cloud and on-prem with Google Cloud Virtual Private Clouds
,” Ines Envid, Google Product Manager, and Neha Pattan, Software Engineer, will show you how to build a sandbox to run your cloud workloads alongside on-prem applications, as well as how to integrate with GCP’s machine learning, big data and storage services.
More and more, building cloud applications means building mobile applications. In “
Security first for a mobile first strategy
,” director of Android security Adrian Ludwig discusses the multiple layers of protection that the Android platform provides to help keep business and personal information safe.
We do our part on the backend, but it’s up to you to write quality apps. In “
Designing secure UX into your products
,” Google senior developer advocate Mandy Waite discusses best practices you should follow when building apps and services, plus how Google protects against threats like malware and phishing attacks.
At a fundamental level, for many of our customers, keeping their business safe is rooted in protecting email. In “
Trends in data security
,” Gilad Golan, Google Director for Security and Data Protection, and Nicolas Lidzborski, Staff Software Engineer, describe our latest innovations in email security
—
and how you can apply those to your organization.
As an added bonus, we’re also offering a full-day
security bootcamp
before the show.
Register
now to reserve your spot, and see you at NEXT!
Fuzzing PCI express: security in plaintext
Thursday, February 9, 2017
By Julia Hansbrough, Software Engineer
Google recently launched
GPUs
on
Google Cloud Platform
(GCP), which will allow customers to leverage this hardware for highly parallel workloads. These GPUs are connected to our cloud machines via a variety of PCIe switches, and that required us to have a deep understanding of PCIe security.
Securing PCIe devices requires overcoming some inherent challenges. For instance, GPUs have become far more complex in the past few decades, opening up new avenues for attack. Since GPUs are designed to directly access system memory, and since hardware has historically been considered trusted, it's difficult to ensure all the settings to keep it contained are set accurately, and difficult to ensure whether such settings even work. And since GPU manufacturers don't make the source code or binaries available for the GPU's main processes, we can't examine those to gain more confidence. You can read more about the challenges presented by the PCI and PCIe specs
here
.
With the risk of malicious behavior from compromised PCIe devices, Google needed to have a plan for combating these types of attacks, especially in a world of cloud services and publicly available virtual machines. Our approach has been to focus on mitigation: ensuring that compromised PCIe devices can’t jeopardize the security of the rest of the computer.
Fuzzing to the rescue
A key weapon in our arsenal is fuzzing, a testing technique that uses invalid, unexpected or random inputs to expose irregular behavior, such as memory leaks, crashes, or undocumented functionality. The hardware fuzzer we built directly tests the behavior of the PCIe switches used by our cloud GPUs.
After our initial research into the PCIe spec, we prepared a list of edge cases and device behaviors that didn’t have clearly defined outcomes. We wanted to test these behaviors on real hardware, and we also wanted to find out whether real hardware implemented the well defined parts of the spec properly. Hardware bugs are
actually quite common
, but many security professionals assume their absence, simply trusting the manufacturer. At Google, we want to verify every layer of the stack, including hardware.
Our plan called for a fuzzer that was highly specialized, and designed to be effective against the production configurations we use in our cloud hardware. We use a variety of GPU and switch combinations on our machines, so we set up some programmable network interface controllers (NICs) in similar configurations to simulate GPU memory accesses.
Our fuzzer used those NICs to aggressively hammer the port directly upstream from each NIC, as well as any other accessible ports in the network, with a variety of memory reads and writes. These operations included a mixture of targeted attacks, randomness and "lucky numbers" that tend to cause problems on many hardware architectures. We wanted to detect changes to the configuration of any port as a result of the fuzzing, particularly the port's secondary and subordinate bus numbers. PCIe networks with Source Validation enabled are governed primarily by these bus numbers, which dictate where packets can and cannot go. Being able to reconfigure a port's secondary or subordinate bus numbers could give you access to parts of the PCIe network that should be forbidden.
Our security team reviewed any suspicious memory reads or writes to determine if they represent security vulnerabilities, and adjusted either the fuzzer or our PCIe settings accordingly.
We discovered some curiosities. For instance, on one incorrect configuration, some undocumented debug registers on the switch were incorrectly exposed to downstream devices, which we discovered could cause serious malfunctioning of the switch under certain access patterns. If a device can cause out-of-spec behavior in the switch it’s connected to, it may be able to cause insecure routing, which would compromise the entire network. The value of fuzzing is its ability to find vulnerabilities in undocumented and undefined areas, outside the normal set of behaviors and operations defined in the spec. But by the end of the process, we had determined a minimum set of ACS features necessary to securely run GPUs in the cloud.
Let's check out those memory mappings too
When you make use of a GPU on a local computer through the root OS, it has direct memory access to the computer’s memory. This is very fast and straightforward. However, that model doesn't work in a virtualized environment like
Google Compute Engine
.
When a virtual machine is initialized, a set of page tables maps the guest's physical memory to the host's physical memory, but the GPU has no way to know about those mappings, and thus will attempt to write to the wrong places. This is where the Input–output memory management unit (IOMMU) comes in. The IOMMU is a page table, translating GPU accesses into DRAM/MMIO reads and writes. It's implemented in hardware, which reduces the remapping overhead.
This means the IOMMU is performing a pretty delicate operation. It’s mapping its own I/O virtual addresses into host physical addresses. We wanted to verify that the IOMMU was functioning correctly, and ensure that it was enabled any time a device may be running untrusted code, so that there would be no opportunity for unfiltered accesses.
Furthermore, there were features of the IOMMU that we didn't want, like compatibility interrupts. This is a type of interrupt that exists to support older Intel platforms that lack the interrupt-remapping capabilities that the IOMMU gives you. They're not necessary for modern hardware, and leaving them enabled allows guests to trigger unexpected MSIs, machine reboots, and host crashes.
The most interesting challenge here is protecting against PCIe's Address Translation Services (ATS). Using this feature, any device can claim it's using an address that's already been translated, and thus bypass IOMMU translation. For trusted devices, this is a useful performance improvement. For untrusted devices, this is a big security threat. ATS could allow a compromised device to ignore the IOMMU and write to places it shouldn't have access to.
Luckily, there's an ACS setting that can disable ATS for any given device. Thus, we disabled compatibility interrupts, disabled ATS, and had a separate fuzzer attempt to access memory outside the range specifically mapped to it. After some aggressive testing we determined that the IOMMU worked as advertised and could not be bypassed by a malicious device.
Conclusions
Beyond simply verifying our hardware in a test environment, we wanted to make sure our hardware remains secure in all of production. Misconfigurations are
likely the biggest source of major outages in production environments
, and it's a similar story with security vulnerabilities. Since ACS and IOMMU can be enabled or disabled at multiple layers of the stack—potentially varying between kernel versions, the default settings of the device, or other seemingly-minor tweaks—we would be remiss to rely solely on isolated unit tests to verify these settings. So, we developed tooling to monitor the ACS and IOMMU settings in production, so that any misconfiguration of the system could be quickly detected and rolled back.
As much as possible, it's good practice not to trust hardware without first verifying that it works correctly, and our targeted attacks and robust fuzzing allowed us to settle on a list of ACS settings that allow us to share GPUs with cloud users securely. This has resulted in being able to provide GPUs to our customers with a high degree of confidence in the security of the underlying system. Stay tuned for more posts that detail how we implement security at Google Cloud.
Windows and .NET Codelabs: an overview
Monday, February 6, 2017
By
Mete Atamel
, Developer Advocate
Google Developers Codelabs
provide guided coding exercises to get hands-on experience with a wide range of topics such as Android Wear, Firebase and Web.
Google Cloud Platform
(GCP) has its own
section
, with codelabs for Google Compute Engine, Google App Engine, Kubernetes and many more.
We’re always working to create new content, and I’m happy to announce that we now have new codelabs for running Windows and .NET apps on GCP, with their own
dedicated page
. Here’s an overview to help you get started.
First, if you’re a .NET developer, you probably love and use Visual Studio daily.
Install and use Cloud Tools for Visual Studio
teaches you how to install and use our GCP plugin for Visual Studio.
If you're a traditional ASP.NET developer writing apps for Windows Server,
Deploy Windows Server with ASP.NET Framework to Compute Engine
is the first codelab you should try. It teaches you how to deploy a Windows Server with ASP.NET Framework on Compute Engine.
Once you have your Windows Server deployed, you can try Deploy ASP.NET app to Windows Server on Compute Engine. It shows you how to take a simple ASP.NET app and publish it to your Windows Server from Visual Studio. These two codelabs provide a good understanding of traditional ASP.NET development and deployment on GCP.
If you've already made the switch to ASP.NET Core, the new multi-platform version of ASP.NET, then start with
Build and launch an ASP.NET Core app from Google Cloud Shell
to learn how to build and test a basic ASP.NET Core app from Cloud Shell. The whole codelab can be done inside your browser, which is pretty cool!
Afterwards, you can take this app and either deploy to App Engine or to Kubernetes on Google Container Engine. App Engine is definitely the easier path, and
Deploy an ASP.NET Core app to App Engine
can show you the way. If you want to tackle Kubernetes, you can follow
Deploy ASP.NET Core app to Kubernetes on Container Engine
to create a Kubernetes cluster of ASP.NET Core pods.
Regardless of where you deploy your app, you need to manage it, and we have a codelab on PowerShell to help with that:
Install and use Cloud Tools for PowerShell
teaches the use of our PowerShell cmdlets to access and manage GCP resources via PowerShell scripts.
I hope this gives you a good overview of where to start with Windows and .NET codelabs on GCP. We'll be adding more to our dedicated page for
Windows and .NET
, so be sure to check back regularly.
8 must-see sessions for application developers at Google Cloud Next '17
Thursday, February 2, 2017
By Chris Sells, Product Manager, Google Cloud
With 200-plus sessions to choose from at
Google Cloud Next ‘17
on March 8 - 10, there’s a little bit of something for everyone. But if you’re an application developer coming to the show, here are a few sessions in particular that I recommend you check out.
The most popular application development platform on
Google Cloud Platform
(GCP) is Java. If that describes your shop, be sure to check out "
Power your Java workloads on Google Cloud Platform
," with
Amir Rouzrokh
, Product Manager for all things Java on GCP. Amir will show attendees how to deploy a Spring Boot application to GCP, plus how to use
Cloud Tools for IntelliJ
to troubleshoot production problems.
In the past year, we’ve also made big strides supporting Microsoft platforms like ASP.NET on GCP. For a taste, check out Google Developer Advocate Mete Atamel’s talk “
Take your ASP.NET apps to the next level with Google Cloud
,” where he’ll cover how to migrate an ASP.NET app to GCP, how to work with our Powersehll cmdlets and Visual Studio plugins and how to tie into advanced GCP services like
Google Cloud Storage
,
Cloud Pub/Sub
and our Machine Learning APIs. Then there’s "
Running .NET and containers in Google Cloud Platform
" with Jon Skeet and Chris Smith, who will show you the next generation of OSS, cross-platform .NET Core apps running in Containers in
Google App Engine
and in
Kubernetes
. (And if that's still not enough, you can always sign up for the full day
Windows on GCP bootcamp
before the show.)
Speaking of App Engine, here’s your chance to learn all about App Engine flexible environment, our next-generation PaaS offering. In "
You can run that on App Engine?
," Product Manager Justin Beckwith shows you how to easily build production-scale web apps for an expanded variety of application patterns.
We’re also excited to talk more about Apigee, the API management platform we acquired in the fall. At “
Using Apigee Edge to create and publish APIs that developers love
,” Greg Brail, Principal Software Engineer and Prithpal Bhogil, GCP Sales Engineer, will walk developers through how to use Apigee Edge and best practices for building developer-friendly APIs.
Newcomers to GCP may also enjoy Google Cloud Product Manager Omar Ayoub’s session, "
Developing made easy on Google Cloud Platform
", where we’ll provide an overview of all the different libraries, IDE and framework integrations and other tools for developing applications on GCP.
But the hottest application development topic at Next '17 is arguably
Google Cloud Functions
, our event-based computing platform that we announced in alpha last year. For an introduction to Cloud Functions, there’s "
Building serverless applications with Google Cloud Functions
" with Product Manager Jason Polites. Mobile developers should also consider "
Google Cloud Functions and Firebase
", marrying our mobile backend as a service offering with Cloud Functions’ lightweight, asynchronous compute.
Of course, that’s just the tip of the iceberg when it comes to application development sessions. Be sure to check out the
full session catalog
, and
register
sooner rather than later to secure your spot in the most coveted sessions and bootcamps.
Delivering a better platform for your SQL Server Enterprise workloads
Wednesday, February 1, 2017
By Amruta Gulanikar, Product Manager, Google Cloud Platform
Our goal at
Google Cloud Platform
(GCP) is to be the best enterprise cloud environment. Throughout 2016, we worked hard to ensure that Windows developers and IT administrators would feel right at home when they came to GCP: whether it’s
building an ASP.NET application
with their favorite tools like
Visual Studio
and
PowerShell
, or deploying the
latest version of Windows Server
onto
Google Compute Engine
.
Continuing our work in providing great infrastructure for enterprises running Windows, we’re pleased to announce pre-configured images for Microsoft
SQL Server Enterprise
and
Windows Server Core
on Compute Engine. High-availability and disaster recovery are top of mind for our larger customers, so we’re also announcing support for SQL Server AlwaysOn Availability Groups and persistent disk snapshots integrated with Volume Shadow Copy Service (VSS) on Windows Server. Finally, all of our Windows Server images are now
enabled
with Windows Remote Management support, including our Windows Server Core 2016 and 2012 R2 images.
SQL Server Enterprise Edition images on GCE
You can now launch Compute Engine VMs with Microsoft SQL Server Enterprise Edition pre-installed, and
pay by the minute
for SQL Server Enterprise and Windows Server licenses. Customers can also choose to
bring their own licenses
for SQL Server Enterprise.
We now support
pre-configured images
for the following versions in Beta:
SQL Server Enterprise 2016
SQL Server Enterprise 2014
SQL Server Enterprise 2012
Supported SQL Server images available on Compute Engine (click to enlarge)
SQL Server Enterprise
targets mission-critical workloads by supporting more cores, higher memory and important enterprise features, including:
In-memory tables and indexes
Row-level security and encryption for data at rest or in motion
Multiple read-only replicas for integrated HA/DR and read scale-out
Business intelligence and rich visualizations on all platforms, including mobile
In-database advanced analytics with R
Combined with Google’s world-class infrastructure, SQL Server instances running on Compute Engine benefit from
price-to-performance
advantages, highly
customizable VM sizes
and state-of-the-art networking and security capabilities. With
automatic sustained use discounts
and the prospect of retiring hardware and associated maintenance on the horizon, customers can achieve total costs lower than those of other cloud providers.
To get started,
learn how to create SQL Server instances easily on Google Compute Engine
.
High-availability and disaster recovery for SQL Server VMs
Mission-critical SQL Server workloads require support for high-availability and disaster recovery. To achieve this, GCP supports Windows Server Failover Clustering (WSFC) and SQL Server AlwaysOn Availability Groups. AlwaysOn Availability Groups is SQL Server’s flagship HA/DR solution, allowing you to configure replicas for automatic failover in case of failure. These replicas can be readable, allowing you to offload read workloads and backups.
Compute Engine users can now configure AlwaysOn Availability Groups. This includes configuring replicas on VMs in different isolated zones as described in these
instructions
.
A highly available SQL Server reference architecture using Windows Server Failover Clustering and SQL Server AlwaysOn Availability Groups (click to enlarge)
Better backups with VSS-integrated persistent disk snapshots for Windows VMs
Being able to take snapshots in coordination with Volume Shadow Copy Service ensures that you get application-consistent snapshots for persistent disks attached to an instance running Windows -- without having to shut it down. This feature is useful when you want to take a consistent backup for VSS-enabled applications like SQL Server and Exchange Server
—
without affecting the workload running on the VMs.
To get started with VSS-enabled persistent disk snapshots, select
Snapshots
under the Cloud Console Compute Engine page. There you'll see a new check-box on the disk snapshot creation page that allows you to specify whether a snapshot should be VSS-enabled.
(click to enlarge)
This feature can also be invoked via the gcloud SDK and API, following these
instructions
.
Looking ahead
GCP’s expanded support for SQL Server images and high availability are our latest efforts to improve Windows support on Compute Engine, and to build a cloud environment for enterprise Windows that leads the industry. Last year
we expanded our list of pre-configured images
to include SQL Server Standard, SQL Server Web and Windows Server 2016, and announced comprehensive
.NET developer solutions
, including a
.NET client library
for all GCP APIs through NuGet. We have lots more in store for the rest of 2017!
For more resources on Windows Server and Microsoft SQL Server on GCP, check out
cloud.google.com/windows
and
cloud.google.com/sql-server
. And for hands-on training on how to deploy and manage Windows and SQL Server workloads on GCP, come to the
GCP NEXT ‘17 Windows Bootcamp
. Finally, if you need help migrating your Windows workloads, don’t hesitate to
contact us
. We’re eager to
hear your feedback
!
SLOs, SLIs, SLAs, oh my - CRE life lessons
Tuesday, January 31, 2017
By AJ Ross and Adrian Hilton, Customer Reliability Engineers, and Dave Rensin, Director of Customer Reliability Engineering
Last week
on
CRE life lessons
, we discussed how to come up with a precise numerical target for system availability. We term this target the Service Level Objective (SLO) of our system. Any discussion we have in future about whether the system is running sufficiently reliably and what design or architectural changes we should make to it must be framed in terms of our system continuing to meet this SLO.
We also have a direct measurement of SLO conformance: the frequency of successful probes of our system. This is a Service Level Indicator (SLI). When we evaluate whether our system has been running within SLO for the past week, we look at the SLI to get the service availability percentage. If it goes below the specified SLO, we have a problem and may need to make the system more available in some way, such as running a second instance of the service in a different city and load balancing between the two.
Why have an SLO at all?
Suppose that we decide that running our
aforementioned Shakespeare service
against a formally defined SLO is too rigid for our tastes; we decide to throw the SLO out of the window and make the service “as available as is reasonable.” This makes things easier, no? You simply don’t mind if the system goes down for an hour now and then. Indeed, perhaps downtime is normal during a new release and the attending stop-and-restart.
Unfortunately for you, customers don’t know that. All they see is that Shakespeare searches that were previously succeeding have suddenly started to return errors. They raise a high-priority ticket with support, who confirms that they see the error rate and escalate to you. Your on-call engineer investigates, confirms this is a known issue, and responds to the customer with “this happens now and again, you don’t have to escalate.” Without an SLO, your team has no principled way of saying what level of downtime is acceptable; there's no way to measure whether or not this a significant issue with the service. and you cannot terminate the escalation early with “Shakespeare search service is currently operating within SLO.” As our colleague Perry Lorier likes to say, “if you have no SLOs, toil is your job.”
The SLO you run at becomes the SLO everyone expects
A common pattern is to start your system off at a low SLO, because that’s easy to meet: you don’t want to run a 24/7 rotation, your initial customers are OK with a few hours of downtime, so you target at least 99% availability
—
1.68 hours downtime per week. But in fact, your system is fairly resilient and for six months operates at 99.99% availability
—
down for only a few minutes per month.
But then one week, something breaks in your system and it’s down for a few hours. All hell breaks loose. Customers page your on-call complaining that your system has been returning 500s for hours. These pages go unnoticed, because on-call leaves their pagers on their desks overnight, per your SLO which only specifies support during office hours.
The problem is, customers have become accustomed to your service being always available. They’ve started to build it into their business systems on the assumption that it’s always available. When it’s been continually available for six months, and then goes down for a few hours, something is clearly seriously wrong. Your excessive availability has become a problem because now it’s the expectation. Thus the expression, “An SLO is a target from above and below”
—
don’t make your system very reliable if you don’t intend and commit to it to being that reliable.
Within Google, we implement periodic downtime in some services to prevent a service from being overly available. In the
SRE Book
, our colleague Marc Alvidrez tells a story about our internal lock system
—
Chubby. Then, there’s the set of test front-end servers for internal services to use in testing, allowing those services to be accessible externally. These front-end servers are convenient but are explicitly not intended for use by real services; they have a one business day support SLA, and so can be down for 48 hours before the support team is even obligated to think about fixing them. Over time, experimental services that used those front-ends started to become critical; when we finally had a few hours of downtime on the front-ends, it caused widespread consternation.
Now we run a quarterly planned-downtime exercise with these front-ends. The front-end owners send out a warning, then block all services on the front-ends except for a small whitelist. They keep this up for several hours, or until a major problem with the blockage appears; the blockage can be quickly reversed in that case. At the end of the exercise the front-end owners receive a list of services that use the front-ends inappropriately, and work with the service owners to move them to somewhere more suitable. This downtime exercise keeps the front-end availability suitably low, and detects inappropriate dependencies in time to get them fixed.
Your SLA is not your SLO
At Google, we distinguish between a Service-Level Agreement (SLA) and a Service-Level Objective (SLO). An SLA normally involves a promise to someone using your service that its availability should meet a certain level over a certain period, and if it fails to do so then some kind of penalty will be paid. This might be a partial refund of the service subscription fee paid by customers for that period, or additional subscription time added for free. The concept is that going out of SLA is going to hurt the service team, so they'll push hard to keep it within SLA.
Because of this, and because the principle availability shouldn’t be much better than the SLO, the SLA is normally a looser objective than the SLO. This might be expressed in availability numbers: for instance, an availability SLA of 99.9% over 1 month with an internal availability SLO of 99.95%. Alternatively the SLA might only specify a subset of the metrics comprising the SLO.
For example, with our Shakespeare search service, we might decide to provide it as an API to paying customers in which a customer pays us $10K per month for the right to send up to one million searches per day. Now that money is involved, we need to specify in the contract how available they can expect the service to be, and what happens if we breach that agreement. We might say that we'll provide the service at a minimum of 99% availability, following the definition of successful queries given previously. If the service drops below 99% availability in a month, then we'll refund $2K; if it drops below 80% then, we'll refund $5K.
If you have an SLA that's different from your SLO, as it almost always is, it’s important for your monitoring to measure SLA compliance explicitly. You want to be able to view your system’s availability over the SLA calendar period, and easily see if it appears to be in danger of going out of SLA. You'll also need a precise measurement of compliance, usually from logs analysis. Since we have an extra set of obligations (in the form of our SLA) to paying customers, we need to measure queries received from them separately from other queries (we might not mind dropping queries from non-paying users if we have to start load shedding, but we
really
care about any query from the paying customer that we fail to handle properly). That’s another benefit of establishing an SLA
—
it’s an unambiguous way to prioritize traffic.
When you define your SLA, you need to be extra-careful about which queries you count as legitimate. For example, suppose that you give each of three major customers (whose traffic dominates your service) a quota of one million queries per day. One of your customers releases a buggy version of their mobile client, and issues two million queries per day for two days before they revert the change. Over a 30-day period you’ve issued approximately 90 million good responses, and two million errors; that gives you a 97.8% success rate. You probably don’t want to give all your customers a refund as a result of this; two customers had all their queries succeed, and the customer for whom two million out of 32 million queries were rejected brought this upon themselves. So perhaps you should exclude all “out of quota” response codes from your SLA accounting.
On the other hand, suppose you accidentally push an empty quota specification file to your service before going home for the evening. All customers receive a default 1000 queries per day quota. Your three top customers get served constant “out of quota” errors for 12 hours until you notice the problem when you come into work in the morning, and revert the change. You’re now showing 1.5 million rejected queries out of 90 million for the month, a 98.3% success rate. This is all your fault: counting this as 100% success for 88.5M queries is missing the point and a moral failure for measuring the SLA.
Conclusion
SLIs, SLOs and SLAs aren’t just useful abstractions. Without them you cannot know if your system is reliable, available, or even useful. If they don’t tie explicitly back to your business objectives then you have no idea if the choices you make are helping or hurting your business. You also can’t make honest promises to your customers.
If you’re building a system from scratch, make sure that SLIs, SLOs and SLAs are part of your system requirements. If you already have a production system but don’t have them clearly defined then that’s your highest priority work.
To summarize:
If you want to have a reliable service, you must first define “reliability.” In most cases that actually translates to availability.
If you want to know how reliable your service is, you must be able to measure the rates of successful and unsuccessful queries; these will form the basis of your SLIs.
The more reliable the service, the more it costs to operate. Define the lowest level of reliability that you can get away with, and state that as your Service Level Objective (SLO).
Without an SLO, your team and your stakeholders cannot make principled judgements about whether your service needs to be made more reliable (increasing cost and slowing development) or less reliable (allowing greater velocity of development).
If you’re charging your customers money you'll probably need an SLA, and it should be a little bit looser than your SLO.
As an SRE (or DevOps professional), it's your responsibility to understand how your systems serve the business in meeting those objectives, and, as much as possible, control for risks that threaten the high-level objective. Any measure of system availability that ignores business objectives is
worse than worthless
because it obfuscates the actual availability, leading to all sorts of dangerous scenarios, false senses of security and failure.
For those of you who wrote us thoughtful comments and questions from our last article, we hope this post has been helpful. Keep the feedback coming!
N. B.
Google Cloud Next '17
is fewer than seven weeks away.
Register now
to join Google Cloud SVP Diane Greene, Google CEO Sundar Pichai, and other luminaries for three days of keynotes, code labs, certification programs, and over 200 technical sessions. And for the first time ever, Next '17 will have a dedicated space for attendees to interact with Google experts in Site Reliability Engineering and Developer Operations.
Guest post: building IoT applications with MQTT and Google Cloud Pub/Sub
Monday, January 30, 2017
By Rick Erickson, Co-founder and EVP and Paul Lundberg, CTO, Agosto
[Editor’s note: Today we hear from Agosto, a Google Cloud Premier Partner that has been building products and delivering services on
Google Cloud Platform
(GCP) since 2012, including Internet of Things applications. Read on to learn about Agosto’s work to build an MQTT service broker for Google Cloud Pub/Sub, and how you can incorporate it into your own IoT applications.]
One of our key practice areas is Internet of Things (IoT). Using the many components of GCP, we’ve helped customers rapidly move their ideas from product concept to launch.
Along the way, we evaluated several IoT platforms and repeatedly came to the conclusion that we’d be better off staying on the GCP stack than a single IoT platform with costly licensing hooks and closed-source practices. Our clients also like being able to build scalable, functional prototypes using pre-existing and standard reference architectures and tools.
One of the many challenges we faced along the way was picking an efficient transport for two-way messaging between “things” and GCP. After evaluating a number of emerging and mature protocols, we settled on Message Queuing Telemetry Transport (MQTT). The MQTT protocol has been around since the early 2000’s and is now an ISO Standard. Originated in 1999 by Andy Stanford-Clark and Arlen Nipper, it's lightweight, has solid documentation and has tens of thousands of production deployments. Furthermore, many existing pre-IoT or “Machine to Machine” projects already use MQTT as their transport from embedded device to the back-office. With MQTT, we’ve been able to increase velocity and reduce complexity for our IoT products and services.
MQTT is a great transport protocol, but it can be challenging to manage at scale, particularly when it comes to scaling message storage and delivery systems. As one of the earliest Google partners to develop a set of reusable tools, reference architectures and methods for accelerating IoT products to market, we’ve been impressed with
Google Cloud Pub/Sub
, a durable, low-latency and scalable service for handling many-to-many asynchronous messaging. But Cloud Pub/Sub uses HTTPS to transfer data. Over numerous small requests, all those HTTP headers add up to a lot of extra data
—
a no-go when you’re dealing with a constrained device that communicates over a mobile network, and where you pay for each byte in mobile data charges, battery usage
—
or both.
We needed to bridge the gap between IoT-connected devices and Cloud Pub/Sub, and began investigating ways to connect MQTT to Cloud Pub/Sub using and extending
RabbitMQ
.
After initial load tests showed this approach was viable, Google asked Agosto to develop an open-source, highly performant MQTT connection broker that integrates with Cloud Pub/Sub. With low network overhead (Agosto has seen up to 10x less compared to HTTPS in scenarios we've tested) and high throughput, MQTT is a natural fit for many scenarios.
The resulting message broker integrates messaging between connected devices using a MQTT client and Cloud Pub/Sub; RabbitMQ performs the protocol conversion for two-way messaging between the device and Cloud Pub/Sub. This means administrators of the RabbitMQ compute infrastructure don't have to concern themselves with managing the durability of the data, or scaling storage.
Our message broker can support both small and very large GCP projects. For example, with smaller projects and IoT prototypes, you can rapidly deploy a single node of Agosto’s MQTT to Pub/Sub Connection Broker supporting up to 120,000 messages per minute for as little as $25/month for the compute costs. Larger production deployments with load-balanced brokers can support millions of concurrent connections and much higher throughput.
Download the broker, follow the instructions and learn more about leveraging MQTT and GCP for your IoT project.
GitHub:
https://github.com/Agosto/gcp-iot-adapter
And if you're looking for a more customized implementation of our MQTT to Pub/Sub Connection broker, visit our website to learn more about
our offerings
.
Don't Miss Next '17
Use promo code NEXT1720 to save $300 off general admission
REGISTER NOW
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Labels
Announcements
56
Big Data & Machine Learning
91
Compute
156
Containers & Kubernetes
36
CRE
7
Customers
90
Developer Tools & Insights
80
Events
34
Infrastructure
24
Management Tools
39
Networking
18
Open Source
105
Partners
63
Pricing
24
Security & Identity
23
Solutions
16
Stackdriver
19
Storage & Databases
111
Weekly Roundups
16
Archive
2017
Feb
Jan
2016
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Feed
Subscribe by email
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow