If you’re not familiar with Cloud9, you should be! Cloud9 is a development environment in the cloud that offers both a rich code editor and Ubuntu command line with sudo rights. With Cloud9 your development environment is entirely online, allowing you to code from any machine and freeing you from the hassle of managing a local environment.
If you’re not familiar with Cloud9, you should be! Cloud9 is a development environment in the cloud that offers both a rich code editor and Ubuntu command line with sudo rights. With Cloud9 your development environment is entirely online, allowing you to code from any machine and freeing you from the hassle of managing a local environment.



Now, you can easily create a new Cloud9 workspace connected with a Google Cloud Platform project. Your GCP-ready Cloud9 workspace comes preinstalled with the Cloud SDK and gcloud command line tool, and allows you to build and deploy your application to Google App Engine directly within the IDE. To learn how, view Cloud9’s documentation.




Getting started


Getting started is easy; first, authenticate with Google in Cloud9. Then, create a workspace for your Cloud Platform project (make sure you’ve created a project in Cloud Platform first). The workspace is configured to store and access your remote source code in Cloud Source Repositories.






Using gcloud and Google Cloud SDK


The Google Cloud SDK comes pre-installed on your workspace's virtual machine and is automatically configured to access your project on Cloud Platform.








Edit, build and deploy directly from Cloud9


With Cloud9, you can edit your project’s code and push changes back to your cloud source repository. When you’re ready, build and deploy to App Engine directly from the IDE.










What’s next for Cloud9 and Cloud Platform




While Cloud9 currently offers support for App Engine Java-based applications, over the next few weeks, they’ll be adding support for additional programming languages and features. If you have questions or comments, please visit Cloud9’s community site.



Want to see it in action? See how quickly you can set up a Cloud Platform project in Cloud9.





We’re very pleased to share Cloud9’s support for Cloud Platform and we’re excited for the languages and features to come. Stay tuned!



- Posted by Chris Sells, Product Manager
Labels: ,

In the coming months, we will be working closely with Red Hat to integrate and deliver OpenShift Dedicated — Red Hat’s managed container application platform — to customers on Google Cloud Platform. This will make adopting containers easier for customers.
In the coming months, we will be working closely with Red Hat to integrate and deliver OpenShift Dedicated — Red Hat’s managed container application platform — to customers on Google Cloud Platform. This will make adopting containers easier for customers.



We are committed to helping you get the most out of cloud, whether it be purely public, or a hybrid of public and private. Being the Open Cloud and growing investments in open source tools like Kubernetes are two facets of this. Our collaboration with Red Hat is another.



Together, we’ve made Google Compute Engine (GCE) a certified environment for Red Hat offerings, and have worked closely to unlock the power of containers through the Kubernetes project and the creation of the Cloud Native Computing Foundation. We’re now deepening this relationship to integrate OpenShift Dedicated with Google Cloud Platform (GCP) services. In this initial phase, you’ll have access to improved support for containers using Kubernetes and OpenShift, as well as access to powerful GCP services designed to help you make better use of data.




Helping Customers Adopt and Operationalize Containers




Both Google and Red Hat have been hearing a consistent story from enterprise customers, who’ve told us that they plan to move containers from experimental projects to supporting production workloads. In doing so, they aim to add:




  • Improved security: Confidence that containerized applications are developed, deployed and maintained on validated platforms with appropriate provenance and governance.

  • Services and ecosystem: Delivering lifecycle services and open interfaces for partners to give developers and operators the ability to build and execute a broad array of microservice based applications.

  • Dynamic scheduling: Provide frictionless resources and management to enable flexible deployment of containers as workloads change.

  • Storage: Resilient access to application data regardless of container deployment locality.

  • Cross cloud portability and hybrid deployments: Consistent container deployment frameworks, resources and platforms wherever development and deployment occurs.




We’ve heard these requests and believe the combination of Google Cloud Platform and Kubernetes plus Red Hat OpenShift will help.



OpenShift inherits easy portability between and across environments using Kubernetes, enabling hybrid cloud deployments. Red Hat plans to offer OpenShift Dedicated (its managed OpenShift cloud service) on Google Cloud Platform. This service is underpinned by Red Hat Enterprise Linux, and marries Red Hat’s enterprise-grade container application platform with Google’s 10 years of operational expertise around containers. This allows you to accelerate development and enable your developers to focus on application creation rather than operational overhead.



Additionally, we’re pleased to announce that Google’s working on integrating Google Cloud Platform services (including big data, analytics and storage services) with OpenShift Dedicated, with the goal of enabling Red Hat customers to natively access these Google Cloud Platform offerings.




The Best of Open Source and Cloud




With the increasing breadth and maturity of Google Cloud Platform’s offerings, we’re well-suited to complement and integrate with on-premise enterprise infrastructure. Our expanding relationship with Red Hat makes this especially true for enterprise-focused developers in need of stable, more secure and open source solutions that include Google’s cloud services and global infrastructure footprint. If you’re interested in learning more about our plans for OpenShift Dedicated on Google Cloud Platform or becoming a beta tester, please let us know here.



- Posted by Martin Buhr, Product Manager, Google Cloud Platform 
Labels: ,

Editor's update February 9, 2016: The Dataflow submission to the Apache Incubator was accepted on February 1, 2016, and the resulting project is now called Apache Beam.



Imagine if every time you upgrade your servers you had to learn a new programming framework and rewrite all your applications. That might sound crazy, but it’s what happens with big data pipelines.
Editor's update February 9, 2016: The Dataflow submission to the Apache Incubator was accepted on February 1, 2016, and the resulting project is now called Apache Beam.



Imagine if every time you upgrade your servers you had to learn a new programming framework and rewrite all your applications. That might sound crazy, but it’s what happens with big data pipelines.



It wasn't long ago that Apache Hadoop MapReduce was the obvious engine for all things big data, then Apache Spark came along, and more recently Apache Flink, a streaming-native engine. Unlike upgrading hardware, adopting these more modern engines has generally required rewriting pipelines to adopt engine-specific APIs, often with different implementations for streaming and batch scenarios. This can mean throwing away user code that had just been weathered enough to be considered (mostly) bug-free, and replacing it with immature new code. All of this just because the data pipelines needed to scale better, or have lower latency, or run more cheaply, or complete faster.



Adjusting such aspects should not require throwing away well-tested business logic. You should be able to move your application or data pipeline to the appropriate engine, or to the appropriate environment (e.g., from on-prem to cloud) while keeping the business logic intact. But, to do this, two conditions need to be met. First, you need a portable SDK, which can produce programs that can execute on one of many pluggable execution environments. Second, that SDK has to expose a programming model whose semantics are focused on your workload and not on the capabilities of the underlying engine. For example, MapReduce as a programming model doesn’t meet the bill (even though MapReduce as an execution method might be appropriate in some cases) because it cannot productively express low-latency computations.



Google designed Dataflow specifically to address both of these issues. The Dataflow Java SDK has been architected to support pluggable “runners” to connect to execution engines, of which four currently exist: data Artisans created one for Apache Flink, Cloudera did it for Apache Spark, and Google implemented a single-node local execution runner as well as one for Google’s hosted Cloud Dataflow service.



That portability is possible because the Dataflow programming model is focused on real-life streaming semantics, like real event time (as opposed to the time at which the event arrives), and real sessions (as opposed to whatever arbitrary boundary the batch cycle imposes). This allows Dataflow programs to execute in either batch or stream mode as needed, and to switch from one pluggable execution engine to the other without needing to be rewritten.



Today we’re taking another step in this collaboration. Along with participants from Cloudera, data Artisans, Talend, Cask and PayPal, we sent a proposal for Dataflow to become an Apache Software Foundation (ASF) Incubator project. In this proposal the Dataflow model, Java SDK, and runners will be bundled into one incubating project with the Python SDK joining the project in the future. We believe this proposal is a step towards the ability to define one data pipeline for multiple processing needs, without tradeoffs, which can be run in a number of runtimes, on-premise, in the cloud, or locally. Google Cloud Dataflow will remain as a “no-ops” managed service to execute Dataflow pipelines quickly and cost-effectively in Google Cloud Platform.









With Dataflow, you can write one portable data pipeline, which can be used for either batch or stream, and executed in a number of runtimes including Flink, Spark, Google Cloud Dataflow or the local direct pipeline.



We're excited to propose Dataflow as an Apache Incubator project because we believe the Dataflow model, SDK and runners offer a number of unique features in the open-source data space.






  • Pipeline first, runtime second – With the Dataflow model and SDKs, you focus first on defining your data pipelines, not how they'll run or the characteristics of the particular runner executing them.

  • Portability – Data pipelines are portable across a number of runtime engines. You can choose a runtime based on any number of considerations, such as performance, cost or scalability.

  • Unified model – Batch and streaming are integrated into a unified model with powerful semantics, such as windowing, ordering and triggering.

  • Development tooling – The Dataflow SDK contains the tools you need to create portable data pipelines quickly and easily using open-source languages, libraries and tools.






To understand the power of the Dataflow model, we recommend this article on the O’Reilly Radar: The World Beyond Batch: Streaming 102. For more information about Dataflow, you can also:










We're grateful to the Apache Software Foundation and community for their consideration of the Dataflow proposal and look forward to actively participating in open development of Dataflow.



- Posted by Frances Perry (Software Engineer) and James Malone (Product Manager)
Labels:

Popular mobile games can attract millions of players and generate terabytes of game-related data in a short burst of time. This places extraordinary pressure on the infrastructure powering these games and requires scalable data analytics services to provide timely, actionable insights in a cost-effective way.
Popular mobile games can attract millions of players and generate terabytes of game-related data in a short burst of time. This places extraordinary pressure on the infrastructure powering these games and requires scalable data analytics services to provide timely, actionable insights in a cost-effective way.



To address these needs, a growing number of successful gaming companies use Google’s web-scale analytics services to create personalized experiences for their players. They use telemetry and smart instrumentation to gain insight into how players engage with the game and to answer questions like: At what game level are players stuck? What virtual goods did they buy? And what's the best way to tailor the game to appeal to both casual and hardcore players?







A new reference architecture describes how you can collect, archive and analyze vast amounts of gaming telemetry data using Google Cloud Platform’s data analytics products. The architecture demonstrates two patterns for analyzing mobile game events:




  • Batch processing: This pattern helps you process game logs and other large files in a fast, parallelized manner. For example, leading mobile gaming company DeNA moved to BigQuery from Hadoop to get faster query responses for their log file analytics pipeline. In this GDC Lightning Talk video they explain the speed benefits of Google’s analytics tools and how the team was able to process large gaming datasets without the need to manage any infrastructure.



  • Real-time processing: Use this pattern when you want to understand what's happening in the game right now. Cloud Pub/Sub and Cloud Dataflow provide a fully managed way to perform a number of data-processing tasks like data cleansing and fraud detection in real-time. For example, you can highlight a player with maximum hit-points outside the valid range. Real-time processing is also a great way to continuously update dashboards of key game metrics, like how many active users are currently logged in or which in-game items are most popular.





Some Cloud Dataflow features are especially useful in a mobile context since messages may be delayed from the source due to mobile Internet connection issues or batteries running out. Cloud Dataflow's built-in session windowing functionality and triggers aggregate events based on the actual time they occurred (event time) as opposed to the time they're processed so that you can still group events together by user session even if there's a delay from the source.


But why choose between one or the other pattern? A key benefit of this architecture is that you can write your data pipeline processing once and execute it in either batch or streaming mode without modifying your codebase. So if you start processing your logs in batch mode, you can easily move to real-time processing in the future. This is an advantage of the high-level Cloud Dataflow model that was released as open source by Google.









Cloud Dataflow loads the processed data into one or more BigQuery tables. BigQuery is built for very large scale, and allows you to run aggregation queries against petabyte-scale datasets with fast response times. This is great for interactive analysis and data exploration, like the example screenshot above, where a simple BigQuery SQL query dynamically creates a Daily Active Users (DAU) graph using Google Cloud Datalab.







And what about player engagement and in-game dynamics? The BigQuery example above shows a bar chart of the ten toughest game bosses. It looks like boss10 killed players more than 75% of the time, much more than the next toughest. Perhaps it would make sense to lower the strength of this boss? Or maybe give the player some more powerful weapons? The choice is yours, but with this reference architecture you'll see the results of your changes straight away. Review the new reference architecture to jumpstart your data-driven quest to engage your players and make your games more successful, contact us, or sign up for a free trial of Google Cloud Platform to get started.



Further Reading and Additional Resources






- Posted by Oyvind Roti, Solutions Architect



Labels: ,

We recently announced our global user conference in March, GCP NEXT 2016. This will be the largest gathering of the Google Cloud Platform community, where you can meet with developers like you who are using Google Cloud Platform to build amazing applications. You’ll be able to explore the latest cloud technologies, get hands-on with our platform and learn directly from those who built it.
We recently announced our global user conference in March, GCP NEXT 2016. This will be the largest gathering of the Google Cloud Platform community, where you can meet with developers like you who are using Google Cloud Platform to build amazing applications. You’ll be able to explore the latest cloud technologies, get hands-on with our platform and learn directly from those who built it.



We’d love for you to tell us your story at NEXT. Submit a proposal to our call for speakers by January 15, 2016 at 11:59pm PST and share your Google Cloud Platform project and experiences. Submissions should align to one of our four technical track topics:




  • Data and Analytics: Learn how Google Cloud Platform can help you build more intelligent applications and make better, more timely decisions.



  • Infrastructure and Operations: See how Google’s infrastructure — including our networks, storage, security, data center operations and DevOps tools — gives you scale, security and reliability.



  • App and Services Development: Understand how different components of Google Cloud Platform can work together to help you develop and deploy powerful apps.



  • Solutions Showcase: Learn how our customers and other developers are using Google Cloud Platform in production.




And for those of you who’d rather sit back and learn from other developers, we look forward to seeing you there, too. Register today and get our early bird rate.



To keep up to date on GCP NEXT 2016, follow us on Google+, Twitter, and LinkedIn.



- Posted by Julia Ferraioli, Developer Advocate, Google Cloud Platform
Labels:

If you’re a heavy user of Google Cloud Platform, you probably already know about Google Cloud SDK, the powerful command line tool for working with all things Cloud Platform, available for Windows, Linux and OS X. In this post, the Cloud SDK engineering team, based in New York City, shares their favorite power features of the Google Cloud command-line interface (CLI).
If you’re a heavy user of Google Cloud Platform, you probably already know about Google Cloud SDK, the powerful command line tool for working with all things Cloud Platform, available for Windows, Linux and OS X. In this post, the Cloud SDK engineering team, based in New York City, shares their favorite power features of the Google Cloud command-line interface (CLI).




#1. Using cloud service emulators


Whether you’re on your commute without connectivity, want a fast reliable way of running tests, or just want to test your application on your development machine without talking to a remote service, gcloud emulators is your friend. Emulators provide a local mock implementation of Google Cloud services, so you can develop and test core functionality. Currently the Cloud SDK includes Google Cloud Datastore and Google Cloud Pub/Sub emulators, with more to come.



You can start an emulator, such as the emulator for Cloud Datastore, like so:



$ gcloud beta emulators datastore start
...
[datastore] To connect, set host to http://localhost:8967/datastore
[datastore] Admin console is running at http://localhost:8851/_ah/admin



Now, you’ve got a datastore running on your local machine! The API is available at the port on localhost, listed above.



Client libraries such as gcloud-node, gcloud-ruby, gcloud-python, and gcloud-java can be configured to use this local emulator by respecting the DATASTORE_LOCAL_HOST

environment variable.



The gcloud emulators command has a neat little trick for automatically setting environment variables like this for each service.



$ $(gcloud beta emulators datastore env-init)
$ echo $DATASTORE_HOST 
http://localhost:8967



The emulator comes with a simple, web-based console which is also available on localhost. Read more in the gcloud emulators documentation.



(Pro tip from Vilas, Engineer on the Cloud SDK)




#2. Type like the wind with autocompletion




Another power feature of the Cloud CLI is tab auto-completion. You can tab-autocomplete gcloud subcommands, and also for many entities such as your instances and zones. Try it out!



(Pro tip from Mark, Engineer on the Cloud SDK)




#3. Using --format to filter, sort and transform output to CSV, JSON, and more




The Cloud CLI gives you a lot of information about your environment, which you might often want to use as input to another script or program. The --format flag provides an easy way to massage the output into a format that makes sense.



Here’s an example using the --format flag to list the zone and IP of your Google Compute Engine instances in CSV format, sorted by zone and name.



$ gcloud compute instances list \
   --format='csv(zone:sort=1,name:sort=2,networkInterfaces[0].networkIP)' 
   > list.csv



You can then open the CSV file in a viewer such as Google Sheets:



This is just a taste of what --format supports. You can also expose data in JSON and tabular format, and use projections to select, sort and filter your data. Read more in the Google Cloud SDK reference for the --format flag to learn some more neat tricks you can do with --format.



(Pro tip from Glenn, Engineer on the Cloud SDK)




#4. Using the gcloud tool with PowerShell




If you’re a user of PowerShell, then often it’s handy to work with PowerShell objects. Some tweaks to the --format command allow you to do this. For example, if you use this command to list all Compute Engine instances in Asia:



PS> (gcloud compute instances list --format=json | Out-String | 
ConvertFrom-Json) | ?{$_.zone -match 'asia'} | select name

name
----
asia-ops-1
asia-ops-2



You can use this command to restart all the instances in Asia:



PS> (gcloud compute instances list --format=json | Out-String |
ConvertFrom-Json) | ?{$_.zone -match ’asia’} | %{gcloud compute instances
reset $_.name --zone $_.zone}



(Pro tip from Valentin, Engineer on the Cloud SDK)




#5. Easily ssh to your Compute Engine instances




The gcloud CLI offers a number of ways to easily use secure shell to access any Linux-based Compute Engine instances you have. For example, you can run:



$ gcloud compute ssh my-instance-name



This command automatically connects you through SSH to any instance that can be accessed publicly. No more SSH keygen or looking up external IPs!



You can go a step further with:



$ gcloud compute config-ssh



This command creates an alias for each of your Compute Engine instances to your ~/.ssh/config file. These aliases are then available to system utilities that also use SSH, such as ssh, scp and sftp. Now you can type commands from your terminal such as:



$ ssh myvm.asia-east1-c.myproject



or



$ sftp myvm.asia-east1-c.myproject



Read more about the gcloud compute ssh command or the gcloud compute config-ssh command

in the documentation.



(Pro tip from Stephen, Engineer on the Cloud SDK)



- Posted by the Google Cloud SDK team
Labels: ,

Pay less, compute more!



Developers running cloud-based apps and services will find out whether it’s a happy new year or not once they take a look at their bill. In case you’ve been reading recent announcements and were wondering, rest assured: Google continues to be the price/performance leader in public cloud.
Pay less, compute more!



Developers running cloud-based apps and services will find out whether it’s a happy new year or not once they take a look at their bill. In case you’ve been reading recent announcements and were wondering, rest assured: Google continues to be the price/performance leader in public cloud.



As you can see, we’re anywhere from 15-41% less expensive than AWS for compute resources, after their reduction. We use automatic Sustained Usage Discounts and our new Custom Machine Types to ensure that we’re presenting exact spec-to-spec comparisons here, something AWS can’t match.





*You might have noticed that the price is a little closer for high-memory instance types: AWS provides a particularly high ratio of ram to CPU on these, so for our comparison instance we’re using one with 4 cores rather than 2… and it’s still over 15% less expensive.



While price cuts sound appealing on the surface, when you unpack the specifics of Amazon’s pricing model, it can be an unpleasant surprise. We often hear from customers who are locked into contracts and aren’t eligible for the new rates, or are stuck with instances that no longer fit their needs.



We designed Google Cloud Platform pricing to be as flexible and beneficial to our customers as possible. You’re not required to make a long-term commitment to a price, machine class or region ahead of time. Our combination of lower list prices, sustained use discounting, no prepaid lock-in, per-minute billing, Preemptible VM’s and Custom Machine Types offers a structural price advantage that’s unmatched in the industry. For a detailed look at our pricing versus Amazon’s see this post on understanding cloud pricing.



As you consider which cloud provider to build and host your apps with in 2016, check out our TCO Tool. Explore how different combinations of development and production instances, as well as environmental assumptions, change the total cost of a real-world application hosted in the cloud — and be sure to compare it to our competitors!



Happy New Year!



- Posted by Miles Ward, Global Head of Solutions, Google Cloud Platform
Labels: ,

Today we hear from Olivier Favorel, Senior Manager, Airline IT at Amadeus. Operating in 195 countries, Amadeus is a leading technology company dedicated exclusively to the global travel industry. When an increase in CPU consumption of just 100 microseconds can mean thousands of dollars of extra hosting, Amadeus turned to Google Cloud Platform to offer new alternatives to its airline customers. ...
Today we hear from Olivier Favorel, Senior Manager, Airline IT at Amadeus. Operating in 195 countries, Amadeus is a leading technology company dedicated exclusively to the global travel industry. When an increase in CPU consumption of just 100 microseconds can mean thousands of dollars of extra hosting, Amadeus turned to Google Cloud Platform to offer new alternatives to its airline customers.



At Amadeus, we develop the technology that will shape the future of travel. To understand the business needs of our customers and partners, we’re highly focused on the trends impacting airlines. One main trend is the exponential growth of consumers browsing and shopping for airline products across digital channels.



Airline “look-to-book” ratios, or the average number of search requests before a flight reservation is actually made, were previously as low as 10:1. Today, these can easily run to 1000:1. Moreover, demand is never constant, thus managing demand fluctuations requires the ability to anticipate strong traffic peaks and make necessary capacity arrangements — a challenging task for airlines. In order to cope with the pressure of ever increasing online shopping transactions, shopping engines have developed cache-based solutions. However, cache-based systems have certain limitations, as they don’t accurately reflect an airline’s sophisticated revenue management policies.



Large network carriers are investing in advanced revenue management solutions to capture maximum traveler value and generate revenue. Maximizing revenue requires real-time capability to process every shopping request and make the right “flight availability” (availability of seats in a particular fare class) offer at an optimal price. Furthermore, it’s crucial for airlines to display consistent offers across various shopping platforms to capture every sales opportunity. Cache-based systems conflict with real-time revenue optimization, thus hindering airlines’ merchandising and personalization capabilities to make the right product offer to the right customer at the right time for the right price.



Given the challenge to maintain accurate and consistent airline offers across all distribution channels, how can we ensure high performance in dynamic content distribution for massive volumes?



With the help of Google Cloud Platform, Amadeus has developed a unique cloud-based solution, Amadeus Airline Cloud Availability. The solution offloads the processing of shopping transactions outside the airline reservation system, where the booking and payment are finally performed. This solution can be deployed in any public or private cloud, bringing airline offers closer to the shopping platform serving online travel agencies, meta searches or global distribution systems, while taking full advantage of more efficient solutions.




Figure 1: Amadeus Airline Cloud Availability architecture













This solution helps airlines efficiently manage the huge increase in search and shopping traffic.



We conducted a pilot of Amadeus Airline Cloud Availability in Google Cloud Platform from February to July 2015, together with Lufthansa. The objective of the pilot was two-fold:




  • Demonstrate the scalability and performance of flight availability requests using Google Compute Engine. Amadeus is currently handling requests for 4M+ flights per second in its private data center in Munich, for more than 140 airline carriers. This traffic increases by 50% every year.

  • Contain infrastructure cost of flight availability traffic.




The flight availability requests are handled by a farm of C++ backends accessing data through a Couchbase cluster, a distributed NoSQL store that hosts the airline flight and fare details. CPU consumption is a critical indicator for these kinds of large scale applications; an increase in CPU consumption of 100 microseconds per transaction translates into several thousands of dollars in extra hosting costs over a one-year period.



The initial deployment of our solution in Compute Engine was seamless thanks to the intuitive console and vast set of pre-installed Linux images (CentOS 7.1 in our case). First flight availability backend instances were ready to accept traffic only two hours after our initial connection.






The 1,500 cores challenge




Amadeus and Google engineering teams worked hand-in-hand to get the most out of a pre-allocated capacity of 1,500 cores spread over 3 regions (Central US, Western Europe and East Asia), each region being fed by airline data thanks to Couchbase Cross-Datacenter (XDCR) replication protocol.



Our mission was to increase the volume of flight availability requests processed per dollar. Several actions were undertaken:




  • Reducing the CPU path-length per transaction thanks to several C++ low level optimizations, and usage of Google’s tcmalloc memory allocator.

  • Increasing the IO throughput towards Couchbase data store to keep our application cores busy. We were quite impressed by the stability and very low latency of the internal Compute Engine network (stable sub-millisecond round-trip to Couchbase cluster nodes).

  • Enabling NOOP scheduler on VMs hosting our Couchbase cluster (optimal IO scheduling pattern to increase throughput to SSD drives).

  • Adjusting the VMs size (CPU/Memory ratio) to ensure that our servers were running constantly between 85-90% CPU usage (n1-highcpu-16 for application servers and n1-highmem-4 for Couchbase cluster nodes).









Figure 2: GCP Console and Performance Reports


The results




Pilot objectives were achieved much faster than initially planned, thanks to the flexibility of GCP and the reactivity of Google support teams.



The overall throughput of flight availability requests processed by 1,500 cores was doubled after only three months of joint effort.






Going further





We’re now engaging in the second phase of the pilot, aiming at dynamically adjusting the hardware capacity to the fluctuating shopping demand, further tuning the size of our VMs and leveraging the benefits of Compute Engine Preemptible VMs (“low-cost VMs” as we like to call them):




  • Dynamic capacity adjustment is being implemented thanks to Kubernetes (Google’s container orchestration and cluster management solution) that’s being rolled out in the pilot framework to dynamically spawn or shut down application VMs in line with flight availability traffic fluctuation. Kubernetes is shipped by our PaaS partner, Red Hat, as part of their OpenShift offer (we’re building our internal application platform, Amadeus Cloud Services, on top of these strategic products, to ensure our independence to the underlying IaaS provider). Per-minute billing of instances further optimizes the hosting costs.

  • Preemptible VMs, released in May 2015, run at a much lower price than standard VMs (70% off) but might be terminated, or preempted, by Compute Engine if it requires access to those resources for other tasks. Our plan is to oversize the number of computation VMs by 10% and use exclusively preemptible instance types, assuming that a fraction of those VMs will be terminated on a daily basis but still keeping our overall processing power at the required level to handle the flight availability traffic. Significant cost savings are anticipated with this new feature as well.

  • Custom machine types, released in November 2015, are being setup to replace our standard instance types (n1-highcpu-16 and n1-highmem-4). Custom VMs will be sized with only the required amount of cores and minimal memory requirement (per GB). The objective is to avoid any waste of CPU/memory.







Return on experience





Our journey on GCP was very exciting and impressed us for the following reasons:




  • Performance: Network latency, throughput and stability have astonishing performance. Also, the on-going migration of VMs to next-generation Intel architecture (Haswell) in many regions will bring even more CPU gains to flight availability request processing.

  • Stability: We faced very few VM outages over the 6-month pilot duration. The maintenance notification process is working great and the live VM migration is really transparent.

  • Monitoring: The Stackdriver framework is awesome to report both system metrics (CPU, Memory, IOs) and user-defined KPIs (like the rate of airline flights processed per second). Coupled with an efficient alerting system and the “Cloud Console” mobile app, we rapidly ended up with a production-grade monitoring solution.

  • Pace of innovation: During the six month duration of the pilot, three major announcements were made that helped our project: introduction of preemptible VMs, rollout of custom machine types and most importantly a 15% price drop in May 2015.







Summary




The pilot in the Google Cloud Platform changed our approach to performance optimization, from a pure CPU cost angle to an infrastructure driven approach (the efficiency is what matters in the end). GCP proved to be a very efficient sandbox environment for internal benchmarking, and we have no doubt that it will become a natural hosting solution for more Amadeus applications in the future.
Labels: ,