Posted by Michael Shields, Technical Lead, Time Team



As if 2016 wasn’t long enough, this year, a leap second will cause the last day of December to be one second longer than normal. But don’t worry, we’ve built support for the leap second into the time servers that regulate all Google services.Read More

Posted by Michael Shields, Technical Lead, Time Team



As if 2016 wasn’t long enough, this year, a leap second will cause the last day of December to be one second longer than normal. But don’t worry, we’ve built support for the leap second into the time servers that regulate all Google services.

Even better, our Network Time Protocol (NTP) servers are now publicly available to anyone who needs to keep local clocks in sync with VM instances running on Google Compute Engine, to match the time used by Google APIs, or for those who just need a reliable time service. As you would expect, our public NTP service is backed by Google’s load balancers and atomic clocks in data centers around the world.

Here’s how we plan to handle the leap second and keep things running smoothly here at Google. It’s based on what we learned during the leap seconds in 2008, 2012 and 2015.



Leap seconds compensate for small and unpredictable changes in the Earth's rotation, as determined by the International Earth Rotation and Reference Systems Service (IERS). The IERS typically announces them six months in advance but the need for leap seconds is very irregular. This year, the leap second will happen at 23:59:60 UTC on December 31, or 3:59:60 pm PST.



No commonly used operating system is able to handle a minute with 61 seconds, and trying to special-case the leap second has caused many problems in the past. Instead of adding a single extra second to the end of the day, we'll run the clocks 0.0014% slower across the ten hours before and ten hours after the leap second, and “smear” the extra second across these twenty hours. For timekeeping purposes, December 31 will seem like any other day.



All Google services, including all APIs, will be synchronized on smeared time, as described above. You’ll also get smeared time for virtual machines on Compute Engine if you follow our recommended settings. You can use non-Google NTP servers if you don’t want your instances to use the leap smear, but don’t mix smearing and non-smearing time servers.



If you need any assistance, please visit our Getting Help page.



Happy New Year, and let the good times roll.

Posted by Quoc Truong, Software Engineer



If you're managing Google Cloud Platform (GCP) resources from the command line on Windows, chances are you’re using our Cloud Tools for PowerShell. Thanks to PowerShell’s powerful scripting environment, including its ability to pipeline objects, you can efficiently author complex scripts to automate and manipulate your GCP resources.Read More

Posted by Quoc Truong, Software Engineer



If you're managing Google Cloud Platform (GCP) resources from the command line on Windows, chances are you’re using our Cloud Tools for PowerShell. Thanks to PowerShell’s powerful scripting environment, including its ability to pipeline objects, you can efficiently author complex scripts to automate and manipulate your GCP resources.



However, PowerShell has historically only been available on Windows. So even though you had an uber-sophisticated PowerShell script to set up and monitor multiple Google Compute Engines and Google Cloud SQL instances, if you wanted to run it on Linux, you would have had to rewrite it in bash!



Fortunately, Microsoft recently released an alpha version of PowerShell that works on both OS X and Ubuntu, and we built a .NET Core version of our Tools on top of it. Thanks to that, you don’t have to rewrite your Google Cloud PowerShell scripts anymore just to make them work on Mac or Linux machines.



To preview the bits, you'll have to:

1. Install Google Cloud SDK and initialize it.


2. Install PowerShell.


3. Download and unzip Cross-Platform Cloud Tools for PowerShell bits.

Now, from your Linux or OS X terminal, check out the following commands:

# Fire up PowerShell.
powershell


# Import the Cloud Tools for PowerShell module on OS X.
PS > Import-Module ~/Downloads/osx.10.11-x64/Google.PowerShell.dll


# List all of the images in a GCS bucket.
Get-GcsObject -Bucket "quoct-photos" | Select Name, Size | Format-Table

If running GCP PowerShell cmdlets on Linux interests you, be sure to check out the post on how to run an ASP.NET Core app on Linux using Docker and Kubernetes. Because one thing is for certain — Google Cloud Platform is rapidly becoming a great place to run — and manage — Linux as well as Windows apps.



Happy scripting!

Posted by Grace Mollison, Solutions Architect





Using Google Cloud Deployment Manager is a great way to manage and automate your cloud environment. By creating a set of declarative templates, Deployment Manager lets you consistently deploy, update and delete resources like ...Read More

Posted by Grace Mollison, Solutions Architect





Using Google Cloud Deployment Manager is a great way to manage and automate your cloud environment. By creating a set of declarative templates, Deployment Manager lets you consistently deploy, update and delete resources like Google Compute Engine, Google Container Engine, Google BigQuery, Google Cloud Storage and Google Cloud SQL. As one of the less well known features of Google Cloud Platform (GCP), let’s talk about how to use Deployment Manager.



Deployment Manager uses three types of files:

• a Configuration file written in YAML


• Template files, based on Jinja 2.7.3 or Python 2.7


• Schema files, which define a set of rules that a configuration file must meet if it wants to use a particular template

Using templates is the recommended method of using Deployment Manager, and requires a configuration file as a minimum. The configuration file defines the resources you wish to deploy and their configuration properties such as zone and machine type.



Deployment manager supports a wide array of GCP resources. Here's a complete list of supported resources and associated properties, which you can also retrieve with this gcloud command:

$ gcloud deployment-manager types list

Deployment Manager is often used alongside a version control system into which you can check in the definition of your infrastructure. This approach is commonly referred to as "infrastructure as code" It’s also possible to pass properties to Deployment Manager directly using gcloud command, but that's not a very scalable approach.

Anatomy of a Deployment Manager configuration

To understand how things fit together, let’s look at the set of files that are used to create a simple network with two subnets and a single deployed instance.

The configuration consists of three files:

• net-config.yaml - configuration file


• network.jinja - template file


• instance.jinja - template file

You can use template files as logical units that break down the configuration into smaller and reusable parts. Templates can then be composed into a larger deployment. In this example, network configuration and instance deployment have been broken out into their own templates.

Understanding templates

Templates provide the following benefits and functionality:

• Composability, making it easier to manage, maintain and reuse the definitions of the cloud resources declared in the templates. In some cases you may not want to recreate the end-to-end configuration as defined in the configuration file. In that case, you can just reuse one or more templates to help ensure consistency in the way in which you create resources.


• Templates written in your choice of Python or Jinja2. Jinja2 is a simpler but less powerful templating language than Python. It uses the same syntax as YAML but also allows the use of conditionals and loops. Python templates are more powerful and allow you to programmatically generate the contents of your templates.


• Template variables – an easy way to reuse templates by allowing you to declare the value to be passed to the template in the configuration file. This means that you can change a specific value for each configuration without having to update the template. For example, you may wish to deploy your test instances in a different zone to your production instances. In that case, simply declare within the template a variable that inherits the zone value from the master configuration file.


• Environment variables, which also help you reuse templates across different projects and deployments. Examples of an environment variable include things like the Project ID or deployment name, rather than resources you want to deploy.

Here’s how to understand the distinction between the template and environment variables. Imagine you have two projects where you wish to deploy identical instances, but to different zones. In this case, name your instances based on the Project ID and Deployment name found from the environment variables, and set the zone through a template variable.

A Sample Deployment Manager configuration

For this example, we’ve decided to keep things simple and use templates written in Jinja2.

The network file

This file creates a network and its subnets whose name and range are passed through from the variable declaration in net-config.yaml, the calling configuration file.

The “for” subnet loop repeats until it has read all the values in the subnets properties. The config file below declares two subnets with the following values:

Subnet name	IP range
web	10.177.0.0/17
data	10.178.128.0/17

The deployment will be deployed into the us-central1 region. You can easily change this by changing the value of the “region” property in the configuration file without having to modify the network template itself.

The instance file

The instance file, in this case "instance.jinja," defines the template for an instance whose machine type, zone and subnet are defined in the top level configuration file’s property values.

The configuration file

This file, called net-config.yaml, is the main configuration file that marshals the templates that we defined above to create a network with a single VM.

To include templates as part of your configuration, use the imports property in the configuration file that calls the template (going forward, the master configuration file). In our example the master configuration file is called net-config.yaml and imports two templates at lines 15 - 17:

The resource network is defined by the imported template network.jinja.

The resource web-instance is defined by the imported template instance.jinja.



Template variables are declared that are passed to each template. In our example, lines 19 - 27 define the network values that are passed through to the network.ninja template.

Lines 28 to 33 define the instance values.

To deploy a configuration, pass the configuration file to Deployment Manager via the gcloud command or the API. Using gcloud command, type the following command:

$ gcloud deployment-manager deployments create net --configuration net-config.yaml

You'll see a message indicating that the deployment has been successful

You can see the deployment from Cloud Console.

Note that the instance is named after the deployment specified in instance.jinja.

The value for the variable “deployment” was passed in via the gcloud command “create net” where “net” is the name of the deployment



You can explore the configuration by looking at the network and Compute Engine menus:

You can delete a deployment from Cloud Console by clicking the delete button or with the following gcloud command:

$ gcloud deployment-manager deployments delete net

You'll be prompted for verification that you want to proceed.

Next steps

Once you understand the basics of Deployment Manager, there’s a lot more you can do. You can take the example code snippets that we walked through here and build more complicated scenarios, for example, implementing a VPN that connects back to your on premises environment. There are also many Deployment Manager example configurations on Github.



Then, go ahead and start thinking about advanced Deployment Manager features such as template modules and schemas. And be sure to let us know how it goes.

Posted by Hanan Youssef, Product Manager, Google Cloud Platform



Google Cloud Platform’s focus on infrastructure excellence allows us to provide great price-performance and access to the latest hardware innovations. Working closely with hardware vendors, we help guide new advancements in data center technology and the speed at which ...Read More

Posted by Hanan Youssef, Product Manager, Google Cloud Platform



Google Cloud Platform’s focus on infrastructure excellence allows us to provide great price-performance and access to the latest hardware innovations. Working closely with hardware vendors, we help guide new advancements in data center technology and the speed at which Google Cloud Platform (GCP) customers can use them.



Yesterday, Google Cloud announced a strategic alliance with Intel that builds on our long-standing relationship developing data center technology. Today, we're excited to announce that Google Compute Engine will support Intel’s latest Custom Cloud solution based on the next-generation Xeon Processor (codenamed Skylake) in early 2017.



The upcoming Xeon processor is an excellent choice for graphics rendering, simulations and any CPU intensive workload. At launch, Compute Engine customers will be able to utilize the processor’s AVX-512 extensions to optimize their enterprise-class and HPC workloads. We'll also add support for additional Skylake extensions over time.



You'll be able to use the new processor with Compute Engine’s standard, highmem, highcpu and custom machine types. We also plan to continue to introduce bigger and better VM instance types that offer more vCPUs and RAM for compute- and memory-intensive workloads.



If you’d like to be notified of upcoming Skylake beta testing programs, please fill out this form.

Posted by Todd Prives, Product Manager





Google Cloud continues its push into media and entertainment since completing the acquisition of online video platform Anvato and a collaboration with Autodesk at NAB ...Read More

Posted by Todd Prives, Product Manager





Google Cloud continues its push into media and entertainment since completing the acquisition of online video platform Anvato and a collaboration with Autodesk at NAB earlier this year. Media use cases like multi-screen video transcoding, livestreaming to global audiences and 3D rendering power demand from customers in every industry where video and creative content is used, from advertising to education and beyond.



Today we're announcing the release of an expansion of ZYNC Render to support two new host platforms: SideFX Houdini and Maxon Cinema 4D. Both integrations will open the cost and productivity benefits for animators to leverage the power of Google Cloud Platform (GCP) to bring their projects to life, bringing massive scalability and compute access to the animation industry.

ZYNC is a turnkey rendering solution for boutique to mid-size studios that allows artists to focus on content creation. It does this through plugins to popular modeling and animation that software artists already use, offering one-stop access to powerful compute, storage and software licenses on GCP.



Users of Houdini, a leading package for complex 3D effects, can now utilize up to 32,000 cores on GCP for their rendering projects. To purchase a traditional render farm of this size is often well beyond the resources of most small to mid-size studios. Instead, artists can render on-demand, paying on a per-minute basis with the full economic benefits of GCP.



The Maxon Cinema 4D integration, a popular package for creating motion graphics, marks the first time Maxon has enabled its product through a cloud rendering service. As artists create more complicated scenes for commercial work and feature films, on-demand, scalable cloud rendering has emerged as a critical tool for studios trying to meet tight deadlines.



The media team at Google Cloud is excited to bring cloud-based rendering to the animation industry. We continue to be driven by empowering creative professionals with world-class infrastructure, giving even the smallest studio equal resources to rival the largest production houses.

Posted by Doron Meirfeld, Head of DevOps at JFrog and Mansirman Singh, Solutions Engineer at JFrog



Editor's Note: Today we hear from our partner JFrog, which recently refactored its Artifactory SaaS offering onto Google Cloud Platform (GCP), making it possible for joint JFrog and GCP customers to co-locate their development and production environments. Read on for more details about how JFrog architected and optimized the service to run on GCP. ...Read More

Posted by Doron Meirfeld, Head of DevOps at JFrog and Mansirman Singh, Solutions Engineer at JFrog



Editor's Note: Today we hear from our partner JFrog, which recently refactored its Artifactory SaaS offering onto Google Cloud Platform (GCP), making it possible for joint JFrog and GCP customers to co-locate their development and production environments. Read on for more details about how JFrog architected and optimized the service to run on GCP.



JFrog Artifactory SaaS is a universal artifact repository hosted in the cloud. Our customers use it as the one-stop-binary-shop for their Docker registry, and repositories for Maven, npm, PyPi, Nuget and more. It offers freedom of choice in several dimensions supporting all major package formats, CI servers and build tools. We recognized the need to add Google Cloud Platform (GCP) to offer more choice, and support organizations already using GCP so they could co-locate all their cloud services. We set up Artifactory SaaS hosted on GCP using Google Cloud Storage as its massively scalable object store.

JFrog Artifactory SaaS deployment architecture

While setting up an enterprise-grade cloud service can get complicated, GCP offers an extensive range of services to make it easier. The architecture we developed is constructed as four layers based on GCP services, mainly Google Compute Engine and Cloud Storage. The four layers are:

• Network Load Balancers, balance requests coming from the outside world into the front-end web Nginx stacks


• Web servers based on stacked Nginx servers are responsible for internal application load balancing and proxying requests


• Artifactory application servers


• Data and Metadata Management using Google Cloud SQL and Cloud Storage. Cloud SQL manages the Artifactory internal database fundamental to the product’s checksum-based storage, and Cloud Storage is the application’s massively scalable object store, where all the binary artifacts are actually hosted.

Onboarding and Provisioning

To onboard new customers quickly and easily, we developed a scripted provisioning service. As soon as you register for a free trial, the JFrog Store triggers the provisioning mechanism to automatically set up and configure all layers in the service architecture, helping you get up and running virtually instantly. This structured and efficient onboarding mechanism made it easier for us to adapt our internal provisioning solution by simply swapping out API calls and replacing them with relevant GCP API calls such as those in the Compute Engine and Cloud Storage services.



This structured and efficient onboarding process made it easier for us to adopt GCP as a new additional underlying platform for the service.

The four tenets of an enterprise-grade service

In designing our setup, we wanted to ensure it would meet the requirements of any enterprise:



Scalability

Every layer in the architecture can be quickly scaled by our provisioning mechanism to meet any load requirements. Whether adding Compute Engines to the network or web service layers, or adding Artifactory instances, any element can be scaled up on demand without incurring any downtime to the system, and storage scales automatically as needed.



High availability

Our years of experience with our Artifactory SaaS taught us that one of the most critical issues that enterprises are concerned about when considering a cloud service is availability. Since an artifact repository is a vital resource for a development organization, any downtime can quickly translate to dollars lost. Our architecture takes availability to the extreme with redundancy implemented at every level, resulting in a setup with no single point-of-failure. The whole system is redundantly deployed on multiple distinct zones, so even if there is a general failure in one zone, the system will failover to another zone and continue to provide service to customers. As demand increases for JFrog Artifactory SaaS on GCP, we have complete flexibility to quickly set up redundant installations in additional zones as needed and on-demand.



Disaster recovery capabilities

To support disaster recovery, we utilize the built-in capability for multi-region storage on Cloud Storage and replicate Artifactory’s Cloud SQL database that contains both application and customer data. This allows us to failover to a recovery region in case of an outage in the main active region with no noticeable impact to users.



Security

The system maintains a clear separation at every level for both customers on a dedicated Artifactory SaaS installation and those on multi-tenanted installations. Dedicated customer installations use separate virtual devices at each layer of the architecture for maximum security. For customers in a multi-tenant environment, our provisioning mechanism automatically creates clearly separated folders, Artifactory servers and web server configurations as well as a dedicated filestore in Cloud Storage buckets.

Lessons learned

While JFrog Artifactory SaaS was already a mature and robust service, we learned a few lessons while migrating the service to GCP.



Tweak to peak

While a service may run on top of different infrastructures, each cloud provider has its nuances. To optimize JFrog Artifactory SaaS on GCP, we focused on tweaking resource allocation such as number of threads and more in each layer to get optimal performance.



Offload to buckets

Working with Cloud Storage buckets made it much easier for us to manage the service than when using traditional storage solutions such as NFS. Things like monitoring folder sizes and storage capacity was a non-issue since these functions are provided by Cloud Storage buckets. On the whole, our service got “lighter.”



Setting up JFrog Artifactory SaaS on top of GCP was a great decision for us. Our past experience with Artifactory SaaS helped us in migrating and modifying our cloud service to a new platform while maintaining the same high quality of service. As a leader in binary artifact management, we take "universality" as a guiding principle, and believe that hosting our service on GCP is a great way to serve our customers. We'll continue to grow with Google Cloud as more services are added to enhance scalability, availability and reliability.



Open source developer? Get a free ride

JFrog’s tools are made by developers for developers. We are part of the OSS community and strive to provide it with the best vehicle to ride in. Together with the Google Cloud team, JFrog is happy to sponsor repositories free of charge (including Artifactory SaaS on top of a JFrog sponsored GCP infrastructure) for open source projects. Browse to the registration form and feel free to submit your request.



Read more about JFrog Artifactory on GCP. And may the frog be with you!

Posted by Chris Sells, Product Manager



In tomorrow’s enterprise, developers and IT professionals work in an environment characterized by technological choice: open, cross-platform development where the best tools are easily within reach. As enterprises adopt public and hybrid cloud at an accelerated pace, Google Cloud is committed to fostering an open platform.Read More

Posted by Chris Sells, Product Manager



In tomorrow’s enterprise, developers and IT professionals work in an environment characterized by technological choice: open, cross-platform development where the best tools are easily within reach. As enterprises adopt public and hybrid cloud at an accelerated pace, Google Cloud is committed to fostering an open platform.



In this spirit of enabling developers and their organizations, Google is pleased to be joining the Technical Steering Group of the .NET Foundation. .NET is a key component in the modern enterprise, and the Google Cloud Platform (GCP) team has worked hard to ensure that .NET has first-class support on Google’s infrastructure, including excellent infrastructure for Windows. For years, Google has offered .NET libraries for more than 200 of its cloud services. More recently, we’ve built native GCP support for Visual Studio and PowerShell.



Google is already an active contributor to .NET, including heavy involvement in the ECMA specification for C#. Joining the Technical Steering Group for the .NET Foundation expands our participation. We’re excited to work with the industry to contribute to .NET as an excellent open platform for developers in the enterprise. Don’t hesitate to contact us if we can help you bring your .NET workloads to GCP!

Posted by Rob Craft, Group Lead for Google Cloud Machine Learning



Google Cloud Machine Learning is one of our fastest growing products areas. Since we first announced our machine learning offerings earlier this year, we’ve released a steady stream of new APIs, tools and services to help you harness the power of machine learning. We’ve seen machine learning transform users ...Read More

Posted by Rob Craft, Group Lead for Google Cloud Machine Learning



Google Cloud Machine Learning is one of our fastest growing products areas. Since we first announced our machine learning offerings earlier this year, we’ve released a steady stream of new APIs, tools and services to help you harness the power of machine learning. We’ve seen machine learning transform users’ experiences, accelerate business operations by solving problems that have existed for decades and delight us with novel applications.



The key to creating a culture of innovation is having the right team, technology and strategy in place. To further these efforts, we’re excited to announce the creation of a new Google Cloud Machine Learning group that will be focused exclusively on delivering cloud-based machine learning solutions to all businesses. Fei-Fei Li and Jia Li, two world-renowned researchers in the subject of machine intelligence, will lead this new group.



Building a centralized team within Google Cloud will accelerate our ability to deliver machine learning products and services to enterprise customers in every industry. Today also marks an exciting next step in Google Cloud’s product commitment to make machine learning more accessible for all businesses. We’re excited to introduce:

• A brand new Cloud Machine Learning API to help people find careers


• New hardware options to accelerate your machine learning workloads


• Improved efficiencies and expanded features for our Cloud Translation, Cloud Vision and Cloud Natural Language APIs

Introducing Google Cloud Jobs API

Machine learning presents new opportunities to solve some pretty difficult business problems. Since so much of what every business achieves depends on great employees, how can we help there? What if we could use machine learning to change the nature of finding jobs and hiring people? We think we can.



Hiring is one of the hardest things organizations do. Part of the difficulty comes from a lack of industry standards to define and describe occupations and how they align to specific skills. Over the past year, Google has developed a new machine-learning model that has the potential to greatly improve the recruitment efforts of any company. We call this the Google Cloud Jobs API. It provides businesses with Google-strength capabilities to find, match and recommend relevant jobs to candidates.



In order to provide the most relevant recommendations to job seekers, Cloud Jobs API uses machine learning to understand how job titles and skills relate to one another and what job content, location, and seniority are the closest match to a jobseeker’s preferences. You can learn more about how it works here.



The API is intended for job boards, career sites and applicant tracking systems. Early adopters of Cloud Jobs API are Jibe, Dice and CareerBuilder.

“Large enterprises have come to expect that integrating new solutions takes month or years, and these long implementation cycles are a major roadblock in delivering innovation. Jibe was able to seamlessly deploy the Google Jobs API as a turnkey machine learning solution for one of our customer's career sites in a matter of 3 weeks, and we expect that implementation time to shrink for future customers.” - Joe Essenfeld, Founder and CEO at Jibe

Dice, a career website that serves opportunities for technology and engineering professionals, is a launch tester of the API to help job candidates browse over 80,000 tech job listings. Tech jobs tend to be complex and skill specific. For example, if a tech professional enters "front-end engineer" in a job search without using typical Boolean standards, search results will also return UI engineer, UI developer, web developer, and UX engineer. Complicated, right? By using the API, Dice will be able to better understand a candidate’s background and preferences and match the tech pro to the right roles.

CareerBuilder, using a prototype that they created with Cloud Jobs API in just 48 hours, found improved, more accurate results when compared to its existing search algorithm. In one test, CareerBuilder chose a top 100 term, “part time,” and compared results using the Google Cloud Jobs API versus their existing solution. Jobs API returned a richer set of results by applying an expanded set of synonyms including “PT.” Another test showcased how Jobs API can refine search results. CareerBuilder has one of the largest repositories of healthcare industry jobs. CareerBuilder tested the terms “CNA psych” (Certified Nurses Assistant) against a dataset and reduced the results returned — delivering only CNA roles in a psychiatric setting — to notably increase accuracy for the job seeker. Based on these results, CareerBuilder is making plans to leverage the API for its customers in the near future.



Cloud Jobs API is now available in a limited alpha. To learn more, visit the Cloud Jobs API page.

Welcoming GPUs for Google Cloud Platform

Machine learning greatly benefits from fast and reliable hardware, and as hardware advances so do the capabilities of machine learning. This is exactly why Google continues to harness hardware innovations that can help to accelerate machine learning applications.



Beginning in 2017, Google Cloud will offer more hardware choices for businesses that want to use Google Cloud Platform (GCP) for their most complex workloads, including machine learning. For Google Compute Engine and Google Cloud Machine Learning, businesses will be able to use GPUs (Graphics Processing Units) that are highly-specialized processors capable of handling the complexities of machine learning applications. Making GPUs available in Google Cloud means that you can focus on solving challenging computational problems while accessing GPU machines from anywhere and only paying for what you need.



In other words, you'll be able to strap your ML-powered applications to a rocket engine, resulting in faster and more affordable machine learning models. To learn more, visit our GPU page.

Making Cloud Vision API affordable for everyone

Google has been leveraging the latest hardware and tuned algorithms to significantly improve the performance of our Cloud Machine Learning services. Cloud Vision API now takes advantage of Google’s custom TPUs, our custom ASIC built for machine learning, to improve performance and efficiency. These improvements have enabled us to reduce prices for Cloud Vision API by ~80%. By offering the API at a more affordable price-point, more organizations than ever will be able to take advantage of Cloud Vision API to power new capabilities.



Along with the price reductions, we have made significant improvements to our image recognition capabilities over the last six months. For example: the logo detection feature can identify millions of logos and label detection can identify an expanded number of entities, such as landmarks and objects in images.



Since we first introduced Cloud Vision API, we’ve been very happy with the positive feedback and creativity customers have in using it to power their experiences. Since the beta release, businesses have analyzed well over a billion images. Image analysis, the core capability of Vision API, is fundamentally changing how businesses operate and interact with their end-users. We have thousands of customers using the product to do amazing things. For example, the e-discovery firm Platinum IDS uses Cloud Vision API to power content relevancy for millions of paper and digital files and deliver its new e-Discovery app, and Disney has leveraged Vision API as the basis of innovative marketing campaigns.

Now offering Cloud Translation API Premium

Most recently, Google announced the launch of our Google Neural Machine Translation system (GNMT) that uses state-of-the-art training techniques and runs on TPUs to achieve some of the largest improvements for machine translation in the past decade. Now, Google Cloud is offering these capabilities to all partners, developers and businesses with a Premium edition of Cloud Translation API (formerly Google Translate API).



This new edition provides:

• Highest-quality model that reduces translation errors by more than 55%-85% on several major language pairs


• Support for up to eight languages (English to Chinese, French, German, Japanese, Korean, Portuguese, Spanish, Turkish) and 16 language pairs. We'll support more languages in the near future.

The Premium edition is tailored for users who need precise, long-form translation services. Examples include livestream translations, high volume of emails and detailed articles and documents. The Standard edition continues to offer translation in over 100 languages and price-performance that's ideal for short, real-time conversational text.

To make Cloud Translation API more affordable, we also decreased the price of the standard edition for higher usage volumes. Please visit our pricing page for more information.

Graduating Cloud Natural Language API to general availability

Cloud Natural Language API, our text analysis machine learning service, is now generally available for all businesses. Based on valuable feedback shared by beta testers such as Evernote, a productivity service used by over 200 million people to store billions of notes and attachments, we're releasing new features:

• Expanded entity recognition to increase the accuracy at which the API identifies the names of things, such as people, companies, or locations in the text.


• Granular sentiment analysis with expanded language support to provide sentiment analysis at the sentence level and not just within a document or record.


• Improved syntax analysis with additional morphologies such as number, gender, person and tense, to improve coreference resolution required of advanced NLP tasks.

Click here to learn more about the technical details and to see the Cloud Natural Language API in action.



Our team is hard at work to enable new machine learning scenarios in the upcoming months. This year alone, we introduced brand new APIs and a fully-managed platform that are now available for all businesses to use. And as users explore our existing machine learning ecosystem, Google continues to invest in research and models that will bring new scenarios to life. We’re committed to quickly delivering new machine learning solutions for businesses in 2017 and beyond. Stay tuned for what’s next.

Posted by John Barrus, Product Manager, Google Cloud Platform





CPU-based machines in the cloud are terrific for general purpose computing, but certain tasks such as rendering or large-scale simulations are much faster on specialized processors. Graphics Processing Units (GPUs) contain hundreds of times as many computational cores as CPUs and are great at accelerating risk analysis, studying molecular binding or optimizing the shape of a turbine blade. If your CPU-based instance feels like a Formula One race car but you’re in need of a rocket, you’re going to love our new cloud GPUs.Read More

Posted by John Barrus, Product Manager, Google Cloud Platform





CPU-based machines in the cloud are terrific for general purpose computing, but certain tasks such as rendering or large-scale simulations are much faster on specialized processors. Graphics Processing Units (GPUs) contain hundreds of times as many computational cores as CPUs and are great at accelerating risk analysis, studying molecular binding or optimizing the shape of a turbine blade. If your CPU-based instance feels like a Formula One race car but you’re in need of a rocket, you’re going to love our new cloud GPUs.



Early in 2017, Google Cloud Platform will offer GPUs worldwide for Google Compute Engine and Google Cloud Machine Learning users. Complex medical analysis, financial calculations, seismic/subsurface exploration, machine learning, video rendering, transcoding and scientific simulations are just some of the applications that can benefit from the highly parallel compute power of GPUs. GPUs in Google Cloud give you the freedom to focus on solving challenging computational problems while accessing GPU-equipped machines from anywhere. Whether you need GPUs for a few hours or several weeks, we’ve got you covered.



Google Cloud will offer AMD FirePro S9300 x2 that supports powerful, GPU-based remote workstations. We'll also offer NVIDIA® Tesla® P100 and K80 GPUs for deep learning, AI and HPC applications that require powerful computation and analysis. GPUs are offered in passthrough mode to provide bare metal performance. Up to 8 GPU dies can be attached per VM instance including custom machine types.

Google Cloud GPUs give you the flexibility to mix and match infrastructure. You’ll be able to attach up to 8 GPU dies to any non-shared-core machine, whether you’re using an n1-highmem-8 instance with 3 TB of super-fast Local SSD or a custom 28 vCPU virtual machine with 180 GB of RAM. Like our VMs, GPUs will be priced per minute and GPU instances can be up and running within minutes from Google Cloud Console or from the gcloud command line. Whether you need one or dozens of instances, you only pay for what you use.



During an early access program, customers have been running machine learning training, seismic analysis, simulations and visualization on GPU instances. Startup MapD gets excellent results with a GPU-accelerated database.

"These new instances of GPUs in the Google Cloud offer extraordinary performance advantages over comparable CPU-based systems and underscore the inflection point we are seeing in computing today. Using standard analytical queries on the 1.2 billion row NYC taxi dataset, we found that a single Google n1-highmem-32 instance with 8 attached K80 dies is on average 85 times faster than Impala running on a cluster of 6 nodes each with 32 vCPUs. Further, the innovative SSD storage configuration via NVME further reduced cold load times by a factor of five. This performance offers tremendous flexibility for enterprises interested in millisecond speed at over billions of rows."



- Todd Mostak, MapD Founder and CEO

The Foundry, a visual effects software provider for the entertainment industry has been experimenting with workstations in the cloud.

"At The Foundry, we're really excited about VFX in the cloud, and with the arrival of GPUs on Google Cloud Platform, we'll have access to the cutting edge of visualisation technology, available on-demand and charged by the minute. The potential ramifications for our industry are enormous.."



- Simon Pickles, Lead Engineer, Pipeline-in-the-Cloud

Tell us about your GPU computing requirements and sign up to be notified about GPU-related announcements using this survey. Additional information is available on our webpage.

Posted by Paul Rossman, Google Cloud Developer Relations



Understanding the nature of the universe isn't a game for the resource-constrained. Today, we probe the very structure of matter using multi-billion dollar experimental machinery, hundreds of thousands of computing cores and exabytes of data storage. Together, the European Center for Nuclear Research (CERN) and partners such as Fermilab built the Large Hadron Collider (LHC), the world's largest particle collider, to recreate and observe the first moments of the universe.Read More

Posted by Paul Rossman, Google Cloud Developer Relations



Understanding the nature of the universe isn't a game for the resource-constrained. Today, we probe the very structure of matter using multi-billion dollar experimental machinery, hundreds of thousands of computing cores and exabytes of data storage. Together, the European Center for Nuclear Research (CERN) and partners such as Fermilab built the Large Hadron Collider (LHC), the world's largest particle collider, to recreate and observe the first moments of the universe.



Today, we're excited to announce that Google Cloud Platform (GCP) is now a supported provider for HEPCloud, a project launched in June 2015 by Fermilab’s Scientific Computing Division to develop a virtual facility providing a common interface to local clusters, grids, high-performance computers and community and commercial clouds. Following the recommendations from a 2014 report by the Particle Physics Project Prioritization Panel to the national funding agencies, the HEPCloud project demonstrates the value of the elastic provisioning model using commercial clouds.



The need for compute resources by the high-energy physics (HEP) community is not constant. It follows cycles of peaks and valleys driven by experiment schedules and other constraints. However, the conventional method of building data centers is to provide all the capacity needed to meet peak loads, which can lead to overprovisioned resources. To help mitigate this, Grid federations such as the Open Science Grid offer opportunistic access to compute resources across a number of partner facilities. With the appetite for compute power expected to increase over 100-fold over the next decade, so too will the need to improve cost efficiency with an “elastic” model for dynamically provisioned resources.



With Virtual Machines (VMs) that boot within seconds and per-minute billing, Google Compute Engine lets HEPCloud pay for only the compute it uses. Because the simulations that Fermilab needs to perform are fully independent and parallelizable, this workload is appropriate for Preemptible Virtual Machines. Without the need for bidding, Preemptible VMs can be up to 80% cheaper compared to regular VMs. Combined with Custom Machine Types, Fermilab is able to double the computing power of the Compact Muon Solenoid (CMS) experiment by adding 160,000 virtual cores and 320 TB of memory in a single region, for about $1400 per hour.



At SC16 this week, Google and Fermilab will demonstrate how high-energy physics workflows can benefit from the elastic Google Cloud infrastructure. The demonstration involves computations that simulate billions of particles from the CMS detector (see fig. 1) at the LHC. Using Fermilab’s HEPCloud facility, the goal is to burst CMS workflows to Compute Engine instances for one week.

Fig. 1: The CMS detector before closure (credit: 2008 CERN, photo: Maximilien Brice, Michael Hoch, Joseph Gobin)

The demonstration also leverages HTCondor, a specialized workload management system for compute-intensive jobs, to manage resource provisioning and job scheduling. HTCondor manages VMs natively using the Compute Engine API. In conjunction with the HEPCloud Decision Engine component, it enables the use of the remote resources at scale at an affordable rate (fig. 2). With half a petabyte of input data in Google Cloud Storage, each task reads from the bucket via gcsfuse, performs its computation on Preemptible VMs, then transports the resulting output back to Fermilab through the US Department of Energy Office of Science's Energy Sciences Network (ESnet), a high-performance, unclassified network built to support scientific research.

Fig. 2: The flow of data from the CMS detector to scientific results through the CMS, HEPCloud and Google Cloud layers. Image of CMS event display © CERN by McCauley, Thomas; Taylor, Lucas; the CMS Collaboration is licensed under CC BY-SA 4.0.

The demonstration shows that HTCondor, HEPCloud and GCP all work together to enable real HEP science to be conducted in a cost-effective burst mode at a scale that effectively doubles the current capability. The Fermilab project plans to transition the HEPCloud facility into production use by the HEP community in 2018.

“Every year we have to plan to provision computing resources for our High-Energy Physics experiments based on their overall computing needs for performing their science. Unfortunately, the computing utilization patterns of these experiments typically exhibit peaks and valleys during the year, which makes cost-effective provisioning difficult. To achieve this cost effectiveness we need our computing facility to be able to add and remove resources to track the demand of the experiments as a function of time. Our collaboration with commercial clouds is an important component of our strategy for achieving this elasticity of resources, as we aim to demonstrate with Google Cloud for the CMS experiment via the HEPCloud facility at SC16.” 

- Panagiotis Spentzouris, Head of the Scientific Computing Division at Fermilab

If you're at SC16, stop by the Google booth and speak with experts on scalable high performance computing or spin up your own HTCondor cluster on Google Cloud Platform for your workloads.

Posted by Mete Atamel, Developer Advocate



Google Cloud Shell, my favorite development tool for Google Cloud Platform, just got more awesome with two new features.



First, we recently ...Read More

Posted by Mete Atamel, Developer Advocate



Google Cloud Shell, my favorite development tool for Google Cloud Platform, just got more awesome with two new features.



First, we recently integrated Eclipse Orion, an online code editor, with Cloud Shell. If you're not a Vim or Emacs fan, Orion is a welcome addition to Cloud Shell. It enables you to edit code right inside the browser with basic syntax highlighting and minimal effort.



Second, we added .NET Core command line interface tools to Cloud Shell. This is a new cross-platform toolchain for developing .NET Core applications and it can be used to create, run and publish .NET Core apps from the command line.



With these two new additions, you can easily create an ASP.NET Core web app in Cloud Shell, edit it using Orion and run it on a Linux instance (yes, ASP.NET Core apps run on Linux!) right inside your browser. Let's take a look at the steps involved.

Create an ASP.NET web app in Cloud Shell

From Google Cloud Console, click on the Cloud Shell icon on the top right section of the toolbar to activate Cloud Shell:

And, create a folder for our HelloWorld app:

$ mkdir HelloWorldAspNetCore
$ cd HelloWorldAspNetCore

Then, create a skeleton ASP.NET Core web app using the dotnet command:

$ dotnet new -t web
Created new C# project in /home/atameldev/HelloWorldAspNetCore.

Use Orion to change the default port

By default, ASP.NET Core web apps run on port 5000. Let's change that to a more conventional port for web apps, port 8080.

public static void Main(string[] args)
{
    var host = new WebHostBuilder()
                .UseKestrel()
                .UseContentRoot(Directory.GetCurrentDirectory())
                .UseIISIntegration()
                .UseStartup()
                .UseUrls("http://*:8080")
                .Build();


    host.Run();
}

Open Orion by clicking Files and then Launch Code Editor at the top of Cloud Shell. Once the code editor is open, find Program.cs in the folder:

Add the UseUrls line to bind to port 8080:

Run the ASP.NET web app

We're almost ready to run our app but we need to restore dependencies first. This will download all the NuGet dependencies for our app:

$ dotnet restore
…
log  : Restore completed in 16298ms.

Finally, run our app:

$ dotnet run
…
Now listening on: http://*:8080
Application started. Press Ctrl+C to shut down.

At this point, our app is running on Linux! Don’t believe it? You can convince yourself by visiting the web preview and Preview on port 8080:

You'll see the default ASP.NET Core webpage in a new tab.

Next steps

There! You’ve just created and launched an ASP.NET Core app from inside Cloud Shell, without once leaving your browser. But that’s not all you can do. You can take your newly created ASP.NET Core app, containerize it using Docker and deploy it to Google App Engine or you can let Kubernetes manage it all for you in Google Container Engine. It’s easy to see why Cloud Shell is my favorite GCP development tool.

Posted by Dave Rensin, Director of Customer Reliability Engineering, and Adrian Hilton, Software Engineer, Site Reliability Engineering



Editor’s note: Left unchecked, poor software architecture decisions are the most common cause of application downtime. Over the years, Google Site Reliability Engineering has learned to spot code that could lead to outages, and strives to identify it before it goes into production as part of its production readiness review. With the introduction of Customer Reliability Engineering, we’re taking the same best practices we’ve developed for internal systems, and extending them to customers building applications on Google Cloud Platform. This is the first post in a series written by CREs to highlight real-world problems  ...Read More

Posted by Dave Rensin, Director of Customer Reliability Engineering, and Adrian Hilton, Software Engineer, Site Reliability Engineering



Editor’s note: Left unchecked, poor software architecture decisions are the most common cause of application downtime. Over the years, Google Site Reliability Engineering has learned to spot code that could lead to outages, and strives to identify it before it goes into production as part of its production readiness review. With the introduction of Customer Reliability Engineering, we’re taking the same best practices we’ve developed for internal systems, and extending them to customers building applications on Google Cloud Platform. This is the first post in a series written by CREs to highlight real-world problems — and the steps we take to avoid them.





Distributed Denial of Service (DDoS) attacks aren’t anything new on the internet, but thanks to a recent high profile event, they’ve been making fresh headlines. We think it’s a convenient moment to remind our readers that the biggest threat to your application isn’t from some shadowy third party, but from your own code!



What follows is a discussion of one of the most common software architecture design fails — the self-inflicted DDoS — and three methods you can use to avoid it in your own application.

Even distributions that aren’t

There’s a famous saying (variously attributed to Mark Twain, Will Rogers, and others) that goes:

“It ain’t what we don’t know that hurts us so much as the things we know that just ain’t so.”

Software developers make all sorts of simplifying assumptions about user interactions, especially about system load. One of the more pernicious (and sometimes fatal) simplifications is “I have lots of users all over the world. For simplicity, I’m going to assume their load will be evenly distributed.”



To be sure, this often turns out to be close enough to true to be useful. The problem is that it’s a steady state or static assumption. It presupposes that things don’t vary much over time. That’s where things start to go off the rails.



Consider this very common pattern: Suppose you’ve written a mobile app that periodically fetches information from your backend. Because the information isn’t super time sensitive, you write the client to sync every 15 minutes. Of course, you don’t want a momentary hiccup in network coverage to force you to wait an extra 15 minutes for the information, so you also write your app to retry every 60 seconds in the event of an error.



Because you're an experienced and careful software developer, your system consistently maintains 99.9% availability. For most systems that’s perfectly acceptable performance but it also means in any given 30-day month your system can be unavailable for up to 43.2 minutes.



So. Let’s talk about what happens when that’s the case. What happens if your system is unavailable for just one minute?



When your backends come back online you get (a) the traffic you would normally expect for the current minute, plus (b) any traffic from the one-minute retry interval. In other words, you now have 2X your expected traffic. Worse still, your load is no longer evenly distributed because 2/15ths of your users are now locked together into the same sync schedule. Thus, in this state, for any given 15-minute period you'll experience normal load for 13 minutes, no load for one minute and 2X load for one minute.



Of course, service disruptions usually last longer than just one minute. If you experience a 15-minute error (still well within your 99.9% availability) then all of your load will be locked together until after your backends recover. You'll need to provision at least 15X of your normal capacity to keep from falling over. Retries will also often “stack” at your load balancers and your backends will respond more slowly to each request as their load increases. As a result, you might easily see 20X your normal traffic (or more) while your backends heal. In the worst case, the increased load might cause your servers to run out of memory or other resources and crash again.



Congratulations, you’ve been DDoS’d by your own app!



The great thing about known problems is that they usually have known solutions. Here are three things you can do to avoid this trap.

#1 Try exponential backoff

When you use a fixed retry interval (in this case, one minute) you pretty well guarantee that you'll stack retry requests at your load balancer and cause your backends to become overloaded once they come back up. One of the best ways around this is to use exponential backoff.



In its most common form, exponential backoff simply means that you double the retry interval up to a certain limit to lower the number of overall requests queued up for your backends. In our example, after the first one-minute retry fails, wait two minutes. If that fails, wait four minutes and keep doubling that interval until you get to whatever you’ve decided is a reasonable cap (since the normal sync interval is 15 minutes you might decide to cap the retry backoff at 16 minutes).



Of course, backing off of retries will help your overall load at recovery but won’t do much to keep your clients from retrying in sync. To solve that problem, you need jitter.

#2 Add a little jitter

Jitter is the random interval you add (or subtract) to the next retry interval to prevent clients from locking together during a prolonged outage. The usual pattern is to pick a random number between +/- a fixed percentage, say 30%, and add it to the next retry interval.



In our example, if the next backoff interval is supposed to be 4 minutes, then wait between +/- 30% of that interval. Thirty percent of 4 minutes is 1.2 minutes, so select a random value between 2.8 minutes and 5.2 minutes to wait.



Here at Google we’ve observed the impact of a lack of jitter in our own services. We once built a system where clients started off polling at random times but we later observed that they had a strong tendency to become more synchronized during short service outages or degraded operation.



Eventually we saw very uneven load across a poll interval — with most clients polling the service at the same time — resulting in peak load that was easily 3X the average. Here’s a graph from the postmortem from an outage in the aforementioned system. In this case the clients were polling at a fixed 5-minute interval, but over many months became synchronized:

Observe how the traffic (red) comes in periodic spikes, correlating with 2x the average backend latency (green) as the servers become overloaded. That was a sure sign that we needed to employ jitter. This monitoring view is also significantly under-counting the traffic peaks because of its sample interval. Once we added a random factor of +/- 1 minute (20%) to each retry the latency, traffic flattened out almost immediately, with the periodicity disappearing:

and the backends were no longer overloaded. Of course, we couldn’t do this immediately — we had to build and push a new code release to our clients with this new behavior, so we had to live with this overload for a while.



At this point, we should also point out that in the real world, usage is almost never evenly distributed — even when the users are. Nearly all systems of any scale experience peaks and troughs corresponding with the work and sleep habits of their users. Lots of people simply turn off their phones or computers when they go to sleep. That means that you'll see a spike in traffic as those devices come back online when people wake up.



For this reason it’s also a really good idea to add a little jitter (perhaps 10%) to regular sync intervals, in addition to your retries. This is especially important for first syncs after an application starts. This will help to smooth out daily cyclical traffic spikes and keep systems from becoming overloaded.

#3 Implement retry marking

A large fleet of backends doesn’t recover from an outage all at once. That means that as a system begins to come back online, its overall capacity ramps up slowly. You don’t want to jeopardize that recovery by trying to serve all of your waiting clients at once. Even if you implement both exponential backoff and jitter you still need to prioritize your requests as you heal.



An easy and effective technique to do this is to have your clients mark each attempt with a retry number. A value of zero means that the request is a regular sync. A value of one indicates the first retry and so on. With this in place, the backends can prioritize which requests to service and which to ignore as things get back to normal. For example, you might decide that higher retry numbers indicate users who are further out-of-sync and service them first. Another approach is to cap the overall retry load to a fixed percentage, say 10%, and service all the regular syncs and only 10% of the retries.



How you choose to handle retries is entirely up to your business needs. The important thing is that by marking them you have the ability to make intelligent decisions as a service recovers.



You can also monitor the health of your recovery by watching the retry number metrics. If you’re recovering from a six-minute outage, you might see that the oldest retries have a retry sequence number of 3. As you recover, you would expect to see the number of 3s drop sharply, followed by the 2s, and so on. If you don’t see that (or see the retry sequence numbers increase), you know you still have a problem. This would not be obvious by simply watching the overall number of retries.

Parting thoughts

Managing system load and gracefully recovering from errors is a deep topic. Stay tuned for upcoming posts about important subjects like cascading failures and load shedding. In the meantime, if you adopt the techniques in this article you can help keep your one minute network blip from turning into a much longer DDoS disaster.

Posted by Morgan McLean, Product Manager for Stackdriver Trace





Google Cloud Platform customers can now analyze changes to their applications’ latency profiles through Google Cloud Console and on their Android devices ...Read More

Posted by Morgan McLean, Product Manager for Stackdriver Trace





Google Cloud Platform customers can now analyze changes to their applications’ latency profiles through Google Cloud Console and on their Android devices, with iOS support coming soon.

Using the latency reports feature, developers can:

• View the latency profiles of their application’s endpoints


• Compare the latency profile of their application between different times or versions


• Observe if a report is flagged as having a major or minor latency shift

This functionality, along with the full suite of Stackdriver Trace features on the web-based Cloud Console, is available for all projects hosted on Google App Engine and any projects on Google Compute Engine and Google Container Engine that use the Stackdriver Trace SDKs (currently available for Node.js and Java). The latency reports can be accessed through the Analysis Reports tab within the Stackdriver Trace section of Cloud Console, or from the Trace tab of the Cloud Console mobile app. Links to endpoint-specific reports are found in the analysis report column of the Trace List page and under the Reports heading on individual traces in the Cloud Console.



Here’s an example of what you’ll see in Cloud Console for a project that's capturing trace data:

(click to enlarge)

You’ll observe the following in the mobile app:

Latency reports are automatically generated for the endpoints with the highest traffic in each project; each of these reports compares each endpoint’s current latency profile to the prior week’s.



In the web-based console, selecting New Report allows you to create custom reports to observe the latency profile of a particular endpoint, or to compare the performance of an endpoint between different times and versions (see here for more details).



Some reports are flagged as having major or minor changes, which indicates that the latency distribution across percentiles is substantially different between the two versions or times being compared. These are often worth investigating, as they can represent changes in a service’s underlying performance characteristics.

(click to enlarge)

Each web-based report contains a graph of the endpoint’s latency distribution across percentiles. The auto analysis example above compares the latency profiles of a given endpoint over one week. As indicated by the graph and the “major change” text, the endpoint’s latency distribution has changed significantly over this time period.



The table at the bottom of the report shows that the application’s latency has increased in the 90th percentile and lower, while it has decreased in the higher percentile cases. This distinction is important: a simple comparison of the mean latencies between times A and B shows little change, but the report correctly identifies that the service is now considerably faster for the worst 10% of requests.



Here’s an example of a similar report in the mobile app, with a similar percentile comparison grid:

This feature will be available for the iOS Cloud Console app shortly.



For more information on how to create and understand latency reports, see this page. Let us know what you think about this feature, either by commenting here or through the send feedback button in Cloud Console.