Google Cloud Platform Blog
Announcing the Google Cloud Platform Podcast
Wednesday, February 24, 2016
The
Google Cloud Platform
team is constantly updating and releasing new products, and sometimes there isn’t enough time in the day to sit down in front of a screen and read through everything.
We’re fixing that with the
Google Cloud Platform Podcast
—
where you can listen to all the new and exciting things that are happening on Google Cloud during your commute, at the gym, mowing the lawn, cooking dinner, or whenever you feel the desire to learn something new about our platform.
Google Cloud Platform Developer Advocates (left to right) Mark Mandel and Francesc Campoy
This weekly production is hosted by myself,
Mark Mandel
and my partner in crime
Francesc Campoy
. We are two members of the Google Cloud Platform
Developer Advocacy team
.
The show includes a weekly news roundup and community contributed questions, deep dives into interesting technical topics, such as
Big Data
,
Kubernetes
and
HTTP/2
, interviews with Google Cloud product managers
—
most recently
Ram Ramanathan
for the Cloud Vision API and
Chris Sells
for Developer Experience
—
and
conversations with customers such as Shine Technologies
to see how they're using Cloud Platform in the wild.
You can subscribe to the podcast to listen to all the episodes already published as well as those to come via
RSS
and
iTunes
.
We’ve got big plans for more upcoming episodes, so stay tuned!
-
Posted by Mark Mandel, Developer Advocate and Francesc Campoy Flores, Developer Advocate
Spotify chooses Google Cloud Platform to power data infrastructure
Tuesday, February 23, 2016
It’s not every day you move a 75 million+ user company from a home-grown infrastructure to the cloud. But if you use Spotify, more and more of your musical experience will be delivered by
Google Cloud Platform
over the coming weeks and months
—
we’re partnering on an ambitious project to move Spotify’s backend into GCP.
Spotify aims to make music special for everyone. Today, the company hosts more than
2 billion playlists
and gives consumers access to more than 30 million songs. Users can search for music across any device by artist, album, genre, playlist or record label, while features like Discover Weekly suggest personalized playlists for millions of people around the world.
While Spotify had engineers running its core infrastructure and buying or leasing data-center space, PC hardware and networking gear to provide a seamless experience for users — time and again it asked whether the tradeoff of resources that could otherwise focus on innovative features and software, was worth it.
Recently Spotify decided it didn’t want to be in the data center business, and chose Cloud Platform over the public cloud competition after careful review and testing. The company split their migration to Cloud Platform into two streams: a services track and a data track. Spotify runs their products on a multitude of tiny microservices, several of which are now being moved from on premise data centers into Google’s cloud using our Cloud Storage,
Compute Engine
and other products.
With Compute Engine, teams can rely on consistent performance from ultra high IOPS SSD and local SSD storage capabilities. And with autoscaling, they can build resilient and cost-efficient applications that use just the right amount of resources necessary at any given time. For storage, Spotify is now implementing
Google Cloud Datastore
and
Google Cloud Bigtable
. This rich fabric of storage services lets engineers work on complex back end logic, instead of focusing on how to store the data and maintain databases. Spotify is also deploying Google’s Cloud Networking services, such as
Direct Peering
,
Cloud VPN
and
Cloud Router
, to transfer petabytes of data. This results in a fast, reliable and secure experience for users around the globe.
On the data side of things, the company is adopting an entirely new technology stack. This includes moving from Hadoop, MapReduce, Hive and a series of home-grown dashboarding tools, to adopting the latest in data processing tools, including
Google Cloud Pub/Sub
,
Google Cloud Dataflow
,
Google BigQuery
, and
Google Cloud Dataproc
.
With BigQuery and Cloud Dataproc, data teams can run complex queries and get answers in a minute or two, rather than hours. This lets Spotify perform more frequent in-depth, interactive analysis, guiding product development, feature testing and more intelligent user-facing features. To gather and forward all events to its ecosystem, Spotify is using Cloud Pub/Sub, Google’s global service for messaging and streaming data. This gives teams the ability to process hundreds of thousands of messages per second, in a reliable no-ops manner. And to power its ETL workloads, Spotify is deploying Cloud Dataflow, Google’s data processing service. This lets the company rely on a single cloud-based managed service for both batch and stream processing.
What makes us most excited to work with Spotify is their company-wide focus on forward-looking user experiences. Now that they’ve begun using Google Cloud Platform, we can’t wait to see what Spotify builds next.
Join us for the
GCP NEXT 2016
opening keynote, where we’ll feature a
talk
from Nicholas Harteau, VP of Engineering and Infrastructure at Spotify. You can also attend Spotify-led technical
sessions
where you can learn more about how they’re deploying Google Cloud BigQuery and Dataflow.
- Posted by Guillaume Leygues, Lead Sales Engineer, Google Cloud Platform
Google seeks new disks for data centers
Tuesday, February 23, 2016
Today, during my keynote at the
2016 USENIX conference on File and Storage Technologies
(FAST 2016
), I’ll be talking about our goal to work with industry and academia to develop new lines of disks that are a better fit for data centers supporting cloud-based storage services. We're also releasing a
white paper
on the evolution of disk drives that we hope will help continue the decades of remarkable innovation achieved by the industry to date.
But why now? It's a fun but apocryphal story that the width of Roman chariots drove the spacing of modern train tracks. However, it is true that the modern disk drive owes its dimensions to the 3½” floppy disk used in PCs. It's very unlikely that's the optimal design, and now that we're firmly in the era of cloud-based storage, it's time to reevaluate broadly the design of modern disk drives.
The rise of cloud-based storage means that most (spinning) hard disks will be deployed primarily as part of large storage services housed in data centers. Such services are already the fastest growing market for disks and will be the majority market in the near future. For example, for YouTube alone, users upload over 400 hours of video every minute, which at one gigabyte per hour requires more than one petabyte (1M GB) of new storage
every day
or about 100x the Library of Congress. As shown in the graph, this continues to grow exponentially, with a 10x increase every five years.
At the heart of the paper is the idea that we need to optimize the
collection
of disks, rather than a single disk in a server. This shift has a range of interesting consequences including the counter-intuitive goal of having disks that are actually a little more likely to lose data, as we already have to have that data somewhere else anyway. It’s not that we want the disk to lose data, but rather that we can better focus the cost and effort spent trying to avoid data loss for other gains such as capacity or system performance.
We explore physical changes, such as taller drives and grouping of disks, as well as a range of shorter-term firmware-only changes. Our goals include higher capacity and more I/O operations per second, in addition to a better overall total cost of ownership. We hope this is the beginning of both a new chapter for disks and a broad and healthy discussion, including vendors, academia and other customers, about what “data center” disks should be in the era of cloud.
- Posted by Eric Brewer, VP Infrastructure, Google
Google Cloud Dataproc managed Spark and Hadoop service now GA
Monday, February 22, 2016
Today
Google Cloud Dataproc
, our managed
Apache Hadoop
and
Apache Spark
service, says goodbye to its beta label and is now generally available.
When analyzing data, your attention should be focused on insights, not your tools. Often, popular tools to process data, such as Apache Hadoop and Apache Spark, require a careful balancing act between cost, complexity, scale, and utilization. Unfortunately, this means you focus less on what is important
—
your data
—
and more on what should require little or no attention
—
the cluster processing it.
We created our managed Spark and Hadoop cloud service, Google Cloud Dataproc, to rectify the balance, so that using these powerful data tools is as easy as 1-2-3.
Since Cloud Dataproc
entered beta last year
, customers have taken advantage of its speed, scalability, and simplicity. We’ve seen them create clusters from three to thousands of virtual CPUs, using our
Developers Console
and
Google Cloud SDK
, without wasting time waiting for their cluster to be ready.
With integrations to
Google BigQuery
,
Google Cloud Bigtable
, and
Google Cloud Storage
, which provide reliable storage independent from Dataproc clusters, customers have created clusters only when they need them, saving time and money, without losing data. Cloud Dataproc can also be used in conjunction with
Google Cloud Dataflow
for real-time batch and stream processing.
While in beta, Cloud Dataproc added several important features including
property tuning
, VM metadata and tagging, and
cluster versioning
. In general availability, just like in beta, new versions of Cloud Dataproc, with new features, functionalities and software components, will be frequently released. One example is support for
custom machine types
, available today.
Cloud Dataproc tips the scale of running Spark and Hadoop in your favor by lowering cost and complexity while increasing scalability and productivity
Cloud Dataproc minimizes two common and major distractions in data processing
—
cost
and
complexity
by providing:
Low-cost.
We believe two things
—
using Spark and Hadoop should not break the bank and that you should pay for what you actually use. As a result, Cloud Dataproc is priced at only 1 cent per virtual CPU in your cluster per hour, on top of the other Cloud Platform resources you use. Moreover, with per-minute billing and a low 10-minute minimum, you pay for what you actually use, not a rounded (up) approximation.
Speed
. With Cloud Dataproc, clusters do not take 10, 15, or more minutes to start or stop. On average, Cloud Dataproc start and stop operations take 90 seconds or less. This can be a 2-10x improvement over other on-premises and IaaS solutions. As a result, you spend less time waiting on clusters and more time hands-on with data.
Management
. Cloud Dataproc clusters don't require specialized administrators or software products. Cloud Dataproc clusters are built on proven Cloud Platform services, such as
Google Compute Engine
,
Google Coud Networking
, and
Google Cloud Logging
to increase availability while eliminating the need for complicated hands-on cluster administration. Moreover, Cloud Dataproc supports
cluster versioning
, giving you access to modern, tested, and stable versions of Spark and Hadoop.
Cloud Dataproc makes two often problematic needs in data processing easy
—
scale
and
productivity
by being:
Easy
. You can create, monitor, and delete Cloud Dataproc clusters and jobs directly through
Google Developers Console
and Cloud SDK. For more advanced use cases, you can use the
Cloud Dataproc REST API
with a programming language, such as
Python
, to
programmatically interact
with Cloud Dataproc without hassle.
Modern
. Cloud Dataproc is frequently updated with new
image versions
to support new software releases from the Spark and Hadoop ecosystem. This provides access to the latest stable releases while also ensuring backward compatibility. For general availability we're releasing image version 1.0.0 with support for Hadoop 2.7.2, Spark 1.6.0, Hive 1.2.1, and Pig 0.15.0. Support for other components, such as
Apache Zeppelin (incubating)
are provided in our
GitHub repository
for initialization actions.
Integrated
. Cloud Dataproc has built-in integrations with other Cloud Platform services, such as BigQuery, Cloud Storage, Cloud Bigtable, and
Google Cloud Logging
so you have more than just a Spark or Hadoop cluster — you have a complete data platform. You can also use Cloud Dataproc
initialization actions
to extend the functionality of your clusters.
Our growing
partner ecosystem
offers certified support from several third-party tools and service partners. We're excited to collaborate with technology partners including
Arimo
,
Attunity
,
Looker
,
WANdisco
, and
Zoomdata
to make working in Cloud Dataproc even easier. Service providers like
Moser
,
Pythian
, and
Tectonic
are on standby to provide expert support during your Cloud Dataproc implementations. Reach out to any of our partners if you need help getting up and running.
To learn more about Cloud Dataproc, visit the
Cloud Dataproc site
, follow our
getting started guide
, take a look at a code example of how you can
predict keno outcomes with Cloud Dataproc
, or submit your questions and feedback on
Stack Overflow
.
-
Posted by James Malone, Product Manager
Sports Authority handles 2,000 transactions per second with Google Cloud Platform
Friday, February 19, 2016
Athletic gear, much like all apparel categories, is quickly shifting to an online sales business.
Sports Authority
, seeing the benefits that cloud could offer around agility and speed, turned to
Google Cloud Platform
to help it respond to its customers faster.
In 2014, Sports Authority’s technical team was asked to build a solution that would expose all in-store product inventory to its ecommerce site,
sportsauthority.com
, allowing customers to see local store availability of products as they were shopping online. That’s nearly half a million products to choose from in over 460 stores across the U.S. and Puerto Rico.
This use case posed a major challenge for the company. Its in-store inventory data was “locked” deep inside a mainframe. Exposing millions of products to thousands of customers, 24 hours a day, seven days a week would not be possible using this system.
The requirements for a new solution included finding the customer’s location, searching the 90 million record inventory system and returning product availability in just the handful of stores nearest in location to that particular customer. On top of that, the API would need to serve at least 50 customers per second, while returning results in less than 200 milliseconds.
Choosing the right cloud provider
At the time this project began, Sports Authority had already been a
Google Apps for Work
(Gmail, Google Sites, Docs) customer since 2011. However, it had never built any custom applications on
Google Cloud Platform
.
After a period of due diligence checking out competing cloud provider options, Sports Authority decided that
Google App Engine
and
Google Cloud Datastore
had the right combination of attributes — elastic scaling, resiliency and simplicity of deployment — to support this new solution.
Through the combined efforts of a dedicated project team, business partners and three or four talented developers, it was able to build a comprehensive solution on Cloud Platform in about five months. It consisted of multiple modules: 1) batch processes, using Informatica to push millions of product changes from its IBM mainframe to
Google Cloud Storage
each night, 2) load processes — python code running on App Engine, which spawn task queue jobs to load Cloud Datastore, and 3) a series of SOAP and REST APIs to expose the search functionality to its ecommerce website.
Sports Authority used tools including
SOAPUI
and
LOADUI
to simulate thousands of virtual users to measure the scalability of SOAP and REST APIs. It found that as the number of transactions grew past 2,000 per second, App Engine and Cloud Datastore continued to scale seamlessly, easily meeting its target response times.
The company implemented the inventory locator solution just in time for the 2014 holiday season. It performed admirably during that peak selling period and continues to do so today.
This screenshot shows what customers see when they shop for products on the website — a list of local stores, showing the availability of any given product in each store
When a customer finds a product she's interested in buying, the website requests inventory availability from Sports Authority’s cloud API, which provides a list of stores and product availability to the customer, as exhibited in the running shoe example above.
In-store kiosk
As Sports Authority became comfortable building solutions on Cloud Platform, it opened its eyes to other possibilities for creating new solutions to better serve its customers.
For example, it recently developed an in-store kiosk, which allows customers to search for products that may not be available in that particular store. It also lets them enroll in the loyalty program and purchase gift cards. This kiosk is implemented on a
Google Chromebox
, connected to a web application running on App Engine.
This image shows the in-store kiosk that customers use to locate products available in other stores.
Internal store portal
Additionally, it built a store portal and task management system, which facilitates communication between the corporate office and its stores. This helps the store team members plan and execute their work more efficiently, allowing them to serve customers better when needs arise. This solution utilizes App Engine, Cloud Datastore and
Google Custom Search
, and was built with the help of a local Google partner,
Tempus Nova
.
This screenshot shows the internal store portal that employees use to monitor daily tasks.
Learning how to build software in any new environment such as Cloud Platform takes time, dedication and a willingness to learn. Once up to speed, the productivity and power of Google Cloud Platform allowed the Sports Authority team to work like a software company and build quickly while wielding great power.
-
Posted by Jon Byrum, Product Marketing Manager, Google Cloud Platform
Google Cloud Vision API enters Beta, open to all to try!
Thursday, February 18, 2016
Today, we're announcing the beta release of
Google Cloud Vision API
. Now anyone can submit their images to the Cloud Vision API to understand the contents of those images
—
from detecting everyday objects (for example, “sports car,” “sushi,” or “eagle”) to reading text within the image or identifying product logos.
With the beta release of Cloud Vision API, you can access the API with location of images stored in
Google Cloud Storage
, along with existing support of embedding an image as part of the API request. We’re also announcing
pricing
for Cloud Vision API and added additional capabilities to identify the dominant color of an image. For example, you can now apply Label Detection on an image for as little as $2 per 1,000 images or Optical Character Recognition (OCR) for $0.60 for 1,000 images. Pricing will be effective, starting March 1st.
Cloud Vision API supports a broad set of scenarios from:
Insights from your images
: Powered by the same technologies behind
Google Photos
, Cloud Vision API detects broad sets of objects in your images
—
from flowers to popular landmarks
Inappropriate content detection
: Powered by
Google SafeSearch
, Cloud Vision API moderates content from your crowd sourced images by detecting different types of inappropriate content.
Image sentiment analysis
: Cloud Vision API can analyze emotional attributes of people in your images, like joy, sorrow and anger, along with detecting popular product logos.
Text extraction
: Optical Character Recognition (OCR) enables you to detect text within your images, along with automatic language identification across a broad set of languages.
Since we announced the
limited preview
of Google Cloud Vision API in early December, thousands of companies have used the API, generating millions of requests for image annotations. We're grateful for your feedback and comments and have been amazed by the breadth of applications using Cloud Vision API.
PhotoFy
, a social photo editing and branding app, moderates over 150,000 photos a day created by a wide audience. Before the Cloud Vision API was available, CTO Chris Keenan said that protecting these branded photos from abuse was almost impossible. With the Cloud Vision API, PhotoFy can flag potentially violent and adult content on user created photos in line with their abuse policies
Marsal Gavaldà, Director of Engineering for machine intelligence over at
Yik Yak
, a location-based social network, ran over a million images through the Cloud Vision API. The company was impressed with the accuracy of its feature detectors and content analyzers and the precision and recall of the text extraction in multiple languages. The number of objects that can be identified with the Cloud Vision API is an order of magnitude greater than comparable services from other cloud providers.
During the beta timeframe, each user will have a quota of 20 million images/month. As such, Cloud Vision API is not intended for real-time mission critical applications. You can access the
documentation
, with samples and tutorials showing usage of the API in Python and Java
1
, along with mobile app
samples
for Android and iOS.
Google Cloud Vision API
is our first step on the journey to enable applications to see, hear and make information in the world more useful. We welcome customers to join us on the journey and start using the API today. You can reach us with questions or feedback
here
.
We couldn't resist showing you our favorite robot again.
- Posted by Ram Ramanathan, Product Manager, Google Cloud Platform
1
Java is the registered trademark of Oracle and/or its affiliates.
Google and Red Hat announce cloud-based scalable file servers
Thursday, February 18, 2016
Google Cloud Platform
brings you lots of ways to store, access and archive your data, including
Google Cloud Storage
,
DataStore
and
BigQuery
. In some cases, there's a need to be able to access a POSIX compatible shared file system across a fleet of your cloud instances. To support these use cases with a robust scale out and scale up solution, Google Cloud Platform and Red Hat are proud to announce the availability of Red Hat Gluster Storage on Google Compute Engine.
Red Hat Gluster Storage offers a highly available and fault tolerant shared file system that can scale vertically and horizontally. Red Hat Gluster Storage makes use of compute instances with disks attached in order to provide a distributed, scale-out file system. Users of Red Hat Gluster Storage on GCE can take advantage of the performance, scalability and flexibility of our
Persistent Disks
.
Disks used for your Red Hat Gluster Storage installation can be chosen based on various performance and cost tradeoffs. For example, you can choose to use standard Persistent Disks for data that does not require high I/O throughput or use the more performant SSD Persistent Disks for your IOPS hungry workloads, such as media rendering. Each node in your cluster can leverage
disks of up to 64TB in size
and
with up to 15,000 IOPS
.
In order to protect mission critical data, Red Hat Gluster Storage enables users to synchronously replicate their files across multiple zones in the same region while at the same time asynchronously replicating them to a separate region for disaster recovery. In the example architecture below, we're using us-east1-b as our primary zone with a hot standby in us-east1-c:
For more information on getting started with Red Hat Gluster Storage on GCE click
here
.
- Posted by Vic Iglesias, Cloud Solutions Architect
Don't Miss Next '17
Use promo code NEXT1720 to save $300 off general admission
REGISTER NOW
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Labels
Announcements
56
Big Data & Machine Learning
91
Compute
156
Containers & Kubernetes
36
CRE
7
Customers
90
Developer Tools & Insights
80
Events
34
Infrastructure
24
Management Tools
39
Networking
18
Open Source
105
Partners
63
Pricing
24
Security & Identity
23
Solutions
16
Stackdriver
19
Storage & Databases
111
Weekly Roundups
16
Archive
2017
Feb
Jan
2016
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Feed
Subscribe by email
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow