Google Cloud Platform Blog
Mapping your knowledge to Google Cloud Platform
Monday, December 21, 2015
Google Cloud Platform
(GCP) is growing all the time and we love introducing
new products
and features and getting them into your hands. This rapid pace of innovation does mean that there is always something new to learn about and this can take up a lot of your time. We also know that GCP isn’t the only cloud platform you’re using or have used, and it’s important that we help you leverage that experience to get up to speed fast.
Our goal is to make it easier for you to stay on top of the services we offer and help you map your existing expertise to GCP at the same time. To that end, we are happy to release a new
whitepaper
that we have created to help you apply your existing knowledge and expertise to GCP.
This document is the first part of an on-going series. We start with the basics on how to complete base level operations, followed by a deep dive into the virtual compute platforms and the underlying networks. In the coming months we’ll add more information on how to work with storage, data, containers, and much more.
We hope you find this useful in learning about GCP. Please tell us what you think and what else you would like us to add. And don't forget to use our
free trial
to try out the things you've learned!
-
Posted by Peter-Mark Verwoerd, Cloud Solutions Architect
Announcing NEXT 2016: Join us for what’s next for cloud
Friday, December 18, 2015
Last June, we kicked off Google Cloud Platform Next, and so many of you wanted to come that we had to move to a different location to accommodate everyone! Today, we’re excited to announce
GCP NEXT 2016
in San Francisco: the event created
specifically for you
to learn about Google Cloud Platform directly from those who built it. You’ll also hear from developers and organizations that use Google Cloud Platform to build and run their businesses.
At GCP NEXT 2016, you’ll see how cutting-edge features in Google Cloud Platform will help you build powerful, reliable and intelligent applications at any scale.
Set your calendars (or your
DeLorean
) for March 23 - 24, 2016 to join us at GCP NEXT, where you’ll:
Hear about the latest in Google Cloud Platform product developments
Watch in-depth product demos and exclusive talks from SVPs Diane Greene and Urs Hölzle.
Try your hand at different parts of the platform in code labs
Get hands-on experience with our platform with immersive tutorials, led by Google engineers and advocates.
Enjoy the NEXT Playground
Play with fun demos powered by Google Cloud Platform, and find out how they were made to see what’s possible with cloud. Chat and exchange ideas with other technologists in the hallway track.
Learn the fundamentals of Google Cloud Platform
Need to learn the basics of our platform or want to refresh your skills? Join us before GCP NEXT for a full-day, instructor-led Bootcamp on March 22.
Join us for some fun at the after-party
Network with our community and attend our after-party, NEXT After Dark.
We’ll also host tracks that dive a little deeper into specific areas of cloud, so you can learn more about topics that interest you most. This year, we’re opening up a
call for speakers
, and we hope to see you submit a session proposal! Share your project or experiences using our platform in one of our featured tracks:
Data and Analytics
Data is key to intelligent applications and decision making. Learn how Google Cloud Platform can help you build more intelligent applications and make better, more timely decisions.
Infrastructure and Operations
Learn how Google’s infrastructure — including our networks, storage, security, data center operations and DevOps tools — gives you scale, security and reliability. Sessions in this track will cover popular tooling, common DevOps patterns and how to manage at any scale.
App and Services Development
Want to understand how different components of Google Cloud Platform can work together in a variety of configurations? In this track, we'll discuss topics such as app architecture, development, deployment and continuous integration.
Solutions Showcase
Listen to some of our customers talk about how they’re using Google Cloud Platform in production. From cloud-native startups to enterprises in the process of migrating to cloud, they'll tell you about their experiences powering everything from mobile applications to mission-critical deployments. Hear about practical solutions, patterns, and lessons that you can apply to your own applications.
We look forward to seeing you at
GCP NEXT 2016
. If you can’t make it in person, catch sessions via livestream. Registration opens today, and the call for speakers closes on January 15, 2016; make sure to get your proposal submitted in time!
cloud.google.com/Next2016
To keep up to date on GCP NEXT 2016, follow us on
Google+
,
Twitter
, and
LinkedIn
.
-
Posted by Julia Ferraioli, Developer Advocate, Google Cloud Platform
Meeting the challenge of financial data transformation
Thursday, December 17, 2015
Today’s guest post comes from Salvatore Sferrazza and Sebastian Just from FIS Global, an international provider of financial services and technology solutions. Salvatore and Sebastian tell us how Google Cloud Dataflow transforms fluctuating, large-scale financial services data so that it can be accurately captured and moved across systems.
Much software development in the capital markets (and enterprise systems in general) revolves around the transformation, enrichment and movement of data from one system to another. The unpredictable nature of financial market data volumes, often driven by volatility, exacerbates the pain of scaling and posting data when and where it’s needed for daily trade reconciliation, settlement and regulatory reporting. The implications of technology missteps within such crucial business processes range from missed business opportunities to undesired risk exposure to regulatory non-compliance. These activities must be relentlessly predictable, repeatable and measurable to yield maximum value to stakeholders.
While developers rely on the Extract, Transform and Load (ETL) activities that are so crucial to processing data, they now face limits in terms of the speed and efficiency of ETL as the amount of transactions grows faster than they can process it. As shortened settlement durations and the Consolidated Audit Trail (CAT) loom on the horizon, financial services institutions need simple, fast and powerful approaches to quickly scale and ultimately mitigate time-sensitive risks and operational costs.
Traditionally, developers have considered the activities around ETL data an unglamorous yet necessary dimension of building software products for encapsulating functions that are core to every tier of computing. So when data-driven enterprises are tasked with harvesting insights from massive data sets, it’s quite likely that ETL, in one form or another, is lurking nearby. But in today’s world, data can come from anywhere and in any format, creating a series of labor, time and intellectual challenges. While there may be hundreds of ways to solve the problem, few provide the efficiency and effectiveness so needed in our “big data” world — until recently.
The
Google Cloud Dataflow
service and its associated software development kit (SDK) provides a series of powerful tools for a myriad of data transformation duties. Designed to perform data processing tasks of any size in a managed services environment, Google Cloud Dataflow simplifies the mechanics of large-scale transformation and supports both batch and stream processing using the same programming model. In our latest
white paper
, we introduce some of the main concepts behind building and running applications that use Dataflow, then get “hands on” with a job to transform and ingest options market symbol data before storing the transformations within a Google BigQuery data set.
In short, Google Cloud Dataflow allows you to focus on data processing tasks and not cluster management. Rather than asking you to guess the right cluster size, Dataflow automatically scales up or down horizontally as much as needed for your exact processing requirements. This includes scaling all the way down to zero when there is no work, so you’re never paying for an idle cluster. Dataflow also alleviates the pain of writing ETL jobs by standardizing the process of implementing application requirements. As a result, you’ll be able to focus on the data transformations you need to make rather than on the processing mechanics themselves. This not only provides greater flexibility, lower latency and enhanced control of ETL jobs; it offers built-in cost management and ties together other useful Google Cloud services. Beyond common ETL, Dataflow pipelines may also include inline computation ranging from simple counting to highly complex, multi-step analysis. In our experience with the service so far, it can potentially remove much of the work from engineers within financial institutions and regulatory organizations, while providing elasticity to the entire process and ensuring accuracy, scale, performance and cost efficiency.
As market volatility and reporting requirements drive the need for accuracy, low latency and risk reduction, transforming and interpreting market data in a big data world is imperative to trading efficiency and accessibility. Every second counts. With a more cost-effective, real-time and scalable method of processing an ever-increasing volume of data, financial institutions will be able to address specific requirements and volumes at hand while keeping up with the demands of a rapidly evolving global financial system. We hope our experience, as captured in the
technical white paper
, will prove useful to others in their quest for the more effective way to process data.
Please see this paper’s GitHub page for the complete and buildable
project source code
.
-
Posted by Salvatore Sferrazza, Principal at FIS and Sebastian Just, Manager at FIS
Monitoring Container Engine with Google Cloud Monitoring
Thursday, December 17, 2015
You’ve decided to adopt a microservice architecture and containerize your application. Congrats! But how will you monitor it? To solve that problem, we've worked to make
Google Container Engine
and
Google Cloud Monitoring
fit together like peas in a pod.
When you launch your Container Engine cluster, you can enable Cloud Monitoring with one click. Check it out!
Information will be collected about the CPU usage, memory usage and disk usage for all of the containers in your cluster. This information is annotated and stored in Cloud Monitoring, where you can choose to either access it via the API or in the Cloud Monitoring UI. From Cloud Monitoring, you can easily examine not only the container level resource usage but also see this aggregated across pods and clusters.
If you head over to the
Cloud Monitoring dashboard
and click on the Infrastructure dropdown, you can see a new option for Container Engine.
If you have more than one cluster with monitoring enabled, you'll see a page listing the clusters in your project along with how many pods and instances are in them. However, if you only have one cluster, you'll be directed straight to details about it, as shown below.
This page gives you a view of your cluster. It lists all the pods running in your cluster, recent events from the cluster, as well as resource usage aggregated across the nodes in your cluster. In this case, you can see that this cluster has the system components in it (DNS, UI, logging and monitoring) as well as the frontend and redis pods from the
guestbook tutorial
in the Container Engine documentation.
From here, you can easily drill down to the details of individual pods and containers, where you'll see metadata about the pod and its containers, such as how many times they've been restarted, along with metrics about the pod's resource usage.
But this is just the first piece. Since Cloud Monitoring makes heavy use of tags (the equivalent of Container Engine's labels), you can create groups based on how you've labeled your containers or pods. For example, if you're running a web app in a replication controller, you may have all of your frontend web containers labeled with “role=frontend.” In Cloud Monitoring, you can now create a group “Frontend” that matches all resources with the tag role and the value frontend.
You can also make queries that aggregate across pods without needing to create a group, making it possible to visualize the performance of an entire replication controller or service on a single graph. You can do this by creating a new dashboard from the top-level menu option named Dashboards, and adding a chart. In the example below, you can see the aggregated memory usage of all the php-redis frontend pods in the cluster.
With these tools, you can create powerful alerting policies that trigger when the aggregate across the group or any container within the group violates a threshold, for example, using too much memory. You can also tag your group as a cluster so that Cloud Monitoring's cluster insights detection will show outliers across the set of containers when they're detected, potentially helping you to pinpoint cases where your load isn't evenly distributed or nodes don't have even workloads.
And since this is all based on tags, it will update automatically, even as your containers move across the nodes of your cluster, even if you're auto-scaling and adding and removing nodes over time.
We have a lot more work planned to continue to integrate Container Engine and Cloud Monitoring and make it easy to collect your application and service metrics as well as system metrics that you can use today.
Do you have ideas of what we should do to make things better? Let us know by sending feedback through the Cloud Monitoring console or directly at
monitoring-and-logs-feedback@google.com
. You can find more information on the available metrics
in our docs
.
-
Posted by Alex Robinson, Software Engineer, Google Container Engine and Jeremy Katz, Software Engineer, Google Cloud Monitoring
Cloud Audit Logs to help you with audit and compliance needs
Wednesday, December 16, 2015
Not having a full view of administrative actions in your Google Cloud Platform projects can make it challenging and slow going to troubleshoot when an important application breaks or stops working. It can also make it difficult to monitor access to sensitive data and resources managed by your project. That’s why we created Google Cloud Audit Logs, and today they’re available in beta for
App Engine
and
BigQuery
. Cloud Audit Logs help you with your audit and compliance needs by enabling you to track the actions of administrators in your Google Cloud Platform projects. They consist of two log streams:
Admin Activity
and
Data Access
.
Admin Activity
audit logs contain an entry for every administrative action or API call that modifies the configuration or metadata for the related application, service or resource, for example, adding a user to a project, deploying a new version in App Engine or creating a BigQuery dataset. You can inspect these actions across your projects on the
Activity page
in the Google Cloud Platform Console.
Data Access
audit logs contain an entry for every one of the following events:
API calls that read the configuration or metadata of an application, service or resource
API calls that create, modify or read user-provided data managed by a service (e.g. inserting data into a dataset or launching a query in BigQuery)
Currently, only BigQuery generates a Data Access log as it manages user-provided data, but ultimately all Cloud Platform services will provide a Data Access log.
There are many additional uses of Audit Logs beyond audit and compliance needs. In particular, the BigQuery team has put together a collection of
examples
that show how you can use Audit Logs to better understand your utilization and spending on BigQuery. We’ll be sharing more examples in future posts.
Accessing the Logs
Both of these logs are available in
Google Cloud Logging
, which means that you’ll be able to view the individual log entries in the
Logs Viewer
as well as take advantage of the many logs management capabilities available, including
exporting the logs
to Google Cloud Storage for long-term retention, streaming to BigQuery for real-time analysis and publishing to Google Cloud Pub/Sub to enable processing via Google Cloud Dataflow. The specific content and format of the logs can be found in the Cloud Logging
documentation
for Audit Logs.
Audit Logs are available to you at no additional charge. Applicable charges for using other Google Cloud Platform services (such as BigQuery and Cloud Storage) as well as streaming logs to BigQuery will still apply. As we find more ways to provide greater insight into administrative actions in GCP projects, we’d love to hear your feedback. Share it here:
gcp-audit-logging-feedback@google.com
.
Posted by Joe Corkery, Product Manager, Google Cloud Platform
Google Cloud Shell will be free through 2016!
Wednesday, December 16, 2015
In October,
we announced
the launch of
Google Cloud Shell
, a Google Cloud Platform feature that lets you manage your infrastructure and applications from the command line in any browser. At that time we committed that Cloud Shell beta would be free through 2015, and today we have
extended this to the end of 2016!
With the holiday season upon us, you might not always have access to the computer you use to manage your application daily. With Cloud Shell, it just takes one click in the console to get temporary, quick access to a VM hosted and managed by Google that includes the most common tools needed to manage GCP pre-installed. If you need to store something between sessions, you’ll have 5GB of storage space.
Cloud Shell in GCP Cloud Console
We’ve seen strong enthusiasm around these new capabilities from the community:
“Cloud shell, the new UI, and the depth of each service and it’s documentation puts @googlecloud on top for me. Quality over quantity”
-
@SageProgramming
“Cloud shell + container engine from @googlecloud make quick work of configuring @kubernetesio projects. Nothing to install but a browser!”
-
@nissyen
But you also told us that a free beta period through the end of 2015 was too short. With that in mind, we’re excited to extend the free beta period for another year, until the end of 2016.
Here are just a few of the things you can try out in Cloud Shell during this period:
Build and deploy a NodeJs application
Create and manage Google Compute Engine instances
Manage Google Cloud Storage data
We hope you give it a try and welcome your feedback or interest in volunteering for a user experience research study, please email us at
gcp-shell-feedback@google.com
.
Posted by Cody Bratt, Product Manager
BigQuery cost controls now let you set a daily maximum for query costs
Tuesday, December 15, 2015
Today we’re giving you better cost controls in BigQuery to help you manage your spend, along with improvements to the streaming API, a performance diagnostic tool, and a new way to capture detailed usage logs.
BigQuery is a Google-powered supercomputer that lets you derive meaningful analytics in SQL, letting you only pay for what you use. This makes BigQuery an analytics data warehouse that’s both powerful and flexible. Those accustomed to a traditional fixed-size cluster – where cost is fixed, performance degrades with increased load, and scaling is complex – may find granular cost controls helpful in budgeting your BigQuery usage.
In addition, we’re announcing availability of BigQuery access logs in Audit Logs Beta, improvements to the Streaming API, and a number of UI enhancements. We’re also launching Query Explain to provide insight on how BigQuery executes your queries, how to optimize your queries and how to troubleshoot them.
Custom Quotas: No fear of surprise when the bill comes
Custom quotas
allow you to set daily quotas that will help prevent runaway query costs. There are two ways you can set the quota:
Project wide: an entire BigQuery project cannot exceed the daily custom quota.
Per user: each individual user within a BigQuery project is subject to the daily custom quota.
Query Explain: understand and optimize your queries
Query Explain
shows, stage by stage, how BigQuery executes your queries. You can now see if your queries are write, read or compute heavy, and where any performance bottlenecks might be. You can use BigQuery Explain to optimize queries, troubleshoot errors or understand if
BigQuery Slots
might benefit you.
In the BigQuery Web UI, use the “Explanation” button next to “Results” to see this information.
Improvements to the Streaming API
Data is most valuable when it’s fresh, but loading data into an analytics data warehouse usually takes time. BigQuery is unique among warehouses in that it can easily ingest a stream of up to 100,000 rows per second per table, available for immediate analysis. Some customers even stream 4.5 million rows per second by sharding ingest across tables. Today we’re bringing several improvements to BigQuery Streaming API.
Streaming API in EU locations
. It’s not just for the US anymore: you may now use the Streaming API to load data into your BigQuery datasets residing in EU.
Template tables
is a new way to manage related tables used for streaming. It allows an existing table to serve as a template for a streaming insert request. The generated table will have the same schema, and be created in the same dataset and project as the template table. Better yet, when the schema of the template table is updated, the schema of the tables generated from this template will also be updated.
No more “warm-up” delay
. After streaming the first row into a table, we no longer require a warm-up period of a couple of minutes before the table becomes available for analysis. Your data is available immediately after the first insertion.
Create a paper trail of queries with Audit Logs Beta
BigQuery Audit Logs
form an audit trail of every query, every job and every action taken in your project, helping you analyze BigQuery usage and access at the project level, or down to individual users or jobs. Please note that Audit Logs is currently in Beta.
Audit Logs can be filtered in Cloud Logging, or exported back to BigQuery with one click, allowing you to
analyze
your usage and spend in real-time in SQL.
With today’s announcements, BigQuery gives you more control and visibility. BigQuery is already very easy to use, and with recently launched products like
Datalab
(a data science notebook integrated with BigQuery), just about anyone in your organization can become a big data expert. If you’re new to BigQuery, take a look at the
Quickstart Guide
, and the first 1TB of data processed per month is on us. To fully understand the power of BigQuery, check out the
documentation
and feel free to ask your questions using the “
google-bigquery
” tag on Stack Overflow.
-
Posted by Tino Tereshko, Technical Program Manager
Don't Miss Next '17
Use promo code NEXT1720 to save $300 off general admission
REGISTER NOW
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Labels
Announcements
56
Big Data & Machine Learning
91
Compute
156
Containers & Kubernetes
36
CRE
7
Customers
90
Developer Tools & Insights
80
Events
34
Infrastructure
24
Management Tools
39
Networking
18
Open Source
105
Partners
63
Pricing
24
Security & Identity
23
Solutions
16
Stackdriver
19
Storage & Databases
111
Weekly Roundups
16
Archive
2017
Feb
Jan
2016
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Feed
Subscribe by email
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow
Follow