FIS, a global financial technology and services firm and frequent leader of the FinTech Top 100 list, recently ran a load test of their system on Google Cloud Platform to process, validate and link U.S. stock exchange market events. FIS used Google Cloud Dataflow and Google Cloud Bigtable to process 25 billion simulated market events in 50 minutes, generating some impressive statistics in the process.
Cloud Bigtable achieved read rates in excess of 34 million events per second and 22 million event writes per second using 3500 Cloud Bigtable server nodes and 300 n1-standard-32 VMs with Cloud Dataflow. Additionally, Cloud Bigtable provided sustained rates of over 22 million event reads per second and 16 million event writes per second for extended periods of time.
Moreover, Cloud Bigtable was also able to achieve significant I/O bandwidth rates during the load test: read bandwidth peaked at 34 GB/s while write bandwidth peaked at 18 GB/s. Cloud Bigtable sustained significant bandwidth for input and output for 30 minutes as well: 22 GB/s for reads and 13 GB/s for writes.
For FIS, these performance capabilities make it possible to process an entire day’s worth of U.S. equities and options data and make it available for analysis within four hours.
For the complete set of benchmark results, see these slides. You can see a more detailed description of the overall system architecture presented by Neil Palmer and Todd Ricker from FIS and Carter Page, engineering manager for Google Cloud Bigtable:
We look forward to working with other innovative companies like FIS to help them address data processing challenges with the performance, scalability and NoOps approach that Cloud Bigtable provides.
- Posted by Misha Brukman, Product Manager for Google Cloud Bigtable
Google Cloud Datastore is a highly-scalable NoSQL database for web and mobile applications. Today we’re announcing much simpler pricing, and as a result, many users will see significant cost-savings for this database service.Read More
Google Cloud Datastore is a highly-scalable NoSQL database for web and mobile applications. Today we’re announcing much simpler pricing, and as a result, many users will see significant cost-savings for this database service.
Along with the simpler pricing model, there’ll be a more transparent method of calculating stored data in Cloud Datastore. The new pricing and storage calculations will go into effect on July 1st, 2016. For the majority of our customers, this will effectively result in a price reduction.
New pricing structure
We’ve listened to your feedback and will be simplifying our pricing. The new pricing will go into effect on July 1st, 2016, regardless of how you access Datastore. Not only is it simpler, but also the majority of our customers will see significant cost savings. This change removes the disincentive our current pricing imposes on using the powerful indexing features, freeing developers from over-optimizing index usage.
We’re simplifying pricing for entity writes, reads and deletes by moving from internal operation counting to a more direct entity counting model as follows:
Writes: In the current pricing, writing a single entity translated into one or more write operations depending on the number and type of indexes. In the new pricing, writing a single entity only costs 1 write regardless of indexes and will now cost $0.18 per 100,000. This means writes are more affordable for people using multiple indexes. You can use as many indexes as your application needs without increases in write costs. Since on average the vast majority of Entity writes previously translated to more than 4 write operations per entity, this represents significant cost savings for developers.
Reads: In the current pricing, some queries would charge a read operation per entity retrieved plus an extra read operation for the query. In the new pricing, you'll only be charged per entity retrieved. Small ops (projections and keys-only queries) will stay the same in only charging a single read for the entire query. The cost per Entity read stays the same as the old per operation cost of $0.06 per 100,000. This means that most developers will see reduced costs in reading entities.
Deletes: In the current pricing model, deletes translated into 2 or more writes depending on the number and type of indexes. In the new pricing, you'll only be charged a delete operation per entity deleted. Deletes are charged at the rate of $0.02 per 100,000. This means deletes are now discounted by at least 66% and often by more.
Free Quota: The free quota limit for Writes is now 20,000 requests per day since we no longer charge multiple write operations per entity written. Deletes now fall under their own free tier of 20,000 requests per day. Over all, this means more free requests per day for the majority of applications.
To coincide with our pricing changes on July 1st, Cloud Datastore will also use a new method for calculating bytes stored. This method will be transparent to developers so you can accurately calculate storage costs directly from the property values and indexes of the Entity. This new method will also result in decreased storage costs for the majority of customers.
Our current method relies heavily on internal implementation details that can change, so we’re moving to a fixed system calculated directly from the user data submitted. As the new calculation method gets finalized, we’ll post the specific details so developers can use it to estimate storage costs.
Building what’s next
With simpler pricing for Cloud Datastore, you can spend less time micro-managing indexes and focus more on building what’s next.
Google Compute Engine provides Persistent Disks to use as the primary block storage for your virtual machine instances. Provisioning the appropriate size of block storage has been a challenge for many cloud and on-premise customers because it requires planning for future data growth and performance needs. When a virtual machine runs out of space, there was no easy way to scale the size of your block storage.Read More
Google Compute Engine provides Persistent Disks to use as the primary block storage for your virtual machine instances. Provisioning the appropriate size of block storage has been a challenge for many cloud and on-premise customers because it requires planning for future data growth and performance needs. When a virtual machine runs out of space, there was no easy way to scale the size of your block storage.
Today we're announcing general availability of online resizing for Persistent Disks. It’s as easy as a button click or a single API call. It doesn’t cause any downtime to Google Compute Engine instances and doesn’t require snapshotting. It applies to all Persistent Disks, including the recently announced 64 TB volumes.
With the introduction of this feature, Persistent Disk capacity planning becomes much simpler. Persistent Disks can be provisioned based on immediate needs and increased in size later when you require more space or performance1. Instead of implementing a complex workflow that would take the system offline — such as snapshot the disk, restore the snapshot to a larger device then bring back online again — there's a single command that makes physical devices larger. The device immediately has higher IOPS and throughput limits. After you resize a disk that's already mounted on a VM instance, resize the file system. Usually it's as simple as running resize2fs on Linux or resizing partitions in Windows Disk Manager.
Internally we've been using online disk resizing with Cloud SQL Second Generation. It has enabled automatic growth of Persistent Disks used by Google Cloud SQL with no downtime.
We hope you enjoy the new feature!
- Posted by Igor Belianski, Software Engineer, Google Compute Engine
1 Persistent Disk performance depends on the size of the volume and the type of disk you select. Larger volumes can achieve higher I/O levels than smaller volumes.
The “Using IAM Securely” guide will help you to implement IAM controls securely by providing a checklist of best practices for the most common areas of concern when using IAM. It categorizes best practices into four sections:
Least privilege - A set of checks that assist you in restricting your users or applications to not do more than they're supposed to.
Managing Service Accounts and Service Account keys - Provides pointers to help you manage both securely.
Policy Management - Some checks to ensure that you're implementing and managing your policies appropriately.
Cloud Platform resources are organized hierarchically and IAM policies can propagate down the structure. You're able to set IAM policies at the following levels of the resource hierarchy:
Organization level. The Organization resource represents your company. IAM roles granted at this level are inherited by all resources under the organization.
Project level. Projects represent a trust boundary within your company. Services within the same project have a default level of trust. For example, App Engine instances can access Cloud storage buckets within the same project. IAM roles granted at the project level are inherited by resources within that project.
Resource level. In addition to the existing Google Cloud Storage and Google BigQuery ACL systems, additional resources such as Google Genomics Datasets and Google Cloud Pub/Sub topics support resource-level roles so that you can grant certain users permission to a single resource.
The diagram below illustrates an example of a Cloud Platform resource hierarchy:
The “Designing Resource Hierarchies” guide provides examples of what this means in practice and has a handy checklist to double-check that you're following best practice.
A Service Account is a special type of Google account that belongs to your application or a virtual machine (VM), instead of to an individual end user. The “Understanding Service Accounts” guide provides answers to the most common questions, like:
What resources can the service account access?
What permissions does it need?
Where will the code assuming the identity of the service account be running: on Google Cloud Platform or on-premises?
This guide discusses what the implications are of making certain decisions so that you have enough information to use Service Accounts safely and efficiently.
We’ll be producing more IAM best practice guides and are keen to hear from customers using IAM or wanting to use IAM on what additional content would be helpful. We’re also keen to hear if there are curated roles we haven’t thought of. We want Cloud Platform to be the most secure and the easiest cloud to use so your feedback is important to us and helps us shape our approach. Please share your feedback with us at:
Snapchat security engineer, Subhash Sankuratripati, took the stage at GCP NEXT in San Francisco this week, to share his company’s best practices for running securely at scale on Google Cloud Platform ...Read More
Snapchat security engineer, Subhash Sankuratripati, took the stage at GCP NEXT in San Francisco this week, to share his company’s best practices for running securely at scale on Google Cloud Platform. And when we say at scale, we mean at scale!
Snapchat has over 100 million daily users and supports 8 billion videos, viewed daily. The company runs about 100 separate GCP projects, each requiring different permissions for who at the company can do what on which GCP resources.
Until recently, Snapchat engineers exclusively used viewer/editor roles and built their own stopgaps to manage resources on the platform. With the launch of IAM Roles in beta, Snapchat now uses this service to set fine grained permissions it needs to help secure its users’ data. Essentially, Snapchat operates on the principle of least privilege.
The company is working on using our new iam.setpolicy feature to create what it calls Access Control List leases or “ACL leases.” These leases temporarily grant access to resources only when someone needs them, then the policy tears them down when the lease is over, for example:
AccessControlService can iam.SetPolicy
When bob@ needs access, AccessControlService adds bob@ to policy
AccessControlService removes bob@ after 1 hour
Like the nature of Snapchat itself, the company wants to treat access to its cloud resources as ephemeral for maximum security. Snapchat has implemented this leasing model for certain resources and privileges and is striving towards a goal where developers are granted access to the resources when they need it and for however long they need it.
Snapchat’s using the new Organizational Node, which sits above projects and manages GCP resources. This prevents shadow projects from being created, giving the company more control over all projects and the permissions of members associated to those projects. Sankuratripati said he’s also doing data siloing based on role using IAM Roles and is testing the IAM Service Account API, which can be used by programs to authenticate to Google and make API calls.
The possibilities this opens up are endless, according to Sankuratripati. He said microservice to microservice authentication would mean an even larger reduction in what his engineers can manage directly, locking down access to resources even further. Snapchat's strategy is essentially to ensure its developers have enough freedom to get their job done, but not enough to get themselves into trouble.
Stay tuned for more resources coming soon on using IAM on Cloud Platform and as you check out these services, please share your feedback with us at GCP-iam-feedback@google.com.
- Posted by Jo Maitland, Managing Editor, Google Cloud Platform
GCP NEXT 2016 — San Francisco — Attendees at Google Cloud Platform’s user conference this week got a chance to hear from two of the company’s leaders — Joe Kava, VP of data center operations and Niels Provos, distinguished engineer for security and privacy ...Read More
GCP NEXT 2016 — San Francisco — Attendees at Google Cloud Platform’s user conference this week got a chance to hear from two of the company’s leaders — Joe Kava, VP of data center operations and Niels Provos, distinguished engineer for security and privacy — on how the company designs, builds, operates and secures its data centers globally. They shared some of the secret sauce that makes Google's data centers so unique and what this means for GCP customers running inside them.
Security and data protection
Google’s focus on security and protection of data is a key design criteria. Our physical security features a layered security model, including safeguards like custom-designed electronic access cards, alarms, vehicle access barriers, perimeter fencing, metal detectors and biometrics. The data center floor features laser beam intrusion detection. Our data centers are monitored 24/7 by high-resolution interior and exterior cameras that can detect and track intruders. Access logs, activity records and camera footage are available in case an incident occurs.
Data centers are also routinely patrolled by experienced security guards who have undergone rigorous background checks and training (look closely and you can see a couple of them in this 360 degree data center tour). As you get closer to the data center floor, security measures increase. Access to the data center floor is only possible via a security corridor which implements multi-factor access control using security badges and biometrics. Only approved employees with specific roles may enter. Less than one percent of Google employees will ever set foot in one of our data centers.
We employ a very strict end-to-end chain of custody for storage, tracking everything from cradle to grave, from the first time a HD goes into a machine until it’s verified clean/erased or destroyed. Information security and physical security go hand-in-hand. Data is most vulnerable to unauthorized access as it travels across the Internet or within networks. For this reason, securing data in transit is a high priority for Google. Data traveling between a customer’s device and Google is encrypted using HTTPS/TLS (Transport Layer Security). Google was the first major cloud provider to enable HTTPS/TLS by default.
We build our own hardware and monitoring systems
Google servers don’t include unnecessary components such as video cards, chipsets or peripheral connectors, which can introduce vulnerabilities. Our production servers run a custom-designed operating system (OS) based on a stripped-down and hardened version of Linux. Google’s servers and their OS are designed for the sole purpose of providing Google services. Server resources are dynamically allocated, allowing for flexibility in growth and the ability to adapt quickly and efficiently, adding or reallocating resources based on customer demand.
For these teams to be successful they must have advanced, real-time visibility into the status and functionality of our infrastructure. As you might know, Google is obsessed with data, which is a bit of an understatement. To aid our teams, we've built monitoring and controls systems for all functional areas, from the servers, storage and networking systems, to the electrical distribution, mechanical cooling systems and security systems. We're monitoring all aspects of performance and operations from “chip to chiller.”
Using machine learning to optimize data center operations
To help in this endeavor, we’re using our machine learning / deep learning algorithms for data center operations. As you can imagine, our data centers are large and complex, with electrical, mechanical and controls systems all working together to deliver optimal performance. Because of the sheer number of interactions and possible settings for these systems, it's impossible for mere mortals to visualize how best to optimize the data center in real time. However, it's fairly trivial for computers to crunch through these possible scenarios and find the optimal settings.
Over the past couple years we've developed this algorithm and trained it with billions of data points from our sites all over the world. We now use this machine learning model to help visualize the data so the operations teams can set up the data center electrical and cooling plants for the optimal, most efficient performance on any given day considering up to 19 independent variables that affect performance. This helps the team identify discontinuities or efficiency inflection points that aren't intuitive.
Powered by renewable energy
On the energy side, we're committed to powering our infrastructure with renewable energy. We're the world's largest private investor in renewable energy. To date we've invested more than $2 billion in renewable energy Power Purchase Agreements. These PPA's are very important because (1) we're buying the entire output of wind and solar farms for long periods, typically 10-20 years, (2) these wind farms are on the same power grids as our data centers, and (3) wind farms and data centers sharing power grids gives the project developer the financial commitment they need to get the project built, so we know our investment is adding renewable power to the grid that wouldn’t otherwise have been added.
For cooling, we've redesigned our fundamental cooling technology on average about every 12-18 months. Along the way, we've developed and pioneered innovations in water-based cooling systems such as seawater cooling, industrial canal water cooling, recycled / grey water cooling, stormwater capture and reuse, rainwater harvesting and thermal energy storage. We've designed data centers that don't use water-based solutions, instead using 100% outside air cooling. The point is there's no "one size fits all" model here. Each data center is designed for the highest performance and highest efficiency for that specific location.
Google employees operate our data centers, not third parties
The norm in the industry is for the design and building contractor to drop off a set of owners manuals and drawings along with the keys to the front door and wish the operator of the data center good luck! All too often these operations teams aren't employed by the owner, but rather an outsourced low-bidder. This is not the case at Google. Our employees manage and operate our data centers. If there's one certainty in data center operations, it's that problems and faults will always happen in the middle of the night — typically on Sundays — when nobody else is around to help :-)
Engineering + operation teams are combined
We also take a different approach to the people we hire and how they run our data centers. Our engineers and operations professionals come from very diverse backgrounds but all have a common trait — they're systems thinkers. Many of our team members come from mission critical environments, like the Navy nuclear submarine program, where mistakes can be catastrophic — they understand how systems interact together. Further, we've built regional site teams at all our data center campuses comprised of the engineers responsible for the design and construction, working side-by-side with the operations teams. Together these integrated teams are responsible for building capacity, commissioning the systems and providing 7x24 operations. This gives us an unparalleled level of ownership of our infrastructure.
- Posted by Joe Kava, VP, Data Center Operations, Google
We’re excited to share the Google Data Center 360° Tour — a YouTube 360° video that gives you an unprecedented and immersive look inside one of our data centers.
There are several ways to view this video ...Read More
We’re excited to share the Google Data Center 360° Tour — a YouTube 360° video that gives you an unprecedented and immersive look inside one of our data centers.
There are several ways to view this video:
On desktop using Google Chrome — use your mouse or trackpad to change your view while the video plays
YouTube app on mobile — move your device around to look at all angles while the video plays
And the most immersive way to view — using Google Cardboard (currently supported by the Android YouTube app only, iOS support is coming soon!) Load the video in the YouTube app and tap on the Cardboard icon when the video starts to play. Insert your phone in Cardboard and look around.
A little background . . .
Several months ago, those of us on the Google Cloud Developer Advocacy Team had a rare opportunity to tour the Google data center in The Dalles, Oregon. Many of us had seen other non-Google data centers in our careers, but this experience was beyond anything we ever imagined. We were blown away by the scale, the incredible attention to security and privacy, and the amazing efforts to make the data center extremely efficient and green. Additionally, we were proud to meet some of the brilliant people that design, build and maintain these data centers.
If you are a Google Cloud Platform customer, then this is your data center as much as it is our data center, so we want you to experience what we experienced.
We hope you enjoy it!
- Posted by Greg Wilson, Head of Developer Advocacy, Google Cloud Platform
GCP NEXT 2016 — SAN FRANCISCO — Today is a big day for Google Cloud Platform (GCP). We’re excited to welcome thousands of developers and IT pros from around the world to San Francisco for our annual cloud computing conference. During the event, we'll be announcing major enhancements to the platform as well as sharing important progress in how we’re putting the same infrastructure that powers Google, to work for your business.Read More
GCP NEXT 2016 — SAN FRANCISCO — Today is a big day for Google Cloud Platform (GCP). We’re excited to welcome thousands of developers and IT pros from around the world to San Francisco for our annual cloud computing conference. During the event, we'll be announcing major enhancements to the platform as well as sharing important progress in how we’re putting the same infrastructure that powers Google, to work for your business.
In the last year, cloud has gone from being the untrusted option to being seen as a more secure option for many companies. We know that compliance, support and integration with existing IT investments is critical for businesses trying to use public cloud services to accelerate into new markets. So what is Google doing to help?
We’re combining the 15+ years of ground-breaking, applied computer science in distributed computing, data management and machine learning that powers Google with the capabilities businesses need to safely adopt cloud today.
We couldn’t be prouder of the amazing set of customers that have chosen Cloud Platform to take advantage of this continuous innovation at Google. Best Buy, Disney Consumer Products & Interactive Media, Domino’s Pizza, FIS Global, Spotify, Macy’s, Pocket Gems, Wix, Atomic Fiction, JDA, Heineken and many more companies are using GCP because they believe that the key to staying in front is staying innovative.
For our customers, cloud means no longer having to think about data centers, servers, storage and networking. Instead, they're able to focus on creating amazing applications, products and services for their customers knowing that Google is taking care of the infrastructure powering their business.
Data center expansion
Customers around the world are quickly scaling up their applications on Cloud Platform, made possible by our global network. In addition to our network, we’re adding more regions as our customer base becomes larger, more diverse and accustomed to running their workloads on cloud.
Hybrid cloud management and operational efficiency
Partners are a critical piece of our strategy to helping customers run their applications on Google Cloud. In the last year, we've rolled out a new partner program, our ecosystem has more than doubled in size and we're seeing innovations in terms of how partners build solutions on our platform to help customers adopt cloud.
Last week, we had over 300 GCP ecosystem partners join us at TeamWork 2016, our annual Global Partner Conference. We rolled out several new partner programs and incentives at the event, all designed to accelerate partner success and stimulate partner innovations on our platform. The energy coming out of the event was great to see, and we're excited for partners to be at the core of everything we do. Our goal is to build a partner economy, not just a channel and to put partners at the center of our strategy.
On that note, we’re happy to announce that BMC, Pivotal, Red Hat, SAP, Splunk, Tenable Network Security, Veritas and many other enterprise ISVs are working hard to integrate their software with GCP, enabling customers to leverage the skills and software they're already familiar with to manage and monitor their resources on Cloud Platform. System Integrators are also a key aspect of our partner strategy. Accenture, CI&T, Cloud Technology Partners, PA Consulting and PwC are important partners helping enterprises move to Google Cloud.
Enterprise feature enhancements
Moving on to features we know you care about, audits and compliance are big concerns for enterprise cloud buyers — beyond these are administration, setting policies and tracking and control of the environment.
Audit Logging
We'll be launching Audit Logging before the end of May, to enable you to answer the question of “who did what, where, and when?” on Cloud Platform. This launch provides the core infrastructure needed for individual Google Cloud services to provide immutable audit logs along with multiple initial service integrations, including Google App Engine, BigQuery, Dataflow, IAM for Projects and Service Accounts, as well as API Credentials. Audit logs are delivered to the Cloud Console Activity Stream as well as to Stackdriver Logging, from where they can easily be archived in Google Cloud Storage, streamed to BigQuery for analysis, or exported via Google Cloud Pub/Sub to a variety of partners, such as Splunk, for additional interrogation. This launch marks the beginning of an ongoing process in which we'll be continuously rolling out audit logs to the rest of Cloud Platform.
IAM Roles
Securely controlling access to Google Cloud resources is important to you. We know that the existing owner/editor/viewer roles are not granular enough for all your resource management needs. That’s why we’ve created a set of new IAM (Identity and Access Management) roles now launched to beta.
IAM allows you to assign permissions to your Google Cloud resources through IAM roles, which are defined as a collection of permissions — owner/editor/viewer gave users permissions to all resources in a project. These new roles allow you to grant more granular permissions to specific types of resources in a project. This is the first of the many launches we have planned to enhance IAM capabilities on GCP. In the coming months, we'll add more roles and the ability to define your own custom roles.
Customer Supplied Encryption Keys
The ability to control and manage your own encryption keys is another capability that you've asked us for. We announced customer supplied encryption in beta for Compute Engine last July and we'll be graduating to GA shortly. Cloud Storage also supports bringing your own encryption keys to secure storage resources, currently in beta.
Networking
On the cloud networking front, we've improved the flexibility of both cross-cloud interconnect and intra-cloud network segmentation options so that you can federate network transport and optimization services, as well as support workload portability across hybrid cloud environments. GCP delivery of Subnetworks, Cloud Router, Cloud VPN and IAM network roles take advantage of this flexibility and enforce dynamic network and security policies with programmatic control and real-time app/user context, eliminating low-level configuration complexity through intelligent automation. Our software-defined Cloud Load Balancer simplifies global cloud service delivery with a single Virtual IP (VIP), while delivering best-in-class auto-scaling and speed (see more here). This resilient BGP-enabled anycast network infrastructure also paves the way for services like Cloud CDN, which uses our distributed edge cache infrastructure to optimize user experience for rich media applications.
Commitment to openness and running containers at scale
Google and Cloud Platform are innovating in computer science in the open. We're fully committed to contributing our learnings back to the community, with some notable, revolutionary examples: Hadoop MapReduce, Spanner, Software-Defined Networking, Kubernetes, Dataflow and TensorFlow for machine learning, among hundreds more. We recently joined the Open Compute Project to drive standards in IT infrastructure; we're a sponsor of the OpenStack Foundation and Cloud Native Computing Foundation, and would urge customers for whom openness is important to look at our long track record as a contributing member of the software community.
At the heart of our open source contributions in cloud is Kubernetes, an open source system for automating deployment, scaling and operations of containerized apps. Recently we announced Kubernetes 1.2, which includes two important updates for enterprises working with containers. The cluster size has been increased 400% to 1,000 nodes and 30,000 containers per cluster; and we added support for TLS for secure communication and L7 for http-based traffic routing, providing a straightforward way to integrate into custom networking environments. Because our fully managed container service, Google Container Engine (GKE) is built on Kubernetes, customers using this service automatically inherit all the latest functionality.
The future of cloud is just beginning
From advancements in machine learning and containers, to better ways to monitor, manage and secure cloud workloads, we’re taking big steps forward to change how businesses compute. But that’s just the start of this next wave of cloud innovation. More and more developers, startups and companies large and small are discovering the benefits of a different kind of cloud, one that challenges convention through continuous innovation, while protecting and leveraging existing IT investments.
We look forward to your feedback at GCP NEXT and hope you’ll join us on this exciting journey!
- Posted by Brian Stevens, VP, Product Management, Google Cloud Platform
GCP NEXT 2016 — SAN FRANCISCO — Hundreds of different big data and analytics products and services fight for your attention as it's one of the most fertile areas of innovation in our industry. And it’s no wonder; the most amazing consumer experiences are driven by insights derived from information. This is an area where ...Read More
GCP NEXT 2016 — SAN FRANCISCO — Hundreds of different big data and analytics products and services fight for your attention as it's one of the most fertile areas of innovation in our industry. And it’s no wonder; the most amazing consumer experiences are driven by insights derived from information. This is an area where Google Cloud Platform has invested almost two decades of engineering, and today at GCP NEXT we’re announcing some of the latest results of that work. This next round of innovation builds on our portfolio of data management and analytics capabilities by adding new products and services in multiples key areas:
Machine Learning:
We're on a journey to create applications that can see, hear and understand the world around them. Today we've taken a major stride forward with the announcement of a new product family: Cloud Machine Learning. Cloud Machine Learning will take machine learning mainstream, giving data scientists and developers a way to build a new class of intelligent applications. It provides access to the same technologies that power Google Now, Google Photos and voice recognition in Google Search as easy to use REST APIs. It enables you to build powerful Machine Learning models on your data using the open-source TensorFlow machine learning library:
Cloud Machine Learning makes it easy for you to build sophisticated, large scale machine learning models in a short amount of time. It’s portable, fully managed and scalable. Cloud Machine Learning works with data in many formats and is well integrated with other Cloud Platform products such as Google Cloud Dataflow, Google BigQuery, Google Cloud Dataproc, Google Cloud Storage, and Google Cloud Datalab. You can easily build predictive analytics models using your own training data. For example, a financial services app that predicts values using regression models, or a classification service for images. Cloud Machine Learning will take care of everything from data ingestion through to prediction. The result: now any application can take advantage of the same deep learning techniques that power many of Google’s services.
Pre-trained Machine Learning models like Google Translate API and Cloud Vision API are being joined today by Google Cloud Speech API. We're very excited to now offer a full set of APIs that help your applications see, hear and translate. Cloud Speech API brings the same advanced neural network technology that powers voice search in the Google app and voice typing in Google Keyboard. It demonstrates speech-to-text conversion in 80+ languages with unparalleled accuracy, especially in noisy environments, and is blazingly fast. The technology that has already empowered developers to add speech to their Chrome and Android is now available for any application in real-time streaming or batch mode.
Big Data and Analytics:
Doing big data the cloud way means being more productive when building applications, with faster and better insights, without having to worry about the underlying infrastructure. To further this mission, we recently announced the general availability of Cloud Dataproc, our managed Apache Hadoop and Apache Spark service, and we’re adding new services and capabilities today:
BigQuery continues to push the limits of what it means to be a fully managed Analytics Data Warehouse. Today we're announcing lots of exciting new features that make analytics faster, cheaper and easier to use:
Long Term Storage automatically drops the price of your storage by 50% after 90 days.
Automatic Table Partitions (coming soon) simplifies the way you store and query data by partitioning your table by date and querying the date ranges you want.
The new Capacitor storage engine, which accelerates many queries by up to 10x, and Poseidon, a new mechanism that improves data ingest and export speed by 5x.
The new Public Datasets Program to help our community host, share and analyze public datasets.
All these features will come to you automagically, without any upgrades or downtime — the way fully managed is meant to be.
We recently announced Google Data Studio 360, a new reporting and data visualization product that allows you to unify all analytics workflows into one tool. Data Studio allows you to access, transform and share datasets from several different sources such as Google Analytics, BigQuery, and Google Sheets. You can then visualize all of the different sources in the same report and collaborate with your peers to build beautiful and flexible reports. Google Data Studio is available in private Beta (i.e., invite only) for Google Analytics Premium customers.
Open Source:
Our Cloud Machine Learning offering leverages Google’s cutting edge machine learning and data processing technologies, some of which we’ve recently open sourced:
TensorFlow, Google’s latest machine learning system, is the #1 Machine Learning project on GitHub today. We’re continuing to develop this ecosystem. For example, you can now use TensorFlow Serving with another of our open-source projects, Kubernetes, to scale and serve ML models. The Cloud Machine Learning product extends these capabilities, enabling you to build powerful Machine Learning models with your data on Cloud Platform.
Earlier this year, we partnered with a number of organizations, including data Artisans, Cloudera, Talend and others to submit the Dataflow model, SDKs and runners for popular OSS distributed systems to the Apache Incubator. This new incubating project, known as Apache Beam, allows you to define portable, powerful and simple data processing pipelines that can execute in either streaming or batch mode.
The Cloud Platform team is super excited about 2016. Some of the biggest brands in the world like Spotify, Atomic Fiction and Khan Academy are using our big data services. We look forward to seeing the innovative ways you’ll use these new products. To get started, visit cloud.google.com/.
- Posted by Fausto Ibarra, Director, Product Management
GCP NEXT 2016— SAN FRANCISCO — We’re excited to introduce Google Stackdriver, a unified monitoring, logging and diagnostics service that makes ops easier, whether you’re running applications on ...Read More
GCP NEXT 2016— SAN FRANCISCO — We’re excited to introduce Google Stackdriver, a unified monitoring, logging and diagnostics service that makes ops easier, whether you’re running applications on Google Cloud Platform (GCP), Amazon Web Services (AWS)1, or a combination of the two.
Stackdriver is the first service to include rich dashboards, uptime monitoring, alerting, log analysis, tracing, error reporting and production debugging, across GCP and AWS, in a single, unified offering. This combination significantly reduces the time that teams spend finding and fixing issues in production.
A unified view across cloud platforms
If you're running an application that spans two or more infrastructure platforms, you’re not alone. We’ve found teams using hybrid infrastructure for a variety of reasons, whether you’re replicating across cloud providers for higher availability, migrating from one cloud to another, or simply choosing the services that best meet the need of each application or component.
To support teams who choose to use GCP and AWS, Stackdriver offers native monitoring, logging and error reporting for both. With Stackdriver, you could start with a single dashboard to monitor the health of an application that's split across clusters on GCP and AWS.
Stackdriver Console for hybrid environment
Likewise, you can define an alerting policy to notify you if either cluster reaches capacity.
Alerting policy incorporating GCP and AWS capacity metrics
You can search for errors in your AWS and EC2 logs in a single interface.
Logs Viewer - search by GCP or AWS service
Finally, Stackdriver will send you error reports when new errors are detected within applications running on either platform:
Strong support for AWS is an essential part of Stackdriver. If you’re running a web application behind an Elastic Load Balancer, for example, Stackdriver provides you with a comprehensive view of the health of that cluster with no setup, including configuration information, uptime, recent events and summary metrics as well as per-availability zone and per-host breakdowns.
The same support for AWS is maintained throughout Stackdriver, from IAM-based setup and API integration to preconfigured dashboards for widely used AWS services to support for SNS Topics as an alerting mechanism and more.
Eliminate data silos, fix problems faster
Stackdriver drastically reduces the number of disparate tools necessary to identify and troubleshoot issues. Within Stackdriver, you can configure uptime checks that monitor the availability of your application endpoints. You can incorporate logs and metrics from your cloud platforms, systems, databases, messaging queues, web servers and application tier into the same monitoring system. You can maintain critical context, such as the timeframe of an issue, as you follow an issue across the monitoring, logging and diagnostics components. For many customers, this will eliminate the need to manually correlate this information across five or more disconnected tools, saving valuable time during incidents and outages.
Your team’s primary starting point might be a summary dashboard that provides an at-a-glance view into the health of your application. That view can include metrics from your cloud platform, system agents, uptime checks, logs and more.
Sample Custom Dashboard with AWS and GCP metrics
Stackdriver can alert your team when issues occur. To avoid dealing with alerts from many different systems when a single issue occurs, you can define alerting policies in Stackdriver that trigger when multiple conditions are true, such as a URL failing an uptime check and latency increases by over 30 percent over a 15-minute period.
Alerting policy with ELB Uptime Check and Latency Threshold
When you discover an issue, Stackdriver helps you follow the trail to the root cause. For example, upon receiving an error report for your Google App Engine application, you may choose to view a summary dashboard, drill down to traces of the latency per URL that your application is serving, and ultimately view logs of specific requests.
Stackdriver Trace Overview
You can also take advantage of integrations with an ecosystem of services to extend the value of Stackdriver. For example, you can stream Stackdriver logs to BigQuery to perform ad-hoc analysis. Likewise, you can use Google Cloud Datalab to perform ad-hoc visualization of time series data. Finally, you can choose among a variety of alerting integrations to ensure that your team receives alert notifications in the appropriate format, including Slack, HipChat, Campfire, and PagerDuty.
Get started in 2 minutes, nothing to maintain or scale
Getting started with Stackdriver is easy. Once you create your account and configure integration with AWS (if applicable), Stackdriver will automatically discover your cloud resources and provide an initial set of metrics and dashboards. From there, you can create uptime checks and deploy our open source agents (packages of Collectd for metrics, Fluentd for logs) to get deeper visibility into your virtual machines, databases, web servers and other components in just a couple of commands.
Stackdriver is built on top of the same powerful technologies that provide monitoring, logging and diagnostics for Google, so you can rest assured that Stackdriver will scale with you as your environment grows. And since Stackdriver is a hosted service, Google takes care of the operational overhead associated with monitoring and maintaining the service for you.
Try Google Stackdriver free during Beta
We're excited to introduce Google Stackdriver and hope you find it valuable in making ops easier — whether you're running on AWS, GCP or both. The service is currently in Beta. Learn more and try it for free at http://cloud.google.com/stackdriver.
Please note that we’ll continue to support existing Stackdriver customers and work closely with them to migrate to Google Stackdriver once it’s generally available.
- Posted by Dan Belcher, Product Manager
1 "Amazon Web Services" and "AWS" are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.↩
Integration with rich third-party ops solutions is important for customers, and we know that many of you are already using these tools to manage hybrid operations in private and public clouds. With that in mind, these partnerships are focused on delivering:
Easy configurable integration of Cloud Platform with partners
New and complementary capabilities, specifically around Security Information and Event Management (SIEM) and compliance reporting
Google Cloud Platform and Splunk
Cloud Platform’s integration with Splunk Enterprise provides insights on operations and leverages Splunk’s unique capabilities around Security Information and Event Management (SIEM).
Turn on the integration by configuring real-time streaming of log data via the Google Cloud Pub/Sub API — a powerful and reliable messaging service responsible for routing data between applications at scale. Once you've configured the integration to stream all of the log data to your Splunk account, you can access the full richness of Splunk Enterprise capabilities. See the details on the partnership here.
Let’s take a scenario in which a network administrator would like to monitor sensitive configuration changes to rules on your network. When an employee in your organization changes firewall rules on any server, the activity is logged by the compute service. The integration between Splunk Enterprise and Stackdriver Logging allows you to monitor such activities and get alerted in real-time. Splunk automatically identifies such interesting trends and anomalies on your system activity data. When you get alerted, you can see the chart, drill down to the actual log entry and rest any undesirable changes that might put your system at risk, thus making it more secure.
Google Cloud Platform and BMC
Our enterprise customers are increasingly building and operating hybrid and even multi-cloud environments. As you migrate existing services and launch new ones on Cloud Platform, we want to ensure that you have access to established solutions like those from BMC that provide a single pane of glass to manage and monitor applications across deployment environments and help with compliance, security and governance.
To kick off this strategic partnership, BMC demonstrated at GCP NEXT an advance version of its Cloud Lifecycle Management product managing and repairing a workload running on Cloud Platform. The company also showcased how Cloud Platform applications can be monitored simultaneously with on-premise installations using BMC TrueSight.
You can learn more about BMC’s suite of solutions here.
Google Cloud Platform and Tenable
We understand that to secure your organization you have to know what applications and workloads are running in it, and who and what devices are trying to access it. Tenable helps secure Cloud Platform with SecurityCenter Continuous View (SecurityCenter CV). This solution supports both on-premises and cloud deployments like Cloud Platform. As a result, organizations familiar with this tool can employ a single technology for monitoring hybrid environments, thereby eliminating the need to buy, deploy and manage multiple tools.
To get started, you'll need to install SecurityCenter CV and create a service account within Cloud Platform and assign permissions to the Tenable service account you plan to use for the Pub/Sub topic. Then publish the logs to the appropriate topic that your Tenable account will subscribe to and you'll see the log and event data in SecurityCenter CV.
Our vision at Cloud Platform is to create a strong ecosystem of partners that provide flexibility and richness of tools, giving you choice and eliminating constraints. Today, we expanded our partners in the ops domain. Please visit the Google Stackdriver Partner page to see the full details on our existing and new ops partners and how you can start using them today.
Please send any feedback to stackdriver-feedback@google.com.
- Posted by Deepak Tiwari and Joe Corkery, Product Managers, Google Cloud Platform
We recommend deploying a Kubernetes-based solution. In the example below, we'll analyze incoming tweets using Google Cloud Pub/Sub (Google’s fully-managed real-time messaging service that allows you to send and receive messages between independent applications) and Google BigQuery (Google's fully managed, no-ops, low cost analytics database). This can be the starting point for incorporating social insights into your own services.
Step 0: If you don’t have a GCP account already, please sign-up for Cloud Platform, setup billing and activate APIs.
Step 1: Next you'll setup a service account. A service account is a way to interact with your GCP resources by using a different identity than your primary login and is generally intended for server-to-server interaction. From the GCP Navigation Menu, click on "Permissions."
Once there, click on "Service accounts."
Click on "Create service account," which will prompt you to enter a service account name. Provide a name relevant to your project and click on "Furnish a new private key." The default "JSON" Key type should be left selected.
Step 2: Once you click "Create," a service account “.json” will be downloaded to your browser’s downloads location.
Important: Like any credential, this represents an access mechanism to authenticate and use resources in your GCP account — KEEP IT SAFE! Never place this file in a publicly accessible source repo (e.g., public GitHub).
Step 3: We’ll be using the JSON credential via a Kubernetes secret deployed to your OpenShift cluster. To do so, first perform a base64 encoding of your JSON credential file:
$ base64 -i ~/path/to/downloads/credentials.json
Keep the output (a very long string) ready for use in the next step, where you’ll replace ‘BASE64_CREDENTIAL_STRING’ in the pod example (below) with the output just captured from base64 encoding.
Important: Note that base64 is encoded (not encrypted) and can be readily reversed, so this file (with the base64 string) is just as confidential as the credential file above.
Step 4: Next you’ll create the Kubernetes secret inside your OpenShift Cluster. A secret is the proper place to make sensitive information available to pods running in your cluster (like passwords or the credentials downloaded in the previous step). This is what your pod definition will look like (e.g., google-secret.yaml):
You’ll need to make two minor tweaks for the solution to work on your OpenShift cluster:
For any pods that need to access Google APIs, modify the pod to create a reference to the secret. The environment variable “GOOGLE_APPLICATION_CREDENTIALS” needs to be exported to the pod — more info on how they work:
Finally, anywhere the solution instructs you to use "kubectl," replace that with the equivalent OpenShift command "oc."
That’s it! If you follow along with the rest of the steps in the solution, you’ll soon be able to query (and see) tweets showing up in your BigQuery table — arriving via Cloud Pub/Sub. Going forward with your own deployments, all you need do is follow the above steps of attaching the credential secret to any pod where you use Google Cloud SDKs and/or access Google APIs.
Join us at GCP Next!
If you’re attending GCP Next and want to experience a live ‘hands-on’ walk-thru of this and other solutions, please join us at the Red Hat OpenShift Workshop. Hope to see you there! If not, don’t miss all the Next sessions online.
- Posted by Sami Zuhuruddin, Solutions Architect, Google Cloud Platform
The public cloud is a network, not just a collection of data centers. Our global network has allowed us to build products that billions of users around the world can depend on. Whether you’re in Taipei or Tijuana, you can get Gmail, Search, Maps or your ...Read More
The public cloud is a network, not just a collection of data centers. Our global network has allowed us to build products that billions of users around the world can depend on. Whether you’re in Taipei or Tijuana, you can get Gmail, Search, Maps or your Google Cloud Platform services with Google speed and reliability.
We’re adding to this global network for Cloud Platform customers by expanding our roster of existing Cloud Platform regions with two more — both operational later this year:
US Western region in Oregon
East Asia region in (Tokyo) Japan
As always, each region has multiple availability zones, so that we can offer high-availability computing to our customers in each locale.
These are the first two of more than 10 additional GCP regions we'll be adding to our network through 2017. This follows the opening last year of a US east coast region in South Carolina for all major GCP services.
We’re opening these new regions to help Cloud Platform customers deploy services and applications nearer to their own customers, for lower latency and greater responsiveness. With these new regions, even more applications become candidates to run on Cloud Platform, and get the benefits of Google-level scale and industry leading price/performance.
The Japan region will be in beta for at least a month. You can fill out this survey to sign up for the beta, and we’ll notify you as soon as it’s ready. If you're interested in Oregon, please fill out this survey to be notified.
To learn how to make the best use of Cloud Platform regions for your application needs, please see the Geography and Regions details page.
We’re excited to announce that the Node.js runtime on Google App Engine is going beta. Node.js makes it easy for developers to build performant web applications and mobile backends with JavaScript. App Engine provides an easy to use platform for developers to build, deploy, manage and automatically scale services on Google’s infrastructure. Combining Node.js and App Engine provides developers with a great platform for building web applications and services that need to operate at Google scale.Read More
We’re excited to announce that the Node.js runtime on Google App Engine is going beta. Node.js makes it easy for developers to build performant web applications and mobile backends with JavaScript. App Engine provides an easy to use platform for developers to build, deploy, manage and automatically scale services on Google’s infrastructure. Combining Node.js and App Engine provides developers with a great platform for building web applications and services that need to operate at Google scale.
Getting started
Getting started with Node.js on App Engine is easy. We’ve built a collection of getting started guides, samples, and interactive tutorials that walk you through creating your code, using our APIs and services and deploying to production.
When running Node.js on App Engine, you can use the tools and databases you already know and love. Use Express, Hapi, Parse-server or any other web server to build your app. Use MongoDB, Redis, or Google Cloud Datastore to store your data. The runtime is flexible enough to manage most applications and services — but if you want more control over the underlying infrastructure, you can easily migrate to Google Container Engine or Google Compute Engine for full flexibility and control.
var gcloud = require('gcloud')({ projectId: 'my-project', keyFilename: 'keyfile.json' });
var vision = gcloud.vision(); vision.detectText('./image.jpg', function(err, text) { if (text.length > 0) { console.log('We found text on this image...'); } });
Services like the Vision API allow you to take advantage of Google’s unique technology in the cloud to bring life to your applications.
Advanced diagnostic tooling
Deploying Node.js applications to Cloud Platform is just the first step. During the lifespan of any application, you’ll need the ability to diagnose issues in production. Google Cloud Debugger lets you inspect the state of Node.js applications at any code location without stopping or slowing it down. You can set breakpoints, and analyze the state of your application in real time:
When you’re ready to address performance, Google Cloud Trace will help you analyze performance by collecting end-to-end latency data for requests to App Engine URIs and additional data for round-trip RPC calls to App Engine services like Datastore, and Memcache.
NodeSource partnership
Along with the Cloud Debug and Trace tools, we’re also announcing a partnership with NodeSource. NodeSource delivers enterprise-grade tools and software targeting the unique needs of running server-side JavaScript at scale. The N|Solid™ platform extends the capabilities of Node.js to provide increased developer productivity, protection of critical applications and peak application performance. N|Solid and Cloud Platform make a great match for running enterprise Node.js applications. You can learn more about using N|Solid on Cloud Platform from the NodeSource blog.
Committent to Node.js and open source
At Google, we’re committed to open source. The new core node.js Docker runtime, debug module, trace tools, gcloud NPM module, everything — all open source:
We’re thrilled to welcome Node.js developers to Cloud Platform, and we’re committed to making further investments to help make you as productive as possible. This is just the start — keep your ear to the ground to catch the next wave of Node.js support on Cloud Platform.
Follow