Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.



Last Tuesday we announced an exciting set of changes to ...Read More

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.



Last Tuesday we announced an exciting set of changes to Google BigQuery making your experience easier, faster and more powerful. In addition to new features and improvements like table wildcard functions, views, and parallel exports, BigQuery now features increased streaming capacity, lower pricing, and more.







1000x increase in streaming capacity

Last September we announced the ability to stream data into BigQuery for instant analysis, with an ingestion limit of 100 rows per second. While developers have enjoyed and exploited this capability, they've asked for more capacity. You now can stream up to 100,000 rows per second, per table into BigQuery - 1,000 times more than before.



For a great demonstration of the power of streaming data into BigQuery, check out the live demo from the keynote at Cloud Platform Live.



Table wildcard functions

Users often partition their big tables into smaller units for data lifecycle and optimization purposes. For example, instead of having yearly tables, they could be split into monthly or even daily sets. BigQuery now offers table wildcard functions to help easily query tables that match common parameters.



The downside of partitioning tables is writing queries that need to access multiple tables. This would be easier if there was a way to tell BigQuery "process all the tables between March 3rd and March 25th" or "read every table which names start with an 'a'". You can do this with this release.



TABLE_DATE_RANGE() queries all tables that overlap with a time range (based on the table names), while TABLE_QUERY() accepts regular expressions to select the tables to analyze.



For more information, see the documentation and syntax for table wildcard functions.



Improved SQL support and table views

BigQuery has adopted SQL as its query language because it's one of the most well known, simple and powerful ways to analyze data. Nevertheless BigQuery used to impose some restrictions on traditional SQL-92, like having to write multiple sub-queries instead of simpler multi-joins. Not anymore, now BigQuery supports multi-join and CROSS JOIN, and improves its SQL capabilities with more flexible alias support, fewer ORDER BY restrictions, more window functions, smarter PARTITION BY, and more.



A notable new feature is the ability to save queries as views, and use them as building blocks for more complex queries. To define a view, you can use the browser tool to save a query, the API, or the newest version of the BigQuery command-line tool (by downloading the Google Cloud SDK).



User-defined metadata

Now you can annotate each dataset, table, and field with descriptions that are displayed within BigQuery. This way people you share your datasets with will have an easier time identifying them.



JSON parsing functions

BigQuery is optimized for structured data: before loading data into BigQuery, you should first define a table with the right columns. This is not always easy, as JSON schemas might be flexible and in constant flux. BigQuery now lets you store JSON encoded objects into string fields, and you can use the JSON_EXTRACT and JSON_EXTRACT_SCALAR functions to easily parse them later using JSONPath-like expressions.



For example:

SELECT json_extract_scalar(
    "{'book': { 
        'category':'fiction', 
        'title':'Harry Potter'}}", 
    "$.book.category");

Fast parallel exports

BigQuery is a great place to store all your data and have it ready for instant analysis using SQL queries. But sometimes SQL is not enough, and you might want to analyze your data with external tools. That's why we developed the new fast parallel exports: With this feature, you can define how many workers will be consuming the data, and BigQuery exports the data to multiple files optimized for the available number of workers.



Check the exporting data documentation, or stay tuned for the upcoming Hadoop connector to BigQuery documentation.



Massive price reductions

At Cloud Platform live, we announced a massive price reduction: Storage costs are going down 68%, from 8 cents per gigabyte per month to only 2.6, while querying costs are going down 85%, from 3.5 cents per gigabyte to only 0.5. Previously announced streaming costs are now reduced by 90%. And finally, we announced the ability to purchase reserved processing capacity, for even cheaper prices and the ability to precisely predict costs. And you always have the option to burst using on-demand capacity.



I want to take this space to celebrate the latest open source community contributions to the BigQuery ecosystem. R has its own connector to BigQuery (and a tutorial), as Python pandas too (check out the video we made with Pearson). Ruby developers are now able to use BigQuery with an ActiveRecord connector, and send all their logs with fluentd. Thanks all, and keep surprising us!



-Posted by Felipe Hoffa, Developer Advocate

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.



Earlier this week, we announced a preview of Google Cloud DNS ...Read More

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.



Earlier this week, we announced a preview of Google Cloud DNS, an authoritative Domain Name System (DNS) service that gives developers a highly available, reliable, and inexpensive way to publish DNS zones and records. Using a simple standards-based API, developers can create and manage their own DNS records. The service's globally distributed nameservers respond to DNS queries to help efficiently route user traffic to servers and web applications.



Cloud DNS can be used to name hosts, webservers and other internet resources, including Google Compute Engine virtual machines, and Google Cloud Storage buckets. You can also use this service for zones and records for systems hosted in your datacenters and remote offices. Cloud DNS serves from more than 20 locations globally, and is based on the same principles and similar infrastructure implemented in Google’s authoritative service used for all Google properties which services billions of queries daily. Cloud DNS’ global, anycast-based network of DNS nameservers responds to end user queries from an optimal location, resulting in low DNS query latency, thereby increasing the access speed and improving the overall experience of end users.



Cloud DNS offers a self-service sign up with an affordable pay-as-you-go model. You pay for the number of managed zones plus records, and for the number of queries serviced for those zones and records. The service is competitively priced at $0.40 per one million queries per month for the first billion queries, and $0.20 per hosted zone per month for the first 25 zones. More details are available at http://developers.google.com/cloud-dns.



You’re welcome to start using the service today by logging into the Google Developers Console and enabling the Cloud DNS API from the APIs menu as shown here:





-Posted by Surbhi Kaul, Product Manager

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.



For many developers, building a cloud-native application begins with a fundamental decision. Are you going to build it on Infrastructure as a Service (IaaS), or Platform as a Service (PaaS)? Will you build large pieces of plumbing yourself so that you have complete flexibility and control, or will you cede control over the environment to get high productivity?Read More

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.



For many developers, building a cloud-native application begins with a fundamental decision. Are you going to build it on Infrastructure as a Service (IaaS), or Platform as a Service (PaaS)? Will you build large pieces of plumbing yourself so that you have complete flexibility and control, or will you cede control over the environment to get high productivity?



You shouldn’t have to choose between the openness, flexibility and control of IaaS, and the productivity and auto-management of PaaS. Describing solutions declaratively and taking advantage of intelligent management systems that understand and manage deployments leads to higher availability and quality of service. This frees engineers up to focus on writing code and significantly reduces the need to carry a pager.



Today, we’re introducing Managed Virtual Machines and Deployment Manager. These are our first steps towards enabling developers to have the best of both worlds.

Managed Virtual Machines

At Google Cloud Platform Live we took the first step towards ending the PaaS/IaaS dichotomy by introducing Managed Virtual Machines. With Managed VMs, you can build your application (or components of it) using virtual machines running in Google Compute Engine while benefiting from the auto-management and services that Google App Engine provides. This allows you to easily use technology that isn’t built into one of our managed runtimes, whether that is a different programming language, native code, or direct access to the file system or network stack. Further, if you find you need to ssh into a VM in order to debug a particularly thorny issue, it’s easy to “break glass” and do just that.



Moving from an App Engine runtime to a managed VM can be as easy as adding one line to your app.yaml file:



vm:true



At Cloud Platform Live, we also demonstrated how the next stage in the evolution of Managed VMs will allow you to bring your own runtime to App Engine, so you won’t be limited to the runtimes we support out of the box.



Managed Virtual machines will soon launch in Limited Preview, and you can request access here starting today.



Introducing Google Cloud Deployment Manager

A key part of deploying software at scale is ensuring that configuration happens automatically from a single source of truth. This is because accumulated manual configuration often results in “snowflakes” - components that are unique and almost impossible to replicate - which in turn makes services harder to maintain, scale and troubleshoot.



These best practices are baked into the App Engine and Managed VM toolchains. Now, we’d like to make it easy for developers who are using unmanaged VMs to also take advantage of declarative configuration and foundational management capabilities like health-checking and auto-scaling. So, we’re launching Google Cloud Deployment Manager - a new service that allows you to create declarative deployments of Cloud Platform resources that can then be created, actively health monitored, and auto-scaled as needed.



Deployment Manager gives you a simple YAML syntax to create parameterizable templates that describe your Cloud Platform projects, including:

• The attributes of any Compute Engine virtual machines (e.g. instance type, network settings, persistent disk, VM metadata).


• Health checks and auto-scaling


• Startup scripts that can be used to launch applications or other configuration management software (like Puppet, Chef or SaltStack)

Templates can be re-used across multiple deployments. Over time, we expect to extend Deployment Manager to cover additional Cloud Platform resources.



Deployment manager enables you to think in terms of logical infrastructure, where you describe your service declaratively and let Google’s management systems deploy and manage their health on your behalf. Please see the Deployment Manager documentation to learn more and to sign up for the Limited Preview.



We believe that combining flexibility and openness with the ease and productivity of auto-management and a simple tool-chain is the foundation of the next-generation cloud. Managed VMs and Deployment Manager are the first steps we’re taking towards delivering that vision.



Update on operating systems

We introduced support for SuSE and Red Hat Enterprise Linux on Compute Engine in December. Today, SuSE is Generally Available, and we announced Open Preview for Red Hat Enterprise Linux last week. We’re also announcing the limited preview for Windows Server 2008 R2, and you can sign up for access now. Windows Server will be offered at $0.02 per hour for the f1-micro and g1-small instances and $0.04 per core per hour for all other instances (Windows prices are in addition to normal VM charges).



Simple, lower prices

As we mentioned on Tuesday, we think pricing should be simpler and more closely track cost reductions as a result of Moore’s law. So we’re making several changes to our pricing effective April 1, 2014.



First, we’ve cut virtual machine prices by up to 53%:

• We’ve dropped prices by 32% across the board.


• We’re introducing sustained-use discounts, which lower your effective price as your usage goes up. Discounts start when you use a VM for over 25% of the month and increase with usage. When you use a VM for an entire month, this amounts to an additional 30% discount.

What’s more, you don’t need to sign up for anything, make any financial commitments or pay any upfront fees. We automatically give you the best price for every VM you run, and you still only pay for the minutes that you use.



Here’s what that looks like for our standard 1-core (n1-standard-1) instance:



Finally, we’ve drastically simplified pricing for App Engine, and lowered pricing for instance-hours by 37.5%, dedicated memcache by 50% and Datastore writes by 30%. In addition, many services, including SNI SSL and PageSpeed are now offered to all applications at no extra cost.



We hope you find these new capabilities useful, and look forward to hearing from you! If you haven’t yet done so, you can sign up for Google Cloud Platform here.



-Posted by Navneet Joneja, Senior Product Manager

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.



Yesterday, we unveiled a new set of developer experiences for ...Read More

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.



Yesterday, we unveiled a new set of developer experiences for Google Cloud Platform that are inspired by the work we’ve done inside of Google to improve our developers productivity. We want to walk you through these experiences in more detail and how we think they can help you focus on developing your applications and growing your business.



Isolating production problems faster

Understanding how applications are running in production can be challenging and sometimes it’s unavoidable that errors will make it into production. We’ve started adding new capabilities to Cloud Platform this week that make it easier to isolate and fix production problems in your application running on Google App Engine.



We are adding a ‘Diff’ link to the Releases page (shown below) which will bring you to a rollup of all the commits that happened between deployments and the source code changes they introduced. Invaluable when you are trying to isolate a production issue.

You can see here where Brad introduced an error into production.

But, looking at source changes can still be like looking for a needle in a haystack. This is why we’re working to combine data from your application running in production and link it with your source code.



In the new and improved logs viewer, we aggregate logs from all your instances in a single view in near real time. Of course, with high traffic levels, just browsing is unlikely to be helpful. You can filter based on a number of different criteria including regular expressions and the time when you saw an issue. We’ve also improved the overall usability by letting you scroll continuously rather than viewing 20 logs per page at a time.



Ordinarily, debugging investigations from the logs viewer would require you to find the code, track down the right file and line and ensure it’s the same version that was deployed in production. This can cause you to completely lose context. In order to address this, we’ve added links from stack traces to the associated code causing them for push-to-deploy users. In one click you are brought to the source viewer at the revision causing the problem with the associated line highlighted.



Shortening your time to fix

You may have noticed the ‘Edit’ buttons in the source viewer. We’re introducing this because we think that finding the error in production is only part of the effort required when things go awry. We continue to ask ourselves how we can make it even faster for you to fix problems. One of the ways we do this inside Google today is to make the fix immediately from the browser.



So, we’re bringing this to you by making it possible to edit a file in place directly from the source viewer in the Console. You can simply click the ‘Edit’ button, make your changes to that file, and click the ‘Commit’ button. No local setup required. We also know that sometimes fixes aren’t that simple and we’re investigating ways we can make it easy to seamlessly open your files in a desktop editor or IDE directly from the Console.

Fix simple problems directly in the browser

Trigger your push-to-deploy setup instantly

Since we’ve integrated this with your push-to-deploy configuration, the commit triggers any build and test steps you have to ensure your code is fully vetted before it reached production. Further, since we’ve built this on top of Git, each change is fully attributed and traceable in your Git repository. This is not SSHing into a machine and hacking on code in production.



An integrated ecosystem of tools you know

Speaking of using other editors, we believe that you are most productive when you have access to all the tools you know and love. That’s why we’re committed to creating a well integrated set of experiences across those tools, rather than than asking you to switch.



With the Git push-to-deploy feature we launched last year, we wanted to make it easy for you to deploy your application utilizing standard Git commands while giving you free private hosted Git repositories. In addition, we understand that many of you host your source code on GitHub and in fact so does the Google Cloud Platform team.



As you can see from the new ‘Releases’ page we showed this week, we’re introducing the ability to connect your GitHub repository to push-to-deploy. We will register a post-commit hook with GitHub to give you all the deployment benefits without moving your source code. Just push to the master branch of your GitHub repository and your code will be deployed!

Push-to-deploy now supports Java builds

Next, we’re excited to introduce new release types for you to use with push-to-deploy. As you can see above, this project is set up to trigger a build, run all your unit tests, and if they all pass, deploy.



Taking a peek under the covers, we utilize a standard Google Compute Engine virtual machine that you own running in your project to perform this build and test. In this case, Google has automatically provisioned the machine with Maven and Jenkins and everything you need to build and run your tests. Your build and tests can be as complex as they need to be and they can use all the resources available on that machine they need to run. What’s more is that all the builds will be done on a clean machine ensuring reliable, repeatable builds on every release.



We’re starting with Maven-based Java builds, but working to release support for other languages, test frameworks and build systems in the future.

The release history showing build, test and deployment status

Tying this together, we’ve simplified getting started on your projects by introducing the new ‘gcloud’ command in the Google Cloud SDK. The Google Cloud SDK contains all the tools and libraries you need to create and manage resources on the Cloud Platform. We recently posted some Cloud SDK tips and tricks.



Now, using the ‘gcloud init’ command, we make setting up a local copy of your project fast by finding your project’s associated Git repository and downloading the source code, so all you have to do is change directories and open your favorite editor.



Once you are initialized, all you need to do is start writing code. After you’re done, running ‘git push’ will push your changes back to the remote repository and trigger your push-to-deploy flow. Then, all your changes will also be available to view on the ‘Source’ page in the Console.



We’ll be rolling out these features to you in stages over the coming weeks, but if you are interested in being a trusted tester for any of them, please contact us here. We’re very excited to hear how you’re using these tools and what features we can build on top of them to make your developer experience even more seamless and fast. We’re just getting started.



-Posted by Cody Bratt, Product Manager

Updated at 12:52 PM PST to include App Engine pricing information



Editors note: Tune in to Google Cloud Platform Live for more information about our announcements. And join us during our 27-city Google Cloud Platform Roadshow which kicks off in Paris on April 7. ...Read More

Updated at 12:52 PM PST to include App Engine pricing information



Editors note: Tune in to Google Cloud Platform Live for more information about our announcements. And join us during our 27-city Google Cloud Platform Roadshow which kicks off in Paris on April 7.



Today, at Google Cloud Platform Live we’re introducing the next set of improvements to Cloud Platform: lower and simpler pricing, cloud-based DevOps tooling, Managed Virtual Machines (VM) for App Engine, real-time Big Data analytics with Google BigQuery, and more.



Industry-leading, simplified pricing

The original promise of cloud computing was simple: virtualize hardware, pay only for what you use, with no upfront capital expenditures and lower prices than on-premise solutions. But pricing hasn’t followed Moore's Law: over the past five years, hardware costs improved by 20-30% annually but public cloud prices fell at just 8% per year.



We think cloud pricing should track Moore’s Law, so we’re simplifying and reducing prices for our various on-demand, pay-as-you-go services by 30-85%:

• Compute Engine reduced by 32% across all sizes, regions, and classes.


• App Engine pricing is drastically simplified. We've lowered pricing for instance-hours by 37.5%, dedicated memcache by 50% and Datastore writes by 33%. In addition, many services, including SNI SSL and PageSpeed are now offered to all applications at no extra cost.


• Cloud Storage is now priced at a consistent 2.6 cents per GB. That’s roughly 68% less for most customers.


• Google BigQuery on-demand prices reduced by 85%.

Sustained-Use discounts

In addition to lower on-demand prices, you’ll save even more money with Sustained-Use Discounts for steady-state workloads. Discounts start automatically when you use a VM for over 25% of the month. When you use a VM for an entire month, you save an additional 30% over the new on-demand prices, for a total reduction of 53% over our original prices.

Sustained-Use Discounts automatically reward users who run VMs for over 25% of any calendar month

With our new pricing and sustained use discounts, you get the best performance at the lowest price in the industry. No upfront payments, no lock-in, and no need to predict future use.



Making developers more productive in the cloud

We’re also introducing features that make development more productive:

• Build, test, and release in the cloud, with minimal setup or changes to your workflow. Simply commit a change with git and we’ll run a clean build and all unit tests.


• Aggregated logs across all your instances, with filtering and search tools.


• Detailed stack traces for bugs, with one-click access to the exact version of the code that caused the issue. You can even make small code changes right in the browser.

We’re working on even more features to ensure that our platform is the most productive place for developers. Stay tuned.



Introducing Managed Virtual Machines

You shouldn't have to choose between the flexibility of VMs and the auto-management and scaling provided by App Engine. Managed VMs let you run any binary inside a VM and turn it into a part of your App Engine app with just a few lines of code. App Engine will automatically manage these VMs for you.



Expanded Compute Engine operating system support

We now support Windows Server 2008 R2 on Compute Engine in limited preview and Red Hat Enterprise Linux and SUSE Linux Enterprise Server are now available to everyone.



Real-Time Big Data

BigQuery lets you run interactive SQL queries against datasets of any size in seconds using a fully managed service, with no setup and no configuration. Starting today, with BigQuery Streaming, you can ingest 100,000 records per second per table with near-instant updates, so you can analyze massive data streams in real time. Yet, BigQuery is very affordable: on-demand queries now only cost $5 per TB and 5 GB/sec reserved query capacity starts at $20,000/month, 75% lower than other providers.



Conclusion

This is an exciting time to be a developer and build apps for a global audience. Today we’ve focused a lot on productivity, making it easier to build and test in the cloud, using the tools you’re already familiar with. Managed VMs give you the freedom to combine flexible VMs with the auto-management of App Engine. BigQuery allows big data analysis to just work, at any scale.



And on top of all of that, we’re making it more affordable than it’s ever been before, reintroducing Moore’s Law to the cloud: the cost of virtualized hardware should fall in line with the cost of the underlying real hardware. And you automatically get discounts for sustained use with no long-term contracts, no lock-in, and no upfront costs, so you get the best price and the best performance without needing a PhD in Finance.



We’ve made a lot of progress this first quarter and you’ll hear even more at Google I/O in June.



-Posted by Urs Hölzle, Senior Vice President

Today’s post comes from Brett Porter, CTO of MaestroDev, a DevOps Automation service for enterprise-class companies.



MaestroDev helps software teams automate every task in a software delivery cycle, so that agile and continuous integration teams can ship higher-quality code faster. Every step of the software development process -- build, test, release, and deploy -- can be managed with a few drags and clicks. Our customers have already chosen their favorite tools; MaestroDev makes those tools work harder.Read More

Today’s post comes from Brett Porter, CTO of MaestroDev, a DevOps Automation service for enterprise-class companies.



MaestroDev helps software teams automate every task in a software delivery cycle, so that agile and continuous integration teams can ship higher-quality code faster. Every step of the software development process -- build, test, release, and deploy -- can be managed with a few drags and clicks. Our customers have already chosen their favorite tools; MaestroDev makes those tools work harder.



To say performance and high availability are important to us is an understatement. We cater to software development teams which are often distributed around the world, so we really have no choice but to perform and scale consistently and to provide 24/7 availability. To achieve this level of performance in a cost-effective manner, last quarter we began moving all of our servers to Google Compute Engine.



Immediately, we experienced significant time savings. Compute Engine helps us quickly deploy large clusters of virtual machines, and then scale up or down quickly. With Compute Engine, it takes about a minute to stand up a complete development environment. For example, one of our customers, the International Health Terminology Standards Development Organisation (IHTSDO), recently moved from a fixed amount of hardware hosted in a datacenter to our SaaS offering on Compute Engine. They immediately realized more build capacity, reduced risk from machine failures, and 80% cost reduction.



A key benefit of Compute Engine is that it allows us to provide a cost-effective product to our customers. Compute Engine charges in granular increments called Sub-Hour Billing, which means we’re able to offer our customers a unique, consumption-based pricing model that is ideal for today’s software development teams. Other orchestration tools charge a license fee and a per-agent fee, effectively pricing out SMBs and small teams at large companies, and artificially limiting throughput performance of development teams, often when they need processing power the most. Our consumption pricing allows granular cost increments, so that users can start small, pay as they go, and scale up gracefully -- only paying for the service as they actually use it. We wouldn’t have been able to offer this subscription plan without the flexibility of Compute Engine.



Google’s partnership with RightScale was the icing on the cake. RightScale is a leader in enterprise multi-cloud management, and our products are deeply integrated. Together we can automate the deployment of products into RightScale environments, and RightScripts can dynamically scale new dev/test environments as needed. Google’s partnership with RightScale means our customers can simply point existing Rightscale templates to Compute Engine, without wasting time manually re-configuring instances.



We’re a fairly lean organization here at MaestroDev, so the performance, scalability, and cost-effectiveness of Compute Engine enable us to tackle problems for enterprise-level software development teams around the world.



-Contributed by Brett Porter, CTO, MaestroDev

Editor’s note: Today’s guest blog comes from Andrew First, co-founder and CTO at Leanplum, a platform for optimizing the mission-critical metrics of mobile apps. Leanplum processes billions of events per day to provide a faster and more effective way for developers to optimize mobile applications. ...Read More

Editor’s note: Today’s guest blog comes from Andrew First, co-founder and CTO at Leanplum, a platform for optimizing the mission-critical metrics of mobile apps. Leanplum processes billions of events per day to provide a faster and more effective way for developers to optimize mobile applications.



Leanplum makes it easy for product managers and marketers to create personalized customer experiences, conduct on-the-fly A/B tests and see immediate results with powerful analytics. We offer an intuitive dashboard, so anyone, not just engineers, can use A/B testing to make informed decisions.



We hit the ground running in 2012 thanks to Google Cloud Platform. As a startup, we deal with a lot of uncertainty on a day-to-day basis. You need to be prepared for unexpected spikes or dips in traffic, without affecting performance; you can’t spend too much money upfront in case you need it for a rainy day; and you can’t rely on developers to spend time configuring servers. Cloud Platform has allowed us to implement and scale seamlessly, and to save money with its pay-as-you-go pricing model.



We started by building our infrastructure on top of Google App Engine, and more recently added Google BigQuery, Google Cloud Datastore, Google Cloud Storage and Google Compute Engine.



BigQuery lets us run arbitrary queries on arbitrary data sets, and like App Engine, autoscales as needed. It has improved our customer response time by allowing us to query over our logs in seconds whenever we receive a support call. Meanwhile, Cloud Datastore lets us store vast amounts of structured data, and Cloud Storage provides secure, scalable storage. We use Compute Engine to take advantage of more powerful cores for processing large amounts of data when generating reports. The pricing model works well for us, as we’re only charged by the minute after the first 10.



From our extensive use of Cloud Platform, we’ve discovered a number of tricks to make it easier to develop on the Cloud Platform:

• By default, App Engine indexes every field in your datastore entities. By indexing only the fields that you need, you can reduce datastore costs by an order of magnitude.


• Run a separate version of your application as a staging instance, so you can test out new features. You can run multiple versions of your app simultaneously, so that every engineer has their own application running at the same time to test new features or to show demos.


• Use AppStats to find bottlenecks in your application. Using AppStats, we were able to reduce the latency of our API by an order of magnitude.


• Use Traffic Splitting to roll out risky changes. Fittingly, we are an A/B testing company that uses A/B testing on our own backend. When we roll out substantial backend changes, we can test them out on a small percentage of traffic to make sure it works well, and to measure the impact on latency and number of instances used.


• Use BigQuery to analyze your logs. We wrote a custom mapreduce to collect our logs and dump them into BigQuery. This gives us the ability to gather statistics on latencies for different RPC calls, frequencies and types of errors, and to resolve customer issues by being able to query over our logs in seconds.

By giving us the tools to develop, iterate and scale quickly, Cloud Platform provided us more time to focus on the product, which is priceless for any startup.



-Contributed by Andrew First, co-founder and CTO, Leanplum

Google is known for creating scalable high performance systems. In a recent blog post, we demonstrated how Google Cloud Platform can rapidly provision and scale networking load to handle ...Read More

Google is known for creating scalable high performance systems. In a recent blog post, we demonstrated how Google Cloud Platform can rapidly provision and scale networking load to handle one million requests per second. A fast front end without a fast backend has limited use, so we decided to demonstrate a backend serving infrastructure that could handle the same load. We looked at popular open source building blocks for cloud applications and choose Cassandra, a NoSQL database designed for scale and simplicity.



Using 330 Google Compute Engine virtual machines, 300 1TB Persistent Disk volumes, Debian Linux, and Datastax Cassandra 2.2, we were able to construct a setup that can:

• sustain one million writes per second to Cassandra with a median latency of 10.3 ms and 95% completing under 23 ms


• sustain a loss of ⅓ of the instances and volumes and still maintain the 1 million writes per second (though with higher latency)


• scale up and down linearly so that the configuration described can be used to create a cost effective solution


• go from nothing in existence to a fully configured and deployed instances hitting 1 million writes per second took just 70 minutes. A configured environment can achieve the same throughput in 20 minutes.

All data presented in this post is using Cassandra Quorum commit (writes must be complete in at least 2 nodes), triple replication, and data encrypted at rest. With commit ONE Compute Engine was able to sustain ~1.4M writes per second, with a latency median of 7.6ms and 95th percentile of 21ms. However, for this post we wanted to focus on quorum commit which improves reliability.



Setup

We deployed a 300 data node cluster running DataStax Cassandra 2.0 on Compute Engine as the backend. To generate the load we used 30 virtual machines that wrote 3 billion small records (170 bytes each) into Cassandra using cassandra-stress. The following depicts the setup used:

You can find the instructions on how to reproduce the results by following the setup instructions.



Results

With 15,000 concurrent clients Cassandra was able to maintain 10.5ms median latency (8.3ms with 12,000 clients), and 95th latency percentile at 23ms. Here is how the solution scales as the number of concurrent clients grows:



Below we show a graph of the throughput versus 95th percentile latency which quickly achieves very good response times after Cassandra initializes its internal state, and Java warms up its heap and memory mapped files table. This test was run longer than the minimal time required to hit over 1M writes per second in order to show the sustained throughput:



In addition to looking at top end performance we also looked at resiliency. We removed ⅓ of the cluster nodes and it remained functional and serving more than 1M writes per second. Median latency held at 13.5ms, 95th percentile at 61.8ms, and 994.9th percentile at 1,333.5ms. We consider those numbers very good for a cluster in distress, proving Compute Engine and Cassandra can handle both spiky workloads and failures.



Conclusion

Tuning the workload costs $5 per hour (on a 3 node cluster), and the minimal test required to hit one million writes per second takes 1 hour and 10 minutes at a cost of $330 USD when run in March 2014.



Putting it all together, this means the Google Cloud Platform was able to sustain one million Cassandra writes per second at a cost of $0.07 USD per million writes.



-Posted by Ivan Santa Maria Filho, Performance Engineering Lead

Real time bidding (or programmatic buying), is one of the fastest growing methods of buying and selling ads online, and is predicted to account for 25% of all display spending by next year ...Read More

Real time bidding (or programmatic buying), is one of the fastest growing methods of buying and selling ads online, and is predicted to account for 25% of all display spending by next year. Marketers are embracing this model as it allows them to connect to audiences at scale quickly and we've been making ongoing investments to help them grow their programmatic businesses. Today we're continuing those investments by introducing functionality that enables marketers to conduct their real-time bidding on the DoubleClick Ad Exchange (AdX) directly from Google Cloud Platform.



Real-time bidding (RTB) enables ads to be bought and sold on a per-impression basis, through an instantaneous auction, similar to financial markets. Using RTB, ad buyers can increase return on investment since it offers the ability to show the right ad to the right person at the right time.



Speed is critical to success in programmatic buying and many exchanges, including the DoubleClick exchange, require that bidders respond to an ad request within a certain time limit in order to preserve the real-time nature of the marketplace. Real time bidding transactions typically happen within 100 milliseconds from a user visiting a website. Starting today, Ad Exchange users hosting on Google Compute Engine will always receive 100 milliseconds for bid request processing and free network transit from Google Cloud Platform to all Ad Exchange trading locations. This effective increase in processing time, combined with the strengths of Google Compute Engine will make Google Cloud Platform a great choice for DoubleClick Ad Exchange buyers.



To ensure that AdX customers have the best possible experience on Google Cloud Platform, we now offer complimentary Gold support for Google Cloud Platform to qualified customers hosting their bidding infrastructure for AdX. You can find more information on this offer as well as instructions on how to sign up here.



For customers who would like to jump start their real-time bidding using a framework provided by Google, Open Bidder provides you the real-time bidding infrastructure. You can sign up for access to Open Bidder here.



For those looking to build their own bidders without the use of our OpenBidder framework, we’ve also published a white paper on how to build real-time bidding solutions on Google Cloud Platform.



If you are interested in learning more about Open Bidder, real-time bidding on DoubleClick Ad Exchange, and bidder hosting on Google Cloud Platform, please join us at an AdX Developer Day in Google’s New York City or San Francisco office on April 9. You can register for the event here.



-Posted by Scott Spencer, Director of Product Management, Doubleclick Ad Exchange



* For trading on the DoubleClick Ad Exchange Hong Kong trading location, please contact Ad Exchange Sales.

Command-line tools are empowering: they deliver speed, control, traceability, scripting and automation capabilities to the developer’s workflow. Here are a few tips and tricks on how to start working with, and how to get more out of Google Cloud Platform’s command-line tools.Read More

Command-line tools are empowering: they deliver speed, control, traceability, scripting and automation capabilities to the developer’s workflow. Here are a few tips and tricks on how to start working with, and how to get more out of Google Cloud Platform’s command-line tools.



Simplified Tool Installation

For streamlined tool distribution, all Google Cloud Platform command-line tools are bundled in Google Cloud SDK, so you only need to obtain Cloud SDK to get command-line access to App Engine, Compute Engine, Cloud Storage, Cloud SQL, BigQuery and other products.



Here’s how to install Cloud SDK:

• If you’re running on Windows, download Cloud SDK zip and launch install.bat script,


• If you’re running on Linux or Mac OS run the following command in your shell/Terminal:

curl https://dl.google.com/dl/cloudsdk/release/install_google_cloud_sdk.bash | bash

When the installer completes, restart your shell/Terminal/Command Prompt window and you should be able to run the main Cloud SDK command-line tool, called gcloud.

.

Component Management

One of the new features installed with Cloud SDK is a component manager, accessible via gcloud components command group. The component manager allows you to add, remove and update Cloud Platform tools without ever needing to re-download the SDK.



Run gcloud components list to see the list of available or installed components, gcloud components update component-id to install or update a particular component or gcloud components --help for more information.



Tip: if you’re using more than one programming language for App Engine development, use Cloud SDK component manager to install multiple App Engine SDKs side-by-side. For example, if you wanted to add Python and Java support for App Engine, run gcloud components update gae-python gae-java.



Staying Up-to-Date

To ensure that you always have the latest versions of Cloud Platform tools, Cloud SDK will notify you if any updates are available when you run individual commands. For example, if Cloud SQL releases an update, the following message will be appended after running Cloud SQL commands:

$ gcloud sql instances list
my-developers-console-project:sql-instance

There are available updates for some Cloud SDK components. To install them, please run:
$ gcloud components update

You can then perform the update in place using the component manager by running gcloud components update, as described above.



Tip: if you want to stick with the current version of the tools that you have installed, you can disable these notifications by running gcloud config set --section component_manager disable_update_check.



Unified Authentication into the whole Google Cloud Platform

Google Cloud SDK provides a unified authentication model across all of the command-line tools. You only need to complete the authentication flow via gcloud auth login once, and you will be authenticated into all tools simultaneously.



Tip: Cloud SDK also allows you to use multiple accounts by repeating gcloud auth login flow. Run gcloud auth list to see all credentialed accounts, and gcloud config set account to set the active account.



Project Initialization and Push-to-Deploy

Every new project created on Google Developers Console now comes with a free private Git repository, where you can store your source code and use it for your App Engine app deployments via push-to-deploy.



To quickly initialize your local environment, run cloud init project-id. This command will set-up the version control system, clone the source code onto your machine and will set-up push-to-deploy, all in one command. Here’s a workflow example:

1. Create “my-awesome-app” application in Google Developers Console.


2. Install Cloud SDK and run gcloud init my-awesome-app.


3. Change directory to my-awesome-app/default and add some code. (Have a look at some sample projects if you need inspiration.)


4. Commit the code by running git commit -a -m “First commit for my awesome app” and deploy it by running git push origin master.


5. Your app should now be serving live at https://my-awesome-app.appspot.com!

General Usage Tips

Command autocompletion



If you’re running on Mac or Linux and you’ve chosen to enable command-line autocompletion during Cloud SDK’s install, then you should be able to use the Tab key to complete the commands under gcloud in your shell or Terminal.



For example, typing “gclo + Tab + con + Tab + l + Tab” will expand to “gcloud config list“. Similarly, pressing the Tab key twice should show all possible options in ambiguous cases, e.g. typing “gclo + Tab + co + Tab + Tab” should show “config components”.



Tip: this works for flags too! Type “gcloud sql instances create - + Tab + Tab“ (don’t forget the dash at the end!) to see all possible flags that you can pass for a new Cloud SQL instance.



Interactive mode



To further simplify scripting and automation, Cloud SDK allows you to call various gcloud commands from Python scripts. To experiment with this, run gcloud interactive to start an interactive Python shell.



For example, to get the currently set default project from gcloud config list (without scraping the console output), run gcloud interactive to get into the interactive Python mode and paste the gcloud.config.list()[’core’][’project’] command. Python interpreter should return you the string containing the default project. You can then use a similar approach to get the currently set project from your Python scripts.



See the interactive mode documentation for more details.



Help and support

If you’re ever unsure about how to use one or another gcloud command, try appending “--help“ to the end of a command. This works at all command nesting levels, e.g.

• gcloud --help


• gcloud sql --help


• gcloud sql backups --help


• gcloud sql backups list --help

If you have any further questions post them on Stack Overflow using “gcloud” tag, send us a message at google-cloud-sdk@googlegroups.com (our discussions and support forum), or come and chat to us live on #gcloud channel at freenode IRC network.



Note that other Cloud Platform command-line tools are still being integrated into gcloud, so stay tuned and check back our blog soon for more good news about Cloud SDK!



-Posted by Manfred Zabarauskas, Product Manager

(cross posted from Google Developers Blog)



We strive to make our APIs accessible to anyone on any platform: ReST, HTTP and JSON mean that from nearly any language on nearly any hardware, you can call any of our APIs. However, to be truly useful on many platforms, it helps to have a client library -- one that packs a lot of functionality like handling auth, streaming media uploads and downloads, and gives you native language idioms.Read More

(cross posted from Google Developers Blog)



We strive to make our APIs accessible to anyone on any platform: ReST, HTTP and JSON mean that from nearly any language on nearly any hardware, you can call any of our APIs. However, to be truly useful on many platforms, it helps to have a client library -- one that packs a lot of functionality like handling auth, streaming media uploads and downloads, and gives you native language idioms.



Today, we are announcing General Availability of the Google APIs Client Library for .NET.



This library is an open-source effort, hosted at NuGet, that lets developers building on the Microsoft® .NET Framework to integrate their desktop or Windows Phone applications with Google’s services. It handles OAuth 2.0 integration, streaming uploads and downloads of media, and batching requests. For more than fifty Google APIs, it is the easiest way to get access for any Windows developer. Whether you are plugging Google Calendar into your .NET Framework-based application, translating text in a Windows Phone app or writing a PowerShell script to start Google Compute Engine instances, the Google APIs Client Library for .NET can save you tons of time.



Want to try it out? Visit the Getting Started tutorial. Want to hear more about about using Google’s services from .NET? Follow the Google APIs Client Library for .NET blog here.



Posted by Dan Ciruli, Product Manager

Google Compute Engine enables you to rapidly bring up a fleet of virtual machines. Once this virtual datacenter is up, you want to efficiently install, configure, and execute the software that runs your services. As your service grows and your engineers develop new features, you'll regularly need to deploy changes. Installing and configuring your software manually can lead to systems that are difficult and expensive to maintain and cannot be replicated for capacity expansion, development, or testing.Read More

Google Compute Engine enables you to rapidly bring up a fleet of virtual machines. Once this virtual datacenter is up, you want to efficiently install, configure, and execute the software that runs your services. As your service grows and your engineers develop new features, you'll regularly need to deploy changes. Installing and configuring your software manually can lead to systems that are difficult and expensive to maintain and cannot be replicated for capacity expansion, development, or testing.



Over the last decade, a vibrant ecosystem of open source tools has emerged to manage the complexity of large-scale compute deployments. These tools allow you to deploy changes more rapidly, recover faster from failures, and take unused resources out of service, enabling you to keep your services' uptime high and operational costs low.



The paper Compute Engine Management with Puppet, Chef, Salt, and Ansible helps you understand how resource deployment on Google Compute Engine works, and how you can use open source software management tools to manage your compute infrastructure.



Google provides the basic building blocks for Compute Engine resource management with the Compute Engine API and the gcutil command line tool. However, these are built for resource management rather than software management. As the complexity of your compute infrastructure grows, you may benefit from tools designed for software management.



Puppet, Chef, Salt, and Ansible are configuration management tools that provide software and resource management. They are open source and support Google Compute Engine. If your organization already uses one of these tools for managing other systems, we hope to help you get started using it with Google Compute Engine. If you are not already using any of these tools, consider reading Compute Engine Management with Puppet, Chef, Salt, and Ansible as a minimal primer to begin evaluating them.



-Posted by Matt Bookman, Solutions Architect

Editor’s note: Today’s guest blog comes from Leon Campise, co-founder of PeopleFun, a Dallas-based creator of social games including Word Chums. Before launching PeopleFun, the company’s founders developed Age of Empires, a highly rated and popular strategy game. ...Read More

Editor’s note: Today’s guest blog comes from Leon Campise, co-founder of PeopleFun, a Dallas-based creator of social games including Word Chums. Before launching PeopleFun, the company’s founders developed Age of Empires, a highly rated and popular strategy game.



We like keeping our PeopleFun team small in size – we’re at eight people today. We also like staying focused on creating engaging games, like Word Chums, instead of building infrastructure. Our business model works best when we can avoid having roles like network engineers on staff at PeopleFun.



If you’re growing a start-up company it’s important to focus on your core intellectual property, and outsource everything else where it’s cost effective. Designing, building-out and managing your own software development and production system is expensive. When we launched PeopleFun in 2012, Google App Engine was our choice for a cloud-based development platform that would let us scale up to millions of users and would easily handle any kind of computing challenge we could come up with.



For many multiplayer games, the backend infrastructure is only used to maintain “game state” – that is, tasks like the level a player has reached. The platform is not doing any computing.



With App Engine, we get much more than a place to store data: We’re using the platform to let players play on multiple devices at the same time and across different mobile platforms, to manage push notifications to players, and to instantly identify which of their Facebook friends are also players so they can connect with them socially. Building these capabilities demands a great deal of flexibility and horsepower from the platform -- App Engine provides that. Plus, it delivers sub-second response times for all of these actions, which makes players happier with our games.



Word Chums currently has over 300,000 active users a month, and growing. We also have another game, MixTwo, that’s popular with puzzle-game fans, and have a planned launch of a third game this summer. To make sure App Engine could handle our anticipated growth we test-drove it for performance and cost, simulating the impact of one million users and high numbers of players per minute.



Google has an excellent reputation for scaling systems, so we weren’t too surprised that App Engine handled a heavy load of users with no loss in performance and in a way that was cost-effective for us.



It’s a relief to know that we don’t have to re-design parts of our game server every time we cross a new threshold of user volume. We’ve all experienced this on previous systems: address one set of performance issues at 100,000 users, then retool it when you get to 500,000 users, and so on. App Engine handles all of the scaling issues seamlessly so our team can focus on functionality and content.



We pride ourselves on our ability to create games that people find challenging and want to play again and again. App Engine frees time for us to focus on making the user experience even better, while handling all the heavy computing tasks of our games.



-Contributed by Leon Campise, co-founder, PeopleFun

Today, we're bringing Google Compute Engine customers two highly requested features for Persistent Disks. First is the ability to create and delete root persistent disk volumes in the same command that creates and deletes VMs. The second is an easier way to grow a Persistent Disk volume. In addition, we are increasing the maximum IOs per second that a VM can issue to a Persistent Disk volume by 50% on reads and by over 500% on writes.Read More

Today, we're bringing Google Compute Engine customers two highly requested features for Persistent Disks. First is the ability to create and delete root persistent disk volumes in the same command that creates and deletes VMs. The second is an easier way to grow a Persistent Disk volume. In addition, we are increasing the maximum IOs per second that a VM can issue to a Persistent Disk volume by 50% on reads and by over 500% on writes.



Create and delete a root Persistent Disk with an instance

Many customers appreciated the convenience of being able to create or delete a VM instance with a single API call that scratch disk provided, and we've just added that single-API simplicity for creating or deleting VM instances that use root persistent disks. Check out Starting an Instance in the API and Setting the Auto-delete State of a Root Persistent Disk for full details.



Resizing Persistent Disks

Customers want to be able to start with a conservative size and IO limit and then grow that volume as the data and IO needs increase. They also want to be able to do this without having to migrate all the data to a new volume.



As of today, that is now possible. Following the instructions here, you can now snapshot a volume and restore that snapshot to a volume with a larger size. That’s it! The larger volume automatically gets higher IO caps because we reserve more storage for that volume. IO caps are described here. Please note that you will need to extend the partition and filesystem inside the VM as described in the instructions to make use of the extra space.



This feature allows you to be more conservative with your Persistent Disk provisioning and allows you to quickly and painlessly grow the volume later if you find you need more space or IO. Try it out and let us know how it works for you.



Increasing VM Performance To Persistent Disk

Our persistent disk offering is unique because developers can get a high level of performance with low cost block storage. Persistent Disk volumes can perform up to 0.3 read IOPS and 1.5 write IOPS for each GB of allocated space. So, a 1 TB volume can do up to 300 read IOPS and 1500 write IOPS. Previously, there was an additional IOPS cap at the VM level of 2000 read IOPS and 2400 write IOPS regardless of volume size. Those caps have been removed for all VM sizes. Larger instance types with greater IO capability can now see persistent disk performance grow linearly all the way up to the maximum volume size of 10TB. For example, an 8 core VM with a 10TB volume now will be able to execute up to 3000 read IOPS or 15000 write IOPS all still at the price of $0.04/GB/month with no extra IO charges.



-Posted By Jay Judkowitz, Product Manager

Google’s global Cloud Platform Developer Roadshow is coming to a city near you. As many of you know, on March 25, we will be making major product announcements at Google Cloud Platform Live ...Read More

Google’s global Cloud Platform Developer Roadshow is coming to a city near you. As many of you know, on March 25, we will be making major product announcements at Google Cloud Platform Live. The Roadshow will kick off immediately following this event and will visit 26 cities around the world. If you’d like to attend, register here.



In the roadshow, we will be talking about new approaches to computing that enable you to move beyond traditional divisions of PaaS and IaaS. We will also show how we are creating a developer experience that enables you to work more efficiently as you build, test, and deploy your code.



This is a great opportunity to see behind the scenes of the world's biggest cloud and engage with the international Google Cloud Platform team.



The Roadshow will be visiting Europe, Asia, and North America. We hope you can join us.



-Posted by Greg DeMichillie, Director of Product Management

It would be a strong statement to say that lipstick can change your life, but when it comes to supporting your data analysts who use Apache Pig, we think that ...Read More

It would be a strong statement to say that lipstick can change your life, but when it comes to supporting your data analysts who use Apache Pig, we think that Netflix Lipstick can make a big difference.



For some of us, when we see sights like this:



we get a little rush of adrenaline and know we are in the zone to debug, analyze, and optimize.



But for most, a little graphical visualization is in order. Lipstick makes life easier for all of us to understand our Pig data flows, recognize what's inefficient, and fix what is just plain incorrect.



Get a high-level look at the sequence of Hadoop jobs executing as a result of your Pig script. Watch the flow of data in real-time:



Pop-up a sampling of the output from one stage of the pipeline:



In doing so, data analysts are able to quickly observe mistakes and inefficiencies in their Pig jobs. A common observation is that data rows or columns are filtered much later than needed. Eliminating those data elements earlier produces more efficient Pig jobs. This reduces time to completion as well as cost.



I first saw Lipstick at a Netflix OSS Meetup and thought it was a great tool to increase data analyst and software engineering productivity.



If you are running Pig jobs on Google Compute Engine, we've got instructions to help you run Netflix Lipstick on Google Compute Engine. If you are not yet using Hadoop on Google Compute Engine, we have resources to help you get started.



-Posted by Matt Bookman, Solutions Architect

As part of our continued investment in Google Cloud Platform, we have worked hard to build a great community of partners. These organizations provide everything from hands-on deployment and technical support to customized application development. Since its inception, we have welcomed 161 partners into the ...Read More

As part of our continued investment in Google Cloud Platform, we have worked hard to build a great community of partners. These organizations provide everything from hands-on deployment and technical support to customized application development. Since its inception, we have welcomed 161 partners into the Partner Program.



Today, we are expanding our Partner Program in order to recognize our top service and technology partners and provide the means for any company to qualify as a Registered Company. The Partner Program’s three tiers are highlighted below:

• Premier Partner - all the benefits of the core partner program and access to premier level services.


• Authorized Partner - core partner program with branding, relationship management and access to online resources and training.


• Registered Company - entry level status with access to online resources and training.

These tiers are the first steps of many to further develop our partner community so we can provide the best possible experiences for everyone out there while working hand in hand with those companies that make it possible.



If you are ready to join our community and help bring Google Cloud Platform to everyone please learn more and apply here.



If you’re looking for technology or services partners with experience with Google Cloud Platform, check out our partner directory where you can find partners who can help you move to the cloud.



-Posted by Mark Hodgson, Head of Global Partner Programs

Today's guest post is from Felipe Rubim, Head of Technology at CI&T. A Top Google Cloud Platform Partner, CI&T builds scalable predictive applications and custom software for fortune 500 companies. As far as PHP and Drupal applications, CI&T has built and maintains more than 400 Drupal enterprise websites globally with an army of 300+ Drupal developers. In this post, Felipe introduces a white paper detailing a test CI&T conducted on Google Cloud Platform and the potential for Drupal Enterprise apps running on Google App Engine PHP. The link to the complete methodology and findings can be found here. ...Read More

Today's guest post is from Felipe Rubim, Head of Technology at CI&T. A Top Google Cloud Platform Partner, CI&T builds scalable predictive applications and custom software for fortune 500 companies. As far as PHP and Drupal applications, CI&T has built and maintains more than 400 Drupal enterprise websites globally with an army of 300+ Drupal developers. In this post, Felipe introduces a white paper detailing a test CI&T conducted on Google Cloud Platform and the potential for Drupal Enterprise apps running on Google App Engine PHP. The link to the complete methodology and findings can be found here.



Is Google App Engine ready for Drupal and if it is, can it be used for enterprise class applications?



Based on our experience working with Drupal and Google App Engine, we felt confident that the combination of the two would have the potential to cause a major shift in the content management portfolio of enterprises. Our goal was to test to see if App Engine was indeed ready to run Drupal for world class Enterprises. First, we tested features frequently requested by our customers, especially in the digital marketing arena: consumer-facing websites, portals for sales teams, doctors and healthcare professionals and intranets.



Methodology:

We evaluated the last three major releases of Drupal, and determined that Drupal 7 was optimally suited for use with App Engine and suitable for enterprise workloads. Drupal 8 was assessed as promising but at present not sufficiently mature, and Drupal 6 deployments on App Engine were not recommended as detailed in the white paper.



Findings:

• Scalability - We determined that Drupal websites running on top of App Engine scale beautifully. This is especially useful for digital marketing campaigns.


• Security - We found App Engine to be in good shape for web portals handling sensitive data.


• Versioning - We determined having different versions of your site without going through extra infrastructure set up to be a big game changer.


• Drupal Multisite - We concluded this was easy to set up and that it works as expected.


• Drush - We found that in order to make Drush work with App Engine, some tweaks are needed when connecting to CloudSQL from another set up (e.g. Google Compute Engine - GCE).


• Search - We concluded that installing and configuring Apache Solr in a GCE instance is the way to go for implementing enterprise Drupal search. We created a tutorial for that.


• SSO - We also tested SSO with Google Apps and provided a code snippet/tutorial

Is Google App Engine ready for Drupal and if it is, can it be used for enterprise class applications?



Our tests showed that App Engine PHP is a powerful alternative for Drupal deployments and is moving fast towards being a solid platform for Drupal Enterprise class sites and portals, particularly with Drupal 7.



In our experiments we identified that components from Drupal, App Engine, GCE and Google Apps could be synergically valuable in several scenarios, like:

• Google Drive and Drupal Workflow - leveraging Drupal custom workflows and auditing/logging capabilities vastly improves Google Drive and Drupal Content/Files Workflow/authoring with automatic publishing of approved documents,


• App Engine Task Queue - to replace scheduled cron tasks with App Engine Task Queue, and then execute background work (export data from a Drupal user table, campaign/contest registration, etc.) and leverage App Engine interface for management, debugging and logging;


• Flexible and extensible stack with an integrated Google Compute Engine approach - The usage of Compute Engine as a complementary component of the solution allows better flexibility to add more features and integration, as well as increased automation for the site management. More details on these features and benefits described below;

We believe Enterprises can reap multiple benefits by moving their Drupal sites to Google App Engine/Google Cloud Platform. Here are the key features and benefits, higher value first:



Auto-scale: By far, the number one benefit. This is particularly critical for Drupal digital marketing/campaign web sites, where unexpected traffic can easily break ill-prepared hosting setups.



Worry-free infrastructure: No need to remove/add servers, configuration or load balance management. Big headaches averted.



Integration with Google Apps: The potential to leverage the integration with Google products family in a number of use cases, is a big plus.



Application Management: By leveraging the Google SDKs along with its simple but powerful application configuration and management dashboard.



Potential cost savings: Considering you pay only for the resources you use, this is a great benefit. This is especially true when moving from an IaaS or traditional hosting model, where resources are needed to maintain the infrastructure portfolio.



Enterprise to Enterprise: GCP/App Engine are based on Google’s infrastructure, which also host Google Products. That means all the features and measures applicable to Google Products (such as security requirements) will also apply to your Drupal Site.



For the complete study, please see the link below:

http://www.ciandt.com/us-en/card/is-google-app-engine-PHP-ready-for-drupal



-Contributed by Felipe Rubim, Head of Technology at CI&T

Bobby Murphy, CTO and co-founder of Snapchat, will be joining Google Senior Vice President Urs Hölzle on stage during the keynote of Google Cloud Platform Live. Bobby will be sharing his experience building one of the world’s most popular apps on Google Cloud Platform.Read More

Bobby Murphy, CTO and co-founder of Snapchat, will be joining Google Senior Vice President Urs Hölzle on stage during the keynote of Google Cloud Platform Live. Bobby will be sharing his experience building one of the world’s most popular apps on Google Cloud Platform.



Snapchat’s users share over 400 million snaps everyday.



Google Cloud Platform Live is taking place 3 weeks from today. Bobby will be joining a great line up of other speakers, including Google Senior Fellow Jeff Dean. We’ll announce new features, take a deep dive on tips, tricks and technology, and share our vision for the future of cloud computing.



We are nearly at capacity in all our locations, but for people who are still interested in attending you can request an invitation to join us in-person in San Francisco or attend a livestream of the keynote at Google New York City, Google Seattle or Google London. And for those who can’t make it in person, you can always tune in online.



-Posted by Greg DeMichillie, Director

Aggregating numbers by geo location is a powerful way to analyze your data, but not an easy task when you have millions of IP addresses to analyze. In this post, we'll check how we can we use ...Read More

Aggregating numbers by geo location is a powerful way to analyze your data, but not an easy task when you have millions of IP addresses to analyze. In this post, we'll check how we can we use Google BigQuery to quickly solve this use case using a publicly available dataset.



We take the developer community seriously and it’s a great way for us to see what your use cases are. This is where I found a very interesting question: 'user2881671' on Stack Overflow had created a way to transform IP address into geographical locations in BigQuery, and asked for help optimizing their query. We worked out an optimized solution there, and today I'm happy to present an even better solution.



For example, if you want to peek at what are the top cities contributing modifications to Wikipedia, you can run this query:

SELECT COUNT(*) c, city, countryLabel, NTH(1, latitude) lat, NTH(1, longitude) lng
FROM (
 SELECT
   INTEGER(PARSE_IP(contributor_ip)) AS clientIpNum,
   INTEGER(PARSE_IP(contributor_ip)/(256*256)) AS classB
 FROM
   [publicdata:samples.wikipedia]
 WHERE contributor_ip IS NOT NULL
   ) AS a
JOIN EACH [fh-bigquery:geocode.geolite_city_bq_b2b] AS b
ON a.classB = b.classB
WHERE a.clientIpNum BETWEEN b.startIpNum AND b.endIpNum
AND city != ''
GROUP BY city, countryLabel
ORDER BY 1 DESC

We can visualize the results on a map:





You can do the same operation with your own tables containing ipv4 IP addresses. Just take the previous query and replace [publicdata:samples.wikipedia] with your own table, and contributor_ip with the name of its column containing ipv4 addresses.



Technical details



First, I downloaded the Creative Commons licensed GeoLite City IPv4 made available by MaxMind in its .csv format. There is a newer database available too, but I didn't work with it as it's only available in binary form for now. I uploaded its 2 tables into BigQuery: blocks and locations.



To get better performance later, some processing was needed: For each rule I extracted into a new column its class B prefix (192.168.x.x) and generated duplicate rules for segments that spanned more than one B class. I also joined both original tables, to skip that step when processing data. In the StackOverflow question 'user2881671' went even further, generating additional rules for segments without a location mapping (cleverly using the LAG() window function), but I skipped that step here (so addresses without a location will be skipped rather than counted). In total, only 32,702 new rows were needed.



The final query JOINs the class B prefix from your IP addresses with the lookup table, to prevent the performance hit of doing a full cross join



You can find the new table with the BigQuery web UI, or using the REST based API to integrate these queries and dataset with your own software.



To get started with BigQuery, you can visit our check out our site and the "What is BigQuery" introduction. You can post questions and get quick answers about BigQuery usage and development on Stack Overflow. Follow the latest BigQuery news at www.reddit.com/r/bigquery. We love your feedback and comments. Join the discussion on +Google Cloud Platform using the hashtag #BigQuery.



-Posted by Felipe Hoffa, Developer Programs Engineer



This post includes GeoLite data created by MaxMind, available from http://www.maxmind.com, distributed under the Creative Commons Attribution-ShareAlike 3.0 Unported License.