cloud-test: October 2015

Camio offers AI-based, real-time video monitoring that learns user preferences using Google Cloud Platform

Friday, October 30, 2015

Today’s guest post comes from Carter Maslan, CEO of Camio, a startup that converts real-time video into useful information. barking and pillow-fluffingGoogle App EngineGoogle Compute EngineGoogle Cloud DatastoreGoogle Cloud StorageGoogle Cloud Platform for StartupsGreylock PartnersGoogle BigQuery,Google Cloud MonitoringGoogle Cloud BigtablePosted by Carter Maslan, CEO, Camio

Try the new cloud management console

Thursday, October 29, 2015

Improved Navigation:

Global navigation: We’ve re-architected the navigation so that you can access features in several different ways. Open the menu to see all Cloud Platform offerings in one consolidated place or use the keyboard shortcut to quickly enter into search based navigation.

Local navigation: We’ve streamlined the navigation layout of each feature to allow complete focus within that space and maximized all screen real estate.

Search: Newly-added navigational search allows you to jump to any feature, page or API in the console from a persistent search box.

Customization: Focus the console to reflect only the services you actually use by “pinning” them to the top bar for fast one-click access from anywhere.

Increased visibility: Identify and address issues from a configurable dashboard of your resources and services.

console.cloud.google.comTry the beta console

Posted by Stewart Fife, Product Manager, Google Cloud Platform

Create metrics, alerts and dashboards based on your Google Cloud logs

Wednesday, October 28, 2015

Google Cloud LoggingGoogle Cloud Monitoring

Logs-based Metrics let you create a metric from a logs filter expression that you build in Cloud Logging. You can use these metrics to build dashboards and create alerting policies in Cloud Monitoring.

Advanced Logs Filters provide a rich filter syntax to select log entries and create logs-based metrics.

Charts to Logs allow you to quickly go from charts to the corresponding logs, where you can investigate and resolve issues.

Logs-based Metricsself.handle_exception.

Metrics

Create alert from metric

“We have been very pleased with the Logs-based Metrics solution. We were sitting on a lot of valuable log data but had been missing insights and there was no good way to gather metrics from them. We experimented with some open source solutions, but it's a lot of work to manage it for the amount of logs we produce. We like that Cloud Logging is a fully managed service with great performance, scalability and feature set. We can now add new metrics on a running production system which has helped us diagnose issues and monitor new signals." - Arjen van der Ende, Engineer at Q42Advanced Logs Filters

Charts to Logs

cloud-logging-feedback@google.comPosted by Deepak Tiwari, Product Manager, Google Cloud Platform

Add backend logic to real-time data with Firebase and Google App Engine

Tuesday, October 27, 2015

Use Firebase and Google App Engine in an Android App

Firebase — a platform for building mobile apps, offering realtime data storage and synchronization, user authentication, and more.

Android Studio — an Android development environment based on IntelliJ IDEA.

Cloud Tools for Android Studio — a set of tools included with Android Studio, that integrate with Google Cloud Platform services.

Google App Engine — an environment for running application code in the cloud; Platform as a service (PaaS).

Posted by Benjamin Wulfe, Firebase

Automating configuration management with Google Cloud Deployment Manager and Puppet

Monday, October 26, 2015

Google Cloud Deployment ManagerPuppetsigning up a free trial
Install and configure the Google Cloud SDKinstall and configure the Google Cloud SDK
Set up a Puppet masteropen Click-to-Deploy Puppet in the Developers Console
Create a Deployment Manager configuration fileweb-server.yaml
resources:
- name: managed-by-puppet
type: compute.v1.instance
properties:
   zone: us-central1-f
   machineType: https://www.googleapis.com/compute/v1/projects/<your_project>/zones/us-central1-f/machineTypes/f1-micro
   disks:
   - deviceName: boot
     type: PERSISTENT
     boot: true
     autoDelete: true
     initializeParams:
       sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-8-jessie-v20150818
   networkInterfaces:
   - network: https://www.googleapis.com/compute/v1/projects/<your_project>/global/networks/default
     accessConfigs:
     - name: External NAT
       type: ONE_TO_ONE_NAT
   tags:
     items:
       - http-server
   metadata:
     items:
     - key: startup-script
       value: |
         #!/bin/bash
         apt-get install -y puppet

         cat <<EOF >> /etc/puppet/puppet.conf

         [agent]
         server = <puppet_master_instance_name>
         EOF

         systemctl enable puppet          systemctl restart puppet

When you run Deployment Manager using this template, Deployment Manager will create a new Debian-based f1-micro instance, then run the startup script defined by the startup-script attribute in the metadata section of the template. Here, the startup script is configured to perform the following actions:

Install Puppet

Edit the Puppet configuration file to point at your Puppet master instance

Connect your instance to the Puppet master instance as a Puppet node

Create and connect a new instance$ gcloud deployment-manager deployments create managed-by-puppet --config web-server.yamlf1-micromanaged-by-puppet
Approve the pending certificate request

Navigate to the VM Instances page in your Developers Console.

Click the SSH button next to your Puppet master instance to connect to your instance via a browser-based SSH terminal.

In the SSH terminal, run the following command to see a list of outstanding Puppet certificate requests:

$ sudo puppet cert list

The output should be similar to the following, with only one request listed: [evan@puppet1-puppet-master c2d]# puppet cert list "managed-by-puppet.c..internal" (SHA256) 4D:B3:C2:33:38:

Sign that request by running the following command, replacing (your_project) with your project name:

$ sudo puppet cert sign                                                managed-by-puppet.c.(your_project).internal

Delete your deployment and Puppet master

Navigate to the Deployments page in your Developers Console:

Click the trash can icon next to the deployment.

Navigate to the VM Instances page in your Developers Console.

Check the box next to your Puppet master instance.

Click the Delete button at the top of the page.

Posted by Evan Brown (@evandbrown), Solutions Architect

Digital agency POSSIBLE London creates Heineken Rugby Studio using Google Cloud Platform to include fans in the action

Thursday, October 22, 2015

Today’s guest post comes from Mandhir Gidda, CTO UK at POSSIBLE London, a global digital agency that brings results-driven campaigns to some of the world’s most dynamic brands. In this post, Mandhir describes how POSSIBLE London, in partnership with Heineken, built the Heineken Rugby Studio as a go-to destination for content related to the Rugby World Cup in London, which runs September 18 to October 31. Rugby World CupRugby Studio

Google App EngineFirebaseGoogle Cloud PlatformPosted by Mandhir Gidda, CTO UK, POSSIBLE London

Speed with Ease – NoSQL on the Google Cloud Platform

Wednesday, October 21, 2015

Today’s guest bloggers are Lynn Langit – Independent Cloud and Big Data Architect named the Google Cloud Developer Expert for the past three years and recognized as an AWS Community Hero, Microsoft MVP (SQL Server), MongoDB Master and ACM Distinguished speaker – and David Haley, startup engineer and former Google Cloud team member who was the first engineer on the Google Cloud Platform Americas sales team.
Validating 4 Million TPS using AerospikeDB with only 20 GCE nodesDecember 2014 benchmarkAerospikeleveraging SSDsfour times the performance at half the price was measuredkey-value store
New Benchmark Validated – TL; DR – 6 Simple Steps to Speed
About Aerospike Scaling:

Aerospike handles the equivalent of sharding automatically.

There’s no downtime. You can add new capacity (data volume and throughput) dynamically. Simply add additional nodes and Aerospike will automatically rebalance the data and traffic.

You set the replication factor to 2 or more and configure Aerospike to operate with synchronous replication for immediate consistency and durability.

If a server goes down, Aerospike handles failover transparently; the Aerospike client makes sure that your application can read or write replicated data automatically.

You can run purely in RAM, or take advantage of Flash/SSDs for storage. Using Flash/SSDs results in a slight penalty on latency (~0.1ms longer on average), but allows you to scale with significantly smaller clusters and at a fraction of the cost.

Benchmark Testing on GCE – TL; DR – look ma’ it’s fast
Setup to Test Aerospike on Compute Engine
Step OneSETUP Compute Engine Environmenthttps://cloud.google.com/https://cloud.google.com/sdk/

NoteTo replicate the 4 million TPS benchmark, you’ll need to enable project billing and then request an increase in the default quota limits for a GCP project, as you’ll need to increase the default CPUs/project limit from 24 to 160.

To do this, from within your new project in the GCP console > Compute > Compute Engine > Quotas and then click on the "Change Request" link at the top of the page. Fill out the form and click send. It generally takes around 24 hours for this type of request to be processed.

Step Two

SETUP Compute Engine instances for the Aerospike server clusterAerospike Java client

LOGIN to your Google Cloud accountSETUP variable values as for your test.your-project-name

CREATE 20 GCE instances

Step Three UPDATE the Aerospike config files

Step Four

SETUP Compute Engine instances for the Aerospike client VMs. To perform the test at the scale for benchmark validation, I created 20 client GCE VMs. For the benchmark work, we chose machine types of n1-highcpu-8 because the benchmark tool workload uses 256 threads in order to appropriately saturate the server nodes. I used the same custom image with Aerospike pre-installed to create the clients as I used when creating Aerospike server nodes and identified these nodes via the tag value “benchmark-client.”

Step Five

CREATE AEROSPIKE CLUSTER by booting the servers: Start the asd daemon on each server using the aerospike.config file. You run lines 64-74 to complete this task, also shown below.

Step Six

Start the AMC (GUI) Console: Although the Aerospike benchmark tool does generate text-based (log) output, I find that "seeing" the power via the Aerospike Management Console (GUI) is a fun part of testing. A portion of a sample AMC console is shown above.

To connect to the AMC console, you’ll need to find the public IP of "as-server-1" and then open a browser using http://(public_IP_of_server-1):8081. Next you’ll enter the private (internal) IP in the dialog box that opens on top of the AMC as follows: http://(internal_IP_of_server-1):3000.

Run Tests using the Benchmark Tools

The Java client for Aerospike includes the Aerospike benchmark testing tools. Because we installed the Java client on our base image, all client instances include the benchmark tools – we simply need to “turn on the tool” to perform a load test. There are a number of parameters that you can use with the benchmark tool (detailed at the end of this section). To perform the test, I set the load parameters to 1 Million Keys and 256 client threads. Run the test. First l generated the load for the insert benchmark, and then I ran the test with a different load type.

Shown below is the first result in the AMC – it’s actually considerably MORE than previous 1 Million TPS – it’s 4 Million! Of note is that only 50 GB (of the total available 350 GB) of RAM available to the cluster is being used for this workload. This first test uses 100% reads for the workload.

For this next test, I ran the benchmark tool with the configuration set to 100% WRITES. This test resulted in a 1 Million TPS as shown in the AMC console image below. Again, of interest, is that only 50 GB (of the total 350 GB) or RAM is being used to run this workload.

Benchmark Tool Script Details

The host in the “-h” option of the benchmark tool must be used to test against the Aerospike host node. Benchmark options used in the current tests were:

10 million keys (-k 10000000)

50 byte string values for each key (-o S: 50) – 150 byte into 3 bins + key -> 200 byte objects

3 bins or columns each of size 50 and insert key size of 20 bytes (-b 3 -l 20)

Two different loads:

100% write (-w I)

100% read (-w RU, 100)

Before every test run, data was populated using the insert workload option (-w I)

For each client instance, 256 threads give the maximum throughput in the in-memory database case (-z 256).

For more about benchmark flags, use benchmarks -help

Clean Up

When you’ve completed your testing, you’ll want to stop the test. You may also want to stop the Aerospike servers, delete disks and shutdown all VM instances. Run the referenced script from line 130 to the end (line 165) to stop the benchmark tests, stop all of the servers, delete all disks and shutdown all instances.

Conclusions and Next Steps

I found the combination of Aerospike and GCP to be a winner for these reasons:

Powerful, scalable NoSQL Database – I was able to set up and test with ease

Quick setup and low cost to test – for this configuration the cost looks to be around 4k/month – this is for up to 4 million reads per SECOND, many customers won’t need this much power, so running fewer servers would reduce costs

Flexibility of cloud platform – scaling up or down is trivial given the sharding of Aerospike and the speed of working with Compute Engine VMs

speed with (n) number of servers, to further simplify management and thereby reduce overall cost?" Also, I could alter the test parameters to match whatever type(s) of real-world workloads I need. For example, I may test a common 80% read, 20% write mixed workload as client requests dictate in the future.

Note: On Compute Engine Machine Types. I chose the n1-standard-8 series of instances, as these are “middle-of-the-road” instance types and provide a good instance-to-value ratio. Of course, Compute Engine offers a variety of instance sizes and types – a grid summarizing choices is shown below.