cloud-test: October 2011

ProdEagle - Analyzing your App Engine apps in real-time

Thursday, October 27, 2011

Today’s post comes to us from Andrin von Rechenberg of MiuMeet who has developed an easy to use analysis framework, ProdEagle, for App Engine apps. ProdEagle enables you to easily count and visualize events to help better understand both performance and usage of your site.

ProdEagle allows you to monitor your system, lets you analyse in real-time what is going on and can alert you if something goes wrong.

The story

We are a small start-up that created an incredibly fast growing social dating platform called MiuMeet. We used App Engine from the start and within 6 months we had over 1 million registered users - the system scaled beautifully. One of the world’s leading online dating companies invested in our start-up and gave us a very good piece of advice: Before you build new features, you need to understand what’s actually going on in your system. Of course we thought we knew exactly what is going on in our system. We had the App Engine Dashboard and we had Google Analytics and we ran a couple of daily MapReduces to collect some statistics. But honestly, compared to today, we didn’t have any idea what was going on.

Understanding your system

From our MapReduces we knew that most users who accessed our dating platform used our Android app and only very few users used our iPhone app. We weren’t exactly sure what the reason for that was, but we wanted to improve the experience for our iPhone users. So we had to start measuring what was different for iPhone vs Android users.

But how do you do this? You start recording everything that happens in your system. And by “everything” we don’t mean the status code of an HTTP response but human understandable events, like “an Android user starts a conversation”, “the average length of a conversation”, “how often iPhone users log in” and so on. We realized that iPhone users are much less active and reply less often - not because they don’t want to, but because they don’t know immediately that they got a new message. Android users are informed by Push-Notifications, iPhone users via email. We did not expect that this would make such a big difference, but now it’s pretty obvious what the next feature we should build to improve the iPhone experience. Our conclusions are based on hard numbers and not guesses.

We used ProdEagle to collect and analyse the data that allowed us to realize this. ProdEagle can count, display and compare events in real-time and can even alert you if something is going wrong.

How to measure events

Let’s start with a simple Python example. We would like to record the devices from which messages are sent in our system. All we have to do is add the green lines to our sendMessage function:

import prodeagle

def sendMessage(from, to, text, device):

mailbox = getMailbox(to)

mailbox.addMessage(from, text)

prodeagle.counter.incr("Message." + device)

This one line of code and a few clicks in the ProdEagle dashboard allows us to create the following graphs:

*The numbers in these graphs are just examples.

Analysing your system in “real-time”

With ProdEagle you can count whatever you want. The advantage over traditional analytics systems is that you don’t have to wait for a day for the data to appear. Data appears within 1 minute in your dashboard. This means that you can monitor your system almost in real-time and set up alerts if something goes wrong.

In the following example we measure how long it takes to execute a datastore query:

def searchPeople(query):

timestamp = time.time()

query.execute()

prodeagle.counter.incr("Search.People.Count")

prodeagle.counter.incr("Search.People.Latency",

time.time() - timestamp)

In the ProdEagle Dashboard we can specify that “Search.People.Latency” should be divided by “Search.People.Count” and plot it as a graph. Additionally we can set up an email alert that fires if the latency was more than 4000ms for 5 minutes:

If the latency is above all the red dots you get an email alert.

Being able to measure latency in real-time is particularly useful when you are trying to optimize your system and want to try out different strategies to answer queries.

The magic behind ProdEagle

Whenever you call prodeagle.counter.incr(counter_name)the ProdEagle library increments a counter in memcache - this shouldn’t add any noticeable latency to your app, unless it is the first call made by a serving instance. Every few minutes, the ProdEagle service that we run collects all these memcache counters and persists them, creates the graphs and alerts you if something is wrong.

Obviously, the system isn’t robust if your memcache gets flushed. To address this, ProdEagle uses 1024 dummy counters to check when your memcache gets flushed and to approximate the inaccuracy of the data in your graphs. This is based on the assumption that elements with the same size in memcache are freed in a “least-recently-used” fashion and that the 1024 dummy counters hit all memcache shards. We have used ProdEagle for MiuMeet for a couple of months now and our approximated accuracy was never below 99.8%.

I like! How can I get it?

ProdEagle is currently completely free of charge. You can download the python client libraries and sign up for your dashboard on www.prodeagle.com.

App Engine SSL for Custom Domains in Testing

Wednesday, October 19, 2011

The long awaited SSL for Custom Domains is entering testing and we are now looking for trusted testers. If you are interested in signing up to test this feature, please fill in this form.

We will be offering two types of SSL service, Server Name Indication (SNI) and Virtual IP (VIP). SNI will be significantly less expensive than VIP when this service is launched, however unlike VIP it does not work in all browsers that support SSL. VIP is a premium service with a dedicated IP and full browser support. Both VIP and SNI support wildcard certificates and certificates with alternate names.

We look forward to making this widely available as soon as possible and as always we welcome your feedback in the group.
Posted by The App Engine Team

App Engine 1.5.5 SDK Release

Tuesday, October 11, 2011

2011 has seen some exciting releases for App Engine. As the days get shorter, the weather gets colder, and all that Halloween candy starts tempting everyone in the grocery store, we’ve been hard at work on our latest action packed release.

Premier Accounts
When choosing a platform for your most critical business applications, we recognize that uptime guarantees, easy management and paid support are often just as important as product features. So today we’re launching Google App Engine premier accounts. For $500 per month (not including the cost to provision internet services), you’ll receive:

Premium support (see the

Technical Support Services Guidelines for details).

A 99.95% uptime Service Level Agreement (see the draft agreement, the final agreement will be in the signed offline agreement).

The ability to create an unlimited number of apps on your premier account domain.

No minimum monthly fees per app. Pay only for the resources you use.

Monthly billing via invoice.

To sign up for a premier account, please contact our sales team at appengine_premier_requests@google.com.

Python 2.7
PIL? NumPy? Concurrent requests? Python 2.7 has it all, and today we’re opening up Python 2.7 as an experimental release. We’ve put together a list of all the known differences between the current 2.5 runtime and the new runtime.

Overall Changes
We know that bumping up against hard limits can be frustrating, and we’ve talked all year about our continued push to lift our system limits. With this release we are raising several of these:

Request Duration: The frontend request deadline has been increased from 30 seconds to 60 seconds. We’ve increased the maximum URLFetch deadline to match from 10 seconds to 60 seconds.

File limits: We’ve increased the number of files you can upload with your application from 3,000 to 10,000 files, and the file size limit has also been increased from 10MB to 32MB.

API Limits: Post payloads for URLFetches are now capped at 5MB instead of 1MB.

We’re also announcing several limited preview features and trusted tester programs:

Cloud SQL Preview: We announced last week that we are offering a preview of SQL support in App Engine. Give it a try and let us know what you think.

Full-text Search: We are looking for early trusted testers for our long anticipated Full-Text Search API. Please fill out this form if you’re interested in trying it out.

Conversion API: Ever wanted to convert from text to PDF in your App? Then consider signing up as a trusted tester for the Conversion API.

Datastore

Cross Group (XG) Transactions: For those who need transactional writes to entities in multiple entity groups (and that's everyone, right?), XG Transactions are just the thing. This feature uses two phase commit to make cross group writes atomic just like single group writes.

Platform Improvements

Experimental Google Cloud Storage Integration: As Google Cloud Storage, formerly Google Storage for Developers, graduates from labs, we are improving our integration by adding access via the files API.

Prediction API: Another of our friends graduating from labs is the Prediction API. Check out some of the App Engine examples in their getting started guide.

Of course, these are just the high level changes. This release is packed full of features and bug fixes, and as always, we welcome your feedback in the group.

Posted by The App Engine Team

Google Cloud SQL: Your database in the cloud

Thursday, October 6, 2011

Cross-posted from the Google Code BlogGoogle Cloud SQL

No maintenance or administration - we manage the database for you.

High reliability and availability - your data is replicated synchronously to multiple data centers. Machine, rack and data center failures are handled automatically to minimize end-user impact.

Familiar MySQL database environment with JDBC support (for Java-based App Engine applications) and DB-API support (for Python-based App Engine applications).

Comprehensive user interface for administering databases.

Simple and powerful integration with Google App Engine.

let us knowPosted by Navneet Joneja, Product Manager for Google Cloud SQL

Google Cloud Platform Blog

ProdEagle - Analyzing your App Engine apps in real-time

App Engine SSL for Custom Domains in Testing

App Engine 1.5.5 SDK Release

Google Cloud SQL: Your database in the cloud

Don't Miss Next '17

Free Trial

GCP Blogs

Labels

Archive

Feed

Subscribe by email

Company-wide

Products

Developers