cloud-test: Scaling SongPop to 60 million users with App Engine and Google Cloud Storage

Google Cloud Platform Blog

Scaling SongPop to 60 million users with App Engine and Google Cloud Storage

Monday, February 4, 2013

Continuing our Developer Insights series, today’s guest bloggers are Olivier Michon, CTO, and Alexis Hanicotte, software engineer, from Fresh Planet, maker of the popular mobile application SongPop. SongPop is a social application where players compete to be the fastest to guess the name of a song or artist.

SongPop is a social mobile app where players compete on who will be the fastest to guess a songs’ artist or title. It is a huge success for our very small team of just 6 engineers. We now have more than 60 million users, were the number 5 most downloaded iOS game of 2012, and went from 0 to more than 10k queries/second on our servers in less than 6 months. This has been made possible in large part because we run on Google App Engine.

App Engine allows us to quickly prototype, iterate and release our games. We’ve been accumulating experience with the platform since 2009, but really saw the power of autoscaling once SongPop became a hit.

Our experience scaling with App Engine

SongPop was released in May 2012. During our growth to our first 100,000 daily active users (DAU), our App Engine backend scaled smoothly. This allowed us to spend our time making actual improvements to the game experience, while our user growth continued at a rapid pace.

We opened a Premier Account with App Engine around the time when we reached 100,000 DAU which gave us access to live customer support. It came in handy when we encountered two downtimes, with one lasting just 10 minutes and the other for 1 hour. But we were able to promptly reach Premier Support and had Google engineers investigate these issues with us.

Once we reached 500,000 DAU, we applied a variety of optimization ideas to reduce latency. For example, we used to have user data spread over many models, but we combined them into a single entry in the Datastore to reduce read operations. We also often needed the list of a user’s opponents, so instead of querying every time, we cached this result using Memcache. It only took one engineer and just 4 days of work to reduce the latency with these optimizations.

As we reached the milestone of 1 million DAU, some Datastore queries (used to find random opponents in the game) showed high latencies and a high rate of timeouts. We had to enforce deadlines, implement better fallbacks and identified, with the help of Premier Support, that degraded performance came from the fact that our queries were relying on many different indexed properties. So the solution was simple - either add a composite index with all the properties we needed or combine into a single one.

Using App Engine + Google Cloud Storage

For each game session, our users need to download song samples in order to play. It is critical that this data gets delivered fast and reliably wherever the user is located. We chose to use Google Cloud Storage for this use case. It has proved to provide high performance content delivery, as we have been able to serve 17 terabytes/day of songs and images worldwide.

In addition to its reliability, Cloud Storage is great because of its integration with App Engine. We can easily read and write files from our application to Cloud Storage using the same syntax as we would use to write local files (using Python). We found it intuitive and convenient because you do not have to manage opaque keys to retrieve your files (just use the path you specified), and you can browse your files through a directory-like structure. Cloud Storage also allows you to manage access rights, can be used with Google BigQuery, and it is priced affordably compared to other solutions we considered.

Architecture diagram of SongPop

Long live App Engine!

When we speak to other game developers, we always recommend that they use App Engine. We’ve used other services such as EC2 from Amazon Web Services for other games before, but we’ve found App Engine to be a better service for our needs. We don’t want to spend time setting up servers and load balancing, when we could instead use that time to build great games and let our service provider handle the infrastructure for us.

When we compare the development of SongPop to stories of other apps, we’re thankful that App Engine allowed us to have only one engineer working full-time on the backend portion of our app. Even better, he was able to do additional work on adding new features to the game instead of solely focusing on infrastructure issues. With App Engine, scaling our game was easy.

Others things we want to share

Do not worry if documented resources limits and rates look too small, they are just there to make sure one app does not abuse any resource, but they can scale. Most of our limits have been increased by 18-fold! We had days where we made 230,000,000 UrlFetch API calls, for instance.

The Location headers is a really great feature, because it is easily accessible for a wide variety of use cases, such as selecting users’ opponents or building their game profile.