cloud-test: High Replication Datastore: A Solid Choice for Enterprise-class Applications

Google Cloud Platform Blog

High Replication Datastore: A Solid Choice for Enterprise-class Applications

Tuesday, March 29, 2011

WebFilings delivers a revolutionary cloud-based application that enables companies to collaboratively build and file financial reports with the SEC.
Built on the Google App Engine platform, companies of all sizes use WebFilings software to take control of their financial reporting processes. WebFilings greatly reduces the time and stress required to file financial reports with the SEC.
Enterprise Requirements

As an enterprise class application supporting many Fortune 100 customers with strict financial reporting requirements, we simply cannot afford to have maintenance outages or service disruptions. Google App Engine has proven to be a robust platform for us to build on and has allowed us to scale seamlessly to support a rapid increase in customer demand. We’re pleased to report that a recent innovation by Google has allowed us to achieve an even higher level of availability and performance.

This recent innovation is the High Replication Datastore or HR Datastore. HR Datastore allows us keep our application live even during system upgrades and other maintenance operations that Google may be performing. In addition, it delivers a new level of reliability that has given our application significantly more consistent response times and 100% availability since making the move.

Decisions... Decisions...

We were originally a bit hesitant to move to HR Datastore because it was a relatively new offering from the App Engine team. Even though the underlying technology has been around for a long time, we weren’t sure initially how the eventual consistency model might affect our application. Well, we did our homework and read the Google posts on HR Datastore, followed the App Engine news groups, pinged Googlers via the App Engine Office Hours Chat on IRC, and did some prototyping with entity groups and eventually consistent queries. From this we got a better handle on how entity groups worked to enforce strong consistency with ancestor queries and also learned of the potential performance issues of doing large parallel db.get() operations across multiple entity groups. We also reviewed our code, and made a few changes to our application where we needed to use entity groups to have strong consistency. In the end, we found the move to be much less work than we had anticipated.

The Migration

The actual migration to HR Datastore was a fairly simple process. Essentially, we just set up a a new appspot instance, and selected the High Replication Datastore option during creation. Then we migrated our data to the new instance.

During our migration process we:

Created our new instance and deployed a new version of code to work with the new entity groups
Scheduled our maintenance window with our customers
Put our current master/slave instance into read-only mode
Copied the data from our master/slave instance to our HR Datastore instance. (Due to the size of our data, this was accomplished with some help from the App Engine team)
Ran a migration mapreduce to iterate over a few entities that now needed to be in entity groups. These were saved with a new parent making them part of the entity group. This enables queries to be consistent.
Implemented a site alias that routes customers to the new instance, webfilings-hrd.appspot.com, from webfilngs.appspot.com. Note: This feature is not generally available so you should contact the App Engine team directly if you need similar functionality.

In short, we went through a complete release and testing cycle and got our application migrated to HR Datastore in under a month. The actual migration happened overnight when we moved a litte over 200GB of data to our new HR Datastore instance, executed our migration mapreduce, and did our verification testing in about five hours. Since moving to HR Datastore, we are wondering why we ever even thought twice. Our customers and our Customer Success Team that works with customers every day have given us feedback that the application is performing better than ever and we have had no downtime since the move.

A Solid Choice

The High Replication Datastore was designed with the needs of large enterprises in mind. HR Datastore provides maximum uptime, multiple datacenter redundancy, consistent response times, along with scalability and performance that has met the needs of our fast growing company. It does take some planning to make the move, however. You should be prepared to review your existing code and test thoroughly beforehand. We found that this effort alone had a significant payback in terms of overall application performance and reliability. Even though the average latency on HR Datastore calls can sometimes be a little bit longer, we were able to optimize and end up with an overall experience that was better for the end user.

If you have an application that has these same demands then my only question for you is “When are you moving to HR Datastore?”. If you are like us, you will be very glad you took advantage of this latest innovation from the Google team.
Posted by Dave Tucker

Director of Product Development

WebFilings (www.webfilings.com)