More members of the App Engine team are hitting the road in the next twoweeks. If you'll be at any of the following events we'd love to meet and talkshop! Here's where you can find us:
Wednesday March 18th 5:00 pm - 6:00 pm New York City
Pete Koomen will be at title="CommunityOne East" href="http://developers.sun.com/events/communityone/2009/east/">CommunityOne East participating in the Closing Panel Discussion – "Enterprises andthe Cloud". The panel will be moderated by David Berlind, Editor-At-Large ofInformationWeek, and the panelists include Adam Gross, Vice President ofDeveloper Marketing, Salesforce and Lew Tucker, Vice President and CTO, CloudComputing, Sun Microsystems, Inc.
Wednesday March 25th 9:00 am - 12:20 pm Chicago
Joe Gregorio will beteaching a half-day tutorial at PyCon 2009 entitled href="http://us.pycon.org/2009/tutorials/schedule/1AM6/">An Introduction to Google App Engine. From the session description:"Google App Engine allows you to build scalable web applications and host themon Google hardware. This hands on tutorial will walk you through the steps ofbuilding a basic web application, from setting up the SDK through to using themajor APIs that App Engine provides."
Thursday March 26th 1:30 pm - 2:30 pm Philadelphia
Joe Gregorio will be giving thepresentation "Under the Covers of the App Engine Datastore" on March 26that the Emerging Technologies for the Enterprise conference in Philadephia. From the session description: "This talk walks through theinternals of the App Engine Datastore, how data is stored, how indexes arebuilt, and how queries are executed. We then discuss how thosearchitectural decisions make the App Engine datastore scalable and finallygo into how those constraints affect how you design your applicationsschema so your application can take full advantage of thatscalability."
Wednesday April 1st Time TBD New York City
For the second time in two weeks we return to New York Citywhere Joe Gregorio will be giving the same "Under the Covers of the AppEngine Datastore" talk at the Cloud Computing Expo in New York City.
Posted by Joe Gregorio, App Engine Team
A number of exciting App Engine-related snippets have made the rounds over the past couple of weeks. Here's a quick summary:
Jaiku moves to App Engine
Last April, Jaiku's official blog confirmed that Jaiku was being ported to App Engine. We are pleased to announce that not only is Jaiku now hosted on App Engine, but substantial progress is being made on the related open source effort to build out the Jaiku-based microblogging platform, Jaiku Engine, which allows organizations, groups, and invidivuals to deploy their own microblogging service to App Engine. For more details, check out the official Jaiku blog posting.
Trender apps launched on App Engine
Jaiku isn't the only application to launch on App Engine in recent weeks. Enter NYT Trender and Flickr Trends which allow users to view the popularity of search terms over time based on occurrence in the New York Times and Flickr respectively. Both mashups display relevant photos for the terms entered, adding to the richness of the experience.
New remote_api article available
Googler Nick Johnson recently published a handy guide introducing the remote_api module which debuted with release 1.1.9 of the App Engine SDK. The module allows remote access to an application's datastore and Nick's guide describes how to set up and use it with plenty of sample code to help you build your own interactive console and map framework. Also, in case you missed them when they were first posted, the articles on paging and avoiding datastore contention using sharded counters are great reads as well, particularly if you're interested in building a highly scalable and efficient apps.
App Engine at WeekendApps
Over the weekend of Feb. 20th, 130 developers descended on the Googleplex in order to design, build, and launch an OpenSocial-based application as part of the WeekendApps - OpenSocial event. While OpenSocial was the star of the show, App Engine garnered a lot of interest itself with BuddyPoke! creator Dave Westwood and App Engine advocate Fred Sauer in attendance. In the end, over 14 new applications were presented, at least 3 of which were built on App Engine. Of these three, the team behind Buddy Quiz! won a Google I/O pass for "best product vision" and another, Yumie Date, recently received an Editor's Pick promotion from MySpace. Neither team had ever used App Engine prior to the event and one had never even worked with Python.
Posted by Jason Cooper, Google App Engine Team
Several of the App Engine team members are excited to be heading to Austin, Texas for SXSW Interactive 2009. SXSW always brings together an talented variety of web developers, entrepreneurs, and visionaries. If you'll be there too, we'd love to meet and talk shop!If you are interested in participating in an App Engine User Study while at SXSW, please fill out this form.
Sunday, March 14th10:00 pm - 1:00 amThe App Engine team is joining forces with the Blogger and Reader teams for a bash "Read. Write. Drink." at Six Lounge (117 W 4th St @ Colorado). Come share a beer and tales of website building and hosting with us! Bring your SXSW Interactive Badge, or find an App Engine team member at the conference and ask for an invitation. Before the party, Engineers from the App Engine team, including Tech Lead, Kevin Gibbs, are having "office hours" from 8-10pm at Halcyon Coffeeshop (218 W 4th St) where we'll hang out, sip coffee and talk shop - we want to answer your tough technical questions and to help you learn if App Engine is a good fit for your application or business. We'll have some T-shirts, and copies of our new cheat sheet!
Tuesday March 17th10:00 - 11:00 am, Room 8Kevin Gibbs, App Engine's Tech Lead, will be participating in the panel "Cloud Computing: Defending the Undefinable" along with Yousef Khalidi, Distinguished Engineer at Microsoft and Werner Vogels, the CTO at Amazon.com. Here's the panel description:"The brave new world of cloud computing is radically changing how we build web applications. What is a platform, what is a service, and how will the future of web applications be built? More importantly, how do these various clouds compare, and what do the differences mean? Are they ready for your world-rockin' startup? In this panel, we'll get nerdy with technical details, you'll yell at us, and we'll argue why your app should already be in the cloud."
Monday March 16th9th Annual 20x2 Event7:00 - 9:30 PM, The Parish on 6th St in Austin20x2 brings together 20 different participants from all walks of creative life and each person has two minutes to answer/interpret the same question before a live audience. The question for this show is "What's It Gonna Take?" App Engine team member Lindsey Simon will be speaking.
If you're in the area, stop by! If not, you can always leave feedback in our Google Group!
We're psyched to announce that developers can now purchase additional computing resources on App Engine, enabling apps to scale beyond our free quotas. This has been our most requested improvement to App Engine and we're thrilled to deliver it, as promised.
When we released App Engine last April, it was completely free to use, but each application was limited to a set of fixed resource usage quotas. The free quotas are still there, but now you can grow beyond them.
You can now set a daily budget for your app that represents the maximum amount you're willing to pay for computing resources each day. You allocate this budget across CPU, bandwidth, storage, and email, and you pay for only what your app consumes beyond the free thresholds -- prorated up to the nearest penny. We've put together a handy screencast to help explain the process:
App Engine remains free to get started. However, along with many performance improvements over the past ten months, we've learned that we overestimated our initial free quota values. Therefore, in 90 days we will be reducing the free quota resources. We believe these new levels will continue to support a reasonably efficient application serving around 5 million page views per month, completely free.
The pricing for resources beyond those free quotas is:
Data stored in the datastore incurs additional overhead, depending on the number of indexes, as well as the number (and size) of associated properties. This overhead can be significant in some cases, and it's something that we have been underreporting up until now. Thus, you may notice an increase in the amount of data stored by your application that is listed in the Admin Console.
To decrease the impact of this change in the way we calculate storage usage, we've doubled the free storage quota to 1GB. To learn more about how our quotas work, upcoming changes to our quota levels, how they relate to billing, and overall resource consumption limitations, see the Quotas section of the docs. The Purchasing Additional Quota section has more detail about how to purchase additional computational resources for your app, and there's also a Billing FAQ. We've also made some changes to our terms of service to include language around payments, fees, and disallowing the use of multiple applications to avoid incurring fees.
As always, we welcome feedback in our discussion group, and don't forget to check out the App Engine sessions at our upcoming developer conference, Google I/O!
Building a massive, distributed datastore which can service requests at an extremely high throughput is something that we've focused on at Google. We created something called Bigtable that underlies the datastore in App Engine. The design for Bigtable focused on scalability across a distributed system so it may operate a bit differently than databases you've worked with before, such as not supporting joins. This isn't an accident -- when you build a system that can scale to the size that Bigtable can there's no way to do a general purpose join on data sets that size and still have them be performant.
Google isn't alone in offering an non-Relational datastore to enable scaling. For example, Amazon has SimpleDB:
A traditional, clustered relational database requires a sizable upfront capital outlay, is complex to design, and often requires a DBA to maintain and administer. Amazon SimpleDB is dramatically simpler, requiring no schema, automatically indexing your data and providing a simple API for storage and access.
There are also a range of non-relational open source datastores now available such as CouchDB and Hypertable. Those are just two examples, there are many more.
While you might think this is all new, it's actually a bit of a return to the past. You see, there was a time when "RDBMS" wasn't always the answer regardless of what the question was. At the time Codd published his paper, "A Relational Model of Data for Large Shared Data Banks," there were many different approaches to datastores. It was only in the '80s that relational databases won the majority of the mindshare. Having settled on a single metaphor the industry has developed many tools and techniques to make developing on a relational database easier.
Unfortunately that majority mindshare is also a problem because while RDBMS' are useful in many situations, they are not useful in all situations. Their dominance in the mindshare means that useful alternatives aren't used, and huge amounts of time and money can be wasted trying to force non-relational problems into a relational model.
We are in the middle of a renaissance in data storage with the application of many new ideas and techniques; there's huge potential for breaking out of thinking about data storage in just one way. Michael Stonebraker pointed out in his paper, "One Size Fits All": An Idea Whose Time Has Come and Gone, that there are common datastore use cases, such as Data Warehousing and Stream Processing that are not well served by a general purpose RDBMS and that abandoning the general purpose RDBMS can give you a performance increase of one or two orders of magnitude.
It's an exciting time, and the takeaway here isn't to abandon the relational database, which is a very mature technology that works great in its domain, but instead to be willing to look outside the RDBMS box when looking for storage solutions.
Michael Bernstein is doing something I find very interesting. Not only is he creating applications on App Engine, he creates a new one every Wednesday. His site, appropriately named Web App Wednesday, is where he is creating a new web application every Wednesday, starting first with the site itself, followed by F/OSS License Info, a single-page site that displays information about the most common Open Source software licenses. Another recent app, LinQR , is a URL shortening service and QR code generator. Good luck to Michael and we look forward to seeing what he comes up with next Wednesday.
If you're doing something cool with App Engine please let us know! We'd love to highlight it here on the blog or even have you write up an article explaining what you've done, like this new article from Omar Abdelwahed on Creating a Facebook App with Google App Engine and Best Buy Remix.
Speaking of articles, Aral Balkan has written an article on his backup and restore utility and our own Joe Gregorio published an article on a design for popularity ranking using voting and time-based decay.
We're very excited today to announce that we've raised limits on several App Engine operations:
These changes were inspired and driven by a great deal of developer feedback, and we're not done! Please let us know what you'd like to see next on App Engine in our Google Group!
Use promo code NEXT1720 to save $300 off general admission
Follow