This post is another entry in our on-going series of guest posts from App Engine developers. Today we have the Guardian reporting in on how they're using App Engine as part of their Open Platform.
There has been a lot of debate recently about the future of news online, some people like The Times of London and New York Times think paywalls are the answer, others are not so sure. At The Guardian we have a clear alternative and vision, we call it mutualisation. It's all about reducing the barriers between our journalists and readers and between our organisation and our partners. For software developers and online businesses we have a product sitting just in this space which we call the Open Platform.
The Open Platform has been in beta for the last year and now, today it is officially "Open for Business". Open for developers to take and co-monetize our content with us, building new applications for distributing and consuming news. It is also open for partners to build new experiences which blend content from The Guardian and their own functionality and in some cases brands into the heart of The Guardian's award winning website.
Today we're launching a brand new product and framework called MicroApps which the diagram above describes. However, just as Google dogfoods its new products before launch, so do we, and we wanted to share some of the things we've been building as MicroApps using Google AppEngine for the storage and application development part.
With 36 million unique and very engaged readers, everything we make has to scale, which is why AppEngine is ideal. With it's highly scalable architecture and features such as task queues, built for creating loosely coupled apps, and memcache, AppEngine makes an excellent companion platform for MicroApps in which the apps can run anywhere in the cloud. The examples presented here range from new ways to find content which others are finding exciting or interesting, to live responses to the TV debates to ways to bring together all of the tweets of our journalists on specific subjects.
Zeitgeist
Many people are interested in what is hot in the news and what is trending. We think this is interesting, but something a bit more interesting is in the things which are unexpectedly popular. Dan Catt used AppEngine to pull together signals from sources such as OneRiot, Twitter, comments in our system and our own internal stats to create an index of Zeitgeisty-ness. For the full "science bit" please see this overview article. We're using AppEngine for the storage and processing of a lot of data, with several task queues in play to create a page which responds over time during the current day showing you what people's attention think is the most interesting content. At the end of the day these are all archived and available so you can see what the Zeitgeist was on a particular day. The MicroApp framework allows us to host all of the application on AppEngine and display it in the network pages as if it were coming out of the content management system. This allows us to build data heavy applications using the economy of scale of cloud computing platforms while not affecting the scalability of the core site. If you want to see exactly how we made Zeitgeist, Dan talks about it here.
Election Debate Live Poll
There was recently a general election in the UK and for the first time in the UK there were a series of debates about the party platforms from the leaders of the three main parties. We felt that it would be good to capture the mood of the nation by allowing people to simply click a "+" or a "-" as to whether they liked or disliked what a leader said. This was then to be displayed as a sentiment "worm" alongside the video. Michael Brunton Spall created an AppEngine app which allowed us to scale rapidly and sustainably for the duration of the debates. At one point in time in the second debate the AJAX calls from the buttons were peaking at around 750 requests per second with an average of 454 requests per second. In total over the hour and a half of the debate over 2.7 million requests were made on the AJAX voting services which made heavy use of memcache and sharded counters. This sort of temporary load would have required a vast amount of additional temporary infrastructure for us and planning. AppEngine made that simple.
Twitter Collation
Just before The Edinburgh Festival, a really rather huge arts festival in the UK, we had a hackday and Stephen Abbott made a suggestion for something that it would be nice to see was an app that would allow the tweets about the festival of a collection of journalists and performers we trusted to be pulled together in one stream. This would need a double white list to be created, one where only if they were in the collection and they mentioned the keyword would your tweet make it into the stream. This is clearly to avoid spam and unwanted messages and to filter out journalists from talking about what they'd had for breakfast (unless it was festival related). In 24 hours using AppEngine Chris Thorpe had built a machine which fanned out queries from regular cron job which hauled in the stream of each journalist and performer (each in a single task queue action) and then processed each tweet asynchronously (again each in a single task queue action). The app performed well and was then industrialized as a MicroApp by Michael Brunton Spall and Lisa van Gelder and has been used throughout the general election and to bring together tweets from journalists within sections. We're now following and processing the streams of a large number of our journalists and are creating views of those streams with multiple watched keywords. The view of the stream can then be inserted into pages just like the one below using AppEngine and our MicroApp framework with memcache used to keep the response time fast.
Random Guardian
A flippant comment, in a serious question from Chris Thorpe to Clay Shirky, when he visited us recently at The Guardian, about where the new serendipity was in news, contained the phrase "should we just build Chatroulette for news?" He said that yes, we should definitely do that and in about 2 hours and using the beta of our new APIs, Daniel Vydra had done just that in AppEngine. The application has been hugely successful at introducing people to parts of the site they've never visited before and has led to people being mildly addicted to discovering things they never knew they needed to know, just at the click of a button. We're open sourcing the code for it very soon, so you too can start experimenting.
The MicroApp platform wouldn't exist without the hard work and brilliance of a team including Matthew Wall, Matt McAlister, Stephen Dunn, Mike Bracken, Graham Tackley, Stephen Wells and Simon Willison. We're now open for business and to prove it we're holding a hackday where we're making some very rapid prototypes for media agencies. Who knows, they may like them so much as concepts they commission them and we'll have to put them into production. That'll be fine for us, with the tools we use and the platform we have we know we have an environment to scale.
Photo by Dan Catt
App Engine has a storied history at Google I/O. Two years ago, we launched Google App Engine to at the inaugural I/O. It’s been two amazing years for App Engine and we’re very excited to be back at this year’s third annual I/O with the announcement of Google App Engine for Business. Google App Engine for Business lets organizations build and maintain their applications on the same scalable architecture that powers Google applications, with added management and support features tailored specifically for the enterprise.
Google App Engine for Business introduces a number of new features that our enterprise customers have been asking for, including:
With these new features, we’re making it easier for businesses to take advantage of the core benefits of Google App Engine: easy development using languages you already know (Java and Python); simple administration, with no need to worry about hardware, patches or backups; and effortless scalability, automatically getting the capacity you need when you need it.
Google App Engine for Business is currently in preview, opened to a limited number of enterprises. Learn more about App Engine for Business, how you can sign-up, and check our roadmap to follow features as they become available.
It’s been a busy past few months for the App Engine team and we’re doing a ton of great stuff at I/O this year.
First up, we are happy to announce version 1.3.4 of the App Engine SDK for Python and Java developers. This release has a number of additional features including:
With 1.3.4, we are also announcing we will be deprecating support for Java 1.5. Developers using 1.5 will now see warnings when building their applications informing them that they should upgrade their version of Java. 1.5 will be supported for at least the next two releases of the SDK and 1.5 will continue to be supported on the App Engine service with previous versions of the SDK for three years.
We’re also demoing a few upcoming features of App Engine at I/O as part of our sessions:
There will be opportunities for you to experiment with both APIs in the near future, so keep an eye on the Google Group and this blog for updates.
We’re also excited to announce that we will be adding App Engine to the Google Qualified Developer program in the near future. If you’re interested in becoming a qualified developer, and helping to develop the curriculum for future programs, you can visit the Developer Qualifications page and sign up to be a member of our first class!
Finally, if you are one of the developers joining us at I/O, make sure to check out the App Engine I/O sessions, visit the App Engine Developers in the Developer Sandbox, and swing by the office hours to talk with the App Engine team, or just to hang out! We look forward to seeing you there.
Use promo code NEXT1720 to save $300 off general admission
Follow