Last year, we featured a guest post by Jeffery Rosen, of Wolfire Games about the Humble Indie Bundle, an awesome promotion for a good cause: spreading awareness of indie games, and supporting charity. For 12 days, you could name your own price for a bundle of awesome independent games, with proceeds split as you wish between the game developers and two deserving charities, Child's Play and the Electronic Frontier Foundation. They hosted the Bundle on App Engine, to great effect - read the original blog post to learn more.

Last year, we featured a guest post by Jeffery Rosen, of Wolfire Games about the Humble Indie Bundle, an awesome promotion for a good cause: spreading awareness of indie games, and supporting charity. For 12 days, you could name your own price for a bundle of awesome independent games, with proceeds split as you wish between the game developers and two deserving charities, Child's Play and the Electronic Frontier Foundation. They hosted the Bundle on App Engine, to great effect - read the original blog post to learn more.


The good news is, they're doing it again, with the same App Engine infrastructure, and a new set of games! There's only 4 days left, so be sure to check them out if you're interested.



App Engine version 1.4.0. is here! It’s our most significant release of the year for the App Engine SDK, including a number of very big features that we know developers have been eagerly awaiting:

App Engine version 1.4.0. is here! It’s our most significant release of the year for the App Engine SDK, including a number of very big features that we know developers have been eagerly awaiting:



  • The Channel API - A bi-directional channel for communicating directly with user browsers by pushing notifications directly to the JavaScript running on the client, eliminating the need for polling. This service makes it easy to build real-time applications such as multi-player games, chat rooms, or any collaboration centric app and is built on the same Google infrastructure that powers Google Talk.

  • Always On - For high-priority applications with low or variable traffic, you can now reserve instances via App Engine's Always On feature. Always On is a premium feature costing $9 per month which reserves three instances of your application, never turning them off, even if the application has no traffic. This mitigates the impact of loading requests on applications that have small or variable amounts of traffic.

Screenshot of the Instances page in the App Engine Admin Console with Always On enabled.
  • Warm Up Requests - This feature reduces time to serve requests by anticipating the need for more instances and loading them before user traffic is sent to the new instance. It can be enabled for all applications through app.yaml or appengine-web.xml and is enabled by default for applications that have purchased Always On. Once enabled, warm up requests will be sent whenever possible to load new instances of your application before it begins serving user traffic.


As well, we’ve spent a lot of time this release on reducing or removing the limitations of some of App Engine’s existing APIs



  • No more 30-second limit for background work - With this release, we’ve significantly raised this limit for offline requests from Task Queue and Cron: you can now run for up to 10 minutes without interruption.

  • Increased API Call Size Limits - A new API architecture has allowed us to start lifting the 1MB size limits on many of the App Engine APIs. To start, the following APIs have been changed:

    • Response size limits for URLFetch have been raised from 1MB to 32MB.

    • Memcache batch get/put can now also do up to 32MB requests.

    • Image API requests and response size limits have been raised from 1MB to 32MB.

    • Mail API outgoing attachments have been increased from 1MB to 10MB


As you can imagine, some of these changes drastically expand the scope of applications that can be easily built using App Engine so download the SDK while it’s hot!


Keep an eye out for more blog posts on how you can take advantage of the new features to build your apps very soon. And we’ve got a few more big features coming very soon, such as a High Replication Datastore, so keep an eye on the App Engine roadmap and stay tuned.



The following post is a guest post by Chris Bunch, a Computer Science Ph.D. student at the University of California, Santa Barbara. He is one of the student leads on the AppScale project, an open source Google App Engine compatible hosting solution led by Professor Chandra Krintz. Chris has developed and maintained AppScale as a research project over the last two years with fellow student lead Navraj Chohan and others.
...

The following post is a guest post by Chris Bunch, a Computer Science Ph.D. student at the University of California, Santa Barbara. He is one of the student leads on the AppScale project, an open source Google App Engine compatible hosting solution led by Professor Chandra Krintz. Chris has developed and maintained AppScale as a research project over the last two years with fellow student lead Navraj Chohan and others.



---------

Over here at the UCSB Racelab, we've complained endlessly about finding a web framework we actually could use. For a long time we thought we just wouldn't be able to find it - many were so-so or good but only after a substantial learning curve. So imagine our surprise back in April 2008 when we heard about what we thought would be just-another-web-framework provided by Google in the Python version of App Engine. But after giving it a try, we were smitten. We finally found a web framework that (1) we could actually use on non-trivial projects
and (2) we could teach in nine-week classes without having students lose half the time with the idiosyncrasies of the programming language involved or the web framework itself. Furthermore, the minimalistic APIs make it simple to get work done: it did for us exactly what we needed and nothing else.



Yet as researchers and hackers-at-heart there was one thing that we really wanted to do with App Engine that we couldn't do: run it on a whole bunch of our machines and tinker with it. A similarly-minded hacker named Chris Anderson had released AppDrop, which was a modified version of the App Engine SDK that hooked up to PostgresSQL and run in Amazon EC2, but only ran over a single machine. So after much discussion, we came up with the following short list of things we wanted to do with App Engine:






  • We wanted to run it on our own virtual machines or those running in Eucalyptus or Amazon EC2 in order to investigate how we can optimally harness cloud infrastructures in our cloud platform.


  • Tons of new datastores have emerged as part of the "NoSQL" movement, and we need a mechanism to evaluate their performance under controlled experiments as well as traditional databases such as MySQL. We also need a platform that supports the ability to add new data storage mechanisms so that when developers tout the features of their new datastore, we can download it and evaluate it under similar circumstances as other datastores.


  • One of the reasons we love Google App Engine is the simple set of APIs provided, but we also wanted to use that as a starting
    point where we could add new APIs and control the environment in which they run.

  • We love that Google App Engine "just works". You don't know where it's running and how it's running, but you can see that it is running, and we wanted to make sure that whatever we developed, that it did the same. We wanted to develop something that automatically deployed your App Engine app and configured everything for you. Expert users should be able to have more control over the system, but the system should be able to handle your app from the moment you deploy it to the moment you tear it down.


  • It had to be open-source - just like how we wanted something to tinker with and run experiments on, we wanted it to be something that you could tinker with too. We wanted you to be able to add in support for a database you're interested in and see how it performs, and we wanted you to be able to add in APIs that you think would be interesting to have an easy-to-use web framework interact with.




So with that in mind, we created AppScale, an open-source cloud platform
for Google App Engine applications. Here's how we did it:




We took the standard three-tier web deployment approach and clearly segmented each tier into a specific component in the system: an AppLoadBalancer routes users to their applications, an AppServer runs the user's App Engine app, and an AppDB handles database interactions. Each have clearly defined roles in the system and are controlled by an AppController, a daemon that runs on each machine, monitors each component, and controls the specific order in which services are started. It writes all the configuration files for each service and coordinates services between the other AppControllers in the deployment. For those interested, we detail the specifics on the original AppScale implementation in this paper.




We also wanted to embody the principle of "standing on the shoulders of giants", and as such, we employ open-source software as often as possible, where appropriate. Our AppLoadBalancer employs the nginx web server as well as the haproxy load balancer to ensure high performance. Our Memcache API implementation uses memcache under the hood, while our MapReduce API uses Apache Hadoop, which we added to give App Engine
users running over AppScale the ability to run Hadoop MapReduce jobs from within their web applications.



Because we were able to keep the database support abstracted away from the other components in the system, we were able to add support for nine different data storage solutions within AppScale: HBase, Hypertable,
MySQL, Cassandra, Voldemort, MongoDB, MemcacheDB, Scalaris, and SimpleDB. Many of these databases have seen interest in recent years but have been hard to measure under comparable conditions, and vary greatly. To give a few examples, they vary in the query languages they provide, their topologies (e.g., master / slave, peer-to-peer), data consistency policies, and end-user library interfaces. This has made it non-trivial for the community to objectively determine scenarios in which one database performs better or worse than another and investigate why, but under AppScale, deploying all these databases is done automatically with no interaction from the user. And because AppScale is open-source, if a developer doesn't like the particular interface we use for a database, they can improve on it and give back to the community. We've used
AppScale internally to evaluate the performance of Google App Engine applications on these datastores as well as developed an App Engine app, Active Cloud DB, that exposes a RESTful API that developers can use
to access these datastores from any programming language or web framework.



Finally, the most important lesson we learned was the value of incremental development. Our core development team fluctuates between two to three developers, so from the first meeting we had, we knew that
our very first release couldn't support every App Engine API nor could it run nine databases seamlessly. Therefore, we started off with support for the two BigTable clones, HBase and Hypertable, as well as support
for just the Datastore API, the URL Fetch API, and the Users API within App Engine. From there, we learned what datastores people actually wanted to see support for as well as what APIs people wanted to use. We
were also able to add APIs within App Engine apps deployed to AppScale to be able to run virtual machines under the EC2 API, while also running computation under the MapReduce API.



But developing AppScale was certainly not a cakewalk for us. Over the course of the last two years, five major issues (some technical and some not) have arisen within the project:





  1. Writing software that works without knowing ahead of time how many machines will be in the system proved initially to be difficult to grasp, but in many cases we were able to reduce the number of variations that could occur and use that to provide some predictability with respect to how we configure and deploy databases and applications.


  2. We couldn't assume that the AppScale administrator has access to DNS; without it, a number of APIs and features are extremely difficult to implement. Load balancing is much more difficult, and many APIs that are tied to host names must be tied to one machine in the system, else they don't work properly. VLAN tagging shows some promise to alleviate these problems, but right now is far from being deployed inexpensively and easily.


  3. The source code for the Java version of App Engine isn't publicly available, so we had to spend a lot of time decompiling the SDK, modifying it to use our database and our API implementations instead of the SDK implementations, and recompiling it. All of these were non-trivial and greatly added to the time it took for us to deploy a version of AppScale with Java App Engine support.


  4. Not all users want a pre-built virtual machine image, so ensuring that building the AppScale environment was done right every time was a top priority. We had to limit ourselves to Ubuntu Jaunty for many releases, and only recently were we able to expand to include Karmic and Lucid, which still make up a microcosm of the distributions available in the Linux world. Adding the ability to install AppScale via apt-get in these specific Linux distributions has also been a crucial step in making sure that users could easily and quickly install AppScale for use.


  5. Both undergraduates and graduate students here at UCSB have done projects involving AppScale, which means that the number and experience levels of developers working on AppScale is completely unpredictable at a given moment in time. Oftentimes the projects they work on are only tangentially related to features that users want, and the time scales that they are available to work for is vastly different than most software engineers are used to.



All of these problems are greatly exacerbated by only having a two-to-three person core developer team, but this also makes the AppScale project particularly interesting to work on. Despite having worked on AppScale for two years, there are still tons of interesting problems to work on and we still love the Python App Engine web framework as much as we did when we first picked it up. And of course, AppScale is open-source, under the New BSD License, so feel free to download it and tinker around like we have! Check out AppScale at:




http://appscale.cs.ucsb.edu




http://code.google.com/p/appscale



-- Chris

Today, we’re releasing version 1.3.8 of the App Engine SDK. Whether you’re a Java or a Python developer, this release includes several exciting new features for improving monitoring, performance, and maintenance tasks.

Today, we’re releasing version 1.3.8 of the App Engine SDK. Whether you’re a Java or a Python developer, this release includes several exciting new features for improving monitoring, performance, and maintenance tasks.


Instances Console


This release includes a new page in the Admin Console, called the Instances page. This page allows you to view information about all server instances currently in use by your application. This information can be useful in debugging your application and also understanding its performance characteristics. There’s no configuration needed for this feature. Just click the “Instances” link on the left hand navigation of the Admin Console to see Average QPS, latency, and memory for an instance.

Screenshot of the instances page of the Admin Console

Task Queue Improvements


This release also has a couple new Task Queue features: First, the maximum bucket size that you can specify during queue configuration is now 100, up from 50. Second, we’ve added a new "Run Now" button to the Task Queues section of the Admin Console that enables developers to run a task immediately. This can be very helpful for debugging your tasks in production.


Builtins Directives


This release contains a new feature for Python apps: builtin handlers that allow you to quickly and easily enable standard functionality in your application without adding additional code to your codebase. The libraries available today are remote_api, appstats, and the datastore_admin feature (see below). For example, to use the remote_api with your application, simply add the following to your app.yaml file:


builtins:
- remote_api: on

If you are already using the remote api endpoint your app, you can choose to remove the entry in the handlers section of your app.yaml and use the above directive instead to simplify your app.yaml file.


Support for builtin handlers is not yet available for Java applications, but will be available in an upcoming release.


Delete all (or a part) of your application’s data


Note: this feature is currently only available by default for Python; see the note below for ways to use it with Java application.


Today, we are releasing an experimental addition to the admin console which provides a simple UI for delete all entities, or all entities of a given kind, in your datastore. To enable this functionality, simply enable the following builtin in your app.yaml file:


builtins:
- datastore_admin: on

Adding these lines to app.yaml enables the “Datastore Admin” page in your app’s Admin Console, where you can see all of the entity types you are able to delete:


Screenshot of the datastore delete builtin UI

Be aware that these deletes are issued by your application (you can read about how the handler works by looking at this code file in the SDK). For this reason, your application will use resources, most significantly CPU, for the deletions you issue which will count towards your application’s daily resource budget.


Datastore delete is currently available only with the Python runtime. Java applications, however, can still take advantage of this feature by creating a non-default Python application version that enables Datastore Admin in the app.yaml. Native support for Java will be included in an upcoming release.


Python Pre-compilation on by Default


Finally, the python pre-compilation feature we announced in 1.3.5 is now turned on for all new python application uploads using the 1.3.8 SDK by default. If you wish to disable this feature, just specify the flag --no-precompilation on the appcfg.py command line when uploading your app.


This release also contains a few more small features and bug fixes. You can read about the full release in our release notes in Python and Java. As always, your feedback in the forums is appreciated (and had a significant influence on this release!).





It's conference season, and we're going around the world again! This fall, Google team members will be speaking about App Engine at events in more than 10 countries! In addition to the regular conference circuit, Google is also hosting Developer Days and DevFests in a variety of locations to bring not only App Engine but a bunch of Google technologies directly to you!




Fall 2010:




We look forward to meeting you at one (or more) of these events!


Posted by Wesley Chun, Google App Engine team

Last week we announced our 1.3.6 release with lots of new, exciting features to try out. After it was released, developers dug into all the new functional and discovered a pair of issues that required a quick fix. Today we are releasing version 1.3.7 of the SDK in both Java and Python to fix two issues that shipped with last week’s release 1.3.6.

Last week we announced our 1.3.6 release with lots of new, exciting features to try out. After it was released, developers dug into all the new functional and discovered a pair of issues that required a quick fix. Today we are releasing version 1.3.7 of the SDK in both Java and Python to fix two issues that shipped with last week’s release 1.3.6.


The first issue was with the Python SDK, which was missing the functionality to assign a namespace based on the Google Apps domain. With this release the function google_apps_namespace() should exist and function as documented.


The second issue we fixed was with the Java SDK. The getServingUrl() function for the new dynamic image resizing service was throwing a SecurityException in the SDK. This issue has now been fixed and this function should work as expected.


Thank you to all the people who reported these issues. Please visit the download page to get the new version of the SDK.


Today marks the 1.3.6 release of App Engine for Java and Python. In this release we have made available several exciting new features, relaxed quota and datastore limitations, and added various issue fixes.

Today marks the 1.3.6 release of App Engine for Java and Python. In this release we have made available several exciting new features, relaxed quota and datastore limitations, and added various issue fixes.



Multi-tenant Apps Using the Namespaces API

We are pleased to announce support for multi-tenancy for applications via the Namespaces API. With multi-tenancy, multiple client organizations (or “tenants”) can all run the same application, segregating data using a unique namespace for each client. This allows you to easily serve the same app to multiple different customers, with each customer seeing their own unique copy of the app. No changes in your code are necessary to use this API-- just a little extra configuration. Further, the API is also designed to be very customizable, with hooks into your code that you can control, so you can set up multi-tenancy in any way you choose.



Check out our application examples for Java and Python to demonstrate how to use the Namespaces API in your application. The API works will all of the relevant App Engine APIs (Datastore, Memcache, and Task Queues). Check out our docs for Java and Python to learn more.



High-Performance Image Serving

This release also includes a new, high-performance image serving system for your applications, based on the same infrastructure we use to serve images for Picasa. This feature allows you to generate a stable, dedicated URL for serving web-suitable image thumbnails. You simply store a single copy of your original image in Blobstore, and then request a high-performance per-image URL. This special URL can serve that image resized and/or cropped automatically, and serving from this URL does not incur any CPU or dynamic serving load on your application (though bandwidth is still charged as usual). It’s easy to use, just call the Python function get_serving_url, or the Java function getServingUrl and supply a Blob key (with optional serving size and/or crop arguments), and you can now serve dozens or hundreds of thumbnails on a single page with ease. To enable high performance image serving in your deployed application, you'll need to enable billing.



Custom Error Pages

Since launch, many developers have asked to be able to serve custom error pages instead of those automatically served by App Engine. We are happy to announce today we are supporting static HTML error pages that can be served for you automatically for over quota, DoS, timeout and other generic error cases, that you previously could not control. You can configure custom error handlers in your app.yaml or appengine-web.xml file. Check out the Java or Python docs for more information.



Increased Quotas

We have also continued the trend of lifting some system limitations that have been in place since launch. The Datastore no longer enforces a 1000 entity limit on for count and offset. Queries using these will now safely execute until they return or your application reaches the request timeout limit. Also, based on your feedback, we have raised nearly all of the burst quotas for free apps to the same level as the burst quotas for billed apps. Check out the docs for more information on quota limits.



Other features included in 1.3.6:
  • Java developers can now use the same app.yaml configuration file that App Engine uses for Python applications instead of appengine-web.xml (if preferred).
  • You can now pause task queues via the Admin Console interface.
  • Dashboard graphs in the Admin Console will now begin showing up to 30 days worth of data.
  • Content-Range headers are now supported with the Blobstore API.
You can read all about these features in the App Engine documentation for Java and Python. The new versions of the SDK can be found on our downloads page.

In addition to all of the new features, we’ve included several bug fixes which you can read all about in the release notes for Java and Python.



-- Posted by the App Engine team




Greetings App Engine Developers!




My name is Dan Murray. I'm a Founder and Managing Director at WebFilings http://www.webfilings.com. In March 2010 my company launched a first-of-its-kind SaaS solution that dramatically improves the process of developing and filing financial reports with the Securities Exchange Commission (SEC). Google App Engine provides the foundation for our backend. Since our launch, my company is experiencing what can only be described as hyper-growth. Everyday, as our team manages the rapid growth of our user community, I am thankful we chose App Engine when we started WebFilings.




Greetings App Engine Developers!




My name is Dan Murray. I'm a Founder and Managing Director at WebFilings http://www.webfilings.com. In March 2010 my company launched a first-of-its-kind SaaS solution that dramatically improves the process of developing and filing financial reports with the Securities Exchange Commission (SEC). Google App Engine provides the foundation for our backend. Since our launch, my company is experiencing what can only be described as hyper-growth. Everyday, as our team manages the rapid growth of our user community, I am thankful we chose App Engine when we started WebFilings.




Our Challenge




Each quarter, thousands of public and private companies in the US report their earnings to the SEC. The SEC reporting process is complex, time-consuming, and expensive. Reporting requirements and accounting codes are in a state of constant flux. The effects of misreported finances can be extremely costly to a company both financially and in reputation.




Prior to starting WebFilings, my partners and I spoke with dozens of companies and listened to them describe their approach to SEC reporting. While each company has its own unique challenges and methods, the general theme was the same: today's word processors and email just don't cut it for financial reporting. SEC reporting requires more sophisticated collaboration and data management capabilities than current tools provide.




We formed WebFilings almost immediately after these meetings and started building our product. We knew our software would manage a large amount of very granular data for which history and revision control are important. Individual data entities would be interrelated and a change to one may require a cascade change to hundreds, thousands, or more entities. Our product needed rich collaboration capabilities to allow multiple authors to work on a single report simultaneously and communicate effectively throughout the reporting process. Finally, because our software would manage sensitive, pre-release financial information, our product had to be state-of-the-art when it came to information security.




Why Google App Engine?




We started working with Google App Engine right after it launched in 2008. Since we started, our development team has grown from a few guys in a basement to an extensive worldwide organization. From the beginning, the choice of using GAE for our backend was obvious. App Engine provides many benefits to a business building a large-scale, data intensive product on a short timeline.





App Engine gives us a flexible, object-oriented datastore that enables us to iterate in a way that is just not possible with traditional database and ORM software. When combined with the ease of deployment to the cloud, we are able to develop at a rapid pace and deliver timely, incremental solutions for our customers' emerging needs.




In addition, App Engine appspot gives us a secure place for our customers' data. It's been clear from the beginning that the App Engine Team works very hard to plug each and every possible security hole. Large organizations all over the world are switching to Google Apps. Real peace-of-mind comes with Google's reputation for data security. I can think of no other company in the world that takes security more seriously and actually walks-the-walk.




Finally, our costs with Google App Engine have been unbelievably low. In past ventures, I got used to huge startup costs and large expenditures each month for hardware infrastructure and support staff. That just isn't the case with App Engine. We don't need a large IT staff to support our product because App Engine scales automatically. Our costs have been and continue to be orders of magnitude lower than anything I've ever experienced.




How we built our implementation




I had always heard that building a word processor is one of the hardest things you can ever do as a software engineer. Well now I know for a fact that it is. In addition to building a word processor, we set out to create a spreadsheet editor as well. Both applications run in a web browser and are interoperable in a way that enables you to "link" very granular data between them. Add in the ability to share documents and track revisions among a large team of collaborating editors, and building WebFilings software has easily been one of the most challenging endeavors during my 20+ years in software.




Using WebFilings rich editor to file financial reports





Google App Engine has been absolutely key to our success in ramping up development quickly and building our product in a very short timeframe. We built our product using App Engine's python runtime environment. The combination of python and App Engine's datastore enabled us to take an iterative and incremental approach to building our backend.




WebFilings software is based upon a multi-tier, service-oriented cloud mashup as shown below.




WebFilings' architecture






Our presentation tier consists of three end-user applications. Our editor and reader applications are implemented using the Adobe Flex SDK and delivered to the users desktop using Adobe Flash Player 10. Our administration application is AJAX-based and implemented with HTML, CSS, Javascript, and JQuery.




Our editor and reader applications combine a traditional word processor with a spreadsheet editor. Each contains logic that controls the content, layout, and display of formatted text. The foundation of these applications is Adobe’s Text Layout Framework, an extensible, open source, ActionScript library that leverages the text engine in Adobe Flash Player 10.





Our administrative application contains logic that controls CRUD operations for accounts and users. Through our administration application one administers usernames, passwords and password policy, user account access, subscription levels, and various account and document permissions.





Our business tier is developed using Google App Engine. Web services are implemented using python and the django web application framework. Our Flex/Flash-based editor and reader communicate with the business tier by exchanging serialized binary representations of WebFilings business objects through a Django-based web services API. The AJAX-based administration application communicates with the business tier using JQuery. Service requests and responses are sent via HTTPS.





The business tier implements logic in two sub-tiers. The first of these sub-tiers is responsible for basic business logic that provides methods for foundation and often repeated operations such as creation of a new document. The second of these sub-tiers is responsible for business processes that chain together basic business services to accomplish larger, more complex operations such as sharing a users’ changes to a document.





Business service and process methods access data through the App Engine Data Services API. This API provides a python-based abstraction layer consisting of convenience methods that unify the data tier API with security sandbox methods designed to prevent unauthorized data access across Google appspot instances.




WebFilings end-user application interoperates with read-only search services and transient translation services hosted using Amazon EC2. These web services include an XBRL taxonomy search service, and XBRL validation service, an XBRL instance document viewer, and an EDGAR HTML translation service.





The XBRL taxonomy search service, and XBRL validation service, and EDGAR HTML translation service use Apache Tomcat and the Java Servlet API. In addition, the XBRL taxonomy search service uses MySQL and the Hibernate ORM SDK for storage, access, and indexing of the SEC’s public taxonomy data.





The XBRL instance document viewer is a Microsoft .NET Framework-hosted implementation of the U.S. Security Exchange Commission (SEC) XBRL reference viewer.





We've been fortunate that the App Engine Team continues to churn out innovations at a rapid pace. AppStats has helped us identify problem areas and fine tune performance. Task Queues have given us a good solution for processing large amounts of data and helped us reduce request latency. Memcache has provided us up to 10x performance increase for critical areas of our application. We've been able to absorb these innovations quickly without having to re-architect.





Future Apps on App Engine
As WebFilings grows, we will continue to expand our product and the web services that support it. When we find opportunities to build new applications there is no doubt that Google App Engine will be at the core.





Please visit http://www.webfilings.com to learn more about WebFilings software. Our team is growing very fast and we're always looking for top software talent. Check out our Careers page If you want a new and exciting challenge with one of the best software teams around.





We recently announced the Mapper API, a first step in a broader effort to provide full MapReduce capabilities on App Engine. While we still have some work ahead of us, there’s already a lot that can be done with today’s Mapper API. One area of particular interest is report generation.

We recently announced the Mapper API, a first step in a broader effort to provide full MapReduce capabilities on App Engine. While we still have some work ahead of us, there’s already a lot that can be done with today’s Mapper API. One area of particular interest is report generation.




Most applications create and maintain large numbers of detail records: entities in data models, transaction history or event logs. In order to glean useful information out of these vast data sets, your application has to iterate over your entities, summarize and breakdown the results. That’s where the Mapper API comes in.




The Mapper API uses the Task Queue to enable your application to rapidly iterate over its data sets, whether small or very, very large. The API takes care of the tedious bookkeeping involved in efficiently scheduling and keeping track of all those tasks. The Task Queue, in turn, automatically ‘pushes’ the work to your app.




Christopher O’Donnell (@markitecht), a technology product designer and developer from Cambridge, Massachusetts, was kind enough to share with us some of the motivations and implementation details behind his own project. In a new guest article, Modern Funnel Analytics Using Mapper, Christopher makes extensive use of the Mapper API to illustrate the process of generating rollup reports. With his approach, he's able to provide both summarized results and drill down capabilities.




If you like Christopher’s article, you should also checkout out the many other interesting articles on the Google App Engine home page. And, if you have an interesting App Engine article or story you'd like to share, let us know. We'd love to hear about it.




Posted by , App Engine Team


"The single coolest thing about this project is that it was possible to go from a state of knowing nothing whatsoever about App Engine or Python (other than the mile-high view) to having a working and useful application inside of eight hours. We're long-time geeks but we're not geniuses. For us to pick up a new language, a new SDK, a new environment, a new way of doing things, and produce anything of value at all in such a short time speaks volumes about the value, potential, and quality of App Engine and Python.
After installing the App Engine SDK, yes, the very first thing I did was your online tutorial. I did "Hello World" to find my feet then continued into webapp, since a clean URL handler with easy ways to get at HTTP variables seemed essential. Then I immediately jumped into learning about data storage. And wow, what an enlightenment that turned out to be! Goodbye SQL, don't think I'm going to miss ya.... :-)
Since the app's purpose is to manage just ~400 simple objects representing Cycle Hire Stations, each of which contains only Plain Old Data types — no object references or anything possibly messy — I felt I knew enough to implement it now, and so I dived in. And it was so easy! I started with a handler to rebuild the datastore from scratch. Then I wrote a "get" type of handler to retrieve information about groups of hire stations (returning the data in JSON). Finally I wrote an "update" handler so that updated information about cycle hire stations could be posted, and that was it. Job done.
One thing that initially confounded me was an HTTP 500 error caused by our "reset" handler exceeding the 30-second request limit. For a while I was ready to despair; HTTP 500s to anyone with much ASP experience usually means a hideous low-level bug somewhere! However, once we discovered the problem, this was easy to fix by splitting the work into multiple requests (/reset1, /reset2, etc.) It's an admin function that only we'd ever be using, so no harm done and no need to work out anything more clever.
I know we've barely scratched the surface of what can be done with App Engine. We've yet to use Memcache, background tasks, batched updates, or anything beyond simple cloud-based data storage. But that simple thing alone seemed then, and still seems, not far short of miraculous. To not have to worry about databases, servers, uptime, upgrades and above all scaling... to not have to think about any of that at all is such an immense freedom. I'm completely hooked on it and am unlikely to go back to my traditional server tools of MySQL and PHP.
Update Oct 2010: It's been several months since we introduced you to Little Fluffy Toys and their exciting Google App Engine story. Since then, their app has continued getting rave reviews. We're happy to let you know that there is a great follow-up to this story from the Android and UI/UE/UX perspective, and you can find it here: http://blog.radioactiveyak.com/2010/10/android-app-surgery-cycle-hire-widget.html

This post is another entry in our ongoing series of guest posts contributed by App Engine developers. Today we partner with Little Fluffy Toys Ltd to tell the story of how they were able to learn App Engine (plus Python) and launched their service paired with their Android application in less than a week!
Introduction
Last week, Little Fluffy Toys Ltd (LFT) launched an Android app to help its users find bicycles and rental locations in London. While this story doesn't sound particularly phenomenal, how they accomplished this using Google App Engine (and Android) makes their application and its launch one of the most exciting success stories so far in 2010.
The development team at LFT were able to quickly come up-to-speed on learning a new programming language and development environment in order to build and launch the App Engine backend service for their Android mobile app to the world in less than one week. The executive summary:
  • Attended 1-hour Thursday night presentation on Google App Engine (Jul 22)

  • Started to learn Python and App Engine on Saturday afternoon

  • Launched live service Wednesday, announcing their Android app with an App Engine backend (Jul 28)

Before we get to the good stuff, a brief backgrounder on the project which spawned the application: metropolitan bicycle-sharing systems, specifically London's. Based on the success and popularity of the Paris Vélib' system, the Barclays Cycle Hire scheme originated mid-November back in 2008 from its mayor, a strong cycling proponent.
The system launched on July 30, 2010; however a month before the project reached its completion, a call for apps was given by the mayor seeking independent developers so that there would be a variety of mobile and web apps available by show time. Enter Little Fluffy Toys Ltd which ended up creating the Cycle Hire Widget app for Android. They needed a backend system to manage the data, found App Engine, and the rest was history. Shortly after going live with their app and the launch of London's bicycle rental system, I had a chance to discuss how their project came together with the help of App Engine.
NOTE: Cycle Hire Widget is only available in the Android Market if you are located within the UK. If you wish to view the application from outside the UK, please download and install it from within your Android browser via this link — bear in mind that the distances to your nearest rental/hire location will be ridiculous!
App Engine and LFT
As we mentioned above, the sole purpose of their App Engine app is to receive data from and provide data to the Android app running on users' mobile phones. The App Engine stored data is "global" for all mobile clients out there, and this includes names, locations, and dynamic specifics related to each bike station such as the data found in the app's screenshot below. Take note all the valuable data that is provided in real-time by App Engine:
If you're a bit familiar with App Engine, you would no doubt have heard about it as a platform for web applications, but this is a use case highlighting App Engine for a non web-based server-side application where no part of the app is user-facing save for what gets sent back to the mobile app. While this type of app doesn’t get much press, it’s more common than people think.
I met Kenton Price, the director and chief architect of Little Fluffy Toys Ltd, at a developer event recently, and he seemed to think that App Engine would be the right tool for the Cycle Hire Widget. It turned out to be true(!), and when I asked about LFT's needs and how they were met by App Engine after their successful product launch, here is what he had to say:
"As you know, we were massively against the clock with the launch of the cycle hire scheme, and we needed something we could get going with fast that would effortlessly scale to perhaps tens of thousands of mobile users. App Engine seemed the perfect choice from what we had read of it before the meeting, and after your presentation it was obviously the way to go. Your recommendation to use Python was scary given neither of us knew a thing about it, but then again we only knew Java from Android not from web development so we didn't have the domain knowledge of building Java web services. So we went with Python, and it worked out really well. I'm astounded how we actually delivered this product in a very short space of time when we both have full schedules working on projects for our clients and other demanding outside interests. Particularly satisfying was having a solution that was agile and flexible enough to enable us to display live cycle availability data within hours of it becoming unexpectedly available at the launch, so we were live in the field with real-time data that very same launch morning, a feature our competitors are still struggling to replicate."
Development experience
Reuben Harris, LFT's chief technical officer, is the lead App Engine developer for the Cycle Hire Widget. He had a great experience even though he was new to Python as well as App Engine. What excited him the most, and what was his development experience like? He tells his story here:
"The single coolest thing about this project is that it was possible to go from a state of knowing nothing whatsoever about App Engine or Python (other than the mile-high view) to having a working and useful application inside of eight hours. We're long-time geeks but we're not geniuses. For us to pick up a new language, a new SDK, a new environment, a new way of doing things, and produce anything of value at all in such a short time speaks volumes about the value, potential, and quality of App Engine and Python.
After installing the App Engine SDK, yes, the very first thing I did was your online tutorial. I did "Hello World" to find my feet then continued into webapp, since a clean URL handler with easy ways to get at HTTP variables seemed essential. Then I immediately jumped into learning about data storage. And wow, what an enlightenment that turned out to be! Goodbye SQL, don't think I'm going to miss ya.... :-)
Since the app's purpose is to manage just ~400 simple objects representing Cycle Hire Stations, each of which contains only Plain Old Data types — no object references or anything possibly messy — I felt I knew enough to implement it now, and so I dived in. And it was so easy! I started with a handler to rebuild the datastore from scratch. Then I wrote a "get" type of handler to retrieve information about groups of hire stations (returning the data in JSON). Finally I wrote an "update" handler so that updated information about cycle hire stations could be posted, and that was it. Job done.
One thing that initially confounded me was an HTTP 500 error caused by our "reset" handler exceeding the 30-second request limit. For a while I was ready to despair; HTTP 500s to anyone with much ASP experience usually means a hideous low-level bug somewhere! However, once we discovered the problem, this was easy to fix by splitting the work into multiple requests (/reset1, /reset2, etc.) It's an admin function that only we'd ever be using, so no harm done and no need to work out anything more clever.
I know we've barely scratched the surface of what can be done with App Engine. We've yet to use Memcache, background tasks, batched updates, or anything beyond simple cloud-based data storage. But that simple thing alone seemed then, and still seems, not far short of miraculous. To not have to worry about databases, servers, uptime, upgrades and above all scaling... to not have to think about any of that at all is such an immense freedom. I'm completely hooked on it and am unlikely to go back to my traditional server tools of MySQL and PHP.
To see Reuben's work in action, check out this video demonstrating how to use the Cycle Hire Widget app while roaming the streets of London seeking a bike to rent/hire or park:
Conclusion
Since the launch, the Cycle Hire Widget has gotten rave reviews from CNET, The Guardian, and The Londonist. It has even been featured by the Press Association of the UK and Ireland! One user commented on Android Market: "Can't really think of a way to make it better," a sentiment reflected in its very high feedback rating. It certainly does sound like quite a success. What does the future look like? I asked Kenton about how LFT came about as well as how they're looking to improve their succeeding offerings, and here's what he had to say:
"We formed Little Fluffy Toys Ltd as a vehicle for Android development where we do consultancy work as well as our own stuff like the Cycle Hire Widget and Social Wallpaper. Whilst all custom development enquiries are very welcome, we're also interested in hearing from people or organisations that would like us to customise Cycle Hire Widget for their particular domain, whether it's cycles with availability in another city, coffee shops with opening hours in a geographic region, or dieting group meetings at pertinent times nearby. You name it, there are a gazillion applications for it!"
Well here on the App Engine team, we're happy for Kenton and his team on being able to implement the server-side solution they needed in such a short period of time on App Engine, and better yet, to help out a worthy cause. Google itself is a socially responsible company that applauds efforts like Barclays Cycle Hire, so we're proud that technologies we provide such as Android and App Engine can be used to help make London and the Earth more sustainable!
Posted by Kenton Price & Reuben Harris, Little Fluffy Toys Ltd, and Wesley Chun, App Engine team

Back in March, we announced that all App Engine apps hosted on appspot.com would be reachable over IPv6, from ISPs enrolled in the Google over IPv6 program.



Today, we're adding the same service to all domains hosted by ghs.google.com. If your App Engine app runs on your own domain name using Google Apps, then be sure to review our earlier blog post, which discusses the caveats of parsing client IP addresses, and how to use the SixXS Gateway to test your site over IPv6.

Back in March, we announced that all App Engine apps hosted on appspot.com would be reachable over IPv6, from ISPs enrolled in the Google over IPv6 program.



Today, we're adding the same service to all domains hosted by ghs.google.com. If your App Engine app runs on your own domain name using Google Apps, then be sure to review our earlier blog post, which discusses the caveats of parsing client IP addresses, and how to use the SixXS Gateway to test your site over IPv6.



Currently, the Google over IPv6 program presents AAAA records only to ISPs within our whitelist. This helps ensure that our services remain reliable, by reducing exposure to clients with broken IPv6 connectivity. As the maturity of IPv6 deployment improves, we expect this whitelist to grow, eventually disappearing as it encompasses the entire Internet.



That said, we know that some of you are fans of IPv6 who would prefer to skip the whitelist, and offer dual-stack service to everyone immediately. To make your domain bypass the whitelist, just change your CNAME pointer from ghs.google.com to ghs46.google.com. Please note that, while ghs46 will cause your App Engine site to receive a larger fraction of its traffic over IPv6, our studies show that it may make your site slow or unreachable for up to 0.1% of clients across the Internet, so use this alternate domain with care.



As before, if you experience any problems with IPv6 serving and App Engine, please report it in the App Engine issue tracker or post about it in the App Engine discussion group.





If you have been running an App Engine application over the last few months, you will have noticed that performance of the Datastore has drastically improved. In fact, it quickly improved after the blog post we wrote in May acknowledging the performance issues. We chose to wait until this point to ensure that we could maintain the performance and quality of service. Thanks to a huge effort from the Datastore engineering team and our Production team, the metrics over the past month have made us confident that we have returned to a level of performance that we are proud to offer. With that in mind, we are now going to re-enable Datastore CPU usage billing.

If you have been running an App Engine application over the last few months, you will have noticed that performance of the Datastore has drastically improved. In fact, it quickly improved after the blog post we wrote in May acknowledging the performance issues. We chose to wait until this point to ensure that we could maintain the performance and quality of service. Thanks to a huge effort from the Datastore engineering team and our Production team, the metrics over the past month have made us confident that we have returned to a level of performance that we are proud to offer. With that in mind, we are now going to re-enable Datastore CPU usage billing.



On Tuesday, August 3rd, we will re-enable charging for Datastore CPU usage.



This does not mean that we are stopping work on Datastore performance and reliability-- quite the opposite, in fact. The issue has forced the team to step back and prioritize our long term roadmap for the Datastore. While we’re happy with our performance now, there are some very large changes to the Datastore in the pipeline that will continue to improve the App Engine platform. These include features such as a more highly replicated Datastore (now part of the roadmap) and improvements to the index building pipeline.



Thank you all for your patience and continued use of App Engine during this time. We’re excited to continue to serve our apps’ continued growth.



At Google I/O we announced the Mapper API. Built completely on top of public App Engine APIs today, this API is only the first component of App Engine’s MapReduce toolkit, but can be extremely useful on its own.

At Google I/O we announced the Mapper API. Built completely on top of public App Engine APIs today, this API is only the first component of App Engine’s MapReduce toolkit, but can be extremely useful on its own.



The Mapper API can already be of use to many developers who would otherwise need to build their own tool for doing large scale data manipulation. In addition to taking care of the distribution of these jobs over task queues, it provides the ability to store state, batch datastore writes via mutation pools, and ships with an easy to use administrative interface for job management, all optimized for the constraints of App Engine’s dynamic serving environment. Some examples of the types of operations that work with minimal configuration with this tool:


  • Report Generation
  • Large scale migration of entity properties
  • Datastore cleanup
  • Computing statistics and metrics
For an introduction to the Mapper API, watch Mike Aizatsky’s video from Google I/O, where he demonstrates building a source code indexer. The slides can be downloaded here, and the video is below:









The App Engine team has also written a few great articles on how to use the Mapper API.
  • For Python developers, take a look at the Python Mapper API post on Nick Johnson’s blog.
  • For Java developers, Ikai Lan has written a great post about the Java Mapper API, which takes some design cues from Hadoop’s API and includes several examples of common operations such as large scale modification of properties or batch delete.


When you’re ready to jump in and start using the tool, head over to the project homepage on Google Code. You’ll want to check out the “Getting Started” page for the language you’re using:






Happy Mapping!




- Fred, Mike, Ikai, Nick + the App Engine team

Fresh from our exciting news from Google I/O last quarter (launch of Google App Engine for Business and new product developments), we now look ahead to the mid-summer months. Since I/O, we've already appeared at a number of events including the Google DevFest Australia and Philippines, PyCon AU, and the Malaysia Open Source Conference, with many more coming this month. There are several well-known large conferences (and many smaller ones too) that our team members will be attending and/or speaking at, so hopefully we'll get to meet you at one of them!

Fresh from our exciting news from Google I/O last quarter (launch of Google App Engine for Business and new product developments), we now look ahead to the mid-summer months. Since I/O, we've already appeared at a number of events including the Google DevFest Australia and Philippines, PyCon AU, and the Malaysia Open Source Conference, with many more coming this month. There are several well-known large conferences (and many smaller ones too) that our team members will be attending and/or speaking at, so hopefully we'll get to meet you at one of them!









In addition to the ones listed above, Google will be hosting more DevFests in Asia and South America later this year. Keep an eye out for specific dates and locations on the Google DevFest home page. There are also 5 larger global Google Developer Day events in the fall. We would love to meet with App Engine developers at a DevFest, Developer Day, or any of the other conferences above!




See you around the world!






We have two scheduled upcoming maintenance periods for the Datastore:



  • July 7th, 2010 - 5pm-6pm Pacific Time (July 8th, 0:00 GMT) - Later today

  • July 14th, 2010 - 5pm-6pm Pacific Time (July 15th, 0:00 GMT)

We have two scheduled upcoming maintenance periods for the Datastore:



  • July 7th, 2010 - 5pm-6pm Pacific Time (July 8th, 0:00 GMT) - Later today

  • July 14th, 2010 - 5pm-6pm Pacific Time (July 15th, 0:00 GMT)




As part of our on going Datastore reliability work, we'd like to inform developers that during the period between the two maintenance events listed above, we are expecting that Datastore performance will be impacted and applications will see higher read/write latencies. This maintenance and the higher latency in between are an unfortunate, but necessary step in our long-term plan towards completing the work to address the datastore latency issues (as described in our recent blog post). We recognize the inconvenience to applications and their users caused by this. For this reason, we continue to offer free datastore CPU usage until we are confident we have cleared these issues.




You can read more about the currently scheduled maintenance periods on the notification to the Downtime Notify list. As a reminder, be sure to sign up with the Downtime Notify list for alerts and updates on scheduled (or unplanned) maintenance, and follow @app_engine on Twitter for ongoing announcements.

Today we are happy to announce the 1.3.5 release of the App Engine SDK for both Python and Java developers.


Due to popular demand, we have increased the throughput of the Task Queue API, from 50 reqs/sec per app to 50 reqs/sec per queue. You can also now specify the amount of storage available to the taskqueue in your app, for those with very large queues with many millions of tasks. Stay tuned for even more Task Queue scalability improvements in the future.

Today we are happy to announce the 1.3.5 release of the App Engine SDK for both Python and Java developers.


Due to popular demand, we have increased the throughput of the Task Queue API, from 50 reqs/sec per app to 50 reqs/sec per queue. You can also now specify the amount of storage available to the taskqueue in your app, for those with very large queues with many millions of tasks. Stay tuned for even more Task Queue scalability improvements in the future.


Additionally, in this release we’ve also added support for precompilation of Python source files to match the same feature we launched for Java last year. For Python, you can now use precompilation to speed up application loading time and to reduce CPU usage for new app instances. You can enable precompilation by including the following lines in your app.yaml file:


derived_file_type:

- python_precompiled

This will start offline precompilation of Python modules used by your app when you deploy your application. Currently precompliation is off by default for Python applications, but it will be enabled by default in some future release. (Java precompilation has been enabled by default since the release of 1.3.1.)


To give you a taste of what this feature is like, we tested this on a modified version of Rietveld (which included a copy of Django 1.0.4 in the app directory, and which did not use the datastore in its base url). The latency and CPU usage results for the initial load of the application, after uploading a new version of the app and requesting the homepage, were:



Before precompilation enabled:

Test 1: 1450ms 1757cpu_ms

Test 2: 1298ms 1523cpu_ms

Test 3: 1539ms 1841cpu_ms

After precompilation enabled:

Test 1: 805ms 669cpu_ms

Test 2: 861ms 702cpu_ms

Test 3: 921ms 803cpu_ms

Of course, any individual app’s performance will vary, so we recommend that you experiment with the setting for your application. Please submit your feedback and results to the support group!


In addition to Task Queues and Python precompilation, we have made a few changes to the Blobstore in 1.3.5 We have added file-like interfaces for reading Blobs. In Python, this is supported through the BlobReader class. In Java, we have implemented the BlobstoreInputStream class, which gives an InputStream view of the blobs stored in Blobstore.


More information on this release can be found by reading the release notes for Python and Java.





Hello App Engine Developers!

My name is Praveen Alavilli ( @ppalavilli) and I work as a developer evangelist for PayPal's X Platform at PayPal.com. I want to take the opportunity to introduce you to our new ...


Hello App Engine Developers!

My name is Praveen Alavilli (@ppalavilli) and I work as a developer evangelist for PayPal's X Platform at PayPal.com. I want to take the opportunity to introduce you to our new open source toolkit for Google App Engine that provides an easy way to integrate your Java apps running on App Engine with the new PayPal's Adaptive Payments API. Currently this is a Java toolkit explicitly for App Engine Java, but a Python version will be coming out soon.



Background

The Adaptive Payments API provides a set of core services offered by the PayPal X Open Global Payments Platform to enable developers to embed payments into their applications, services, and platforms. The Adaptive Payments APIs offer several new payments functionality like Split Payments and Preapprovals, that enable developers to implement a variety of monetization models - freemium, subscriptions, pay-per-use, value-added-services, micro-transactions, e-commerce, etc. in their applications built and running on the App Engine. Whether you are building an application for Businesses to process back-end disbursements or payouts to affiliates, or building a social or gaming app for Facebook / Twitter / Open Social, or building a Desktop gadget for premium content, or building a Geolocation app that only helps users find places and people around where they are but even help in transacting them, or building a Web2.0 AJAX app that mashes up content and services, or several more use cases enabled by App Engine, now you can use the PayPal X toolkit to enable payments in them as it fits the needs.



Using the toolkit

Getting started with App Engine toolkit is easy. You can either checkout the source code from svn and import it into your Eclipse project or simply download the prebuilt jar file and include it in your application's class path (/WEB-INF/lib). Similar to other APIs that you might have used, you would need PayPal API Credentials to authenticate your API requests. With the toolkit you simply create an "APICredential" object from one of your application initialization methods (in most cases from your Servlet init() method) and load the API Credentials that you have obtained from the PayPal Sandbox. (Please refer to PayPal's Sandbox guide for more detailed information on how to obtain them).

// Obtain the credentials from your configs 
credentialObj = new APICredential();
credentialObj.setAPIUsername(getServletConfig()
.getInitParameter("PPAPIUsername"));
credentialObj.setAPIPassword(getServletConfig()
.getInitParameter("PPAPIPassword"));
credentialObj.setSignature(getServletConfig()
.getInitParameter("PPAPISignature"));

// setup your AppID from X.com
credentialObj.setAppId(getServletConfig()
.getInitParameter("PPAppID"));

// setup your Test Business account email
// in most cases this would be associated with API Credentials
credentialObj.setAccountEmail(getServletConfig()
.getInitParameter("PPAccountEmail"));

// add required error condition checks
//....


Once the APICredentialObj is initialized successfully, save it in the application's local context so you do not need to reinitialize it on every request.

At its core, the Adaptive Payments API provides 5 generic API methods: Pay, Pay Details, Preapproval, Preapproval Details, Cancel Preapproval, Refunds, and Convert Currency. The toolkit provides the base API Request classes required to make those API calls. To make it even simpler, the toolkit also provides a few functional API wrapper classes that not only abstracts the APIs in terms of the functionality exposed (SimplePay, ChainedPay, ParallelPay, CreateSimplePreapproval, CreatePreapprovalForPeriodicPayments, PreapprovedChainedPay, etc.) but also provides a few exceptions that helps in handling errors more easily than the generic API Responses.



A Simple example

To give you an example, let's say you are building a SaaS model application on App Engine that you charge your customers based on their usage. While on-boarding customers to use your app, you can use the Preapproval API to obtain authorization from your customers to charge them for the app/service based on their usage in the future. In this case you can simply use the 'CreateSimplePreapproval' to create and send a request as below:
try {

// CreateSimplePreapproval request to setup a simple
// preapproval with no Payment Period Set
CreateSimplePreapproval simplePreapproval =
new CreateSimplePreapproval();

// set the API Credentials object (as given in the code above)
simplePreapproval.setCredentialObj(credentialObj);

// starting date in yyyy-MM-dd format
simplePreapproval.setStartingDate("2010-06-25");

// ending date in yyyy-MM-dd format
// in this case it's for an year from the starting date
simplePreapproval.setEndingDate("2011-06-25");

// set max total amount of all Payments
simplePreapproval.setMaxTotalAmountOfAllPayments(52.00);

// set max amount for each payment - let's say $1
simplePreapproval.setMaxAmountPerPayment(1.00);

// set max number of payments allowed - (52 weeks)
simplePreapproval.setMaxNumberOfPayments(52);

// set where to send the user in case of a cancellation
simplePreapproval.setCancelUrl(req.getRequestURL() +
"?action=preapproval&cancel=1");

// set where to return the user after successful approval
simplePreapproval.setReturnUrl(req.getRequestURL() + "?
return=1&action=preapproval&preapprovalKey=${preapprovalKey}");

/* Set other required fields */
// ... //

// set memo for user's transaction history
simplePreapproval.setMemo("Preapproval for GAE Sample");

// send the request
PreapprovalResponse preapprovalResponse =
simplePreapproval.makeRequest();

} catch (//...exceptions go here...//) {
// handle exceptions as necessary

}


In this case, when the 'AuthorizationRequiredException' is thrown, your application would need to redirect the user to PayPal for authorizing the preapproval. The 'getAuthorizationUrl' method takes care of building the PayPal authorization url along with the Preapproval Key returned by the Preapproval API. Once the user authenticates and authorizes the preapproval request on PayPal.com, the user will be redirected back to your 'returnUrl' along with the preapprovalKey, which your application can verify by using the 'PreapprovalDetailsRequest' and store it in it's own app engine data store securely. From that point onwards, whenever the user needs to be charged for their usage of the application, the application can use one of the 'PreapprovedSimplePay' or 'PreapprovedParallelPay' or 'PreapprovedChainedPay' classes to make a payment on behalf of the user. Please refer to the sample apps to understand how to use the other classes provided by the toolkit to make Parallel,Chained or Simple Payments. For eg. the PicMartServlet.java in the PicMart sample app shows how you can make a parallel payment to two receivers at the same time.



Further examples

As mentioned earlier, if you are building a social game where users can buy digital goods (eg. micro-payments) and virtual currencies while playing a game, you can use the same Preapproval API to obtain authorization from them so you could charge their PayPal accounts as and when needed, without requiring them to re-enter their payment information or redirect to PayPal for authorizing the payments.

Other examples of using the toolkit can include:

  • An application that lets merchants or enterprises pay their suppliers, or manage affiliate networks

  • Applications to enable property owners to collect rental payments from tenants

  • With a mult-merchant marketplace, a simple payroll app enables employee salary payments in multiple countries

  • ...



As you can see, the toolkit's support of the Adaptive Payments APIs can easily enable all of these usecases.



Where to find more info

Please look at the sample code provided in the samples directory. The AdaptiveRequests.java class in AdaptiveSample and AdaptiveSampleFnAPI shows how to use the helper classes to send and receive requests. The PicMart sample app provides a simple example of how the Parallel Payment can be used in a Photo Printing app that let's users to buy prints of the pictures from Picasa Album of the photographer. This sample app uses the Picasa APIs to fetch the album and picture information from Picasa.



To learn more about the PayPal X Toolkit for App Engine, please visit: http://code.google.com/p/paypalx-gae-toolkit/ and to learn more about Adaptive Payments, please visit: https://www.x.com/community/ppx/adaptive_payments. You can find a lot of resources for developers like documentation, technical spec, sample apps, code, sdks, technical forums, technical support, etc.. on PayPal X Developer Network web site (https://www.x.com).

Posted by Praveen Alavilli (@ppalavilli), PayPal X Developer Network.

This is a guest post from Jeffrey Rosen, of Wolfire Games.



Recently, Wolfire Games, joined by four other independent video game developers and two charities, put on a pretty unusual promotion: The Humble Indie Bundle. For twelve days, you could have gone to the promotional site and paid what you want (divided any way you choose among the developers, the EFF, and the Child's Play Charity) for a bundle of independent video games, all compatible with Mac, Windows, and Linux.

This is a guest post from Jeffrey Rosen, of Wolfire Games.



Recently, Wolfire Games, joined by four other independent video game developers and two charities, put on a pretty unusual promotion: The Humble Indie Bundle. For twelve days, you could have gone to the promotional site and paid what you want (divided any way you choose among the developers, the EFF, and the Child's Play Charity) for a bundle of independent video games, all compatible with Mac, Windows, and Linux.



The reason why I've been invited to write this guest blog post is because the site was built on Google App Engine and, according to Google Analytics, it served about 3.4 million pageviews (1 million unique visitors) in the 12 days it was live. It grossed over $1.2 million. Depending on who you ask, this is quite a lot of traffic, and we exceeded the free App Engine quota. Google sent us a bill for a grand total of $71.56.



I've been a long time advocate of Google App Engine on the Wolfire Blog, even when traffic was tiny, for a number of reasons. I really like its ease of use, and the fact that Google engineers, including Guido himself (the creator of Python), are the ones making sure everything works at 4 AM, and Google's infrastructure is responsible for data durability and security. The pay-as-you-go price combined with a generous free quota is icing.



As an independent video game developer, it's pretty crazy to have these resources at my disposal. While I'm confident in my Python web coding ability, I am definitely not confident in being a Linux sys-admin and commiting to an expensive server that may or may not handle the load properly (especially scary when critical data is involved). This upfront risk and steep learning curve would probably have been the straw that broke the camels back and turned the bundle into vaporware.



How did App Engine perform?



According to Google Analytics we did about 3.4 million pageviews (1 million unique visitors) during the bundle period on the Wolfire site (heavily concentrated around the first and last days). The traffic came in huge surges from Reddit, Slashdot, Digg, and a plethora of gaming and tech sites, and people seemed to really enjoy pushing the server by refreshing the page to update the real-time statistics. App Engine performed as advertised: additional servers were seamlessly brought up as needed and shut down when appropriate.



One thing that was really neat about the bundle is that it had a bunch of real-time statistics about purchases along with a leaderboard of the top contributions. There were many interesting donations, for example, $3333.33, $1337, $327.67, and $313.37 and watching the average donation go up and down was really addictive. I think being able to refresh the page to see the sales coming in was a really big draw for the bundle. This feature was really easy to implement on App Engine using a sharding counter. I'm really looking forward to native comet support in App Engine so the next bundle can have this update completely live, without polling or refreshing. Memcache and the datastore is built for stuff like this, so it was very fast.



How was developing on App Engine?



One awesome thing about App Engine is how it handles the deployment for you transparently and seamlessly. I felt comfortable developing, testing, and ultimately deploying new features to the site, even while it was serving say, 100 visitors per second. App Engine handles all the details of seamlessly bringing down the servers and getting the new ones in place even under intense loads. Because of this, I was able to hack on new features to the site as necessary. For example, I expanded the statistics so that you could see the platform specific average payments in real-time. That feature was started, tested, and deployed on maybe the third day of the bundle. Similarly, I spontaneously added an unplanned sixth game (Samorost 2) to the bundle on the fifth day and deployed minor tweaks dozens of times throughout the promo.



One thing that the bundle made clear to me is that raw Python code is much faster than you might expect. After I finished building the prototype of the bundle, I was getting ready to heavily optimize everything: each hit to the Humble Bundle page executed a few hundred lines of my Python code, doing all sorts of calculations for the statistics and inserting that into a massive Django template. Initially I thought that there would be no way this would be scalable and I'd need to aggressively cache the whole page. However, when I looked at the estimated CPU usage (tip: if you look at the request headers of a page while logged in as an admin, App Engine gives you some estimates on how much it will charge you for 1000 requests and how long it will take) it was something like 0.00375 CPM and was executing in milliseconds. Despite what I had assumed to be inefficient, unoptimized code, the CPM was so ridiculously low and it was executing so fast, I decided not to even bother optimizing the backend further. The only thing that was taking any time at all were the few memcache calls to retrieve the real-time list of tweets, real-time raw sales data, and other interesting stats. In hindsight though, I could have easily shaved off 30 ms by combining all the memcache requests into a single RPC with memcache.get_multi.



Was there anything App Engine could improve?



While I am really satisfied with App Engine and credit its existence and my familiarity with it for the existence of the Humble Indie Bundle in the first place, there were a few gotchas:



Due to a scheduled maintenance period on the datastore, the bundle was down for about 20 minutes (we need to store metadata about people's orders before we forward them to PayPal, Amazon Payments, or Google Checkout). Normally a message pops up to inform people when downtime occurs is going on, but this didn't activate for some reason, so we had to handle a lot of email about a blank page when people were trying to place their orders. I understand that Google is addressing this though.



The features that I would like to ask for are mostly all "on deck" on the roadmap: SSL for custom domains, comet support, and longer background tasks in particular. I'd also like to see App Engine focus on utilizing Google's infrastructure by serving non-static requests geographically like "web acceleration" services from CDNs. There are some trade-offs due to the nature of the datastore, but I think it can be done.



Thanks for having me on the blog! People can find me on the Wolfire Blog where we talk about more technical and game design topics, and check out our upcoming game Overgrowth.