Just in time for spring, we have a new App Engine release to bring our Java and Python runtimes even closer to parity. For Python, we’ve launched a test library to match the existing Java testing framework, and for Java we’ve introduced the Deferred and Remote APIs. This release also introduces a new Blobstore writing feature, the experimental release of the Prospective Search API in Python, and a few goodies for Task Queue and Cron users.

Just in time for spring, we have a new App Engine release to bring our Java and Python runtimes even closer to parity. For Python, we’ve launched a test library to match the existing Java testing framework, and for Java we’ve introduced the Deferred and Remote APIs. This release also introduces a new Blobstore writing feature, the experimental release of the Prospective Search API in Python, and a few goodies for Task Queue and Cron users.


Python



  • Prospective Search API: The experimental Prospective Search API allows Python runtime users to detect and take action on datastore entities that match certain criteria when they are written. For the experimental release, users will be allowed 10,000 subscriptions with the Prospective Search API. Pricing will be announced once the feature is fully launched.

  • Testbed Unit Test Framework: The Testbed suite for Python provides an easy interface for using App Engine API stubs in integration tests similar to the previously existing Java Testing Framework. You can create tests for your application that do not rely on calling App Engine production services, which speeds up the time your tests take to complete, and eliminates dependencies for your test on external services. Here at Google, we’re well known for our testing culture, and we hope that this API will allow you to develop more stable code more quickly.


Java



  • Concurrent Requests: Until now, Java applications relied on starting additional instances to dynamically scale up for higher traffic levels. Now with support for concurrent requests, each application instance may serve multiple user requests at the same time. To start, ensure your application’s code is threadsafe, then enable concurrent requests by adding the <threadsafe> flag to your appengine-web.xml.

  • Java Remote API and Deferred API support: The Remote API and Deferred API libraries have been supported in Python for awhile, and now they can be used with Java, too! The Remote API allows you to perform operations on your application’s datastore from your local machine. This is particularly useful for work that is not well-suited to App Engine’s request/response model. The Deferred API allows users to more easily write and execute ad hoc tasks. Our docs contain more information and examples on how to use the Remote API and Deferred API in Java App Engine.


New and changed APIs



  • Files API: The new Files API in Python and Java allow you to programatically read and write data using Blobstore. This API can be used to generate reports, export data, or do anything that your heart desires that requires large, binary objects.


Serving Changes



  • Task Queue and Cron update: We’ve addressed some of your top requested items for Task Queues and Cron. With this release, you can now configure the specific application version to which a task queue or cron job will send requests. For those wanting to schedule cron jobs with a range syntax like “every 5 minutes from 11:00 to 17:00”, that too is available in the 1.4.3 release. Last, but not least, the Admin Console Task Queues page now displays a more accurate estimate of queue size for queues containing more than 2000 tasks.


Coming soon


Finally, we have a pre-announcement about the 1.4.4 release. In 1.4.0, we introduced a feature that allowed users to download code that they’ve deployed to App Engine. In 1.4.2 we released admin roles allowing for Owners. When 1.4.4 is released, we plan on including a change that allows both the person who uploaded the code to download it, as well as the Owner(s) of the project (as listed in the Admin Console). In preparation for this, be sure to properly assign roles to all developers on your App Engine application in the Admin Console. Alternatively, you will still be able to permanently disable the code download feature for your application.


That’s it for now, for additional information read the full release notes, including all new features and issues fixed for Java and Python. All feedback is welcome and encouraged in our groups.


Posted by Greg D'alesandre, The App Engine Team






















Posted by Dave Tucker

Director of Product Development

WebFilings (www.webfilings.com)




WebFilings delivers a revolutionary cloud-based application that enables companies to collaboratively build and file financial reports with the SEC.
Built on the Google App Engine platform, companies of all sizes use WebFilings software to take control of their financial reporting processes. WebFilings greatly reduces the time and stress required to file financial reports with the SEC.




Enterprise Requirements

As an enterprise class application supporting many Fortune 100 customers with strict financial reporting requirements, we simply cannot afford to have maintenance outages or service disruptions. Google App Engine has proven to be a robust platform for us to build on and has allowed us to scale seamlessly to support a rapid increase in customer demand. We’re pleased to report that a recent innovation by Google has allowed us to achieve an even higher level of availability and performance.




This recent innovation is the High Replication Datastore or HR Datastore. HR Datastore allows us keep our application live even during system upgrades and other maintenance operations that Google may be performing. In addition, it delivers a new level of reliability that has given our application significantly more consistent response times and 100% availability since making the move.




Decisions... Decisions...

We were originally a bit hesitant to move to HR Datastore because it was a relatively new offering from the App Engine team. Even though the underlying technology has been around for a long time, we weren’t sure initially how the eventual consistency model might affect our application. Well, we did our homework and read the Google posts on HR Datastore, followed the App Engine news groups, pinged Googlers via the App Engine Office Hours Chat on IRC, and did some prototyping with entity groups and eventually consistent queries. From this we got a better handle on how entity groups worked to enforce strong consistency with ancestor queries and also learned of the potential performance issues of doing large parallel db.get() operations across multiple entity groups. We also reviewed our code, and made a few changes to our application where we needed to use entity groups to have strong consistency. In the end, we found the move to be much less work than we had anticipated.




The Migration

The actual migration to HR Datastore was a fairly simple process. Essentially, we just set up a a new appspot instance, and selected the High Replication Datastore option during creation. Then we migrated our data to the new instance.


During our migration process we:

  1. Created our new instance and deployed a new version of code to work with the new entity groups
  2. Scheduled our maintenance window with our customers
  3. Put our current master/slave instance into read-only mode
  4. Copied the data from our master/slave instance to our HR Datastore instance. (Due to the size of our data, this was accomplished with some help from the App Engine team)
  5. Ran a migration mapreduce to iterate over a few entities that now needed to be in entity groups. These were saved with a new parent making them part of the entity group. This enables queries to be consistent.
  6. Implemented a site alias that routes customers to the new instance, webfilings-hrd.appspot.com, from webfilngs.appspot.com. Note: This feature is not generally available so you should contact the App Engine team directly if you need similar functionality.

In short, we went through a complete release and testing cycle and got our application migrated to HR Datastore in under a month. The actual migration happened overnight when we moved a litte over 200GB of data to our new HR Datastore instance, executed our migration mapreduce, and did our verification testing in about five hours. Since moving to HR Datastore, we are wondering why we ever even thought twice. Our customers and our Customer Success Team that works with customers every day have given us feedback that the application is performing better than ever and we have had no downtime since the move.




A Solid Choice

The High Replication Datastore was designed with the needs of large enterprises in mind. HR Datastore provides maximum uptime, multiple datacenter redundancy, consistent response times, along with scalability and performance that has met the needs of our fast growing company. It does take some planning to make the move, however. You should be prepared to review your existing code and test thoroughly beforehand. We found that this effort alone had a significant payback in terms of overall application performance and reliability. Even though the average latency on HR Datastore calls can sometimes be a little bit longer, we were able to optimize and end up with an overall experience that was better for the end user.

If you have an application that has these same demands then my only question for you is “When are you moving to HR Datastore?”. If you are like us, you will be very glad you took advantage of this latest innovation from the Google team.



Posted by Dave Tucker

Director of Product Development

WebFilings (www.webfilings.com)




SpotCloud.com is the realization of a dream originating more than 7 years ago when we first started Enomaly. SpotCloud was born from a desire to change the way cloud computing is bought and sold for the better by overcoming the inefficiencies (including large capital commitments and low utilization rates) that beset the traditional models. SpotCloud is the first multi-supplier, multi-buyer model of cloud computing, all built on Google App Engine.




SpotCloud.com is the realization of a dream originating more than 7 years ago when we first started Enomaly. SpotCloud was born from a desire to change the way cloud computing is bought and sold for the better by overcoming the inefficiencies (including large capital commitments and low utilization rates) that beset the traditional models. SpotCloud is the first multi-supplier, multi-buyer model of cloud computing, all built on Google App Engine.





SpotCloud is the first marketplace where service providers can sell their unutilized or under-utilized IaaS cloud services, and where buyers can shop competitively for these services on the basis of quality, price, and geography across a federated global pool of capacity with a single, consistent set of access and management mechanisms.



SpotCloud provides its users with the following main services:

  • VM image repository & VM image provisioning services
  • REST API for cloud service users, with management and monitoring functionality
  • REST API for cloud service providers
  • REST client to access service provider API endpoints
  • Single-point unified access & authentication to all underlying cloud services
  • Federated management portal, providing management & monitoring over all connected cloud services
  • Cloud service quality, reliability, and performance measurement instrumentation and rating
  • Runtime management service (to enable time-limited instances)
  • Billing, payment, and settlement services



Why Google App Engine?

We realized early on that a traditional data center infrastructure -- even one built on our own ECP technology -- was not a fit for a platform which needs to be global from day one. We also had a strong preference for Python, given that large parts of our existing IaaS software stack is built using it. Cost was also very important: SpotCloud needs to manage potentially very large Virtual Machines, delivering them to a globally distributed group of providers and managing them in near-real time across this extended footprint. With Google App Engine we have the power, flexibility, and global breadth of the Google infrastructure at our disposal. This was a key selling feature for us. Also the cost was practically impossible to beat.



How App Engine powers SpotCloud

As you can see from the list of services provided by SpotCloud, most of our application actually runs in the background. This is made possible by App Engine task queues and cron jobs. We need to poll for stage changes across hundreds (soon thousands) of service providers -- we need a near-real-time view of instance states, available virtual hardware profiles, cloud utilization data, etc. These things are triggered by cron schedules. Each of these tasks goes into a queue. We're able to execute these tasks to allow us to keep our view of service provider resources up to date without interrupting the front-end UI and API workflows.



Another challenge is executing tasks at a particular point in the future. Running instances deployed out to a provider through SpotCloud may expire, and renewal may or may not be allowed depending on the seller’s settings. So we need a way to terminate them if their time is up. We can use the task queue API to execute this in the future. We're able to ensure our data is consistent by adding tasks to the queue in a transactional way.



Buyers upload the appliances they want to provision to the market. So we obviously have a big storage requirement. App Engine’s data store allows us to do this in a highly performant, cost-effective manner. SpotCloud can then distribute these appliances out to providers on demand. We're able to do this because of the low-cost bandwidth Google provides.



We're big fans of Django at Enomaly, and we use it in SpotCloud. We've been able to loosely couple Django with the App Engine data store, along with other App Engine components and services like urlfetch. This means that we're not "locked-in" with App Engine. The App Engine API is structured in such a way that we're able to extend any web framework we want with App Engine components that help us scale.



SpotCloud Success

Since announcing the SpotCloud private beta at the end of last year, we’ve been overwhelmed by the amount of interest we’ve gathered, and have had large numbers of service providers from all over the globe join our beta. We now have more than 10,000 servers from 40+ countries represented in the marketplace.



Since we opened the marketplace to buyers and sellers early in 2011, the market has been off to a roaring start -- as noted by The Economist, ComputerWorld, and many others. Google App Engine has performed up to our (high!) expectations, serving an ever-increasing transaction rate in the market without a hiccup. We’re looking forward, as the market ramps up even further, to seeing if we can make App Engine break a sweat! Sign up today as a cloud capacity buyer or seller at www.spotcloud.com.



Posted by Reuven Cohen, Founder & Chief Technologist, Enomaly Inc.
Labels:













This post is another entry in our ongoing series of guest posts contributed by App Engine developers. Today we partner with Jason Collins and Shawn Rusaw of VendAsta Technologies who discuss their simplistic workflow management system for Python App Engine called Fantasm. Fantasm is very simple to start using but incredibly powerful. So powerful in fact, that they couldn't fit it into one blog post. So for more detail, see the Implementing Workflows on App Engine with Fantasm article in our documentation.




Most software systems of reasonable size need to implement workflows: a series of processing steps and decision points over an artifact, typically a document. You are likely using workflows even if you don't realize it; often processing is built up iteratively over time until you're left with a complex system with lots of subtleties. These systems can mysterious and difficult to manage or extend.




A formal workflow engine can offer help in these instances. It allows developers to break the steps and decision points into manageable and testable chunks, so that processing is predictable and measurable. Workflow engines can offer visibility into system operations and metrics around execution times and failure points. They provide retry mechanisms and allow tasks to be distributed among multiple computers.




We have developed Fantasm to be just such a workflow engine. Fantasm is a Python library that allows you to specify your tasks and steps between those tasks in a YAML configuration file. It uses a finite state machine model where the tasks are states and the steps are transitions. As a developer, you implement the code that executes inside of a state and are only responsible to return an event that is used to identify the appropriate transition to the next state. It is similar to the App Engine Pipeline API but is more simplistic and serves as a user-friendly introduction to the concept of workflows in App Engine apps.




Fantasm has become the central processing tool within our organization and we hope it can be of help to yours. For a more in-depth look at Fantasm (including examples!), please see the full Implementing Workflows on App Engine with Fantasm article in the App Engine documentation.




Posted by Wesley Chun, App Engine team

Labels:

[Cross-posted from the Europe Developer Blog]


As you may have heard, the UK is gearing up for a very special wedding on April 29, when Prince William and Catherine Middleton will be married in Westminster Abbey in London.

[Cross-posted from the Europe Developer Blog]


As you may have heard, the UK is gearing up for a very special wedding on April 29, when Prince William and Catherine Middleton will be married in Westminster Abbey in London.


Unlike many previous Royal Weddings, this event will have its own website – and we're honoured that St. James’s Palace has chosen to use Google’s computing infrastructure to power the site.


The site, which just went live at www.officialroyalwedding2011.org, is hosted on Google App Engine, which allows developers easily to build and host their web applications on Google's own computing infrastructure. It's a great way to run apps quickly, more securely, and at scale, which makes it ideal for such an important national occasion. The site will be regularly updated by St. James’s Palace in the run up to the wedding day.


This was a great team effort between Accenture, who built the site, the web design agency Reading Room who led on the design and creative work, and the App Engine team.


We hope it proves to be the perfect marriage of tradition and modernity as we approach the big day.