cloud-test: April 2008

Tips on writing scalable apps

Monday, April 28, 2008

Posted by Ken Ashcraft, Software Engineer

Avoid contention on datastore entities. If every request to your app reads or writes a particular entity, latency will increase as your traffic goes up because reads and writes on a given entity are sequential. One example construct you should avoid at all costs is the gobal counter, i.e. an entity that keeps track of a count and is updated or read on every request. There are some interesting ways of simulating this behavior that don't require reads/writes on every request, and we'll talk about a handy way to cache entities for reads below.

Avoid large entity groups. Any two entities that share a common ancestor belong to the same entity groups. All writes to an entity group are sequential, so large entity groups can bog down popular apps quickly if there are a lot of writes to that group. Instead, use small, localized groups in your design.

Write sparingly. Writes are more expensive than reads; keep this in mind when designing your data model. If you can avoid a write--it's best to do so.

Define a main() function for code reuse. Instances of your app are kept running for a certain period after each request, so there's a chance that any request will hit an already-running app. Apps that define main as in the following example:


def main(): 

   application = webapp.WSGIApplication(_URLS, debug=True)

   wsgiref.handlers.CGIHandler().run(application)



if __name__ == '__main__':

   main()

can reuse global variables, so although there's no guarantee that any given request will hit a running app instance, you can take advantage of this to non-deterministically cache entities and other expensive values. This works simply because App Engine will detect if main() is present in an already-loaded script and call it instead of re-importing.

Profile your code. You can use the Python profiler to profile your code's performance.

Google Group

Google

Google App Engine + Google Data APIs: A Match Made on the Web

Wednesday, April 23, 2008

Posted by Marzia Niccolai, Google App Engine Team
Google Data APIsPython client libraryAuthSubarticleGoogle App Engine sample code

Google

Google App Engine Hack-a-thons!

Monday, April 21, 2008

Posted by Dick Wall, Google App Engine TeamLearn about Google App EngineBuild With Us, or Build Your OwnWhat Do I Need?When and Where?New York Hack-a-thonSan Francisco Hack-a-thon

Google

Google App Engine Helper for Django

Friday, April 18, 2008

Posted by Matt Brown, Google App Engine Team

The ability to use most manage.py commands

A BaseModel class that appears the same as the standard Django Model class.

The ability to serialize and deserialize model instances to JSON, YAML and XML.

Access to Django's test framework with a test datastore and support for fixtures.

http://code.google.com/p/google-app-engine-django

Google

Getting your data on, and off, of Google App Engine

Monday, April 14, 2008

Posted by Kevin Gibbs, Software Engineer
One upcoming feature of Google App Engine that's worth mentioning is large-scale data import and export. We know that many of you have large apps that already run on other platforms, and getting all of that data into the Datastore API is a challenge. And, conversely, we also know that a lot of you want to make sure it's easy to move off Google App Engine in the future, in case you need to switch to a different platform.

With Google App Engine, you own all the data in your app. As stated in our terms, you always have the right to get your data out of Google App Engine at any point. We wouldn't have it any other way.

To that end, we would like to solicit feedback on what form you would like the data exporter feature to take. We've started a thread in the discussion group about this, and we'd love to hear what you think. XML output? CSV transform? AtomPub? RDF? Let us know!

Google

We're up and running!

Friday, April 11, 2008

Posted by Pete Koomen, Product Manager
Google App Engine has been live for several days now and we're thrilled with the response we've gotten so far! We filled our initial batch of accounts within a few hours after Monday night's Campfire One. And today, we've just invited another 10,000 developers to try it out.

We're excited to see the beginnings of a developer community forming in our Google Groups! We're actively reading your posts and responding with help where we can, and it's great to see some of the more experienced members of the community start helping others out too!

We've also seen a lot of feature and bug reports on our Issue Tracker, including requests for Ruby, Java, Perl, PHP, and Fortran. Here are are some of the general areas we're focusing on right now:

Support for more languages. We're obviously huge Python fans, but we recognize that there are other great languages out there that developers use to build web applications.

Support for offline processing. Right now Google App Engine is great for web apps that do all of their processing in response to user requests, but what about apps that need to perform scheduled tasks or larger-scale data migration? We'd like to support those apps too.

Support for large files. Google App Engine currently imposes a limit of 1MB on all requests, both inbound and outbound. We're interested in providing efficient support for much larger uploads and downloads.

Billing for additional quota. During the preview release period, all apps are restricted to a set of free resource quotas. Although Google App Engine will always be free to get started, we plan on allowing developers to purchase additional resources in the future, while paying only for what they use.

Judging by the the feedback we've gotten so far, these are areas you're interested in as well--we'll keep you posted. Keep letting us know what you think about the Google App Engine and where you think it should be headed!

Google

Introducing Google App Engine + our new blog

Monday, April 7, 2008

Posted by Paul McDonald, Product Manager
At tonight's Campfire One we launched a preview release of Google App Engine -- a developer tool that enables you to run your web applications on Google's infrastructure. The goal is to make it easy to get started with a new web app, and then make it easy to scale when that app reaches the point where it's receiving significant traffic and has millions of users.

Google App Engine gives you access to the same building blocks that Google uses for its own applications, making it easier to build an application that runs reliably, even under heavy load and with large amounts of data. The development environment includes the following features:

Dynamic webserving, with full support of common web technologies

Persistent storage (powered by Bigtable and GFS with queries, sorting, and transactions)

Automatic scaling and load balancing

Google APIs for authenticating users and sending email

Fully featured local development environment

Google App Engine packages these building blocks and takes care of the infrastructure stack, leaving you more time to focus on writing code and improving your application.

Today's launch is a preview release -- we're by no means feature-complete, and we're giving you early access because we really want your feedback. This preview of Google App Engine is available for the first 10,000 developers who sign up, and we plan to increase that number in near future.

During this preview period, applications are limited to 500MB of storage, 200M megacycles of CPU per day, and 10GB bandwidth per day. We expect most applications will be able to serve around 5 million pageviews per month. In the future, these limited quotas will remain free, and developers will be able to purchase additional resources as needed.

If you'd like to try it out, sign up for access to the preview release and then download the SDK to get started. The best place to give us feedback is in the discussion group. Future announcements from us will appear there and/or this blog.