WOW - Showing Live Temperature Data Across Europe

Read More

Today's post is contributed by Edward Hartwell Goose of PA Consulting, who is working on an App for the UK’s Met Office to report everyone's favorite bit of small talk, the weather. We hope you find the discussion of his team's experience using App Engine illuminating.

The UK’s Met Office is one of the world’s leading organisations in weather forecasting, providing reports throughout the day for the UK and the rest of the world. The weather information they provide is consumed by a variety of industries from shipping to aircraft, and powers some of the UK’s leading media organisations, such as the BBC.

Although the Met Office is the biggest provider of weather data, they aren’t the only ones collecting information. Thousands of enthusiasts worldwide collect their own weather data, from a wide variety of weather stations - either simple temperature sensors or highly sophisticated stations that rival the Met Office’s own equipment. The question of course is: how do you harness the power of this crowd?

Enter the Weather Observations Website

The Met Office and our team from PA Consulting worked together to answer this question late last year. The end result was The Weather Observations Website, or “WOW” for short. In the 3 months since launch on the 1st June 2011, WOW has recorded 5.5 million weather reports from countries throughout the world. Furthermore, we can retrieve current reports in sub second times, providing a real time map of worldwide weather. We haven’t got the whole globe covered just yet and the UK carries the most sites, but most countries in Western Europe are reporting. We also have reports from a medley of countries throughout the world, from Mauritius, Brazil and as far away as New Zealand. We even have one site reporting at regular intervals in Oman (it’s hot!).

Better yet, as a development team of 2, since launch, we’ve spent almost no time at all doing anything but casual monitoring of WOW. No one carries a pager, and the one time we did have problems (we underestimated demand, and our quota ran out), I was able to upgrade the quota in a minute or so. And I did it from my sofa. On my phone.

How good is that?

WOW - Showing Live Temperature Data Across Europe

Lessons Learnt Building for App Engine

We learnt a lot building WOW. A huge amount in fact. And we’d love to share our insights with you. We’d also love to tell you what we love about App Engine - and why you should use it too. And so you know we’re honest - we’ll tell you what we don’t like too!

Firstly - the good stuff. We think App Engine is a fantastic tool for prototyping and any team working in an agile environment. We’re big fans of SCRUM at PA Consulting, and App Engine was a dream to work with. Compared to some of our colleagues working with difficult build procedures and environments, our full release procedure never took more than 5 minutes. Better yet, App Engine’s deployment tools all hook up with ANT and CruiseControl (our continuous build system), allowing us to run unit tests and deploy new code on every check-in to our code repository. This allowed us as developers to get on with what we do best: develop!

The APIs are great too. The documentation is fantastic and is regularly updated. We use all but the Channel API at the moment (sadly, it uses Google Talk Infrastructure behind the scenes and this gets blocked by some corporate environments). If I could offer any advice to a budding App Engine developer it would be to thoroughly read the documentation, and place the following three questions and their solutions at the forefront of everything you do:

        1. Can I do it in the background? - Task Queue API

        2. Can I use the low-level API? - Datastore API

        3. Can I cache the result? - Memcache API

These three principles have given WOW the performance it has, and will allow it to scale effectively over the coming years. We’ve got some big changes coming over the next couple of months too that should provide even higher performance as well as exciting new features.

Future Developments

So, how about improvements App Engine could make? There are definitely a few, although how important they are will depend on the problems you’re trying to solve.

To begin with, improvements to some of the non technical elements are needed before App Engine becomes truly mainstream. Needing a credit card to pay is a showstopper for some organisations. Backup and restore is also missing, unless you implement it yourself. This is perfectly possible, but can add significant man days to your development effort if your data structure is complex and fast changing.

We also struggled with how to estimate quota to begin with too. One of the brilliant features of App Engine is how easy it is to spool up (and down) new instances to deal with demand. Unfortunately, this also means it can be quite easy to accidentally spool up too many instances and burn quota quickly. Although this has never affected the actual data, it can cause an unpleasant spike in the amount of dollars spent. We also had a similar problem with a MapReduce job getting stuck overnight that caused a scare the next morning. Hopefully the new monitoring API should provide a bit more visibility of these issues as well as automatic email notifications to help catch these issues.

Aside from that, other features will probably depend on the application you’re trying to build. Built in support for geo-queries would be invaluable for WOW. Currently we use an external library, but this adds some extra overhead on our development. Another common feature request is full text search which is essential for projects dealing with large text corpa. Both of these features would allow us to provide better search facilities for our users - for example search by site name or geographic location. These queries can be implemented in App Engine as it is now, but achieving optimal performance and optimal cost are difficult problems that we struggle to complete ourselves.

Final Thoughts

Overall, we’re really impressed by App Engine. The App Engine team regularly releases new versions, and although it does have limitations it has allowed us to concentrate on what really matters to us - the weather. We know WOW will scale without any problems, and we don’t have to worry about any of the hardware configuration or system administration that can easily consume time. Our small team of developers spends all of their time understanding the business problems and improving WOW.

We’re really looking forward to taking WOW forward in the future, we hope you can join us: http://wow.metoffice.gov.uk.

Edward Hartwell Goose (@edhgoose)

Developer

PA Consulting

Read More

Buenos días desde Buenos Aires!! Wow, we are just blazing through the summer! We had an exciting Google I/O in May where we saw the launch of the Go runtime, introduction of the Backends feature, and confirmed Google's commitment to the platform with our plans to leave preview mode, not to mention the five releases since that time. We then hit the road this summer with team members appearing around the globe, including mega events like the Cloud Computing Expo, EuroPython, and OSCON.



At Google I/O, many App Engine team members gave in-depth technical sessions. Alfred Fuller, an App Engine Datastore engineer, revealed how the High Replication datastore gives greater reliability and availability for users. Michael Handler, one of App Engine’s talented Site Reliability Engineers, discussed how App Engine works in production. At OSCON, the largest open source event in the world, yours truly gave a three-hour App Engine overview and workshop to help new users get up to speed, and earlier in the year at PyCon, described how Python users can avoid “vendor lock-in” via the Django-nonrel project by porting their applications from webapp to Django, allowing them to move on to or off of App Engine as they please with only minor configuration changes. We participate in events like these because we like to communicate with users to find out what they like & don’t like about the platform, and we often take their suggestions to heart!



This fall we have a full schedule, appearing at many developer events worldwide, including Google Developer Days and Google DevFests. At the Google events, we'll recap App Engine and those announcements we made at I/O, and you’ll learn how to build exciting applications using App Engine and other Google cloud technologies as well as how to build games in the cloud. Here are the dates, events, and locations that we will be visiting:



Fall 2011

Sep 19-20 - Google Developer Day Argentina - Buenos Aires - Wesley Chun, Chris Schalk

Sep 19-23 - Strata - New York - Chris Schalk

Sep 23-24 - PyCon Argentina - Junín - Wesley Chun

Sep 26-30 - Congresso Brasileiro de Software - São Paulo - Wesley Chun

Sep 29-Oct 1 - Python Brasil[7] - São Paulo - Wesley Chun

Oct 8-9 - Silicon Valley CodeCamp - Los Altos Hills - Wesley Chun

Oct 10 - Google Developer Day Russia - Moscow - Iein Valdez, Fred Sauer

Oct 18 - Google Developer Day Czech Republic - Prague - Iein Valdez, Fred Sauer

Oct 20 - Google DevFest France - Paris - Iein Valdez, Johan “Proppy“ Euphrosine

Nov 1 - Google Developer Day Japan - Tokyo - Takashi Matsuo, Johan “Proppy“ Euphrosine

Nov 8 - Google Developer Day Australia - Sydney - Chris Schalk, Johan “Proppy“ Euphrosine

Nov 12 - Google DevFest Singapore - Singapore - Chris Schalk, Wesley Chun

Nov 13 - Google Developer Day Israel - Tel-Aviv - Michael Manoochehri

Nov 16 - Google DevFest Indonesia - Jakarta - Chris Schalk, Wesley Chun

Nov 19 - Google Developer Day Germany - Berlin - Michael Manoochehri, Wesley Chun



If these aren't close enough to you, keep an eye out on this list as we'll add new events and locations as they are scheduled. A calendar with all of Google's developer events is also available. We look forward to meeting you soon!



Posted by Wesley Chun (@wescpy), Google cloud Developer Relations team

Pictarine is a photo management web application, launched in 2010, that allows people to easily manage and share all of their photos from Flickr, Picasa, Facebook, Twitter and other sites. Pictarine developers Guillaume Martin and Maxime Rafalimanana have contributed the following post discussing their experiences using Google App Engine and Google Web Toolkit. ...Read More

Pictarine is a photo management web application, launched in 2010, that allows people to easily manage and share all of their photos from Flickr, Picasa, Facebook, Twitter and other sites. Pictarine developers Guillaume Martin and Maxime Rafalimanana have contributed the following post discussing their experiences using Google App Engine and Google Web Toolkit.



From the start, we used Google technologies in developing Pictarine and we wanted to share our experience with them so far. In this post, we will shed some light on the weaknesses and strengths we found in Google Web Toolkit (GWT) and Google App Engine. We will also discuss how we leveraged GWT to build a new technology that allows Pictarine to seamlessly display photos from the computer directly into the browser. The following diagram is an overview of how our application works.

Building a mashup in the cloud with Google App Engine



The Pictarine team is made of a web designer and two developers who previously worked mainly with Java based enterprise technologies and had a little experience with web technologies. When we started the project in early 2009, we were quite open on learning new languages like Python or Ruby, but when App Engine announced that Java would be supported, we were really excited to give Google App Engine a try.



The first few months, learning about the App Engine environment was quite easy and dare I say fun. Testing our code on Google’s servers from Eclipse IDE was only one click away. So we built our first prototype fast and we quickly decided to adopt App Engine. Then we started to build the core of our application: the engine that uses the API from Flickr, Picasa, Facebook to fetch the users’ photos. This is where we hit the first limitations of App Engine. Most users have a lot of photos on these services and retrieving them can take some time. But App Engine has strict limits on how long a request should last: an outgoing HTTP request cannot last more than 10 seconds and cannot process a request for more than 30 seconds. So while building our architecture we found ourselves writing approximately one third of our code dealing with these limitations: paginating our requests, creating background tasks to store data in small batches, etc.



In early 2010, when we launched our alpha version, everything went smoothly. We had some good press coverage and App Engine met our expectations in handling our first users. During 2010, we worked on implementing new features requested by our users, and during this period of time we were really impressed by the way App Engine evolved. Many of the limitations were lifted and great new features were added. We are now able to use Task Queues for requests that last up to 10 minutes, which we fully use to sync our users’ photos and albums. One of the features we like the most is the Channel API, a push notification system that allows us to instantly show a photo in every connected browser as soon as it is uploaded.
App Engine is still not perfect but has greatly improved and when we see its roadmap, we are quite confident it will continue to improve.



Building a fresh photo experience with Google Web Toolkit 



When we started Pictarine, we wanted a fast, distraction free interface that would allow our users to focus on their photos. We wanted the interface to adapt to the screen resolution, displaying a lot of photos on large screens and fewer on small ones. We wanted it to be automatically updated when new comments or new photos are added. We wanted a web application. As we eliminated Flash quite quickly (based on our user experience...) we started to look at tools to build HTML/CSS/Javascript applications. We settled quickly on GWT: while coding in Java, with all the tools we are used to (even the debugger), we could produce optimized Javacript that would run in every browser! When we started with GWT, it was already 3 years old, so we had few complaints about it. The main issue was that we had to always keep in mind that the Java code we produced was ultimately translated to Javascript. So some Java methods, such as the Reflection API, are not allowed. Another thing that was not obvious to us when we started with GWT was that a java developer needs an intimate knowledge of HTML/CSS if he/she wants to go beyond the basic user interface provided by the GWT widgets.



What we really like about GWT in our architecture is the ability to share code between client and server: we can use the same Photo or Album class on the client and the server and the GWT RPC system allows us to automatically share the same Java object on both side. We can also have the same data validation code on both sides: we can alert the user immediately on errors and still validate the data on the server just in case.



Another great feature we like about GWT is its handling of internationalisation. From the beginning we wanted to build a website available for all Internet users, so supporting English as well as our native language (French) was almost obligatory. Fortunately, GWT makes it really easy to generate centralized localization files so that we just have to translate.



Finally, to illustrate how great Javascript generation is, when IE9 came out, we waited a few weeks for GWT to support it and our application was compatible after a recompile! Of course, the IE9 team also did a good job with their HTML5/CSS3 engine.



Building an universal uploader 



After the launch of our alpha in 2010, our users were able to see and share their photos from Flickr, Picasa, Facebook. But they still had to put their photos on these websites first before coming to Pictarine. This limitation quickly became the first request on our feedback system. We needed to let our users do everything from Pictarine, including uploading photos. Uploading many photos from a website is still not a trivial process. Most websites choose Flash to allow users to upload multiple files at once, but our experience with it was that it often crashed after a while. Some use Java applets, but they are never well integrated and always look odd. At Pictarine we decided to tackle this problem by using Java Applet for their stability across all platforms but without using it to render photos or folders.



We have built a technology that uses the GWT RPC mechanism to talk to a Java Applet: photos, upload progression are rendered in HTML/CSS and the applet takes care of photos resizing and uploading. Sharing a photo from a camera is now a one-step process. This technology also allows users to browse their local files directly in their browser and it is fully integrated in our design.



We believe that this new use of Java applets can help blur the line between the Desktop and the Cloud by seamlessly integrating desktop files in any web application.

In conclusion, we can say that we are really happy with the choices we made with App Engine and GWT. App Engine is a great service that perfectly handled the spike in traffic we saw right after articles on Mashable and Lifehacker were published. So we recommend it to every lean startup out there who loves developing in Java, Python or Go.

Posted by The App Engine Team

It’s been a busy four weeks for us (you may have heard), but we are still on track with our new monthly release schedule. Today we have new SDKs for you, with some new features as well as some bug fixes.

Read More

It’s been a busy four weeks for us (you may have heard), but we are still on track with our new monthly release schedule. Today we have new SDKs for you, with some new features as well as some bug fixes.

Overall Changes

• Blobstore API - We’ve introduced an option to specify a limit on your blob upload size. This feature will allow you to expose blob uploads to your users, while still being able to ensure that their uploads won't exceed your preferred limits.


• Datastore Query Improvements - We’re also continuing our theme from the past few releases to tune our Datastore query planner to give users more flexibility. Starting with 1.5.4, queries with equality filters on multiple properties will now continue scanning up to the 30 second Datastore query deadline. Many of these queries that used to generate an error due to inefficient indexes will now succeed.


• Datastore Write Ops in the SDK - In response to the feedback from our recent Side By Side Bills release, we are now displaying the number of write ops needed to store an entity in the SDK dataviewer. Write ops include both the entity write as well as the index writes that are executed when an entity is added. You can always reduce the number of write ops by turning your indexed properties into unindexed properties, just make sure you don’t reference those properties in any of your queries! (Java, Python).

Java

• Prospective Search API - We’ve released the experimental Java version of our Prospective Search API. Prospective Search lets you detect and take action on datastore entities that match certain criteria when they are written.

Python

• Memcache - You can now make calls asynchronously with the Memcache API. With Asynchronous Memcache, your application does not need to block on calls to the Memcache API and can continue processing a request instead of waiting for a response from Memcache. Fear not Java friends, we’ll have matching functionality for you in an upcoming release.

Full release notes can be found in the usual location (Java, Python), and we’re always listening for your feedback in the groups. Happy coding!

Posted by The App Engine Team

Last week we rolled out side-by-side billing to give you a more detailed preview of how you’ll be affected by the price changes that we announced in May. We received a variety of feedback and have made a few important changes based on it. Our intent is to be as open and transparent about the changes as possible and to give you enough time to prepare. In that spirit, our Engineering Director ...Read More

Last week we rolled out side-by-side billing to give you a more detailed preview of how you’ll be affected by the price changes that we announced in May. We received a variety of feedback and have made a few important changes based on it. Our intent is to be as open and transparent about the changes as possible and to give you enough time to prepare. In that spirit, our Engineering Director has also shared some of his personal thoughts.



We understand that the new rates surprised some of you. We’ve been listening closely to your feedback, and we wanted to share an update on the changes we’re making to help ensure you have an accurate picture of how the new pricing will affect your app. Although prices will increase, we’re confident that you’ll find App Engine still provides great value.



Based on your feedback we’re taking the following steps:

• Extended review period: We’re now giving you almost eight weeks before introducing the new pricing. You now have until November 1 to configure and tune your application to manage your costs. 


• Increased free Instance Hours: We are increasing the number of free Instance Hours from 24 to 28. This means that people who are using a free app to try out App Engine can run a single instance all day with a few spikes and still remain below our free quota.  This will be reflected in the comparison bills soon.


• Extended discount: We’ll continue to offer the 50% discount on instance prices until December 1st, at which time Python 2.7 should be available. Python 2.7 will include support for concurrent requests, which could further lower your costs. 


• Faster Usage Reports: We appreciate the importance of quickly being able to see the effect your tuning has on your bill and starting today we’ll provide your Usage Report (and the included comparison bills) within one day instead of the previous three. 


• Better analysis tools: We are working on better ways for you to model the cost of your apps. We will add the “billing” line into the instances graph on the Admin Console. We’re adding datastore billing information into the dev console to making it easier for you to track how the changes you make affect your bill, which should also help lower the cost. 


• Premier accounts: we know a lot of our customers are eagerly awaiting Premier accounts to get operational support, offline billing, unlimited accounts, and the SLA. So we will not wait until November 1st for this, but rather launch Premier accounts as soon as possible.  If you are interested in a Premier account, please contact us at appengine_premier_requests@google.com.

We also wanted to share some of the main ways for you to lower your bill and get a better sense of the true cost of App Engine:

• Set Max Idle instances: Setting Max Idle Instances to a lower level will help lower your costs as we will only charge for idle instances up to the maximum you set. This could impact your performance so it’s worth reading up on the ramifications. 


• Always-On reflected in bills: Currently the side-by-side bills still include the cost of always-on even though it will be retired when the new pricing launches (to be replaced by min idle instances). We’re working on a fix for this. Until then you can comfortably subtract 48 instance hours per day from the estimate. 


• Reserved instance hours: The simplest way to lower the charge for instance hours is to consider using reserved instance hours. They are 37.5% cheaper than on-demand, but you do need to commit to a certain number of them over the course of a week. 


• Managing resources: Check out this article, which provides more helpful advice on how to efficiently manage your resources and lower costs. 

We launched App Engine in preview three years ago to make it easier for you to build, maintain and scale web applications. Since then, we’ve added many features like Java and Go language support, created a High Replication Datastore, and added many other APIs. And there’s a lot more cool stuff to come. It’s heartening to us that so many developers- 150,000 and counting- have chosen to use App Engine to run their apps. While we can’t continue to offer App Engine at our original prices, we can commit to listening to your feedback, acting on it, and working hard to give you a great platform for your apps at the most competitive price possible. If you find that’s not the case, or have any questions about this at all, please feel free to contact us at appengine_updated_pricing@google.com.



The App Engine Team