Nate Silva

Jun 24 2008 ∞

My thoughts on Google App Engine

I recently tried Google App Engine.

Pros and cons

Pros

  • The canned environment is really nice. It’s great to focus on development instead of setting up infrastructure.
  • Love Python. Like the Django-based templating engine.
  • Bandwidth allocation is reasonable for small- or medium-sized apps.
  • The data store is great!
    • Because it’s non-relational there’s no ORM layer to get in the way. Once you understand it, it’s intuitive.
    • Reading from the datastore is nice and fast. (But not writing — see next point.)

Cons

  • The data store sucks!
    • There’s no way to import/export data.
      • So you have to write your own.
    • Writing to the datastore is extremely slow, so if you write your own import routine, you need to break up your imports so they don’t time out. It took a couple hours to manually load around 2,000 items.
    • The Bulk Data Upload utility is essentially useless, because it can’t handle Unicode and it times out when importing large amounts of data. If I wasn’t importing large amounts of data, I wouldn’t need a bulk upload utility, no?
  • During beta, the quotas are so restrictive that it’s only useful for trivial applications at this time.
    • You can’t retrieve more than 1,000 rows from a query (and doing so would probably time-out your request).
    • Apps are limited to 500 MB total storage and 1,000 files.
    • Developers can only create three apps and you can’t delete or rename an app, so use your three wishes wisely.

Other

  • If you’re using Google Apps For Your Domain to authenticate, you need to separately set the app up in your domain, which is poorly documented. It seems obvious, but it was confusing when I tried it.

Moving apps to GAE

I considered moving three in-house apps to GAE, but none were feasible.

Build query tool

The first is a build query tool. It lists which components are present in our software builds, organized by category.

Builds have anywhere from 10 to 1,500 components. Our users sometimes pull up the larger builds. The list loads quickly because it’s plain text.

With GAE there’s no way to retrieve it (1,000 row limit) and even if there were, it would probably time out. We could get around this by introducing a search feature, and showing just subsets of the data. But we would lose the ability to browse over the entire contents of a build at once.

And then there’s the nightmare of updating the datastore. Let’s say we add a new build that incorporates a few hundred components. How are we going to update the datastore in an automated fashion? It’s certainly possible with HTTP POSTs but it would take ages because of timeout issues and how slow it is to write to the datastore.

File upload utility

The second is an upload utility. Files are uploaded and then “processed,” which involves storing them into an Amazon S3 bucket and writing a database record.

This was out of the question due to the 10 MB file size limit. And the 500 MB overall limit might be an issue — we occasionally get giant files up to ~750 MB. (The 1,000 maximum file count is fine; once files are processed they would be deleted from the GAE application space.)

Finally, getting a responsive progress bar during an upload is a tricky problem that generally requires some help from the server-side. I don’t know how to do it with GAE, or if it’s even possible. Of course it’s possible to upload without a progress bar, but users need the reassurance of a progress bar when dealing with huge files.

Historical data utility

The third is a historical data utility. It contains old (going back to 1995!) customer records, notes and invoices. These come from an obsolete accounting system that has been decommissioned, so the web UI is the only way our users can get to it. While not frequently used, it is sometimes handy to pull up old data and chart trends.

Application wise, this should work with App Engine. The row, data size and other limitations don’t come into play.

Unfortunately, there are a few hundred thousand rows in the database. It’s around 125 MB so it would not hit quotas but uploading that data could take weeks.

Summary

Bottom line: I would love to use Google App Engine, especially for in-house apps where we can authenticate against our Google Apps domain. But it’s too limited right now — at least for the types of apps I tried. I’m still on the lookout for other apps that might be a better fit.

Page 1 of 1