Reuters is carrying a article today talking about the fact that Microsoft is getting ready for customers to move to the cloud for all sorts of applications. Not surprisingly, they figure the first major migration will be Exchange mailboxes and that they really don’t care if someone buys the software or pays a monthly fee to host it on Microsoft’s infrastructure. Given that Exchange hosting in the cloud has been available for years from a whole range of companies, this isn’t much of a stretch. I’ve never worked with a company that didn’t hate all the hassle of hosting their own Exchange farm of computers.

The real interesting question is what they plan to do about Microsoft Office. Granted, cloud migration of major desktop applications requires a persistent, fast online connection but it is almost a guaranteed outcome of the current trend. Throw in offline capability like Adobe Air and it won’t be long before most of us are creating, storing and sharing our documents on the net.

In a somewhat related article, The Guardian reports that a US court has ruled that border agents can search your laptop as your cross into the United States. British border security has a similar option. While there are lots of options for encrypting and hiding information, the whole issue sort of becomes moot if you have nothing on your laptop in the first place and everything is up on the cloud. Even better if that cloud is in a country that doesn’t respond to search warrants! Not that we’d ever do anything illegal of course…it is just the point of the matter.

As a heavy user of databases, both in my smaller ventures like TravelGator and also in my consulting work for large financial services companies helping them figure out how to do nasty jobs like trade surveillance, I’ve spent hours (days?) trying to figure out how to load, analyze and extract data from large databases in an efficient manner (and before the market open the next day). Unfortunately, traditional databases like Oracle and MySQL make this job very difficult . They required significant of tuning to get queries to run fast and are completely inadequate for anything that is adhoc in nature. Even the industry leading ETL frameworks have all kinds of limitations when it comes to loading large datasets that require a lot of cleaning and normalization.

So it is fun to see that startups haven’t completely given up on trying to do the database better, even given the large players in the space like Oracle, SAS and others.. Aster has released a product that came out of a Stanford PhD project (why is it always Sanford…?) that is attempting to do for databases what Google has done for search - make the database scale horizontally to thousands of generic, off the shelf servers.

Apparently they are already working with MySpace to load and analyze 100 terabytes of data that they’ve collected from the users. If this thing works it might help move the database out of the Oracle era of small incremental improvements and into the future.

Thanks to Read Write Web for bringing them to my attention.

Techcrunch did a little digging into the Amazon earnings call and has come to the conclusion that a large percentage of the computing power being deployed by Amazon web services is being used by large corporations, not startups.

While I didn’t validate this analysis, I’m willing to take it at face value. That being said, it is really surprising. The banks I’ve consulted for over the last decade or so would rather cut off their left arm than expose their data externally. So this situation has me really curious…what exactly are they doing on the Amazon web services platform and how did they get permission from IT Security, IT Risk, Compliance, and Legal to do it? (Seriously, that is what would be required.)

I’m looking forward to more information becoming public on the types of analysis that is being performed. Call me puzzled.

You know a trend is about to go mainstream when Microsoft announces an initiative to support it. This morning’s NYT article reminded me of Bill Gates’ mid-’90s epiphany that the Internet was going to be important. Better late than never, I guess, and you have to give Microsoft credit for elbowing their way into significant (if not leading) market shares after arriving late to the party.

Cloud computing is a particularly interesting reversal for Microsoft, and one that makes me feel somewhat ambivalent. The company’s leadership has always been PC-centric, and while this made them look archaic when they resisted the Internet wave, there was always a “power to the little people” aspect of their approach that I found appealing. Keep in mind that these are people who grew up in an era when nobody owned their own computer, if you wanted to use one you had to go to your nearest mainframe and get in line for your share of its precious processing cycles.

Of course in most ways the Internet actually increased the power of individuals. By giving anyone with a cheap PC the ability to not only consume but produce content across nearly any physical boundaries, it’s hard not to argue we’ve been liberated by it. And while the worries about electronic privacy have moved beyond the curmudgeons and into a small (though vocal) concerned minority, nobody seriously argues that the risks outweigh the benefits.

Cloud computing somehow feels different, and carries many echoes of our time-sharing mainframe past. While most consumers have no idea what a mainframe is (and to an ordinary user, a “grid” or “cloud” is just a new kind of mainframe), I wonder whether its implications will mesh well with basic human nature, or at least the Western materialistic version of it that I’m familiar with.

Why is iTunes so much more successful than Rhapsody or Pandora? Why is the adoption of Google Apps so slow despite its many advantages? Why do people buy vacation homes that they only visit for one or two weeks a year, when it would be far more economical (never mind convenient) to rent them? When I moved to Manhattan in 2001, it didn’t take long to realize I could rent a car as often as I wanted and it would cost me less than insurance and garage fees if I kept my own. But even though I sold my car, I have many friends who kept theirs, and I often envy them.

Why do we like to own things when there are clear financial and logistical advantages to using shared resources? There are plenty of possible answers, many of them emotional ones like control, security, or pride of possession. I don’t mean to delve into the human psyche too deeply, I only wonder how this quirk of our natures will affect, or be affected by the apparent swing of the computing pendulum back to large communal computing resources. If we’re lucky, we can even have the “thin client” debates of the ’90s all over again.

Not exactly breaking news but I found it interesting that yet another cloud database provider has joined the mix. QuickBase has launched a developers program that promises to allow you to build Web 2.0 applications using their architecture and the Adobe Flex framework.

I’ve not really played around with Adobe Flex but they have a demo video up on the site that has tweaked my interest so if I can carve a few hours out of a weekend I may see if I can get it up and running. I am dreading trying to figure out yet another object model though.

With all the majors clamoring for developers to join their cloud initiatives (Microsoft…where are you in this game?) I’m still struggling to figure out the real angle here. I get the advantage of someone else managing all your scaling but I’m still not sure that a major site who might really have the traffic to justify a large cloud is going to be willing to give up this much control over their infrastructure. That reduces these efforts to prototyping platforms which I don’t think is a scalable vision nor the intent of Google, Amazon and Intuit.

That being said, I am very interested to see if an ecosystem of applications starts to sprout on these platforms. Now that would be very cool - have a great idea that builds on the basic building blocks of Web 2.0 (social networking, messaging, etc.)? No problem, rent or borrow someone else’s functionality and integrate it with your own. Sort of the open source model but rather than downloading source to your own servers, plug into application services that sit on the cloud. Even better if you can plug into data services to get reference data that your application requires.

Now that would be interesting….

So the big announcement yesterday was that Google is entering the hosted application space with their Google Apps Engine. So far in this space, Amazon has been generating the most press with their web services offerings and specifically S3, their hosted database server (which incidentally went down again recently).

While the blogsphere has lit up on the topic of cloud computing based on Google’s announcement, as someone who has launched an Internet based startup and would in theory be a target for these services, I continue to struggle with them conceptually as I’ve written before.

In launching TravelGator, the things we struggled with were not the things that Google or Amazon is trying to solve. Granted, perhaps once our traffic is at a level that requires resources to handle millions of users we might need some of their infinite scalability, but I’d argue at that point a company has the resources to manage their own infrastructure and actually wouldn’t necessarily want to suffer through the problems that full outsourcing can cause.

That being said, the real problems that an emerging or mid-market online company struggles with have nothing to do with large scalability. Ours included:

1. Feeds integration - we get data from lots of partners in lots of formats (some of it really dirty). It was a HUGE task to write all the code to standardize it and even to date I’ve not found a good infrastructure to handle it. Even cool tools like SnapLogic don’t seem to be able to handle complex feed integrations where the data is excessively dirty.
2. Search - OK, here is a place that Google should be able to help. The issue we ran into though is that the tools that are available for searching structured data rather than unstructured data are very difficult to get working correctly. If you want to search your site text - no problem - but try figuring out a way to take a search for “Hilton” and return “Hilton Head”, every Hilton Hotel and anything else with the word “Hilton” in it all organized and filtered by the type of data being returned. It is really difficult (and no, straight SQL doesn’t work in this case).
3. SEO - again, the folks at Google should know something about this. The dos and don’t for search engine optimization are well known but very few application stacks automate it (Ruby on Rails being one exception but as a framework it limits your flexibility in numerous other ways).
4. Front end design - this is a real killer. The libraries and tools that exist for doing true, template based website design are stuck in in 1990s. It is about time someone created a front end, WYSIWYG editor that was a powerful as PhotoShop and Illustrator and creates clean, HTML code that can be easily integrated with an application server. This is a problem worth solving as much of the cost for developing a site goes into the creation of the user interface.
5. Ad serving - much to our surprise, third party ad serving engines aren’t very sophisticated. There is an opportunity here for a innovative company to abstract away all the ad networks and affiliate networks and allow sites to properly monetize their page views. A better solution should exist and should be part of the application framework.

I could go on but these are the biggies. Databases and Python engines are all very well and good, but to really get me to jump out of my seat I’d have to see all the other services come together into true building blocks. Then I’d get really excited.

I’ve been a bit delinquent in putting up new posts lately as I’ve been distracted with some other projects but more important I’ve just not found anything that I wanted to write about. There is certainly a lot going on out there as is clear from the latest TechCrunch postings from the Next Web Conference in Amsterdam but so far I’m not seeing much in the way of new trends.

That being said, there are a couple of things that I’ve written about previously that seem to be gaining steam.

Widgets are in general, picking up steam. It seems like daily (hourly?) you hear of yet another Facebook widget or iGoogle widget that is going to change your life. No some of these are pretty cool, don’t get me wrong, but I’m beginning to wonder whether we are seeing a fundamental break between a self contained, fully functional site like WebMD that solves a specific problem and a world in which everything is a collection of much more specialized widgets that you access from lots of different channels - and ideally follow you around from channel to channel.

This is worth more thought (and postings!) in the future. I’m also waitig for a widget search engine to launch - finding all these things is a major pain today.

The other rumor I thought was particularly interesting was the one about Google launching a cloud database solution that would compete with the one started awhile back by Amazon. As I’ve written before, the key with these services is to get true generic, scalability without sacrificing the real power of the database. As every developer knows, often you need to do some funky stuff to get a database to perform which is why I’m really curious as to whether the cloud computing approach will work for the majority of web sites out there - which are the obvious target for these sort of services.

The other really challenge here is not so much getting the database up and running (and scaling), but actually designing and loading it with data. That might be the value add that really gets these services off the ground. Imagine a situation in which you could just grab a set of database components from a central library (say a list of the cities of the world) for use on your web page and someone else would keep them up to date. Now there is a service that would be worth paying for. Oh yeah, and what about loading those feeds.

More to come on this trend I predict.

Amazon announced that they have released to limited Beta a new web service called Simple DB. In a nutshell, Amazon is offering to host databases for other companies. Rather than setting up a large, clustered database environment yourself, you can simply grab some space on Amazon’s servers and point your application at them - viola, instance high performance database.

While in a nutshell this is simply a managed database service (available from any number of managed hosting services companies), I do expect Amazon will add some additional tools and services around it to make it compelling. TechCrunch seems to think that this will a rather attractive service for Web startups. As someone who has recently launched a web startup (TraveGator), I’m not so sure.

Amazon is offering to manage all the normal database complexity that usually you need a database administrator to handle, including indexes, backups, etc. While having someone else handle clustering and backups would sure be a nice (trust me…we have sunk many hours into tuning our MySQL configuration), I’m not convinced it will actually work. Startups who have a lot of data, i.e. those who would be attracted to this sort of thing, are also the ones who will need to get down into the guts of the data to actually get their site to perform.

Call me a skeptic, but I just don’t think that Amazon can host a cluster where they know nothing about the nature of the data and make queries actually perform properly. They are also limiting queries to a rather simple language (what if you need a ten table outer join?) and to a maximum of 5 seconds - which will work fine for real time web queries but is going to be lousy when you need to bolt an indexer like Lucene on the data to do anything more complex.

I love the idea, but my concern is that in making it simple, Amazon has also made it pretty useless. I guess time will tell the right way for the database to move to the cloud and whether this is it. My bet is it isn’t.