My Own Pirate Radio

Businesses that offer content online have a real challenge on their hands with search engines.

On the one hand, content providers need search engines in order to be discovered. (Yes, people also discover sites through word of mouth and advertising, but search engines are undoubtedly the primary discovery mechanism for web sites. Heck, I even navigate using the search box… it’s often faster than remembering where I stashed a bookmark.)

On the other hand, search engines continually raise the bar in terms of the quality and quantity of information displayed on search results pages. And to a content provider, that can be dangerous.

Consider this Google Maps search, for example: “Terroni restaurant Toronto”. A few years ago the search results page just contained a list of possible matches plotted onto a map. Now that same results page shows a map, yellow-pages type information (business hours, type of food served, payment types accepted), links to relevant web pages, and links to relevant reviews on review web sites. Furthermore the reviews themselves are summarized in a five-star rating graphic.

Microsoft’s Live Search is a little clunkier in this scenario, but offers pretty much the same content.

It’s useful, right? Yes indeed… very convenient for customers, and great for businesses like Terroni’s. But if you’re running one of the sites that provided the reviews summarized by Google — Zagats, Where.ca, and NowToronto.com, in this case — how should you feel about this?

The answer depends on your business model.

If your business relies on web traffic to your web site, and your content can be comprehensively summarized within search engine results, you’re in trouble. For instance, if I skim the review snippets, and I like the aggregate rating (5 stars in this case… surprise, surprise), I may just write down the street address and dispense with visiting the review sites. From Terroni’s point of view that’s just fine… they get my business when I walk through the front door. But from the review site point of view that is a business opportunity lost, because I didn’t click through.

Conversely if your business model relies on something else for revenue — customer subscriptions, let’s say — then this sort of disintermediation is not such a big deal.

This is the latest incarnation of the old OS/Applications cat-and-mouse game: Apple and Microsoft enhance every new release of their operating systems with a bunch of new features, thus commoditizing some of the functionality within the apps that run atop the OS. The app developers therefore have to “move up” and differentiate their products by adding new features that aren’t yet offered by the operating system. The same thing will play out in the content provider / search engine space.

Lessons for content providers:

1. Make your content indexable by search engines so that your site can be discovered easily.
2. Make sure your content cannot be easily and completely summarized in a search result.
3. Differentiate your content, and make sure people know you have something different to offer.
4. If possible, select a business model that does not rely entirely on search engine traffic.
5. Never stop improving, because you know the search engine companies won’t stop chasin’ ya

Researcher Elizabeth Van Couvering recently published research results on how search engine employees think and talk about their work.

In the face of rising controversy about search engine results—that they are too restrictive, too comprehensive, lacking in certain areas, over-represented in others—this article presents the results of in-depth interviews with search engine producers, examining their conceptions of search engine quality and the implications of those conceptions.

She interviewed employees of Google, Yahoo!, MSN, Ask Jeeves, AOL, Excite, Lycos, Infoseek, and WebCrawler, and her report is remarkable in terms of the insider verbatim quotes. (The search industry is notoriously secretive, so getting people to reveal their inner thoughts about it is challenging.) Her analysis reveals that search industry insiders consistently portray themselves as fighting a war, that they justify much of their work in scientific terms — even when trying to deal with unquantifiable issues like public good — and that they understand in a deep way how their business is driven by profit motives.

Her work is on target. And not just for search; her observations apply to every technical group I’ve been part of or collaborated with, both within and outside of Microsoft. We all used similar language “schemas”. We talked about marketplace war, about science for science’s sake. We often shrouded ideology — “censorship is inherently bad” — in technical terminology — “I suppose the algorithm decided to demote that result because its relevance dropped during our last crawl. We don’t decide; the algorithm decides.”. Or, “Sorry, we can’t implement that feature, the current architecture doesn’t allow it and it would take way too long.” Frankly, it was often easier to fight such technical arguments than to fight the underlying ideological battles, because the latter approach generally ended in stalemate. So technical language became a tool for keeping non-techies from influencing product design, and it shielded teams from struggling with uncomfortable thoughts like, “Does any of my work have a negative impact on society?”. I believe this shielding pattern reoccurs broadly in many science and technology fields.

Elizabeth’s methodology — analyzing the actual words people speak — reminds me of ‘The Four Horsemen’: Why Marriages Fail, a radio story that aired on NPR in 2005. It’s an interview with a researcher who talks to couples about their relationships, then analyzes their word choice and emotional tone to predict whether they will eventually divorce or not. Apparently there are four strong predictors of divorce: criticism, defensiveness, contempt and stonewalling. And words that convey contempt for a partner are the strongest predictors of all.

The NPR piece is designed for a mass audience. Elizabeth’s piece is written for an academic audience. I recommend both.

Dane Glasgow is moving on.

See also my previous post on this topic.

Lots happening in search recently, especially in the local search segment.

• On Friday Google announced it acquired DoubleClick for $3.1B, beating out Yahoo!, Time Warner (AOL’s owner), and Microsoft in the bidding. This gives Google ability to place display ads (graphical ads) on sites, in addition to their existing text ads. In an ironic twist, both Microsoft Corp. and AT&T Inc. called for antitrust regulators to investigate the deal.
• Two days ago Google struck a deal with Clear Channel Communications to sell airtime on their 675 radio stations to Google’s advertisers. Clear Channel is America’s largest radio broadcaster.
• Also this week, Yahoo! expanded its agreement with a consortium of newspaper publishers to run their content on Yahoo!’s websites, and to place advertisements on the sites of the newspapers.
• In late March the News Corporation and NBC Universal struck a video content distribution deal with AOL, Yaho, MSN and MySpace. Some are casting this as an ill-fated effort to compete with YouTube (I agree), while the dealmakers claim it is more about creating a shared network of content and ads.

It’s neat seeing all the old media companies come around, albeit belatedly, to doing deals with the new media companies. It’s nice for the old media companies; they get a handy cash infusion, some positive PR, and new ad inventory to sell. But this is just a short-term fix. The new media companies get the better part of this deal, because:

1. They get to tap into the local sales force of the old media companies. This reduces a huge barrier to success in local search: the cost of creating and running a local advertising sales force, a.k.a. “feet on the street”.
2. They get to demonstrate their reach, innovation, and cost efficiencies to a new advertising base. (Google and Yahoo! are surely saying nice things about not trying to steal away advertisers from their new radio, television and newspaper publishing partners, but you’ve got to be skeptical about that in the medium to long term. Ad dollars are shifting online, fast. These deals only grease the skids.)

For an overview of US local search market trends, see LennAnn Prescott’s Hitwise blog post.

BusinessWeek also has more commentary on Google’s recent moves in the ad business.

A year and a half after launching their new search engine, Microsoft’s US market share in web search has declined from 13% to 9%, while Google and Yahoo! both gained. Microsoft is treading water while the market overall has grown. Here are the share numbers.

Yesterday brought more bad news for the Windows Live Search team: VP Christopher Payne is leaving Microsoft to start his own company.

It’s hard to imagine how to spin this one positively. Christopher was one of the first to pitch Bill and Steve on building a new search service in-house rather than continuing to outsource to Yahoo!. He played a huge role in recruiting key people to the team, creating a vision, and integrating marketing, business and customer focus into what was undoubtedly a pure engineering team. And as you can imagine, his business performance goals were tied in no small part to gains in market share and profitability. So this is not just the departure of the chief cheerleader and visionary; it’s a sign that success in search is not coming nearly as fast as Microsoft had hoped.

Having been on the team myself I have mixed feelings about this changing of the guard, and I imagine many of the people I used to work with are feeling the same way. When a key leader heads for the door you stop and question what keeps you there. Is it your passion for the technology and business challenges? Your allegiance to the company, the leaders, the team you helped build? The paycheck? How does that stack up against the negatives of the job? (Unless things have changed radically, the main down sides were: unattainable performance expectations pushed down from above, grueling hours, and a maddening culture of prima donna behavior that was tolerated and even rewarded.) Most importantly, how does all that measure up against time you could be spending doing something else? It’s worth doing that math once in a while.

Of course, the search battle is not over. In fact, it’s barely begun. But it will continue to be uphill for Microsoft. From the Seattle PI, March 5: “You have to think that, with the heavy lifting behind it, Microsoft has got to be in a good position to execute, and execute well,” Nielsen NetRatings’ Cassar said. “But they’re competing against two of the strongest companies, between Yahoo and Google, and so Microsoft can’t just execute well. Microsoft needs to execute in an extraordinary fashion in order to just keep pace.”

Good luck to all the warriors.

update March 8: Danny Sullivan gives some history on search at Microsoft.

Blogging for me is a hobby. I’ve only been doing it a short while, and it’s not my day job or something I try to promote. I’m pretty sure most of my readers are friends and family who humor me by visiting on occasion. So this morning, on seeing my blog traffic running at 10x its normal level, my initial reaction was fear, uncertainty, and doubt. Read the rest of this entry »

Announced today on the del.icio.us blog. In hindsight it was only a matter of time before one of the big three bought them.

Looksmart owns Furl. Who’s next, I wonder?