March 26, 2010

Microsoft fell so far behind Google in the search engine market because it failed to retrieve relevant results for a long line of less popular queries, a senior Microsoft executive told the crowd at Search Engine Strategies show here March 25.

Such was the key reason Yusuf Mehdi, senior vice president of the Online Audience Group for Microsoft Bing, offered for why Google, is light years ahead of Microsoft in the search market. Google commands 65 percent of the U.S. share search market compared to 11.5 percent for Microsoft Bing.

Mehdi, responding to a keynote host's observation that Microsoft was late to the Internet and search, said "we missed the boat early on, that the focus was about the long tail. We actually focused a lot on the head of the queries.... It turned out the long tail was much more important."

For Microsoft, focusing on the head instead of the "long tail" meant that it returned popular queries and failed to satisfy less common queries. The long tail of queries ended up yielding more sizeable traffic and therefore more money for Google over the last 11-plus years. The result was a decade of search futility; just 10 short months ago, Microsoft's share of the search market had fallen to 8 percent. Bing has gained 3.5 percentage points of search since June 2009. Google has remained at its position, even growing a tad, while Yahoo has slide, with Bing gobbling the company's lost share.

Mehdi said that a year ago, four new URLs were created every second of every day. "Think about the explosion in the long tail. You have to crawl, index and make sense of that. On any given a month, one-third of queries that show up on Bing, it's the first time we've ever seen that query. A huge chunk of those, we'll never see again. They're like gone with the wind. The challenge of being able to be up to speed to understand that new flow of data and to be able to index the right thing so you can respond in sub-second time is a very, very hard problem." This scale challenge Mehdi described is a big reason why Google has extended and maintained its massive search share. That and the massive digital advertising market Google has fostered to make $23 billion a year.

"The problem gets harder when it comes to new forms of unstructured data," Mehdi continued. "How do I get to things that are closed off, such as Twitter feeds, or YouTube videos, or things that you can't crawl easily. What happens when you can't understand the URL?" Mehdi said the tide is turning back around to the head of popular queries. This is why Microsoft has partnered with Twitter, Wolfram Alpha and, as of today, popular location-sharing service Foursquare.

While search was originally meant for navigation, with Google's PageRank ranking algorithm fortify results with relevance, search is becoming an engine to fuel transactions and advanced research, including queries of multiple sentences instead of just keywords. If the old search query was about finding the right Web site, the new line of queries is more like "Where does President Obama stand on healthcare," and "What are the causes of a certain type of pancreatic cancer," Mehdi explained.

Current search, isn't designed to keep up with humans' thinking patterns on search and suffers from ambiguity. Bing, he said, is looking to solve the challenge of deciphering user intent. This means finding what users are searching for even though the consumers' query words don't match their idea of what they want to find.

"It's more of a dialogue with the consumer," Mehdi said. "We are about understanding user intent, and in mapping the intent into tasks and into actions." Microsoft executives are certainly on the same page in this regard. Mehdi's comments echoed those made by Bing Director Stefan Weitz to eWEEK recently in stories here and here.

Source:-http://www.eweek.com/