One Kiwi's quest for a better search algorithm

One Kiwi's quest for a better search algorithm

Sick of Googling 'apple' only to be shown Macs or pictures of Gwyneth Paltrow’s baby? One Kiwi researcher is working on technology to help fine-tune search engines so they understand what users are actually searching for, and getting attention from Microsoft and Google.

Daniel CrabtreeDr Daniel Crabtree, 27, has created an algorithm that would improve the likes of Google and Yahoo's understanding of a desired search term.

“Search engines still throw up mixed results with ambiguous search terms. If you searched for ‘jaguar’ for instance, it could refer to the animal, the car, even an Apple operating system or ‘80s video game console.

“Search engines currently don’t deal with that ambiguity because they simply search for web pages that contain the words you’ve entered.”

Crabtree says the algorithms he developed group pages together to separate different interpretations, and use statistic language models to capture the intended meaning of the query.

“One aspect of the model for instance is that it recognises word order. If you typed in ‘New Zealand air’, for example, it would cluster results around air quality rather than Air New Zealand which is the search result you get from Google.”

Crabtree says this has been a lengthy and difficult process. He started researching this in 2005 and published papers on the topic between 2005 and 2008.

“When you sit down and try to solve a difficult problem head-on you often hit road-blocks and dead-ends, but eventually if you try hard enough you come up with something really complicated that sort of works.”

He has had informal meetings with Google, Microsoft and Yahoo, who were interested in his work and hiring him. However, Crabtree says he had no interest in being an employee so no formal job offers were ever made.

Crabtree is also working on a new web concept, codenamed Project Mandelbrot, which will improve how people find and share information online.

Idealog has been covering the most interesting people, businesses and issues from the fields of innovation, design, technology and urban development for over 12 years. And we're asking for your support so we can keep telling those stories, inspire more entrepreneurs to start their own businesses and keep pushing New Zealand forward. Give over $5 a month and you will not only be supporting New Zealand innovation, but you’ll also receive a print subscription and a copy of the new book by David Downs and Dr. Michelle Dickinson, No. 8 Recharged (while stocks last).