By Nate Anderson | Published: January 07, 2008 – 02:01AM CT
Wikia Search was announced at the end of 2006, but it wasn’t clear at the time how the system would work (the for-profit Wikia is not affiliated with Wikipedia). There was much talk about “openness” and “user control” and “transparency,” but the details were elusive. In July 2007, the project came into some focus when Wikia acquired the Grub indexing system and released it under an open-source license. But only in the last few weeks has Wikia Search entered even a “pre-alpha” state suitable for limited user testing.
Wales says that Wikia Search will be open to users in two ways. First, anyone can rank search results by using a simple five-star system. Second, the backend systems that power Wikia Search will all be open-source projects like Grub, and anyone with a bent for coding can get their hands dirty with the actual search algorithms that operate in the background.
To Wales, this algorithmic transparency is key to what Wikia Search wants to do. Other search engines keep their algorithms hidden in an attempt to keep site operators from gaming the system and competitors from seeing how it works. But Wales believes that it’s important to know how the algorithm works, since each algorithm makes an editorial statement.
This isn’t the way that most people think about algorithms, which seem like the ultimate expressions of unbiased machine processing. Wales turns to an example to make his point: searching for “Thomas Jefferson” might return a page of ten links. Those links ostensibly represent the most important information about him, but are they? Who says so? The way the algorithm was tuned and constructed means that even these results are “an editorial statement” about the sorts of pages that are important. Algorithms of this kind aren’t neutral any more than the people who create them are.
While it’s fine to run a search engine this way, Wales thinks that users want more; they want real transparency about the way that the engine produces results, and they want a hand in tuning these results.
With search engines attaining such importance as gateways to information of all kinds, Wales could well be on to something, but he knows that Wikia Search won’t be ready for regular use for a couple of years. The goal isn’t necessarily to supplant established players like Yahoo and Google, but that doesn’t mean Wales is aiming low. He sees “some hope that we can generate better quality search results” in time. Until then, Wikia Search’s progress will be open for all to see.