Wikia acquires Grub distributed search indexing system

Wikia acquires Grub distributed search indexing system

By Ryan Paul | Published: July 30, 2007 – 08:17AM CT

Wikia, the company created by Wikipedia founder Jimmy Wales, has acquired the Grub distributed indexing system from LookSmart and is preparing to distribute Grub’s code under an open-source license. Wikia plans to use Grub for its user-driven search engine, which is still under development.

Originally created in 2000, Grub leverages the distributed computing model to crawl the web and index pages. Users install a specialized client application on their computer, which then automatically performs indexing while idle and transmits page data back to a centralized repository. In this manner, volunteers will contribute the raw computing power that performs the indexing.

Wikia is resurrecting Grub as an open source project and hopes to work with the open source software community to create ports of the Grub client—which currently only runs on Windows—to other operating systems. Wikia hopes that the modular nature of Grub and the availability of source code will make it possible for users to add features and help improve the system’s performance.

In addition to leveraging volunteer computing power for automated indexing, Wikia’s search engine will also attempt to take advantage of human power for index editing and refinement. According to Wales, users of the Wikia search engine will be involved in adding and removing links, removing spam, and policing other users much like the participatory model used by Wikipedia today.

“The desire to collaborate and support a transparent and open platform for search is clearly deeply exciting to both open source and businesses,” said Wales in a statement. “Look for other exciting announcements in the coming months as we collectively work to free the judgment of information from invisible rules inside an algorithmic black box.”

Wikia’s search engine isn’t yet available for use, but the project’s mission is articulated on the Wikia search page. With goals like transparency, community, quality, privacy, and interoperability, Wikia’s search service seems promising at first glance, but despite the potential value, there are many problems that the company will face when the search engine launches.

Search engine ranking has significant financial implications for many companies, so it’s likely that Wikia’s user-driven search engine will face constant attempts at manipulation. Keeping the spammers and search engine optimization hackers at bay is sure to be a taxing endeavor. Considering the vehemence with which Wikipedia users have traditionally opposed using ads rather than donations to fund Wikipedia, it’s not entirely clear that an ad-based commercial project like Wikia’s search engine will attract the same degree of user involvement.

Distributed computing is a highly unusual approach to indexing, but it’s also consistent with Wikia’s participatory model. Regardless of whether or not Wikia’s search engine succeeds, the company’s willingness to experiment with unconventional approaches could spur innovation and change the landscape of the search engine market.

Discuss Print

Comments RSS

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: