Thesauri for Information Retrieval Part. 1

Thesauri have been widely used in information retrieval in recent years. They are built in software to facilitate users to retrieve information on websites or content management systems. Currently, law firms and consultant companies have integrated thesauri into their websites. Thesauri could be also used to automatically index contents in databases. Most commercial databases all have thesauri to help users increase search effectiveness, such as STN, WilsonWeb. However, there are no standards for current thesauri developers to adopt or compose concepts consistently. The same concept could be displayed differently in different thesauri for different purposes. For example, "knowledge management software" could be splitted into "knowledge", "management" and "software"; it could be also broken down to "Knowledge management" and "software". The first case could happen in a general thesaurus, second one might possibly happen in a thesaurus of Information Science. People who want to integrate the above thesauri into their software will ask a question, which one is more appropriate for my system?

Except for specific purposes, most thesauri should be interoperable with software to maximize the benefit of thesauri to certain extent regardless of different domains. Sometimes, it even takes longer to customize a thesaurus for local systems than develop a thesaurus from scratch. In another word, how we could let current content management systems easily adopt available thesauri? We need a standard to standardize the way we create concepts in thesauri.

Ranking Terms in a Thesaurus Database

As I have discussed in the previous posting, a thesaurus is very useful tool for users to efficiently search information in a database. The purpose that users look up the thesaurus is to find the right concept for their search. Do users really need a ranking scale to indicate the relevancy of the term they are looking for?

If the ranking makes sense to the users, it would be worthy doing so. For instance, when users search chemicals in databases, such as STN and Dialog, they would prefer to look up the term in the thesaurus first. By looking at the term indexed by the database producer, users would know how to create their search strategy. The ranking scale used by these databases is the number of records linked to the term. A term in a database could be ranked quite differently from it in another database.

If the ranking is based on a word partially matching with the indexing term, it would confuse users. For example, if users get the same ranking scale of different indexing terms because of partial word matching, users would conclude that these indexing terms are all equally relevant. That's not true. Some indexing terms are linked with more relevant records than others, how the thesaurus would help users to decide which term should be used to perform the search?

I would love to see a new systematical ranking system will be adopted in a thesaurus database.

WilsonWeb Thesaurus Database

WilsonWeb is a hybrid database with subjects of humanities, social science, education, business, applied science and technology. The thesaurus database is a very useful tool for searchers. For instance, if you are not sure the term indexed in the database, you can search thesaurus database, then you will get the list for all related subjects and related terms with the number of linked records. This is a preliminary search, it will give you some hints to search WilsonWeb with indexing terms.

However, WilsonWeb covers subjects in different disciplines, the same concept could have different meanings in different domains. It would be a huge amount of works to create a hierachic structure in its thesaurus database, or people should call it taxonomy, which will really narrow down the search terms. It's always a challenge for database producers that what kind of thesaurus should be offered to users. Database producers would think cost and effectiveness are the key to solve this problem.

Today, taxonomy has been a main information technology to make search engine more intelligent. Law firms, R & D, and consulting companies have begun building their own taxonomy to enhance the searchability of web search engines, which greatly saves searchers' time with more relevant search results. The problem most people have today is too much information exists, how are they able to find the needed information? Taxonomy could assist companies to organize information and make information easily searchable for users. SLA website and Askus.com are good examples of web content with the benefit of taxonomy.