En studie av evalueringar av webbaserade söktjänsters återvinningseffektivitet
The aim of this thesis is to describe and critically investigate eight different evaluations of the retrieval effectiveness of webbased search engines. The questions to be answered in this investigation are: - What kind of relevance judgements have been used? - Which criteria have been used when judging the relevance of a document? - Which measures have been used? - How many queries have been used? - How were the queries constructed? - What document cut-off value has been used? - Has hypothes testing been applied? - What kind of webbased search engines have been included in the evaluations? The study showed that although the evaluations investigate the same phenomena, they are very different from each other in certain aspects. Generally the study showed that precision is the preferred measure in comparison to recall in the chosen evaluation even though all the included evaluations have constructed unique formulas for calculating precision. Some attempts to measure relative recall have been performed but they all suffer from different defects. The study also showed that the only reliable way to judge the relevance of a document is to judge the content of the document thanks to the unpredictable and dynamic nature of the web. Also the use of more complicated queries seem to influence the results of the evaluations in a way that reward certain search engines and punish others. The methods chosen also seem to be modelled for the robot-driven search engines as the subject directories investigated always scored remarkably low precision values in comparison.