Utvärdering av ett IR-system i en informationssökningsprocess ett holistiskt perspektiv
The research objective of this thesis was to conduct a holistic evaluation of an operational information system. By holistic evaluation we mean that an equal focus was placed on both the system system perspective and its users user perspective, in the actual environment where the system and its users function contextuality. In addition, the methodological objective of the study was to test a new research approach in a real life setting. The participants of the study were twenty newspaper journalists employed at Göteborgs-Posten GP. The information system NewsLink is a manually indexed full-text database containing all articles published in GP since 1994. It employs Boolean search logic and offers a choice between showing the retrieved result ranked either by date or order of relevance. Our evaluation methodology consists of triangulation pre-search questionnaires; search log; post-interviewing and novel interactive performance measures the Ranked Half-Life measure, as well as Satisfaction and Novelty perception by users, supplementing Precision. It involves traditional algorithmic performance measures, usefulness situational relevance, and non-binary relevance judgements in an operational ranked IR system seen in the context of 1 work task fulfilment, and 2 other information resources. Our findings show that the system worked well for its users, but would gain considerably from the improvement of its interface. In sum, we found the method employed to be well suited for evaluating information systems in a real life setting as it ensures both realism and control.