Skip navigation

Which vertical search engines are relevant? Understanding vertical relevance assessments for web queries

Which vertical search engines are relevant? Understanding vertical relevance assessments for web queries

Zhou, Ke, Cummins, Ronan, Lalmas, Mounia and Jose, Joemon M. (2013) Which vertical search engines are relevant? Understanding vertical relevance assessments for web queries. In: Proceedings of WWW 2013, May 13-17, Rio de Janeiro, Brazil. ACM, New York, NY, USA, pp. 1557-1567. ISBN 9781450320351

Full text not available from this repository.

Abstract

Aggregating search results from a variety of heterogeneous sources, so-called verticals, such as news, image and video, into a single interface is a popular paradigm in web search. Current approaches that evaluate the effectiveness of aggregated search systems are based on rewarding systems that return highly relevant verticals for a given query, where this relevance is assessed under different assumptions. It is difficult to evaluate or compare those systems without fully understanding the relationship between those underlying assumptions. To address this, we present a formal analysis and a set of extensive user studies to investigate the effects of various assumptions made for assessing query vertical relevance. A total of more than 20,000 assessments on 44 search tasks across 11 verticals are collected through Amazon Mechanical Turk and subsequently analysed. Our results provide insights into various aspects of query vertical relevance and allow us to explain in more depth as well as questioning the evaluation results published in the literature.

Item Type: Conference Proceedings
Title of Proceedings: Proceedings of WWW 2013, May 13-17, Rio de Janeiro, Brazil.
Additional Information: [1] First published: May 2013. [2] Published as: Zhou, Ke, Cummins, Ronan, Lalmas, Mounia and Jose, Joemon M. (2013) Which vertical search engines are relevant? Understanding vertical relevance assessments for web queries. In: Proceedings of WWW 2013, May 13-17, Rio de Janeiro, Brazil. International World Wide Web Conferences Steering Committee, pp. 1557-1567. [3] ISBN: 978-1-4503-2035-1 (Proceedings); 978-1-4503-2038-2 (Companion). [4] This paper was first presented at the 22nd International World Wide Web Conference, (WWW'13), held from 13-17 May 2013 in Rio de Janeiro, Brazil.
Uncontrolled Keywords: aggregated search, vertical selection, evaluation, user study, relevance assessment
Subjects: Q Science > QA Mathematics > QA76 Computer software
Pre-2014 Departments: School of Computing & Mathematical Sciences
Related URLs:
Last Modified: 14 Oct 2016 09:24
Selected for GREAT 2016: None
Selected for GREAT 2017: None
Selected for GREAT 2018: None
Selected for GREAT 2019: None
URI: http://gala.gre.ac.uk/id/eprint/10093

Actions (login required)

View Item View Item