Second Workshop on Empirical Evaluation of Adaptive Systems

in conjunction with UM2003

Del Missier, F. and Ricci, F. (2003). Understanding Recommender Systems: Experimental Evaluation Challenges. In: Weibelzahl, S. and Paramythis, A. (eds.). Proceedings of the Second Workshop on Empirical Evaluation of Adaptive Systems, held at the 9th International Conference on User Modeling UM2003, Pittsburgh.

The paper discusses some significant issues in the empirical evaluation of interactive recommender: the role of experiments, the contingent and constructive nature of users’ interaction strategies, and the generalizability of the results. We propose to adopt as the main evaluation goal the construction of a situation-specific account of the user-system behavior, and we suggest applying the context matching approach to cope with the contingencies of the user behavior. To make clear the limitations of high-level single step experimental evaluations, we present a critical analysis of a case study, a pilot evaluation of a travel recommender system (ITR). The examination of this study shows the danger of overlooking the detailed aspects of user-system interaction, and underlines the need to iteratively refine the evaluation hypotheses and design when no detailed model of the user-system behavior is initially available.