F.A.Q. on Evaluation

The term "adaptive system" is used in various contexts. However, this site is dedicated to all interactive systems that adapt their behaviour to each individual user, on the basis of nontrivial inferences from information about that user (Jameson, 2001).

The term empirical evaluation is seen in a very broad sense in this context. It includes all kind of empirical studies that assess the quality of the adaptive systems, be it in a classical controlled experimental setting, an early exploratory study, or real world observations of running systems. Even studies with simulated (hypothetical) users are considered if they focus on the user model.

Well, empirical valuations have several advantages and provide interesting opportunities, but certainly there are also limitations (Weibelzahl, Lippitsch & Weber, 2002). By evaluating your adaptive system you may be able to demonstrate that it is actually working in the expected way and that it is thus useful. Speaking from a philosophy of science perspective, an evaluation validates your theoretical assumptions about how to infer certain user characteristics and how to use them to improve the interaction.

You might want to read a book on experimental design and statistical analysis first. The tutorial by David Chin also provides a good overview. Next you might want to get to know the problems and pitfalls that arise in the evaluation of adaptive systems. Several frameworks addressed (at least some of) these problems and serve as good guideline for empirical studies.

Several workshops on empirical evaluation and countless discussions resulted in one important claim: There is a lack of empirical evaluations in this scientific community (as it is the fact in many other communities), and we need to take actions to encourage more high quality studies. You are just looking at one of these actions. We strongly believe that this site could encourage researchers to conduct an empirical study by providing easy access to material that enables them to plan, design, conduct, analyse and report an empirical study. Moreover, the evaluation of user modeling systems includes several tricky theoretical questions which are still unsolved. This site is supposed to be an starting point for discussions and theoretical considerations about empirical research.

Generating FAQs is a lot of work. Thus, we hesitated to produce very many items here without knowing much about the actual information need. However, we are happy to get input from others. Feel free to submit as many questions and answers as you want to by email and we will make them available on this page.