Literature : Theory

Here you can find a list of papers that are concerned with theoretical issues of the evaluation of adaptive systems.

Quick index:

Brusilovsky, P., Karagiannidis, C., and Sampson, D. (2001). Benefits of Layered Evaluation of Adaptive Applications and Services. In S. Weibelzahl, D. Chin and G. Weber (Eds.) Empirical Evaluation of Adaptive Systems. Proceedings of workshop held at the Eighth International Conference on User Modeling in Sonthofen, Germany, July 13th, 2001, pp. 1-8, Freiburg: Pedagogical University of Freiburg

Comment

Revisited version of a prior study which uses the layered evaluation approach to explain empirical results.

Herder, E. (2003). Utility-Based Evaluation of Adaptive Systems. In: Weibelzahl, S. and Paramythis, A. (eds.). Proceedings of the Second Workshop on Empirical Evaluation of Adaptive Systems, held at the 9th International Conference on User Modeling UM2003, Pittsburgh, pp. 25-30.

Comment

Utility-based evaluation is introduced as a technique to compare different systems or modeling approaches.

Höök K. (2000). Steps to take before intelligent user interfaces become real. Interacting with computers, 12(4), 409-426

Comment

Some ideas about problems with the evaluation of user modeling systems.

Magoulas, G. D., Chen, S. Y. and Papanikolaou, K. A. (2003). Integrating Layered and Heuristic Evaluation for Adaptive Learning Environments. In: Weibelzahl, S. and Paramythis, A. (eds.). Proceedings of the Second Workshop on Empirical Evaluation of Adaptive Systems, held at the 9th International Conference on User Modeling UM2003, Pittsburgh, pp. 5-14.

Comment

Proposal of how to integrate heuristic evaluation into the layered evaluation approach in order to facilitate the diagnosis of design problems in early stages.

Masthoff, J. (2002). The evaluation of adaptive systems. In Patel, N. V. (Ed.), Adaptive evolutionary information systems. Hershey, PA: Idea Group Publishing.

Comment

Overview of issues that arise when evaluating an adaptive system. Cognitive modeling is introduced as a way to save resources. Sorry, there is no online version of this book chapter, but you may want to ask the author to send you a copy.

Ortigosa, A., and Carro, R.M. (2003). The Continuous Empirical Evaluation Approach: Evaluating Adaptive Web-Based Courses. In Brusilovsky, P., Corbett, A.,de Rosis, F. (Eds.) User Modeling 2003. Lecture Notes in Computer Science 2702, Springer-Verlag, pp. 163-167

Comment

The idea put forward in this paper is to identify possible fails of web-based adaptive courses by analyzing the behavior of users and feeding this back to the course designers.

Paramythis, A., Totter, A. & Stephanidis, C. (2001). A Modular Approach to the Evaluation of Adaptive User Interfaces. In Stephan Weibelzahl, David Chin, and Gehard Weber (Eds.) Empirical Evaluation of Adaptive Systems. Proceedings of workshop held at the Eighth International Conference on User Modeling in Sonthofen, Germany, July 13th, 2001, 9-24, Freiburg: Pedagogical University of Freiburg

Comment

A modular framework for the evaluation of adaptive system from a software engineering perspective.

Tobar, C. M. (2003). Yet Another Evaluation Framework. In: Weibelzahl, S. and Paramythis, A. (eds.). Proceedings of the Second Workshop on Empirical Evaluation of Adaptive Systems, held at the 9th International Conference on User Modeling UM2003, Pittsburgh, pp. 15-24.

Comment

Separating design perspectives in a so-called "Extended Abstract Categorization Map" is proposed as a new framework for AH systems to facilitate evaluation processes.

Totterdell, P., and Boyle, E. (1990). The Evaluation of Adaptive Systems. In Browne, D., Totterdell, P., Norman, M. (Eds.), Adaptive User Interfaces (pp. 161-194). London: Academic Press.

Comment

The first "seeds" of the idea of breaking down adaptation for evaluation purposes can be traced back to this paper. Totterdell and Boyle propose that a number of adaptation metrics be related to different components of a logical model of adaptive user interfaces, to provide what amounts to adaptation- oriented design feedback. Furthermore, they present two types of assessment performed to validate what is termed "success of the user model" (note that, in their case, the "user model" is also responsible for adaptation decision making): "... an assessment of the accuracy of the model's inferences about user difficulties; and an assessment of the effectiveness of the changes made at the interface."

Weibelzahl, S. (2001). Evaluation of adaptive systems. In M. Bauer, P. Gmytrasiewicz, and J. Vassileva (Eds.), User Modeling 2001: Proceedings of the Eighth International Conference, UM2001. (pp. 292-294) (Lecture Notes in Computer Science). Berlin: Springer.

Comment

Very short description of a layered framework for evaluation of adaptive systems.

Weibelzahl, S., Lippitsch, S. & Weber, G. (2002). Advantages, opportunities, and limits of empirical evaluations: Evaluating adaptive systems. Künstliche Intelligenz, 3/02, 17-20.

Comment

A rough sketch of the advantages and disadvantages of an empirical approach. What kind of failures may be uncovered, which cannot be detected? When should I perform an empirical study with my system?