Evaluation of Adaptive Systems
Fundamental Empirical Techniques and Caveats
David Chin (2007). Evaluation 1: Fundamental Empirical Techniques and Caveats. Tutorial presented at the 11th International Conference on User Modeling (UM2007), Corfu, Greece, 2007.
Comment
(From the tutorial's description:) This tutorial will introduce UM researchers to the fundamental techniques of empirical evaluation of user modeling systems. The target audience of this tutorial are new UM researchers who want to learn how to evaluate their UM system. No background in statistics is assumed. Topics will include experiment design, running experiments, and experiment analysis, especially statistical techniques for analysis.
Related links
- Tutorial page on the UM2007 site
- Tutorial slides in PDF format (from the above page)
- Homepage of David Chin
Formative Evaluation Methods for Adaptive Systems
Weibelzahl, S., Paramythis, A., & Masthoff, J (2007). Evaluation 2: Formative Evaluation Methods for Adaptive Systems. Tutorial presented at the 11th International Conference on User Modeling (UM2007), Corfu, Greece, 2007.
Comment
An introduction to Layered Evaluation, and to evaluation methods suited for the formative evaluation of Adaptive Systems.
Related links
- Tutorial page on the UM2007 site
- More detailed page that includes reading material
- Tutorial slides in PDF format (from the conference site)
Problems and Pitfalls in the Evaluation of Adaptive Systems
Weibelzahl, S. (2005). Problems and pitfalls in the evaluation of adaptive systems. In S. Y. Chen & G. D. Magoulas (Eds.), Adaptable and Adaptive Hypermedia Systems (pp. 285-299). Hershey, PA: IRM Press
Comment
This chapter lists several problems and pitfalls that arise when evaluating an adaptive system and provides guidelines and recommendations for workarounds or even avoidance of these problems. Among other things the following issues are covered: relating evaluation studies to the development cycle; saving resources; specifying control conditions, sample and criteria; asking users for adaptivity effects; reporting results. An overview of existing evaluation frameworks shows which of these problems have been addressed in which way.
Evaluating the Effectiveness of User Models by Experiments
David Chin (2003). Evaluating the Effectiveness of User Models by Experiments. Tutorial presented at the Ninth International Conference on User Modeling (UM2003), Johnstown, PA, 2003.
Comment
This tutorial provides a nice overview of setting up and running experiments as well as analysing data statistically Thus, if you want to evaluate your user modeling system, but you you have no experience with empirical studies, this is a good place to start.
Related links
Questionnaires in Usability Engineering
Jurek Kirakowski (without year). Questionnaires in Usability Engineering. A List of Frequently Asked Questions (3rd Ed.). Online: http://www.ucc.ie/hfrg/resources/qfaq1.html. Last accessed December 30, 2003.
Comment
A nice list of Frequently Asked Questions on usability questionnaires that explains the basic concepts (e.g., reliability and validity).
Related links