UM 2007 Tutorial
"Evaluation 2: Formative Evaluation Methods for Adaptive Systems"

Reading material

This page provides pointers to material that is directly related to the tutorial and you may find useful in deisgning your studies. There are two sections, one on traditional methods for evaluating usability, and one on methods specific to adaptive systems.

You may also want to check out the workshops and the tutorials & guidelines pages on EASy-Hub.

Traditional methods for evaluating usability

Overviews of usability methods

• Usability and User experience: a website with many links on the traditional methods covered in this tutorial. Strongly recommended.
• Usability.Net: short explanations of many of the tradional methods covered in this tutorial.
• Task-centered user-interface design, Lewis and Rieman, On-line book. This is a shareware book, that covers some of the traditional methods in this tutorial.

Questionnaires

• Gary Perlman's Web-based user interface evaluation with questionnaires. Provides links to questionnaires that can be used assess a user's view of the usability of a product. Includes a cgi script for running such questionnaires on a website. http://www.acm.org/%7Eperlman/question.html

Focus Groups

• Basics of conducting focus groups by McNamara
• Paper by Nielsen on Focus groups

Cognitive Walkthrough

• Gregory Abowd's summary of Cognitive Walkthrough.
• Darryn Lavery and Gilbert Cockton's Cognitive Walkthrough Usability Evaluation Materials (PDF file).
  Contains: A description of Cognitive Walkthrough; A test to assess your understanding of it; A checklist; Sheets to record usability problems.
• Rowley, D., & Rhoades, D.G. (1992). The cognitive jogthrough: A fast paced user interface evaluation procedure. CHI '92 Proceedings, page 389-395.
• Wharton, C. et al. (1994). The cognitive walkthrough method: A practicioner's guide. In J.Nielsen.and R.L.Mack (Eds.) Usability Inspection Methods. John Wiley & Sons, New York.

Heuristic Evaluations

• Nielsen. J., & Mack, R.L. (Eds.) (1994). Usability Inspection Methods. John Wiley & Sons, New York.
• Nielsen's How to conduct a heuristic evaluation
• Nielsen's ten usability heuristics
• Deniese Pierotti's heuristics from Xerox Corporation.
• Bruce Tognazzini's list of basic principles for interface design

Task-based experiments

• Gary J. Dickelman's Performance support for conducting usability evaluations . A very good set of guidelines on how to perform a user test.
• Chapter 5 of Clayton Lewis' and John Rieman's Task-centered user-interface design. provides a good overview of user testing.
• Statistics glossary

Wizard of Oz

• Wizard of Oz on Wikipedia
• Kelley, J.F., "An iterative design methodology for user-friendly natural language office information applications". ACM Transactions on Office Information Systems, March 1984, 2:1, pp. 26-41.

Simulated Users

• Summary of the GOMS method based on David Kieras' A Guide to GOMS Task Analysis.
• Pages on GOMS at Virginia Tech
• Pages on GOMS at Usability First
• Clayton Lewis and John Rieman's section on Action Analysis (see Sec 4.2)
• Card, S. K., T. P. Moran, A. Newell. (1983). The psychology of human-computer interaction. Hillsdale, NJ, Lawrence Erlbaum Associates.

Methods specific to adaptive systems

Layered Evaluation

• Paramythis, A. & Weibelzahl, S. (2005). A Decomposition Model for the Layered Evaluation of Interactive Adaptive Systems. In Ardissono, L., Brna, P., & Mitrovic, A. (Eds.), Proceedings of the 10th International Conference on User Modeling (UM2005), Edinburgh, Scotland, UK, July 24-29 (pp. 438-442) (Lecture Notes in Computer Science LNAI 3538, Springer Verlag). Berlin: Springer.
• Weibelzahl, S. (2001). Evaluation of adaptive systems. In M. Bauer, P. Gmytrasiewicz & J. Vassileva (Eds.), User Modeling 2001: Proceedings of the Eighth International Conference, UM2001. (pp. 292-294) (Lecture Notes in Computer Science LNAI 2109; © Springer-Verlag). Berlin: Springer.
• Paramythis, A., Totter, A., & Stephanidis, C. (2001). A modular approach to the evaluation of Adaptive User Interfaces. In S. Weibelzahl, D. Chin & G. Weber (Eds.), Proceedings of the Workshop on Empirical Evaluations of Adaptive Systems, held in the context of the 8th International Conference on User Modeling (UM'2001), 13-17 July, Sonthofen, Germany (pp.9-24). Freiburg: Pedagogical University of Freiburg.

Methods in general and pitfalls

• Weibelzahl, S., Lippitsch, S., & Weber, G. (2002). Advantages, opportunities, and limits of empirical evaluations: Evaluating adaptive systems. Kόnstliche Intelligenz, 3/02, 17-20.
• Gena, C. & Weibelzahl, S. (2007). Usability Engineering for the Adaptive Web. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.): The Adaptive Web: Methods and Strategies of Web Personalization, Lecture Notes in Computer Science, Vol. 4321 (© Springer). Berlin: Springer.
• Weibelzahl, S. (2005). Problems and pitfalls in the evaluation of adaptive systems. In S. Chen & G. Magoulas (Eds.). Adaptable and Adaptive Hypermedia Systems (pp. 285-299). Hershey, PA: IRM Press.
• Hφφk, K., Karlgren, J., Waern, A., Dahlback, N., Jansson, C., Karlgren, K., and Lemaire, B. (1996). A glass box approach to adaptive hypermedia. User Modeling and User-Adapted Interaction, 6:157-184, 1996.

User as Wizard

• Masthoff, J. The user as wizard: A method for early involvement in the design and evaluation of adaptive systems. In S Weibelzahl, A Paramythis & J Masthoff (ed), Fifth Workshop on User-Centred Design and Evaluation of Adaptive Systems, associated with AH'06 (Dublin, Ireland), 2006.
• Nguyen, H., Masthoff, J. & Edwards, P. (2007). Modelling a receiver's position to persuasive arguments. In Proceedings of the Persuasive Conference (Stanford, USA), Springer-Verlag, Berlin. © Springer-Verlag
• Masthoff, J. & Gatt, A. (2006). In pursuit of satisfaction and the prevention of embarrassment: Affective state in Group Recommender Systems. User Modeling and User Adapted Interaction, 16, pp281-319.

Task-based Experiments

• Masthoff, J. (2002). The evaluation of adaptive systems. In N. V. Patel (Ed.), Adaptive evolutionary information systems. Idea Group publishing. pp329-347
• Weibelzahl, S., & Weber, G. (2002). Adapting to prior knowledge of learners. In de Bra, P., Brusilovsky, P., & Conejo, R. (Eds.), Proceedings of the second international conference on Adaptive Hypermedia and Adaptive Web Based Systems, Malaga, Spain, AH2002 (pp. 448-451). Berlin: Springer.
• Chin, D.N. (2001). Empirical Evaluation of User Models and User-Adapted Systems. User Modeling and User-Adapted Interaction 11: 181-194, 2001.

Simulated-users

• Masthoff, J. (2002). The evaluation of adaptive systems. In N. V. Patel (Ed.), Adaptive evolutionary information systems. Idea Group publishing. pp329-347
• Masthoff, J. & Gatt, A. (2006). In pursuit of satisfaction and the prevention of embarrassment: Affective state in Group Recommender Systems. User Modeling and User Adapted Interaction, 16, pp281-319.