---

Tutorial slides (handout format)

System leaflets (for the hands-on sessions)

---

Adaptive systems are by definition interactive systems. As such they stand to benefit considerably from a development lifecycle that ensures user involvement from the early design stages, and embraces evaluation, in both formative and summative forms. Evaluating an adaptive system involves a number of specific problems and pitfalls that need to be addressed by the selection of specific methods, techniques and criteria.

Accordingly, the guiding perspective of this tutorial is that adequate evaluation methods, reliable assessment criteria and metrics are prerequisites for improving the quality and usability of the next generations of adaptive systems.

This tutorial aims to raise the awareness of adequate evaluation methods and techniques in order to foster an increase in the quantity and quality of evaluation studies of adaptive systems. It aims to share experiences and guide participants towards a proper evaluation of their own systems.

Participants will achieve the following learning objectives:

• Awareness of the specific problems involved in the evaluation of adaptive systems that differentiate them from their non-adaptive counterparts, and being able to solve or circumvent these problems
• Understanding and being able to apply the principles of layered evaluation of adaptive systems
• Ability to design a targeted evaluation study for an adaptive system (e.g., addressing a given layer and set of criteria) by selecting appropriate methods and criteria

Target Audience

The tutorial participants are expected to be researchers and practitioners working in the field of adaptive hypermedia, user modeling, intelligent tutoring systems, intelligent agents, etc. While experience with evaluation gained in the context of their own work would be of advantage, prior experience is not required. Participants will benefit from getting a rich insight into the current state-of-the-art research in the field, with a view to informing their on-going and future work.

Content

The tutorial will cover the following content:

• Motivation: What are the specific problems involved in the evaluation of adaptive systems?
• Layered evaluation:
  • Why layered evaluation? Rationale of the approach
  • Separation of layers
  • "Slicing" vs. "dicing": separating layers or separating user groups across the layers
  • Discussion of advantages and limitations
• Methods and criteria:
  • Identification of relevant criteria
  • Difficulty in defining appropriate criteria for adaptive systems
  • Examples of methods such as controlled experiment, Wizard of Oz, heuristic evaluation, user simulation and their adaptation or instantiation for adaptive systems
• Typical pitfalls in evaluating adaptive systems
• Available resources

Participants will be pointed towards reading material and existing presentations on evaluation in general (like experimental design, heuristic evaluation for usability testing).

We aim to make the learning experience of this tutorial as interactive and learner-centred as possible. The sessions will use a combination of interactive lectures and practical application of methods to case studies. Participants will be split into groups (of 5-7 participants) to work on a case study of their choice.

Lecture notes will be made available during the tutorial and afterwards on the website.

Date and time

Monday, June 21st 2010, Full Day

Organizers

Dr. Stephan Weibelzahl
National College of Ireland Dublin
Mayor Street IFSC
Dublin 1
Ireland
+353 1 4498 579
sweibelzahl at ncirl.ie
http://www.weibelzahl.de/

Dr. Alexandros Paramythis
Institute for Information Processing and Microprocessor Technology (FIM)
Johannes Kepler University Linz
Altenbergerstr. 69
A-4040 Linz, AUSTRIA
+43 732 2468 8442
alpar at fim.uni-linz.ac.at
http://www.fim.uni-linz.ac.at/staff/paramythis/

Dr. Judith Masthoff
Department of Computing Science
University of Aberdeen
Aberdeen AB24 3UE
Scotland, UK
+44 1224 272299
j.masthoff at abdn.ac.uk
http://www.csd.abdn.ac.uk/~jmasthof/