Fourth Workshop on the
Evaluation of Adaptive Systems

Download the call for papers in simple text format ( 12 KB) or PDF format ( 125 KB).

Page contents

• Introduction
• Thematic Areas
• First EAS Evaluation Challenge
• Workshop Format and Solicited Contributions
• Submission Format
• Important Dates

You are kindly invited to contribute to the Fourth Workshop on the Evaluation of Adaptive Systems (EAS), which follows in the tracks of three very successful workshops held in conjunction with UM2001, UM2003 and AH2004. The workshop's guiding perspective is that adequate methods and reliable criteria are prerequisites towards increasing the quantity and quality of evaluation studies on adaptive systems. The workshop aims to contribute to the exploration and discussion of suitable methods and criteria in various domains with differing user modelling and adaptation techniques. It further aims to encourage researchers to perform evaluation studies with their own systems.

Thematic Areas

The workshop, continuing in the steps of its predecessors, will focus on the following general themes:

• Criteria and methods for evaluating adaptation
  • Which of the existing criteria and methods are appropriate for the evaluation of user models and adaptive systems?
  • What new criteria need to be introduced to specifically cater for the presence of user modelling and adaptation in the evaluated systems?
  • What empirical methods are appropriate (or how do existing methods need to be modified so as to be suitable) for assessment against the new sets of criteria?
     
• Adaptation metrics
  • What aspects of user modelling / adaptation offer themselves to assessment through qualitative or quantitative measures?
  • Can metrics be developed to facilitate the comparison between different (versions of) adaptive systems, or between adaptive and non-adaptive systems?
  • Can metrics be developed to assess adaptation in “absolute” terms (e.g. on the basis of well-defined scales), thus making it possible to conduct studies that do not involve comparisons between systems?
     
• Encouraging and supporting evaluation studies
  • How can we foster an increase in the volume and quality of empirical evaluations of adaptive systems?
  • What are the most common pitfalls that can be identified in previous studies?

Expanding upon the scope of its predecessors, this year’s workshop also has an explicit focus on non-empirical approaches to evaluating user models and systems that employ them as a means towards exhibiting intelligent /adaptive behaviour. Of particular interest are metrics / approaches that would enable one to judge factors such as correctness, effectiveness, efficiency, etc., during early stages of the development cycle of user modelling systems, where, typically, prototypes are still not functional or mature enough to involve users in the evaluation process. Also of interest are approaches that can be employed to lower the inarguably high costs and complexity involved in performing full-scale evaluations of adaptive / intelligent systems.

The workshop will also include a talk delivered by the workshop organisers, on the topic of common pitfalls and problems when using “traditional” approaches to evaluate adaptive systems. This talk is intended as a condensed overview of the results of previous workshops, and will serve both as a mini “tutorial ” on the subject of empirically evaluating adaptive systems, and as a basis for further discussion among participants.

First EAS “Evaluation Challenge”

Further to the above, this workshop will launch the first in a series of “evaluation challenges”. The challenge concerns an adaptive system, which recommends sequences of music clips to groups of users. Participation in the challenge entails proposing an empirical evaluation design purposely created to answer specific design questions regarding the system’s modelling component. A detailed description of the system at hand and the challenge’s requirements can be found on the Evaluation Challenge page of this site.

In short, to participate in the challenge, you need to prepare and submit an entry following the guidelines and using the material in the challenge description. Submitted proposals will go through a typical review process and the selected ones will be presented during a dedicated track in the workshop. In that same track, workshop participants will discuss the merits and shortcomings of different entries, and will vote on the “winning” entry. The winner(s) that will be declared at the end of that session will also be announced as part of the report on the workshop’s outcomes to be delivered during the main UM conference. Finally, the winning proposal will form the basis for the evaluation of a real-world system that is very close to the one described for the challenge (the evaluation will be overseen by members of the workshop’s organising committee), and the winner(s) will be included as co-author(s) on any publications that are based on the evaluation proposal.

This evaluation challenge is an ideal way to contribute to the workshop, if you do not have your own evaluation to report on. Also, an ideal way to share your evaluation expertise, and perhaps be recognized for it by winning! We hope that this challenge will lead to animated interactions during the workshop, focussing the discussion on a real world problem, and would like to invite you to visit the workshop’s web site for additional information.

Workshop Format and Solicited Contributions

We plan to apply a format that facilitates goal-oriented group discussions (among the workshop participants) in addition to the standard paper presentations. The workshop will have two general sessions (to be held in succession), as outlined below.

The first session will be organized around the general research themes discussed earlier, i.e., submissions are to be evaluated in regard to their relevance to / contribution towards the themes, and the discussion during the workshop will be structured around the themes. Participants of the workshop are expected to share their experiences and discuss the issue of what constitutes good and bad practice in studies targeting adaptive systems. Two types of submissions are solicited to achieve this goal: studies and position papers. Each study will be discussed in terms of experience gathered. The submission of "unsuccessful studies" is explicitly encouraged, as these can be very informative for other participants. Position papers can address any of the workshop themes. Authors of such papers are expected to build upon related experience to discuss and propose either new (or reworked existing) metrics / approaches for evaluating adaptation or address the issue of why related studies in the literature are so few and mostly uninformative. The topics to be raised by the authors will be integrated in (and, hopefully, enrich) the aforementioned study-based workshop discussions.

The second session will be organized around the “evaluation challenge”. As already mentioned the papers solicited for this session are proposals on evaluation approaches for a given adaptive system. During the workshop, selected proposals will be presented by their respective authors. Presentations will be followed by a discussion session, where participants will be invited to comment on the merits and drawbacks of individual proposals. The discussion will culminate into a voting session, through which the winning proposal(s) will be selected.

Submission Format

Submissions should not exceed 10 pages for studies / position papers and 3 pages for evaluation challenge entries, and should be formatted according to the Springer LNCS guidelines. Please refer to the Submission page of this site for more information on the submission format.

Workshop papers will be published in full length in the workshop proceedings and presented in talks at the workshop.

Please, submit your paper on-line at the workshop's conference management site (refer to the Submission page for details). If you encounter problems in using the site's tools, you can also submit the paper by e-mail to um2005-workshop@easy-hub.org until Monday, March 13th, 2005. Portable Document Format (*.pdf) and Postscript (*.ps) files are preferred.

Important dates