********************************************************************** ** ** ** Third Workshop on ** ** Empirical Evaluation of Adaptive Systems ** ** ** ** ** ** Held in conjunction with the ** ** Third International Conference on Adaptive Hypermedia and ** ** Adaptive Web-Based Systems (AH2004) ** ** Eindhoven, The Netherlands, August 23 - 26 ** ** ** ** ** ** http://www.easy-hub.org/hub/workshops/ah2004/index.html ** ** ** ********************************************************************** You are kindly invited to contribute to the Third Workshop on Empirical Evaluation of Adaptive Systems, which follows in the tracks of two very successful workshops held in conjunction with UM2001 and UM2003. This workshop intends to explore methods and criteria that can be employed in assessing (interactive) adaptive systems. The perspective taken is that a build-up of knowledge in this direction can serve as a driving force in facilitating and fostering detailed empirical studies with significant results in the field of user modeling something lacking from the literature today. The workshop seeks to cover a wide range of application domains and user modeling techniques and aims to both raise awareness of researchers and practitioners in the area, as well as contribute towards a widely accepted evaluation "toolset" for adaptive systems. Workshop web site: http://www.easy-hub.org/hub/workshops/ah2004/index.html Web sites and proceedings of the two previous workshops: http://www.easy-hub.org/hub/workshops.jsp Background and Motivation Only few publications on user modeling systems report significant evaluation results. One of the reasons that have been identified to be responsible for this lack is that both methods and criteria for assessing the "success" of adaptation are not clear. However, user modeling systems are designed for human-computer interaction and thus, empirical approaches for such assessment are absolutely necessary in order to justify the efforts required to render systems adaptive. This is the third workshop on empirical evaluation of adaptive systems. The workshop's guiding perspective is that adequate methods and reliable criteria are prerequisites towards increasing the quantity and quality of evaluation studies on adaptive systems. The workshop aims to contribute to the exploration and discussion of suitable methods and criteria in various domains with differing user modeling and adaptation techniques. It further aims to encourage researchers to perform evaluation studies with their own adaptive hypermedia systems. Questions The workshop, continuing in the steps of its predecessor, will focus on the following questions: * Which of the existing empirical criteria and methods are appropriate for the evaluation of adaptive hypermedia systems? What new criteria need to be introduced to specifically cater for the presence of adaptation in the evaluated systems? What empirical methods are appropriate (or, how do existing methods need to be modified, so as to be suitable for) assessment against the new sets of criteria? * Can metrics be developed to facilitate the comparison between different (versions of) adaptive systems, or between adaptive and non-adaptive systems? * How can we foster an increase in the volume and quality of empirical evaluations of adaptive systems? What are the most common pitfalls that can be identified in previous studies? Goals and Workshop Format The goals of this workshop are, on the one hand, to consolidate existing knowledge in the area of adaptive hypermedia system evaluation, and, on the other hand, to provide researchers and practitioners in the field with a set of "tools" and guidelines, which could foster future work and facilitate a more homogeneous view of related evaluation activities. The format of the workshop will be such as to facilitate goal-oriented discussions; specifically, there will be short presentations of the accepted papers, followed by group discussion (among the workshop participants). The workshop will be organized around the research questions above, i.e., submissions are to be evaluated in regard to their contribution to the questions, and the discussion during the workshop will be structured around the questions. Participants of the workshop are expected to share their experiences and discuss the issue of what constitutes good and bad practice in studies targeting adaptive systems. Two types of submissions will be solicited to achieve this goal: studies and position-papers. Each study will be discussed in terms of experience gathered. The submission of "unsuccessful studies" is explicitly encouraged, as these can be very informative for other participants. During the workshop we will work towards a list of recommendations for the performance of an evaluation study. Position-papers will be solicited mainly in relation to the second and third questions above. Authors of such papers are expected to build upon related experience to discuss and propose either new (or, reworked existing) metrics for adaptation, or, perhaps more interestingly, address the issue of why related studies in the literature are so few and mostly uninformative. The topics to be raised by the authors will be integrated in (and, hopefully, enrich) the aforementioned study-based workshop discussions. Submission Format As stated above, each submission is required to address at least one of the main workshop questions. Fulfillment of this requirement will be assessed in the course of the review process. Submissions should not exceed 10 pages and should be formatted according to the Springer LNCS guidelines. Please refer to the submission section for more information on the submission format. Workshop papers will be published in full length in the workshop proceedings and presented in talks at the workshop. Please, submit your paper by e-mail to ah2004-workshop@easy-hub.org until Friday May 28th, 2004. Portable Document Format (*.pdf) and Postscript (*.ps) files are preferred. Important dates May 28th, 2004: Submission of papers June 21st, 2004: Notification of authors July 14th, 2004: Delivery of camera ready copy August 23rd, 2004: Workshop date; the main conference will last until August 26th, 2004 Program Committee Peter Brusilovsky, University of Pittsburgh, USA Judith Masthoff, University of Brighton, UK Alexandros Paramythis, Johannes Kepler University Linz, Austria Gerhard Weber, University of Education Freiburg, Germany Stephan Weibelzahl, Fraunhofer IESE, Germany Organizers Dr. Stephan Weibelzahl DOC - Document Engineering Fraunhofer IESE - Institute Experimental Software Engineering Sauerwiesen 6 D-67661 Kaiserslautern Germany +49 (0) 6301 707 255 stephan.weibelzahl@iese.fraunhofer.de Alexandros Paramythis Institute for Information Processing and Microprocessor Technology Johannes Kepler University Linz Altenbergerstr. 69 A-4040 Linz Austria +43 (0) 732 2468 8442 alpar@fim.uni-linz.ac.at