ELRA Workshop on Evaluation at LREC 2008

Looking into the Future of Evaluation

LREC 2008 - 27 May 2008, Marrakech

Held in conjunction with the 6th International Language Resources and Evaluation Conference ()

View/download the Workshop’s Proceedings (PDF)

Workshop Chairing Team

  • Gregor Thurmair, Linguatec Sprachtechnologien GmbH, Germany - chair
  • Khalid Choukri, ELDA - Evaluations and Language resources Distribution Agency, France - co-chair
  • Bente Maegaard, CST, University of Copenhagen, Denmark - co-chair

Workshop Programme

09:00 Welcome and Introduction

09:15 Technology Advancement has Required Evaluations to Change Data and Tasks — and now Metrics Mark Przybocki; NIST, USA

10:00 Explicit and Implicit Requirements of Technology Evaluations: Implications for Test Data Creation Lauren Friedman, Stephanie Strassel, Meghan Lammie Glenn; Linguistic Data Consortium, USA

10:35 Coffee break

11:00 Automated MT Evaluation for Error Analysis: Automatic Discovery of Potential Translation Errors for Multiword Expressions Bogdan Babych, Anthony Hartley; Centre for Translation Studies, University of Leeds, United Kingdom

11:35 Discussion

12:15 Reference-based vs. Task-based Evaluation of Human Language Technology Andrei Popescu-Belis; IDIAP Research Institute, Switzerland

12:50 FEIRI: Extending ISLE’s FEMTI for the Evaluation of a Specialized Application in Information Retrieval Keith J. Miller; The MITRE Corporation, USA

13:25 Lunch

14:30 Discussion

15:15 Evaluating a Natural Language Processing Approach in Arabic Information Retrieval Nasredine Semmar, Laib Meriama, Christian Fluhr; CEA, LIST, Laboratoire d’ingénierie de la Connaissance Multimédia Multilingue, France, NewPhenix, France

15:50 Coffee break

16:20 A Review of the Benefits and Issues of Speaker Verification Evaluation Campaigns Asmaa El Hannani, Jean Hennebert; Department of Computer Science, University of Sheffield, UK, HESSO, Business Information Systems, Switzerland and University of Fribourg, Switzerland

16:55 Field Testing of an Interactive Question-Answering Character Ron Artstein, Sudeep Gandhe, Anton Leuski and David Traum; Institute for Creative Technologies, University of Southern California, USA

17:30 Discussion and conclusions

18:15 Close