QTLaunchPad Workshop at EAMT, Dubrovnik, June 15th 2014

Quality Translation: Where Are We Now, and Where Are We Going?

QTLaunchPad Half-Day Workshop held in conjunction with the 17th Annual Conference of the European Association for Machine Translation (EAMT) in Dubrovnik, Croatia (June 15th 2014)

Programme now available here


  • Bogdan Babych (University of Leeds)
    • Machine Translation evaluation for MT development: Improving MT quality with evaluation-oriented methods [slides]
  • Christian Federmann (Microsoft)
    • Unfortunately, Christian will be unable to attend due to an emergency
  • Kim Harris (text&form)
    • Multidimensional Quality Metrics: The Art of the Science [slides]
  • Maja Popovic (German Research Centre for Artificial Intelligence)
    • The limits of automation in MT evaluation [slides]
  • Lucia Specia (University of Sheffield)
    • Predicting human and machine translation quality [slides]
  • Antonio Toral & Federico Gaspari (CNGL, Dublin City University)
    • Source-language phenomena triggering MQM errors [slides]
  • Jost Zetzsche (International Writers' Group)
    • Are Quality Metrics for Machine Translation the Way to the Heart of the Translator? [slides]

The QTLaunchPad Project

QTLaunchPad is a European Commission-funded project which is preparing a large-scale research and innovation initiative focusing on quality translation technology for the 21st century. This effort is of interest to the whole MT community, and it involves a variety of stakeholders, including academics, researchers, technology developers and integrators, translation buyers, professional translators and language service providers.

QTLaunchPad has developed tools and resources to drive forward the fight against MT quality barriers, such as the Multidimensional Quality Metric (MQM, a new shared and flexible paradigm for human and machine translation quality evaluation and annotation), QuEst (an open-source tool for MT quality estimation) and DELiC4MT (an open-source toolkit for diagnostic MT evaluation). As part of its effort to showcase these technologies and engage with the wider community, QTLaunchPad is presenting the results of the project at high-profile international events, and is organising the Quality Estimation Shared Task during the Ninth WMT Workshop, that will be held at ACL 2014 in Baltimore (USA) on June 26-27, 2014.

The QTLaunchPad Workshop at EAMT 2014

As part of these ongoing engagement and dissemination activities, QTLaunchPad is organising a workshop supported by EAMT which will take place immediately before the Seventeenth Annual EAMT Conference in Dubrovnik (Croatia), on Sunday 15th June 2014. The main aims of this workshop are to present the tools and technologies developed by QTLaunchPad, elicit feedback from (potential) users, and discuss issues revolving around the central concern of quality translation, so that we can exchange opinions on where we are now in this respect, and discuss ways forward as to where we are (or should be) going next.

The workshop programme includes talks given by high-profile speakers with academic and industrial backgrounds, who will share their substantial expertise on several key aspects of quality translation. Special emphasis will be laid on predictive, diagnostic and semi-automatic approaches to translation quality evaluation, exploring the possibilities for bringing closer together the evaluation of human and machine translation, also on the basis of recent quality annotation efforts using the MQM undertaken with industrial partners. The participants will have the opportunity to learn about the innovative applications of the MQM, QuEst and DELiC4MT directly from the researchers who have developed these tools within QTLaunchPad, in close collaboration with representatives from the translation and localisation industry.

In addition to providing training on the latest developments in these areas, the workshop will act as a forum to collect feedback from (potential) users of these new and exciting tools and metrics, and to discuss future desirable directions to enhance quality translation and its evaluation that would benefit the entire community. Participation is free, but workshop participants must register in advance via the EAMT 2014 Conference website (http://hnk.ffzg.hr/eamt2014/registration.html).

Talk Abstracts

Machine Translation evaluation for MT development: Improving MT quality with evaluation-oriented methods by Bogdan Babych
The talk will discuss the link between methods for evaluating vs. improving MT output: interestingly, automated MT evaluation techniques can be transformed into models for MT architectures that work from inside MT engines and improve translation quality. Examples of such methods are Information Extraction and Named Entity recognition from MT output, evaluation-based automated error analysis. MT that uses these models is usually over-rated by the metrics for which it was optimized (e.g., Callison-Burch et al., 2006), so we will always have to develop new ‘surprise’ automated metrics and to characterize the limits of their applicability. However, different usage scenarios for MT often have different quality requirements, such as MT for comprehension, internal communication, or post-editing; MT developers and professional translators often use different assumptions in their evaluation; and there is a need to pre-evaluate resources used for training MT and their comparability with user translation projects – in terms of subject domains and genres. From this perspective the talk will outline requirements for a flexible usability-oriented evaluation methodology that addresses these issues and can be systematically integrated into new models that improve the quality of Machine Translation.

The limits of automation in MT evaluation by Maja Popovic
Human evaluation is a resource-intensive and time-consuming task. Therefore various automatic evaluation approaches have been developed and have become widely used. However, these automatic methods cannot provide all the details yet, and they also require some kind of human judgments for comparison.

Predicting human and machine translation quality by Lucia Specia
This talk will contrast quality estimation of machine versus human translations. Using datasets recently collected within the QTLaunchPad project for both sentence- and word-level prediction, we will present experiments comparing the performance of models for human translation against models for machine translations, as well as models that mix these two types of translations. We will then discuss the main  challenges in automatically predicting the quality of human translations, from data collection to feature engineering.

Are Quality Metrics for Machine Translation the Way to the Heart of the Translator? by Jost Zetzsche
The relationship between professional translators and the machine translation community has been difficult. Creating transparent ways of measuring the quality of MT output might be one way to bridge this gap. This would allow translators to evaluate how reasonable the use of MT is in their daily work and how it can even be used as an effective sales tool for their services. For this to succeed, communication as well as integration into the translator’s technological environment would be essential. The talk will suggest some ways that this might be possible.

Workshop Organisers:

  • John Judge, CNGL, Dublin City University [jjudge AT computing.dcu.ie]
  • Federico Gaspari, CNGL, Dublin City University [fgaspari AT computing.dcu.ie]
  • Ankit Srivastava, CNGL, Dublin City University [asrivastava AT computing.dcu.ie]
  • Antonio Toral, CNGL, Dublin City University [atoral AT computing.dcu.ie]