A method for evaluating MT performance embedded in Cross-Language Instant Messaging (CLIM) systems is presented. A web interface that provided concurrent real- time translation for instant messaging from multiple MT services was developed and used by paid participants to collaborate on a photo identification task. The method showed a task performance benefit due to the availability of multiple translation alternatives.
Ogden, William, Ron Zacharski, Sieun An and Yuki Ishikawa. 2009. User choice as an evaluation metric for web translation services in cross language instant messaging applications. Proceedings of the Machine Translation Summit XII. Ottawa, Canada. (PDF)