Crowdsourcing is one of the most promising trends of recent years. As I’m interested in this topic mainly from the perspective of language technology, I was wondering if we can use the crowd to obtain high-quality translations. Yesterday I found a paper by Omar F. Zaidan and Chris Callison-Burch that discusses crowdsourcing translations from Amazon’s Mechanical Turk.
Crowdsourcing translations is many times cheaper than hiring professional translators, but obviously there’s a risk of obtaining low-quality output. Zaidan and Callison-Burch describe how this trap can be avoided. Their recommendation is to stick to the following pipeline:
“collect multiple translations for each source sentence, collect rank labels for the translations, and finally collect edited versions of the top ranked translations” (p. 8)
First, it is necessary to collect multiple translations for each sentence, because a considerable number of crowdsourced translations is of relatively low quality. In one interesting piece of practical information, Zaidan and Callison-Burch suggest to present sentences as images to the Turkers, in order to avoid simple copy-and-paste into a machine translation system like Google Translate.
Second, all collected translations should be submitted for ranking. Zaidan and Callison-Burch asked US-based Turkers to order several crowdsourced translations for each sentence by fluency, in order to identify the best instances. Finally, these best translations are submitted for post-editing. In this phase, US-based Turkers were asked to edit the translations into more fluent and grammatical sentences.
Instead of having Turkers manually rank sentences, there are also a number of features that can help determine the quality of translations automatically. These include sentence-level features, like the length of sentences and their probability according to a language model, and worker-level features, like the native language and the location of the worker. Despite the predictive accuracy of these features, however, the automation of this phase brought down the general quality of the translations in Zaidan and Callison-Burch’s experiment.
Although crowdsourcing translations is a relatively cheap effort, the translations resulting from the pipeline above are of comparable quality to those of professional translators. Still, it looks like very few companies tap into this source. The short overview here shows that most so-called crowdsourcing translation companies simply rely on a wide network of freelancers. A hole in the market, maybe?
Yves Peirsman 