International Workshop on Spoken Language Translation

 
Home Important dates

Evaluation campaign Submissions

Workshop Contact

References

 
 

Evaluation campaign
>> Evaluation specification

Evaluation Scores

  • Subjective Evaluation (for Chinese → English translations)

  • Automatic Evaluation (BLEU, NIST, WER, PER, METEOR and GTM)


Number of References


Arabic → English, Chinese → English, Japanese → English and Korean → English tracks have 16 English references.
English → Chinese track has only 1 Chinese reference.


Specification: Case and Punctuation marks


The evaluation will be done by submitting your translations to an online evaluation server. We will automatically detect and possibly change (lower casing and removal of punctuation marks) your submission format.
  • Word compounds will always be split (hyphen '-' will be replaced with space, "second-hand" → "second hand").
  • Apostrophes ("I'll", "He's") will not be changed.

Lower Case & Punctuation removed


All evaluations will be done under this condition. Every sentence is evaluated lower-cased, punctuation-removed

Optional: Mixed Case & Punctuation marks


Mixed case evaluation is not obligatory but an option of participants. Every sentence is evaluated mixed-cased with punctuation marks. Punctuations will be separated.



Specification: Segmentation


For English

  • The word segmentation by space in the original training data will be used for text and ASR ouput translation.

For Chinese

  • The data will be segmented using the ASR segmentation.


Specification: Output Format


MT results for ASR output

  • When submitting MT results for the ASR output, please submit both source hypothesis and its tranlation. The output file format should be as follows:

    Source Sentence 1
    Target Sentence 1

    Source Sentence 2
    Target Sentence 2

    ...

MT results for Manual transcriptions

  • One target sentence per line, in the same order of the source sentences given as test set.



Examples


  • Example submitted sentence:
    "They didn't buy any duty-free items, did they?"
  • Automatically changed for lower case/punctuation removed:
    "they didn't buy any duty free items did they"
  • Automatically changed for mixed case/with punctuations:
    "They didn't buy any duty-free items , did they ?"


 
        Copyright(c) 2005 interACT All rights reserved.