Title: Initial Investigations in Evaluation of Speech Synthesis Evaluation
Speaker: Tina Bennett

Evaluation of speech synthesis systems is a challenging problem.
There are several commonly used methods for evaluation, but it is
sometimes unclear how to best utilize them.  We seek to learn how to
conduct evaluations such that they are straightforward and their
results are maximally useful for learning about users' preferences and
making improvements to our systems.  A small evaluation was conducted
to explore this topic.  Volunteers with varying levels of experience
in listening to synthetic speech were included.  Each participant
listened to audio samples from three distinct systems in tests devised
for four genres: news, novels, conversation, and semantically
unpredictable sentences (SUS).  Mean opinion scores were used in the
first three tests, whereas a "type what you hear" format was used for
the SUS test.  Results and insights from the evaluation will be
discussed.