LipReadingSentencesintheWild.pdfVIP

Lip Reading Sentences in the Wild Joon Son Chung1 Andrew Senior2 Oriol Vinyals2 joon@robots.ox.ac.uk andrewsenior@ vinyals@ 1Department of Engineering Science, University of Oxford Andrew Zisserman1,2 az@robots.ox.ac.uk 2Google DeepMind arXiv:1611.05358v1 [cs.CV] 16 Nov 2016 Abstract The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem – unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) a ‘Watch, Listen, Attend and Spell’ (WLAS) network that learns to transcribe videos of mouth motion to characters; (2) a curriculum learning strategy to accelerate training and to reduce over?tting; (3) a ‘Lip Reading Sentences’ (LRS) dataset for visual speech recognition, consisting of over 100,000 natural sentences from British television. The WLAS model trained on the LRS dataset surpasses the performance of all previous work on standard lip reading benchmark datasets, often by a signi?cant margin. This lip reading performance beats a professional lip reader on videos from BBC television, and we also demonstrate that visual information helps to improve speech recognition performance even when the audio is available. 1. Introduction Lip reading, the ability to recognize what is being said from visual information alone, is an impressive skill, and very challenging for a novice. It is inherently ambiguous at the word level due to homophemes – different characters that produce exactly the same lip sequence (e.g. ‘p’ and ‘b’). However, such ambiguities can be resolved to an extent using the context of neighboring words in a sentence, and/or a language model. A machine that can lip read opens up a host of applications: ‘dictating’ instructions or messages to a phone in a noisy environment; transcribing and re-dubbing archival silent ?lms