Automatic Viseme Vocabulary Construction to Enhance Continuous Lip-reading

Adriana Fernandez-Lopez, Federico M. Sukno

2017

Abstract

Speech is the most common communication method between humans and involves the perception of both auditory and visual channels. Automatic speech recognition focuses on interpreting the audio signals, but it has been demonstrated that video can provide information that is complementary to the audio. Thus, the study of automatic lip-reading is important and is still an open problem. One of the key challenges is the definition of the visual elementary units (the visemes) and their vocabulary. Many researchers have analyzed the importance of the phoneme to viseme mapping and have proposed viseme vocabularies with lengths between 11 and 15 visemes. These viseme vocabularies have usually been manually defined by their linguistic properties and in some cases using decision trees or clustering techniques. In this work, we focus on the automatic construction of an optimal viseme vocabulary based on the association of phonemes with similar appearance. To this end, we construct an automatic system that uses local appearance descriptors to extract the main characteristics of the mouth region and HMMs to model the statistic relations of both viseme and phoneme sequences. To compare the performance of the system different descriptors (PCA, DCT and SIFT) are analyzed. We test our system in a Spanish corpus of continuous speech. Our results indicate that we are able to recognize approximately 58% of the visemes, 47% of the phonemes and 23% of the words in a continuous speech scenario and that the optimal viseme vocabulary for Spanish is composed by 20 visemes.

Paper Citation

in Harvard Style

Fernandez-Lopez A. and M. Sukno F. (2017). Automatic Viseme Vocabulary Construction to Enhance Continuous Lip-reading . In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5: VISAPP, (VISIGRAPP 2017) ISBN 978-989-758-226-4, pages 52-63. DOI: 10.5220/0006102100520063

in Bibtex Style

@conference{visapp17,
author={Adriana Fernandez-Lopez and Federico M. Sukno},
title={Automatic Viseme Vocabulary Construction to Enhance Continuous Lip-reading},
booktitle={Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5: VISAPP, (VISIGRAPP 2017)},
year={2017},
pages={52-63},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006102100520063},
isbn={978-989-758-226-4},
}

in EndNote Style

TY - CONF
JO - Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5: VISAPP, (VISIGRAPP 2017)
TI - Automatic Viseme Vocabulary Construction to Enhance Continuous Lip-reading
SN - 978-989-758-226-4
AU - Fernandez-Lopez A.
AU - M. Sukno F.
PY - 2017
SP - 52
EP - 63
DO - 10.5220/0006102100520063