Evaluation of Statistical Text Normalisation Techniques for Twitter

Phavanh Sosamphan, Veronica Liesaputra, Sira Yongchareon, Mahsa Mohaghegh

2016

Abstract

One of the major challenges in the era of big data use is how to ‘clean’ the vast amount of data, particularly from micro-blog websites like Twitter. Twitter messages, called tweets, are commonly written in ill-forms, including abbreviations, repeated characters, and misspelled words. These ‘noisy tweets’ require text normalisation techniques to detect and convert them into more accurate English sentences. There are several existing techniques proposed to solve these issues, however each technique possess some limitations and therefore cannot achieve good overall results. This paper aims to evaluate individual existing statistical normalisation methods and their possible combinations in order to find the best combination that can efficiently clean noisy tweets at the character-level, which contains abbreviations, repeated letters and misspelled words. Tested on our Twitter sample dataset, the best combination can achieve 88% accuracy in the Bilingual Evaluation Understudy (BLEU) score and 7% Word Error Rate (WER) score, both of which are considered better than the baseline model.

Paper Citation

in Harvard Style

Sosamphan P., Liesaputra V., Yongchareon S. and Mohaghegh M. (2016). Evaluation of Statistical Text Normalisation Techniques for Twitter . In Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR, (IC3K 2016) ISBN 978-989-758-203-5, pages 413-418. DOI: 10.5220/0006083004130418

in Bibtex Style

@conference{kdir16,
author={Phavanh Sosamphan and Veronica Liesaputra and Sira Yongchareon and Mahsa Mohaghegh},
title={Evaluation of Statistical Text Normalisation Techniques for Twitter},
booktitle={Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR, (IC3K 2016)},
year={2016},
pages={413-418},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006083004130418},
isbn={978-989-758-203-5},
}

in EndNote Style

TY - CONF
JO - Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR, (IC3K 2016)
TI - Evaluation of Statistical Text Normalisation Techniques for Twitter
SN - 978-989-758-203-5
AU - Sosamphan P.
AU - Liesaputra V.
AU - Yongchareon S.
AU - Mohaghegh M.
PY - 2016
SP - 413
EP - 418
DO - 10.5220/0006083004130418