JUCS - Journal of Universal Computer Science 16(9): 1190-1214, doi: 10.3217/jucs-016-09-1190
LemmaGen: Multilingual Lemmatisation with Induced Ripple-Down Rules
expand article infoMatjaž Juršič, Igor Mozetič, Tomaž Erjavec, Nada Lavrač
‡ Jožef Stefan Institute, Ljubljana, Slovenia
Open Access
Abstract
Lemmatisation is the process of finding the normalised forms of words appearing in text. It is a useful preprocessing step for a number of language engineering and text mining tasks, and especially important for languages with rich inflectional morphology. This paper presents a new lemmatisation system, LemmaGen, which was trained to generate accurate and efficient lemmatisers for twelve different languages. Its evaluation on the corresponding lexicons shows that LemmaGen outperforms the lemmatisers generated by two alternative approaches, RDR and CST, both in terms of accuracy and efficiency. To our knowledge, LemmaGen is the most efficient publicly available lemmatiser trained on large lexicons of multiple languages, whose learning engine can be retrained to effectively generate lemmatisers of other languages.
Keywords
rule induction, ripple-down rules, natural language processing, lemmatisation