Please use this identifier to cite or link to this item: http://hdl.handle.net/10397/3779
Title: The study on automatic Chinese collocation extraction
Authors: Xu, Ruifeng
Keywords: Hong Kong Polytechnic University -- Dissertations
Chinese language -- Word order
Chinese language -- Data processing
Issue Date: 2006
Publisher: The Hong Kong Polytechnic University
Abstract: Collocation is a lexical phenomenon in which two or more words are habitually combined together as some conventional way of saying things. Collocation information is essential to many natural language processing tasks such as word sense disambiguation, machine translation, and information extraction. Most of current works on collocation extraction are statistical based with limited precision and recall because they can not well distinguish word co-occurrences, which are statistically significant, from true collocations, which are of habitual use and are thus either syntactically or semantically significant. The objective of this study is to investigate methods to improve the performance of Chinese collocation extraction algorithms. Different types of collocations are identified. Collocation extraction algorithms are then designed to target on different types of collocations using different features and criteria associated with these different types. In addition to improve statistical based collocation extraction algorithms, additional syntactic and semantic information are also incorporated into the algorithm to further improve the performance of collocation extraction. On the study of the statistical based algorithms, a new algorithm based on bi-directional word bi-grams is designed to help identify collocations with low co-occurrence frequency and are of fixed use. A large scale Chinese collocation answer set is established so that collocation extraction algorithms can be evaluated and compared objectively by using the same training corpus and corresponding answer set. Collocations are then categorized into four types based on their compositionality, substitutability, modifiability and internal association. Based on the characteristics of each type of collocations, a multi-stage window-based collocation extraction is built where the n-gram collocations and different types of bi-gram collocations are separately extracted in different stages using different strategies and different discriminative features. A Chinese shallow treebank, referred to as the PolyU Treebank, is annotated manually to provide syntactic and semantic knowledge to further help collocation extraction. This treebank is also used to train a chunker based on lexicalized Hidden Markov Model (HMM). The chunker provides ways to process running text for collocation extraction. By using the support collocation patterns and reject collocation patterns extracted from the annotated Chinese treebank and parsed running text, syntactic features are employed to further improve the performance of the window-based collocation extraction system. Experimental results show that the use of syntactic patterns can significantly improve the performance of collocation extraction, especially for filtering pseudo collocations. The extracted collocations were applied in the post-processing of a handwritten Chinese character recognition system. Experiments indicate that collocation information can be used in real application to improve the performance of these natural language related applications.
Description: xvi, 196 p. : ill. ; 30 cm.
PolyU Library Call No.: [THS] LG51 .H577P COMP 2006 Xu
URI: http://hdl.handle.net/10397/3779
Rights: All rights reserved.
Appears in Collections:Thesis

Files in This Item:
File Description SizeFormat 
b20593326_link.htmFor PolyU Users 161 BHTMLView/Open
b20593326_ir.pdfFor All Users (Non-printable) 3.02 MBAdobe PDFView/Open
Show full item record

Page view(s)

454
Last Week
2
Last month
Checked on Jan 15, 2017

Download(s)

243
Checked on Jan 15, 2017

Google ScholarTM

Check



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.