A deep learning model to recognize communication-oriented entity of ICT in construction

Abstract

Due to the fragmented nature and complexity of the industry, effective communication is increasingly recognized as a key factor to enable real-time transfer of information and to achieve the success of construction projects. Information and communication technology (ICT) has been recognized as an important determinant to enable and enhance the communication. A large volume of unformatted textual data is available in digital forms in the current web 2.0 era, and it seems no exception for the field of ICT in construction. There is a number of experts who are skilled in the ICT and familiar with the construction industry, leaving a large volume of technical documents (such as handbooks, patents, literature, and reports) embodied with professional and skilled knowledge. This motivates this study to investigate those textual data of ICT in construction. The key component of ICT in construction is the communication functionality, forming the communicating process whereby construction data is coordinated during the whole life cycle of construction projects. Most of the functionalities' specifications are archived as written language in the technical documents of ICT in construction, mainly by referring three types of communication-oriented entity (CE): transferred information, communication models and communication subjects. This study seeks to develop an entity recognition approach that can automatically identify the CEs from raw text and categorize them into pre-defined entity types. Entity recognition (sometimes called concept extraction, term identification or event extraction) has become an important approach in recent informatics studies of the Construction Engineering and Management (CEM) domain, aiming to automatically extract all the entities (sometimes called concepts, textual elements, textual items, or terminologies) from raw text that are relevant to a specific domain. Recently, a small number of CEM studies utilized rule-based and pre-defined vocabularies to enable automatic entity recognition to create digital dictionaries. The problems of these approaches are two-fold. First, these methods require expertise knowledge and enormous efforts for the establishment of rules and vocabularies, which are extremely time-consuming and labor-intensive. In addition, pre-defined vocabularies exclude unknown entities (the entities exist but are not known by the developer), resulting in the missing of numerous relevant entities. Second, those methods always lead to lower accuracy in discerning ambiguous entities (the spellings can appear as an entity at one position and common noun at another position, or appear as different entity types). Those methods typically utilized external lexical databases such as WordNet to discern ambiguous entities, but the performance is still unsatisfactory due to the limited coverage of the lexical databases and ignoring contextual information. To remedy these problems, the proposed approach resorts to the techniques from the realm of deep learning that can utilize the contextual information within raw texts. In recent years, deep learning techniques have been recognized as a powerful tool to aid human beings in solving complex tasks to explore and utilize the unstructured text data in a robust way, in which the embedded information can be extracted in a readable manner. The key merit of deep learning is its capacity to utilize the contextual information within raw texts, rather than the external linguistic resources that are required in previous studies. The proposed deep learning model is developed through the transformer, a novel neural network structure with self-attention mechanism, which enables parallelly computing among the input sequence in memorizing contextual information. In this way, the whole model improves performance compared with traditional RNN in which the contextual information is computed sequentially. Specifically, the objectives of this study are as follows: (1) To develop a deep learning model for entity recognition which can automatically recognize CEs from raw text through utilizing the contextual information. (2) To compile a patent database of ICT in construction and to acquire annotated data as training and testing instances for the deep learning model of communication-oriented entity recognition (CER). (3) To train and validate the deep learning model and to make it tailored for CER, which achieves intelligence to recognize the CEs from technical documents of ICT in construction like human beings to understand the contextual meanings. This study first reviewed relevant research about ICT in construction and entity recognition to identify the research gaps. The basic concepts of CER of ICT in construction were clarified and the technical problems of CER were highlighted. Considering the technical problems embodied in recognizing CEs from the technical documents of ICT in construction, a deep learning approach based on transformer was developed to recognize CEs from the technical documents of ICT in construction, which enables the whole model to understand the contextual meanings. The patents of ICT in construction were compiled as an initial dataset, and the training and testing instances for CER were achieved by manually tagging CEs in each sentence of the database. The validation was carried in two perspectives: (1) The accuracy of the deep learning model in recognizing the CEs was validated, compared to the traditional RNN-based deep learning model; (2) The practical value of recognized CEs was validated by applying the CEs as features to achieve a classification scheme that categorizes patents of ICT in construction into different communication modes. The key findings achieved from this study are as follows. First, a deep learning model was developed based on the transformer for entity recognition that was tailored to CER of ICT in construction. The validation results indicated that it outperformed the traditional RNN-based deep learning model, yielding better performance by 15%. Second, through the training process, the developed model acquired intelligence that is able to recognize a CE by addressing the contextual information that surrounds the CE. Thirdly, a database of patents of ICT in construction was compiled, and the CEs of each patent was recognized by the deep learning model. The recognized CEs are more valuable and informative than common words in indicating the communication patterns of ICT in construction, resulting in better performance when they were used as features for classifying the patents into different communication modes. This study contributes to knowledge in three aspects. First, a deep learning model was developed for entity recognition, which can identify and classify the entities out of common words by utilizing the contextual meanings of the surrounding text, rather than the word-level features and syntactic rules that were used in previous construction informatics studies. In this way, the proposed approach can recognize unknown entities (entities beyond a pre-defined vocabulary) and improve the performance in recognizing ambiguous entities. Second, the developed model utilizes the "self-attention" mechanism to capture the contextual meanings in recognizing the CEs through the training process, leading to better performance compared to the traditional RNN-based model. The trained deep learning model acquires the intelligence like human beings to recognize the CEs by understanding the contextual meanings rather than the word-level meanings. Third, this study contributes an effective NLP approach for the practitioners to access and perceive the communication functionality underlying the patents of ICT in construction. This can help acknowledge how the up-to-data inventions employ and utilize devices to scheme the information flows.