From n-gram to skipgram to concgram

Abstract

Uncovering the extent of word associations and how they are manifested has been an important area of study in corpus linguistics since the 1960s (Sinclair et al. 1970). This paper defines and describes a new way of categorising word association, the concgram, which constitutes all of the permutations of constituency and positional variation generated by the association of two or more words. Concgrams are identified without prior input from the user (other than to set the size of the span) employing a fully automated search that reveals all of the word association patterns that exist in a corpus. This study argues that concgrams represent more fully word associations in a corpus. Most concgrams seem to be non-contiguous, and show both constituency (AB, ACB) and positional (AB, BA) variations. Further studies of concgrams will help in the task of uncovering the full extent of the idiom principle (Sinclair 1987).