Using machine learning to analyze longitudinal data : a tutorial guide and best-practice recommendations for social science researchers

Abstract

This article introduces the research community to the power of machine learning over traditional approaches when analyzing longitudinal data. Although traditional approaches work well with small to medium datasets, machine learning models are more appropriate as the available data becomes larger and more complex. Additionally, machine learning methods are ideal for analyzing longitudinal data because they do not make any assumptions about the distribution of the dependent and independent variables or the homogeneity of the underlying population. They can also analyze cases with partial information. In this article, we use the Household, Income, and Labour Dynamics in Australia (HILDA) survey to illustrate the benefits of machine learning. Using a machine learning algorithm, we analyze the relationship between job-related variables and neuroticism across 13 years of the HILDA survey. We suggest that the results produced by machine learning can be used to generate generalizable rules from the data to augment our theoretical understanding of the domain. With a technical guide, this article offers critical information and best-practice recommendations that can assist social science researchers in conducting machine learning analysis with longitudinal data.