Probing numerical concepts in financial text with BERT models

Abstract

Numbers are notoriously an essential component of financial texts, and their correct understanding is key to an automatic system for efficiently extracting and processing information. In our paper, we analyze the embeddings of different BERT-based models by testing them on supervised and unsupervised probing tasks for financial numeral understanding and value ordering. Our results show that LMs with different types of training have complementary strengths, thus suggesting that their embeddings should be combined for more stable performances across tasks and categories.