Identifying novel customer needs from user-generated content for product development using pre-trained language model

The identification of novel customer needs is crucial for companies to create new products and seise business opportunities in a constantly evolving technological and social landscape. However, traditional methods for identifying emerging needs are costly, time-consuming, and labour-intensive, often resulting in delays in bringing products to market. In recent years, user-generated content (UGC), such as online product reviews, has emerged as a promising alternative source for uncovering novel customer needs. In this paper, we propose a novel approach to identifying customer needs by treating this as a text classification task. Specifically, we leverage the power of the pre-trained language model BERT to analyze and extract insights from UGC, particularly online product reviews. To address the challenge of class imbalance in the data, we developed a regularized dual BERT structure that achieves state-of-the-art performance. Our experiments demonstrate the effectiveness of this structure, showing that it is robust even when dealing with reviews of varying lengths. By using this proposed methodology, companies can quickly and efficiently automate the process of identifying novel customer needs, requiring fewer expert resources. Copyright © 2025 Informa UK Limited, trading as Taylor & Francis Group.