Investment Decision-Making with AI: Exploring limitations of LLMs in analyzing Earnings Transcripts

Abstract

In this applied research report, we investigate the viability of using frontier Large Language Models (LLMs) for the purpose of automating financial sentiment extraction from earnings transcripts of publicly traded companies, a common business analytics workflow used by investment banks such as Citibank in their client advisory service.

We first introduce new innovative LLM-backed financial advisory services that can potentially be adopted by investment banks such as Citibank to offer to their clients and contrast such services with the current such services.

We then outline a methodology and research plan to analyze the accuracy, limitations, and risks of using LLMs in such services. In particular, the plan consists of compiling an annotated dataset from publicly available transcripts, implementing several NLP and LLM pipelines and outlining several research questions and evaluation frameworks. Our methodology is designed to ensure that the entire pipeline, from data collection to model evaluation, is reproducible and can be replicated using the outlined procedures and tools.

After implementing and running our LLM-backed models and analysing the results, this report identifies several limitations of large language models in extracting insights from financial transcripts. Among other findings, we show that LLMs tend to exhibit an optimism bias, demonstrate variable accuracy depending on the type of company analysed, and that model choice significantly impacts performance. We present several recommendations to address these limitations.

In addition, we outline key business risks, privacy concerns, and ethical considerations associated with using LLM-backed sentiment classification services in investment banking contexts.

While our results suggest that fully automated LLM-based sentiment analysis is not currently viable, we propose a more refined hybrid human–AI approach that balances risks and mitigates some of the limitations identified. To support this, we also implement a dashboard that not only assists in evaluating our models but also serves as a prototype tool for the proposed hybrid workflow.