Part V: Natural Language Processing for factor investing¶

Chapter 20: Classical NLP for factor investing¶

120.1 Introduction¶

The preceding chapters of this book have explored a wide array of machine learning tools for factor investing, from penalized regressions (Chapter 5) and tree-based models (Chapter 6) to neural networks (Chapter 7), reinforcement learning (Chapter 16), and genetic algorithms (Chapter 19). All of these methods operate on the same type of input: structured, numerical data organized into a matrix of firm characteristics. The book’s standard dataset of 1,207 US stocks with 93 features (see Chapter 1) epitomizes this paradigm. Yet a vast reservoir of economically relevant information is encoded not in numbers but in words.

Corporate filings, earnings call transcripts, analyst reports, news articles, central bank communications, and social-media posts contain forward-looking statements, managerial tone, risk disclosures, and sentiment cues that are difficult (and often impossible) to capture through numerical characteristics alone. The field of natural language processing (NLP) provides the tools to convert this unstructured text into quantitative signals suitable for factor models and portfolio construction.

The relevance of text for asset pricing is well established empirically. Earlier foundations were laid by Antweiler and Frank (2004), who showed that the volume and disagreement of Internet stock message board posts predict next-day volatility and trading activity, and by Das and Chen (2007), who classified Yahoo! Finance messages about 24 technology stocks and found that aggregate sentiment tracks intraday returns. Building on this foundation, Tetlock (2007) showed that the fraction of negative words in a Wall Street Journal column predicted next-day Dow Jones returns. Loughran and McDonald (2011) demonstrated that generic sentiment dictionaries misclassify financial language and proposed a domain-specific lexicon that has since become the standard.

This chapter focuses on the classical NLP toolkit that has driven text-based factor research for two decades, and that remains the right starting point for any production pipeline. We introduce text as a data class and survey the six source families that recur throughout Part V (Section 20.2), define the preprocessing and bag-of-words representations that every downstream method consumes (Section 20.3), develop the dictionary-based sentiment scoring that opened the field empirically (Section 20.4), introduce topic models for thematic factor discovery (Section 20.5), present named-entity recognition and event extraction for event-driven strategies (Section 20.6), and close with the two factor families that the classical toolkit has produced in mature, peer-reviewed form: disclosure-based factors (Section 20.7) and news and media-flow factors (Section 20.8). The remainder of the NLP toolkit (embeddings, transformers, large language models, and an end-to-end case study) is deferred to the following chapters.

NLP for equity markets is now a vast and rapidly growing literature: corpus surveys identify on the order of 1,000 or more papers since the late 2000s, with roughly 76% appearing after 2021 alone (Lis, 2024). Two structural axes organize the field. The first is a method curve that runs from rule-based lexicons through classical machine learning and dense embeddings to BERT-style encoders and, finally, generative LLMs; Mishev et al. (2020) provide the canonical benchmark spanning this entire arc, testing approximately 30 sentiment models on financial text and finding that transformers dominate accuracy while sacrificing transparency. The second is a source curve: early studies focused on news articles and message boards because they are short and syntactically simple, while longer and more specialized documents such as earnings-call transcripts and 10-K filings only became tractable once transformer-scale encoders arrived around 2019-2020. Two caveats are in order. First, the book’s standard mlfi_us_data panel is numerical only: it does not contain textual data, in keeping with the design of the dataset described in Chapter 1. Shipping a raw text corpus alongside this book would be problematic on two counts. Legally, although individual corporate filings, earnings-call transcripts, and news articles are public, their bulk redistribution sits in a grey zone: publishers and transcript vendors retain copyright on the typeset record, and SEC filings are public but not licence-free when repackaged at scale. Practically, assembling a research-grade corpus of millions of filings, transcripts, and news items is an engineering project (storage, deduplication, OCR for older filings, ticker mapping, point-in-time alignment) that is well outside the scope of a methods textbook. To mitigate both concerns we ship a sentiment-derived companion dataset rather than the raw text itself: for the 437 largest US stocks in mlfi_us_data we processed the full available history of quarterly earnings-call transcripts and recorded the per-call sentiment scores produced by each of the model families introduced across Part V (classical dictionaries, contextual encoders, and an LLM-based scorer), split into prepared-remarks, Q&A-management, and Q&A-analyst buckets. Distributing scores rather than text sidesteps the licensing question and shrinks the artifact from terabytes to a single panel CSV. The empirical tables of Chapter 21 and the case study of Chapter 24 read from this companion panel directly; the methodology subsections of the present chapter use much shorter illustrative samples (a single Coca-Cola Q1 2026 transcript excerpt and a handful of stylised sentences) when the pedagogical point is the technique itself rather than its cross-sectional performance. Second, NLP is a vast and fast-moving field. We do not attempt encyclopedic coverage; instead, we focus on the methods most directly relevant to equity factor investing and refer the interested reader to Jurafsky and Martin (2024) for a comprehensive treatment of the broader NLP landscape, and to the finance-specific survey of Xing, Cambria, and Welsch (2018), which catalogues the text sources (news, filings, message boards, social media), modeling families (lexicon, statistical, deep learning), and applications (return and volatility forecasting, event studies) that underpin the literature Part V draws on.

A note on notation in this chapter. Several equations in this chapter use letters that double as both NLP and asset-pricing indices. We highlight the local redefinitions to avoid confusion. In Equation (20.7), $n$ indexes word positions within a document (not assets) and $N_d$ is the document length. In Equation (20.8), $t$ indexes corpus tokens and $T$ the corpus length (not calendar time). The symbol $\alpha$ appears in two distinct roles: a Dirichlet concentration vector in Equation (20.7) and a portfolio intercept in Equation (20.13); context disambiguates in each case. We retain these letter choices because they are standard in the NLP literature, but we flag them here so that the cross-reference to Chapter 1’s notation is unambiguous.

220.2 Text as a data class¶

Text differs from the structured numerical features of Chapters 2 through 19 in three ways that drive every design choice that follows. It is high-dimensional: a typical 10-K contains 50,000 to 150,000 tokens (the atomic word or word-piece units a text is broken into, defined precisely in Section 20.3.1) drawn from a vocabulary of tens of thousands of distinct types, so any representation must compress aggressively before reaching a tractable feature space. It is semantic: identical numbers (“revenue grew 5%”) carry very different signals depending on the surrounding context (guidance versus historical, prepared remarks versus Q&A, peer comparison versus stand-alone), so encodings that ignore surrounding context (the classical bag-of-words representation defined in Section 20.3.2, which simply counts how often each word appears) throw away information that the contextual encoders of Chapter 22 (transformer models whose representation of each word depends on the entire sentence) preserve. And it is irregularly timed: filings appear on quarterly schedules with embargoed release windows, earnings calls are clustered in two-week reporting waves, news arrives continuously with intra-day spikes, and social-media posts arrive in bursts driven by events; this temporal heterogeneity dictates how text-derived features should be aligned to the monthly factor frequency of the book’s dataset.

These three properties motivate a data-class view of text. Rather than treating “text” as a single source, we partition the universe of financial text into six classes that differ along the dimensions that matter for factor investing: producer (firm, analyst, journalist, regulator, retail investor), publication frequency, regulatory status, signal-to-noise ratio, coverage breadth, and typical document length. Each class supports a distinct set of metrics and a distinct set of factor strategies, and each has accumulated its own peer-reviewed empirical literature.

FIGURE 20.1 organizes the six text-data classes along three axes (data class, document or data type, metrics or items examples) following the standard practitioner taxonomy used in industry research. We now describe each row in turn, summarizing what the class is, who produces it, what use cases it enables, and which published evidence anchors it in the empirical literature.

2.120.2.1 Corporate filings (10-K, 10-Q, 8-K, Forms 3/4/5, 13-F)¶

Filings submitted to the U.S. Securities and Exchange Commission are the most regulated text class in the financial system. The annual 10-K and quarterly 10-Q reports contain audited financial statements wrapped in extensive textual discussion, including the Management’s Discussion and Analysis (MD&A), risk factors, and forward-looking statements. The 8-K reports material events as they occur; Forms 3, 4, and 5 disclose insider transactions; and 13-F filings reveal quarterly institutional positions above the reporting threshold. These documents are long (typically 100 to 300 pages for a 10-K), uniformly formatted, infrequent (quarterly to annual), and legally vetted, which makes them high-signal but slow-moving. The classical evidence on filings runs through Loughran and McDonald (2011), whose finance-specific sentiment dictionary corrects the systematic misclassification of generic lexicons on legal and accounting prose; Li (2008), who showed that 10-K readability (Fog index) predicts earnings persistence; Cohen, Malloy, and Nguyen (2020), who documented the Lazy Prices anomaly whereby year-over-year textual changes in 10-Ks generate a long-short return of roughly 23.5% per annum; Bushee, Gow, and Taylor (2018), who decompose linguistic complexity into obfuscation and information components; Hoberg and Phillips (2016), who build text-based industry classifications from 10-K product descriptions; Campbell, Chen, Dhaliwal, Lu, and Steele (2014), who quantify the information content of mandatory risk factor disclosures; and Lopez-Lira (2023), who extracts risk factors directly from 10-K text and prices them in the cross-section.

2.220.2.2 Listed-company communications (annual reports, IR websites, conference presentations)¶

Beyond mandatory SEC filings, listed companies maintain a parallel stream of voluntary communications: glossy annual reports, investor-relations website content, sustainability reports, capital-markets-day decks, and conference presentations to sell-side analysts. These materials are less constrained by accounting standards but governed by Regulation Fair Disclosure, which prohibits selective release of material information. They are useful for extracting forward-looking commentary that is downplayed or absent in SEC filings: total addressable market estimates, net sales by geography and product segment, segment-level growth narratives, capex guidance, and product-pipeline disclosures. The empirical evidence here is thinner than for SEC filings but growing: Sautner, van Lent, Vilkov, and Zhang (2023) construct firm-level climate change exposure measures by combining earnings calls with IR communications, and Webersinke, Kraus, Bingler, and Leippold (2022) pretrain ClimateBERT on a large corpus of corporate sustainability reports and use it to score voluntary climate disclosures across firms.

2.320.2.3 Earnings call transcripts¶

Quarterly earnings conference calls are perhaps the richest single-source text class for equity factor investing. Each call has two parts with very different information content: prepared remarks (10 to 30 minutes), in which management reads a scripted account of the quarter, and the Q&A session (typically 30 to 60 minutes), in which sell-side analysts question executives in real time. Transcripts are publicly available with a delay measured in hours and cover essentially every US-listed company that holds a quarterly earnings call. Larcker and Zakolyukina (2012) identified linguistic markers of deceptive language by executives who subsequently restated earnings; Hassan, Hollander, van Lent, and Tahoun (2019) extracted firm-level political risk scores from transcripts that load on lobbying activity and predict the cross-section of returns; Sautner et al. (2023) measured climate exposure at the firm level; Kim, Muhn, and Nikolaev (2024) used a generative LLM to distil firm-level risk assessments from earnings-call transcripts that explain return volatility and firms’ risk-mitigating actions better than dictionary-based measures; Hafez, Matas, Gomez, and Kangrga (2021) showed that document sentiment, event sentiment, and disclosure transparency are three largely orthogonal signals extractable from the same transcript; Zhao (2018) demonstrated that the Q&A portion of the call carries substantially more cross-sectional alpha than the prepared remarks; and Yang, Ng, Smyth, and Dong (2020) introduced HTML, the hierarchical transformer architecture that has since become standard for long-form transcript modeling.

2.420.2.4 Public news and newswires¶

News is the most-studied text class in the literature and the dominant data source in every method era surveyed by Mishev et al. (2020). It is heterogeneous in publisher (Wall Street Journal, Reuters, Dow Jones, Bloomberg, regional newspapers, financial blogs), frequency (intra-day to daily), and coverage (broad market commentary versus firm-specific news). The canonical empirical results start with Tetlock (2007), who showed that the negative-word share in a single daily WSJ column predicts next-day Dow returns; Tetlock, Saar-Tsechansky, and Macskassy (2008), who generalized the finding to firm-specific news; Antweiler and Frank (2004) on Internet message-board volumes; Engelberg and Parsons (2011), who isolated the causal effect of local newspaper coverage on local trading via weather-driven distribution disruptions; Hillert, Jacobs, and Mueller (2014), whose interaction of past returns with press coverage drives the momentum factor; Heston and Sinha (2017), who built a news-tone factor not subsumed by price momentum; Garcia (2013) on the recession-dependence of news sentiment effects; Calomiris and Mamaysky (2019) on novelty-weighted sentiment in a 51-country sample; Manela and Moreira (2017), who construct the NVIX disaster-concerns index from front-page articles since 1890; and Ke, Kelly, and Xiu (2020), who push the out-of-sample (OOS) predictive frontier with their SESTM (Sentiment Extraction via Screening and Topic Modeling) supervised text learner over Dow Jones news. Two early methodological studies anchor the historical foundation of the literature for this data class: Devitt and Ahmad (2007), working with cohesion-based analysis of financial news, were among the first to demonstrate rigorously that textual tone carries price-relevant information beyond prices themselves, and Schumaker et al. (2012), in one of the canonical early studies in Decision Support Systems, showed that lexicon-based sentiment over news articles produces reliable predictive signals for stock returns.

2.520.2.5 Social media (Twitter/X, StockTwits, Reddit, message boards)¶

Social media is the youngest and noisiest text class in the taxonomy, characterised by very short documents (Twitter’s 280-character limit), heavy slang and emoji usage, severe class imbalance (most posts are neutral chatter), and intermittent retail-sentiment regimes that can shift abruptly during specific events (GameStop 2021, AMC 2021, COVID 2020). Despite these challenges, peer-reviewed evidence on its information content is robust: Bollen, Mao, and Zeng (2011) reported aggregate Twitter mood as a leading indicator of DJIA moves; Chen, De, Hu, and Hwang (2014) used Seeking Alpha posts to predict next-quarter earnings; Bartov, Faurel, and Mohanram (2018) showed that pre-announcement Twitter sentiment predicts both earnings surprises and announcement-day returns, particularly for low-coverage names; Xu and Cohen (2018) released the StockNet dataset (88 stocks, two years of tweets and prices) that has become a standard NLP benchmark; Zheludev, Smith, and Aste (2014) provide a negative-control finding that the lead-lag effect of Twitter on the S&P 500 is detectable for high-volume cashtags but absent at the aggregate index level; Sawhney, Agarwal, Wadhwa, and Shah (2020) introduced graph-attention models that combine tweet content with inter-company correlation graphs; and Albladi, Islam, and Seals (2025) survey the post-transformer state of the art across the broader Twitter sentiment literature.

2.620.2.6 Patents and intellectual property filings¶

The final class in our taxonomy, patents, is the least developed in the equity factor investing literature but is included here because it sits at the intersection of two robust signals: corporate R&D effort (a quality factor proxy) and innovation novelty (an asset-pricing factor in its own right). Patent abstracts describe what the invention does, patent claims define what is legally protected, and the patent citation graph captures intellectual debt and influence. Useful firm-level metrics derived from patent text include the patent-family size (how broadly an invention is protected internationally), expiration timelines (which affect future cash flows), novelty keywords (which proxy for breakthrough innovation), the count of citing patents (an empirical proxy for technology-moat strength), and concept exposure to specific technology themes. The flagship paper in this area, Kogan, Papanikolaou, Seru, and Stoffman (2017), measures patent value from text and shows that text-derived novelty predicts future cash flows and stock returns; although outside the dominant news and transcript literature of this chapter, the methodology is a natural extension of the techniques developed in Sections 20.3 through 20.7 and an open avenue for factor researchers working at the intersection of IP economics and asset pricing.

320.3 Text as data: preprocessing and representation¶

3.120.3.1 From raw text to clean tokens¶

Before any quantitative analysis can begin, raw text must be transformed into a structured representation. This preprocessing pipeline is the foundation of all NLP methods, classical and modern alike, and its quality directly affects downstream performance. We outline the standard steps below.

Tokenization is the process of splitting a string of text into individual units called tokens. In the simplest case, tokens are words separated by whitespace, but financial text often requires more nuanced treatment. Consider the string "Q3 2023 EPS beat by $0.12": a naive whitespace tokenizer would treat "$0.12" as a single token, while a more sophisticated tokenizer might separate the currency symbol from the number. The nltk library provides several tokenizers suited to different use cases.

The tokenizer isolates the dollar sign and the percent sign as their own tokens, which is useful for a downstream rule that wants to identify currency or magnitude expressions, but it also splits "18" from "%", so any pipeline that wants the original "18%" magnitude back must reassemble adjacent tokens (or use a custom regular-expression tokenizer).

Stop-word removal eliminates high-frequency words that carry little semantic content: “the,” “is,” “and,” “of.” Removing these words reduces the dimensionality of the representation and focuses subsequent analysis on content-bearing terms. However, in financial text, some seemingly generic words can carry meaning (e.g., “will” in forward-looking statements), so the choice of stop-word list requires domain judgment.

Eight content-bearing tokens survive out of seventeen original tokens. The pronouns (“we,” “our”), auxiliaries (“are,” “to,” “as”), and the article (“a”) are dropped; “balance sheet” and “court decision” survive as the substantive concepts.

Stemming and lemmatization reduce words to their root forms. Stemming applies crude suffix-stripping rules (“running” becomes “run,” “consolidation” becomes “consolid”), while lemmatization uses morphological analysis to produce valid dictionary forms (“better” becomes “good,” “are” becomes “be”). For financial applications, lemmatization is generally preferred because it preserves interpretability: “diversified” and “diversification” both reduce to “diversify” rather than the truncated stem “diversif.”

Lemmatization keeps real dictionary words but only normalises plurals (consumers → consumer) unless we pass the right part-of-speech tag for verbs; the Porter stemmer is more aggressive and collapses inflections but produces non-words such as consum and even. Neither method handles better → good without a richer morphological resource (NLTK’s WordNet lemmatizer with pos='a' would).

Handling financial jargon. Domain-specific vocabulary poses particular challenges. Ticker symbols (AAPL, MSFT), financial abbreviations (EPS, EBITDA, P/E), and compound terms (free-cash-flow, mark-to-market) require custom tokenization rules. Negation handling is especially important: “not profitable” should not be treated as two independent tokens in a bag-of-words model. We discuss this further in Section 20.4.3.

The substitution pass collapses multi-word financial terms into single tokens before the generic tokenizer runs, preserving them as atomic units downstream. The negation cue is rewritten with a NEG_ prefix on the modified word, so that a bag-of-words sentiment scorer that does not understand syntax still sees NEG_slowing as a distinct feature rather than positive slowing plus a discarded no.

We illustrate the preprocessing pipeline on six real excerpts taken from the Coca-Cola Q1 2026 earnings call transcript (Coca-Cola, henceforth referenced by its NYSE ticker KO). In practice, the corpus would consist of thousands of 10-K filing excerpts, earnings call paragraphs, or news articles; here a single call already shows the variety of language a sentiment pipeline must cope with.

The preprocessing reduces each sentence to its content-bearing lemmas. Notice that “consumers” becomes “consumer,” “results” becomes “result,” and common words like “the,” “a,” “by” are removed. The plural “segments” is normalised to “segment,” and the proper nouns “Coca,” “Cola,” “Cherry,” and “Coke” survive as lowercase tokens that a downstream entity-recognition pass (Section 20.6) would tag back as brand names. The resulting token lists are the raw material for the bag-of-words representation introduced in the next subsection.

3.220.3.2 Bag-of-words and term frequency¶

The simplest numerical representation of a text document is the bag-of-words (BoW) model: a vector whose length equals the vocabulary size $V$ and whose $k$-th entry counts how many times word $k$ appears in the document. This representation discards word order entirely, treating “the dog bit the man” and “the man bit the dog” identically.

Let $d$ denote a document and $\mathcal{V} = \{w_1, \ldots, w_V\}$ the vocabulary. The raw term-frequency vector is:

where $\text{count}(w_k, d)$ is the number of occurrences of word $w_k$ in document $d$.

Given a corpus of $D$ documents, the document-term matrix (DTM) $\mathbf{X} \in \mathbb{R}^{D \times V}$ stores the term frequencies (or raw counts) for all documents. Each row is a document, each column is a word. This matrix is typically very sparse, because any single document uses only a small fraction of the total vocabulary.

Raw term frequency overweights common words. The term frequency-inverse document frequency (TF-IDF) weighting addresses this by down-weighting terms that appear in many documents:

where $\text{df}(w_k)$ is the number of documents containing word $w_k$. The logarithmic term penalizes words that appear across most documents (e.g., “company,” “quarter”) while boosting words that are distinctive to a few documents (e.g., “goodwill impairment,” “FDA approval”). TF-IDF is arguably the most widely used text representation in applied finance, owing to its simplicity and effectiveness.

We now construct the TF-IDF matrix from the same six KO Q1 2026 excerpts preprocessed above.

The full 6 by 30 TF-IDF matrix is extremely sparse, since each document uses only a small fraction of the vocabulary; the per-document top-4 view above is the empirically useful summary. The highest-weighted terms are those that are both frequent within a document and rare across the corpus. Doc 4 illustrates the construction cleanly: “mix” appears twice in that single document and nowhere else in the corpus, so it dominates the row with a weight of 0.76. Doc 5 is similar for “cherry”, which appears twice in Doc 5 alone. The macroeconomic vocabulary of Doc 1 (“consumer”, “inflation”, “greater”, “due”) and the forward-looking vocabulary of Doc 3 (“expects”, “full”, “guidance”) split cleanly along the prepared-remarks-versus-outlook axis that the chapter exploits throughout.

This representation, while simple, already captures meaningful structure. Documents about cash flows (Doc 2, Doc 4) share the terms “cash” and “flow,” while documents about losses (Doc 1) and earnings beats (Doc 5) are clearly separated. The challenge, of course, is that bag-of-words ignores word order and context entirely, a limitation we address in the sections that follow.

420.4 Sentiment analysis with dictionaries¶

4.120.4.1 The lexicon approach¶

The most direct way to extract economic meaning from text is sentiment analysis: classifying the tone of a document as positive, negative, or neutral. The simplest and most widely used approach in finance relies on pre-compiled sentiment dictionaries (also called lexicons): curated lists of words labeled with their emotional or evaluative polarity.

Early work in computational linguistics used general-purpose lexicons such as the Harvard General Inquirer (GI), which classifies English words into psychological categories including positive and negative tone. However, Loughran and McDonald (2011) showed that general-purpose lexicons perform poorly on financial text. Almost three-quarters of the words classified as “negative” by the Harvard GI are not negative in a financial context. The word “tax,” for instance, is labeled negative by the GI, yet it carries no evaluative tone in an earnings report. Similarly, “liability,” “cost,” and “capital” are flagged as negative despite being neutral financial terminology.

The Loughran-McDonald (LM) dictionary was specifically constructed for financial documents. The authors analyzed 10-K filings from 1994 to 2008 and identified six sentiment categories: negative, positive, uncertainty, litigious, strong modal, and weak modal. The negative word list (containing terms such as “loss,” “impairment,” “decline,” “adverse,” “default”) has become the de facto standard for financial sentiment analysis. Subsequent studies have largely confirmed that domain-specific dictionaries outperform generic ones for financial text; the survey by Kearney and Liu (2014) provides a thorough review of these findings.

Given a dictionary with positive word set $\mathcal{P}$ and negative word set $\mathcal{N}$, the simplest sentiment score for a document $d$ is the net tone:

where $\#(d \cap \mathcal{P})$ counts the number of words in $d$ that belong to the positive list. This score ranges from -1 (entirely negative) to +1 (entirely positive). Alternative normalizations divide by total word count rather than the sum of sentiment words, but Equation (20.3) is the most common in the literature. Jegadeesh and Wu (2013) showed that giving every LM-listed word equal weight in this formula is suboptimal, because some terms carry much stronger return-predictive content than others. They proposed a market-based term-weighting scheme that estimates each word’s informativeness from its association with stock returns around 10-K filing dates, and reported that this data-driven weighting materially improves the out-of-sample predictive power of the tone score relative to the uniform weighting implicit in Equation (20.3).

We demonstrate the Loughran-McDonald approach using six sentences in the style of (and partly drawn from) the KO Q1 2026 earnings call already introduced in Section 20.3.1. In practice, researchers download the LM dictionary from the authors’ website and apply it to large corpora of filings or news articles.

The dictionary correctly identifies the dominant tone in most cases. Doc 0, 3, and 5 are scored as strongly positive, while Doc 1, 4 are strongly negative. However, Doc 2 illustrates the negation problem: “operating losses did not materialize, and margins held firm” is a positive statement, yet the dictionary scores it as -1.00 because “losses” is counted as a negative word while the negation “did not materialize” is ignored entirely. The dictionary cannot read the difference between “losses occurred” and “losses did not materialize” because it sees only a bag of words.

FIGURE 20.2 (panel a): dictionary-based sentiment scores for six KO-style sentences. Positive tone is shown in green, negative in red. Doc 2 illustrates the negation problem: a positive statement (“losses did not materialize, margins held firm”) receives a strongly negative score because the dictionary sees “losses” but cannot read “did not materialize.”

The visualization in FIGURE 20.2 (panel a) makes the limitations of the dictionary approach immediately apparent. While the method works well for straightforward sentences, it systematically misclassifies negated statements. In Chapter 22, we will see that transformer-based models handle negation naturally through contextual representations.

4.220.4.2 From sentiment scores to factor signals¶

The seminal application of textual sentiment to asset pricing is Tetlock (2007), who studied the “Abreast of the Market” column in the Wall Street Journal. Using the Harvard GI to measure the fraction of negative words, Tetlock found that high media pessimism predicted downward pressure on market prices, followed by a reversion to fundamentals. Crucially, the effect was concentrated in small stocks with high individual investor ownership, consistent with a behavioral explanation in which noise traders overreact to negative tone.

This finding opened a productive line of research. Tetlock et al. (2008) extended the analysis to firm-level news and showed that the fraction of negative words in stories about individual firms predicted lower subsequent earnings, suggesting that journalists incorporate fundamentally relevant information. Loughran and McDonald (2011) refined the measurement using their domain-specific dictionary and confirmed that negative tone in 10-K filings predicts higher post-filing return volatility, lower earnings, and wider bid-ask spreads. Engelberg and Parsons (2011) pushed the literature a step further on causal identification: they used weather-driven disruptions to local newspaper delivery as an instrument and showed that local press coverage of S&P 500 earnings announcements roughly doubles local trading activity, providing direct evidence that the media is not merely reporting prices but shaping them. Hillert, Jacobs, and Mueller (2014) then connected sentiment to a classical anomaly by showing that firms with high media coverage exhibit substantially stronger momentum returns, with the effect largest for glamour stocks followed by many journalists, suggesting that press attention amplifies the behavioural frictions that sustain cross-sectional momentum.

To construct a news-tone factor for the cross-section, one typically proceeds as follows:

1. For each firm $n$ at time $t$, collect all relevant text documents (news articles, filings, transcripts) published during the period $(t-1, t]$.

2. Compute a sentiment score for each document using Equation (20.3) or a variant.

3. Aggregate across documents, possibly weighting by recency or source prominence, to obtain a firm-level sentiment signal $s_{t,n}$.

4. Use $s_{t,n}$ as a cross-sectional factor alongside the traditional characteristics from Chapter 1.

To make the factor scale-invariant across periods we standardize the score cross-sectionally each rebalancing date,

where the cross-section at date $t$ has $N_t$ firms. The portfolio implementation that we use throughout this chapter is the equal-weighted long-short decile spread,

with $\mathcal{D}_t^{10}$ and $\mathcal{D}_t^{1}$ the top and bottom deciles of $z_{t,n}$, and $r_{t \to t+h, n}$ the $h$-period total return on firm $n$. The cross-sectional information coefficient at horizon $h$, which we report extensively in the case study of Chapter 24, is the Spearman rank correlation between the standardized signal and the forward return computed within each cross-section and then averaged across dates,

Equations (20.4) through (20.6) supply the language we need to compare sentiment methods quantitatively: an information coefficient and a decile-spread Sharpe are the two summary statistics that the empirical tables of Chapter 21 and the case study of Chapter 24 are organized around.

The aggregation step is important and non-trivial. Calomiris and Mamaysky (2019) showed that novelty-weighted sentiment, which down-weights language that merely repeats prior disclosures, is more informative for predicting returns than raw sentiment. The intuition is simple: a 10-K filing that repeats last year’s risk factors verbatim contains less new information than one with substantially revised language.

The choice of aggregation window is equally consequential. Hafez and Xie (2012) constructed several variants of a firm-level sentiment index from a commercial firm-level news sentiment dataset on a universe of 4,144 US stocks over 2000 to 2011, comparing absolute (net positive minus negative news count) and relative (positive share of total) measures across 1-day, 7-day, and 90-day rolling windows. They found that the 90-day window dominates: it smooths transient noise and captures the slower-moving component of sentiment that is most strongly associated with future returns. Using their preferred event-level score with a 90-day window, the average 3-day return spread between firms with positive and negative sentiment changes reaches 1.39%, with most of the divergence occurring on the day the news is released. A long-short strategy built on this signal generates information ratios of 2 to 3 and a daily hit ratio close to 60% before transaction costs. These figures should be read with the usual caveats (in-sample tuning, look-ahead in the universe construction, and gross-of-cost performance), but they illustrate a robust pattern: properly aggregated firm-level sentiment carries information beyond the contemporaneous return. An early explicit demonstration of this came from Zhang and Skiena (2010), who combined blog and news sentiment from a broad online corpus and showed that the resulting aggregate signals could be converted into profitable daily trading strategies across a large equity universe; their results were noisy at the individual-stock level but robust in aggregate, reinforcing the case for the cross-sectional sorting approach described above.

4.320.4.3 Limitations of dictionary methods¶

Dictionary-based sentiment analysis is transparent, fast, and reproducible. It is also fundamentally limited by what we might call context blindness. A dictionary treats each word independently, ignoring the syntactic and semantic structure of the sentence.

The most frequently cited limitation is negation. The sentence “The company did not experience any significant losses” is objectively positive, yet a dictionary approach that counts “losses” as negative and ignores “not” will assign a negative score. Various heuristic fixes exist (e.g., flipping the polarity of sentiment words that follow a negation cue within a fixed window), but they are fragile and incomplete.

Intensifiers and diminishers pose a related problem. “Slightly disappointing” and “extremely disappointing” carry very different magnitudes, but a dictionary assigns both the same polarity. Similarly, conditional and hypothetical language is common in financial text: “If revenue were to decline, the company might need to restructure” expresses a possibility, not a fact, yet a dictionary would count “decline” and “restructure” as negative.

Sarcasm and implicit sentiment are rare in formal financial documents but appear in social media and analyst commentary. A tweet reading “Great job losing another billion dollars” is clearly negative, but a dictionary would score “great” as positive.

Finally, domain drift is a practical concern. Language evolves, and the connotations of words change over time. A dictionary compiled from 1990s filings may not accurately capture sentiment in 2020s filings that discuss “pandemic,” “supply chain disruptions,” or “generative AI.” This static nature is a fundamental limitation that motivates the data-driven approaches of Chapters 21 through 23. Lis (2024) provides a useful perspective on how much of this gap has actually been closed: reviewing 71 empirical studies of investor sentiment in asset pricing models published between 2000 and 2021, the author documents that more complex sentiment measures do not reliably translate into better return prediction, and that survey-based, lexicon-based, and transformer-based signals often sit within overlapping error bars once implementation frictions are accounted for. The implication is not that modern methods are useless, but that the marginal gain per unit of modelling effort is smaller than the headlines of individual papers suggest.

A concrete out-of-sample replication of the central Loughran and McDonald (2011) claim is available on a single large-cap name from the case-study corpus, KO. We score the prepared-remarks bucket of 68 KO earnings calls covering October 2006 through February 2026 with four sentiment methods (Harvard General Inquirer, the pre-LM baseline used by Tetlock (2007); the LM dictionary; VADER, the rule-based lexicon of Hutto and Gilbert (2014); and FinBERT, included here as a contextual-encoder benchmark that anticipates Chapter 22), and regress the resulting per-call score on the five-day post-call cumulative abnormal return (CAR) computed against the S&P 500 ETF (SPY).

Panel (b) of FIGURE 20.2 (KO single-name replication): the left sub-panel shows the scatter of KO five-day post-call CAR against the prepared-remarks LM tone, with the OLS regression line and its t-statistic. The right panel reports the slope t-statistics of the same regression for all four methods. On this single-name sample (n=68), LM achieves a univariate t-statistic of +1.04 with R²=0.016, ahead of GI at -0.43 and VADER at +0.31. FinBERT, the only contextual-encoder method in the comparison, leads at t=+1.52 with R²=0.034. None of these four single-name slopes is statistically significant at the 5% level (|t|<1.96), which is exactly the point: at a single large-cap name there is simply too little variation across 68 calls for any sentiment method to clear conventional significance, regardless of how sophisticated the model is. What the figure does illustrate is the qualitative hierarchy that recurs at scale: LM outranks GI on the sign of the slope (consistent with the Loughran and McDonald (2011) thesis that finance-domain dictionaries beat generic ones), and FinBERT in turn outranks LM (consistent with the contextual-encoder lift documented in Chapter 22).

520.5 Topic modeling¶

5.120.5.1 Latent Dirichlet Allocation¶

While sentiment analysis reduces text to a single dimension (positive vs. negative), many financial applications require a richer representation that captures what the text is about, not just its tone. Topic models provide such a representation by discovering latent thematic structure in a corpus.

The most widely used topic model is Latent Dirichlet Allocation (LDA), introduced by Blei et al. (2003). The generative story behind LDA is elegant:

1. Each document is a mixture of $J$ latent topics, with mixing proportions $\boldsymbol{\theta}_d \sim \text{Dir}(\boldsymbol{\alpha})$.

2. Each topic $j$ is a distribution over the vocabulary, $\boldsymbol{\phi}_j \sim \text{Dir}(\boldsymbol{\beta})$.

3. For each word position in document $d$:

   • Draw a topic assignment $z \sim \text{Multinomial}(\boldsymbol{\theta}_d)$.

   • Draw a word $w \sim \text{Multinomial}(\boldsymbol{\phi}_z)$.

The Dirichlet priors $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ control the sparsity of the topic mixtures and word distributions, respectively. A small $\alpha$ encourages documents to concentrate on a few topics; a small $\beta$ encourages topics to concentrate on a few words.

Formally, the joint distribution of a corpus of $D$ documents, given hyperparameters $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$, factorizes as

where $N_d$ is the length of document $d$ and $w_{d,n}, z_{d,n}$ are the $n$-th word and its latent topic assignment. Posterior inference targets the marginal $p(\mathbf{z} \mid \mathbf{w}, \boldsymbol{\alpha}, \boldsymbol{\beta})$, which has no closed form; in practice we estimate it by collapsed Gibbs sampling or variational Bayes. The key output for financial applications is the document-topic matrix $\boldsymbol{\Theta} \in \mathbb{R}^{D \times J}$, where $\theta_{d,j}$ represents the proportion of document $d$ that belongs to topic $j$. These topic proportions can serve directly as features in a factor model.

5.220.5.2 Financial applications¶

Topic models have found several applications in equity investing. We highlight two prominent examples.

Risk theme discovery. Israelsen (2016) applied LDA to the risk-factor sections (Item 1A) of 10-K filings and discovered interpretable risk themes such as “litigation risk,” “regulatory risk,” “commodity price risk,” and “technology risk.” Firms with high exposure to specific risk topics exhibited different return patterns, suggesting that topic proportions capture economically meaningful variation in firm-level risk that is not fully captured by traditional factors.

Text-derived factors beyond Fama-French. Lopez-Lira (2023) used topic models on news articles to construct text-based factors and showed that these factors span a significant portion of the cross-sectional variation in returns. The key insight is that topics capture latent themes (e.g., “energy crisis,” “trade war,” “pandemic recovery”) that cut across traditional sector and style classifications.

The number of topics $J$ is a hyperparameter that must be chosen by the researcher. Common choices range from 10 to 50, depending on the granularity desired. Model selection criteria such as perplexity and topic coherence scores (Roder et al. (2015) survey the space of coherence measures) provide quantitative guidance, but human inspection of the top words per topic remains essential.

To make this concrete, we fit an LDA model to a small corpus of financial news headlines. In a production setting, the corpus would be much larger (thousands of articles per month), and the number of topics would be tuned using coherence metrics.

The topics are only partially interpretable, which is itself the lesson of running LDA on a corpus this small. Topic 2 is the cleanest: it groups the rate-and-macro headlines (the Fed hike, the GDP slowdown, housing starts, retail sales, and bond yields all load on it most heavily). Topic 1 collects energy and central-bank vocabulary (oil, crude, and the European Central Bank), and Topic 0 gathers an equities-and-safe-haven cluster (stocks, strong guidance, gold, and safe-haven demand). The separation is imperfect, with several headlines splitting their mass across topics, because fifteen short headlines give the model too little co-occurrence evidence to carve clean themes; the same procedure on tens of thousands of documents produces the sharp, stable topics that the technique is known for. We visualize the topic proportions for each headline in FIGURE 20.3.

LDA topic proportions for 15 financial headlines. Each row shows the estimated mixture of three topics for one headline. The three columns correspond to the equities-and-safe-haven, energy-and-central-bank, and rates-and-housing clusters identified above.

The heatmap in FIGURE 20.3 shows how each headline distributes its mass across the three topics. The rate-and-macro headlines load most heavily on Topic 2, as expected: “Federal Reserve raises interest rates”, “Housing starts decline as mortgage rates reach 20-year high”, and “Bond yields fall as markets price in rate cuts” all peak there. The energy headlines (“Oil prices surge”, “Crude oil futures fall”) concentrate on Topic 1, and the equities headlines (“Gold prices rise as investors seek safe haven assets”, “Semiconductor stocks jump”) on Topic 0, though several headlines spread their proportions fairly evenly, a direct visual signal of the weak topic separation that a fifteen-document corpus produces.

5.320.5.3 Extensions¶

Two extensions of standard LDA deserve mention. Dynamic topic models (Blei and Lafferty, 2006) allow topic distributions to evolve over time, which is natural for financial applications where the salience of themes (e.g., “pandemic,” “inflation,” “AI”) changes across market regimes. Structural topic models (STM) introduced by Roberts et al. (2014) incorporate document-level metadata (e.g., publication date, firm sector, filing type) as covariates that influence the topic proportions. This allows the researcher to ask questions such as “Do firms in the energy sector discuss regulatory risk more than firms in technology?” while controlling for other observable characteristics. A further extension is to couple topic assignments with sentiment scoring rather than treating the two as independent pipelines. Nguyen and Shirai (2015) showed, in the context of social-media message-board posts, that conditioning sentiment on the latent topic (their Joint Sentiment-Topic model, TSLDA) substantially improves next-day return-direction prediction relative to topic-agnostic sentiment; the intuition is that the word “risk” carries very different valence in a macro-rates discussion than in an earnings-surprise thread. This topic-conditioned sentiment approach anticipates the contextualised representations produced by word embeddings (§20.7) and, ultimately, by transformer models (§20.11), where meaning is determined jointly by the word and its full surrounding context.

620.6 Named entity recognition and event extraction¶

6.120.6.1 Principle¶

While sentiment and topics capture the tone and theme of text, many financial applications require identifying specific entities and events mentioned in documents. Named entity recognition (NER) is the task of locating and classifying named entities in text into predefined categories such as persons, organizations, locations, dates, and monetary values.

For financial text, relevant entity types include:

• Organizations: company names, ticker symbols, regulatory bodies.

• Persons: CEO names, board members, central bank officials.

• Monetary values: deal sizes, revenue figures, price targets.

• Dates: earnings dates, merger deadlines, policy meeting dates.

• Events: mergers and acquisitions, CEO departures, product launches, regulatory actions.

NER enables structured information extraction from unstructured text. For example, given the headline “Microsoft acquires Activision Blizzard for USD 68.7 billion,” an NER system identifies “Microsoft” and “Activision Blizzard” as organizations, “USD 68.7 billion” as a monetary value, and the entire sentence as describing an acquisition event. This structured output can then be linked to firm identifiers in a financial database.

The spaCy library ships pre-trained NER pipelines that resolve this example out of the box. The small English model (en_core_web_sm) is sufficient for short headlines and runs in milliseconds; the larger transformer-based pipeline (en_core_web_trf) is more accurate on long, jargon-heavy financial prose but slower.

Each detected span has a .label_ (entity type) and character offsets into the source text. Downstream code can filter for ORG spans to link to ticker symbols, for MONEY to extract deal sizes, and for DATE to align the event to a calendar bar. The output also illustrates a typical failure mode: the small model misclassifies Activision Blizzard as PERSON rather than ORG, because the proper noun pattern matches its prior on first-last-name pairs. The larger transformer pipeline (en_core_web_trf) usually fixes this. For finance-specific entity types not covered by the off-the-shelf models (e.g., CUSIP, ISIN, distinct CEO roles), or to harden the recall on company names like the one just misclassified, the spaCy EntityRuler lets one inject regex- or gazetteer-based rules ahead of the statistical NER component; the resulting hybrid pipeline keeps the recall of the pretrained model and adds the precision of hand-curated patterns for the long-tail entities that matter for equity research.

6.220.6.2 Applications in equity research¶

NER and event extraction support several equity research workflows:

Building event databases. By systematically applying NER to news feeds, researchers can construct databases of corporate events (M&A announcements, CEO changes, patent filings, lawsuits) at a scale and speed that would be impossible manually. These event databases enable event studies in the tradition of MacKinlay (1997) and event-driven trading strategies.

Entity-level sentiment. In a news article that mentions multiple companies, the overall sentiment of the article may differ from the sentiment associated with each individual firm. For example, “Apple’s gain is Samsung’s loss” is positive for Apple but negative for Samsung. Entity-level sentiment links the tone of specific sentences to the firms they mention.

Supply chain and network analysis. By extracting company mentions and their relationships from news and filings, researchers can map supply chain networks, competitive relationships, and industry linkages. Hoberg and Phillips (2016) used textual similarity of 10-K business descriptions as a measure of product market proximity, constructing “text-based network industries” that outperform SIC and NAICS codes for predicting cross-sectional returns.

Knowledge graphs as factor sources. A more ambitious use of entity extraction is to build a knowledge graph: a directed graph whose nodes are companies, people, products, events, and abstract concepts, and whose edges are co-mentions, supplier-customer relationships, partnerships, or thematic exposures extracted from news and filings at scale. Each security can then be assigned a set of concept exposure scores that measure how strongly it is connected (directly or indirectly) to a given concept, and aggregated to the cross-section those scores become a candidate factor. Knowledge-graph signals are fundamentally complementary to the bag-of-words and transformer-based features discussed in Sections 20.3 to 20.6: they operate on structured relationships extracted from text rather than on the text itself, and their alpha depends on the quality of entity disambiguation and edge construction rather than on sentiment lexicons or contextual embeddings. The peer-reviewed empirical literature on stand-alone KG-derived factors is comparatively thin and is dominated by work that feeds the graph into a downstream predictor rather than using concept exposures directly. A natural extension of this entity-extraction pipeline to patent filings, cross-referencing entity mentions with the USPTO patent database, would enable the text-based novelty factors of Kogan et al. (2017) to be constructed algorithmically at scale, though this application is outside the scope of the current chapter.

As a worked example of entity extraction, we demonstrate NER using the spaCy library, which provides pre-trained models for English text.

The NER model correctly identifies all key entities: the date of the announcement, the organizations involved (Microsoft, Activision Blizzard), the CEO’s name, the deal value, and the stock price change. In a production pipeline, these extracted entities would be linked to a master security database (mapping “Microsoft Corporation” and “Microsoft” to the same ticker MSFT) and stored in a structured event table for downstream analysis.

720.7 Disclosure-based factors¶

Corporate disclosures, 10-K and 10-Q filings, and earnings-call transcripts give rise to a self-contained family of equity factors built directly on the language of those documents. The signals reviewed below are not NLP-flavoured versions of value or quality factors; they are anomalies in their own right, identified by the accounting-finance literature and indexed by what the disclosure says, by how it is written, by what has changed in it relative to prior periods, and by what management chooses to discuss on the associated conference call. We organise the subsection around four mechanisms, readability and obfuscation, deceptive language, lazy-prices-style disclosure changes, and structured transcript analytics, and close with the quantity dimension of disclosure.

Readability and obfuscation. Li (2008) showed that annual report readability, measured by the Fog Index (a function of average sentence length and the proportion of complex words), predicts subsequent earnings. Less readable filings are associated with lower and more volatile future earnings, consistent with the hypothesis that managers strategically obscure bad news through complex language. The Fog Index has since been used as a quality signal in factor models: firms with unusually opaque filings may be concealing deteriorating fundamentals. Bushee, Gow, and Taylor (2018) refined the interpretation by decomposing linguistic complexity in firm disclosures into an obfuscation component (complexity that managers use to conceal unfavourable information) and an information component (complexity that reflects the genuine difficulty of the underlying economics). They show that only the obfuscation component is associated with lower trading volume and higher bid-ask spreads around 10-K filings, while the information component actually helps analysts incorporate complex facts into their forecasts. For factor investors, the practical lesson is that raw Fog Index scores can be noisy as stand-alone signals, and benefit from being residualised against proxies for business complexity before being used as a quality input.

Deceptive language. Larcker and Zakolyukina (2012) analyzed the linguistic content of CEO and CFO presentations during earnings conference calls and identified markers of deceptive language: executives who are subsequently found to have misstated earnings use fewer self-references (“I”), more general knowledge references (“as you know”), and more extreme positive emotion words. While these signals are too noisy for stand-alone trading strategies, they can augment traditional quality metrics (such as accruals and earnings persistence) to flag firms with elevated accounting risk.

A complementary out-of-sample illustration comes from Faccia, McDonald, and George (2024), who applied the off-the-shelf TextBlob polarity and subjectivity scorers to the annual reports of three well-known accounting failures, Wirecard, Tesco, and Under Armour, across a multi-year window straddling each fraud period, with each firm’s pre-fraud years used as its own baseline. The case-by-case trajectories are instructive precisely because they are heterogeneous. Tesco’s polarity rose from around 0.11 in the three years preceding the GBP 263 million profit overstatement to 0.148 in the 2014 fraud year, and its subjectivity rose alongside it (from roughly 0.37 in 2011-2012 to 0.60 in 2013 and 0.46 in 2014), an inflated-optimism pattern that is consistent with the deliberate-framing reading of Larcker and Zakolyukina (2012). Wirecard’s polarity stayed essentially flat around 0.07 across 2014-2018, with only a marginal dip in the 2018 fraud year, suggesting that long-running accounting manipulation can coexist with linguistically unremarkable annual reports. Under Armour’s polarity dipped in 2014 (a year before the 2015-2016 SEC investigation window) and then drifted back up across the fraud years, producing no monotonic relationship to the timing of the manipulation. The cross-section of three cases is much too small for inferential claims, but the pattern reinforces the Larcker and Zakolyukina (2012) caveat that polarity-based forensic signals are too noisy to operate standalone: directional inferences from a single firm’s tone trajectory point in opposite directions across firms. What the exercise does suggest is that forensic NLP and quality factors share a common substrate, and that a scalable version of the Faccia et al. study (thousands of restatements, automated extraction, and a within-firm pre-event baseline of the kind the paper constructs for each case) is an obvious complement to accruals-based accounting-quality scores.

Lazy prices and disclosure changes. Cohen, Malloy, and Nguyen (2020) documented one of the cleanest and most surprising text-based anomalies in the accounting-finance literature. They argued that investors pay insufficient attention to year-over-year changes in the language of 10-K and 10-Q filings, a behavioural phenomenon they call “lazy prices”. Using a simple cosine-similarity measure between successive filings from the same firm, they sorted stocks into quintiles of textual change and found that a long-short portfolio buying firms whose filings barely changed and selling firms whose filings changed the most earned roughly 188 basis points per month, or about 22% per year, net of standard factor exposures. The effect is concentrated in the negative tail: large text changes, which typically coincide with newly disclosed risks or litigation, are followed by persistent underperformance that takes several months to impound into prices. Cohen, Malloy, and Nguyen (2020) further isolate the mechanism. The predictability is concentrated in changes within the Risk Factors and Management Discussion and Analysis sections of the 10-K, and especially in changes that mention executive turnover (CEO or CFO transitions), litigation, or Loughran-McDonald negative-sentiment words; the 188 basis-point monthly alpha quoted above is in fact strongest when the sort is restricted to changes in the Risk Factors section alone (t-statistic of 2.76 over their 1995-2014 sample), and changes that explicitly reference litigation and lawsuits underperform the non-changers by roughly 71 basis points per month on their own. The asymmetry is striking: roughly 86 percent of substantive year-over-year changes carry a negative-sentiment loading, while the 14 percent that read positively predict significantly positive future returns, so the long-short result is driven by a specific tilt toward newly disclosed negative information rather than by aggregated text churn. The same construction predicts future earnings surprises, future news announcements, and even firm-level bankruptcies, anchoring the return predictability in real operational realisations rather than in a pure mispricing artefact. In practice the cosine-similarity-plus-sort construction is built on TF-IDF or document-embedding representations of the filings (see Section 20.7) rather than on raw binary word vectors, and the sorting step is combined with the usual industry and size controls.

Systematic transcript analytics. A more recent illustration of how structured NLP pipelines can extract multiple independent signals from the same call is provided by Hafez et al. (2021), who used a transcripts archive (roughly 500,000 conference-call documents from 2003 to 2021) to construct three orthogonal indicators: an event sentiment score that aggregates the tone of key events detected in the call, a document sentiment score obtained by applying a sentence-level machine learning classifier across the full transcript, and a transparency score that counts the number of market-moving events disclosed by management. On the US universe, the document sentiment signal alone delivered information ratios above 1.1 for mid/large-cap and above 2.0 for small-cap names over holding periods of a few days to a week, and combining the three signals into a single strategy lifted these figures to 1.4 and 2.3, respectively. Two conclusions generalize beyond this particular study. First, the same transcript can support multiple largely orthogonal factors (event-level tone, document-level tone, and disclosure intensity), so researchers should resist the temptation to collapse text into a single sentiment scalar. Second, transcript signals are largely additive to news-based signals: overlaying transcripts on a standalone news-analytics strategy improved the US information ratio across nearly all holding periods, consistent with the view that earnings calls carry information that does not fully propagate into daily news flow.

Quantity-of-disclosure factors. Beyond tone and readability, the quantity of disclosed text is informative in its own right. Campbell, Chen, Dhaliwal, Lu, and Steele (2014) study the mandatory Item 1A risk-factor disclosures introduced by the SEC in 2005 and show that firms tailor their disclosure language to their actual risk profile rather than rely on boilerplate: longer risk-factor sections are associated with higher market beta, higher idiosyncratic volatility, wider bid-ask spreads, and stronger post-filing return volatility, and the categorical content of disclosed risks (financial, legal, tax, systematic) maps in cross-section to the corresponding realised risk dimensions. Kravet and Muslu (2013) examine the complementary investor-reaction side and report that year-over-year changes in risk-disclosure length are positively associated with post-filing increases in market-implied volatility, with the trading-volume reaction around the 10-K filing date, and with the dispersion of analyst forecasts. Newly added risk factors drive the response more strongly than restated language, an attention pattern that echoes the Cohen, Malloy, and Nguyen (2020) “lazy prices” mechanism reviewed earlier in this subsection. The practical implication for factor construction is that both the level and the year-over-year change in risk-section length carry information, and that any text-quantity factor should be controlled for industry and filing-date effects to absorb the secular upward drift in 10-K length that the broader disclosure literature has documented.

820.8 News and media-flow factors¶

News articles, financial-media coverage, and the broader flow of public information about firms give rise to a second self-contained family of equity factors, distinct in mechanism and information horizon from the disclosure-based signals of Section 20.7. The constituent signals are constructed from the tone, volume, and breadth of media coverage rather than from corporate documents themselves, and they enter factor portfolios on their own merits, not as text-based proxies for price-based momentum. The natural reference benchmark for these signals is the price-driven momentum factor of Chapter 3, against which the news-based variant is shown to carry independent information; the rest of this section traces that argument from the early single-source evidence to the recent multi-source pipelines.

Tetlock (2007) established that media pessimism predicts negative returns, particularly for small stocks. Garcia (2013) extended this finding using a century of financial news data and showed that the predictive power of sentiment is concentrated in recessions, when investor uncertainty is highest. Chan (2003) found that stocks with news (regardless of tone) exhibit momentum, while stocks without news tend to show reversals.

These findings suggest that news-driven and price-driven return predictability (Chapter 3) capture partially distinct phenomena. A factor strategy that goes long in stocks with positive recent news sentiment and short in stocks with negative sentiment generates returns that are not fully explained by traditional momentum factors. Heston and Sinha (2017) confirmed this using a large dataset of news articles tagged with firm-level sentiment: news-based predictability is strongest at the weekly frequency and decays over one to two months.

The breadth of the underlying news source matters as much as the modeling choice. Hafez and Xie (2014) compared a sentiment indicator built on Dow Jones newswires alone with one built on a broader corpus of more than 22,000 web publications, applied as an overlay on a short-term reversal strategy on the Russell 3000. The two sources prove complementary: Dow Jones is timelier (60 to 80% of novel stories appear before the market open, against less than 50% on the web) and dominates regulatory and credit-related categories, while the web edition is broader for operational news and stabilizes monthly company coverage near 95%. Adding web content to the Dow Jones signal lowers signal turnover and lifts the information ratio of the combined strategy by up to 70%. A second result is more sobering: the predictive content of the sentiment indicator is concentrated in small and mid-cap names. The 10-day return spread between the top and bottom sentiment quintiles reaches roughly 0.38% on Russell 2000 stocks but is statistically indistinguishable from zero on Russell 1000 stocks, consistent with the view that large-cap prices already impound public news efficiently. Practitioners should therefore restrict text-based momentum strategies to the segment of the cross-section where news travels slowly enough to be tradable.

News-implied volatility. A different slice of the same media-sentiment literature focuses on text as a proxy for macro uncertainty rather than firm-level tone. Manela and Moreira (2017) built a monthly news-implied volatility (NVIX) index that spans more than a century, from 1890 to 2009, by training a support-vector regression on the front-page text of the Wall Street Journal and using the fitted model to reconstruct a VIX-like uncertainty measure in the pre-option era. Two headline findings emerge from their analysis. First, NVIX has substantial out-of-sample explanatory power for the realised VIX over the short post-1986 window in which both series exist, with out-of-sample $R^2$ on the order of 0.2. Second, a one-standard-deviation increase in NVIX predicts an excess return on the aggregate US market of roughly 3.3% over the following twelve months, consistent with a variance-risk-premium interpretation in which text-based uncertainty is compensated in expected returns. The broader point for factor investors is that text can proxy for a latent state variable (here, uncertainty) whose own numerical proxy is only available for a fraction of the sample, extending classical time-series tests into pre-option-market history without relying on volatility data that did not exist.

920.9 Coding exercises¶

1. Download the Loughran-McDonald sentiment word lists from the authors’ website. Apply the dictionary to a set of 10-K filing excerpts (available from SEC EDGAR) and compute the net tone for each filing. Group the filings by sector and compare the distribution of sentiment scores across sectors. Do some sectors systematically exhibit more negative tone?

2. Using the gensim library, train an LDA topic model with 10 topics on a corpus of at least 500 financial news headlines (e.g., from a public dataset on Kaggle). Inspect the top-10 words for each topic and assign interpretable labels. Then merge the topic proportions as features in a simple return prediction model (e.g., linear regression) and evaluate whether any topics have significant predictive power for next-month returns.

10References¶

Albladi, A., Islam, M., and Seals, C. (2025). Sentiment analysis of Twitter data using NLP models: A comprehensive review. IEEE Access, 13:30444-30468.

Antweiler, W. and Frank, M. Z. (2004). Is all that talk just noise? The information content of Internet stock message boards. Journal of Finance, 59(3):1259-1294.

Bartov, E., Faurel, L., and Mohanram, P. S. (2018). Can Twitter help predict firm-level earnings and stock returns? The Accounting Review, 93(3):25-57.

Blei, D. M. and Lafferty, J. D. (2006). Dynamic topic models. In Proceedings of the 23rd International Conference on Machine Learning, pages 113-120.

Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993-1022.

Bollen, J., Mao, H., and Zeng, X. (2011). Twitter mood predicts the stock market. Journal of Computational Science, 2(1):1-8.

Bushee, B. J., Gow, I. D., and Taylor, D. J. (2018). Linguistic complexity in firm disclosures: Obfuscation or information? Journal of Accounting Research, 56(1):85-121.

Calomiris, C. W. and Mamaysky, H. (2019). How news and its context drive risk and returns around the world. Journal of Financial Economics, 133(2):299-336.

Campbell, J. L., Chen, H., Dhaliwal, D. S., Lu, H., and Steele, L. B. (2014). The information content of mandatory risk factor disclosures in corporate filings. Review of Accounting Studies, 19(1):396-455.

Chan, W. S. (2003). Stock price reaction to news and no-news: Drift and reversal after headlines. Journal of Financial Economics, 70(2):223-260.

Chen, H., De, P., Hu, Y. J., and Hwang, B.-H. (2014). Wisdom of crowds: The value of stock opinions transmitted through social media. Review of Financial Studies, 27(5):1367-1403.

Cohen, L., Malloy, C., and Nguyen, Q. (2020). Lazy prices. Journal of Finance, 75(3):1371-1415.

Das, S. R. and Chen, M. Y. (2007). Yahoo! for Amazon: Sentiment extraction from small talk on the web. Management Science, 53(9):1375-1388.

Devitt, A. and Ahmad, K. (2007). Sentiment polarity identification in financial news: A cohesion-based approach. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 984-991.

Engelberg, J. E. and Parsons, C. A. (2011). The causal impact of media in financial markets. Journal of Finance, 66(1):67-97.

Faccia, A., McDonald, J., and George, B. (2024). NLP sentiment analysis and accounting transparency: A new era of financial record keeping. Computers, 13(1):5.

Garcia, D. (2013). Sentiment during recessions. Journal of Finance, 68(3):1267-1300.

Hafez, P. A. and Xie, J. (2012). Short-term stock selection using news based indicators. RavenPack Quantitative Research, SSRN Working Paper 2155679.

Hafez, P. A. and Xie, J. (2014). Web news analytics enhance stock portfolio returns. RavenPack Quantitative Research, SSRN Working Paper 2423362.

Hafez, P. A., Matas, R., Gomez, F., and Kangrga, M. (2021). Three trading signals generated systematically from earnings conference calls. RavenPack Quantitative Research, SSRN Working Paper 4323558.

Hassan, T. A., Hollander, S., van Lent, L., and Tahoun, A. (2019). Firm-level political risk: Measurement and effects. Quarterly Journal of Economics, 134(4):2135-2202.

Heston, S. L. and Sinha, N. R. (2017). News vs. sentiment: Predicting stock returns from news stories. Financial Analysts Journal, 73(3):67-83.

Hillert, A., Jacobs, H., and Mueller, S. (2014). Media makes momentum. Review of Financial Studies, 27(12):3467-3501.

Hoberg, G. and Phillips, G. (2016). Text-based network industries and endogenous product differentiation. Journal of Political Economy, 124(5):1423-1465.

Hutto, C. J. and Gilbert, E. (2014). VADER: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the 8th International AAAI Conference on Web and Social Media (ICWSM 2014), 8(1):216-225.

Israelsen, R. D. (2016). Does common analyst coverage explain excess comovement? Journal of Financial and Quantitative Analysis, 51(4):1193-1229.

Jegadeesh, N. and Wu, D. (2013). Word power: A new approach for content analysis. Journal of Financial Economics, 110(3):712-729.

Jurafsky, D. and Martin, J. H. (2024). Speech and Language Processing. 3rd edition (online draft).

Ke, Z. T., Kelly, B. T., and Xiu, D. (2020). Predicting returns with text data. NBER Working Paper 26186.

Kearney, C. and Liu, S. (2014). Textual sentiment in finance: A survey of methods and models. International Review of Financial Analysis, 33:171-185.

Kim, A. G., Muhn, M., and Nikolaev, V. (2024). From transcripts to insights: Uncovering corporate risks using generative AI. arXiv preprint arXiv:2310.17721.

Kogan, L., Papanikolaou, D., Seru, A., and Stoffman, N. (2017). Technological innovation, resource allocation, and growth. Quarterly Journal of Economics, 132(2):665-712.

Kravet, T. and Muslu, V. (2013). Textual risk disclosures and investors’ risk perceptions. Review of Accounting Studies, 18(4):1088-1122.

Larcker, D. F. and Zakolyukina, A. A. (2012). Detecting deceptive discussions in conference calls. Journal of Accounting Research, 50(2):495-540.

Li, F. (2008). Annual report readability, current earnings, and earnings persistence. Journal of Accounting and Economics, 45(2-3):221-247.

Lis, S. (2024). Investor sentiment in asset pricing models: A review of empirical evidence. arXiv preprint arXiv:2411.13180.

Lopez-Lira, A. (2023). Risk factors that matter: Textual analysis of risk disclosures for the cross-section of returns. SSRN Working Paper 3313663.

Lopez-Lira, A. and Tang, Y. (2023). Can ChatGPT forecast stock price movements? Return predictability and large language models. SSRN Working Paper 4412788.

Loughran, T. and McDonald, B. (2011). When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks. Journal of Finance, 66(1):35-65.

MacKinlay, A. C. (1997). Event studies in economics and finance. Journal of Economic Literature, 35(1):13-39.

Manela, A. and Moreira, A. (2017). News implied volatility and disaster concerns. Journal of Financial Economics, 123(1):137-162.

Mishev, K., Gjorgjevikj, A., Vodenska, I., Chitkushev, L. T., and Trajanov, D. (2020). Evaluation of sentiment analysis in finance: From lexicons to transformers. IEEE Access, 8:131662-131682.

Nguyen, T. H. and Shirai, K. (2015). Topic modeling based sentiment analysis on social media for stock market prediction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 1354-1364.

Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., and Rand, D. G. (2014). Structural topic models for open-ended survey responses. American Journal of Political Science, 58(4):1064-1082.

Roder, M., Both, A., and Hinneburg, A. (2015). Exploring the space of topic coherence measures. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining, pages 399-408.

Sawhney, R., Agarwal, S., Wadhwa, A., and Shah, R. R. (2020). Deep attentive learning for stock movement prediction from social media text and company correlations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 8415-8426.

Sautner, Z., van Lent, L., Vilkov, G., and Zhang, R. (2023). Firm-level climate change exposure. Journal of Finance, 78(3):1449-1498.

Schumaker, R. P., Zhang, Y., Huang, C.-N., and Chen, H. (2012). Evaluating sentiment in financial news articles. Decision Support Systems, 53(3):458-464.

Tetlock, P. C. (2007). Giving content to investor sentiment: The role of media in the stock market. Journal of Finance, 62(3):1139-1168.

Tetlock, P. C., Saar-Tsechansky, M., and Macskassy, S. (2008). More than words: Quantifying language to measure firms’ fundamentals. Journal of Finance, 63(3):1437-1467.

Webersinke, N., Kraus, M., Bingler, J. A., and Leippold, M. (2022). ClimateBERT: A pretrained language model for climate-related text. In Proceedings of the AAAI 2022 Fall Symposium: The Role of AI in Responding to Climate Challenges.

Xu, Y. and Cohen, S. B. (2018). Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), pages 1970-1979.

Xing, F. Z., Cambria, E., and Welsch, R. E. (2018). Natural language based financial forecasting: A survey. Artificial Intelligence Review, 50(1):49-73.

Yang, L., Ng, T. L. J., Smyth, B., and Dong, R. (2020). HTML: Hierarchical transformer-based multi-task learning for volatility prediction. In Proceedings of The Web Conference 2020, pages 441-451.

Zhao, F. (2018). Alpha unscripted: The message within the message in earnings calls. S&P Global Market Intelligence, Quantamental Research, SSRN Working Paper 4852473.

Zhang, W. and Skiena, S. (2010). Trading strategies to exploit blog and news sentiment. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), 4(1):375-378.

Zheludev, I., Smith, R., and Aste, T. (2014). When can social media lead financial markets? Scientific Reports, 4:4213.