Chapter 22: Transformers and pre-trained language models for finance¶

122.1 Introduction¶

The embeddings of Chapter 21 lift bag-of-words representations into a dense, distributional space, but they remain context-blind: every word has exactly one vector regardless of the surrounding sentence. For financial text this is a binding limitation. Volatility in an options-pricing discussion sits in a different region of the language space than volatility in a political-stability discussion; exposure across market risk, media attention, and environmental contamination should occupy three different points, not one. Resolving this limitation requires an architecture in which a token’s representation depends on the entire input sequence rather than on the token’s type alone.

The transformer architecture introduced by Vaswani et al. (2017) provides exactly that. Each token is encoded conditional on every other token in the sequence through a self-attention mechanism, and stacking such layers produces representations in which the contextual encoder is the model rather than a fixed table. We develop the architecture from first principles, cover the transfer-learning paradigm that has made pre-trained language models practical, and survey the finance-specific pre-trained models built on it. The decoder-only and encoder-decoder variants that underpin large language models are introduced here, but their full treatment is deferred to Chapter 23.

The methodological progression of this chapter parallels the architectural progression of the field between 2017 and 2020: attention as a generic mechanism (2017), BERT as the canonical pre-trained encoder (2018), and finance-specific pre-trained models (2019 onward).

222.2 Why attention changed everything¶

The classical methods of Chapter 20 represent text as a bag of words, and the embeddings of Chapter 21 as static vectors, both discarding the sequential structure of language. Recurrent neural networks (RNNs) and their variant, long short-term memory networks (LSTMs) introduced by Hochreiter & Schmidhuber (1997), addressed this by processing text sequentially, maintaining a hidden state that evolves as each word is read (see Chapter 7 for neural network foundations). Concretely, the LSTM cell at position $t$ takes the previous hidden state $\mathbf{h}_{t-1}$ and the new input $\mathbf{x}_t$ and produces a new cell state $\mathbf{c}_t$ and hidden state $\mathbf{h}_t$ through four gates,

The forget gate $\mathbf{f}_t$, input gate $\mathbf{i}_t$, and output gate $\mathbf{o}_t$ control how much of the previous cell state is retained, how much new information is written, and how much of the resulting state is exposed to downstream layers, while the fourth gate, the candidate $\mathbf{g}_t$, supplies through a $\tanh$ the new content that the input gate then writes into the cell. The architecture solves the vanishing-gradient problem of vanilla RNNs, which Pascanu et al. (2013) formalise as the reason recurrent networks are hard to train over long horizons, but the sequential dependence (each $\mathbf{h}_t$ requires $\mathbf{h}_{t-1}$) prevents parallelisation over the time dimension and limits effective context length in practice to a few hundred tokens. These two limitations are exactly what attention was designed to overcome. A first generalisation came from Bahdanau et al. (2015), who introduced an additive attention mechanism in the context of neural machine translation: instead of compressing an input sentence into a single fixed-length hidden state, their model produced a context vector that was a soft-weighted alignment over all source tokens, with the weights themselves learned. This idea, that the model should learn where to look, is the conceptual bridge between recurrent architectures and the transformer, and it is the reason every modern transformer paper traces its lineage to the 2015 attention-for-translation construction rather than to the 2017 self-attention paper alone.

Before describing why attention displaced recurrent architectures, it is useful to know how high the ceiling actually was. Sohangir et al. (2018) provided an early systematic comparison of CNN, LSTM, and Doc2Vec models on financial sentiment over StockTwits messages and showed that CNNs comfortably outpaced classical bag-of-words baselines, but then stalled: adding depth or width to the convolutional architecture yielded diminishing returns, with no model clearing 60% accuracy on out-of-sample financial sentiment. That plateau, arguably as much as any single architectural flaw, is often seen as having helped push the field toward attention-based models. Correia et al. (2022) later benchmarked seven CNN, LSTM, GRU, and hybrid architectures on a stock-sentiment classification task and reported test accuracies that plateau around 69%, with the best model achieving roughly 74% training accuracy. When the resulting signal was pushed through a trading simulation, the top models generated a return on investment of approximately 4.8% over the test horizon. These numbers set the empirical baseline that transformer-based models had to beat, and they illustrate why even carefully engineered pre-transformer sequence models left a large amount of information on the table.

We can see this ceiling directly, rather than take it on trust, by training a genuine LSTM sentiment classifier on a standard financial benchmark. We use the Financial PhraseBank of Malo et al. (2014), a corpus of financial-news sentences each hand-labelled negative, neutral, or positive by annotators.

The corpus holds 4,846 sentences; we hold out 30% for testing and build the vocabulary from the training split alone, so no test word inflates the model’s apparent knowledge. We then encode each sentence as a fixed-length sequence of token indices and train the single-layer LSTM of Equation (22.1), with an embedding layer feeding the recurrence and a linear head reading its final hidden state, as a three-class classifier.

The result is a textbook illustration of the plateau. On held-out sentences the LSTM reaches only 66.6% accuracy, barely above the 59.4% share of the majority (neutral) class, even as its 82.5% training accuracy shows that the spare capacity is being spent memorising the training set rather than generalising. That wide gap between a strong training fit and a mediocre test score is a familiar sign of a model that has largely exhausted the generalisable structure it can extract from the data, and 66.6% sits squarely in the plateau band that Sohangir et al. (2018) and Correia et al. (2022) mapped out with far more elaborate architectures. The two structural limitations of recurrent models that attention was built to remove bear stating precisely:

1. Vanishing gradients. Information from early words in a long sequence is progressively diluted as the hidden state is updated, making it difficult to capture long-range dependencies. A sentence like “The company, which was founded in 1987 and has since expanded into 47 countries across six continents, reported strong earnings” requires linking “company” to “reported” across many intervening tokens.

2. Sequential bottleneck. Because each hidden state depends on the previous one, RNN computation cannot be parallelised across sequence positions, making training slow on modern hardware (GPUs and TPUs are designed for parallel computation).

The transformer architecture of Vaswani et al. (2017), whose title deliberately announced that attention is all you need, resolved both problems at once by dispensing with recurrence altogether. Bahdanau et al. (2015) had added attention on top of a recurrent encoder, the 2017 construction kept the attention and threw the recurrence away. In place of a hidden state passed from one position to the next, self-attention lets every token attend to every other token in the sequence simultaneously: in the example above, “company” attends directly to “reported” regardless of the distance between them, which collapses the long path that a recurrent network must traverse one step at a time. Because the positions no longer form a chain, they can also be computed in parallel, so the same construction removes the vanishing-gradient dilution and the sequential bottleneck together. It is this combination, rather than any single trick, that let attention-based models break through the accuracy ceiling documented above and, in the sections that follow, scale to corpus sizes no recurrent model could digest. We now turn to the mechanism itself.

322.3 Inside the transformer¶

The transformer consists of stacked layers, each containing a multi-head self-attention mechanism and a position-wise feed-forward network. We describe the core components.

Scaled dot-product attention. Given a sequence of $T$ token representations, the attention mechanism computes three matrices from the input: queries $\mathbf{Q}$, keys $\mathbf{K}$, and values $\mathbf{V}$, each obtained by linear projection. The attention output is:

where $d_k$ is the dimension of the key vectors, and the division by $\sqrt{d_k}$ prevents the dot products from growing too large. The softmax is applied row-wise, so each row of the output is a weighted average of the value vectors, with weights determined by the compatibility between the query and each key. Intuitively, Equation (22.2) computes how much each token should “attend to” every other token.

Multi-head attention. Rather than computing a single attention function, the transformer uses $H$ parallel attention heads, each with its own learned projection matrices. This allows the model to attend to different types of relationships simultaneously: one head might capture syntactic dependencies (subject-verb agreement), while another captures semantic relationships (entity co-reference). The outputs of all heads are concatenated and linearly projected:

where $\text{head}_h = \text{Attention}(\mathbf{Q}\mathbf{W}_h^Q, \mathbf{K}\mathbf{W}_h^K, \mathbf{V}\mathbf{W}_h^V)$.

Positional encoding. Because self-attention is permutation-invariant (it treats the input as a set, not a sequence), the transformer adds positional encodings to the input embeddings to inject information about token order. The original transformer uses sinusoidal functions of different frequencies:

where $t$ is the position and $i$ is the dimension index. Modern transformers often use learned positional embeddings instead.

Feed-forward networks and residual connections. After each attention layer, a position-wise feed-forward network (a two-layer MLP with ReLU activation) transforms each token representation independently,

Each sub-layer (multi-head attention or feed-forward network) is wrapped in a residual connection, following the deep-residual-learning idea of He et al. (2016), and a layer-normalisation step, following Ba et al. (2016),

so that gradients flow back through the identity branch even when the sub-layer is poorly conditioned. The composition $\text{LayerNorm} \circ (\mathbf{x} + \text{Sublayer}(\mathbf{x}))$ is what gives the transformer its trainable depth: stacking many such blocks (12 for BERT-base, 96 for GPT-3, and roughly 60 to 130 for the largest openly documented models of the mid-2020s, such as the 126 layers of Llama 3.1 405B or the 61 layers of the mixture-of-experts DeepSeek-V3) without the residual-plus-normalisation pattern would lead to vanishing or exploding activations within a handful of layers. It is worth noting that depth itself has largely plateaued: the exact layer counts of the largest proprietary systems are not publicly disclosed, but the recent growth in capability has come far more from width and from mixture-of-experts routing, which adds parameters without adding depth, than from ever-taller stacks.

These components (attention, the position-wise feed-forward network, and the residual-plus-normalisation wrapper) assemble into the two stacks of the original architecture of Vaswani et al. (2017), shown in FIGURE 22.2: an encoder that maps the input sequence to a set of contextual representations, and a decoder that consumes those representations to generate an output sequence one token at a time.

FIGURE 22.2 makes the two data paths explicit. On the encoder side, each block applies multi-head self-attention followed by a feed-forward network, with every sub-layer wrapped in the residual add-and-normalise step of Equation (22.6), and the block is repeated N times. On the decoder side, an analogous stack inserts two extra pieces: a masked self-attention, so that position $t$ cannot attend to tokens that come after it, and a cross-attention sub-layer whose queries come from the decoder but whose keys and values come from the encoder output, which is how information flows from the input sequence to the generated one. A final linear projection followed by a softmax turns the top representation into a probability distribution over the vocabulary. The reader has met the words “encoder” and “decoder” before, in the autoencoder of Chapter 7, but the meaning here is different: the transformer’s encoder and decoder do not compress and then reconstruct a single input, they map one sequence into another through attention.

We illustrate how self-attention resolves lexical ambiguity using a pre-trained transformer model.

The key observation is that the BERT embeddings for “bank” differ substantially between the two sentences, even though the word is identical. In the financial sentence, the embedding captures the meaning of “bank” as a financial institution (influenced by context words like “earnings,” “quarterly,” “reported”). In the river sentence, the embedding reflects the geographical meaning (influenced by “river,” “walked,” “sunset”). This is precisely the contextual understanding that static word embeddings (Chapter 21) cannot provide.

As shown in FIGURE 22.1, the transformer concentrates attention from “bank” on the words “earnings” and “reported,” which disambiguate the financial meaning. This attention pattern demonstrates the mechanism by which transformers build contextual representations. As we discuss in Section 24.5, attention visualisation is also a useful interpretability tool for understanding what a model focuses on when making predictions.

422.4 Architecture variants¶

The original transformer of Vaswani et al. (2017) contains both an encoder (which processes the input sequence) and a decoder (which generates the output sequence). Subsequent work has shown that using only the encoder or only the decoder yields powerful models for different tasks:

• Encoder-only (BERT family). The encoder processes the entire input sequence bidirectionally, meaning each token can attend to both left and right context. This is ideal for understanding tasks: sentiment classification, named entity recognition, and textual similarity. BERT Devlin et al., 2019 is the most prominent example.

• Decoder-only (GPT family). The decoder processes tokens autoregressively, attending only to previous tokens (not future ones). This is ideal for generation tasks: text completion, summarisation, and dialogue. GPT-2, GPT-3 Brown et al., 2020, and subsequent models follow this paradigm.

• Encoder-decoder (T5, BART). The full encoder-decoder architecture is used for sequence-to-sequence tasks: translation, abstractive summarisation, and question answering. Lewis et al. (2020) train BART as a denoising autoencoder in this family, and Raffel et al. (2020) showed that many NLP tasks can be framed in this text-to-text format.

The essential difference between the encoder-only and decoder-only families is not the depth of the stack but the attention mask each one applies, illustrated in FIGURE 22.3.

The contrast in FIGURE 22.3 explains why the two families specialise as they do. Bidirectional attention lets an encoder condition each token’s representation on the whole sentence, which is exactly what a sentiment classifier or a named-entity tagger needs, but it makes the model unusable for generation, since predicting the next word while already being allowed to see it is a degenerate task. The causal mask of the decoder removes that leakage, at the cost of context: each representation is built from the left only. For financial NLP, encoder-only models are most commonly used for classification and feature extraction, while decoder-only models (LLMs) are used for generation, summarisation, and zero-shot analysis. The encoder-only family is developed in the sections that follow; the decoder-only models that power large language models are the subject of Chapter 23.

522.5 Transfer learning and fine-tuning¶

Training a transformer from scratch requires enormous computational resources: BERT was pre-trained on 3.3 billion words (the entire English Wikipedia plus the BookCorpus) using 64 TPU chips for several days. Few financial practitioners have access to such resources. The solution is transfer learning: pre-train a model on a large general-purpose corpus to learn the structure of language, then fine-tune it on a small task-specific dataset to specialise it for the application of interest. Howard & Ruder (2018) established this pre-train-then-fine-tune paradigm for NLP text classification, showing that a language model pre-trained on general text could be adapted to a downstream task with only a modest labelled sample.

The pre-training phase teaches the model general linguistic knowledge: grammar, semantics, world knowledge, and reasoning patterns. The fine-tuning phase adapts this knowledge to a specific task (e.g., financial sentiment classification) using a much smaller labelled dataset, typically a few hundred to a few thousand examples. This approach works remarkably well because linguistic structure is largely shared across domains; what changes between general English and financial English is primarily vocabulary and domain-specific conventions.

622.6 BERT and masked language modelling¶

The value of bidirectional context had already been demonstrated by Peters et al. (2018), whose ELMo model derived context-dependent word representations from a bidirectional LSTM language model; BERT brought this idea into the transformer. BERT (Bidirectional Encoder Representations from Transformers; Devlin et al., 2019) is pre-trained using two objectives:

1. Masked language modelling (MLM). Randomly select 15% of the input tokens and train the model to predict them from context. In practice, of the selected positions, 80% are replaced with a [MASK] token, 10% with a random token, and 10% are left unchanged, a scheme that reduces the mismatch between pre-training (which sees [MASK]) and fine-tuning (which does not). For example, given “The [MASK] reported quarterly [MASK],” the model must predict “company” and “earnings.” This forces the model to build rich bidirectional representations.

2. Next sentence prediction (NSP). Given two sentences concatenated with a [SEP] separator token, predict whether the second follows the first in the original text. This teaches the model about discourse coherence.

Formally, let $\mathbf{x} = (x_1, \ldots, x_T)$ be an input token sequence and $\mathcal{M} \subset \{1, \ldots, T\}$ the random subset of masked positions (typically $|\mathcal{M}| = 0.15\, T$). The MLM objective is the masked negative log-likelihood,

where $\mathbf{x}_{\setminus \mathcal{M}}$ denotes the input sequence with the positions in $\mathcal{M}$ replaced by a [MASK] token, and $\theta$ are the transformer parameters. The combined pre-training loss adds the binary next-sentence-prediction objective, $\mathcal{L}_{\text{BERT}} = \mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{NSP}}$. After pre-training, the model’s internal representations (the output of the final transformer layer) serve as powerful features for downstream tasks. For classification, a simple linear layer is added on top of the [CLS] token embedding (a special token prepended to every input) and fine-tuned with labelled data.

722.7 Finance-specific models¶

General-purpose models like BERT sometimes struggle with financial text because of domain-specific vocabulary and conventions. Several finance-adapted models have been proposed:

FinBERT is in fact a family of three closely related models that are easily confused. The original Araci (2019) version further pre-trained BERT on Reuters TRC2-financial news and fine-tuned it on the Financial PhraseBank sentiment benchmark. Yang et al. (2020) released a separate FinBERT trained from scratch on a much larger 4.9-billion-token corpus of corporate reports, earnings-call transcripts, and analyst reports, which has become the de facto open-source FinBERT in practitioner pipelines. Huang et al. (2023) then introduced a third FinBERT released alongside their Contemporary Accounting Research paper, pre-trained on a still broader financial corpus and benchmarked on a wider battery of finance NLP tasks. All three significantly outperform general-purpose BERT and dictionary methods on financial sentiment benchmarks, and the differences between them matter mostly when reproducing prior numbers: the choice of pre-training corpus drives most of the residual gap.

ClimateBERT Webersinke et al., 2022 is adapted for climate-related text, trained on climate disclosures, IPCC reports, and sustainability publications. It enables automated analysis of climate risk in corporate filings, a task of growing importance for ESG-oriented investors.

Domain adaptation is not confined to encoder models. Wu et al. (2023) train BloombergGPT, a decoder-only large language model, from scratch on a corpus that mixes financial and general-purpose text, producing a finance-specialised counterpart to the general generative models taken up in Chapter 23.

A parallel strand of research applies transformer architectures specifically to earnings-call transcripts, which pose a distinctive challenge because of their length (often thousands of tokens) and mixed register (formal prepared remarks followed by unscripted Q&A). Yang et al. (2020) address this with HTML (Hierarchical Transformer-based Multi-task Learning), a model that segments a transcript into utterance-level blocks, applies a local transformer within each block, and then applies a second transformer across blocks to capture discourse-level structure. Trained jointly to predict both short-term volatility and price direction, HTML outperformed flat BERT and RNN baselines on earnings-call benchmarks, and it established the hierarchical transformer as the canonical architecture for long financial documents. The fact that the academic transcript literature Yang et al., 2020 and the practitioner transcript literature surveyed in Chapter 20 converge on the same genre of text underscores the importance of earnings calls as a signal source.

The common thread is domain adaptation: starting from a general pre-trained model and further training on domain-specific text. This approach combines the linguistic knowledge from general pre-training with the specialised vocabulary and conventions of finance. It is worth emphasising a division of labour that the architecture motivates: these contextual encoder models excel at classification tasks (sentiment, topic, entity type), where a single label is attached to the full input. For similarity tasks across pairs of sentences or documents (peer-group construction, filing-change detection, cross-asset analogies), the sentence-transformer family of Section 21.4 is the more appropriate tool, because its contrastive training objective calibrates cosine distances in a way that the masked-language-modelling objective of standard BERT does not.

We apply FinBERT to the same six KO-flavoured sentences used in the Loughran-McDonald demonstration of Section 20.4 and compare its classifications directly against the dictionary approach.

The contrast with the dictionary approach (Section 20.4) is striking. FinBERT correctly classifies Doc 2 (“Operating losses did not materialize, and margins held firm”) as positive with 89.3% confidence, whereas the Loughran-McDonald dictionary scored it as -1.00. The model’s contextual understanding of negation is its key advantage: it processes “losses did not materialize” as a single negated expression rather than counting “losses” in isolation.

The two methods agree on five of six sentences. The disagreement on Doc 2 highlights the negation-handling advantage of transformer-based models. In a large-scale evaluation this type of error correction accumulates, though the resulting lift over a well-tuned lexicon is typically modest.

822.8 When to use FinBERT (and when not to)¶

The practical gains of contextual representations are largest precisely for the failure modes catalogued in Section 20.4: negation, conditional language, and context-dependent polarity, cases in which the sentiment carried by a word depends on the words around it. Because FinBERT encodes each token in the context of the whole sentence rather than scoring it in isolation, it resolves many of the cases that a fixed lexicon systematically mishandles, and on standard financial-sentiment benchmarks it is usually the more accurate of the two. That advantage is genuine, but it is bounded, and it is worth being precise about where it stops.

Fine-tuned models are not a panacea. They require labelled training data, which is expensive to create for finance; they can be sensitive to the distribution of that data, so a model fine-tuned on news may underperform on earnings calls; and they are computationally heavier than dictionary lookups. For text that is relatively formulaic (standardised filings, for instance) and whose vocabulary is stable, dictionary methods remain a cost-effective baseline, a point the survey of textual analysis in accounting and finance by Loughran & McDonald (2016) makes at length. More pointedly, FinBERT is no longer the frontier it once was: Fatouros et al. (2023) report that a general-purpose large language model (ChatGPT), simply prompted, outperforms FinBERT on financial-sentiment classification by roughly a third and correlates more strongly with subsequent returns, so the domain-specific fine-tune no longer dominates a general model that can be steered with a well-written prompt. A second and more unsettling line of criticism concerns robustness. Leippold (2023) shows that financial sentiment scores can be adversarially manipulated: paraphrasing a sentence with a language model flips a keyword classifier’s verdict in close to 99% of negative cases, and although context-aware encoders looked comparatively resistant in that study, Turetken & Leippold (2026) go on to show that the transformer sentiment models themselves, FinBERT included, can be fooled by targeted attacks. The concern is not merely academic, since firms increasingly craft disclosures in the knowledge that machines will read them, so any deployed sentiment model operates in a setting where the very text it scores is partly written to move the score. The reasonable conclusion is not that FinBERT should be avoided, but that its output is one noisy, potentially gameable signal among several, best combined with dictionary and market-based measures rather than trusted on its own.

922.9 Coding exercises¶

1. Load a pre-trained FinBERT model with the transformers library and extract the [CLS] token embedding for each of the six example sentences of Section 22.7. Compute the pairwise cosine-similarity matrix of the embeddings, identify the pair of sentences the model considers most similar, and comment on whether the grouping reflects the topic of the sentences or their sentiment.

10References¶