Reinforcement learning

Due to its increasing popularity within the Machine Learning community, we dedicate a chapter to reinforcement learning (RL). In 2019 only, more than 25 papers dedicated to RL have been submitted to (or updated on) arXiv under the q:fin (quantitative finance) classification. Applications to trading include Xiong et al. (2018) and Théate & Ernst (2020). Market microstructure is a focal framework (Wei et al. (2019), Ferreira (2020), Karpe et al. (2020)).
Moreover, an early survey of RL-based portfolios is compiled in Sato (2019) (see also Zhang et al. (2020)) and general financial applications are discussed in Kolm & Ritter (2019), Meng & Khushi (2019), Charpentier et al. (2023) and Mosavi et al. (2020). This shows again that RL has recently gained traction among the quantitative finance community.^Like neural networks, reinforcement learning methods have also been recently developed for derivatives pricing and hedging, see for instance Kolm & Ritter, 2019, and @du2020deep.

While RL is a framework much more than a particular algorithm, its efficient application in portfolio management is not straightforward, as we will show. For a discussion on the generalization ability of RL algorithms, we refer to Packer et al. (2018) and Ghosh et al. (2021).

19.1Theoretical layout¶

19.1.1General framework¶

In this section, we introduce the core concepts of RL and follow relatively closely the notations (and layout) of Sutton & Barto (2018), which is widely considered as a solid reference in the field, along with Bertsekas (2017). One central tool in the field is called the Markov Decision Process (MDP, see Chapter 3 in Sutton & Barto (2018)).

MDPs, like all RL frameworks, involve the interaction between an agent (e.g., a trader or portfolio manager) and an environment (e.g., a financial market). The agent performs actions that may alter the state of environment and gets a reward (possibly negative) for each action. This short sequence can be repeated an arbitrary number of times, as is shown in the figure below.

Given initialized values for the state of the environment ($S_0$) and reward (usually $R_0=0$), the agent performs an action (e.g., invests in some assets). This generates a reward $R_1$ (e.g., returns, profits, Sharpe ratio) and also a future state of the environment ($S_1$). Based on that, the agent performs a new action and the sequence continues. When the sets of states, actions and rewards are finite, the MDP is logically called finite. In a financial framework, this is somewhat unrealistic and we discuss this issue later on. It nevertheless is not hard to think of simplified and discretized financial problems. For instance, the reward can be binary: win money versus lose money. In the case of only one asset, the action can also be dual: investing versus not investing. When the number of assets is sufficiently small, it is possible to set fixed proportions that lead to a reasonable number of combinations of portfolio choices, etc.

We pursue our exposé with finite MDPs; they are the most common in the literature and their formal treatment is simpler. The relative simplicity of MDPs helps grasp the concepts that are common to other RL techniques. As is often the case with Markovian objects, the key notion is that of transition probability:

which is the probability of reaching state $s'$ and reward $r$ at time $t$, conditionally on being in state $s$ and performing action $a$ at time $t-1$. The finite sets of states and actions will be denoted with $\mathcal{S}$ and $\mathcal{A}$ henceforth. Sometimes, this probability is averaged over the set of rewards which gives the following decomposition:

The goal of the agent is to maximize some function of the stream of rewards. This gain is usually defined as

i.e., it is a discounted version of the reward, where the discount factor is $\gamma \in (0,1]$. The horizon $T$ may be infinite, which is why $\gamma$ was originally introduced. Assuming the rewards are bounded, the infinite sum may diverge for $\gamma=1$. That is the case if rewards don’t decrease with time and there is no reason why they should. When $\gamma <1$ and rewards are bounded, convergence is assured. When $T$ is finite, the task is called episodic and, otherwise, it is said to be continuous.

In RL, the focal unknown to be optimized or learned is the policy $\pi$, which drives the actions of the agent. More precisely, $\pi(a,s)=\mathbb{P}[A_t=a|S_t=s]$, that is, $\pi$ equals the probability of taking action $a$ if the state of the environment is $s$. This means that actions are subject to randomness, just like for mixed strategies in game theory. While this may seem disappointing because an investor would want to be sure to take the best action, it is also a good reminder that the best way to face random outcomes may well be to randomize actions as well.

Finally, in order to try to determine the best policy, one key indicator is the so-called value function:

where the time index $t$ is not very relevant and omitted in the notation of the function. The index $\pi$ under the expectation operator $\mathbb{E}[\cdot]$ simply indicates that the average is taken when the policy $\pi$ is enforced. The value function is simply equal to the average gain conditionally on the state being equal to $s$. In financial terms, this is equivalent to the average profit if the agent takes actions driven by $\pi$ when the market environment is $s$. More generally, it is also possible to condition not only on the state, but also on the action taken. We thus introduce the $q_\pi$ action-value function:

The $q_\pi$ function is highly important because it gives the average gain when the state and action are fixed. Hence, if the current state is known, then one obvious choice is to select the action for which $q_\pi(s,\cdot)$ is the highest. Of course, this is the best solution if the optimal value of $q_\pi$ is known, which is not always the case in practice. The value function can easily be accessed via $q_\pi$: $v_\pi(s)=\sum_a \pi(a,s)q_\pi(s,a)$.

The optimal $v_\pi$ and $q_\pi$ are straightforwardly defined as

If only $v_*(s)$ is known, then the agent must span the set of actions and find those that yield the maximum value for any given state $s$.

Finding these optimal values is a very complicated task and many articles are dedicated to solving this challenge. One reason why finding the best $q_\pi(s,a)$ is difficult is because it depends on two elements ($s$ and $a$) on one side and $\pi$ on the other. Usually, for a fixed policy $\pi$, it can be time consuming to evaluate $q_\pi(s,a)$ for a given stream of actions, states and rewards. Once $q_\pi(s,a)$ is estimated, then a new policy $\pi'$ must be tested and evaluated to determine if it is better than the original one. Thus, this iterative search for a good policy can take long. For more details on policy improvement and value function updating, we recommend chapter 4 of Sutton & Barto (2018) which is dedicated to dynamic programming.

19.1.2Q-learning¶

An interesting shortcut to the problem of finding $v_*(s)$ and $q_*(s,a)$ is to remove the dependence on the policy. Consequently, there is then of course no need to iteratively improve it. The central relationship that is required to do this is the so-called Bellman equation that is satisfied by $q_\pi(s,a)$. We detail its derivation below. First of all, we recall that

where the second equality stems from Equation 19.3. The expression $\mathbb{E}_\pi[R_{t+1}|S_t=s,A_t=a]$ can be further decomposed. Since the expectation runs over $\pi$, we need to sum over all possible actions $a'$ and states $s'$ and resort to $\pi(a',s')$. In addition, the sum on the $s'$ and $r$ arguments of the probability $p(s',r|s,a)=\mathbb{P}\left[S_{t+1}=s',R_{t+1}=r | S_t=s,A_t=a \right]$ gives access to the distribution of the random couple $(S_{t+1},R_{t+1})$ so that in the end $\mathbb{E}_\pi[R_{t+1}|S_t=s,A_t=a]=\sum_{a', r,s'}\pi(a',s')p(s',r|s,a) r$. A similar reasoning applies to the second portion of $q_\pi$ and:

This equation links $q_\pi(s,a)$ to the future $q_\pi(s',a')$ from the states and actions $(s',a')$ that are accessible from $(s,a)$.

Notably, Equation 19.8 is also true for the optimal action-value function $q_*=\underset{\pi}{\max} \, q_\pi(s,a)$:

because one optimal policy is one that maximizes $q_\pi(s,a)$, for a given state $s$ and over all possible actions $a$. This expression is central to a cornerstone algorithm in reinforcement learning called $Q$-learning (the formal proof of convergence is outlined in Watkins & Dayan (1992)). In $Q$-learning, the state-action function no longer depends on policy and is written with capital $Q$. The process is the following:

Initialize values $Q(s,a)$ for all states $s$ and actions $a$. For each episode:

The underlying reason this update rule works can be linked to fixed point theorems of contraction mappings. If a function $f$ satisfies $|f(x)-f(y)|< \delta |x-y|$ (Lipshitz continuity), then a fixed point $z$ satisfying $f(z)=z$ can be iteratively obtained via $z \leftarrow f(z)$. This updating rule converges to the fixed point. Equation 19.9 can be solved using a similar principle, except that a learning rate $\eta$ slows the learning process but also technically ensures convergence under technical assumptions.

More generally, Equation 19.11 has a form that is widespread in reinforcement learning that is summarized in Equation (2.4) of Sutton & Barto (2018):

where the last part can be viewed as an error term. Starting from the old estimate, the new estimate therefore goes in the ‘right’ (or sought) direction, modulo a discount term that makes sure that the magnitude of this direction is not too large. The update rule in Equation 19.11 is often referred to as ‘temporal difference’ learning because it is driven by the improvement yielded by estimates that are known at time $t+1$ (target) versus those known at time $t$.

One important step of the Q-learning sequence (QL) is the second one where the action $a_i$ is picked. In RL, the best algorithms combine two features: exploitation and exploration. Exploitation is when the machine uses the current information at its disposal to choose the next action. In this case, for a given state $s_i$, it chooses the action $a_i$ that maximizes the expected reward $Q_i(s_i,a_i)$. While obvious, this choice is not optimal if the current function $Q_i$ is relatively far from the true $Q$. Repeating the locally optimal strategy is likely to favor a limited number of actions, which will narrowly improve the accuracy of the $Q$ function.

In order to gather new information stemming from actions that have not been tested much (but that can potentially generate higher rewards), exploration is needed. This is when an action $a_i$ is chosen randomly. The most common way to combine these two concepts is called $\epsilon$-greedy exploration. The action $a_i$ is assigned according to:

Thus, with probability $\epsilon$, the algorithm explores and with probability $1-\epsilon$, it exploits the current knowledge of the expected reward and picks the best action. Because all actions have a non-zero probability of being chosen, the policy is called “soft”. Indeed, then best action has a probability of selection equal to $1-\epsilon(1-\text{card}(\mathcal{A})^{-1})$, while all other actions are picked with probability $\epsilon/\text{card}(\mathcal{A})$.

19.1.3SARSA¶

In $Q$-learning, the algorithm seeks to find the action-value function of the optimal policy. Thus, the policy that is followed to pick actions is different from the one that is learned (via $Q$). Such algorithms are called off-policy. On-policy algorithms seek to improve the estimation of the action-value function $q_\pi$ by continuously acting according to the policy $\pi$. One canonical example of on-policy learning is the SARSA method which requires two consecutive states and actions SARSA. The way the quintuple $(S_t,A_t,R_{t+1}, S_{t+1}, A_{t+1})$ is processed is presented below.

The main difference between $Q$ learning and SARSA is the update rule. In SARSA, it is given by

The improvement comes only from the local point $Q_i(s_{i+1},a_{i+1})$ that is based on the new states and actions ($s_{i+1},a_{i+1}$), whereas in $Q$-learning, it comes from all possible actions of which only the best is retained $\underset{a}{\max} \, Q_i(s_{i+1},a)$.

A more robust but also more computationally demanding version of SARSA is expected SARSA in which the target $Q$ function is averaged over all actions:

Expected SARSA is less volatile than SARSA because the latter is strongly impacted by the random choice of $a_{i+1}$. In expected SARSA, the average smoothes the learning process.

19.2The curse of dimensionality¶

Let us first recall that reinforcement learning is a framework that is not linked to a particular algorithm. In fact, different tools can very well co-exist in a RL task (AlphaGo combined both tree methods and neural networks, see Silver et al. (2016)). Nonetheless, any RL attempt will always rely on the three key concepts: the states, actions and rewards. In factor investing, they are fairly easy to identify, though there is always room for interpretation. Actions are evidently defined by portfolio compositions. The states can be viewed as the current values that describe the economy: as a first-order approximation, it can be assumed that the feature levels fulfill this role (possibly conditioned or complemented with macro-economic data). The rewards are even more straightforward. Returns or any relevant performance metric^e.g., Sharpe ratio which is for instance used in Moody et al., 1998, @bertoluzzo2012testing and @aboussalah2020continuous or drawdown-based ratios, as in @almahdi2017adaptive. can account for rewards.

A major problem lies in the dimensionality of both states and actions. Assuming an absence of leverage (no negative weights), the actions take values on the simplex

and assuming that all features have been uniformized, their space is $[0,1]^{NK}$. Needless to say, the dimensions of both spaces are numerically impractical.

A simple solution to this problem is discretization: each space is divided into a small number of categories. Some authors do take this route. In Yang et al. (2018), the state space is discretized into three values depending on volatility, and actions are also split into three categories. Bertoluzzo & Corazza (2012), Xiong et al. (2018) and Taghian et al. (2020) also choose three possible actions (buy, hold, sell). In Almahdi & Yang (2019), the learner is expected to yield binary signals for buying or shorting. Garcı́a-Galicia et al. (2019) consider a larger state space (8 elements) but restrict the action set to 3 options.^Some recent papers consider arbitrary weights (e.g., Jiang et al., 2017, and @yu2019model) for a limited number of assets. In terms of the state space, all articles assume that the state of the economy is determined by prices (or returns).

One strong limitation of these approaches is the marked simplification they imply. Realistic discretizations are numerically intractable when investing in multiple assets. Indeed, splitting the unit interval in $h$ points yields $h^{NK}$ possibilities for feature values. The number of options for weight combinations is exponentially increasing with $N$. As an example: just 10 possible values for 10 features of 10 stocks yield 10¹⁰⁰ permutations.

The problems mentioned above are of course not restricted to portfolio construction. Many solutions have been proposed to solve Markov Decision Processes in continuous spaces. We refer for instance to Section 4 in Powell & Ma (2011) for a review of early methods (outside finance).

This curse of dimensionality is accompanied by the fundamental question of training data. Two options are conceivable: market data versus simulations. Under a given controlled generator of samples, it is hard to imagine that the algorithm will beat the solution that maximizes a given utility function. If anything, it should converge towards the static optimal solution under a stationary data generating process (see, e.g., Chaouki et al. (2020) for trading tasks), which is by the way a very strong modelling assumption.

This leaves market data as a preferred solution but even with large datasets, there is little chance to cover all the (actions, states) combinations mentioned above. Characteristics-based datasets have depths that run through a few decades of monthly data, which means several hundreds of time-stamps at most. This is by far too limited to allow for a reliable learning process. It is always possible to generate synthetic data (as in Yu et al. (2019)), but it is unclear that this will solidly improve the performance of the algorithm.

19.3Policy gradient¶

19.3.1Principle¶

Beyond the discretization of action and state spaces, a powerful trick is parametrization. When $a$ and $s$ can take discrete values, action-value functions must be computed for all pairs $(a,s)$, which can be prohibitively cumbersome. An elegant way to circumvent this problem is to assume that the policy is driven by a relatively modest number of parameters. The learning process is then focused on optimizing this set of parameters $\boldsymbol{\theta}$. We then write $\pi_{\boldsymbol{\theta}}(a,s)$ for the probability of choosing action $a$ in state $s$. One intuitive way to define $\pi_{\boldsymbol{\theta}}(a,s)$ is to resort to a soft-max form:

where the output of function $\textbf{h}(a,s)$, which has the same dimension as $\boldsymbol{\theta}$ is called a feature vector representing the pair $(a,s)$. Typically, $\textbf{h}$ can very well be a simple neural network with two input units and an output dimension equal to the length of $\boldsymbol{\theta}$.

One desired property for $\pi_{\boldsymbol{\theta}}$ is that it be differentiable with respect to $\boldsymbol{\theta}$ so that $\boldsymbol{\theta}$ can be improved via some gradient method. The most simple and intuitive results about policy gradients are known in the case of episodic tasks (finite horizon) for which it is sought to maximize the average gain $\mathbb{E}_{\boldsymbol{\theta}}[G_t]$ where the gain is defined in Equation 19.3. The expectation is computed according to a particular policy that depends on $\boldsymbol{\theta}$, this is why we use a simple subscript. One central result is the so-called policy gradient theorem which states that

This result can then be used for gradient ascent: when seeking to maximize a quantity, the parameter change must go in the upward direction:

This simple update rule is known as the REINFORCE algorithm. One improvement of this simple idea is to add a baseline, and we refer to section 13.4 of Sutton & Barto (2018) for a detailed account on this topic.

19.3.2Extensions¶

A popular extension of REINFORCE is the so-called actor-critic (AC) method which combines policy gradient with $Q$- or $v$-learning. The AC algorithm can be viewed as some kind of mix between policy gradient and SARSA. A central requirement is that the state-value function $v(\cdot)$ be a differentiable function of some parameter vector $\textbf{w}$ (it is often taken to be a neural network). The update rule is then

but the trick is that the vector $\textbf{w}$ must also be updated. The actor is the policy side which is what drives decision making. The critic side is the value function that evaluates the actor’s performance. As learning progresses (each time both sets of parameters are updated), both sides improve. The exact algorithmic formulation is a bit long and we refer to Section 13.5 in Sutton & Barto (2018) for the precise sequence of steps of AC.

Another interesting application of parametric policies is outlined in Aboussalah & Lee (2020). In their article, the authors define a trading policy that is based on a recurrent neural network. Thus, the parameter $\boldsymbol{\theta}$ in this case encompasses all weights and biases in the network.

Another favorable feature of parametric policies is that they are compatible with continuous sets of actions. Beyond the form 19.17, there are other ways to shape $\pi_{\boldsymbol{\theta}}$. If $\mathcal{A}$ is a subset of $\mathbb{R}$, and $f_{\boldsymbol{\Omega}}$ is a density function with parameters $\boldsymbol{\Omega}$, then a candidate form for $\pi_{\boldsymbol{\theta}}$ is

in which the parameters $\boldsymbol{\Omega}$ are in turn functions of the states and of the underlying (second order) parameters $\boldsymbol{\theta}$.

While the Gaussian distribution (see section 13.7 in Sutton & Barto (2018)) is often a preferred choice, they would require some processing to lie inside the unit interval. One easy way to obtain such values is to apply the normal cumulative distribution function to the output. In Wang & Zhou (2019), the multivariate Gaussian policy is theoretically explored, but it assumes no constraint on weights.

Some natural parametric distributions emerge as alternatives. If only one asset is traded, then the Bernoulli distribution can be used to determine whether or not to buy the asset. If a riskless asset is available, the beta distribution offers more flexibility because the values for the proportion invested in the risky asset span the whole interval; the remainder can be invested into the safe asset. When many assets are traded, things become more complicated because of the budget constraint. One ideal candidate is the Dirichlet distribution because it is defined on a simplex (see Equation 19.16):

where $B(\boldsymbol{\alpha})$ is the multinomial beta function:

If we set $\pi=\pi_{\boldsymbol{\alpha}}=f_{\boldsymbol{\alpha}}$, the link with factors or characteristics can be coded through ${\boldsymbol{\alpha}}$ via a linear form:

which is highly tractable, but may violate the condition that $\alpha_{n,t}>0$ for some values of $\theta_{k,t}$. Indeed, during the learning process, an update in $\boldsymbol{\theta}$ might yield values that are out of the feasible set of $\boldsymbol{\alpha}_t$. In this case, it is possible to resort to a trick that is widely used in online learning (see, e.g., section 2.3.1 in Hoi et al. (2018)). The idea is simply to find the acceptable solution that is closest to the suggestion from the algorithm. If we call $\boldsymbol{\theta}^*$ the result of an update rule from a given algorithm, then the closest feasible vector is

where $||\cdot||$ is the Euclidean norm and $\Theta(\textbf{x}_t)$ is the feasible set, that is, the set of vectors $\boldsymbol{\theta}$ such that the $\alpha_{n,t}=\theta_{0,t} + \sum_{k=1}^K \theta_{t}^{(k)}x_{t,n}^{(k)}$ are all non-negative.

A second option for the form of the policy, $\pi^2_{\boldsymbol{\theta}_t}$, is slightly more complex but remains always valid (i.e., has positive $\alpha_{n,t}$ values):

which is simply the exponential of the first version. With some algebra, it is possible to derive the policy gradients. The policies $\pi^j_{\boldsymbol{\theta}_t}$ are defined by the Equations $(\textbf{Fj})$ above. Let $\digamma$ denote the digamma function. Let $\textbf{1}$ denote the $\mathbb{R}^N$ vector of all ones. We have

where $e^{\textbf{X}}$ is the element-wise exponential of a matrix $\textbf{X}$.

The allocation can then either be made by direct sampling, or using the mean of the distribution $(\textbf{1}'\boldsymbol{\alpha})^{-1}\boldsymbol{\alpha}$. Lastly, a technical note: Dirichlet distributions can only be used for small portfolios because the scaling constant in the density becomes numerically intractable for large values of $N$ (e.g., above 50). More details on this idea are laid out in André & Coqueret (2020).

19.4Simple examples¶

19.4.1Setup and Imports¶

19.4.2Q-learning with simulations¶

To illustrate the gist of the problems mentioned above, we propose two implementations of $Q$-learning. For simplicity, the first one is based on simulations. This helps understand the learning process in a simplified framework. We consider two assets: one risky and one riskless, with return equal to zero. The returns for the risky process follow an autoregressive model of order one (AR(1)): $r_{t+1}=a+\rho r_t+\epsilon_{t+1}$ with $|\rho|<1$ and $\epsilon$ following a standard white noise with variance $\sigma^2$. In practice, individual (monthly) returns are seldom autocorrelated, but adjusting the autocorrelation helps understand if the algorithm learns correctly (see exercise below).

The environment consists only in observing the past return $r_t$. Since we seek to estimate the $Q$ function, we need to discretize this state variable. The simplest choice is to resort to a binary variable: equal to -1 (negative) if $r_t<0$ and to +1 (positive) if $r_t\ge 0$. The actions are summarized by the quantity invested in the risky asset. It can take 5 values: 0 (risk-free portfolio), 0.25, 0.5, 0.75 and 1 (fully invested in the risky asset). This is for instance the same choice as in Pendharkar & Cusatis (2018).

Below we implement a simple Q-learning algorithm from scratch in Python. The landscape of Python libraries for RL is richer than R, with options like Stable-Baselines3, RLlib, and Gymnasium (formerly OpenAI Gym). However, for pedagogical purposes and to match the simplicity of the original example, we implement Q-learning directly.

Now let’s generate the simulated data with an AR(1) process and train our Q-learning agent.

There are 3 parameters in the implementation of the Q-learning algorithm:

• $\eta$, which is the learning rate in the updating Equation 19.11. In our implementation, this is coded as alpha;

• $\gamma$, the discounting rate for the rewards (also shown in Equation 19.11);

• and $\epsilon$, which controls the rate of exploration versus exploitation (see Equation 19.13).

The output shows the Q function, which depends naturally both on states and actions. When the state is negative, large risky positions (action equal to 0.75 or 1.00) are associated with the smallest average rewards, whereas small positions yield the highest average rewards. When the state is positive, the average rewards are the highest for the largest allocations. The rewards in both cases are almost a monotonic function of the proportion invested in the risky asset. Thus, the recommendation of the algorithm (i.e., the policy) is to be fully invested in a positive state and to refrain from investing in a negative state. Given the positive autocorrelation of the underlying process, this does make sense.

Basically, the algorithm has simply learned that positive (resp. negative) returns are more likely to follow positive (resp. negative) returns. While this is somewhat reassuring, it is by no means impressive, and much simpler tools would yield similar conclusions and guidance.

19.4.3Visualization of the Q-function¶

19.4.4Q-learning with market data¶

The second application is based on the financial dataset. To reduce the dimensionality of the problem, we will assume that:

• only one feature (price-to-book ratio) captures the state of the environment. This feature is processed so that it has only a limited number of possible values;

• actions take values over a discrete set consisting of three positions: +1 (buy the market), -1 (sell the market) and 0 (hold no risky positions);

• only two assets are traded: those with fsym_id equal to 3 and 4 - they both have 245 days of trading data.

The construction of the dataset is coded below.

Actions and states have to be merged to yield all possible combinations. To simplify the states, we round 5 times the price-to-book ratios.

We keep the same hyperparameters as in the previous example. Columns below stand for actions: the first (resp. second) number notes the position in the first (resp. second) asset. The rows correspond to states. The scaled P/B ratios are separated by a point (e.g., “2.3” means that the first (resp. second) asset has a scaled P/B of 2 (resp. 3).

The output shows that there are many combinations of states and actions that are not spanned by the data: basically, the $Q$ function has a zero and it is likely that the combination has not been explored. Some states seem to be more often represented, others, less. It is hard to make any sense of the recommendations. Some states are close but the outcomes related to them can be very different. Moreover, there is no coherence and no monotonicity in actions with respect to individual state values: low values of states can be associated to very different actions.

One reason why these conclusions do not appear trustworthy pertains to the data size. With only 200+ time points and many state-action pairs, this yields on average only a few data points to compute the $Q$ function. This could be improved by testing more random actions, but the limits of the sample size would eventually (rapidly) be reached anyway. This is left as an exercise (see below).

19.4.5Visualization of Q-values for Market Data¶

19.5Modern RL Approaches: Deep Q-Networks¶

The tabular Q-learning approach shown above suffers from the curse of dimensionality. Modern approaches use function approximation to represent the Q-function, typically with neural networks. This is the foundation of Deep Q-Networks (DQN), which was popularized by DeepMind’s work on playing Atari games.

Below, we provide a simple implementation of DQN using Keras for the same simulated environment. Keras 3 supports multiple backends including JAX, TensorFlow, and PyTorch. This demonstrates how neural networks can scale Q-learning to continuous or high-dimensional state spaces.

The DQN learns a policy that is consistent with the tabular Q-learning result: invest more when the previous return is positive (exploiting the positive autocorrelation) and invest less when the previous return is negative. The advantage of DQN is that it can handle continuous states without discretization, and it scales to much higher-dimensional problems.

19.6Concluding remarks¶

Reinforcement learning has been applied to financial problems for a long time. Early contributions in the late 1990s include Neuneier (1996), Moody & Wu (1997), Moody et al. (1998) and Neuneier (1998). Since then, many researchers in the computer science field have sought to apply RL techniques to portfolio problems. The advent of massive datasets and the increase in dimensionality make it hard for RL tools to adapt well to very rich environments that are encountered in factor investing.

Recently, some approaches seek to adapt RL to continuous action spaces (Wang & Zhou (2019), Aboussalah & Lee (2020)) but not to high-dimensional state spaces. These spaces are those required in factor investing because all firms yield hundreds of data points characterizing their economic situation. In addition, applications of RL in financial frameworks have a particularity compared to many typical RL tasks: in financial markets, actions of agents have no impact on the environment (unless the agent is able to perform massive trades, which is rare and ill-advised because it pushes prices in the wrong direction). This lack of impact of actions may possibly mitigate the efficiency of traditional RL approaches.

Those are challenges that will need to be solved in order for RL to become competitive with alternative (supervised) methods. Nevertheless, the progressive (online-like) way RL works seems suitable for non-stationary environments: the algorithm slowly shifts paradigms as new data arrives. In stationary environments, it has been shown that RL manages to converge to optimal solutions (Kong et al. (2019), Chaouki et al. (2020)). Therefore, in non-stationary markets, RL could be a recourse to build dynamic predictions that adapt to changing macroeconomic conditions. More research needs to be carried out in this field on large dimensional datasets.

We end this chapter by underlining that reinforcement learning has also been used to estimate complex theoretical models (Halperin & Feldshteyn (2018), Garcı́a-Galicia et al. (2019)). The research in the field is incredibly diversified and is orientated towards many directions. It is likely that captivating work will be published in the near future.

19.7Exercises¶

1. Test what happens if the process for generating returns has a negative autocorrelation. What is the impact on the $Q$ function and the policy?

2. Keeping the same 2 assets as in Section Section 19.4.4, increase the size of RL_data by testing all possible action combinations for each original data point. Re-run the $Q$-learning function and see what happens.

3. Implement SARSA (Equation 19.14) and compare its performance with Q-learning on the simulated AR(1) environment.

4. Modify the DQN implementation to use Double DQN (separate networks for action selection and value estimation) and compare performance.

5. Implement a simple policy gradient method (REINFORCE) for the AR(1) environment using Keras and compare with Q-learning approaches.

19.8Additional Topics: Stable-Baselines3 Integration¶

For more sophisticated RL applications, the Stable-Baselines3 library provides production-ready implementations of various RL algorithms (PPO, SAC, TD3, etc.). Below is an example of how to set up a custom financial environment compatible with the Gymnasium interface.

This environment can be used with Stable-Baselines3 algorithms:

from stable_baselines3 import PPO, SAC

# Using PPO (Proximal Policy Optimization)
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=100000)

# Or using SAC (Soft Actor-Critic) for continuous actions
model = SAC("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=100000)

These modern algorithms handle continuous action spaces natively and include various improvements over basic Q-learning, such as entropy regularization, importance sampling, and policy clipping.