Quants Quest

Signed Dual Attention Explained

2026-10-04T00:00:00+00:00

In this post I will try my best to explain my latest work, the Signed Dual Attention (SDA) mechanism, developed with Tristan Cazenave as part of my PhD and presented at the AI4TS workshop (AAAI 2026).

Table of Content

How Rethinking Attention Led Me to SDA
How Signed Dual Attention Works
2.1 Mathematical Formulation
2.2 Link with Two-Head Attention
Future Directions

How Rethinking Attention Led Me to SDA

This work began with a critical reflection on the Transformer architecture, inspired by my ongoing research on Graph Neural Networks. Attention can, in fact, be viewed as a fully connected graph. From this perspective, the attention matrix can be interpreted as a learned adjacency matrix.

Given my background in statistics and classical time series analysis, one of the most intuitive adjacency matrices that came to mind was the correlation matrix. Indeed, in terms of raw formulation, the similarity matrix corresponds to the correlation matrix when variables are centered and standardized. However, attention does not operate directly on a similarity matrix—it applies a softmax transformation to it.

This distinction leads to a crucial behavioral difference. A correlation matrix used as an adjacency matrix can represent signed relationships, allowing information to propagate with both positive and negative influences depending on the sign of the correlation. In contrast, a standard attention mechanism only permits positive influence. Why does attention still work despite this limitation? Because multi-head attention allows the model to learn multiple types of relationships.

However, this observation motivated me to explore the modeling of signed relationships within a single attention head. Why bother? Because relying on multiple heads to circumvent this theoretical limitation introduces inefficiencies. When two attention heads are used to learn opposite relationships, the corresponding keys and queries governing the same interaction must be learned twice. Furthermore, in conventional architectures, the feature dimension is divided by the number of heads, resulting in more parameters but less expressive capacity per head. It is from this inefficiency that the idea emerged—to design a mechanism that models both aspects of a relationship within a single attention head, using shared parameters.

How Signed Dual Attention Works

To model signed relationships, both positive and negative weights in the attention matrix are required. Previous works have addressed this by applying a $\tanh$ transformation to the similarity scores. However, I believe that the softmax function plays an essential role in ensuring stability and effective focus during attention computation.

The solution I adopted was therefore not to replace the softmax, but to extend it: by applying two softmax functions and constructing an odd function of the similarity matrix, the model can naturally encode both positive and negative affinities while retaining the desirable normalization properties of softmax-based attention.

Mathematical Formulation

SDA extends standard scaled dot-product attention by computing positive and negative attention matrices with shared keys and queries matrices:

\[\begin{align} A^+ &= \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) \\ A^- &= \text{softmax}\left(-\frac{QK^\top}{\sqrt{d_k}}\right) \\ \end{align}\] \[\text{SDA}(Q,K,V) = (A^+ - A^-) V\]

$A^+$ propagates positive signals.
$A^-$ propagates negative signals.

The subtraction ensures antagonistic relationships invert the message.

Link with Two-Head Attention

The Signed Dual Attention (SDA) block can be seen as a constrained form of a two-head self-attention mechanism. Imagine a standard two-head attention layer with parameters defined as

\[(W_1^K, W_1^Q, W_1^V) = (W^K, W^Q, W^V), \qquad (W_2^K, W_2^Q, W_2^V) = (-W^K,\, W^Q,\, -W^V).\]

Let $H_1, H_2 \in \mathbb{R}^{T \times d}$ be the outputs of the two heads. In conventional multi-head attention, their concatenation $H_{\text{cat}} \in \mathbb{R}^{T \times 2d}$ is projected through an output matrix $W^O$ to obtain the final representation $H$ :

\[H = H_{\text{cat}} W^O.\]

If we choose

\[W^O = \begin{bmatrix} I_d \\ I_d \end{bmatrix} \in \mathbb{R}^{2d \times d},\]

the projection simply performs an additive fusion, leading to $H = H_1 + H_2$. Under this configuration, the output of the two-head mechanism aligns exactly with the SDA formulation.

This perspective highlights SDA as a compact and efficient alternative to multi-head attention—able to capture both supportive and antagonistic interactions while using half the parameter count and maintaining the computational footprint of a single-head layer.

Future Directions

Preliminary experiments suggest that SDA behaves differently across datasets depending on the underlying autocorrelation structure, particularly when both positive and negative dependencies coexist. This makes sense since by $W^O$ inforce both aspect of the relationship to have an equal influence, when it can depending of the data not be the case. I see potential in learning adaptive weighting between the positive and negative attention components $A^{+}$ and $A^{-}$ instead of assigning them equal importance. This enhancement could improve performance in settings where negative interactions are weak or primarily noisy.

If you would like to go further, you can read our paper, Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting, on arXiv. It presents the mathematical formulation and experiments behind SDA, exploring how attention can capture both positive and negative dependencies in time series without adding parameters.

Estimating Pi (π) with Monte Carlo Simulation in Python

2026-01-03T08:00:00+00:00

Introduction
Monte Carlo Simulation Basics
Using Random Sampling to Estimate Pi
A Python Implementation
Conclusion
References and Further Reading
Monte Carlo Interview Questions

Introduction

How can we estimate the value of $\pi$? While we know $\pi$ is irrational and cannot be expressed as a finite closed formula, approximations like $3.14$ or $22/7$ are widely used. But what if we want a systematic, simulation-based method to approximate $\pi$ with arbitrary precision?

Monte Carlo methods provide exactly this. By exploiting randomness and statistical principles, we can approximate $\pi$ in a surprisingly elegant way. This tutorial walks step by step through the mathematics and Python code.

Monte Carlo Simulation Basics

Monte Carlo methods rely on random sampling and the central limit theorem (CLT) to estimate expectations. Indeed the CLT states that for $(X_i)_{i=1}^n$ $n$ independent and identically distributed random variables :

\[\frac{1}{\sqrt{n}}\sum_{i=1}^{n}(X_i - \mu) \xrightarrow[]{d} N(0, \sigma^2)\]

If we can find a random variable $X$ such that $\mathbb{E}[X] = \pi$, then simulating $X$ repeatedly and averaging would us an estimate of $\pi$ ! The more samples we take, the closer we get to the true value, with the rate of convergence governed by the variance of $X$.

But this here is a quant blog. Suppose we want the absolute error to be less than $\varepsilon > 0$ with probability at least $1 - \alpha$. Can we solve this problem ? Answer is yes, but with some maths.

Let’s consider $\text{Var}(X) = \sigma^2$, then by the Central Limit Theorem, the sample mean

\[\hat{\pi}_n = \frac{1}{n} \sum_{i=1}^n X_i\]

satisfies

\[\sqrt{n}\,(\hat{\pi}_n - \pi) \xrightarrow[]{d} N(0, \sigma^2).\]

This means that for large $n$, the estimation error is approximately normal with variance $\frac{\sigma^2}{n}$. Hence, the standard error of our estimator is

\[\text{SE}(\hat{\pi}_n) = \frac{\sigma}{\sqrt{n}}.\]

Using the normal approximation, we require

\[\mathbb{P}\big( |\hat{\pi}_n - \pi| < \varepsilon \big) \approx 1 - \alpha.\]

This leads to the condition

\[\frac{\sigma}{\sqrt{n}} z_{1-\alpha/2} \leq \varepsilon,\]

where $z_{1-\alpha/2}$ is the $(1 - \alpha/2)$ quantile of the standard normal distribution (for example, $z_{0.975} \approx 1.96$ for a $95\%$ confidence level).

Rearranging gives the required number of runs:

\[n \geq \left( \frac{z_{1-\alpha/2} \cdot \sigma}{\varepsilon} \right)^2.\]

This formula provides a practical guideline: once we know or estimate the variance $\sigma^2$ of the Monte Carlo variable, we can determine how many simulations are needed to achieve a desired accuracy $\varepsilon$ with a given confidence level $1 - \alpha$.

Using Random Sampling to Estimate $\pi$

In practice, random number generation typically begins with pseudo-random draws from a uniform distribution, which can be transformed into other distributions. In Python, numpy provides a straightforward way to simulate such values.

Here is the key idea: let $X_1$ and $X_2$ be two i.i.d. uniform random variables on $[-1, 1]$. The probability that the point $(X_1, X_2)$ falls inside the unit circle is proportional to the ratio between the area of the circle and the area of the square:

\[\mathbb{P}(X_1^2 + X_2^2 \leq 1) = \frac{\text{Area of Circle}}{\text{Area of Square}} = \frac{\pi r^2}{(2r)^2} = \frac{\pi}{4}.\]

Thus, if we simulate many points uniformly in the square $[-1, 1]^2$ and check the fraction that lies within the unit circle, multiplying this fraction by 4 yields an approximation of $\pi$.

We can for sure also ask ourself : what is the value of $\sigma_X$?

By definition,
$\sigma_X = \sqrt{\mathbb{E}[X^2] - \mathbb{E}[X]^2}$

Here, $X$ is the indicator function of the event ${(X_1, X_2) \in \text{Unit Circle}}$. Since $X$ is an indicator, we have $X^2 = X$, thus :

\[\mathbb{E}[X^2] = \mathbb{E}[X] = \frac{\pi}{4}\]

And so we find :

\[\sigma_X = \sqrt{\frac{\pi}{4} - \left(\frac{\pi}{4}\right)^2} = \sqrt{\frac{\pi}{4} \cdot \frac{4 - \pi}{4}} = \sqrt{\frac{\pi (4 - \pi)}{16}}.\]

Thus,

\[\sigma_X < \frac{1}{2}.\]

A Python Implementation

The implementation is straightforward:

import numpy as np

n = int(1e6)  # number of simulations
x1 = np.random.uniform(-1, 1, n)
x2 = np.random.uniform(-1, 1, n)

# Check which points fall inside the unit circle
inside_circle = x1**2 + x2**2 <= 1

# Estimate of pi
pi_estimate = 4 * inside_circle.mean()
print(pi_estimate)

Running this code with one million points gives a fairly good approximation of $\pi$. Increasing n will further improve accuracy.

Conclusion

Estimating $\pi$ with Monte Carlo is a beautiful demonstration of how probability and geometry intersect. It shows that if we can identify a random variable with an expectation equal to the quantity of interest, then repeated sampling gives us a practical way to approximate it.

References and Further Reading

Monte Carlo Method — Wikipedia
Central Limit Theorem — Wikipedia
Gentle, J. E. (2003). Random Number Generation and Monte Carlo Methods. Springer.

Monte Carlo Interview Questions

1. What is a Monte Carlo simulation, and why is it useful in quantitative finance?

Monte Carlo simulation uses random sampling to approximate complex probabilistic outcomes. In finance, it's used for option pricing, risk management, portfolio optimization, and evaluating scenarios where closed-form solutions are difficult.

2. How does Monte Carlo simulation leverage randomness to approximate deterministic quantities like π?

By generating many random points in a known geometric space (e.g., a square), we can estimate the probability of events (e.g., falling inside a circle). Multiplying this probability by the appropriate factor gives an approximation of π.

3. What role does the Central Limit Theorem play in Monte Carlo methods?

The Central Limit Theorem justifies that the sample mean of independent simulations approximates a normal distribution for large samples, allowing us to compute confidence intervals and error bounds for estimates.

4. How do you determine the number of Monte Carlo simulations needed to achieve a desired accuracy?

Using the standard error formula: $ SE = \sigma / \sqrt{n} $, and desired error $\varepsilon$ at confidence level $1-\alpha$, the required simulations are $ n \ge (z_{1-\alpha/2} \cdot \sigma / \varepsilon)^2 $.

5. How can π be estimated using random sampling in a unit square?

Generate uniform random points in $[-1,1]^2$. Count the fraction inside the unit circle. Multiply this fraction by 4 to estimate π: $$ \pi \approx 4 \times frac{\text{points inside circle}}{\text{total points}} $$

6. What is the variance of the Monte Carlo estimator when estimating π?

Using the indicator variable for a point inside the circle: $ \sigma^2 = \frac{\pi}{4}\left(1 - \frac{\pi}{4}\right) $. The standard error of the mean decreases with $1/\sqrt{n}$.

7. How would you implement a Monte Carlo simulation to estimate π in Python efficiently?

Use vectorized operations in NumPy to generate random points and compute the fraction inside the unit circle. Example:


import numpy as np
n = 1_000_000
x, y = np.random.uniform(-1, 1, n), np.random.uniform(-1, 1, n)
pi_estimate = 4 * ((x**2 + y**2) <= 1).mean()
print(pi_estimate)

8. How can variance reduction techniques improve Monte Carlo estimation?

Techniques like antithetic variates, control variates, and quasi-random sequences reduce estimator variance, improving accuracy without increasing the number of simulations—a critical tool in hedge fund risk modeling.

9. Why is understanding the standard error important in quantitative simulations?

The standard error quantifies the expected fluctuation of the estimate around the true value, allowing quants to construct confidence intervals and assess the reliability of Monte Carlo results.

10. Where are Monte Carlo methods commonly applied in quantitative finance?

Applications include derivative pricing (e.g., options), Value-at-Risk (VaR) calculation, stress testing portfolios, scenario analysis, and simulating complex stochastic processes like interest rates or asset paths.

Is a Random Walk with Drift a Martingale ?

2025-11-12T00:00:00+00:00

As discussed in our previous blog post on martingales and random walks, both concepts are fundamental in financial mathematics and quantitative finance. In financial modelling, however, the most relevant case is the random walk with drift. While it may seem at first that a random walk with drift has little in common with a martingale, the relationship between them is deeper than it appears. In this article, we explore the connection between random walks with drift, martingales, and the stochastic discount factor, to build a clearer understanding of what they truly are.

Martingales, Random Walks and Drifts
From Drift to Martingale
Link with the Stochastic Discount Factor
Conclusion
Further Reading

Martingales, Random Walks and Drifts

Let’s make some definitions here to ensure we all speak the same language (thank God mathematics is universal).

Martingale

A martingale $W_t$ with respect to a filtration $\mathcal{F}_t$ satisfies :

\[\mathbb{E}\left[W_{t+1} \mid \mathcal{F}_t \right] = w_t \quad \forall t \ge 0\]

Intuitively, the best forecast of tomorrow’s value, given all information today, is simply today’s value.

Random Walk

A random walk $W_t$ is a discrete-time stochastic process defined by :

\[W_t = \sum_{i=1}^{t} \varepsilon_i,\]

where ${\varepsilon_i}_{i \ge 1}$ is a sequence of i.i.d. random variables and where in many cases :

\[\mathbb{E}[\varepsilon_i] = 0, \quad \text{and} \quad \text{Var}(\varepsilon_i) = \sigma^2.\]

In these settings, the process satisfies :

\[\mathbb{E}[W_{t+1} \mid \mathcal{F}_t] = W_t,\]

which and the random walk $(W_t)$ is a martingale.

Random Walk with Drift

However, financial modelling requires more often than not to have $\mathbb{E}[\varepsilon_i] = r$, with $r$ a non null constant. To adjust, we can use $\tilde{\varepsilon_i} = \varepsilon_i - r$ to define a new random walk with increments $\tilde{\varepsilon_i}$ that we will note $\tilde{W_t}$. From the relationship we can say that $\tilde{W_t} = W_t - r \times t$ or equivalently $W_t = \tilde{W_t} + r \times t$.

Thus introducing a non zero constant for $\mathbb{E}[\varepsilon_i]$ creates a random walk with drift and modifies the process into : $X_t = \mu t + W_t.$ This can be interpreted as a deterministic linear trend $\mu t$ plus a centered random walk $W_t$.

Quantitatively:

The expected value evolves as $\mathbb{E}[X_t] = \mu t$ showing the systematic accumulation of the drift.
The variance remains stochastic due to the random component : $\text{Var}(X_t) = t \sigma^2$

Hence, a random walk with drift is simply a martingale tilted by a linear deterministic trend $\mu t$.
In financial modeling, removing the drift by working under a risk-neutral measure recovers the martingale property crucial for pricing and arbitrage-free valuation.

import numpy as np
import matplotlib.pyplot as plt
np.random.seed(42)

N = 20          # number of sample paths
T = 2520        # number of time steps
mu = 0.05       # drift
sigma = 1       # volatility

eps = np.random.normal(mu, sigma, (T, N))
X = np.cumsum(eps, axis=0)  # random walk

expected_drift = np.arange(T) * mu

fig, ax = plt.subplots(figsize=(10, 6))
ax.plot(X, alpha=0.75,)
ax.plot(expected_drift, color='red', linestyle='--', linewidth=2, label='Expected Drift')

ax.set_title("Random Walk with Drift Simulations")
ax.set_xlabel("Time Steps")
ax.set_ylabel("Value")
ax.legend()

plt.tight_layout()
plt.show()

From Drift to Martingale

So far, we have described our random walks dynamics under a real-world measure, denoted by $\mathbb{P}$. Under $\mathbb{P}$, a random walk can exhibit a drift. When modelling financial instruments, this drift reflects required expected return for bearing risk. However it is far more convenient to work with martingales when computing expectations, the probability of a given event, … And that why mathematicals tools have been searched to transform random walk with drift into random walk : the change of measure, and more speciafically the risk-neutral measure in finance.

What is the key idea here ? Many real word applications require simply to compute an expectation. For a deriviative for instance, however the model used to describe the behavior of the underlying, Black Scholes for instance, we end up computed to expectation of the final payoff.

For any random real variable $X$, we can express it’s expectation as :

\[\mathbb{E}[X] = \int_{-\infty}^{\infty} f(x)x \mathrm{d}x\]

Changing the measure simply means finding a new density function $f’$ for $X$ such that it becomes a martingale. How is this done rigorously ? That is where the Radon Nikodym Bridge comes by.

The Radon–Nikodym Bridge

The Radon–Nikodym Theorem is the key demonstration behind everything that will follow. Using the Radon Nikodyn variable
$\Lambda_t = \frac{d\mathbb{Q}}{d\mathbb{P}}\Big|_{\mathcal{F}_t}.$ sometimes called the change-of-measure density, reweights probabilities such that expectations under $\mathbb{P}$ are transformed into expectations under $\mathbb{Q}$:

\[\mathbb{E}_{\mathbb{Q}}[X_t] = \mathbb{E}_{\mathbb{P}}[\Lambda_t X_t].\]

Let’s give an intuition for what will happend if use it to transform our random walk with a positive drift into a martingale. $\Lambda_t$ will downweight states of the world that are good and upweights those that are bad, effectively neutralizing the drift. A detailed exposition of this concept can be found in this post, which connects the Radon–Nikodym transformation with random walk intuition.

Ok, great, but you are now probability asking : Is that usefull maths or just some weird theory that will always be pointless ? Well, despite being nice and clever theorem, it does have it moment. Let’s look at a classic application.

A Practical Application

\[W_0 = 0, \quad W_{t+1} = W_t + \mu + \varepsilon_{t+1}, \quad \varepsilon_t \sim \text{i.i.d. } N(0,\sigma^2),\]

Let’s denote $\tau_a$ the hitting time of level $a > 0$ : $\inf{t \ge 0 : W_t \ge a}$.
We must compute the probability $p = \mathbb{P}(\tau_a < \infty)$ that the drifted random walk eventually reaches level $a$.

Consider $M_t = e^{-\theta W_t}$

where $\theta > 0$ is chosen such that :

\[\mathbb{E}[e^{-\theta (\mu + \varepsilon_t)}] = 1.\]

For $\varepsilon_t \sim N(0,\sigma^2)$:

\[\mathbb{E}[e^{-\theta (\mu + \varepsilon_t)}] = e^{-\theta \mu + \frac{1}{2}\sigma^2 \theta^2} = 1 \quad\Longrightarrow\quad \theta = \frac{2\mu}{\sigma^2}.\]

Then $(M_t)_{t=0}^{+\infty}$ is a positive martingale with $M_0 = 1$.

We can define a new probability measure $Q$ on $\mathcal{F}_n$ via

\[\frac{dQ}{dP}\Big|_{\mathcal{F}_n} = M_n\]

This is an exponential change of measure (Esscher transform). Under $Q$, the increments of the random walk become

\[\varepsilon_t^Q \sim N(-\sigma^2 \theta, \sigma^2),\]

so that the drift under $Q$ is

\[\mu - \sigma^2 \theta = \mu - 2\mu = -\mu < 0.\]

By the Radon–Nikodym formula:

\[P(\tau_a < \infty) = \mathbb{E}_P[\mathbf{1}_{\{\tau_a < \infty\}}] = \mathbb{E}_Q\Big[ \frac{dP}{dQ} \mathbf{1}_{\{\tau_a < \infty\}} \Big] = \mathbb{E}_Q\Big[ e^{\theta W_{\tau_a}} \mathbf{1}_{\{\tau_a < \infty\}} \Big].\]

Moreover, $W_{\tau_a} = a$, so

\[P(\tau_a < \infty) = e^{-\theta a}.\]

And finally :

\[P(\tau_a < \infty) = \exp\Big(-\frac{2\mu}{\sigma^2} a\Big)\]

Link with the Stochastic Discount Factor

In finance, the Stochastic Discount Factor (SDF), usually denoted $M_t$, plays exactly the same conceptual role. It is the core of assets pricing and the magic behind factors models (go see the post).

The SDF discounts risky future payoffs back to today’s value, adjusting for both time and risk, ensuring that random walks with drift become martingales. To gain more insight, check out our previous post about random walks.

Conclusion

When a stochastic process exhibits drift, its future values are partially predictable, and it is no longer a martingale. However, there exist systematic ways to transform a drifted process into a martingale, allowing us to restore the powerful martingale property for analysis and computation.

In the context of random walks, introducing a constant drift creates a random walk with drift, but through techniques such as the exponential martingale transform or a change of measure, we can construct a new process that removes the drift while preserving the essential probabilistic structure.

This transformation is not only a theoretical tool: in finance, it underpins the risk-neutral measure and the SDF, where discounted asset prices become martingales. By reweighting probabilities via the Radon–Nikodym derivative, we can consistently price assets without having to explicitly account for the real-world drift of returns.

The SDF Explained: Why Factor Models Actually Work

2025-09-27T00:00:00+00:00

Asset pricing has evolved dramatically over the past several decades, shifting from simple models based on a single market factor to an expansive zoo of multifactor frameworks. But beneath the growing complexity lies a deeper unifying idea: factor models are, at their core, a way to approximate the Stochastic Discount Factor (SDF). This post is not about which factors “work” or how to pick them, it’s about why factor models work at all. Drawing on insights from Cochrane (2005), we’ll explore how the empirical world of factor investing is fundamentally tied to the SDF.

A Historical Perspective
The True Concept Behind Asset Prices
Factor Models as SDFs in Disguise
Conclusion

A Historical Perspective

In the beginning, investors created the Market. The Market was one, with a clear form and simple measure. And the investors said: Let risk be priced. And risk was priced. And they called this the Capital Asset Pricing Model, born in 1964, the first of its kind.

But investors said: Let there be more than one risk, for markets are many and varied. And so in 1992 arose the Fama–French Three-Factor Model. And they called it the Threefold Model, for it spanned size, value, and the market itself. Thus were born factors, and with them the first risk portfolios.

And again the investors spoke: Let us make factors in our own image, many and diverse, to capture the world as we see it. And so began the great multiplication. From momentum in the 1990s, to profitability and investment in the 2010s, factors spread and multiplied, until there was a vast zoo of them, each claiming to price risk and explain return.

But despite the proliferation of models that price the market and explain the cross-section of returns, one central question remains: Where does all this actually come from? After all, divine metaphors aside, no one created the market from thin air with well structured properties and linear pricing properties. In this post, I want to explore one possible line of reasoning that helps explain why factor models work, not just how. My thinking here is largely influenced by the reading of Cochrane (2005).

SDF: The True Concept Behind Asset Prices

When first encountering factors, OLS estimators, shrinkage, and $t$-tests, it can feel rather abstract—especially if the end goal is making an investment decision. Yet one principle is more intuitive and foundational: the law of one price, which underlies most modern asset pricing theory.

The Law of One Price

The Law of One Price states that in an arbitrage-free market, two assets with identical cash flows must have the same price (G. Cassel (1918)). In practice, of course, there are frictions such as limited accessibility, bid-ask spreads, or information asymmetries. Nevertheless, it forms the backbone of asset pricing: two portfolios delivering the same payoffs should have the same price, and two bonds with identical cash flows and risk characteristics should offer the same yield.

From the Law of One Price to the SDF

This idea can be formalized mathematically through the stochastic discount factor (SDF), also called the pricing kernel Cochrane (2005). The SDF is a random variable $m_{t+1}$ that links future payoffs $X_{t+1}$ to current prices $P_t$:

\[P_t = \mathbb{E}_t \left[ m_{t+1} \, X_{t+1} \right]\]

Equivalently, for excess returns $r_{t+1}$ (returns over the risk-free rate), the SDF satisfies:

\[\mathbb{E}_t \left[ m_{t+1} \, r_{t+1} \right] = 0\]

Intuitively, this states that all assets are “fairly priced” given the SDF. The SDF is closely related to risk-neutral probabilities and can be thought of as a weighting factor that adjusts future payoffs to their present value while accounting for risk preferences. Go check this previous post if you want info on that.

The SDF in Practice

In theory, if we knew the distribution of $m_{t+1}$, pricing would be straightforward. This leads to the “everything is factored into the price,” which you can see on Benjamin Channel. But in reality, the SDF is unobservable, so we must approximate it. One approach could be Bayesian: treat the SDF as a latent random variable and update a prior distribution with market realizations over time.

In practice, however, finance has favored simpler approximations. Factor models, estimated via OLS regressions, can be interpreted as linear projections of the SDF onto observed returns. This is why factor models—CAPM, Fama–French, and beyond—capture so much pricing information: they approximate the true, unobservable stochastic discount factor. To known more about the Stochastic Discount Factor and what it’s implies in terms of risk neutral mesures and martingales, go check this previous post

Factor Models as SDFs in Disguise

At this point, you might be thinking: “What’s the link between factor models and the SDF?” If you already know where this is going, congratulations mate.

Estimating a factor model, whether it is CAPM, the Fama–French Five-Factor Model, or a custom model with hundreds of factors selected via LASSO, is, at its core, an SDF estimation problem. This was a revelation for me while reading John Cochrane’s Asset Pricing (Chapter 6).

From Factor Models to the SDF

Suppose we have a factor model:

\[r_{i,t+1} = \sum_{k=1}^K \beta_{ik} F_{k,t+1} + \epsilon_{i,t+1}, \quad i = 1, \dots, N\]

with factors $F_k$ and factor loadings $\beta_{ik}$. We want an implied SDF $m_{t+1}^{\text{impl}}$ that satisfies the SDF condition for all assets:

\[\mathbb{E}_t \left[ m_{t+1}^{\text{impl}} \, r_{i,t+1} \right] = 0 \quad \forall i.\]

Cochrane shows that we can construct it as a linear combination of the factors:

\[m_{t+1}^{\text{impl}} = 1 - \sum_{k=1}^K \lambda_k F_{k,t+1},\]

where the weights $\lambda_k$ solve the system:

\[\mathbb{E}_t \Big[ (1 - \sum_{k=1}^K \lambda_k F_{k,t+1}) \, r_{i,t+1} \Big] = 0, \quad i = 1, \dots, N.\]

In matrix notation, let $R$ be the $N \times 1$ vector of excess returns and $F$ the $N \times K$ factor matrix:

\[\mathbb{E}_t[R] = F \lambda \quad \implies \quad \lambda = (F^\top F)^{-1} F^\top \mathbb{E}_t[R].\]

This provides an explicit SDF estimate from any factor model. Every factor captures a dimension of risk, and the combination of factors approximates the true, unobservable SDF $M_t = 1-\lambda^\top F_t$. This is why factor models are powerful : they encode the pricing kernel in a tractable, estimable form.

Why Use Factor Models Instead of Directly Approximating the Stochastic Discount Factor?

If the true power of a factor model lies in its deep connection to the stochastic discount factor (SDF), why not bypass the factors entirely and approximate the SDF directly to predict future prices?

The idea might be to approximate the SDF $ M $ and then use the fundamental pricing equation $ P_t = \mathbb{E}[M_{t+1} \times P_{t+1}] $ to compute fair prices and identify whether an asset is over or undervalued. However, this approach isn’t as straightforward or as informative as it might seem.

For example, suppose $ M = 1 $ almost surely. Then you’re left with $ P_t = \mathbb{E}[P_{t+1}] $, which offers no new insight. This illustrates that simply knowing the distribution of the SDF is not enough.

What really matters when working with $\mathbb{E}[M_{t+1} R_{t+1}] = 0$ is understanding the joint distribution of $ (M_t, R_t) $ the stochastic discount factor and the asset returns together. This joint distribution is difficult to characterize directly.

Factor models offer a practical solution by capturing the sensitivity of returns to underlying risk factors. This sensitivity gives us a tractable way to understand how returns co-move with the SDF, essentially providing a handle on their joint behavior.

In other words, the key mathematical value of factor models is that they let us approximate the joint dynamics between $ M_t $ and $ R_t $ through the factors $ F $. Let’s see that with some equations :

\[\mathbb{E}_t[R_{t+1} \times M_{t+1}] = 0\]

Which implies

\[\mathbb{E}_t\big[R_{t+1} \times (1 - \lambda^\top F_{t+1})\big] = 0\]

And thus

\[\mathbb{E}_t[R_{t+1}] = \mathbb{E}_t[R_{t+1} F_{t+1}^\top] \lambda\]

When using the OLS estimator, the betas of the stock (an $ N \times K $ matrix) are defined as

\[B_{t+1} = R_{t+1} F_{t+1}^\top (F_{t+1} F_{t+1}^\top)^{-1}\]

Since

\[R_{t+1} F_{t+1}^\top = R_{t+1} F_{t+1}^\top (F_{t+1} F_{t+1}^\top)^{-1} (F_{t+1} F_{t+1}^\top),\]

We get :

\[R_{t+1} F_{t+1}^\top = B_{t+1} F_{t+1} F_{t+1}^\top\]

Noting :

\[\Sigma_{FF}^{t+1} = \mathbb{E}_t[F_{t+1} F_{t+1}^\top]\]

We end up with :

\[\mathbb{E}_t[R_{t+1}] = B_{t+1} \Sigma_{FF}^{t+1} \lambda\]

By starting with basic properties of the stochastic discount factor, we ultimately arrive at a pricing formula that depends only on our factor model. This is powerful because it allows us to shift the focus from estimating relationships across all $ N $ assets to working with a much smaller set of $ K $ factors.

Plenty of work must be then done on estimating $B_{t+1}$, $\Sigma_{FF}^{t+1}$ and $\lambda_{t+1}$. When $ K \ll N $, and assuming the factors are not highly correlated, the factor covariance matrix $\Sigma_{FF}^{t+1}$ is easier to estimate, invert, and work with in general.

This dimensionality reduction offers several practical benefits, including more stable estimates, reduced computational complexity, and better interpretability of risk-return trade-offs.

Conclusion

At first glance, factor models may look like clever statistical tools thrown at return data to extract patterns. But a deeper look reveals something more fundamental: they are not just predictive regressions, they are structured approximations of the Stochastic Discount Factor, the very object at the heart of modern asset pricing. This tells us why factor models work.

So next time you’re running a factor regression, remember: you’re not just crunching numbers, you’re approximating the invisible hand that prices all possible states of the economy.

Got thoughts, questions, or counterpoints? Drop a comment below. I’d love to hear your take.

Merger Arbitrage Explained

2025-09-21T00:00:00+00:00

Disclaimer

This post is for informational and educational purposes only. It does not constitute financial, investment, or trading advice. The author currently holds a position in Warner Bros Discovery (WBD) at the time of writing.

What Is Merger Arbitrage?
Market Prices and Implied Probabilities
A Simple Merger Arbitrage Methodology
Case Study: Warner Bros & Paramount Merger
- 4.1 Deal Overview
- 4.2 Price-Based Probability Estimation
- 4.3 Sensitivity Analysis
Risks and Considerations
Building an Arbitrage Framework
Conclusion

What Is Merger Arbitrage

I’ve held a large position in Warner Bros Discovery (WBD) since 2023, so when news broke of a potential takeover by Paramount/Skydance, it naturally caught my attention. This post is partly motivated by that position, and more broadly by the opportunity to walk through how merger arbitrage can be evaluated when a real-world event unfolds.

Let’s walk through it.

Merger arbitrage is an event-driven investment strategy built around one simple idea: when a company announces it’s acquiring another, there’s a temporary pricing inefficiency and that create an opportunity. Here’s how it works.

When an acquisition is announced, the acquirer offers to buy the target company, usually at a premium. For example, they might offer 24\$ per share for a company that was trading at 15\$. The target’s stock typically jumps on the news… but not all the way to 24$. Why not? Because there’s still risk: the deal might fall through, get delayed, or be blocked by regulators. So the market applies a discount, pricing in the possibility of failure.

This gap between the current market price of the target and the deal price is called the spread. Merger arbitrageurs make bets on it to make an arbitrage.

In a pure cash deal, the arbitrage setup is straightforward: you buy the target company’s shares after the deal is announced and wait. If the deal closes, the acquirer pays cash, and you pocket the difference. If the deal breaks, the stock usually drops, sometimes sharply, and you lose money. So the whole game is about probabilities and payoffs. In this post I will only talk about this kind of deal.

The wider the spread, the more the market is signaling uncertainty. A narrow spread? The market sees a high chance of the deal going through. A wide spread? More risk or at least, more perceived risk.

In some cases, the spread can even go negative, the stock trades above the offer price. That usually means investors are betting on a bidding war or a sweetened deal.

Merger arbitrage isn’t about guessing whether a stock will go up or down. It’s about evaluating a specific corporate event, estimating the probability of success, and judging whether the potential return justifies the risk. It’s closer to probabilistic thinking than traditional investing.

It’s not glamorous. It’s not momentum-driven. But when done systematically with proper sizing, scenario modeling, and discipline it can be a effective strategy (here a first article and a second one documenting this assertion).

Market Prices and Implied Probabilities

When a public company announces it’s being acquired, the offer is almost always above the last trading price, often significantly. That premium is what makes merger arbitrage possible.

Let’s work on a simple example. Imagine a small public company : let’s call it Banana. It is trading at 10\$ per share. One morning, Apple announces it wants to acquire that company for 20\$ a share, in cash. That’s a 100% premium.

What happens to the stock? It jumps, but not to 20$. Maybe it shoots up to 18\$ or 19\$. That difference between the offer price 20\$ and the new trading price (say, 18\$) is called the spread. And it contains valuable information.

The Spreak is the Market-Priced Deal Risk

If the market believed with 100% certainty that the deal would close exactly as proposed, the stock would immediately trade at 20\$. But that almost never happens. There’s always some risk : antitrust issues, financing problems, board pushback, political pressure, or just the chance that the acquirer changes its mind.

So the market prices in that uncertainty. In our example, the 2\$ gap between 18\$ and 20\$ is how the market discounts the possibility that the deal might not happenn or might happen later, or on worse terms.

From that spread, we can actually estimate the implied probability of the deal closing.

A Simple Formula

We can model the current price as a weighted average of two outcomes:

The deal closes, and the target gets 20\$
The deal fails, and the target falls back to its pre-deal price (say, 10\$)

We can write this as:

\[\text{Market Price} = p \times \text{Deal Price} + (1 - p) \times \text{Fallback Price}\]

Where:

$p$ is the implied probability of deal success
Deal Price is the offer amount (e.g. 20\$)
Fallback Price is the estimated value if the deal fails (e.g. 10\$)

Solving for $p$:

\[p = \frac{\text{Market Price} - \text{Fallback Price}}{\text{Deal Price} - \text{Fallback Price}}\]

Plug in the numbers from our example:

\[p = \frac{18 - 10}{20 - 10} = \frac{8}{10} = 0.80\]

So, in this case, the market is pricing in an 80% chance that the deal will close as announced.

Merger Arbitrage Methodology

The implied probability isn’t just academic : it’s your decision-making compass.

If you’re looking at a potential merger arbitrage setup, here’s the process in plain terms:

Step 1: Understand what the market is pricing in.
The current spread between the offer price and the trading price reflects the market’s best guess at the probability the deal will close. Not with certainty, but under prevailing assumptions: risk, timing, regulatory pressure, etc.
Step 2: Decide whether you agree.
This is the core of the strategy. You’re not predicting prices — you’re evaluating scenarios. Is the market underestimating the chance of success? Are people missing a regulatory blocker? Has the risk already been overstated?
Step 3: Think in outcomes, not absolutes.
There are usually two (or more) outcomes: the deal closes, or it doesn’t. Each comes with a probability, a timeline, and a payoff. Your goal is to model the expected value (EV) of the trade, given those inputs.

The Expected Value Framework

Let’s break it into a simple two-outcome model:

Deal Price (D): The price paid if the acquisition closes
Current Price (C): What the target trades at now
Fallback Price (F): Estimated value if the deal fails
Probability of Success (p): Your estimate (or market’s implied)

Then your expected value (EV) per share is:

\[\text{EV} = p \times D + (1 - p) \times F\]

If EV is meaningfully above the current price (after adjusting for time and risk), it’s potentially a good trade. If not, you pass.

A Quick Example

Suppose:

Deal price = 24\$
Current stock price = 19\$
Fallback (if deal fails) = 12\$

Using the formula to back out the implied probability:

\[p = \frac{19 - 12}{24 - 12} = \frac{7}{12} \approx 58\%\]

Now suppose you think, because you are an expert in the industry for example, that the chance of success is more like 70%. Plug it in:

\[\text{EV} = 0.70 \times 24 + 0.30 \times 12 = 16.8 + 3.6 = 20.4\]

The expected value under your assumptions is 20.40\$, which is above the current price 19\$. That suggests a potential edge if your assumptions are more accurate than the market’s. When making a are more complex model, you should take into account also the time required for the deal to go trough, and think more into target and fallback distributions than fixed parameters.

Warner Bros & Paramount Merger

Deal Overview

According to a New York Post report (Sep 19, 2025), Skydance Media, which recently acquired control of Paramount Global, is now exploring a bid to acquire Warner Bros Discovery at a price in the range of 22\$–24\$ per share.

This isn’t a formal offer (yet), but it’s enough to move the market. Want to see it ? Here the Warner Bros stock price over the last days (and bonus the Python code to do it at home) :

import yfinance as yf
import matplotlib.pyplot as plt
import datetime

# Fetch WBD price data
ticker = yf.Ticker("WBD")
start_date = "2025-06-01"
end_date = "2025-09-21"
data = ticker.history(start=start_date, end=end_date)

# Plot
plt.figure(figsize=(10, 5))
plt.plot(data.index, data["Close"], label="WBD Close Price", color="navy")
plt.axvline(datetime.datetime(2025, 9, 10), color='red', linestyle='--', label="Deal Rumor (Sep 19)")
plt.title("WBD Stock Price – June to September 2025")
plt.xlabel("Date")
plt.ylabel("Price ($)")
plt.legend()
plt.grid(True)
plt.tight_layout()
plt.show()

Analyst reactions have been mixed: analysts from TD Cowen have downgraded WBD from Buy to Hold, arguing that without a deal, the stock could fall back to 11\$–12\$, based on fundamentals. (MarketWatch, Sep 20, 2025)

How to get info like that easily at home ? Here again python is your friend. Here a small code enabling you to get recent headlines on any ticker :

import yfinance as yf
from datetime import datetime

# Fetch the lastest news article for Warner Bros Discovery (WBD)
ticker = yf.Ticker("WBD")
news_items = ticker.get_news(count=1)

# Format and print the news like an RSS reader
item = news_item['content']
title = item["title"]
published = item["displayTime"]
link = item["canonicalUrl"]['url']
summary = item['summary']
    
print(f"📌 {title}")
print(summary)
print(f"🔗 {link}\n")

📌 Wells Fargo Raises Warner Bros. Discovery (WBD) Price Target, Sees M&A Potential
Warner Bros. Discovery Inc. (NASDAQ:WBD) ranks among the best communication services stocks to buy now.
Wells Fargo boosted its price target for Warner Bros. Discovery Inc. (NASDAQ:WBD) to 14$ from 13$ on
September 11, retaining an Equal Weight rating on the company’s shares. The revision reflects
Wells Fargo’s belief that Warner Brothers Discovery’s Studios and […]
🔗 https://finance.yahoo.com/news/wells-fargo-raises-warner-bros-075936106.html

So what is the market currently pricing in? Let’s quantify it.

Price-Based Probability Estimation

Let’s plug in some numbers based on the reporting and price action:

Deal price (X): We will use 24\$
Current price (Y): 19\$ (WBD has traded between 18\$–20\$ since the rumor broke)
Fallback price (B): 11\$ (where analysts estimate WBD would trade without a deal)

Using the implied probability formula:

\[p = \frac{Y - B}{X - B} = \frac{19 - 11}{24 - 11} = \frac{8}{13} \approx 61.5\%\]

So based on this spread, the market is implying roughly a 61–62% chance that the deal goes through as rumored.

Again using yfinance and Python you can easily look at the evolution of the merger probability (under this parameters of deal price and fallback of course) :

import yfinance as yf import matplotlib.pyplot as plt

# Fetch WBD price data
ticker = yf.Ticker("WBD")
start_date = "2025-06-01"
end_date = "2025-09-21"
data = ticker.history(start=start_date, end=end_date)

deal_price = 24  # Assumed acquisition price
fallback_price = 11  # Assumed fallback price

data["Implied_Probability"] = (data["Close"] - fallback_price) / (deal_price - fallback_price)

plt.figure(figsize=(10, 4))
plt.plot(data.index, data["Implied_Probability"], label="Implied Deal Probability", color="purple")
plt.axhline(0.5, color='gray', linestyle='--', alpha=0.7)
plt.title("Market-Implied Probability of WBD Acquisition")
plt.xlabel("Date")
plt.ylabel("Probability")
plt.ylim(0, 1)
plt.legend()
plt.grid(True)
plt.tight_layout()
plt.show()

Let’s break down the potential payoff structure:

Upside if deal closes: 5\$ per share → about +26.3% return from 19\$ to 24\$
Downside if deal fails: Loss of 8\$ per share → about –42.1% from 19\$ to 11\$

If the deal closes in 6 months, that’s a 26% return in half a year, or over 50% annualized (not adjusting for taxes or compounding). But this return is priced in. Meaning the risk of dowside exist and is priced. The question you must ask yourself in this situation is more : is the market probability near the truth ? ie is the market efficient ?

Sensitivity Analysis

The final step to fully understand our model is asking : how sensitive are we to changes in assumptions ? Let’s run a few scenarios.

What if the fallback increase ?

Suppose the market is wrong about the downside, and the true fallback price is 14\$, not 11\$. Then:

\[p = \frac{19 - 14}{24 - 14} = \frac{5}{10} = 50\%\]

So in this more optimistic view, the implied probability drops to 50%, because the downside isn’t as severe.

What if the deal price decrease ?

What if the deal gets negotiated down to 22\$ ? Then:

\[p = \frac{19 - 11}{22 - 11} = \frac{8}{11} \approx 72.7\%\]

Ironically, a lower offer price results in a higher implied probability — because the spread narrows.

This shows that the probability estimate is sensitive to both inputs: the upside and the fallback. What if we want to look at it in more scalable fashion ?

Getting the full picture

The 24\$ dollars per share is maybe a good payoff guess, but what about the fallback price ? Warner Bros Stock has move from 8 to 14\$ regularly over the last 2 years. Again (and for the last time) we can use Python to study the sensitivity to this fallback price. What we want to see is the implied probability.

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

# Set deal price
deal_price = 24
# Define fallback and market price ranges
fallbacks = np.arange(8, 12.5, 0.5)  # $8 to $12
prices = np.arange(15, 23.5, 0.5)    # $11 to $24

# Create a matrix to store implied probabilities
spread_matrix = np.zeros((len(fallbacks), len(prices)))

# Compute implied probabilities
for i, b in enumerate(fallbacks):
    for j, y in enumerate(prices):    
        spread_matrix[i, j] = (y - b) / (deal_price - b)
    
# Clip to range [0, 1] for clarity
spread_matrix = np.clip(spread_matrix, 0, 1)
spread_matrix = spread_matrix[::-1, :]
# Plot heatmap
plt.figure(figsize=(12, 6))
sns.heatmap(spread_matrix, annot=True, fmt=".2f", cmap="YlGnBu",
            xticklabels=prices, yticklabels=fallbacks[::-1],
            cbar_kws={'label': 'Implied Probability'})

plt.xlabel("Market Price ($)")
plt.ylabel("Fallback Price ($)")
plt.title("Implied Probability Sensitivity — WBD Deal at 24$")
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

What Makes This Deal Tricky?

Beyond price modeling, this deal carries significant regulatory risk.

According to a Columbia Journal of Law & the Arts article, a Paramount–WBD merger would likely trigger scrutiny under the 2023 DOJ/FTC Merger Guidelines. The combined entity would have substantial market share in theatrical distribution, streaming, and television possibly pushing it into “highly concentrated” territory by antitrust standards.

Key regulatory concerns:

Horizontal overlap in film studios and content production
Vertical integration risks (content + distribution)
Media plurality concerns (CNN + CBS News under one roof)
Public interest tests from the FCC (because of news media and licenses)

So even if Skydance makes a formal offer, the path to closing could be long and far from guaranteed.

Where We Are Now

To sum up:

The market is pricing in ~60% chance of a 24\$ share deal based on current prices and downside estimates
Regulatory headwinds are real and potentially deal-breaking
If you think the market is too cautious, there may be opportunity
If you think the market is too optimistic, it might be time to hedge or trim

The rest depends on how you model risk and whether you trust your estimates more than the market.

Conclusion

The Warner Bros Discovery–Paramount/Skydance case offers a real-time example of how markets digest deal rumors, price in uncertainty, and reflect a collective estimate of risk, all through a single number: the stock price.

This case illustrates a few key points:

Markets imply probabilities through price — not opinion. While analysts debate whether WBD is worth 11\$ or 14\$ without a deal, the stock price incorporates not just that debate, but the likelihood of various outcomes.
Implied probability is not truth : it’s just what the current prices suggest. If you have a differentiated view on the deal’s odds or timing, that’s where alpha can live.
A simple framework goes a long way. By mapping out:
- the potential deal price (upside)
- the fallback value (downside)
- the timeline to closing
- the regulatory and strategic context …you can structure a probabilistic view of the trade, rather than just a gut feeling.

This isn’t forecasting. It’s scenario modeling and it forces clarity of thought.

What we’ve walked through here is a first-principles framework: pricing, spreads, probabilities, payoffs.

But more complex models are absolutely possible:

Multi-scenario trees, with branching outcomes (e.g. competing bids, revised terms, partial deals)
Bayesian updates as new information arrives (regulatory signals, earnings calls, leaks)
Monte Carlo simulations to model a range of regulatory, political, or timing variables
Option-based models to account for volatility in both upside and fallback prices

You don’t need these tools to start — but as the stakes grow, or as you allocate more capital to merger arbitrage, adding depth to your modeling can help improve risk management and return profile.

What’s your take ? Is the market underestimating the regulatory risk here ? Or is this a misunderstood opportunity ? Let’s talk bellow !

Anaconda to Create a Clean Python Environment

2025-09-20T00:00:00+00:00

Setting up a clean and stable Python environment is essential for learning and working on projects. This guide will show you how to use Anaconda to do just that.

🐍 What is Anaconda?

Anaconda is a free Python distribution that includes:

Python interpreter
Popular libraries like NumPy, pandas, matplotlib
Jupyter Notebook
conda environment manager

It’s ideal for students and beginners because it simplifies installation and avoids conflicts between projects. It is compatible with VS CODE ide. An official tutorial exist here.

🔧 Step 1: Install Anaconda

Download Anaconda:
👉 https://www.anaconda.com/download
Choose your OS and download the Python 3.x (64-bit) version.
Run the installer and follow the prompts:
- (Optional) On Windows, check “Add Anaconda to my PATH”
- Select “Just Me” and complete installation.

🧪 Step 2: Create a New Conda Environment

Open Anaconda Prompt (Windows) or Terminal (macOS/Linux):

conda create -n myproject python=3.10

📂 Step 3: Activate Your Environment

After creating the environment, you need to activate it:

conda activate myproject

📚 Step 4: Install Packages

Use conda install to install packages:

conda install numpy pandas matplotlib jupyter

If a package isn’t available in the default conda channel, you can try conda-forge:

conda install -c conda-forge scikit-learn

Or even use pip inside the environment:

pip install seaborn

Now any Python packages you install will be isolated from your base installation.

💻 Step 5: Use the Environment in VS Code

Anaconda integrates smoothly with VS Code. Once your environment is created, you can use it for both Jupyter notebooks and Python scripts.

📝 For Notebooks (.ipynb)

Open VS Code.
Install the extensions:
Python (by Microsoft)
Jupyter (by Microsoft)
Open a .ipynb file or create a new notebook.
In the top-right corner, select the kernel → choose your conda environment (myproject).

Now, your notebook cells will run inside the clean conda environment.

📄 For Python Scripts (.py)

Open a .py file in VS Code.
Look at the bottom-right corner of VS Code → click on the interpreter selector.
Choose the interpreter from your environment (it should appear as something like Python 3.10 (‘myproject’)).

Once selected, running scripts (Run → Run Without Debugging or the green button) will execute in your conda environment.

Random Walk vs Martingale: What’s the Difference

2025-09-12T00:00:00+00:00

Finance students often encounter both random walks and martingales in their studies. At first sight, the two concepts look almost the same : both can be applied to describe the process, that “do not drift on average”. But both concepts are not identical. Understanding the distinction is crucial.

In this article, we’ll clarify the concepts, illustrate them with Python simulations, and explain why the distinction matters in finance.

Random Walk: A Simple Model of Prices
Martingale: A Broader Mathematical Concept
Random Walk vs Martingale: Key Differences
Conclusion
Further Reading

What is a Random Walk in Finance?

A random walk is one of the simplest stochastic processes. In discrete time, it is defined as:

\[X_{t+1} = X_t + \epsilon_{t+1}\]

where $\epsilon_{t+1}$ are independent and identically distributed (i.i.d.) shocks, often assumed to be Gaussian with mean zero and variance $\sigma^2$. The noise distribution can however take different for, including distribution with heavy tails.

Intuition

At each step, the process moves up or down randomly, with no predictable trend. The past brings no information to the future.

Financial interpretation

If stock prices followed a random walk, tomorrow’s price would equal today’s price plus a random shock. This aligns with the Efficient Market Hypothesis : if markets are efficient, new information arrives randomly, and so do price changes.

Python example

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(42)

N, T = 20, 252
eps = np.random.normal(0, 1, (T, N))
X = np.cumsum(eps, axis=0)  # random walk

plt.plot(X)
plt.title("Random Walk Simulations")
plt.show()

What is a Random Walk in Stock Prices?

A martingale is defined in probability theory as a process $X_t$ such that:

\[\mathbb{E}[X_{t+1} \mid \mathcal{F}_t] = X_t\]

where $\mathcal{F}_t$ is the information available up to time $t$.

Intuition

The conditional expectation of tomorrow’s value, given today’s information, is just today’s value. In other words, knowing the past gives you no edge in predicting the future.

Key difference from random walk

A martingale is defined by a conditional expectation property under a probability measure, when a random walk is model of the behavior of $X_t$. This makes the martingale a far more general concept.

Financial Interpretation: The Role of the SDF and Change of Measure

Under the real-world probability measure $\mathbb{P}$, a stock price is generally not a martingale due to the presence of risk premia—investors demand compensation for bearing risk. However, in risk-neutral pricing, we use a different probability measure, called the risk-neutral measure $\mathbb{Q}$, under which discounted asset prices become martingales. This makes the concept universal.

What is the risk neutral probability

This probability transformation is made possible by the Stochastic Discount Factor (SDF), also known as the pricing kernel. The SDF captures how investors value future payoffs relative to today, adjusting for both time value and risk preferences.

If we consider no dividends from $t$ to $t+1$ the connection is formalized as:

\[\forall i \in \Omega : P_t^i = \mathbb{E}^{\mathbb{P}}_t \left[ m_{t+1} \cdot P_{t+1}^i \right]\]

Here, $m_{t+1}$ acts as a weighting function that adjusts future payoffs based on states of the world, giving less weight to “good” states (where marginal utility is low) and more to “bad” states (where it’s high).

But how is this possible ? Let’s define the Radon-Nikodym derivative:

\[\begin{equation} \frac{d\mathbb{Q}}{d\mathbb{P}} = \frac{m_{t+1}}{\mathbb{E}^{\mathbb{P}}[m_{t+1}]} \tag{1} \end{equation}\]

using this ratio we can shift from the real-world measure to the risk-neutral measure:

\[\begin{aligned} \mathbb{E}^{\mathbb{P}}_t \!\left[ m_{t+1}\, P_{t+1}^i \right] &= \mathbb{E}^{\mathbb{P}}_t \!\left[ \frac{d\mathbb{Q}}{d\mathbb{P}} \; \mathbb{E}^{\mathbb{P}}[m_{t+1}] \; P_{t+1}^i \right] \\[1em] &= \mathbb{E}^{\mathbb{P}}_t \!\left[ \frac{d\mathbb{Q}}{d\mathbb{P}} \, P_{t+1}^i \right] \, \mathbb{E}^{\mathbb{P}}[m_{t+1}] \\[1em] &= \mathbb{E}^{\mathbb{Q}}_t \!\left[ P_{t+1}^i \right] \, \mathbb{E}^{\mathbb{P}}[m_{t+1}]. \end{aligned}\]

Using equation (1) we can apply it to the risk free asset such that wih return $e^{r}$:

\[\mathbb{E}^{\mathbb{P}}[m_{t+1}] = e^{-r}\]

We end up with :

\[P_t = e^{-r} \mathbb{E}^{\mathbb{Q}}_t \left[ P_{t+1}^i \right]\]

If we use $e^{rt} P_t$ as the discounted price, we get

\[e^{-rt} P_t^i = e^{-r(t+1)} \mathbb{E}^{\mathbb{Q}}_t \left[ P_{t+1}^i \right]\] \[e^{-rt} P_t^i = \mathbb{E}^{\mathbb{Q}}_t \left[ e^{-r(t+1)} P_{t+1}^i \right]\]

Thus if we note $\tilde{P}_t^i = e^{rt} P_t^i$ we have :

\[\mathbb{E}^{\mathbb{Q}}_t[ \tilde{P}_{t+1}^i] = \tilde{P}_t^i\]

In this new measure $ \mathbb{Q} $, the discounted price of any asset is a martingale, meaning its expected future value (adjusted for time) equals its current price.

This is the mathematical foundation of modern asset pricing and derivatives valuation: we price assets as if investors are risk-neutral, using a change of measure justified by the SDF, even though in reality they are not.

Random Walk vs Martingale: Key Differences

Feature	Random Walk	Martingale
Definition	$X_{t+1} = X_t + \epsilon_{t+1}$, iid shocks	$\mathbb{E}[X_{t+1} \mid \mathcal{F}_t] = X_t$
Distribution	Specific (often Gaussian)	Any distribution, as long as conditional expectation holds
Predictability	No trend, but variance accumulates	No conditional drift; more general
Finance usage	Descriptive model of price series	Mathematical framework for pricing, based on no-arbitrage arguments

When is Martingale not a Random Walk ?

The key difference between a random walk and a martingale process lies in the distributional assumptions. A random walk assumes i.i.d. (independent and identically distributed) increments, while a martingale only requires that the conditional expectation of tomorrow’s value equals today’s value.

To see this distinction, consider the following stochastic process:

\[X_{t+1} = X_t + X_t\epsilon_{t+1}\]

where $\epsilon_{t+1}$ is a zero-mean random shock.

This process is a martingale, because
$\mathbb{E}[X_{t+1} \mid \mathcal{F}_t] = X_t.$
But it is not a random walk, since the increments depend on the current level $X_t$ rather than being i.i.d.

This example also highlights why log prices are often modeled instead of raw prices. Taking logs, we get:

\[\log(X_{t+1}) = \log\!\big(X_t(1+\epsilon_{t+1})\big) = \log(X_t) + \log(1+\epsilon_{t+1}).\]

Now the process $\log(X_t)$ evolves as a random walk with innovations $\log(1+\epsilon_{t+1})$, which are i.i.d. under suitable conditions.

Python illustration

# Simple martingale simulation
X = [10]  # start at 0
for t in range(1, 252):
    # steps are fair, zero expected change
    X.append(X[-1] + np.random.choice([-0.1, 0.1])*X[-1])  

plt.plot(X, label="Martingale-like Process")
plt.title("Martingale Simulation")
plt.legend()
plt.show()

Conclusion

Random walks and martingales may seem similar at first glance, but the distinction is subtle and important. Random walks describe how prices move step by step, while martingales capture a broader property of fair game conditional expectations. Understanding both is fundamental to quantitative finance, stochastic modeling, and financial mathematics. To see a pratical use of random walk models and martingales properties to forecast volatility, check this previous post !

Bitcoin Volatility Estimation with the Parkinson Estimator in Python

2024-09-27T00:00:00+00:00

Introduction
The Parkinson’s Range-Based Method
Implementation on Bitcoin’s Data
Considerations for Practical Use
Conclusion
References and Further Reading

The Parkinson Estimator for Bitcoin Volatility

In this third installment of our series on Bitcoin volatility, we delve into a new volatility estimator, this time based on High and Low data points. This approach not only leverages high and low price points but also offer more precise volatility estimations. To fetch the data, check out this previous post, where we explored how to use the Binance API.

Small recap of the previous articles of this serie :

Part 1 article introduced the basic concepts of volatility, focusing on historical volatility calculations using closing prices from Binance data and the EWMA estimator.
Part 2 article expanded on these concepts by examining the use of econometric model (GARCH) to modelize complex behaviours (ARCH effect).

These analyses provided a groundwork for understanding the complexities of Bitcoin’s price dynamics and set the stage for integrating more sophisticated statistical techniques.

Building on our previous discussions, this article aims to Explore Alternative Data: and Alternative Volatility Estimators that utilize not the close (C) prices but the high (H), low (L).

The estimator we’ll discuss is not my original creation; its roots trace back to Parkinson, who first introduced it to the community in 1980. Since the formal proof has been established over four decades ago, my aim here isn’t to rehash the theory but rather to provide a clear, step-by-step guide to its construction. This tutorial will walk you through the essential concepts, ensuring you gain a practical understanding of how this estimator works and how it can be implemented effectively.

By the end of this post, you’ll gain a deeper understanding of how a volatility estimator can be constructed, making it a valuable guide for quantitative researchers.

The Parkinson’s Range-Based Method

What steps should a quant researcher follow when searching for an estimator ?

Objective: Construct a volatility estimator based on High and Low observations

Modelling: For this post, we will assume the asset price ( P_t ) follows a geometric Brownian motion (I will conceive not being original here).

\[dP_t = \mu P_t \, dt + \sigma P_t \, dW_t\]

where $ \mu $ is the drift, $ \sigma $ is the constant volatility, and $ W_t $ is a Wiener process.

Intuition: Our key assumption here is that the expected logarithmic range ($R$) of the asset should be proportional to its standard deviation.

\[R = \ln \left( \frac{\max_{0 \leq t \leq T} \ln P_t}{\min_{0 \leq t \leq T} \ln P_t}\right) = \ln \left( \frac{M_T}{m_T} \right)\]

Methodology: The standard approach to constructing an estimator typically involves identifying a random variable whose expected value is connected to the theoretical quantity we want to estimate. From there, we leverage the Central Limit Theorem:

\[\frac{1}{\sqrt{n}}\sum_{i=1}^{n}(X_i - \mu) \xrightarrow[]{d} N(0, \sigma^2)\]

Where $ X_i $ are independent and identically distributed (i.i.d.) random variables, with $ E[X_i] = \mu $ and $ \text{Var}(X_i) = \sigma^2 $, and $ n $ is the sample size.

The central limit theorem ensures that as $ n \to \infty $, the sum of the random variables converges in distribution to a normal distribution.

Now that we have the variable $R$, whose expected value should be related to $ \sigma $, the next step is to formally prove this through some mathematical derivations.

The Chicken and the Knife: As a former physics teacher of mine used to say about thermodynamics equations: now you have the knife in your hand, and the chicken is right in front of you. Time to get to work! There’s no simple trick to computing $ E\left[ R \right] $ in this case.

Here, we’re working with an expectation, and as probabilists, we naturally express this as an integral. To move forward, we need the probability distribution of the range, which, luckily, can be derived from the joint distribution of the minimum and maximum values along the paths.

In his 1951 paper, Feller outlines the necessary steps to compute this joint probability distribution. The key steps are as follows:

Step 1: Express the joint distribution in terms of simpler events. The first idea is to express the probability that $ M_t $ and $ m_t $ are below certain levels, say $ M_t \leq a $ and $ m_t \geq b $. This is the probability that the process stays between $ b $ and $ a $ over time $ t $.
Step 2: Use reflection symmetry. A Wiener process has a useful symmetry property: If we reflect the path at any point (like a mirror), the reflected path is also a valid Wiener process. This symmetry can simplify the analysis of the maximum and minimum because what happens above zero is mirrored below.
Step 3: Break the problem into manageable parts. The joint probability of reaching a maximum $ M_t $ and a minimum $ m_t $ can be viewed as a combination of two probabilities:
1. The probability that the process doesn’t cross a certain boundary.
2. The probability that the process reaches a given level for the first time (like hitting a particular maximum or minimum).
Step 4: Utilize known distributions. The distribution of the maximum $ M_t $ is known to follow a specific distribution (related to the normal distribution of the Wiener process). Similarly, the minimum can be handled by symmetry.

We then get the folowing cumulative probability distribution, taking the same notations as 1980 Parkinson’s article :

\[P(R \leq x) = \sum_{n=1}^{\infty} (-1)^{n+1} n \left\{ \text{Erfc} \left( \frac{(n+1)x}{\sqrt{2\sigma}} \right) - 2 \, \text{Erfc} \left( \frac{nx}{\sqrt{2\sigma}} \right) + \text{Erfc} \left( \frac{(n-1)x}{\sqrt{2\sigma}} \right) \right\}\]

For clarity sake, we have here consider $ T = 1 $. Using this, Parkinson (1980) calculates (for $ p \geq 1 $):

$E(R^p) = \frac{4}{\sqrt{\pi}} \Gamma \left( \frac{p + 1}{2} \right) \left( 1 - \frac{4}{2^p} \right) \zeta (p - 1) 2\sigma^2 \tag{10}$ where $ \Gamma(x) $ is the gamma function and $ \zeta(x) $ is the Riemann zeta function.

Particularly, for $ p = 2 $:

\[E(R^2) = 4 \ln(2) \sigma^2\]

Creating the Estimator: We have finally established the desired relationship, linking $ R $ and $ \sigma $. The final step is to apply the central limit theorem to create an estimator for $ \sigma $, considering we have multiple observations.

Suppose we have $ n $ independent observations of the range, denoted by $ R_1, R_2, \dots, R_n $. Using the fact that $ E(R^2) = 4 \ln(2) \sigma^2 $, we can derive an unbiased estimator for $ \sigma^2 $ based on these observations.

The estimator for $ \sigma^2 $ is:

\[\hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^{n} \frac{R_i^2}{4 \ln(2)} = \frac{1}{4 n \ln(2)} \sum_{i=1}^{n} \ln \left(\frac{H_i}{L_i}\right)\]

By the central limit theorem, as the sample size $ n $ increases, the estimator $ \hat{\sigma}^2 $ converges to the true variance $ \sigma^2 $, and its distribution approaches normality. Let’s get moving now on a pratical implemention on Bitcoin data.

Parkinson Volatility Estimator

Binance Bitcoin Data Preparation

As we’ve done in previous articles, we’ll follow a similar data preparation process here. For those needing a refresher, we previously explained how to fetch and store Binance data in an HDF5 file in this post. It’s this dataset we will exploit here.

import pandas as pd 
import numpy as np 
import matplotlib.pyplot as plt 

database_path = "data/crypto_database.h5"
df = pd.read_hdf(database_path, key="BTCUSDT")

# Convert 'Open_Time' to datetime if not already done
df.reset_index(inplace=True)
df['Open_Time'] = pd.to_datetime(df['Open_Time'])

# Set 'Open_Time' as the index
df.set_index('Open_Time', inplace=True)

Close-to-Close Volatility Estimator for Bitcoin

To provide a meaningful comparison, we’ll calculate the Close-to-Close volatility estimator over the same time frame as the Parkinson estimator (1 hour, or 60 data points).

In line with Collin Bennett’s recommendations, we assume a zero-mean model for the returns. Therefore, instead of relying on Python’s built-in .std() function, we’ll manually compute the volatility, excluding any mean adjustment. This approach provides a more accurate representation of volatility under the assumption that returns have a zero mean.

# We will compute returns with the last hour data 
T = 60
df['classic_std'] = df['log_returns'].apply(lambda x:x**2).rolling(T).mean()**0.5

Python Implementation of the Parkinson Volatility Estimator

We then implement the Parkinson Volatility Estimator, which uses the high-low range of prices to capture intraday volatility more effectively.

# Compute the log range 
df['log_range'] = np.log(df['High'] / df['Low'])
# Define a python function for the parkinson volatility 
def parkinson_std(R:float)->float:
    sigma = R / ((4 * np.log(2))**0.5)
    return sigma
# Compute the parkinson standard deviation
df["parkinson_std"] = df["log_range"].apply(lambda x:parkinson_std(x)).rolling(T).mean()

Parkinson vs Classic Volatility in Bitcoin

Now that we’ve computed the Parkinson volatility, let’s visually compare it with our classic close to close volatility estimator to better understand how they behave across different time periods.

# Create a figure with a 1x2 grid layout
fig, ax = plt.subplots(1, 2, figsize=(16, 8))

# Add main title and subtitle to the entire figure
fig.suptitle(
    'Volatility Analysis', 
    fontsize=18, 
    fontweight='bold'
)
fig.text(
    0.5, 0.90, 
    'Comparison of the Parkinson Volatility and the Classic Volatility', 
    fontsize=14, 
    ha='center', 
    style='italic'
)

# Adjust spacing between plots for better layout
plt.subplots_adjust(top=0.85, wspace=0.3)

# Plot 1: Bitcoin price and Parkinson volatility on the left side
ax[0].plot(df['parkinson_std'], label='Parkinson Volatility (Rolling 60)', color='#17becf')  # Teal for Parkinson Volatility
ax[0].set_title('Parkinson Volatility', fontsize=14)
ax[0].set_xlabel('Date')
ax[0].set_ylabel('Parkinson Volatility')
ax[0].legend(loc='upper left')

# Plot 2: Classic standard deviation and percentage difference on the right side
ax[1].plot(df['classic_std'], label='Classic Volatility (Rolling 60)', color='#ff7f0e')  # Orange for Classic Volatility
ax[1].set_title('Classic Volatility', fontsize=14)
ax[1].set_xlabel('Date')
ax[1].set_ylabel('Classic Volatility')
ax[1].legend(loc='upper left')

# Rotate x-axis labels for both subplots for better readability
for axes in ax:
    plt.setp(axes.get_xticklabels(), rotation=45, ha='right')

# Adjust layout to prevent overlap and enhance readability
plt.tight_layout(rect=[0, 0, 1, 0.9])  # Leave space at the top for titles

# Show the plot
plt.show()

Great! It appears that both estimators show fairly similar behavior, which makes sense given that both are designed to be consistent measures of volatility. However, there are bound to be some differences between them. To better understand these variations, let’s calculate the percentage difference between the two estimators:

# Calculate percentage difference between Parkinson and realized volatility
df['volatility_diff'] = 100 * (df['parkinson_std'] - df['close_close_std']) / df['close_close_std']

Let’s focus on specific time periods to examine the differences between the two estimators more closely, as visualizing all the data at once would likely result in a cluttered and unclear display.

Parkinson Volatility During Bitcoin’s 2020 COVID-19 Crash

Background: In March 2020, Bitcoin’s price plummeted by over 50% within a week amidst the global financial turmoil caused by the COVID-19 pandemic.

import matplotlib.pyplot as plt

# Define the time range for the 2020 COVID-19 crash
start_date = '2020-02-01'
end_date = '2020-05-01'

# Slice the DataFrame for this time period
df_2020_crash = df.loc[start_date:end_date]

# Create a figure with a 2x2 grid layout
fig, ax = plt.subplots(2, 2, figsize=(16, 12))

# Add main title and subtitle to the entire figure
fig.suptitle('Bitcoin Price and Volatility Analysis during 2020 COVID-19 Crash', fontsize=18, fontweight='bold')
fig.text(0.5, 0.93, 'Comparison of Parkinson Volatility, Classic Volatility, and Price Evolution', 
         fontsize=14, ha='center', style='italic')

# Adjust spacing between plots for better layout
plt.subplots_adjust(top=0.85, wspace=0.3, hspace=0.4)

# Plot 1: Bitcoin price (close) evolution during 2020 COVID-19 crash
ax[0, 0].plot(df_2020_crash['Close'], label='BTC Price', color='#1f77b4')  # Deep Blue
ax[0, 0].set_title('Bitcoin Price (2020 COVID-19 Crash)', fontsize=14)
ax[0, 0].set_xlabel('Date')
ax[0, 0].set_ylabel('BTC Price (USD)')
ax[0, 0].legend()

# Plot 2: Parkinson volatility during 2020 COVID-19 crash
ax[0, 1].plot(df_2020_crash['parkinson_std'], label='Parkinson Volatility', color='#17becf')  # Teal
ax[0, 1].set_title('Bitcoin Parkinson Volatility (2020 COVID-19 Crash)', fontsize=14)
ax[0, 1].set_xlabel('Date')
ax[0, 1].set_ylabel('Parkinson Volatility')
ax[0, 1].legend()

# Plot 3: Classic standard deviation during 2020 COVID-19 crash
ax[1, 0].plot(df_2020_crash['classic_std'], label='Classic Std (Rolling 120)', color='#ff7f0e')  # Orange
ax[1, 0].set_title('Bitcoin Classic Volatility (Rolling Std, 2020 COVID-19 Crash)', fontsize=14)
ax[1, 0].set_xlabel('Date')
ax[1, 0].set_ylabel('Classic Volatility')
ax[1, 0].legend()

# Plot 4: Percentage difference between Parkinson and Classic volatility
ax[1, 1].plot(df_2020_crash['volatility_diff'], label='% Difference', color='#2ca02c')  # Green
ax[1, 1].plot(df_2020_crash['volatility_diff'].rolling(60*24*7).mean(), label='7-Day Rolling Avg', linestyle='--', color='#d62728')  # Red
ax[1, 1].set_title('Percentage Difference Between Parkinson and Classic Volatility (2020 COVID-19 Crash)', fontsize=14)
ax[1, 1].set_xlabel('Date')
ax[1, 1].set_ylabel('% Difference')
ax[1, 1].legend()

# Adjust layout to prevent overlap and enhance readability
plt.tight_layout(rect=[0, 0, 1, 0.9])  # Leave space at the top for titles

# Show the plot
plt.show()

From the visual analysis of Bitcoin’s price and volatility during the 2020 COVID-19 crash, a few key observations emerge:

Price Trend: The Bitcoin price experienced significant fluctuations, with a steep decline in early March followed by a rapid recovery in April. This reflects the general market instability during the early phases of the pandemic.
Parkinson Volatility: The Parkinson volatility spikes in March, correlating with the sharp price drop, indicating heightened intraday volatility during that period. This volatility seems to stabilize but remains elevated as the price recovers.
Classic Volatility: The classic standard deviation, representing the close-to-close volatility, mirrors the pattern seen in the Parkinson volatility, with spikes during March’s market turbulence. However, it smooths out more in the later periods.
Percentage Difference: The percentage difference between Parkinson and classic volatility shows consistent fluctuations. A notable feature is the elevated percentage difference in mid-March, suggesting that the intraday high-low ranges (Parkinson) captured more volatility compared to the closing prices during the market’s most turbulent moments.

Overall, this period reflects heightened volatility due to the COVID-19 market shock, with significant short-term fluctuations and a high divergence between intraday and closing-price volatility.

Parkinson Volatility During Bitcoin’s 2024 ATH

Background: In 2024, Bitcoin surged to a new all-time high, driven by increasing institutional adoption, macroeconomic instability, and growing recognition of Bitcoin as a hedge against inflation. The price rise was fueled by heightened demand for alternative assets, pushing Bitcoin beyond previous record levels set in 2021.

# Define the time range for the 2024 ATH
start_date = '2024-01-01'
end_date = '2024-04-01'

# Slice the DataFrame for this time period
df_2024_ath = df.loc[start_date:end_date]

# Create a figure with a 2x2 grid layout
fig, ax = plt.subplots(2, 2, figsize=(16, 12))

# Add main title and subtitle to the entire figure
fig.suptitle('Bitcoin Price and Volatility Analysis during 2024 ATH', fontsize=18, fontweight='bold')
fig.text(0.5, 0.93, 'Comparison of Parkinson Volatility, Classic Volatility, and Price Evolution', 
         fontsize=14, ha='center', style='italic')

# Adjust spacing between plots for better layout
plt.subplots_adjust(top=0.85, wspace=0.3, hspace=0.4)

# Plot 1: Bitcoin price (close) evolution during 2024 ATH
ax[0, 0].plot(df_2024_ath['Close'], label='BTC Price', color='#1f77b4')  # Deep Blue
ax[0, 0].set_title('Bitcoin Price (2024 ATH)', fontsize=14)
ax[0, 0].set_xlabel('Date')
ax[0, 0].set_ylabel('BTC Price (USD)')
ax[0, 0].legend()

# Plot 2: Parkinson volatility during 2024 ATH
ax[0, 1].plot(df_2024_ath['parkinson_std'], label='Parkinson Volatility', color='#17becf')  # Teal
ax[0, 1].set_title('Bitcoin Parkinson Volatility (2024 ATH)', fontsize=14)
ax[0, 1].set_xlabel('Date')
ax[0, 1].set_ylabel('Parkinson Volatility')
ax[0, 1].legend()

# Plot 3: Classic standard deviation during 2024 ATH
ax[1, 0].plot(df_2024_ath['classic_std'], label='Classic Std (Rolling 120)', color='#ff7f0e')  # Orange
ax[1, 0].set_title('Bitcoin Classic Volatility (Rolling Std, 2024 ATH)', fontsize=14)
ax[1, 0].set_xlabel('Date')
ax[1, 0].set_ylabel('Classic Volatility')
ax[1, 0].legend()

# Plot 4: Percentage difference between Parkinson and Classic volatility
ax[1, 1].plot(df_2024_ath['volatility_diff'], label='% Difference', color='#2ca02c')  # Green
ax[1, 1].plot(df_2024_ath['volatility_diff'].rolling(60*24*7).mean(), label='7-Day Rolling Avg', linestyle='--', color='#d62728')  # Red
ax[1, 1].set_title('Percentage Difference Between Parkinson and Classic Volatility (2024 ATH)', fontsize=14)
ax[1, 1].set_xlabel('Date')
ax[1, 1].set_ylabel('% Difference')
ax[1, 1].legend()

# Adjust layout to prevent overlap and enhance readability
plt.tight_layout(rect=[0, 0, 1, 0.9])  # Leave space at the top for titles

# Show the plot
plt.show()

In the analysis of Bitcoin’s price and volatility during its 2024 All-Time High (ATH), several insights can be observed:

Price Trend: Bitcoin’s price experienced a significant upward trend leading to its new all-time high, surpassing $70,000 in March 2024. This rally was marked by steady growth throughout January and February, followed by higher volatility as the price approached its peak. The price stabilized but remained elevated after the peak.
Parkinson Volatility: The Parkinson volatility, which measures intraday high-low fluctuations, spiked during moments of rapid price movement, particularly as Bitcoin approached its ATH in mid-March 2024. This indicates increased intraday volatility during periods of strong upward price momentum.
Classic Volatility: The classic volatility (rolling standard deviation of closing prices) follows a similar pattern to Parkinson volatility but appears smoother. It shows volatility peaks aligned with the rapid price increases but captures less extreme intraday swings.
Percentage Difference: The percentage difference between Parkinson and classic volatility fluctuates throughout the period, with notable spikes in volatility divergence as Bitcoin approaches its ATH. The rolling average of the percentage difference shows that intraday volatility (Parkinson) tended to outpace close-to-close volatility (classic) during periods of intense price movement, with the difference narrowing slightly after the ATH.

Overall, this analysis highlights the heightened volatility in both intraday and closing price movements as Bitcoin surged to its 2024 ATH, with Parkinson volatility capturing sharper intraday price changes compared to the classic volatility measure.

Practical Considerations when working on Volatility

A critical decision when implementing a volatility model is selecting the appropriate rolling window length for historical volatility estimation. As Collin Bennett discusses in Trading Volatility, this choice is far from trivial and carries important implications for the accuracy of the model. Specifically, the number of data points used in the rolling window directly impacts the variance of the estimator: a longer window reduces variance but assumes that the underlying market conditions, particularly volatility, remain constant over a longer period.

This assumption can be problematic in practice. In our case, for example, using a rolling window that spans several days or weeks assumes that volatility is relatively stable over that period. However, in fast-moving or highly volatile markets, such an assumption may be hazardous. Short-term market shocks or trends may significantly affect volatility, making a longer window less responsive to sudden changes.

Collin Bennett emphasizes this challenge when he states:

“Choosing the historical volatility number of days is not a trivial choice… while an identical duration historical volatility is useful to arrive at a realistic minimum and maximum value over a long period of time, it is not always the best period of time to determine the fair level of long-dated implieds… volatility mean reverts over a period of c8 months” (Trading Volatility, Bennett).

In essence, while using longer windows may reduce estimator variance, it risks incorporating outdated volatility caused by past events, diminishing the relevance of the estimate for current market conditions. As Bennett suggests, aligning the historical window with implied volatility durations (e.g., 21 trading days for one-month implied volatility) can be a practical approach but should be adjusted to account for events that may no longer impact market conditions.

Conclusion

In this post, we explored the Parkinson estimator as an alternative method for estimating Bitcoin’s volatility, leveraging high and low data points to capture more precise intraday fluctuations. We compared it with the classic close-to-close estimator and observed how the Parkinson estimator provides a better representation of volatility during sharp market movements, such as Bitcoin’s 2020 COVID-19 crash and its 2024 all-time high. While both methods have their merits, the choice of estimator ultimately depends on the nature of the data and the specific trading strategy.

Stay tuned for the next post, where we continue our journey into volatility modeling, exploring more advanced estimators and their practical applications in trading!

I’d love to hear your thoughts! Share your feedback, questions, or suggestions for future topics in the comments below.

References and Further Reading

Parkinson, M. (1980), Estimating the Variance of the Rate of Return for a Security
Feller, W. (1951), The Asymptotic Distribution of the Range of Sums of Independent Random Variables
Great article
Binance API Documentation: Binance API

Feel free to explore these resources to deepen your understanding.

Estimating Bitcoin's Volatility using a GARCH Model

2024-09-23T00:00:00+00:00

Introduction
Motivations
The GARCH Model
Preparing Binance Bitcoin Data for GARCH in Python
Testing Bitcoin Returns for ARCH Effects
Selecting a GARCH Model for Bitcoin
Interpreting the GARCH Parameters
Goodness-of-fit Check
Estimating and Forecasting Volatility
Conclusion

Introduction

In our last post, we discussed the Exponentially Weighted Moving Average (EWMA) method for estimating Bitcoin’s volatility. We used historical data from Binance and implemented the EWMA model to track volatility. To fetch the data, check out this previous post, where we explored how to use the Binance API. This time we’ll take it a step further by introducing the GARCH model, a more sophisticated method used to estimate but also forecast volatility. Let’s dive in!

Motivations

When modeling financial volatility, it’s crucial to choose a model that captures the unique characteristics of asset price returns without unnecessary complexity. While simple models like the random walk with gaussian increments might seem sufficient for short-term forecasts, they miss important features of financial time series.

Let’s consider the random walk model :

\[p_{t+1} = p_t + \epsilon_t, \quad \epsilon_t \sim \mathcal{N}(0, \sigma^2)\]

Then the following empirical properties cannot be modelized :

Volatility Clustering : One of the most well-documented phenomena in financial markets. This refers to the empirical observation that large price changes tend to be followed by large price changes, and small changes tend to be followed by small changes, regardless of the direction of the price movement.
Heavy-Tailed Distributions : Another well-known stylized fact about financial returns. Extreme events (large price changes) happen more often than a normal distribution would predict. Simple volatility models that assume normality tend to underestimate the likelihood of these extreme events.

From a mathematical standpoint:

\[\text{Cov}(\epsilon_t^2, \epsilon_{t+k}^2) > 0 \quad \text{for small } k\] \[\text{Kurt}(\epsilon_t) = \frac{\mathbb{E}[(\epsilon_t - \mu)^4]}{\sigma^4} > 3\]

For any financial time series exhibiting these characteristics (volatility clustering and heavy tails) using a more sophisticated model becomes necessary. Fortunately, we can identify these properties through rigorous statistical tests :

Autocorrelation Tests: We can apply tests like the Ljung-Box Q-test on the squared residuals $\epsilon_t^2$ to detect volatility clustering.
Kurtosis Test: We can perform a Jarque-Bera test to check whether the distribution of the residuals significantly deviates from normality.

Moreover we also have the GARCH model, which is specifically designed to capture and replicate such behaviors. When applied, the GARCH model accounts for these empirical features, allowing the time series to exhibit the volatility clustering and non-normal distributions that are often observed in financial markets. But enough teasing and let’s explain it !

The GARCH Model

What is a GARCH Model?

The GARCH (Generalized Autoregressive Conditional Heteroskedasticity) model was developped by Tim Bollerslev and is a popular choice for financial volatility modeling, especially in markets where volatility tends to cluster over time. Let’s dive in.

We modelize the log returns with :

\[r_t = \log(p_t) - \log(p_{t-1}) \quad \text{where} \quad r_t \sim \mathcal{N}(0, \sigma_t^2)\]

Here, $\sigma_t^2$ is the time-varying volatility, which is gonna be modeled by the GARCH model. A GARCH model need two parameters for it’s definition : p and q. It considers a combination of p lagged variances (past periods’ volatility) and q lagged squared returns (recent price changes), giving it the flexibility to model volatility with a deeper memory. The general form of the GARCH(p, q) model is given by:

\[\sigma_t^2 = \omega + \sum_{i=1}^{q} \alpha_i \cdot r_{t-i}^2 + \sum_{j=1}^{p} \beta_j \cdot \sigma_{t-j}^2\]

Where:

$\sigma_t^2$ is the current variance estimate (volatility squared),
$\omega$ represents the long-term variance (baseline level of volatility),
$\alpha_i$ weights the impact of past squared returns $(r_{t-i}^2)$, capturing how recent market shocks influence volatility,
$\beta_j$ weights the impact of past variances $(\sigma_{t-j}^2)$, reflecting how long-term volatility persists over time.

By adjusting the p and q parameters, the GARCH(p, q) model becomes highly versatile. It can capture both volatility clustering (the tendency for large changes to follow large changes) and mean reversion (volatility eventually returning to a long-term average), which are essential characteristics in financial markets, especially in assets with erratic price movements like cryptocurrencies.

Why Use a GARCH Model Here?

The decision to make a blog post about using GARCH model for estimating Bitcoin’s volatility stems from both academic motivations and empirical necessities. Fitting a GARCH model is not just an exercise in financial theory, but is also one let’s be honnest here.

But the model’s ability to incorporate past variances and shocks into future volatility estimates makes it a compelling choice for researchers and practitioners alike. Academically, it’s exciting to explore how well the GARCH model fits different time series data, especially in markets as volatile as cryptocurrencies, where price swings happen frequently.

In financial time series like Bitcoin’s returns, there are well-documented ARCH effects. Volatility changes over time and exhibits clustering. This means simple models that assume constant volatility will fail to adequately describe the data. Moreover, it seems legit to assume Bitcoin’s return distribution exhibits fat tails, meaning extreme price changes occur more frequently than predicted by a normal distribution. This heavy-tailed behavior is another reason to use a GARCH model.

Preparing the Data

Before we fit a GARCH model, let’s load and clean our data, just like we did in the previous post with EWMA. We’ll again use the Bitcoin data we stored in HDF5 format and ensure the dataset is free of missing values. Because we are gonna use plenty of statistical test, we are gonna have to tackle some hardware limitations. For this modelling we will work with 3 months historic and keep one months for out of sample testing.

Loading Data from HDF5

First, let’s load the Bitcoin data we previously stored in our HDF5 file.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Define the date range 
start_date = '2019-01-01'
end_date = '2019-05-01'
# Load the data from the HDF5 file
df = pd.read_hdf('data/crypto_database.h5', key='BTCUSDT', where=f"index >= '{start_date}' and index <= '{end_date}'")

This will give us access to the Bitcoin candlestick data with 1-minute granularity.

Data Cleaning and Preprocessing

Ensuring data quality is paramount. We’ll check for missing values and ensure that the data types are appropriate.

# Check for missing values
print(df.isnull().sum())

# Convert 'Open_Time' to datetime if not already done
df.reset_index(inplace=True)
df['Open_Time'] = pd.to_datetime(df['Open_Time'])

# Set 'Open_Time' as the index
df.set_index('Open_Time', inplace=True)

Taking a look at the data

Let’s ensure we have good quality data by plotting the evolution of the Bitcoin Price.

# Plot the log returns
plt.figure(figsize=(12, 6))
df['Close'].plot()
plt.title('Bitcoin Price')
plt.xlabel('Date')
plt.ylabel('BTC USDT')
plt.show()

Calculating Log Returns

Let’s calculate here again the log returns using the ‘Close’ price.

# Calculate log returns
df['log_returns'] = np.log(df['Close'] / df['Close'].shift(1))

# Drop the NaN value created by the shift
df.dropna(subset=['log_returns'], inplace=True)

Now, let’s visualize the log returns to get a sense of their behavior.

# Plot the log returns
plt.figure(figsize=(12, 6))
df['log_returns'].plot()
plt.title('Bitcoin Log Returns')
plt.xlabel('Date')
plt.ylabel('Log Return')
plt.show()

Testing for ARCH Effects

In financial time series, it’s crucial to test for ARCH effects (Autoregressive Conditional Heteroskedasticity), which means that the volatility of the series changes over time and is not constant. If ARCH effects are present, we can use models like GARCH to capture the volatility dynamics. For the folowing statistical tests, we will as an example a 3 months period, keeping the last month as an out of sample dataset.

split_date = pd.to_datetime("2019-04-01")
training_data_mask = df.index < split_date
returns = df.loc[training_data_mask, 'log_returns']

Stationarity Test

Before testing for ARCH effects, we must first check if the series is stationary. This is indeed a required features to fit a GARCH model on the serie. We typically can use the Augmented Dickey-Fuller (ADF) test. The null hypothesis of the ADF test is that the series has a unit root, i.e., it is non-stationary.

import pandas as pd
from statsmodels.tsa.stattools import adfuller

# Assuming 'returns' is a Pandas Series of your financial returns
result = adfuller(returns)

print(f'ADF Statistic: {result[0]}')
print(f'p-value: {result[1]}')

ADF Statistic: -53.002111501058444
p-value: 0.0

The p-value is below 0.05, we reject the null hypothesis, implying that the series is stationary.

Autocorrelation Test

Once stationarity is confirmed, we check for autocorrelation in the squared returns. The presence of autocorrelation in squared returns indicates volatility clustering, a key sign of ARCH effects.

To test for autocorrelation, we can use the Ljung-Box Q-test. This test checks for autocorrelation at multiple lags.

from statsmodels.stats.diagnostic import acorr_ljungbox

# Check autocorrelation in squared returns
ljung_box_test = acorr_ljungbox(returns**2, lags=[10], return_df=True)

print(ljung_box_test)

         lb_stat  lb_pvalue
10  34331.948673        0.0

A significant p-value (below 0.05) for the Ljung-Box test means that there is significant autocorrelation in the squared returns, suggesting the presence of ARCH effects.

Normality Test

The Jarque-Bera test assesses whether the skewness and kurtosis of the series significantly deviate from those of a normal distribution. The null hypothesis is that the data follows a normal distribution.

The test checks both skewness and kurtosis:

Skewness indicates asymmetry in the distribution.
Kurtosis measures the tail heaviness of the distribution.

A high Jarque-Bera test statistic suggests that the series is not normally distributed, which is often the case with financial returns, where we observe fat tails and non-symmetric behavior.

from scipy.stats import jarque_bera

# Perform Jarque-Bera test on returns
jb_test_stat, jb_p_value = jarque_bera(returns)

print(f'Jarque-Bera Test Statistic: {jb_test_stat}')
print(f'P-value: {jb_p_value}')

Jarque-Bera Test Statistic: 85728124.76566969
P-value: 0.0

A p-value lower than 0.05 indicates that we reject the null hypothesis, implying that the returns do not follow a normal distribution. This further strengthens the case for using models like GARCH, which can handle non-normal characteristics such as fat tails and volatility clustering.

Model Selection

We now have to select the p (the lag of past volatilities) and q (the lag of past squared returns) orders of our GARCH model. Goal here is thus to find the configuration that best fits the data. To do this, we’ll estimate multiple models by varying the parameters p and q in the GARCH model, ranging from simple to more complex configurations, and compare their AIC and BIC values. The model with the lowest Akaike Information Criterion (AIC) is generally favored when focusing on model fit, while the lowest Bayesian Information Criterion (BIC) is preferred when penalizing more complex models to avoid overfitting.

Some pratical considerations now :

We’ll use Python’s arch package to fit the GARCH model.
We’ll apply a scaling factor of 1440 to the log returns.
We’ll we used a zero mean assumption for the log returns.

Why scale the data ?

This is necessary because GARCH models tend to perform better when the input data is scaled to more manageable levels, especially with high-frequency data like minute-by-minute returns. Without scaling, the small magnitude of log returns can lead to numerical instability in the estimation process. Scaling helps to improve the model’s convergence and interpretability of the parameter estimates. Here we choose to multiply the returns by the number of minutes in the day : 1440.

Why assume a zero drift ?

The argument I will give here was presented by Collin Bennett in his book Trading Volatility. Bennett explains that when calculating volatility, it is often best to assume zero drift. The calculation for standard deviation measures the deviation from the average log return (drift), which must be estimated from the sample. This estimation can lead to misleading volatility calculations if the sample period includes unusually high or negative returns.

For example, if a stock rises by 10% every day for ten days, the standard deviation would be zero because there is no deviation from the 10% average return. However, such trends are unrealistic over the long term, and using the sample log return as the expected future return can distort the volatility estimate. By assuming a zero mean or drift, we prevent the volatility calculation from being influenced by extreme sample returns. In theory, over the long term, the expected return should be close to zero, as the forward price of an asset should reflect this assumption. This is why volatility calculations are typically more accurate when assuming zero drift.

Results

from arch import arch_model

scaling_factor = 60*24 # We get daily returns
min_bic = 1e10
selected_orders = (0, 0)

for p in range(1, 4):
    for q in range(0, 4):
        # GARCH(p,q) model
        model = arch_model(
            df['log_returns']*scaling_factor, 
            mean='Zero', 
            vol='GARCH', 
            p=p, q=q
            )
        res = model.fit(disp='off', last_obs=split_date,)
        print(f'GARCH({p},{q}) AIC: {res.aic}, BIC: {res.bic}')
        if res.bic < min_bic:
            min_bic = res.bic
            selected_orders = (p, q)
p, q = selected_orders
print(f'Selected model with BIC : p={p} and q={q}')

GARCH(1,0) AIC: 269793.01931055624, BIC: 269812.558147921
GARCH(1,1) AIC: 245265.1323021801, BIC: 245294.44055822722
GARCH(1,2) AIC: 244480.07736779848, BIC: 244519.15504252794
GARCH(1,3) AIC: 244168.23045229167, BIC: 244217.0775457035
GARCH(2,0) AIC: 260475.38690308636, BIC: 260504.69515913347
GARCH(2,1) AIC: 245268.00954894855, BIC: 245307.087223678
GARCH(2,2) AIC: 244482.07737045927, BIC: 244530.9244638711
GARCH(2,3) AIC: 244170.230456519, BIC: 244228.8469686132
GARCH(3,0) AIC: 255222.99234444185, BIC: 255262.0700191713
GARCH(3,1) AIC: 245269.1326656593, BIC: 245317.97975907114
GARCH(3,2) AIC: 244484.07805419483, BIC: 244542.69456628902
GARCH(3,3) AIC: 244172.23049541027, BIC: 244240.61642618684
Selected model with BIC : p=1 and q=3

After evaluating multiple GARCH(p, q) models, we first select the GARCH(1, 3) model based on both information criterions. Let’s take a look at the estimated parameters then :

# GARCH(1, 3) model
model = arch_model(
    df['log_returns']*scaling_factor,
    mean='Zero',
    vol='GARCH',
    p=p, q=q
    )
res = model.fit(last_obs=split_date, disp='off')
print(res.summary())

                       Zero Mean - GARCH Model Results                        
==============================================================================
Dep. Variable:            log_returns   R-squared:                       0.000
Mean Model:                 Zero Mean   Adj. R-squared:                  0.000
Vol Model:                      GARCH   Log-Likelihood:               -122079.
Distribution:                  Normal   AIC:                           244168.
Method:            Maximum Likelihood   BIC:                           244217.
                                        No. Observations:               129239
Date:                Thu, Sep 26 2024   Df Residuals:                   129239
Time:                        10:34:30   Df Model:                            0
                              Volatility Model                              
============================================================================
                 coef    std err          t      P>|t|      95.0% Conf. Int.
----------------------------------------------------------------------------
omega          0.0180  3.464e-03      5.200  1.991e-07 [1.122e-02,2.480e-02]
alpha[1]       0.1996  1.736e-02     11.497  1.362e-30     [  0.166,  0.234]
beta[1]        0.3958      0.103      3.859  1.140e-04     [  0.195,  0.597]
beta[2]        0.1163      0.127      0.919      0.358     [ -0.132,  0.364]
beta[3]        0.2756  3.697e-02      7.455  9.010e-14     [  0.203,  0.348]
============================================================================

Covariance estimator: robust

This provides us with the key parameters of our model, that we can analyze now !

Interpreting the GARCH Parameters

The GARCH(1, 3) model provides several important parameters that give insight into Bitcoin’s volatility. Below is a breakdown of each key parameter and its interpretation:

Parameter	Value	Description	Interpretation
$\omega$	0.0180	Represents the long-term or baseline variance.	A small, positive, and significant $\omega$ indicates a constant underlying volatility in the returns. Spikes in volatility are largely driven by recent shocks and historical persistence.
$\alpha_1$	0.1996	Captures the impact of recent squared returns ($r_{t-1}^2$) on current volatility.	About 20% of recent price movements influence current volatility. This suggests that large, sudden price changes significantly impact volatility.
$\beta_1$	0.3958	Reflects the persistence of past volatility on current volatility.	Around 40% of previous volatility persists into the current period. This explains the volatility clustering phenomenon, where high volatility tends to follow high volatility.
$\beta_2$	0.1163	Represents the influence of the second lag of volatility on current volatility.	$\beta_2$ is not statistically significant (p-value = 0.358), suggesting the second lag has a minimal effect on current volatility.
$\beta_3$	0.2756	Measures the impact of the third lag of volatility on current volatility.	The third lag is statistically significant with a t-statistic of 7.455. This reinforces the idea that Bitcoin volatility has a long memory.

Key Insights:

Long-term volatility ($\omega$) is relatively low, suggesting that Bitcoin’s baseline volatility is not extreme but driven by recent market shocks.
Recent shocks ($\alpha_1$) have a notable effect, with around 20% of volatility attributed to recent returns.
Volatility persistence ($\beta_1$) is high, reinforcing the clustering of volatility that is often observed in financial markets.
Second lag ($\beta_2$) is not significant, indicating that further-back periods do not heavily influence the current volatility.
Third lag ($\beta_3$) is significant, meaning that some past volatility still impacts the current market, showcasing Bitcoin’s long volatility memory.

Goodness-of-fit Check

After fitting our selected GARCH model, it is however highly recommended to perform a goodness-of-fit check on the residuals to ensure that the model adequately captures the dynamics of your time series. The main goal of such checks is to assess whether the model residuals behave as expected—ideally, they should resemble white noise, meaning they have no autocorrelation and constant variance (homoscedasticity). We will work on the standardized residuals

# Standardized residuals from the GARCH model
std_residuals = res.resid / res.conditional_volatility
# Because of the last obs arguments we have NaNs
std_residuals = std_residuals.dropna()

Let’s vizualize those residuals :

# Plot the standardized residuals from the GARCH model
plt.figure(figsize=(12, 6))
std_residuals.plot()
plt.title('Standardized residuals from the GARCH model')
plt.xlabel('Date')
plt.ylabel('Standardized residuals')
plt.show()

At first glance, the situation looks concerning: we observe clear volatility clustering, with high volatility in the initial months, and numerous extreme values, suggesting the residuals may deviate from normality.

We now need to realize some rigorous statistical testing to confirm that.

Autocorrelation of Residuals

The standardized residuals (the residuals divided by their estimated volatility) should no longer show any significant autocorrelation. We can here again use the Ljung-Box Q-test to test whether there is any remaining autocorrelation in the residuals or squared residuals.

# Ljung-Box test for residuals (no autocorrelation should be present)
ljung_box_res = acorr_ljungbox(std_residuals, lags=[10], return_df=True)
print('Ljung-Box test for residuals')
print(ljung_box_res)

# Ljung-Box test for squared residuals (no remaining ARCH effects should be present)
ljung_box_sq_res = acorr_ljungbox(std_residuals**2, lags=[10], return_df=True)
print('Ljung-Box test for squared residuals')
print(ljung_box_sq_res)

Ljung-Box test for residuals
       lb_stat     lb_pvalue
10  163.434818  6.326210e-30

Ljung-Box test for squared residuals
      lb_stat  lb_pvalue
10  19.082143   0.039232

The p-values are bellow 0.05, this indicates that there is statistically significant autocorrelation in the residuals or squared residuals, suggesting that the GARCH model has not effectively captured all the volatility structure.

Normality of Residuals

The standardized residuals should follow a normal distribution. The Jarque-Bera test can be used for this:

from scipy.stats import jarque_bera

# Perform Jarque-Bera test on standardized residuals
jb_stat, jb_p_value = jarque_bera(std_residuals)
print(f'Jarque-Bera Test Statistic: {jb_stat}')
print(f'P-value: {jb_p_value}')

Jarque-Bera Test Statistic: 5551677.823833454
P-value: 0.0

The p-value bellow 0.05 indicates that the residuals do significantly deviate from normality.

After running both test, the results indicate that the model’s residuals have issues.

What Might Actually Improve the Model?

To improve the fit of the GARCH model, we could consider several adjustments. The following table outlines possible strategies for improving the model:

Adjustment	Description	Benefit
Use a higher-order GARCH	Increase the number of lags in the GARCH terms, such as moving to GARCH(2, 3). This adjustment could better capture the autocorrelation if volatility has a longer memory, thereby modeling both short-term and long-term volatility effects more effectively.	Provides more flexibility in capturing extended memory in the volatility series, which could improve the model’s fit in markets with persistent volatility patterns.
Use an EGARCH or GJR-GARCH model	A GARCH variant like EGARCH or GJR-GARCH can account for asymmetric volatility (where negative returns have a greater effect on volatility than positive returns).	Improves performance in modeling volatility clustering in assets with skewed returns, like Bitcoin.
Change the residual distribution	If the residuals deviate from normality (as indicated by the Jarque-Bera test), switch from a normal distribution to a Student’s t-distribution or Generalized Error Distribution (GED) for the residuals.	Better reflects the non-normal characteristics of financial time series, particularly fat tails, improving the model's ability to handle extreme events and outliers.

But should we really bother ?

Should We Discard the Current GARCH Model?

Although the Ljung-Box tests for residuals and squared residuals show statistically significant autocorrelation, and the Jarque-Bera test indicates that the residuals deviate from normality, these results do not necessarily invalidate the use of the GARCH model. This indeed does not imply that the model is entirely ineffective. GARCH models are specifically designed to capture conditional heteroskedasticity—volatility that changes over time based on past returns and variances. While the remaining autocorrelation suggests there are still some dynamics unexplained by the model, the GARCH framework remains usefull to capture volatility clustering.

Even though the residuals do not follow a perfect normal distribution, the GARCH model can still provide valuable insights by capturing time-varying volatility. If we expect fat-tailed distributions, we could enhance the model by specifying a non-normal distribution (e.g., Student’s t-distribution) for the residuals, which might better reflect the empirical properties of the data.

Financial time series are notoriously difficult to model, and violations of normality or remaining autocorrelation are expected in complex markets like cryptocurrencies. GARCH remains a valuable tool in capturing the essential dynamics of volatility, and with potential enhancements—such as using a different distribution for the residuals or exploring higher-order GARCH models—we can further refine the model for better performance.

In summary, while the current model may not be perfect, it provides a solid foundation for understanding volatility patterns. Refining the model rather than discarding it will likely lead to more accurate and insightful results. Thus, we will keep it that way for the moment, and probably improve it in a latter post.

Estimating and Forecasting Volatility

Once we have select and fit one model, we can use it both to estimate realized volatility and for forecast.

Estimating Realized Volatility

In this subsection, we’ll estimate realized volatility using our GARCH model and compare it to an alternative method like the EWMA estimator. This comparison will give us a clearer picture of how well the GARCH model captures the dynamics of Bitcoin’s volatility compared to simpler approaches.

Using the following code, we extract and plot the conditional volatility from our GARCH model:

# Get conditional volatility
cond_vol = res.conditional_volatility
cond_vol.dropna(inplace=True)

# Plot the conditional volatility
plt.figure(figsize=(12, 6))
cond_vol.plot()
plt.title(f'Conditional Volatility from GARCH({p},{q}) Model')
plt.xlabel('Date')
plt.ylabel('Volatility')
plt.show()

The plot shows how volatility fluctuates over time, capturing periods of heightened risk (volatility clustering) and more stable periods.

The Exponentially Weighted Moving Average (EWMA) model is another method often used to estimate realized volatility, giving more weight to recent observations. To compute and compare the EWMA volatility, we can use the following code:

# Compute EWMA volatility
lambda_ = 0.94  # Decay factor as used in RiskMetrics
ewma_vol = (scaling_factor*returns).ewm(alpha=(1 - lambda_), adjust=False).std()

# Create a figure with two subplots (2 rows, 1 column)
fig, axs = plt.subplots(2, 1, figsize=(12, 12))

# First subplot: GARCH vs EWMA Volatility
axs[0].plot(cond_vol, label='GARCH Conditional Volatility', color='blue')
axs[0].plot(ewma_vol, label='EWMA Volatility', color='orange')
axs[0].set_title('GARCH vs. EWMA Volatility')
axs[0].set_xlabel('Date')
axs[0].set_ylabel('Volatility')
axs[0].legend()

# Second subplot: GARCH vs EWMA Volatility Ratio
axs[1].plot((cond_vol - ewma_vol) / ewma_vol, label='GARCH Conditional Volatility vs EWMA Volatility Ratio', color='blue')
axs[1].plot(((cond_vol - ewma_vol) / ewma_vol).rolling(60*24).mean(), label='Rolling Mean', color='red')
axs[1].set_title('GARCH vs. EWMA Volatility Ratio')
axs[1].set_xlabel('Date')
axs[1].set_ylabel('Volatility Ratio')
axs[1].legend()

# Adjust layout to prevent overlap
plt.tight_layout()
# Display the figure
plt.show()

Observation: The GARCH model consistently shows higher volatility estimates compared to the EWMA model.
Explanation: This is a typical feature of the GARCH model, which captures both the recent volatility and its persistence over time. Unlike EWMA, which gives more weight to recent data while smoothing out fluctuations, GARCH is designed to account for longer-term volatility clustering, making it more responsive to periods of heightened market stress.
Observation: The EWMA model displays smoother, more stable volatility estimates over time.
Explanation: EWMA assigns exponentially decreasing weights to past data, which dampens the impact of older, more volatile periods. This explains why EWMA produces lower volatility values, as it focuses more on recent changes without accounting for the persistent volatility that the GARCH model captures.

Forecasting Volatility

The GARCH(1,3) model allows us to generate conditional forecasts over future periods. Let’s expirement forecasting the volatility on unseen data.

forecasts = res.forecast(horizon=60, start=split_date)
forecasted_variance = forecasts.variance
forecasted_volatility = np.sqrt(forecasted_variance)    

# Plot the conditional volatility
plt.figure(figsize=(12, 6))
forecasted_volatility["h.60"].plot()
plt.title(f'Forecast Volatility from GARCH({p},{q}) Model')
plt.xlabel('Date')
plt.ylabel('Volatility')
plt.show()

The plot shows how volatility is expected to behave over the next 60 minutes. Spikes in forecasted volatility would indicate heightened market risk, while a more stable line would suggest a calmer market.

Conclusion

In this post, we explored the use of the GARCH model for estimating and forecasting Bitcoin’s volatility. Compared to simpler models like EWMA, GARCH provides a more robust framework for capturing the dynamic nature of financial markets, particularly in handling volatility clustering and fat tails—two key characteristics often present in cryptocurrency markets.

We began by discussing the motivations for moving beyond simple models, then delved into the mechanics of the GARCH(1, 3) model, explaining how it captures both recent volatility shocks and their persistence. We demonstrated how to fit the GARCH model, perform essential goodness-of-fit tests, and interpret the resulting parameters. We encountered some issues with the residuals, but despite these imperfections our GARCH model still provides valuable insights into Bitcoin’s volatility structure and that potential improvements, like changing the residual distribution or testing higher-order models, could further refine the model.

We showed how the GARCH model can be used both to estimate realized volatility and to generate volatility forecasts for the next hour. This ability to project future volatility is incredibly useful for traders, risk managers, and anyone involved in high-frequency trading environments where knowing short-term risk is critical.

In the next post, we will dive deeper into volatility forecasting by exploring alternative data points like Low and High. Stay tuned for the final part of our series as we continue to explore the fascinating world of volatility modeling for Bitcoin !

Additional Resources

Code Repository: GitHub Link
Adviced Reading: John Hull’s Options, Futures, and Other Derivatives and Collin Bennet’s Trading Volatility

Feel free to check out the GitHub repository for the complete code and try experimenting with different parameters to see how they affect volatility estimates.

I’d love to hear your thoughts! Share your feedback, questions, or suggestions for future topics in the comments below.

Happy analyzing, and as always, may your trading strategies be well-informed!

Bitcoin Volatility Estimation with EWMA in Python

2024-09-20T00:00:00+00:00

Introduction

In our previous post, we explored how to fetch and store Binance candlestick data using HDF5, laying the groundwork for efficient data management in cryptocurrency research. Efficient data handling is crucial when dealing with large datasets, especially in the fast-paced world of crypto trading.

Now, we’re taking the next step in our journey by delving into the analysis of this data. This is the first part of a three-part series where we’ll dive deep into estimating Bitcoin’s volatility using different methods.

Terminology Alert : Volatility measures how much and how quickly the value of an asset changes.

It’s important to note that volatility is not directly observable. It’s a hidden variable that must be estimated from market data. Its value can vary depending on the estimation model used, making it a crucial yet complex component in financial analysis.

There are two main types of volatility:

Realized Volatility: Calculated from past market prices, it reflects the asset’s actual price movements over a specified period.
Implied Volatility: Derived from the market price of a traded derivative, especially options. It represents the market’s expectation of future volatility.

Why this difference ? Well, it happens volatility is far easier to predict than returns themselves. For options or other financial products, the future value of the volatility (meaning the future behaviour of the asset) is more usefull. Thus, the market will use the volatility expectancy for those products. In this post, we will focus on the realized volatility.

Why Analyze Bitcoin’s Realized Volatility ?

Understanding realized volatility is crucial, especially when dealing with highly volatile assets like Bitcoin. Realized volatility measures the degree of variation in an asset’s past price movements over a specific period. Studying it provides valuable insights into the asset’s risk profile and helps in making informed decisions. Here’s why analyzing realized volatility is essential:

Risk Assessment and Management: Realized volatility is a key indicator of the risk associated with an asset. By analyzing how much Bitcoin’s price has fluctuated in the past, traders and investors can gauge the potential risk of future price movements.
Strategic Trading Decisions: Studying realized volatility can help traders develop effective trading strategies by recognizing periods of high or low volatility to choose appropriate trading approaches (e.g., trend following in high volatility, mean reversion in low volatility).
Option Pricing and Derivative Valuation: Since volatility is a hidden variable and not directly observable, realized volatility serves as an essential estimate.
Getting Risk-Adjusted Returns: Evaluating whether the potential returns justify the risks based on realized volatility.

In essence, studying realized volatility is not just an academic exercise—it’s a practical necessity. It equips traders and investors with the knowledge to navigate the complexities of the financial markets, particularly the dynamic and often unpredictable cryptocurrency landscape. By acknowledging that volatility is a hidden variable dependent on estimation models, we can appreciate the importance of selecting appropriate methods to measure and interpret it. This understanding ultimately leads to better risk management, more effective trading strategies, and informed decision-making in the pursuit of financial goals.

Preparing Binance Bitcoin Data for EWMA in Python

Before we can estimate volatility, we need to prepare our data. The database we will use is the one we created in this post.

Loading Data from HDF5

First, let’s load the Bitcoin data we previously stored in our HDF5 file.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 

# Load the data from the HDF5 file
df = pd.read_hdf('data/crypto_database.h5', key='BTCUSDT')

This will give us access to the Bitcoin candlestick data with 1-minute granularity.

Data Cleaning and Preprocessing

Ensuring data quality is paramount. We’ll check for missing values and ensure that the data types are appropriate.

# Check for missing values
print(df.isnull().sum())

# Convert 'Open_Time' to datetime if not already done
df.reset_index(inplace=True)
df['Open_Time'] = pd.to_datetime(df['Open_Time'])

# Set 'Open_Time' as the index
df.set_index('Open_Time', inplace=True)

Taking a look at the data

Let’s ensure we have good quality data by plotting the evolution of the Bitcoin Price.

# Plot the log returns
plt.figure(figsize=(12, 6))
df['Close'].plot()
plt.title('Bitcoin Price')
plt.xlabel('Date')
plt.ylabel('BTC USDT')
plt.show()

Calculating Bitcoin Log Returns in Python

Logarithmic returns, or log returns, can be preferred over simple returns for several reasons:

Time Additivity: Log returns are additive over time, which simplifies calculations.
Normality: They often better approximate a normal distribution, which is a common assumption in financial models.
Symmetry: Log returns treat upward and downward movements symmetrically.

The formula for log returns is:

\[r_t = \ln\left(\frac{P_t}{P_{t-1}}\right)\]

Where:

$ r_t $ is the log return at time $ t $.
$ P_t $ is the price at time $ t $.

Let’s calculate the log returns using the ‘Close’ price. While this approach is common, it’s important to note that volatility estimation doesn’t have to rely solely on the ‘Close’ price; other data points like the ‘Open’, ‘High’, and ‘Low’ during a time interval are also meaningful and can provide additional insights. However, for simplicity in this post, we’ll only use the ‘Close’ price.

# Calculate log returns
df['log_returns'] = np.log(df['Close'] / df['Close'].shift(1))

# Drop the NaN value created by the shift
df.dropna(subset=['log_returns'], inplace=True)

Now, let’s visualize the log returns to get a sense of their behavior.

# Plot the log returns
plt.figure(figsize=(12, 6))
df['log_returns'].plot()
plt.title('Bitcoin Log Returns')
plt.xlabel('Date')
plt.ylabel('Log Return')
plt.show()

Estimating Bitcoin Volatility with EWMA

Introducing the EWMA Method

The Exponentially Weighted Moving Average (EWMA) is a powerful statistical technique that assigns exponentially decreasing weights to past observations, giving more significance to recent data points. This responsiveness to the latest market information makes EWMA particularly valuable in volatile markets like cryptocurrency, where conditions can change rapidly.

EWMA was popularized by J.P. Morgan in the early 1990s through their development of the RiskMetrics system. RiskMetrics provided a standardized framework for measuring and managing market risk, introducing EWMA as an effective method for estimating volatility. John Hull’s book “Options, Futures, and Other Derivatives” discusses EWMA as an effective method for volatility estimation. By emphasizing recent price movements without entirely discarding older data, EWMA captures the persistence and clustering of volatility often observed in financial time series.

The EWMA volatility is calculated using the formula:

\[\sigma_t^2 = \lambda \sigma_{t-1}^2 + (1 - \lambda) r_t^2\]

Where:

$ \sigma_t^2 $ is the variance at time $ t $.
$ \lambda $ is the decay factor (0 < $ \lambda $ < 1).
$ r_t $ is the return at time $ t $.

Setting the Decay Factor (Lambda)

The choice of the decay factor $ \lambda $ determines how quickly the weights decrease. A commonly used value is $ \lambda = 0.94 $, as suggested by RiskMetrics. A higher $ \lambda $ gives more weight to older observations, while a lower $ \lambda $ emphasizes recent returns more heavily.

Step-by-Step Calculation

Using pandas, we can implement the EWMA volatility estimation easily.

# Set the decay factor
lambda_ = 0.94
alpha = 1 - lambda_

# Calculate the EWMA variance
df['ewma_variance'] = df['log_returns'].ewm(alpha=alpha, adjust=False).var()

# Calculate the EWMA volatility
df['ewma_volatility'] = np.sqrt(df['ewma_variance'])

Visualization

Let’s plot the EWMA volatility to visualize how it changes over time.

# Plot the EWMA volatility
plt.figure(figsize=(12, 6))
df['ewma_volatility'].plot()
plt.title('EWMA Volatility of Bitcoin Returns')
plt.xlabel('Date')
plt.ylabel('Volatility')
plt.show()

Analyzing the Results

By analyzing the plot, we can identify periods of sharp volatility spikes, which signal higher risk and potentially greater reward opportunities. These spikes are often triggered by significant market events or news impacting Bitcoin. This behavior is known as volatility clustering, where high-volatility periods are followed by similarly turbulent periods, while low-volatility phases exhibit smaller, more stable fluctuations.

EWMA Volatility: Advantages and Limitations

Responsiveness: EWMA adjusts quickly to changes, making it suitable for volatile markets.
Simplicity: It’s straightforward to implement and doesn’t require complex modeling.
Weighted Data: By giving more importance to recent data, EWMA provides a more current view of market volatility.
Easy Online Learning: EWMA is well-suited for real-time or online learning, where data is processed sequentially. In production environments, where data arrives continuously, this feature ensures the system can easily adapt without reprocessing the entire dataset each time.

These characteristics make EWMA a highly practical tool for production applications, especially in environments where real-time data processing is critical. Yet it has some important limitations that should be considered, especially when thinking about future predictions:

No Predictive Power for Future Volatility: The EWMA estimator is not suitable for predicting future volatility. When simulating future values of the model, the expectation $ E[\sigma_{t+h}] $ tends to 0 as $ h $ increases, meaning the model effectively predicts no future volatility (assuming $ E[ r_t] = 0 $ ). This limitation arises because EWMA only reacts to past data without incorporating any forward-looking information or structural changes in the market.
Constant Decay Factor: The choice of the decay factor $ \lambda $ is critical to the model’s performance. However, EWMA uses a constant $ \lambda $, which assumes that the rate at which past data becomes less relevant is fixed over time. This might not be realistic during periods of sudden market regime shifts, where volatility patterns change dramatically.
Sensitivity to Large Shocks: While EWMA adapts to volatility changes, it might still overreact to short-term shocks. Large price movements (like flash crashes) can disproportionately affect the estimated volatility in the short term, without accounting for whether these movements represent structural shifts or noise.

Lastly, as explained by Collin Bennet in his book Trading Volatility Exponentially weighted volatility models struggle to account for recurring volatility-inducing events like earnings reports. Just before an earnings announcement—when future volatility is expected to rise—the model places the least weight on previous earnings-related volatility spikes. Conversely, right after the earnings event—when future volatility is likely to decrease—the model assigns the most weight to these prior events. However, these models may still have some applicability when analyzing indices.

Due to these limitations, the EWMA method is suitable only for historical analysis and not for forecasting future volatility. More advanced models, like GARCH, are better suited for predicting future volatility because they account for the persistence and reversion tendencies in volatility over time.

Conclusion and Next Steps

In this post, we’ve introduced the concept of volatility and its importance in financial markets, particularly for Bitcoin. We’ve shown how to estimate volatility using the EWMA method, which provides a responsive and practical approach for traders and analysts.

This is just the beginning. In the next posts of this series, we’ll explore more sophisticated methods like GARCH models for volatility estimation.

Stay tuned for deeper insights into the fascinating world of financial volatility analysis!

Additional Resources

Code Repository: GitHub Link
Adviced Reading: John Hull’s Options, Futures, and Other Derivatives and Collin Bennet’s Trading Volatility

Feel free to share your thoughts or ask questions in the comments below. I encourage you to experiment with the code, try different decay factors, and see how it affects the volatility estimates.

I’d love to hear your thoughts! Share your feedback, questions, or suggestions for future topics in the comments below.

Happy analyzing, and as always, may your trading strategies be well-informed!

How to Fetch and Store Binance Data Efficiently Using HDF5 in Python

2024-08-18T00:00:00+00:00

Why HDF5 is Ideal for Storing Cryptocurrency Data

When handling large datasets, especially in research, the choice of file format is crucial. HDF5 (Hierarchical Data Format version 5) is a powerful data model that is ideal for managing and storing enormous amounts of data efficiently. It’s designed to store data in a hierarchical structure, making it a perfect choice for large, complex datasets—like cryptocurrency trading data. Let’s dive into how you can leverage HDF5 to store crypto data fetched from Binance.

Before jumping into the code, let’s discuss why HDF5 is so beneficial:

Efficiency: HDF5 is built for writing and reading speed. It can handle massive datasets much more efficiently than traditional file formats like CSV.
Scalability: It’s designed to store and organize large amounts of data across multiple datasets.
Flexibility: HDF5 files can be easily sliced and diced for different analyses without needing to load the entire dataset into memory.

These characteristics make HDF5 a valuable tool for research, especially when working with high-frequency trading data, which can accumulate quickly.

Want some proof ? Here is an comparaison between .csv and .h5 files for a small dataset.

For the writing speed :

%time df.to_csv("data/speed_test.csv")

CPU times: total: 93.8 ms
Wall time: 123 ms

%time df.to_hdf("data/speed_test.h5", key="data", mode="w")

CPU times: total: 0 ns
Wall time: 14.7 ms

For the reading speed :

%time a = pd.read_csv("data/speed_test.csv")

CPU times: total: 15.6 ms
Wall time: 32 ms

%time b = pd.read_hdf("data/speed_test.h5", key="data")

CPU times: total: 15.6 ns
Wall time: 11.7 ms

As you can see, the csv format can’t compete.

Getting Started with Python Imports

First, let’s set up our environment by importing the necessary libraries:

import requests
import pandas as pd
from datetime import datetime, timedelta
import time

These libraries are fundamental for fetching, processing, and storing cryptocurrency data. requests will help us interact with Binance’s API, while pandas is crucial for data manipulation. datetime and time will manage our data fetching timeframes.

Fetching and Storing Binance Data in HDF5

Next, we’ll create a function to fetch candlestick data from Binance and store it in an HDF5 file. The HDF5 file acts as a database, storing our data in a structured and efficient manner.

def fetch_candlestick_data(symbol, interval, start_date, end_date, limit=10000, hdf5_file='data/crypto_database.h5', key='candlestick_data'):
    url = "https://api.binance.com/api/v3/klines"
    
    # Convert dates to timestamps in milliseconds
    start_ts = int(datetime.strptime(start_date, '%Y-%m-%d %H:%M:%S').timestamp() * 1000)
    end_ts = int(datetime.strptime(end_date, '%Y-%m-%d %H:%M:%S').timestamp() * 1000)

    # Calculate the total number of steps
    total_steps = (end_ts - start_ts) // (limit * 60 * 1000) + 1

    # Initialize step counter
    step_counter = 0
    
    while start_ts < end_ts:
        params = {
            'symbol': symbol,
            'interval': interval,
            'startTime': start_ts,
            'endTime': min(start_ts + limit * 60 * 1000, end_ts),
            'limit': limit
        }
        
        response = requests.get(url, params=params)
        if response.status_code == 200:
            data = response.json()
            if not data:
                break
            df = pd.DataFrame(data, columns=[
                'Open_Time', 
                'Open', 
                'High', 
                'Low', 
                'Close', 
                'Volume', 
                'Close_Time', 
                'Quote_Asset_Volume', 
                'Number_of_Trades', 
                'Taker_Buy_Base_Asset_Volume', 
                'Taker_Buy_Quote_Asset_Volume', 
                'Ignore'
            ])
            
            # Convert timestamps to datetime
            df['Open_Time'] = pd.to_datetime(df['Open_Time'], unit='ns')
            del df['Close_Time']
            # Convert numeric columns to appropriate types
            df['Open'] = pd.to_numeric(df['Open'])
            df['High'] = pd.to_numeric(df['High'])
            df['Low'] = pd.to_numeric(df['Low'])
            df['Close'] = pd.to_numeric(df['Close'])
            df['Volume'] = pd.to_numeric(df['Volume'])
            df['Quote_Asset_Volume'] = pd.to_numeric(df['Quote_Asset_Volume'])
            df['Taker_Buy_Base_Asset_Volume'] = pd.to_numeric(df['Taker_Buy_Base_Asset_Volume'])
            df['Taker_Buy_Quote_Asset_Volume'] = pd.to_numeric(df['Taker_Buy_Quote_Asset_Volume'])

            # Set 'Open Time' as index
            df.set_index('Open_Time', inplace=True)

            # Store DataFrame in HDF5 file
            df.to_hdf(
                hdf5_file, 
                key=key, 
                mode='a', 
                format='table', 
                append=True, 
                data_columns=True,
            )
            
            # Update step counter
            step_counter += 1
            
            # Print progress
            print(f"Step {step_counter}/{total_steps} completed.")
            
            start_ts = data[-1][0] + 1  # Move to the next timestamp
        else:
            print(f"Error: {response.status_code}")
            time.sleep(60)  # Wait a minute before retrying
            continue
        
        time.sleep(1)  # To avoid hitting rate limits

Breaking Down the Function :

API Request: The function sends a request to Binance’s API for candlestick data. We loop through time intervals using the start_ts and end_ts timestamps to ensure we gather data over a specified period.
Data Processing: The JSON response is converted into a pandas DataFrame, making it easier to work with. We format the data by converting timestamps and changing numeric columns to their appropriate types.
HDF5 Storage: Finally, we store the DataFrame in an HDF5 file. This allows us to append new data over time, making it an efficient method for long-term storage.

Running the Data Collection

Now, let’s tie it all together with a main function that initiates the data fetching process:

def main():
    symbol = 'BTCUSDT'
    interval = '1m'  # 1-minute granularity
    start_date = '2020-01-01 00:00:00'
    end_date = '2020-02-01 00:00:00'
    
    # Fetch the data and store it in HDF5
    fetch_candlestick_data(
        symbol, 
        interval, 
        start_date, end_date, 
        hdf5_file='data/crypto_database.h5',
        key=symbol)

if __name__ == "__main__":
    main()

Final Thoughts

By using HDF5, you gain a powerful tool for managing large datasets, ensuring that your research is both scalable and efficient. Whether you’re analyzing historical price data or developing trading strategies, this approach provides a robust foundation for your work.

Moreover, you can expand your database by storing additional cryptocurrency pairs in the same HDF5 file. Accessing them is straightforward—simply use pd.read_hdf() with the desired key, allowing you to seamlessly retrieve and analyze data across multiple assets.

Additional Resources

Code Repository: GitHub Link
Adviced Reading: Yves Hilpish Python for Algorithmic Trading

I’d love to hear your thoughts! Share your feedback, questions, or suggestions for future topics in the comments below.

Happy coding, and may your data always be manageable!

Leveraging Options Data for Stock Sentiment Analysis

2024-07-14T00:00:00+00:00

Introduction

Options data, often overlooked by those who don’t directly trade options, can be a goldmine for assessing market sentiment and gauging potential stock movements. Here, we explore three key options metrics that can serve as a sentiment barometer for any stock trader:

Key Metrics to Watch

Open Interest
Put/Call Ratio
IV Skewness

These metrics provide valuable insights into market dynamics and potential trend shifts in stock prices. If you want to use them easily, feel free to check my app here

Practical Tip : Option data can easily be access for major stock with yfinance for free

import yfinance as yf
import pandas as pd
def get_options_data(ticker):
    # Load the ticker data
    stock = yf.Ticker(ticker)
    # Get available options expirations
    expirations = stock.options
    # Initialize an empty DataFrame to hold all options data
    all_options = pd.DataFrame()
    # Loop through all available expiration dates and get the options data
    for date in expirations:
        # Fetch the call and put data for the current expiration date
        opt = stock.option_chain(date)
        # Combine the call and put data
        current_options = pd.concat([opt.calls, opt.puts])
        current_options['Expiration'] = date  # Add the expiration date to the DataFrame
        # Append the current options data to the all_options DataFrame
        all_options = pd.concat([all_options, current_options])
    # Clearly Identify Call and Puts 
    all_options["Type"] = all_options["contractSymbol"].copy()
    all_options["Type"] = all_options["Type"].apply(lambda x:x.split(ticker)[1][6])
    all_options["Type"] = all_options ["Type"].map({"C":"Call", "P":"Put"})
    return all_options

Understanding Open Interest

Open interest represents the total number of outstanding options contracts that have not been settled. For each new option contract that is opened, open interest increases by one; it decreases by one for every contract that is closed. This number gives traders an idea of the liquidity and size of the market for a particular option.

Some traders use the highest open interest strike price as a marker for potential support or resistance levels in the underlying stock. This concept is tied to the “max pain theory,” which posits that the stock price might gravitate towards the strike price at expiration where the most options (both calls and puts) would expire worthlessly, causing the maximum financial loss to option holders.

Analyzing the Put/Call Ratio

Terminology Alert : The put/call ratio measures the volume of trading in puts versus calls.

\[\text{Put/Call Ratio} = \frac{\text{Volume of Puts Traded}}{\text{Volume of Calls Traded}}\]

This ratio is calculated by dividing the total volume of puts traded by the total volume of calls traded over a specific period, typically one trading day. In practice, the put/call ratio can be derived from options across all strikes and maturities, not just those with the same strike price or expiration date. This comprehensive approach provides a broader view of market sentiment, capturing the general trading behavior towards puts versus calls across the entire options market. Traders use this ratio to gauge overall market sentiment, identifying whether there’s a prevailing bearish or bullish inclination among participants.

A higher ratio indicates more put buying (commonly seen as bearish sentiment), whereas a lower ratio suggests more call buying (typically viewed as bullish sentiment). This ratio can act as a contrarian indicator, hinting at possible reversals if predominantly one type of option outweighs the other significantly.

Traders often look for extremes in this ratio on their trading platforms to assess whether the market sentiment is overly pessimistic or optimistic, which might indicate a potential reversal in the stock’s price movement.

Conclusion

While not everyone may trade options, the data derived from options markets can offer crucial insights into broader market sentiments and potential shifts in stock trajectories. By integrating open interest and put/call ratios into your analysis, you can gain a more nuanced understanding of market dynamics and better prepare for future movements. Remember, these tools should provide additional layers of insight and are not standalone decision-makers.

An Introduction to Pair Trading and Market Neutral Strategies

2024-06-22T00:00:00+00:00

Introduction

I have started working on pairs trading and market-neutral trading strategies in the US equity market over the last few months. Here, I will try to give a clear and concise introduction to the topic and introduce the frameworks I will be delving into more deeply in the next posts.

General Overview

Before speaking of pairs trading, let’s first define market-neutral strategies. A market-neutral strategy aims to provide uncorrelated sources of returns from the market. The Capital Asset Pricing Model (CAPM), even if now outdated, provides a nice framework to explain the intuition:

The return of an asset can be expressed as $r_t - r_f = \alpha + (\beta \times (r_m - r_f)) + \epsilon_t$, with:

$r_f$ the risk-free return
$r_m$ the market return
$\alpha$ the expected excess return
$\beta$ the sensitivity of the asset’s returns to the market returns

With this framework, if you manage to construct a portfolio with a null beta (i.e., the linear combination of the assets within is such that the sum of their betas is null), you then get a portfolio whose returns are uncorrelated from those of the market! The quant researcher’s role is thus to reduce its strategy’s beta as much as possible and to maximize its alpha (i.e., its expected outperformance). Bear and bull markets don’t matter anymore! But what’s the point?

Why do market neutral startegies

Why not just buy OTM calls with one week until expiration and pray to get rich? Joke aside, why not just be a bullish investors and be directional ? Warrent Buffet for sure isn’t a market neutral player.

This is I believe a legitimate question for any investors (motivations from the Quants side are mostly about the challenge, the maths behind and the elegance of it). Well first all, this allows you to perform consistent return (doesn’t mean you will suceed). Market can indeed go upside but also downside. Their is also the diversification play : since Markowitz’s work and modern portfolio theory adoption investor are looking for uncorrelated strategies in order to improve their sharpe or sortino ratio.

About Pairs Trading

Pairs trading is a specific framework with the purpose of creating market-neutral strategies. The way to do this, used by hedge fund quants, is as follows: use a linear combination of two stocks to remove the beta from the market. Let’s consider two stocks: stock A and stock B.

Under the CAPM model (again, this approach is outdated and if you are looking for practical advice, you should work with a multi-factor model at least) you get:

\[\beta_B \times r_t^A - \beta_A \times r_t^B = \beta_B \times \alpha_A - \beta_A \times \alpha_B + \epsilon_t^A + \epsilon_t^B\]

with

$\beta_A$, $\alpha_A$, $r_t^A$, and $\epsilon_t^A$ being the CAPM components for stock A
$\beta_B$, $\alpha_B$, $r_t^B$, and $\epsilon_t^B$ being the CAPM components for stock B

As you can see, the market part has disappeared. Theoretically, buying $\beta_B$ stock A and shorting $\beta_A$ stock B should give you a market-neutral portfolio for any stock. Where’s the hitch? Well, a warning here: an assumption of the CAPM model is that $\epsilon_t$ is an idiosyncratic risk, meaning the correlation between $\epsilon_t^A$ and $\epsilon_t^B$ is null. That’s a big assumption and why I have been warning so much about this framework. We will see more useful frameworks in the next posts.

Terminology Alert: In practice, strategies tend to go long on stock A and short $\frac{\beta_A}{\beta_B}$. Thus, we introduce the hedge ratio $\rho = \frac{\beta_A}{\beta_B}$.

Pair trading is all about working on spreads between the two assets and to construct getting positive returns from it (modern pair trading can of course be involving 3, 4, 5, or even more stock but we will keep simple for now).

Now that we are clear on what market neutral means, let’s get into some of the basic tools.

Some Technical Aspects

Let’s dive into some technical aspects. I will stop using the CAPM model from now on. Despite it being great for gaining some common sense and intuitions, it’s not so good for modeling today’s market.

A common framework for years has been cointegration. Basically, you consider stocks A and B being cointegrated if and only if you can get a stationnary series from a linear combinaison of their log prices (which are not stationnary):

\[\log(P_t^A) = \text{const} + \rho \times \log(P_t^B) + \epsilon_t\]

with $(\epsilon_t)$ being stationary.

In this case, you are able to compute the hedge ratio from a simple regression. Moreover, you can get a confidence interval and p-value to ensure it is significantly non-null. Great! I have to say that linking cointegration with achieving a market-neutral strategy is not straightforward. It indeed requires at least some modeling or further assumptions/requirements. But for now, let’s admit it gets us closer.

But the devil hides in the details. I won’t delve into all the linear regression temporal aspects and its application for sequential data (refer to a handbook on time series prediction for more knowledge), but one has to be very careful. We are indeed working on non-stationary variables.

Terminology Alert: A spurious regression is a regression where the p-value indicates a statistically significant relationship between variables, but this relationship is actually meaningless or random, often due to underlying trends in the data rather than a true correlation.

Let’s consider two stock prices, $(P_t^A)$ and $(P_t^B)$. Let’s model both stock prices logarithm by a random walk and with uncorrelated noise. So we get:

$\log(P_t^A) = \log(P_0^A) + \sum_{i=1}^{t} \epsilon_i^A$ $\log(P_t^B) = \log(P_0^B) + \sum_{i=1}^{t} \epsilon_i^B$

with:

$\epsilon_t^A \sim \text{WN}(0, \sigma_A^2)$ and $\epsilon_t^B \sim \text{WN}(0, \sigma_B^2)$

The issue here is $Var(P_t^A) = t \times \sigma_A^2$, which means we have a non-stationary variable. Thus, the linear regression requirements are not met. If you attempt to do one, the issue is that the usual test statistics (like t-statistics) do not follow their standard distributions under the null hypothesis. P-values will not be interpretable since they need to follow a uniform distribution between 0 and 1 under the null hypothesis.

Doing some simulation with the folowing code :

    import numpy as np
    import pandas as pd
    import statsmodels.api as sm
    import matplotlib.pyplot as plt
    
    def simulate_random_walk(n):
        """ Generate a random walk series. """
        # Random walk starts at zero
        walk = np.zeros(n)
        # Generate random steps, add to previous
        for i in range(1, n):
            walk[i] = walk[i - 1] + np.random.normal()
        return walk
    
    def perform_regression(x, y):
        """ Perform linear regression and return the p-value. """
        x = sm.add_constant(x)  # Adding a constant for the intercept
        model = sm.OLS(y, x)
        results = model.fit()
        return results.pvalues[1]  # p-value for the slope
    
    def main():
        num_simulations = 1000
        num_points = 1000  # Number of points in each random walk
        p_values = []
    
        for _ in range(num_simulations):
            # Generate two independent random walks
            x = simulate_random_walk(num_points)
            y = simulate_random_walk(num_points)
            
            # Perform regression and get the p-value
            p_value = perform_regression(x, y)
            p_values.append(p_value)
    
        # Plotting the distribution of p-values
        plt.hist(p_values, bins=100, edgecolor='black')
        plt.xlabel('P-value')
        plt.ylabel('Frequency')
        plt.title('Distribution of P-values for 100 Regressions of Independent Random Walks')
        plt.show()
    
        # Optionally, return or print p_values or any other statistics
        return p_values
    
    # Call the main function to execute the simulation
    if __name__ == "__main__":
        p_values = main()

You get the folowing graph :

As you can see, despite having two unrelated random walks, we get a significant relationship most of the time.

To summarize, cointegration could be a great tool, allowing us to compute the hedge ratio. However, one must ensure the hypothesis of cointegration is met, meaning hypothesis testing on the stationarity of the residuals is required in order to use it. That’s where cointegration’s best friend comes into play: the stationarity test.

Several stationarity tests exist. They can be used in tandem or on their own. The two main ones are the KPSS and the ADF tests. They allow us to test if a series is likely to be stationary or not. I would advise again to read Time series Analysis if you are looking to go into the maths.

If say you suceed to find two stock which cointegrated prices, you are then able to construct a stationanry portfolio longing one share of stock A and shorting \rhau share of stock B. You can then can exploid the mean reversion properties of a stationnary process. Meaning, you will short this portfolio when it is bellow its long term mean and short it when its above its long term mean (you actually can greatly improve from this but again, we keep it simple for now).

When people use these two tools, the process is typically as follows:

Compute the linear regression
Perform a stationarity test on the residuals
If the residuals are stationary, use the linear regression coefficients to get the hedge ratio
Obtain your market-neutral portfolio
Create entry and exit thresholds to trade on low/mid-frequency

This approach is explained by Gatev et al. in Pairs Trading: Performance of a Relative Value Arbitrage Rule. Trading the spreads is not something I will explain here but known it can be approach with stochastic calculus (OU model, …).

The pratical challenges

So far, we have a well-documented approach, complete with specified mathematics and models, that should enable us to consistently make money. So, what’s the catch? To generate consistent profits, you need to identify several pairs and trade them. I believe the identification phase is one of the most challenging aspects. If one opts for a brute force approach and applies the Augmented Dickey-Fuller (ADF) test to all possible pairs, the number of false positives will likely skyrocket.

The hedge against other players is created in this phase. The question then becomes: How will you define your universe? There are many existing stock universes that involve methods such as clustering, setting thresholds on the Hurst exponents of the spreads, and multiple stationarity tests, among others.

Additionally, there is a second challenge: you should aim to trade pairs with the fastest mean reversion speed. This is also be difficult to achieve and I will leave it an open question for now.

Conclusion

In conclusion, while the theory behind pair trading and market-neutral strategies appears robust, implementing these strategies effectively requires navigating a complex landscape of challenges. Identifying viable pairs, maintaining a truly market-neutral position, and capitalizing on mean reversion speed are just the tip of the iceberg. Each phase, from defining your universe to executing trades, demands rigorous analysis and a critical approach to data. The true test lies not only in the selection and application of models but also in our ability to adapt and refine these strategies in response to evolving market conditions. As we move forward, the subsequent posts will explore more advanced tools and techniques, aiming to enhance our understanding and execution of these sophisticated trading strategies. Stay tuned for a deeper dive into the nuances that can make or break the success of pair trading.

A closed-form filter for binary time series

2023-07-09T00:00:00+00:00

This project is a common work with Raphael Thabut and Eric Vong. It’s a pratical implementation of A closed-form filter for binary time series by Fanaso and al.

Article presentation

Denoising data is always relevant to perform, as in reality, a lot of observations can be noised due to external effects. Most filtering methods to predict and smooth distributions relies on the assumption of the Gaussian nature of the state-space models. This is because we have closed form filters available. Even for a binary state-space model, we require Monte-Carlo strategies for prediction.

This article claims that they have found a closed form filter for the dynamic probit models with a gaussian state variable. To do so, they showed that they belong to a class of distribution, the Skew Unified Normal (SUN), whose parameters can be updated with closed expression.

They then show how the algorithm performs in practice, and how using, the properties of the SUN distribution, allows for online inference for high dimensions.

Zoom on the financial application

The articles gave an application example, using financial time series. Indeed, it focus on the analysis of two times series, the Nikkei 225 (Japanase Stock Index) and the CAC 40 (French Stock Index).

As the latter opens after the former, one can try to use the cointegration effect between the two time series to obtain relevant predictions. Our data will be composed of :

$x_t$ where $x_t = 1$ if the Nikkei 225 opening has increased between the close value and opening value and $x_t = 0$ otherwise
$y_t$ where $y_t = 1$ if the CAC 40 opening has increased between the close value and opening value and $y_t = 0$ otherwise

One important objective for a financial application would be to evaluate $Pr(y_t = 1 \mid x_t)$.

This can be done using a probit model. We try to infer a binary value given two parameters, the trend of the CAC 40 when the Nikkei 225 opening is negative and the shift in the trend if the Nikkei 225 opening is positive. Mathematically, this means :

\[P(y_t = 1 \mid x_t) = \Phi[F_t \theta, 1]\]

with the following notations :

$F_t = (1, x_t)$
$\theta = (\theta_1, \theta_2)$, $\theta_1$ being the trend and $\theta_2$ the shift

In the case of such an application, the purpose of the methods we describe in this report is to estimate the $\theta$ parameter effectively.

The SUN distribution

Gaussian models are omnipresent in mathematical models. They have very strong properties and make models simpler. Although, one important drawback is that real life distribution tends to have some sort of asymetry.

A first extension of the gaussian models, Skew Normals, has been proposed by Arellano-Valle and Azzalini (1996). Due to the fertility of this formulation, a unifying representation, the Skewed Unifyed Normals, has been proposed by them since 2006.

Given $\theta \in R^q$, we say that $\theta$ has a unified skew-normal distribution, noted $\theta \sim {SUN}_{q, h}{(\xi, \Omega, \Delta, \gamma, \Gamma)}$, if its density function $\rho{(\theta)}$ is of form:

\[\phi_q{(\theta - \xi ; \Omega)} \frac{\Phi_{h}{(\gamma + \Delta^T \overline{\Omega}^{-1} \omega^{-1} ({\theta - \xi}); \Gamma - \Delta^T \overline{\Omega}^{-1} \Delta})}{\Phi_h{(\gamma ; \Gamma)}}\]

With:

$\phi_q{(\theta ; \Omega)}$ where $\phi_h$ is the centered gaussian density with covariance $\Omega$ matrix
$\Phi_{h}{(\gamma ; \Gamma)}$ where $\Phi_h$ is the centered gaussian cumulative distribution function with covariance $\Gamma$
$\Omega = \omega \overline{\Omega} \omega$ and $\overline{\Omega}$ is the correlation matrix
$\xi$ controls the location
$\omega$ controls the scale
$\Delta$ defines the dependance between the covariance matrixes $\Gamma, \overline{\Omega}$
$\gamma$ controls the truncature

The term:

\[\frac{\Phi_{h}(\gamma + \Delta^T \overline{\Omega}^{-1} \omega^{-1} (\theta - \xi) ; \Gamma - \Delta^T \overline{\Omega}^{-1} \Delta)}{\Phi_{h}(\gamma ; \Gamma)}\]

induces the skewness as it is rescaled by the term $\Phi_{h}(\gamma ; \Gamma)$

We have also the following distribution equality $\theta \overset{(d)} = \xi + \omega (U_0 + \Delta \Gamma^{-1} U_1)$ with $U_0$ independant of $U_1$ and $U_0 \sim N_q{(0, \overline{\Omega} - \Delta \Gamma^{-1} \Delta^T)}, U_1 \sim TN_h{(0, \Gamma, -\gamma)}$ truncated normal law below $-\gamma$. This representation allows for efficient computing, but we can also apply it to a probit model.

Given $y_t = (y_{1t}, \ldots, y_{mt})^T \in {(0, 1)}^m$ a binary vector and $\theta_t = (\theta_{1t}, \ldots, \theta_{pt})^T \in R^p$ the vectors of parameters. The dynamic probit model is then defined as: `

$p{(y_t \mid \theta_t)} = \Phi_m{(B_t F_t \theta_t ; B_t V_t B_t)}$ $\theta_t = G_t \theta_{t-1} + \varepsilon_t, \varepsilon_t \sim N_p{(0, W_t)}$

With $\theta_0 \sim N_{p}{(a_0, P_0)}$ with $B_t = \mathrm{diag}{(2y_{1t} - 1, \ldots, 2y_{mt} - 1)}$

This representation is equivalent to another formulation (Albert and Chib 1993) as followed:

$p{(z_t \vert \theta_t)} = \phi_m{(z_t - F_t \theta_t; V_t)}$ $\theta_t = G_t \theta_{t-1} + \varepsilon_t, \varepsilon_t \sim N_{p}(0, W_t)$

With $y_{it} = 1{z{it} > 0}$

Then, under such a dynamic, the first step filtering distribution is:

\[(\theta_1 \mid y_1) \sim SUN_{p,m}{(\xi_{1 \mid 1}, \Omega_{1 \mid 1}, \Delta_{1 \mid 1}, \gamma_{1 \mid 1}, \Gamma_{1 \mid 1})}\]

With the following parameters:

$\xi_{1 \mid 1} = G_1 a_0$
$\Omega_{1 \mid 1} = G_1 P_0 G_1^T + W_1$
$\Delta_{1 \mid 1} = \overline{\Omega_{1 \mid 1}} \omega_{1 \mid 1} F_1^T B_1 s_1^{-1}$
$\gamma_{1 \mid 1} = s_1^{-1} B_1 F_1 \xi_{1 \mid 1}$
$\Gamma_{1 \mid 1} = s_1^{-1} B_1 \times {F_1 \Omega_{1 \mid 1} F_1^T + V_1}B_1 s_1^{-1}$
$s_1 = (\mathrm{diag}{(F_1 \Omega_{1 \mid 1}F_1^T + V_1)})^{1/2}$

More generally, if $(\theta_t \mid y_{1:t-1}) \sim SUN_{p,m(t-1)}(\xi_{t-1 \mid t-1}, \Omega_{t-1 \mid t-1}, \Delta_{t-1 \mid t-1}, \gamma_{t-1 \mid t-1}, \Gamma_{t-1 \mid t-1})$ is the filtering distribution at $t-1$, then the predictive distribution at $t$ is $(\theta_t \mid y_{1:t-1}) \sim SUN_{p,m(t-1)}(\xi_{t \mid t-1}, \Omega_{t \mid t-1}, \Delta_{t \mid t-1}, \gamma_{t \mid t-1}, \Gamma_{t \mid t-1})$ with the following parameters:

$\xi_{t \mid t-1} = G_t \xi_{t-1 \mid t-1}$
$\Omega_{t \mid t-1} = G_t \Omega_{t-1 \mid t-1}G_t^T + W_t$
$\Delta_{t \mid t-1} = w_{t-1 \mid t-1}^{-1} G_t w_{t -1 \mid t-1} \Delta_{t-1 \mid t-1}$
$\gamma_{t \mid t-1} = \gamma_{t-1 \mid t-1}$
$\Gamma_{t \mid t-1} = \gamma_{t-1 \mid t-1}$

And at time $t$, the filtering distribution are defined as:

$\xi_{t \mid t} = \xi_{t \mid t-1}$
$\Omega_{t \mid t} = \Omega_{t \mid t-1}$
$\Delta_{t \mid t} = [\Delta_{t \mid t-1}, \overline{\Omega_{t \mid t}} \omega_{t \mid t} F_t^T B_t s_t^{-1}]$
$\gamma_{t \mid t} = [\gamma_{t \mid t-1}^T, \xi_{t \mid t}^T F_t^T B_t s_t^{-T}]$
$\Gamma_{t \mid t} = [[A, B], [B, C]]$
$A = \Gamma_{t \mid t-1}$
$B = s_t^{-1} B_t F_t \omega_{t \mid t} \Delta_{t \mid t-1}$
$C = s_t^{-1} B_t(F_t \Omega_{t \mid t} F_t^T + V_t) B_t s_t^{-1}$
$s_t = (\mathrm{diag}{F_t \Omega_{t \mid t} F_t^T + V_t})^{1/2}$

For the smoothing, it is available with similar forms of equations.

Particles algorithms

Particle filters are a mathematical tool used to estimate the state of a system that is described by a set of uncertain variables. It is a type of recursive Bayesian filter that estimates the state of a system by sampling a large number of “particles,” each of which represents a possible state of the system. The particles are weighted according to their likelihood of being the true state of the system, and the weighted average of the particles is used as an estimate of the state. In this section we describe two particles algorithms we used to estimate the parameters of the targeted SUN distribution : the bootstrap filter, and an optimal filter proposed by the researchers.

To obtain similar results, detailed in the fourth part of the report, we started with the same parameters and used the previously seen properties.

$a_0 = [0, 0]$
$P_0 = 3I_2$
$V_t = 1$

Bootstrap filter

In a bootstrap filter, the state estimates are generated using a bootstrapping process, which involves sampling from the previous estimate of the state distribution with replacement. This means that each sample is drawn from the previous estimate, and the same sample may be drawn multiple times. The new estimate of the state is then obtained by weighting the samples according to their likelihood of being the true state.

In the specific case of our study, using the properties and the assumptions of the model, we implemented the bootstrap filter by following this pseudo code :

Where $W_0$ is taken as $W_0 = 0.01 I_2$ (suggested by a graphical search of the maximum for the marginal likelihood computed under different combinations via the analytical formula)

Independent and identically distributed sampling

A first filter is proposed by the researchers in the article. It uses the main properties of the SUN distribution in order to get independent and identically distributed sampling. This algorithm can be described by the following pseudo code :

Optimal filter

The optimal filter introduced by the researchers can be described by the following pseudo code, using the equations of the SUN distribution and filtering, we perform the steps :

Where $G_t = \Gamma = \overline{\Omega} = I_2$ and $\omega = 0.1$, this filter leverages the SUN distribution properties detailed above. With the starting values, we have:

\[\Gamma = I_2\]

Results and Conclusion

We decided to focus on the convergence of the algorithms. Plotting the convergence of the models parameter $\theta_1$ and $\theta_2$.

We observe similar results as the one in the paper :

$\theta_1$ seems to tend towards -0.5
$\theta_2$ seems to tend towards 1

One may find the same plot but for the boostrap algorithm in the Appendix. We can see the same convergence results : $\theta_1$ seems to tend towards -0.5, $\theta_2$ seems to tend towards 1.

In order to compare the accuracy of the algorithms, Fasano et al used the following procedure :

Compute the corresponding SUN density on a 2000*2000 sized grid of equally spaced values of $\theta_{1t}$ and $\theta_{2t}$ for each $t$ in $1,2,…,97$
Compute the marginal distributions of $\theta_{1t}$ (resp. $\theta_{2t}$) by summing over the values of $\theta_{2t}$ (resp. $\theta_{1t}$) and multiplying by the grid step size (this corresponds to approaching the integral of the joint density on one of the two variables by a rectangle method)
Run each algorithm 100 with respectively $10^3, 10^4,$ and $ 10^5$ particles
Compute the Wasserstein distance between the empirical distribution given by each algorithm and the discretized exact density
for each algorithm, take the median of the 100 Wasserstein distance computed

We also used this method, however due to the extensive computing time the methods required. We restrained ourselves to grids of size $100 \times 100$ and only went up to $t=50$. We ended up with the following table :

Algorithm	$\theta_1 \quad (R=10^3)$	$\theta_1 \quad (R=10^4)$	$\theta_1 \quad (R=10^5)$	$\theta_2 \quad (R=10^3)$	$\theta_2 \quad (R=10^4)$	$\theta_2 \quad (R=10^5)$
Boostrap	0.086	0.036	0.025	0.124	0.069	0.062
Optimal	0.082	0.034	0.024	0.117	0.067	0.061

We can see that our results are similar to the article’s results, and the conclusions drawn out are the same : although mild, we benefit from a better accuracy with the proposed “optimal” filter than the one we observe when using the classic bootstrap filter.
“ We also reproduced a bar plot which shows the frequency at which each algorithm dominates the other accross time.

We can see the “optimal” filter has ranks above the bootstrap filter most of the time: it overperforms the bootstrap filter at frequency 0.59 for $R=10^3$, 0.84 for $R=10^4$, and 0.66 for $R=10^5$.

We also compared the performance of the algorithms throughout the time window by averaging the wasserstein distance over each of the 100 experiments for each algorithm at each time $t$, the graph below shows the results:

We can see that the accuracy of both algorithms decreases with time (let it be noted that here a good accuracy is a low accuracy, although this can sound confusing). Of course, we also notice throughout the study that more particles implies better results.

Finally, and because we are using Monte Carlo methods, we decided to take a look at the variance of our algrithms. We expected a smaller variance for the “optimal” algorithm. To look at the variance of the results, we simulated 100 runs with each algorithm and $10^3$ samples. We then assessed the median variance. We obtained the following results :

Algorithm	Variance of $\theta_1$	Variance of $\theta_2$
Boostrap	0.00525312	0.00937037
Optimal	0.00380095	0.00727529

As expected we have lower variance with the optimal algorithm than the boostrap algorithm. Moreover, we have a relatively small variance : $10^{-3}$ for all coefficients.

Appendix

How to cluster time series within a bayesian framework

2023-07-08T00:00:00+00:00

A model-based clustering of multiple time-series algorithm

This project is a common work with Gabriel Guaquiere. It is a pratical implementation of the methods from Model-Based Clustering of Multiple Time Series by Frühwirth-Schnatter, and S. Kaufmann.
You can check the python implementation on my Github.

Introduction

The purpose of this project is to group a set of time series into different clusters and, in doing so, estimate the statistical model describing the time series of each cluster. To conduct our estimations, we adopt a Bayesian framework and utilize Monte Carlo simulation methods and hidden Markov chains.

Let $(y_{i,t})$, with $t=1,…,T$, be a multiple time series among N other time series $i=1,…,N$. We assume that these series belong to K different clusters, and that all time series belonging to cluster $k$ are described by the same statistical model with a group-specific parameter, $\theta_k$. The membership to group $k$ for each time series $i$ is initially unknown and is estimated simultaneously with the parameters $\theta_k$. Additionally, we assume that the parameters $\theta_k$ are specific to each cluster.

We introduce :

the vector $S = (S_1,S_2,…,S_N)$ where $\forall i \in 1,..,N$ $S_i$ $\in 1,…,K$ indicates the group to which time series $i$ belongs
the vector $\phi = (\eta_1, …, \eta_K)$ where $\eta_k$ indicates the proportion of time series belonging to cluster $k$.

Using an MCMC algorithm, we will iteratively estimate the vector $S$, followed by the vectors $\theta$ and $\phi$.

The Model

Theoretical Perspective

For $i = 1,..,N$, the conditional density of $y_i$ given $\theta_{S_i}$ is written as:

\[p(y_i \mid \theta_{S_i}) = \prod_{t=1,...,T} p(y_{i,t} \mid y_{i,t-1},..,y_{i,0},\theta_{S_i})~~~~~~~~~~~~~~~~~~~~~~(1)\]

Where $p(y_i \mid y_{i,t-1},..,y_{i,0},\theta_{S_i})$ is a known density that depends on the chosen model.

Therefore,

\[p(y_i \mid S_i, \theta_1,...,\theta_K) = p(y_i \mid \theta_{S_i})~~~~~~~~~~~~~~~~~~~~~~(2)\]

Next, we establish a probabilistic model for the variable $S = (S_1,..,S_N)$. We assume that $S_1, S_2,..,S_N$ are pairwise independent a priori, and for all $i = 1,..,N$, we define the prior probability $Pr(S_i = k)$, the probability that time series $i$ belongs to cluster $k$. We assume that for each series $i$, we have no prior knowledge of which cluster it belongs to. Hence,

\[Pr(S_i = k \mid \eta_1,..,\eta_K) = \eta_k~~~~~~~~~~~~~~~~~~~~~~(3)\]

The sizes of the groups $(\eta_1,..,\eta_K)$ are initially unknown and are estimated using the data.

The MCMC Algorithm

The estimation of the parameter vector $\psi = (\theta_1,..,\theta_k,\phi,S)$ using MCMC is done in two steps:

Step 1

We fix the parameters $(\theta_1,..,\theta_K,\phi)$ and estimate $S$.

In this step, we will assign a group $k$ to each time series $i$ using the posterior $p(S_i \mid y,\theta_1,..,\theta_K,\phi)$.

Based on Bayes’ theorem and the above, we know that:

\[p(S_i = k \mid y,\theta_1,..,\theta_K,\phi) \propto p(y_i \mid \theta_k)Pr(S_i = k \mid \phi)~~~~~~~~~~~~~~~~~~~~~~(4)\]

Using equations (1) and (3), we will calculate this posterior for $k = 1,..,K$, and using Python, we will simulate a draw of $S_i$ and assign it to a group $k$.

Step 2

We fix the classification $S$ and estimate the parameter vector $(\theta_1,..,\theta_K,\phi)$.

Conditioned on $S$, the variables $\theta$ and $\phi$ are independent. Since the parameter $\theta_k$ is specific to cluster $k$, we group all time series belonging to group $k$.

Thus, $\theta_k$ is estimated using the posterior (5) and a Metropolis-Hastings algorithm:

\[p(\theta_k \mid y,S_1,..,S_N) = \prod_{i : S_i = k} p(\theta_k \mid y_i) = \prod_{i : S_i = k} p(y_i \mid \theta_k)p(\theta_k)~~~~~~~~~~~~~~~~~~~~~~(5)\]

Where the prior $p(\theta_k)$ depends on the chosen model.

Finally, we estimate $\phi = (\eta_1,..,\eta_k)$ using the posterior (6) and a Metropolis-Hastings algorithm:

$p(\phi \mid S,y) = p(y \mid S,\phi,\theta_1,..,\theta_K)$
$= p(y \mid S,\phi,\theta_1,..,\theta_K) \times p(S \mid \phi) \times p(\phi)~~~~~~~~~~~~~~~~~~~~~~(6)$
$= \prod_{k=1,…,K} \prod_{i : S_i = k} p(y_i \mid \theta_k) \prod_{j = 1,…,N}Pr(S_j \mid \phi)p(\phi)$

Where the prior distribution of $\phi$ is a Dirichlet distribution (4,..,4).

We will estimate $\psi = (\theta_1,..,\theta_k,\phi,S)$ by repeating these two steps P times, after initializing $\psi^{0} = (\theta_1^{0},..,\theta_k^{0},\phi^{0},S^{0})$.

Implementation

From a practical point of view, the likelihood of the estimated ARIMAX(p, 0, d) models was calculated using the statsmodels library.

To avoid a final classification with zero time series in a cluster, we decided to randomly select about ten series in such cases to update the cluster parameters. Otherwise, if the size of a cluster is reduced to zero in the early iterations, the coefficients associated with the model would not be updated.

The two steps described in the previous section were used for a Gibbs sampling algorithm. For each step, a Metropolis-Hastings algorithm was implemented. A random walk was performed to find the model coefficients, with ten successive iterations for each step. We selected this number based on the results we obtained and taking into account the complexity of the final algorithm.

Results

Model 1

The first model is an ARMA(1,1) model of the form:

$y_{i,t} = \alpha_{S_i}y_{i,t-1} + \beta_{S_i}\epsilon_{t-1} + \epsilon_t$

Here, $\theta_k$ = $(\alpha_k,\beta_k)$, where $\alpha_k$ and $\beta_k$ are the AR(1) and MA(1) parameters, respectively, of the time series belonging to cluster $k$.

We set $K = 2$ and $N = 100$, and use the following priors:

$\forall k \in 1,2 : \alpha,\beta \sim \mathcal{N}(0,,\frac{1}{3})\phi \sim \mathcal{D}(4,4)\Pr(S_i = k \mid \eta_1,..,\eta_K) = \eta_k$

Since $\epsilon_t \sim \mathcal{N}(0,\sigma^2)$, we have:

$y_{i,t} \mid y_{i,t-1},..,y_{i,0},\theta_{S_i} \sim \mathcal{N}(\alpha_{S_i}y_{i,t-1} + \beta_{S_i}\epsilon_{t-1}, \sigma^2)$

We are able to calculate the posteriors (4), (5), and (6) and estimate the parameters using the method described in Section 2.

The results obtained after 5000 iterations are as follows:

Coefficients	$\alpha$	$\beta$	$\sigma^2$	Cluster Sizes
True Value	1 and -0.8	0.1 and 0.1	0.1 and 0.1	80 and 20
Estimated Value	1.00 and 0.81	0.08 and 0.11	0.09 and 0.10	80 and 20
Error (%)	0.03% and 0.91%	17.3% and 12.6%	0.19% and 0.95%	0% and 0%

Our model successfully assigned each series to its cluster and accurately estimated the $\alpha$ parameter for each cluster. However, it performed less well in estimating the $\beta$ (MA(1)) parameter. Perhaps with a more appropriate prior, the error could have been reduced. The variance of the residuals was estimated with high precision.

The graphs below illustrate the convergence of the $\alpha$, $\beta$, $\phi$, $\sigma^2$, $S$, and $\phi$ parameters over the iterations for each cluster.

It can be observed that the parameters converge to their true values around the 1000th iteration.

Model 2

The second model is an ARMAX(1,1) model of the form:

$y_{i,t} = \alpha_{S_i}y_{i,t-1} + \beta_{S_i}\epsilon_{t-1} + \gamma_{S_i}x_t + \epsilon_t$

Here, $\theta_k$ = $(\alpha_k,\beta_k,\gamma_k)$, where $\alpha_k$, $\beta_k$, and $\gamma_k$ are the AR(1), MA(1), and the coefficient for the exogenous variable, respectively, of the time series belonging to cluster $k$. The process $x_t$ is a random walk and is observed.

We set K = 3 and N = 100, and use the same priors as in Model 1.

Since $\epsilon_t \sim \mathcal{N}(0,\sigma^2)$, with $\sigma^2$ known, we have:

$y_{i,t} \mid y_{i,t-1},..,y_{i,0},\theta_{S_i} \sim \mathcal{N}(\alpha_{S_i}y_{i,t-1} + \beta_{S_i}\epsilon_{t-1} + \gamma_{S_i}x_t, \sigma^2)$

We are able to calculate the posteriors (4), (5), and (6) and estimate the parameters using the method described in Section 2.

The results obtained after 5000 iterations are as follows:

Coefficients	$\alpha$	$\beta$	$\gamma$	$\sigma^2$	Cluster Sizes
True Value	1, -0.8, 0.5	0.1, 0.1, 0.5	0.1, 0.5, 0.9	0.1 0.1 0.1	60, 20, 20
Estimated Value	0.99 -0.75 -0.50	0.11, 0.03 0.55	0.091, 0.497, 0.886	0.099 0.097 0.098	60, 20, 20
Error (%)	0.07% 6.88% 0.56%	14.29% 70.58% 9.35%	9.4%, 0.57%, 1.54%	0.46% 2.03% 1.32%	0%, 0%, 0%

Our model successfully assigned each series to its cluster, and it accurately estimated the $\alpha$ and $\gamma$ parameters within each cluster. However, it performed less well in estimating the $\beta$ (MA(1)) parameter. Perhaps with a more appropriate prior, the error could have been reduced.

The graphs below illustrate the convergence of the $\alpha$, $\beta$, $\gamma$, $\phi$, $\sigma^2$, $S$, and $\phi$ parameters over the iterations for each cluster.

It can be observed that the parameters converge to their true values around the 1000th iteration.

Conclusion

By using a Bayesian approach and Monte Carlo methods, we successfully grouped time series into different clusters. We observed that the estimations of the AR parameters were very close to the true values, and our algorithms made no classification errors. However, the estimations of the MA parameters were often less accurate. This difficulty arises from the challenges involved in estimating the parameters specific to ARIMA models. Additionally, we assumed that all parameters follow the same prior distribution

Bibliography

Frühwirth-Schnatter, and S. Kaufmann, (2008). Model-based clustering of multiple time-series, Journal of Business and Economic Statistics, 26, 78 – 89

Quants Quest

Signed Dual Attention Explained

Table of Content

How Rethinking Attention Led Me to SDA

How Signed Dual Attention Works

Mathematical Formulation

Link with Two-Head Attention

Future Directions

Estimating Pi (π) with Monte Carlo Simulation in Python

Table of Contents

Introduction

Monte Carlo Simulation Basics

Using Random Sampling to Estimate $\pi$

A Python Implementation

Conclusion

References and Further Reading

Monte Carlo Interview Questions

Is a Random Walk with Drift a Martingale ?

Table of Contents

Martingales, Random Walks and Drifts

Martingale

Random Walk

Random Walk with Drift

From Drift to Martingale

The Radon–Nikodym Bridge

A Practical Application

Link with the Stochastic Discount Factor

Conclusion

Further Reading

The SDF Explained: Why Factor Models Actually Work

Table of Contents

A Historical Perspective

SDF: The True Concept Behind Asset Prices

The Law of One Price

From the Law of One Price to the SDF

The SDF in Practice

Factor Models as SDFs in Disguise

From Factor Models to the SDF

Why Use Factor Models Instead of Directly Approximating the Stochastic Discount Factor?

Conclusion

Merger Arbitrage Explained

Disclaimer

Table of Contents

What Is Merger Arbitrage

Market Prices and Implied Probabilities

The Spreak is the Market-Priced Deal Risk

A Simple Formula

Merger Arbitrage Methodology

The Expected Value Framework

A Quick Example

Warner Bros & Paramount Merger

Deal Overview

Price-Based Probability Estimation

Sensitivity Analysis

What if the fallback increase ?

What if the deal price decrease ?

Getting the full picture

What Makes This Deal Tricky?

Where We Are Now

Conclusion

Anaconda to Create a Clean Python Environment

🐍 What is Anaconda?

🔧 Step 1: Install Anaconda

🧪 Step 2: Create a New Conda Environment

📂 Step 3: Activate Your Environment

📚 Step 4: Install Packages

💻 Step 5: Use the Environment in VS Code

📝 For Notebooks (.ipynb)

📄 For Python Scripts (.py)

Random Walk vs Martingale: What’s the Difference

Table of Contents

What is a Random Walk in Finance?

Intuition

Financial interpretation

Python example

What is a Random Walk in Stock Prices?

Intuition

Key difference from random walk

Financial Interpretation: The Role of the SDF and Change of Measure

What is the risk neutral probability