Achieving Financial Excellence: Portfolio Optimization in Python with Mean-Variance and Black-Litterman Models

This tutorial aims to guide you through the process of creating a portfolio optimization tool using Python. We will fetch historical stock log returns through the yfinance library and employ techniques like Mean-Variance Optimization or the Black-Litterman Model to find the optimal allocation of assets. We will use all SP100 tickers from Wikipedia as our dataset.

Photo by Campaign Creators on Unsplash

By the end of this tutorial, you will have a comprehensive understanding of portfolio optimization and how to implement it in Python using the cvxpy library. We will cover the following topics:

1. Introduction to Portfolio Optimization
2. Fetching Historical Stock Data with yfinance
3. Preprocessing the Data
4. Mean-Variance Optimization
5. The Black-Litterman Model
6. Building the Portfolio Optimization Tool

1. Introduction to Portfolio Optimization

Portfolio optimization is the process of selecting the best allocation of assets in a portfolio to achieve a desired objective. The objective can be maximizing returns, minimizing risk, or finding a balance between the two. The goal is to find the optimal combination of assets that maximizes returns while minimizing risk.

There are several techniques for portfolio optimization, but two popular approaches are Mean-Variance Optimization and the Black-Litterman Model.

Mean-Variance Optimization is based on the idea that investors are risk-averse and seek to maximize returns for a given level of risk. It involves calculating the expected returns and covariance matrix of the assets in the portfolio and finding the weights that minimize the portfolio’s variance.

The Black-Litterman Model is an extension of Mean-Variance Optimization that incorporates investor views and market equilibrium assumptions.

It allows investors to express their opinions on the expected returns of assets and combines them with market expectations to find the optimal portfolio allocation.

In this tutorial, we will explore both Mean-Variance Optimization and the Black-Litterman Model and implement them using the cvxpy library.

2. Fetching Historical Stock Data with yfinance

Before we can start optimizing our portfolio, we need historical stock data for the assets in our portfolio. We will use the yfinance library to fetch this data directly in Python.

To install yfinance, run the following command:

!pip install yfinance

Once installed, we can use the library to fetch historical stock data. Let’s start by importing the necessary libraries and fetching the data for all SP100 tickers from Wikipedia.

import yfinance as yf
import pandas as pd
import numpy as np

# Fetch SP100 tickers from Wikipedia
url = 'https://en.wikipedia.org/wiki/S%26P_100'
tables = pd.read_html(url)
sp100_tickers = tables[2]['Symbol'].tolist()

# Fetch historical stock data
start_date = '2018-01-01'
end_date = '2023-07-30'

data = yf.download(sp100_tickers, start=start_date, end=end_date)['Adj Close'].dropna(how='all', axis=1)

In the code above, we first fetch the SP100 tickers from the Wikipedia page using the pd.read_html() function. We then extract the ticker symbols from the table and store them in the sp100_tickers list.

Next, we define the start and end dates for the historical data we want to fetch. In this example, we are fetching data from January 1, 2018, to June 30, 2023.

Finally, we use the yf.download() function to fetch the historical stock data for all the tickers in the sp100_tickers list. We specify the start and end dates using the start and end parameters and we select the 'Adj Close' column of the resulting DataFrame.

Let’s take a look at the first few rows of the data:

data.head()

First rows of the data. Created by Author

The DataFrame contains the adjusted closing prices for each ticker on each trading day within the specified date range. We will use this data to calculate the log returns and perform portfolio optimization.

3. Preprocessing the Data

Before we can perform portfolio optimization, we need to preprocess the data. This involves calculating the log returns of the assets and handling any missing values.

Let’s start by calculating the log returns. Log returns are commonly used in finance because they have several desirable properties, including additive properties and symmetry.

# Calculate log returns
returns = data.pct_change().dropna().apply(lambda x: np.log(1 + x))

In the code above, we use the pct_change() function to calculate the percentage change between consecutive rows of the DataFrame. We then drop the first row, which contains NaN values, using the dropna() function. Finally, we apply the np.log(1 + x) function to each element of the DataFrame to calculate the log returns.

Next, let’s handle any missing values in the data. Missing values can occur due to corporate actions such as stock splits or mergers. We will use the fillna() function to replace missing values with the mean of the respective column.

# Fill NaN values with mean return
returns = returns.fillna(returns.mean())

Now that we have preprocessed the data, we are ready to perform portfolio optimization.

4. Mean-Variance Optimization

Mean-Variance Optimization is a popular technique for portfolio optimization. It involves finding the weights that minimize the portfolio’s variance for a given level of expected return.

To implement Mean-Variance Optimization, we will use the cvxpy library. cvxpy is a Python-embedded modeling language for convex optimization problems. It provides a simple and intuitive syntax for formulating and solving optimization problems.

Let’s start by installing cvxpy:

!pip install cvxpy

Once installed, we can use cvxpy to perform Mean-Variance Optimization. Here's an example of how to do it:

import cvxpy as cp

# Number of assets
n = len(returns.columns)

# Expected returns
mu = returns.mean().values

# Covariance matrix
Sigma = np.cov(returns, rowvar=False).astype(np.float64) 

# Variable for the portfolio weights
w = cp.Variable(n)

# Portfolio expected return
expected_return = mu @ w

# Portfolio variance
variance = cp.quad_form(w, Sigma)

# Target expected return
target_return = 0.05

# Problem constraints
constraints = [
    cp.sum(w) == 1,  # Sum of weights equals 1
    expected_return >= target_return  # Target expected return constraint
]

# Problem objective
objective = cp.Minimize(variance)

# Solve the problem
problem = cp.Problem(objective, constraints)
problem.solve()

# Optimal portfolio weights
optimal_weights = w.value

In the code above, we first define the number of assets in the portfolio (n), the expected returns (mu) and the covariance matrix (Sigma) using the preprocessed data.

Next, we define a cp.Variable object w to represent the portfolio weights. We then define the portfolio expected return (expected_return) as the dot product of the expected returns and the weights and the portfolio variance (variance) as the quadratic form of the weights and the covariance matrix.

We also define a target expected return (target_return) and specify the problem constraints. In this example, we require the sum of the weights to equal 1 and the expected return to be greater than or equal to the target return.

Finally, we define the problem objective as minimizing the portfolio variance and solve the problem using problem.solve(). The optimal portfolio weights are stored in the w.value attribute.

Let’s visualize the optimal portfolio weights:

import matplotlib.pyplot as plt

# Plot optimal portfolio weights
plt.figure(figsize=(10, 6))
plt.bar(returns.columns, optimal_weights)
plt.xlabel('Asset')
plt.ylabel('Weight')
plt.title('Optimal Portfolio Weights')
plt.xticks(rotation=90)

plt.show()

Figure 1: Optimal portfolio weights. Created by Author

The bar plot shows the optimal weights for each asset in the portfolio. The weights represent the proportion of the portfolio allocated to each asset. As expected, the weights sum up to 1.

5. The Black-Litterman Model

The Black-Litterman Model is an extension of Mean-Variance Optimization that incorporates investor views and market equilibrium assumptions. It allows investors to express their opinions on the expected returns of assets and combines them with market expectations to find the optimal portfolio allocation.

To implement the Black-Litterman Model, we need to specify the investor views and the market equilibrium assumptions. We also need to estimate the covariance matrix of the asset returns.

Let’s start by estimating the covariance matrix using the Ledoit-Wolf shrinkage method. This method is commonly used to improve the estimation of the covariance matrix when the number of assets is large compared to the number of observations.

from sklearn.covariance import LedoitWolf

# Estimate covariance matrix using Ledoit-Wolf shrinkage
cov_estimator = LedoitWolf()
cov_estimator.fit(returns)
Sigma = cov_estimator.covariance_

In the code above, we use the LedoitWolf class from the sklearn.covariance module to estimate the covariance matrix. We fit the estimator to the log returns data and store the estimated covariance matrix in the Sigma variable.

Next, let’s specify the investor views and the market equilibrium assumptions. For simplicity, we will assume that the investor has a neutral view on all assets and that the market is in equilibrium.

# Investor views
views = np.zeros(n)

# View uncertainty
view_uncertainty = np.eye(n) * 0.1

# Market equilibrium returns
market_equilibrium_returns = returns.mean().values

In the code above, we initialize the investor views (views) as an array of zeros. We also specify the view uncertainty (view_uncertainty) as a diagonal matrix with small values. Finally, we set the market equilibrium returns (market_equilibrium_returns) as the mean returns of the assets.

Now, let’s implement the Black-Litterman Model using cvxpy:

# Variable for the portfolio weights
w = cp.Variable(n)

# Portfolio expected return
expected_return = mu @ w

# Portfolio variance
variance = cp.quad_form(w, Sigma)

# Investor views
P = np.eye(n)
Q = views

# View uncertainty
Omega = view_uncertainty

# Market equilibrium returns
Pi = market_equilibrium_returns

# Black-Litterman model
tau = 0.1
Omega_inv = np.linalg.inv(Omega)
Sigma_inv = np.linalg.inv(Sigma)

# Posterior expected returns
posterior_return = cp.inv_pos(cp.inv_pos(tau * Sigma_inv) + P.T @ Omega_inv @ P) @ (cp.inv_pos(tau * Sigma_inv) @ Pi + P.T @ Omega_inv @ Q)

# Problem constraints
constraints = [
    cp.sum(w) == 1,  # Sum of weights equals 1
    expected_return >= target_return  # Target expected return constraint
]

# Problem objective
objective = cp.Minimize(variance)

# Solve the problem
problem = cp.Problem(objective, constraints)
problem.solve()

# Optimal portfolio weights
optimal_weights = w.value

In the code above, we first define the variables and parameters for the Black-Litterman Model. We then calculate the posterior expected returns (posterior_return) using the Black-Litterman formula.

Next, we specify the problem constraints and objective and solve the problem using problem.solve(). The optimal portfolio weights are stored in the w.value attribute.

Let’s visualize the optimal portfolio weights:

# Plot optimal portfolio weights
plt.figure(figsize=(10, 6))
plt.bar(returns.columns, optimal_weights)
plt.xlabel('Asset')
plt.ylabel('Weight')
plt.title('Optimal Portfolio Weights (Black-Litterman Model)')
plt.xticks(rotation=90)

plt.show()

Figure 2: Optimal portfolio weights using the Black-Litterman Model. Created by Author

The bar plot shows the optimal weights for each asset in the portfolio, taking into account the investor views and market equilibrium assumptions. Compared to the Mean-Variance Optimization, the weights may differ due to the additional information incorporated into the model.

6. Building the Portfolio Optimization Tool

Now that we have implemented Mean-Variance Optimization and the Black-Litterman Model, let’s build a portfolio optimization tool that allows us to interactively explore different portfolio allocations.

We will use the ipywidgets library to create interactive widgets and the matplotlib library to visualize the results.

import ipywidgets as widgets

def optimize_portfolio(target_return):
    # Mean-Variance Optimization
    w_mv = cp.Variable(n)
    expected_return_mv = mu @ w_mv
    variance_mv = cp.quad_form(w_mv, Sigma)
    constraints_mv = [
        cp.sum(w_mv) == 1,
        expected_return_mv >= target_return
    ]
    objective_mv = cp.Minimize(variance_mv)
    problem_mv = cp.Problem(objective_mv, constraints_mv)
    problem_mv.solve()
    optimal_weights_mv = w_mv.value

    # Black-Litterman Model
    w_bl = cp.Variable(n)
    expected_return_bl = mu @ w_bl
    variance_bl = cp.quad_form(w_bl, Sigma)
    posterior_return_bl = cp.inv_pos(cp.inv_pos(tau * Sigma_inv) + P.T @ Omega_inv @ P) @ (cp.inv_pos(tau * Sigma_inv) @ Pi + P.T @ Omega_inv @ Q)
    constraints_bl = [
        cp.sum(w_bl) == 1,
        expected_return_bl >= target_return
    ]
    objective_bl = cp.Minimize(variance_bl)
    problem_bl = cp.Problem(objective_bl, constraints_bl)
    problem_bl.solve()
    optimal_weights_bl = w_bl.value

    # Plot optimal portfolio weights
    plt.figure(figsize=(10, 6))
    width = 0.35
    x = np.arange(len(returns.columns))
    plt.bar(x - width/2, optimal_weights_mv, width, label='Mean-Variance Optimization')
    plt.bar(x + width/2, optimal_weights_bl, width, label='Black-Litterman Model')
    plt.xlabel('Asset')
    plt.ylabel('Weight')
    plt.title(f'Optimal Portfolio Weights (Target Return: {target_return:.2%})')
    plt.xticks(x, returns.columns, rotation=90)
    plt.legend()
    
    plt.show()

# Create target return slider
target_return_slider = widgets.FloatSlider(
    value=0.05,
    min=returns.mean().min(),
    max=returns.mean().max(),
    step=0.01,
    description='Target Return:',
    readout_format='.2%',
    layout=widgets.Layout(width='50%')
)

# Create portfolio optimization button
optimize_button = widgets.Button(description='Optimize')

# Define button click event handler
def on_optimize_button_clicked(b):
    target_return = target_return_slider.value
    optimize_portfolio(target_return)

# Register button click event handler
optimize_button.on_click(on_optimize_button_clicked)

# Display widgets
widgets.VBox([target_return_slider, optimize_button])

The code above defines a function optimize_portfolio() that takes a target return as input and performs both Mean-Variance Optimization and the Black-Litterman Model. It then plots the optimal portfolio weights for comparison.

We also create a slider widget target_return_slider to interactively select the target return and a button widget optimize_button to trigger the portfolio optimization. When the button is clicked, the on_optimize_button_clicked() function is called, which retrieves the target return from the slider and calls optimize_portfolio().

Target Return Slider. Created by Author.

To run the portfolio optimization tool, execute the following code:

# As example - 5% of target return
optimize_portfolio(0.05)

This will display the target return slider and the optimize button. Adjust the target return using the slider and click the optimize button to see the optimal portfolio weights for the selected target return.

Figure 3: Optimal portfolio weights using the Black-Litterman Model. Created by Author

Conclusion

In this tutorial, we have explored the concept of portfolio optimization and implemented two popular techniques, Mean-Variance Optimization and the Black-Litterman Model, using Python and the cvxpy library. We fetched historical stock data using the yfinance library, preprocessed the data and performed portfolio optimization to find the optimal allocation of assets.

Portfolio optimization is a powerful tool for investors and financial professionals to make informed investment decisions. By considering the expected returns and risks of different assets, portfolio optimization can help maximize returns while minimizing risk.

I hope this tutorial has provided you with a comprehensive understanding of portfolio optimization and how to implement it in Python. You can further enhance the portfolio optimization tool by adding additional constraints or incorporating other optimization techniques.

Become a Medium member today and enjoy unlimited access to thousands of Python guides and Data Science articles! For just $5 a month, you’ll have access to exclusive content and support as a writer. Sign up now using my link and I’ll earn a small commission at no extra cost to you.