Reproduced from my book “Python for Quantitative Finance“

10.2 Introduction: The Only “Free Lunch”

Harry Markowitz won a Nobel Prize for a simple insight: You should not look at assets in isolation.

If Asset A goes up when Asset B goes down (Negative Correlation), holding both reduces your risk without necessarily reducing your return. This reduction in risk is called Diversification, and it is often called “the only free lunch in finance.”

In this chapter, we stop asking “Is Apple a good stock?” and start asking “Does Apple improve my portfolio?”

import pandas as pd
import numpy as np
import yfinance as yf
import matplotlib.pyplot as plt
import scipy.optimize as sco
 
# Set style
plt.style.use('seaborn-v0_8-darkgrid')

10.3 The Mathematics of MPT

To optimize a portfolio, we need to express its Return and Risk mathematically.

1. Portfolio Return

2. Portfolio Variance (The Tricky Part)

10.4 Simulating the Efficient Frontier

The Efficient Frontier is the set of optimal portfolios that offer the highest expected return for a defined level of risk.

To see it, we will simulate 5,000 random portfolios composed of 4 tech stocks.

Step 1: Data Preparation

We need Expected Returns ($mu$) and the Covariance Matrix ($Sigma$).

# 1. Fetch Data
tickers = ['AAPL', 'MSFT', 'GOOG', 'AMZN']
data = yf.download(tickers, start='2020-01-01', end='2023-12-31')['Close']
returns = data.pct_change().dropna()
 
# 2. Annualized Statistics
# Expected Return (Mean daily return * 252)
mean_returns = returns.mean() * 252
 
# Covariance Matrix (Daily Covariance * 252)
cov_matrix = returns.cov() * 252
 
print("--- Covariance Matrix (Annualized) ---")
print(cov_matrix)

Step 2: The Simulation Loop

We generate random weights, normalize them so they sum to 100%, and calculate the Risk/Return for each combination.

num_portfolios = 5000
results = np.zeros((3, num_portfolios)) # Rows: [Return, Volatility, Sharpe]
 
np.random.seed(42)
 
for i in range(num_portfolios):
    # 1. Generate Random Weights
    weights = np.random.random(4)
    weights /= np.sum(weights) # Normalize to sum to 1
    
    # 2. Calculate Portfolio Return
    p_return = np.sum(mean_returns * weights)
    
    # 3. Calculate Portfolio Volatility (Std Dev)
    # Formula: sqrt(w.T * Cov * w)
    p_std = np.sqrt(np.dot(weights.T, np.dot(cov_matrix, weights)))
    
    # 4. Store Results
    results[0,i] = p_return
    results[1,i] = p_std
    # Sharpe Ratio (assuming Rf=0 for simplicity)
    results[2,i] = results[0,i] / results[1,i]
 
# Convert to DataFrame for easier plotting
results_df = pd.DataFrame(results.T, columns=['Return', 'Volatility', 'Sharpe'])

Step 3: Visualizing the Frontier

The resulting plot usually looks like a bullet or a parabola on its side.

plt.figure(figsize=(10, 6))
 
# Scatter plot colored by Sharpe Ratio
plt.scatter(results_df['Volatility'], results_df['Return'], 
            c=results_df['Sharpe'], cmap='viridis', marker='o', s=10, alpha=0.5)
 
plt.colorbar(label='Sharpe Ratio')
plt.xlabel('Volatility (Risk)')
plt.ylabel('Expected Return')
plt.title('The Efficient Frontier (Monte Carlo Simulation)')
 
# Mark the Best Portfolio found in simulation
max_sharpe_idx = results_df['Sharpe'].idxmax()
max_sharpe_port = results_df.iloc[max_sharpe_idx]
 
plt.scatter(max_sharpe_port['Volatility'], max_sharpe_port['Return'], 
            c='red', s=100, marker='*', label='Max Sharpe (Simulated)')
 
plt.legend()
plt.show()

Interpretation:

• The Edge: The top-left boundary of the blob is the Efficient Frontier. You cannot find a portfolio to the left of this line (it’s impossible to have less risk for that return).
• Inefficient: Any point inside the blob is inefficient. Why hold a portfolio with 20% risk and 10% return if another portfolio exists with 20% risk and 15% return?

Check Your Understanding

• Exercise 10.2 (Frontier Logic):

Looking at the Efficient Frontier plot, why does the curve bend backwards (convex) instead of being a straight line?

(Hint: Think about correlation.)

# Solution 10.1

# No. That formula ignores the Covariance (interaction) between assets.

# The correct formula includes the term: + 2 * w1 * w2 * Cov(A, B).

# If Covariance is negative, the risk is LOWER than the weighted average.

# Solution 10.2

# The bend represents the benefit of Diversification.

# Because assets are not perfectly correlated (Corr < 1), combining them cancels out some risk.

# This “cancellation” pulls the curve to the left (lower volatility), creating the bulge.

10.5 Portfolio Optimization (The Solver)

Simulating 5,000 portfolios gives us a good guess, but it’s not precise. To find the mathematically optimal weights, we need a Solver.

We use the scipy.optimize library. Since scipy only has a “minimize” function, we have to be clever:

• To Maximize Sharpe Ratio: We Minimize the Negative Sharpe Ratio.
• To Minimize Volatility: We Minimize the Standard Deviation.

1. Defining the Objective Functions

First, we create a helper function that takes Weights as input and returns the portfolio’s Return, Volatility, and Sharpe Ratio.

def get_portfolio_stats(weights, mean_returns, cov_matrix, rf_rate=0):
    """
    Returns [Portfolio Return, Portfolio Volatility, Sharpe Ratio]
    """
    weights = np.array(weights)
    port_return = np.sum(mean_returns * weights)
    port_volatility = np.sqrt(np.dot(weights.T, np.dot(cov_matrix, weights)))
    
    # Sharpe Ratio (Return - Rf) / Volatility
    sharpe = (port_return - rf_rate) / port_volatility
    
    return [port_return, port_volatility, sharpe]
 
# Objective Function 1: Negative Sharpe (for Maximization)
def neg_sharpe(weights, mean_returns, cov_matrix, rf_rate=0):
    # We return the negative because scipy minimizes
    return -get_portfolio_stats(weights, mean_returns, cov_matrix, rf_rate)[2]
 
# Objective Function 2: Volatility (for Minimization)
def minimize_volatility(weights, mean_returns, cov_matrix):
    return get_portfolio_stats(weights, mean_returns, cov_matrix)[1]

2. Constraints and Bounds

Optimization requires rules:

• Constraints: The weights must sum to 1 (100% invested).
  • type: eq means “Equality Constraint”.
  • fun: lambda x: np.sum(x) – 1 means the result must be 0.
• Bounds: No short selling (Weights between 0 and 1).
  • If we wanted short selling, we would set bounds to (-1, 1).

# Number of assets
num_assets = len(tickers)
 
# 1. Constraints: Sum of weights = 1
constraints = ({'type': 'eq', 'fun': lambda x: np.sum(x) - 1})
 
# 2. Bounds: Each weight between 0 and 1
bounds = tuple((0, 1) for _ in range(num_assets))
 
# 3. Initial Guess: Equal weights (Start the search here)
init_guess = num_assets * [1. / num_assets,]

3. Running the Optimization

We use the Sequential Least Squares Programming (SLSQP) algorithm, which handles constraints well.

A. Finding the Maximum Sharpe Portfolio

This is the portfolio a rational investor should hold (The “Tangency Portfolio”).

import scipy.optimize as sco
 
# Minimize the Negative Sharpe
opt_sharpe = sco.minimize(neg_sharpe, 
                          init_guess, 
                          args=(mean_returns, cov_matrix), 
                          method='SLSQP', 
                          bounds=bounds, 
                          constraints=constraints)
 
# Extract Results
max_sharpe_weights = opt_sharpe.x
max_sharpe_stats = get_portfolio_stats(max_sharpe_weights, mean_returns, cov_matrix)
 
print("--- Maximum Sharpe Portfolio ---")
print(f"Return:     {max_sharpe_stats[0]*100:.2f}%")
print(f"Volatility: {max_sharpe_stats[1]*100:.2f}%")
print(f"Sharpe:     {max_sharpe_stats[2]:.2f}")
print("\nOptimal Weights:")
for ticker, weight in zip(tickers, max_sharpe_weights):
    print(f"{ticker}: {weight*100:.2f}%")

B. Finding the Minimum Volatility Portfolio

This is the safest possible portfolio on the Efficient Frontier.

# Minimize Volatility
opt_vol = sco.minimize(minimize_volatility, 
                       init_guess, 
                       args=(mean_returns, cov_matrix), 
                       method='SLSQP', 
                       bounds=bounds, 
                       constraints=constraints)
 
min_vol_weights = opt_vol.x
min_vol_stats = get_portfolio_stats(min_vol_weights, mean_returns, cov_matrix)
 
print("\n--- Minimum Volatility Portfolio ---")
print(f"Return:     {min_vol_stats[0]*100:.2f}%")
print(f"Volatility: {min_vol_stats[1]*100:.2f}%")
print(f"Sharpe:     {min_vol_stats[2]:.2f}")

10.6 Visualizing the Optimal Portfolios

Now we can plot these precise points on top of our previous Monte Carlo cloud to see exactly where they sit on the frontier.

plt.figure(figsize=(10, 6))
 
# 1. Plot the Monte Carlo Cloud (from Part 1)
plt.scatter(results_df['Volatility'], results_df['Return'], 
            c=results_df['Sharpe'], cmap='viridis', marker='o', s=10, alpha=0.3)
 
# 2. Plot Max Sharpe (Red Star)
plt.scatter(max_sharpe_stats[1], max_sharpe_stats[0], 
            marker='*', color='red', s=200, label='Max Sharpe (Optimized)')
 
# 3. Plot Min Volatility (Blue Star)
plt.scatter(min_vol_stats[1], min_vol_stats[0], 
            marker='*', color='blue', s=200, label='Min Volatility (Optimized)')
 
plt.title("Efficient Frontier with Optimal Portfolios")
plt.xlabel("Volatility")
plt.ylabel("Return")
plt.colorbar(label='Sharpe Ratio')
plt.legend()
plt.show()

Financial Insight:

• Notice that Max Sharpe is usually higher up the curve (taking more risk for better reward).
• Min Volatility is at the absolute leftmost tip of the “bullet.”
• Often, the optimizer will set some weights to 0.00%. This is MPT telling you: “Amazon is not efficient right now; sell it all.”

# Solution 10.3

# The scipy library function is called ‘minimize’. It finds the lowest value.

# To find the maximum of a function f(x), we must minimize -f(x).

# Solution 10.4

# Constraints: Sum of weights = 0 (instead of 1).

# Bounds: (-1, 1) to allow short selling.

# Note: You would likely also add a “Gross Exposure” constraint (sum of absolute weights = 1 or 2)

# to prevent the optimizer from taking infinite leverage.

10.7 The Capital Asset Pricing Model (CAPM)

Modern Portfolio Theory tells us that there are two types of risk:

1. Idiosyncratic Risk: Specific to a company (e.g., Apple’s CEO quits). This can be diversified away by holding many stocks.
2. Systematic Risk: Specific to the market (e.g., Interest rates rise). This cannot be diversified away.

Since you can easily eliminate Idiosyncratic Risk for free (by buying an index fund), the market should not pay you for taking it. The market only compensates you for taking Systematic Risk.

10.8 Calculating Beta and Alpha

1. Implementation using scipy.stats

We will calculate the Beta of Nvidia (NVDA) relative to the S&P 500 (SPY).

import yfinance as yf
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
 
# 1. Fetch Data (Stock vs. Benchmark)
tickers = ['NVDA', 'SPY']
data = yf.download(tickers, start='2020-01-01', end='2023-12-31')['Close']
returns = data.pct_change().dropna()
 
# 2. Prepare Regression Arrays
# X = Market Returns (Independent Variable)
# Y = Stock Returns (Dependent Variable)
X = returns['SPY'].values
Y = returns['NVDA'].values
 
# 3. Perform Linear Regression
slope, intercept, r_value, p_value, std_err = stats.linregress(X, Y)
 
beta = slope
alpha = intercept
r_squared = r_value ** 2
 
print(f"--- CAPM Regression Results (NVDA vs SPY) ---")
print(f"Beta (Systematic Risk): {beta:.4f}")
print(f"Alpha (Daily Excess):   {alpha:.5f}")
print(f"R-Squared (Fit):        {r_squared:.4f}")

Interpretation:

• Beta: You will likely see a value around 1.5 – 2.0. This means NVDA is “High Beta.” It crashes harder than the market but rallies harder too.
• Alpha: If this is close to zero, NVDA’s returns are fully explained by its risk. If positive, it “beat the market” on a risk-adjusted basis.

2. Visualizing the Regression Line

The regression line is often called the Security Characteristic Line.

plt.figure(figsize=(10, 6))
 
# 1. Scatter Plot of Daily Returns
plt.scatter(X, Y, alpha=0.5, color='blue', label='Daily Returns')
 
# 2. Regression Line (y = mx + b)
# We generate points for the line using the min/max of X
x_line = np.linspace(X.min(), X.max(), 100)
y_line = (beta * x_line) + alpha
 
plt.plot(x_line, y_line, color='red', linewidth=2, label=f'Regression Line (Beta={beta:.2f})')
 
plt.title("CAPM Regression: NVDA vs. S&P 500")
plt.xlabel("Market Return (SPY)")
plt.ylabel("Asset Return (NVDA)")
plt.legend()
plt.grid(True)
plt.show()

3. The Rolling Beta

Just like volatility, Beta is not constant. In a recession, a “Growth Stock” might suddenly behave like a “Value Stock” or vice versa.

# Calculate Rolling 60-Day Beta
# Covariance / Variance
rolling_cov = returns['NVDA'].rolling(window=60).cov(returns['SPY'])
rolling_var = returns['SPY'].rolling(window=60).var()
 
rolling_beta = rolling_cov / rolling_var
 
plt.figure(figsize=(10, 5))
rolling_beta.plot(color='purple', label='60-Day Rolling Beta')
plt.axhline(beta, color='black', linestyle='--', label='Average Beta')
plt.title("Time-Varying Risk: Rolling Beta of NVDA")
plt.legend()
plt.show()

Check Your Understanding

• Exercise 10.5 (Alpha Logic):

Fund Manager A returns 15% this year. The Market returns 10%. The Risk-Free rate is 0%.

However, Manager A has a Beta of 2.0.

According to CAPM, did the manager generate positive Alpha (Skill)?

(Hint: Calculate the Expected Return first).

• Exercise 10.6 (Beta Hedging):

You own a $100,000 portfolio of Tech stocks with a Beta of 1.5.

You are worried about a market crash. You want to short the S&P 500 (Beta = 1.0) to make your portfolio “Market Neutral” (Beta = 0).

How much dollar value of S&P 500 must you short?

# Solution 10.5

# CAPM Expected Return = Rf + Beta * (Rm – Rf)

# E[R] = 0% + 2.0 * (10% – 0%) = 20%.

# The manager delivered 15%.

# Alpha = Actual – Expected = 15% – 20% = -5%.

# Conclusion: The manager UNDERPERFORMED. They took enough risk to earn 20%, but only got 15%.

# Solution 10.6

# To neutralize Beta: (Value_Port * Beta_Port) + (Value_Hedge * Beta_Hedge) = 0

# (100,000 * 1.5) + (Value_Hedge * -1.0) = 0

# 150,000 – Value_Hedge = 0

# Value_Hedge = $150,000.

# You must short $150k worth of SPY to offset the magnified risk of your $100k tech portfolio.

10.9 Mini-Project: The Robo-Advisor Engine

In the real world, clients don’t ask for “The Tangency Portfolio.” They say, “I am conservative,” or “I am aggressive.”

A Robo-Advisor translates these qualitative statements into quantitative constraints.

Objective:

Build a function that takes a Risk Tolerance (Low, Medium, High) and optimizes a portfolio of Stocks, Bonds, and Gold to find the highest return possible for that specific risk level.

The Asset Universe:

1. Stocks (SPY): High Return, High Risk.
2. Bonds (TLT): Low Return, Low Risk, Negative Correlation.
3. Gold (GLD): Non-correlated inflation hedge.

Step 1: Data Setup

We fetch the data and calculate the necessary matrices.

import yfinance as yf
import pandas as pd
import numpy as np
import scipy.optimize as sco
import matplotlib.pyplot as plt
 
# 1. Fetch Data
tickers = ['SPY', 'TLT', 'GLD']
data = yf.download(tickers, start='2020-01-01', end='2023-12-31', progress=False)['Close']
returns = data.pct_change().dropna()
 
# 2. Statistics (Annualized)
mean_returns = returns.mean() * 252
cov_matrix = returns.cov() * 252
 
print("--- Annualized Expected Returns ---")
print(mean_returns)

Step 2: The Optimizer Logic

Instead of maximizing the Sharpe Ratio (which finds only one optimal point), we will Maximize Return subject to a Target Volatility.

• Low Risk: Target Volatility = 5%
• Medium Risk: Target Volatility = 10%
• High Risk: Target Volatility = 15% (or Unconstrained Max Return)

def get_port_stats(weights):
    weights = np.array(weights)
    ret = np.sum(mean_returns * weights)
    vol = np.sqrt(np.dot(weights.T, np.dot(cov_matrix, weights)))
    return ret, vol
 
def robo_advisor(target_volatility):
    num_assets = len(tickers)
    
    # Objective: Minimize Negative Return (aka Maximize Return)
    def neg_return(weights):
        return -get_port_stats(weights)[0]
    
    # Constraints
    constraints = (
        {'type': 'eq', 'fun': lambda x: np.sum(x) - 1},              # Sum weights = 1
        {'type': 'eq', 'fun': lambda x: get_port_stats(x)[1] - target_volatility} # Vol = Target
    )
    
    # Bounds (0% to 100% allocation, no shorting)
    bounds = tuple((0, 1) for _ in range(num_assets))
    
    # Initial Guess
    init_guess = num_assets * [1. / num_assets,]
    
    # Run Optimization
    result = sco.minimize(neg_return, init_guess, method='SLSQP', 
                          bounds=bounds, constraints=constraints)
    
    return result.x

Step 3: Running the Profiles

Let’s see how the allocations shift based on risk tolerance.

# Define Risk Profiles (Target Volatility)
profiles = {
    "Conservative (Vol=5%)": 0.05,
    "Balanced (Vol=10%)": 0.10,
    "Aggressive (Vol=15%)": 0.15
}
 
allocations = {}
 
print("--- Robo-Advisor Recommendations ---")
for profile, target_vol in profiles.items():
    try:
        weights = robo_advisor(target_vol)
        allocations[profile] = weights
        
        # Print Result
        ret, vol = get_port_stats(weights)
        print(f"\n{profile}:")
        print(f"  Exp Return: {ret*100:.2f}%")
        print(f"  Allocation: SPY {weights[1]*100:.0f}% | TLT {weights[2]*100:.0f}% | GLD {weights[0]*100:.0f}%")
        
    except Exception as e:
        print(f"Could not solve for {profile}. Target might be infeasible.")

Interpretation:

1. Conservative: Should be heavy on TLT (Bonds).
2. Aggressive: Should be heavy on SPY (Stocks).
3. Balanced: A mix that leverages the negative correlation between SPY and TLT.

Step 4: Visualizing the Allocations

A pie chart dashboard helps clients understand where their money is going.

fig, axes = plt.subplots(1, 3, figsize=(15, 5))
 
for i, (profile, weights) in enumerate(allocations.items()):
    # We use tickers order from yfinance download (alphabetical: GLD, SPY, TLT)
    # Check your specific download order! yfinance sorts columns alphabetically.
    # In this run: GLD (0), SPY (1), TLT (2)
    labels = ['GLD', 'SPY', 'TLT']
    
    axes[i].pie(weights, labels=labels, autopct='%1.1f%%', startangle=90, colors=['gold', 'green', 'blue'])
    axes[i].set_title(profile)
 
plt.tight_layout()
plt.show()

10.10 Chapter Summary

In this chapter, we optimized the trade-off between Risk and Return.

1. Diversification: We proved mathematically that adding uncorrelated assets reduces portfolio variance without sacrificing return.
2. Efficient Frontier: We used Monte Carlo simulation to visualize the boundary of possible portfolios.
3. Optimization: We used scipy.optimize to solve for specific points on the frontier:
   1. Max Sharpe: The “Tangency Portfolio” (Best risk-adjusted return).
   2. Min Volatility: The safest possible portfolio.
4. CAPM: We decomposed return into Alpha (Skill) and Beta (Market Exposure), learning that the market only pays for Systematic Risk.
5. Robo-Advisor: We built an engine that reverse-engineers the optimal weights for a specific risk target.

Coming Up Next:

We have covered fundamental strategies (Trend Following, Mean Reversion) and Portfolio Theory.

In Chapter 11: Algorithmic Trading and Backtesting, we will build a professional-grade Backtesting Engine. We will learn how to simulate realistic trading (including transaction costs) to prove if a strategy actually makes money.

10.11 Review Questions

• Constraint Logic: In our Robo-Advisor, if you ask for a Target Volatility of 1% (extremely low), the optimizer might fail. Why?

a) The code is buggy.

b) It is mathematically impossible to construct a portfolio with 1% volatility using these specific assets (even the safest asset, Bonds, is more volatile than 1%).

c) You need to add Crypto.

• CAPM in Portfolio Theory: If you add a “High Alpha” stock to your portfolio, what happens to the Efficient Frontier?

a) It does nothing.

b) It shifts the frontier upward (higher return for same risk).

c) It shifts the frontier downward.

• Correlation: Which pair of assets is most effective for diversification?

a) Two assets with correlation +0.9.

b) Two assets with correlation 0.0.

c) Two assets with correlation -0.5.

• Coding Challenge:

Write a scipy constraint dictionary that ensures the weight of SPY (index 1) never exceeds 50% of the portfolio.

Answers

1: (b) Infeasible constraint. You cannot create a portfolio safer than the safest combination of available assets (unless you hold Cash).

2: (b) Shifts upward. Positive Alpha means “Free Return,” improving the efficiency of the entire set.

3: (c) Correlation -0.5. Negative correlation provides the strongest hedging benefit.

4:

# Inequality constraint: 0.5 – weight >= 0 (so weight <= 0.5)

{‘type’: ‘ineq’, ‘fun’: lambda x: 0.5 – x[1]}