Forecasting Financial Markets Testing Strategies
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 1
Overview Measuring Estimator Performance ¾ Measuring Profitability ¾ Equity Curve Measures ¾ Portfolio Performance Measures ¾
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 2
Measuring Estimator Performance Correlation coefficient ¾ Theil’s information coefficient ¾ Akaike information criteria ¾ Schwartz Bayesian information criterion ¾ Average relative variance ¾ Directional change predictor ¾ Bull-bear statistic ¾
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 3
Correlation Coefficient ¾
Most Common Measure of Prediction Accuracy 2 ⎧⎪ n ⎫ ⎨∑ ( yi − y )( f i − f )⎬ ⎪⎩ i =1 ⎭ 2 R = n n 2 2 ∑ ( yi − y ) ∑ ( f i − f ) i =1
¾
i =1
Adjusted R2 = R2 (n+k) / (n-k) Penalizes for model complexity • k is the number of model parameters
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 4
Theil’s Information Coefficient ¾
Advantages vs. standard measures Comparison with naïve forecast (ft+1 = yt ) Considers disproportionate cost of large errors (in MSE) 2
⎛ f t +1 − yt +1 ⎞ ⎜⎜ ⎟⎟ ( FPEt +1 − APEt +1 ) ∑ ∑ yt t =1 ⎝ ⎠ t =1 = U= n −1 2 n − 1 2 ⎛ yt +1 − yt ⎞ APE ∑ t +1 ⎜⎜ ⎟⎟ ∑ t =1 yt ⎠ t =1 ⎝ n −1
n −1
2
FPEt+1 = (ft+1 - yt ) / yt (forecast relative change) APEt+1 = (yt+1 - yt ) / yt (actual relative change) Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 5
Interpretation of Theil’s U-Statistic ¾
U=1 Naïve method is as good as technique being evaluated • FPEt+1 = 0; so ft+1 = yt , as for naïve method
¾
U<1 Technique is better than naïve method • Since FPEt+1 < APEt+1 • Smaller U the better
¾
U>1 Naïve method will produce better results • Since FPEt+1 > APEt+1
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 6
Average Relative Variance (Mean Reversion) ¾
Trivial Predictor: Historical Mean n
Tµ =
∑(y t =1
t
− ft )
n
∑(y t =1
t +1
2
− y)
2
• With Tµ < 1 the forecasting method is making better predictions than simply predicting the mean Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 7
Akaike Information Criteria ¾
Adjusts MSE to take account of complexity of estimator
1 n 2 ⎡n + k ⎤ A = ∑ ( yi − f i ) ⎢ ⎥ − n i =1 n k ⎣ ⎦ K is the # free parameters in estimator
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 8
Bayesian Information Criterion ¾
Adjusts MSE for model complexity
k ⎡1 n 2⎤ B = Ln ⎢ ∑ ( yi − f i ) ⎥ + Ln(n) ⎦ n ⎣ n i =1 K is # free parameters in model (weights)
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 9
Directional Change Predictor ¾
Directional change predictor Correctness of sign predictions
1 n d = ∑ zi n i =1 • zi = 1 if (yt+1 - yt)(ft+1 -yt ) > 0; 0 otherwise ¾
Interpretation D = 1 means perfect prediction of directional changes D > 0.5 is better than tossing a coin D = 0 implies 0% predictive ability • Note: easy to obtain large d in trending market
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 10
The Bear-Bull Statistic ¾
Measures forecasting ability P1 is percentage of correct bull market forecasts P2 is percentage of correct bear market forecasts B = P1 + P2 - 100%
¾
Example: Predictor which is always right on bull and bear calls: • P1 = P2 = 100%; B = 100% Predictor which calls all the bulls (but no bears) • P1 = 100%; P2 = 0%; B = 0%
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 11
Measuring Profitability Net returns ¾ Buy and hold test ¾ Distance from the ideal ¾
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 12
Net Returns ¾
Test of investment strategy Long positions when expected returns are positive Short positions when expected returns are negative n
r = ∑ pt ( yt +1 − yt ) t =1
⎧1 if ( f t +1 − yt ) > 0 ⎪ pt = ⎨− 1 if ( f t +1 − yt ) < 0 ⎪0 if ( f − y ) = 0 t +1 t ⎩ Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 13
Buy and Hold Test ¾
Benchmark to quantify excess returns Tests whether profitability is due to predictive ability or just general market conditions
c + ( yt + n − yt ) r= yt • C is stock dividend or bond coupon
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 14
Distance From the Ideal ¾
Measures returns from trading system against perfect predictor d n
rd =
∑ p (y t =1 n
t
∑| y t =1
t +1
t +1
− yt )
− yt |
Pt as previously defined
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 15
Equity Curve Measures Drawdown ¾ Luck coefficient ¾ Stirling ratio ¾ Risk of ruin ¾
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 16
Drawdown Equity Curve
Cumulative Returns
130 120 110 100
Drawdown
90 80 70 27
25
23
21
19
17
15
13
11
9
7
5
3
1
60
Trades
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 17
Drawdown ¾
Systems with large drawdowns hard to trade Requires lots of capital & confidence!
Smooth equity curve is desirable ¾ Usually harder to obtain than high net return ¾
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 18
Luck Coefficient ¾
How much of total profit was dependent on most profitable (k) trades(s)?
l (k ) =
Maxk {r0 , r1 , ..., rn } n
∑r i =1
¾
i
Large L indicates system success unlikely to be repeatable
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 19
Stirling Ratio ¾
Penalizes average returns for drawdown
1 n ri ∑ n i =1 s= 10 − d i • di is the i-period maximum drawdown. Can be too slow to change • Recalculate frequently
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 20
Risk of Ruin ¾
Probability that capital will be depleted Depends on • Probability of successful trade p • Payoff ratio (av. Win / av. Loss) • Fraction of capital exposed to trading Assume: • Payoff ratio is 1 • We risk all capital • K sequential trades R ~ [(1-p)/p]k
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 21
Portfolio Performance Measures ¾
Sharpe ratio: (rp - rf) / σp • Measures reward to total risk trade-off
¾
Treynor’s measure: (rp - rf) / βp • Excess return per unit of systematic risk
¾
Jensen’s measure: αP = rp - [rf + βp(rM - rf)] • The portfolio’s alpha - abnormal return above that predicted by CAPM
¾
Appraisal ratio: αP / σ(ep) • Abnormal return per unit of specific risk that could be diversified away using a market index portfolio
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 22
Which Measure to Use ¾ ¾ ¾
Suppose you have invested in a portfolio P Case 1: P is your entire investment fund Case 2: P is your active portfolio and: You are also investing in the passive market index portfolio
¾
Case 3: P is one of many portfolios Combined in a large investment fund E.g. You are one of a number of portfolio managers
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 23
Case 1: P Is Your Entire Investment Fund ¾ Compare
P’s Sharpe ratio with other fund:
Passive index fund Professionally managed active funds
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 24
Case 2: P Is Your Active Portfolio ¾
Recall: S2C = S2M + [αP / σ(eP)]2 SC is the Sharpe ratio of the combined portfolio (M and P)
“How much does your active portfolio P add to the Sharpe ratio SM of your passive market index portfolio?” ¾ Use appraisal ratio: [αP / σ(eP)] ¾
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 25
Case 3: P Is One of Many Portfolios P’s contribution to the entire diversified fund is αP ¾ So could use Jensen’s measure (portfolio alpha) ¾
But this takes no account of risk ¾
Better to use Treynor’s measure: (rp - rf) / βp Measure P’s excess return against the systematic risk (beta) rather than the total diversifiable risk (s.d.)
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 26
Lab: Portfolio Performance Measurement ¾
Advise a client in choice of funds Use different performance measures
¾
Excel lab: portfolio performance measurement Complete worksheet See solution worksheet
¾
See written notes and solution
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 27
Portfolio Performance Measurement Solution Fund P Fund Q Benchmark M Sharpe Alpha (Jensen) Beta Treynor σ(e) Appraisal ratio R2
Copyright © 1999-2006 Investment Analytics
0.43 1.63% 0.70 3.97 1.92% 0.85 91.12%
0.49 5.26% 1.40 5.38 9.35% 0.56 63.82%
0.19 0.00% 1.00 1.64 0 0.00 100.00%
Forecasting Financial Markets – Testing Strategies
Slide: 28
Portfolio Performance Measurement Solution ¾
Both P & Q outperform M: Higher Sharpe ratios, positive alphas
¾
Fund Q is preferred: If this fund is the client’s only investment • Higher Sharpe ratio than P As one of a mix of portfolios • Higher Treynor measure than P
¾
P is preferred if used as an active fund In conjunction with a passive index fund • Higher appraisal measure than Q
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 29
Summary: Testing Strategies ¾
Appropriate testing metric depends on application Forecasting Trading system development Portfolio management
¾
Models unlikely to perform equally on every basis E.G. Models with low R2 may generate significant profits Models with good statistical fit may trade badly
¾
Moral Decide objective and testing strategy before modeling!
Copyright © 1999-2006 Investment Analytics
Forecasting Financial Markets – Testing Strategies
Slide: 30