Optimization 101 Rev A

Optimization 101 – Bruce Robinson [email protected]

October 8, 2005

Presentation Overview   



   

Mechanical systems and modeling Fitness – what is it and what are the possibilities Back-fitting as a loaded gun – what are the issues and pitfalls Near-optimal solutions and a high level overview of how its done Walk-forward methodology --- BREAK --Mechanics of the IO optimizer Three functions  





Detailed example of back-fitted optimization Sensitivity analysis and how it predicts robustness - with example Detailed example of walk-forward optimization, and use of sensitivity analysis

How to use it in your trading – some guidelines

Presentation Goal -

System vs. discretionary trading “Dad – are all stops signs like mandatory, or are some just like recommended?” 

There are successful mechanical traders and successful discretionary traders



Both back-test, although differently, and over different time periods



Discretionary traders use the “computer between their ears”



Discretionary traders tend to factor in more complex patterns, but don’t take all signals



Mechanical traders must take all signals to achieve expected results – but most don’t take

Discretionary traders 

Discretionary traders process – Patterns they articulate  Patterns that they do not, OR can not articulate 





Patterns may involve price action alone, or other information such as markets, sectors, age of move, etc. Fall into two categories Setups  Projections of future effects 

Mechanical system traders 

Rationale  





Modern T/A programs have made development much easier There are a number of traps waiting for a developer of trading systems 



Mechanical system can be back-tested In theory, it removes emotions from trading

Back-fitting is one of the worst !

Most system traders will exercise discretion thinking that they can improve on results

Why do we model, backtest, and optimize 

Model   

To define an “edge” in timing, selection, money management A model approximates history Are we modeling an underlying crowd behavior, or a statistical relationship with no cause and effect ? 





Accuracy is required, but not exactness

Back-test   



Storks nesting in post-WWII Copenhagen -> birth rate

To verify that edge, know the odd’s To test across different markets To provide confidence going forward

Optimize  

To maximize or minimize those factors that are most important CAR, MDD, Sharpe, UPI, etc.

What is fitness?   

We have to decide on what we are going to optimize on It can be a single measure, or a weighted combination of measures Some typical candidates     

CAR – compounded annual return CAR / MDD UPI - Ulcer Performance Index – penalize net. volatility only Sharpe ratio – neg. volatility will follow pos. volatility Combination measures 



Fitness = CAR / Trades ^ 0.5 / MDD ^ 2 / UI ^ 0.5

Risk adjusted measures will often need

Trading System Mining Pitfalls

“A fossicker is one who searches (unsystematically) for any object by which to make gain - to dig - to hunt - to bustle about - or to rummage... “



Back-fitting to history usually yields negative surprises

Back-testing ???

2001

2003

TODAY

BACKTEST AND OPTIMIZE



Big Question – What percentage of the back-test return, MDD, etc. should you expect going forward ?

The back-fitting sensitivity problem -

Other issues 

Back-testing 

Different market action vs. recent market action 



Number of signal samples - statistical significance 



Test over bull, bear, sideways markets

Issue for intermediate term systems

Survivorship bias Bad performing stocks fold, or merge  Good funds’ record is bought  Bad funds record is buried 



Selection bias

AI optimization   

AI optimization offers many new capabilities To date we have fitted as much data as possible and “hoped” Mechanization is one of the most important capabilities 

  

Eliminates biases - selection, event knowledge, etc.

Less sensitive solutions can be found for a point in time We can “time travel” to points in the past and look at walk forward performance A “spliced” equity curve can be assembled to look at what would have happened if

Keywords –    

Near optimal Robust Out of sample, walk-forward Many of these concepts were pioneered by Rober Pardo

Near optimality -

Robustness -

Rolling walk forward testing

1998



2000

2002

TODAY

Green segments represent equity that COULD have been achieved !

Anchored walk forward testing

1998



2000

2002

TODAY

Green segments represent equity that COULD have been achieved !

Walk forward issues 

Rolling walk forward – How far back to back-test and optimize over  Attractive to look over recent history  Markets change when you least expect 



Anchored walk forward – Encompasses different markets  Compromises over those different environments 

Walk forward efficiency  





Several approaches to walk forward efficiency including Pardo’s One approach is compound annual return (CAR) of walk forward periods / CAR of solution optimized over entire period The following alternative, though, performs measurements in the same period as a kind of “apples to apples” approach Efficiency (WFE) 



Walk forward period / value for walk forward period when optimized through end of walk forward period

Example –    

LB = Optimize 1/2/1998 – 1/2/2002 WF = Walk forward 1/2/2002 – 1/2/2003 TOT = Optimize 1/2/1998 – 1/2/2003 IDEAL = Segment of TOT for 1/2/2002 – 1/2/2003

Why Pardo Design, Testing, and Optimization of Trading Systems - Robert Pardo  Originally written in 1992  His book is a good set of guidelines  Other authors such as Perry Kaufman have also written about this topic 

Pardo Cliff Notes 

First – Pardo is a guide

Difference between necessary and sufficient  Not A -> Not B  A -> B  Most of Pardo’s guidelines are sufficient  In other words, if you have them, you should have a robust system, BUT, is you don’t, you may still have a robust system. “Overfitting occurs when excessive attention is paid to creating a curve or testing a trading model with past data and not enough attention is paid to judging the predictive value of the curve or trading model.” Much of Pardo’s work deals with over-fitting, how to detect it, and how to avoid it 





Pardo’s operational recommendations  

Know the point at which to abandon a system Models have a “shelf life” 



Systems must have a statistically significant sample size  

 

Re-optimization can be too frequent OR too infrequently

Intermediate systems – as much data as is available Swing systems – multiple markets vs. recent market debate

Monitor the equity curve Trade the equity curve – trend line, MA, MDD, etc.

What is the problem to solve? 







Today we optimize to today, fit, and hope ! Many/most system models are too complex for standard optimization techniques We do not know how a model would have performed in the past as it was optimized and traded forward. “In theory there is no difference between theory and practice. In

System off-roading -

A Solution 





 

What if we had ways to find a near optimal, reasonably robust solution (if one exists) to any system, at any point in time? And, we could GO BACK IN TIME see how such a system would do OOS in a walk forward test. We could piece together those OOS tests over time to get a result that could have been achieved. We would no longer be optimizing and hoping.

BREAK

Artifical Intelligence Optimizations 





The idea is to use AI based search techniques that model techniques in nature to find a nearoptimal, robust solution. Solutions are expected to be the best that can be detected by intelligent searches in a reasonable timeframe. Timeframe for full search is impractical ! (NP Complete)  

RUTTR – 14 variables (9 without 4 levels and 1 lookback) 51 x 51 x 41 x 21 x 21 x 51 x 101 x 15 x 21 x 21 x 18 x 100 x 100 x 5 = 1,442,204,034,769,858,500,000 combinations 1,483,131,035,700,000 combinations without levels and lookback 2,351,489 years at 1,728,000 per day

AB and IO 

AB lends itself to this approach because –   

      

Has very fast backtester Allows user functions Support COM automation by controlling script

IO – product written by Fred Tonetti as AB add-on IO uses particle swarm and differential evolution Manipulates the Optimize() statements in the source and “feeds in” potential solution points Exports performance results and iterates Animates the ongoing equity charts and produces statistical data and charts in browser Also produces numerous CSV files of results Will “dim the lights”

Setup for background IO run -

AB tool used to run IO -

Goals and penalties 

  

Goal assigns a minimum desired value to a fitness field name (AB optimization result column) Values below the goal result in an adjustment factor that is applied to fitness. Two types of scaling – multiplier and exponential Example of multiplier    

Fitness is UPI UPI of solution point = 2 CAR = 15, CAR goal = 20 Fitness adjustment is .75 * 2 = 1.5

Sensitivity  

 

Definition - change in fitness with respect to “distance” from solution Rationale – since change in inputs can’t be accurately predicted, approach is to measure change in output based on change in parameters An average percentage is determined based on samples The solution point fitness is adjusted by a sensitivity goal

The sensitivity adjusted fitness -

IO directives for traditional optimization //IO: StatusWindow: N //IO: SaveCancelled: Y

Disable the directive // Remove in final "production" run //xIO: FitnessTime: 300 5 minute //xIO: SenFinalTests: 1000 limit 1000 samples for sensitivity Penalize result < 85% //IO: SenOptGoalPct: 85 sensitivity //IO: SenOptTests: 10 10 samples to determine //IO: Fitness: UPI sensitiviy CAR goal to prevent high //IO: Goal: CAR: 15 UPI, low CAR //xIO: BegISDate: 01/01/1995 //xIO: EndISDate: 01/01/2001 Walk-forward directives //xIO: EndOSDate: 9/15/2005 disabled //xIO: WFAuto: Anchored: Every: Year

Timing signal logic BuyPrice = SellPrice = ShortPrice = CoverPrice = Close; SetTradeDelays( 0, 0, 0, 0 ); emalen longfilt 0.050, 0.001 ); shortfilt 0.050, 0.001 ); fund fundema Buy Sell fund ); BuyState SellState Buy Sell Short Cover

= Optimize( "emalen", 4, 2, 100, 1 ); = Optimize( "longfilt", 0.014, 0.001, = Optimize( "shortfilt", 0.016, 0.001,

= Close; = EMA( fund, emalen ); = Cross( fund, fundema * ( 1 + longfilt ) ); = Cross( fundema * ( 1 - shortfilt ), No needed in = Flip( Buy, Sell ); this case – but = NOT BuyState; always safe = ExRem( BuyState, SellState ); Turns off = ExRem( SellState, BuyState ); short trading = 0; //Sell; = 0; //Buy;

SetOption( "ReverseSignalForcesExit", True );

Graphical signal logic -

Note that area between green and red requires state logic

State and boolean signal logic 

Area between red and green can’t be described by Boolean logic 





Fund is in that area AFTER BUY and AFTER SELL

Impulse signal occurs when fund crossed up thru green or down thru red EMA’s Design pattern 

 

These impulses are switched into state by the Flip() function Flip() also resolves combinations of signals AB’s trading engine want impulse form which is obtained with the ExRem() function

IO Startup Screen -

Next – IO Optimizer in detail

Traditional optimization -

Portfolio indicator -

Days in drawdown

Equity & signals Regressi on line Current drawdown

Reference ticker

Parameter sensitivity distribution -

Report of traditional backtest -

1 year walk forward optimization -

Walk forward vs. traditional from 2000 -

Walk forward vs. traditional from 1996 -

Comparison of walk-forward to traditional 

  



Primary difference is that the 2003 run-up was not captured by the walkforward Walk-forward actually did better in 2000, and 1997-1998 The traditional back-fit must make compromises to handle all periods But the walk-forward is more sensitive to changes in the early years Note differences in number of trades

Additional IO functionality 



 

  

Perform sensitivity testing on a given system Invoke IO from the command line and run batch directive files Process user defined metrics Turn off sensitivity testing of individual variables Optimization of groups of variables Walk forward on every signal Many more options that are documented in the IO.DOC file

How to use in your trading 

Model your trading ideas 

Don’t incorporate hidden fixed assumptions/filters  



Utilize Optimize() statements for all system parameters – not just indicator parameters 

 



Now, there is no reason not to – we no longer have to fit and hope Use a problematic period – for example, 2001-2002

Use common sense –   



For example - PositionSize, WorstRankHeld, ApplyStops, etc.

Perform IS (in sample) over several types of markets Perform a walk-forward analysis 



FilteredEMA example is a stop and reverse system A system that went flat might allow for better filters and signals with less retracement

Don’t re-optimize too frequently Don’t have short look-back periods Expect to iterate your model

Look at it as improving your odds for success

Conclusions      

The example used was a teaching example, not a practical system IO allows optimization of previously intractable systems But, it’s value is in allowing walk forward optimization of any system This allows you to see how a system would have performed in the past It allows you to see how it may perform in the future and to assess its stability You may find results you didn’t want to find  

 

Some high performance systems may not walk forward well But, others will

“ The future ain't what it used to be “ – Yogi Berra Remember, it is like going …

IO -

Questions and Answers

Preview of Optimization 201  

Knowledge of AFL programming will be needed Construction and optimization of timing signals 



Using IO to turn signal components on and off 



   

Assess portfolio mix

IO optimization of rotational systems 



Explore which components add value

Optimization of ticker selection and weights 



Level vs. state

Special case of rotational system = walk-forward system

Selecting look-back and walk-forward periods Fine-tuning the directives and options Minimizing run time Design patterns Why system design is partly a science and partly an art