Strategy backtesting is a mix of art and science. Quants who rely too much on patterns in data will fall victim to curve fitting, while others create theories to fit their models. Here are leading quants’ perspectives on best practice in strategy backtesting.
1. Have a Common Sense Idea of Your Model
“If you can’t come up with a common-sense explanation, chances are, you’re data mining,” says Mebane Faber (@mebfaber) at Cambria Funds. When Mebane began in quant trading he thought he had found the holy grail via analysing historical returns. That, he says, is a common mistake made by new quants which can be fixed if you know why your model works. His advice? “Come up with a system that fits your personality but is also robust over time.”
2. Use Blind Data to Improve Your Strategy
Deepak Chenoy (@deepakshenoy) at Capital Mind says: “I tend to avoid over-optimization.” Instead, he adds unrelated factors such as volume, open interest or options price sensitivity (vega). “Or I might use my knowledge of near-term events to augment a system.”
Another important step for Deepak, after he’s optimized, is to test on a blind data set for validation.
3. Decide Your Most Important Metrics Before
“Popular performance statistics only reveal a small picture of how the model performed,” says Michael Guan (www.systematicedge.wordpress.com). Michael focuses on additional measures like average drawdown, annual returns, average profit per trade, for “a more multi-dimensional view through time”. Michael believes in multi-dimensional backtesting. “Make as few assumptions as possible and test on a wide variety of assets to ensure statistical robustness.”
4. Make Sure You Are Looking At The Right Data
Blake Woodard of RLF Capital Management (www.rlfcm.com) doesn’t like the way most quants backtest. “When building and backtesting in C++, your results come out as static data.” He prefers “to see every cell and action as you break it down” because that approach “is really good for figuring out the logic behind the model”. Blake still uses advanced Excel plugins to see how each section of his algorithm is affecting results. Transparency is key for Blake.
5. Don’t be afraid to Experiment with Tools
Spencer Connaughton (spenc3rc) is a self-taught 23-year-old quant and managing director of Archivolt Partners. His advice? Each tool or language has its own pros and cons. Experiment with different ways to build the same algorithm.Spencer built a model in Wolfram Alpha Mathematica. “Mathematica lets you play with different variables one at a time and isolate elements, draw graphs and visualize your system. I prototyped the system in Mathematica and built my own backtesting system first in C++. I then moved everything over to Bloomberg for its robustness.” A variety of tools for a variety of results and insights. Try them all says Spencer.
6. Be Ready for Regime Change
Lars Foleide (@zyron), who teaches Quantitative Finance at UC Berkeley, says, “What I focus on is the abnormal events. They can’t be backtested for.” Lars believes that with volatile markets, a strategy should be optimized by monitoring current market events as well as historical results. “I’m looking for Black Swan events. Working out how to have my system ready for a regime change.” Be open to the idea that your strategy might not work in 6 months’ time.
7. When Do You Stop?
“Never,” says Michael Guan. “Strategy development is a continual process.” All our quants agree that the key is a focus on experimentation and continuous monitoring.
