Statistical bias sources and trading strategy construction
By Daniel Fernandez
To give a trading system the highest probability of success under unknown market conditions it is important to understand the sources of bias associated with the system creation process and carry out the necessary steps to reduce these biases to their minimum possible value.
 Example of systems expected from randomness and total number of systems found for a sample data-mining exercise on the USD/JPY (1987-2015 data). Data-mining bias is significantly large. |
Live Vs simulated performance for a portfolio of 20 strategies (EUR/USD, USD/JPY, EUR/JPY) traded through the Swiss National Bank market shock in January 2015. Inaccuracy in simulations comes from a disparity between real/simulated market behavior (large slippage, low liquidity) causing drastically different entries/exits. |
There are mainly three facts that generate statistical bias when building strategies using historical simulations. The first is the fact that the amount of historical data used is not infinite, the second is that the historical data is known in advanced and can be reused infinitely within the strategy creation process and the third is that the historical simulation is not a real market execution of the strategy.
The first fact, that historical data is not infinite, causes a generalization problem within any strategy creation method. The main assumption when building a system from trading history is that the trading history contains some general market behavior that will exist under future market conditions; a behavior that can be measured and then exploited. However a limited data set implies that we are never sure whether what we are finding is a true general behavior of our financial time series or some particular behavior within the limited data set.
This source of bias is most commonly referred to as “curve-fitting bias” as it is associated with a lack of generalization towards real and stable market inefficiencies. Curve-fitting bias can never be completely eliminated – as there is no way to simulate on infinite real market data – but it can be reduced by increasing the amount of data used for model building, either by increasing the amount of financial time series used or increasing the amount of data within an individual time series. In this manner a system built on a year of data on a single currency pair will be much more subject to curve-fitting bias than a strategy built across ten years of data or a strategy built on 1 year on data on 3 separate symbols.
The second fact – that historical data is known in advance and can be reused infinitely within the system creation process – generates a different type of bias which is associated with the probability that a system with a desired statistical profile is found simply due to random chance. By evaluating many different rule combinations – either explicitly through a systematic search process or implicitly through manual chart analysis – a trader introduces statistical bias within the strategy creation process. Every increase in system building complexity carries with it an increase in the probability that a system with the desired historical statistical characteristics is found simply due to random chance.
This type of bias is usually called “data-mining bias” as it is associated with the statistical bias created through the rule search/modification process (the “mining” process). There are several methods to measure this type of bias, including White’s reality check and regression methods. Generally you will want to reduce data-mining bias such that the probability that a system comes from random chance for a given system building methodology is less than 1%. Ideally you will want this to be even lower, creating systems within a 0.0001% confidence that systems do not come from random chance is not unprecedented.
The third fact – that execution in simulated performance is not real execution – creates a rather difficult type of bias that deals with the actual real effect of your trading strategy within the market and the highest possible accuracy of the historical simulations. Since the market is built from actual traders buying and selling financial instruments, historical simulations of your strategy fail to include the effect that your strategy would have had on the market, had it been traded. A trading strategy might have generated a reaction within the market that might have eliminated its potential as a successful trading strategy. Your strategy might have been unable to enter the real market at the simulated quotes due to a lack of enough liquidity, for example. The two ways to attenuate this bias is to simulate using reliable tick data that matches your live trading data source, possibly with a level two order book to evaluate the viability of fills and second, to avoid trading under circumstances where the above problems are even more prominent (low liquidity hours, high volatility news events, etc).
Although the reduction of all the above sources of bias does not guarantee success – as the future is unknown – it does allow for the creation of trading strategies with bigger chances of success under unseen market condition. Building portfolios containing a large number of uncorrelated strategies with low levels of the above mentioned bias sources gives an even higher degree of confidence around potential success under out of sample condition.
