We are often asked about live strategies compared with backtests and simulations, and questions about not getting the same results. Under certain, maybe even most conditions, the results of the 3 should be close or even similar. There are, however, a few factors that can cause a divergence between simulations, backtests, and live strategies - even when running at the same time.
No execution - Backtests and simulations are not connected to a broker, and no execution is taking place. Especially when simulating/backtesting a low-volume asset, with a high amount - an actual trade can affect the price of the asset. The live execution can have a significant effect on the market, that will not be reflected in non-execution modes.
Partial fill - When a limit order on live mode meets a limited offering on the market, only part of the order will be executed. When there is no contact with the market, as happens on simulations and backtests, the orders will be filled fully (which is ideal when trading).
Slippage - There is a dedicated article for slippage issues. Slippage can be caused by low volume and therefore limited execution prices, actual trades that affect the asset's price (which is not happening on simulations and backtests), and more.
Price source - Live and simulated strategies are feeding live data from the brokers. However, different brokers may get their prices from different sources. This is something to be mindful of when comparing the same asset strategy performances between different brokers.
Price type - Each mode has its own engine, and each broker sends a specific price type. Some brokers may work with bid/ask, while others will work with the last trade price. In some cases (brokers) the backtest prices are different than the ones generated in live and simulation mode. This, too, can explain the differences between modes and between brokers.
Execution price - Since no execution is taking place, Capitalise.ai platform generates a price based on the last traded price (sent by the broker). In live mode, however, the price type may vary and probably will be based on the book managed by the broker.
Different periods - As of now, our backtest has a default of 90 previous days. When comparing simulations/live strategies to backtests, one must verify that the periods of the strategies and the hits timings are the same, especially when using timing conditions, percentages (of a bar for example), and trailing stop/take profits.
P/L Reference - exit conditions like “take profit”, “at loss”, “trailing stop”, and similar are referencing the change in P/L, and not the change in the asset’s price. When creating two “similar” strategies on different brokers or modes, gaps in execution times may be explained by different traded amounts and different entry prices on different P/L entry points.
Data corrections - Backtest data includes data corrections. That backtests will reflect changes in assets; historical data fixes, canceled orders, and corporate actions (like stock split/reverse-split), that live and simulated strategies will not.
Fees offset - Some brokers are offsetting the fees from the trade price. For example, if a broker charges a 1$ fee for 100$ trade, and one would want to buy 100 units of 1$ assets, only 99 units will be bought at a price of 99$, while 1$ will be allocated for fees. This does not happen on simulations and backtests.
Orders type - Some order types like MKT are simpler to execute. The platform gets value from the broker/historical data and executes it. However, other orders, like LMT can be the source of differences between strategies since their execution depends on a particular and accurate price. When the data varies between modes or brokers, some LMT orders may execute while others will not.
Going from Backtest to Real
Backtest is a popular tool for testing strategies. Running a backtest over a historical period reveals a lot about the strategy's strengths and weaknesses. However, a backtest cannot predict the strategy’s future behavior and performance. One should not expect the same results between a backtest and simulations/live. Backtests can help you perfect your strategy and optimize your P/L. For example, realizing that working with 5m bars will generate higher profits than 15m bars, or working with MAs rather than EMAs.
If you will reference a price in a backtest, a disclaimer will pop up, clarifying that the backtest works for a minimum of 1 minute. data, rather than tick data on live mode. That means the response time of the strategy (for meeting the conditions) will be on a 1m basis. That can explain delays in execution timing and prices between live/simulations and backtest.
Other points to be mindful of when analyzing a backtest, are no execution (1), slippage (2), the strategy period (7), data corrections (9), and broker fees (10) - which are not inclusive in backtest mode whatsoever, and the type of the order (11) and how it is being filled.
While live and simulation strategies are feeding prices from the broker, we (Capitalise.ai) are getting the backtest data ad-hoc directly from the exchanges/data vendors. This can cause differences in price types and rates (5 & 6) between modes.
Going from Simulation to Real
Simulations are great for testing strategies in real time. Generally, simulations will provide a more accurate picture of the strategy’s performance than a backtest, since it is feeding real-live data. However, simulations have a different engine from live strategies, physically and methodologically.
The first thing to consider when comparing simulations to real strategies is the fact that there is no execution (1). Since real trades are taking an active place in the market, big amounts that will affect a low-volume asset in live mode will not do so in simulation mode. This will cause the actual market condition to be different, as well as the strategies' performances.
Since strategies are being tested in simulation mode, and there is no actual execution, the orders will be “optimally” filled, meaning that orders like market orders will be fully executed. This probably will not be the case in live strategies (2, 11).
Other points of difference to be mindful of are slippage (3), generated prices (6), the periods of the strategies (7), how the P/L is affected in each strategy (8), and the fees discounted from the traded amount- which is not taking place on simulations.
Cross-brokers comparisons
As specified above, different brokers get their data from different sources, which may result in the execution of different prices (4). In addition to the different sources, each broker has its price types (methodology). That means that some brokers will reflect and execute mid prices, some will do so with the last traded price, and others will break it down for bid/ask (6).
Conclusion
This document includes all the different characteristics between strategy modes in Capitalise.ai (backtests, simulations, and live). These differences should not severely affect good strategies. In other words - a good strategy would not make or break as a result of these differences and natural phenomena (due to market behavior or technology limitations).
Automated trading, like manual trading, is not an accurate science. Be mindful of the difference between modes and brokers, and make sure your strategy can endure market changes and variations between modes.