Explainer
In-sample, out-of-sample and walk-forward testing
In-sample data is what you build the rules on, out-of-sample data is held back to test them once, and walk-forward repeats that test through time.
Updated
In-sample: where the rules come from
In-sample data is the history you look at while you write the rules. You see a chart, form an idea, pick a stop, pick a target, run it, and adjust.
Anything shaped on that data will fit it. That is not a flaw in the person. It is how rules get written. It only means the in-sample result tells you how well the rules fit that history, not how they will do on history you have not looked at.
- In-sample
- The data you used to build and adjust the rules.
- Out-of-sample
- Data kept out of sight while you built the rules, used afterwards for a check.
- Walk-forward
- Fit on one window of history, test on the next, slide forward, and repeat.
- Overfitting
- Rules tuned so closely to one stretch of history that they describe its noise as well as its structure.
Out-of-sample: the part you do not touch
Out-of-sample data is history you set aside before you start. You build the rules without looking at it. When the rules are fixed, you run them on that data.
The rule that makes it work is simple. Once you look at the held-back data and change something because of what you saw, it is in-sample. It cannot do the job twice.
Check the held-back data once. If you change the rules after looking, it is no longer held back.
Why held-back periods are spread across the history
The simple way to hold data back is to keep the last few years. The trouble is that those years are one stretch of market. If they happen to be quiet, or fast, the check says something about quiet or fast markets, and nothing else.
So we hold back months, not years. When a strategy starts, we pick about 30% of the months, spread across every year, with at least one in each year. The months are chosen once, are fixed for the life of the strategy, and are the same for every version of it.
Any trade whose entry or exit falls in a held-back month is dropped before a single figure is worked out. The results page shows those months greyed out and labelled "Saved for out-of-sample".
Walk-forward testing
Walk-forward testing repeats the idea through time. Take the first window of history and choose your settings on it. Test those settings on the window right after it. Then slide both windows forward and do it again. At the end you join up all the test windows.
It checks how the way you choose settings holds up as markets change. It does not check one fixed set of settings, and it does not tell you what comes next.
What overfitting looks like
You will rarely see it announced. These are the signs:
- Every condition was added after a specific bad stretch.
- Moving a setting by one tick changes the result a lot.
- The result sits in one year, or in a few trades.
- The held-back check looks nothing like the in-sample result.
Questions and answers
What is the difference between in-sample and out-of-sample?
In-sample data is what you used to build the rules. Out-of-sample data was kept out of sight until the rules were fixed. The first shows how well the rules fit. The second is a check on that.
Is walk-forward testing the same as out-of-sample testing?
It is built on the same idea. Out-of-sample testing usually holds one block back and checks it once. Walk-forward repeats the fit-then-test step many times along the history.
Why hold back months instead of the last few years?
A block of years is one stretch of market. Months spread across every year test the rules in each year of the history, not in a single period.
Does an out-of-sample check show the rules will hold up on new data?
No. It shows whether the rules were tuned only to the months you looked at. New data can still behave differently.