Temporal Holdout Evaluation
Definition
Temporal holdout evaluation is a validation design for time-ordered data.
Instead of randomly splitting observations, choose a cutoff time $:
- observations before or at $ are used to build features or segments
- observations after $ are used to measure future outcomes
This respects the direction of time.
Basic Setup
For customer behavior analysis, let:
- (t_0)$ = customer $ historical information up to cutoff time $
- (t_0, h)$ = customer $ outcome over the next horizon $
Examples of future outcomes are:
7648 Y_i = 1{\text{customer orders within } h \text{ days after } t_0} 7648
or:
7648 Y_i = \text{revenue from customer } i \text{ over the next } h \text{ days} 7648
Why It Matters
A random split can leak future behavior into the training side when data are time ordered.
Temporal holdout avoids this by making sure that all features are computed from information that would have been known at the cutoff.
Empirical Reorder Probability
For a segment $, the reorder probability within horizon $ is estimated as:
7648 \hat P(\text{reorder within } h \mid S) = \frac{1}{|S|}\sum_{i \in S} 1{\text{customer } i \text{ ordered within } h} 7648
This is a sample proportion.
Empirical Expected Revenue
For a segment $, expected future revenue over horizon $ is estimated as:
7648 \widehat{E}[R_h \mid S] = \frac{1}{|S|}\sum_{i \in S} R_{i,h} 7648
where {i,h}$ is customer $ future revenue in the holdout window.
Empirical Churn Probability
If churn is defined as no observed future order after the cutoff, then:
7648 \hat P(\text{churn} \mid S) = \frac{1}{|S|}\sum_{i \in S} 1{\text{no future order observed}} 7648
This is an observed non-return rate, not proof that the customer will never return outside the observation window.