Is it possible to somehow make the CV work with cross_val_predict function. Fore example, if I try:

cv = GapWalkForward(n_splits=3, gap_size=1, test_size=2)
cross_val_predict(estimator=SGDClassifier(), X=X_sample, y=y_bin_sample, cv=cv, n_jobs=6)

it returns an error

ValueError: cross_val_predict only works for partitions

but I would like to have predictions so I can make consfusion matrx and other statistics.

Is it possible to make it work with your cross-validators?

Documentation and examples do not address the splitting of data set into training and test sets.

If using one of the cross validators, does the data set need to be sorted in time order? Is there way to designate a datetime column so the class understands on what basis to sequentially split data?

Just flagging a minor issue:

We found this after poetry update-ing our dependencies, inadvertently bumping scikit-learn to 0.24.0. This broke code we have that uses tscv

relevant scikit-learn source-code from version 0.23.0 https://github.com/scikit-learn/scikit-learn/blob/0.23.0/sklearn/utils/init.py#L274-L275

The method has been made private in scikit-learn 0.24.0: https://github.com/scikit-learn/scikit-learn/blob/0.24.0/sklearn/utils/init.py#L271

I did not investigate further, we pinned scikit-learn to 0.23.0 and that's OK for now, but some refactoring may be in order to move off the private method.

Using TSCV Gapwalkforward successfully with Python 3.7.

Suddenly getting following error:

ImportError Traceback (most recent call last) in 41 #Modeling 42 ---> 43 from tscv import GapWalkForward 44 from sklearn.utils import shuffle 45 from sklearn.model_selection import KFold

~\Anaconda3\envs\py37\lib\site-packages\tscv_init_.py in ----> 1 from .split import GapCrossValidator 2 from .split import GapLeavePOut 3 from .split import GapKFold 4 from .split import GapWalkForward 5 from .split import gap_train_test_split

~\Anaconda3\envs\py37\lib\site-packages\tscv\split.py in 7 8 import numpy as np ----> 9 from sklearn.utils import indexable, safe_indexing 10 from sklearn.utils.validation import _num_samples 11 from sklearn.base import _pprint

ImportError: cannot import name 'safe_indexing' from 'sklearn.utils'

Any insight? I get this when simply importing Gapwalkforward.

When I upgrade to Scikit-learn 0.24.1 I get an issue:

cannot import name 'safe_indexing' from 'sklearn.utils'

This appears to be a change within scikit-learn as indicated here:

https://stackoverflow.com/questions/65602076/yellowbrick-importerror-cannot-import-name-safe-indexing-from-sklearn-utils

No issue using scikit-learn 0.23.2

Would it be possible to issue a new release on PyPI to include the latest changes from this commit which aligns the get_n_splits method signature with the abstract method signature required by GridSearchCV?

https://github.com/WenjieZ/TSCV/blob/f8b832fab1dca0e2d2d46029308c2d06eef8b858/tscv/split.py#L253

This warning should appear for every occurrence. Use standard output instead.

The current implementation is based on legacy K-Fold cross-validation requiring an explicit value for the n_splits parameter. It puts the burden of calculating desired value of n_splits on the user.

A better implementation should allow the user to initiate a GapWalkForward class without specifying the value for n_splits. Instead, it can deduct the right value through the other inputs.

It is theoretically desirable to keep both channels of kickstarting a GapWalkForward class. In practice, however, it is hard to maintain both within a single class. Therefore, I decide to ~~deprecate the n_splits channel~~ implement a new class dubbed GapRollForward in v0.1.0 -- the version after the next.

…e. Now works with GridSearchCV. Otherwise using GapWalkForward as the cross validation class passed to GridSearchCV will fail with "TypeError: get_n_splits() takes 1 positional argument but 4 were given."

Hi guys, this issue occured after the upgrade to 1.1.3

ImportError: cannot import name '_pprint' from 'sklearn.base'

/.venv/lib/python3.10/site-packages/tscv/_split.py:19 in      │
│ <module>                                                                                         │
│                                                                                                  │
│    16 import numpy as np                                                                         │
│    17 from sklearn.utils import indexable                                                        │
│    18 from sklearn.utils.validation import _num_samples, check_consistent_length                 │
│ ❱  19 from sklearn.base import _pprint                                                           │
│    20 from sklearn.utils import _safe_indexing                                                   │
│    21                                                                                            │
│    22                                       

Could you please fix it ?

Kind regards, Jim

There are two ways of defining a derived cross-validator. One is to redefine _iter_test_indices or _iter_test_masks (test viewpoint), and the other is to redefine _iter_train_masks or _iter_train_indices (train viewpoint).

Currently, these two methods assign different semantic meanings to the parameters gap_before and gap_after. The test viewpoint uses the test sets as the reference:

train    gap_before    test    gap_after    train

The train viewpoint uses the training sets as the reference:

test    gap_before    train    gap_after    test

This diverged behavior is ~~not intended~~ inappropriate. The package should insist on the test viewpoint, and hence this PR. It will be enforced in v0.2.

I don't think this issue has touched any users, for the derived classes in this package use _iter_test_indices exclusively (test viewpoint). No users have reported this issue either. If you suspect that you have been affected by it, please reply to this PR.

With contiguous test sets:

cv_orig = GapKFold(n_splits=5, gap_before=1, gap_after=1)

for train_index, test_index in cv_orig.split(np.arange(10)):
    print("TRAIN:", train_index, "TEST:", test_index)


... TRAIN: [3 4 5 6 7 8 9] TEST: [0 1]
... TRAIN: [0 5 6 7 8 9] TEST: [2 3]
... TRAIN: [0 1 2 7 8 9] TEST: [4 5]
... TRAIN: [0 1 2 3 4 9] TEST: [6 7]
... TRAIN: [0 1 2 3 4 5 6] TEST: [8 9]

cv_opt = GapKFold(n_splits=5, gap_before=1, gap_after=1)

for train_index, test_index in cv_opt.split(np.arange(10)):
    print("TRAIN:", train_index, "TEST:", test_index)


... TRAIN: [3 4 5 6 7 8 9] TEST: [0 1]
... TRAIN: [0 5 6 7 8 9] TEST: [2 3]
... TRAIN: [0 1 2 7 8 9] TEST: [4 5]
... TRAIN: [0 1 2 3 4 9] TEST: [6 7]
... TRAIN: [0 1 2 3 4 5 6] TEST: [8 9]

%%timeit
folds = list(cv_orig.split(np.arange(10000)))


... 1.21 s ± 37.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
folds = list(cv_opt.split(np.arange(10000)))


... 4.74 ms ± 44.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

With uncontiguous test sets:

cv_orig = _XXX_(_xxx_, gap_before=1, gap_after=1)

for train_index, test_index in cv_orig.split(np.arange(10)):
    print("TRAIN:", train_index, "TEST:", test_index)


... TRAIN: [5 6 7 8 9] TEST: [0 1 2 3]
... TRAIN: [7 8 9] TEST: [0 1 4 5]
... TRAIN: [3 4 9] TEST: [0 1 6 7]
... TRAIN: [3 4 5 6] TEST: [0 1 8 9]
... TRAIN: [0 7 8 9] TEST: [2 3 4 5]
... TRAIN: [0 9] TEST: [2 3 6 7]
... TRAIN: [0 5 6] TEST: [2 3 8 9]
... TRAIN: [0 1 2 9] TEST: [4 5 6 7]
... TRAIN: [0 1 2] TEST: [4 5 8 9]
... TRAIN: [0 1 2 3 4] TEST: [6 7 8 9]

cv_opt = _XXX_(_xxx_, gap_before=1, gap_after=1)

for train_index, test_index in cv_opt.split(np.arange(10)):
    print("TRAIN:", train_index, "TEST:", test_index)


... TRAIN: [5 6 7 8 9] TEST: [0 1 2 3]
... TRAIN: [7 8 9] TEST: [0 1 4 5]
... TRAIN: [3 4 9] TEST: [0 1 6 7]
... TRAIN: [3 4 5 6] TEST: [0 1 8 9]
... TRAIN: [0 7 8 9] TEST: [2 3 4 5]
... TRAIN: [0 9] TEST: [2 3 6 7]
... TRAIN: [0 5 6] TEST: [2 3 8 9]
... TRAIN: [0 1 2 9] TEST: [4 5 6 7]
... TRAIN: [0 1 2] TEST: [4 5 8 9]
... TRAIN: [0 1 2 3 4] TEST: [6 7 8 9]

%%timeit
folds = list(cv_orig.split(np.arange(10000)))

... 1.23 s ± 75.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
folds = list(cv_opt.split(np.arange(10000)))

... 4.78 ms ± 49.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

From "Advances in Financial Machine Learning" book by Marcos López de Prado the implemented version of Combinatorial Cross Validation with Purging and Embargoing

explaining video: https://www.youtube.com/watch?v=hDQssGntmFA

Thank you for this repository and the implemented CV-methods; especially GapRollForward. I was looking for exactly this package.

I was wondering if you are interested in implementing another CV-Method for time series, called Rep-Holdout. It is used in this evaluation paper (https://arxiv.org/abs/1905.11744) and has good performance compared to all other CV-methods - some of which you have implemented here.

As I understand it, it is somewhat like sklearn.model_selection.TimeSeriesSplit but with a randomized selection of all possible folds. Here is the description from the paper as an image:

The authors provided code in R but it is written very differently than how it needs to look in Python. I adapted your functions to implement it in python but I am not the best coder and it really only serves my purpose of tuning a specific model. Seeing as the performance of Rep-Holdout is good and -to me at least - it makes sense for time series cross validation, maybe you are interested in adding this function to your package?

If for example, I have data without gaps, when and why would I still create a break between my train and validation? I have seen the argument for setting gaps when the period that needs to be predicted may be N days after the train. Are there other reasons? And if so, what is the intuition on knowing how many gaps to include before/after the training set?