1
$\begingroup$

Why do we split the data into two parts, and then split those segments into training and testing data? Why do we have two sets of data for each training and test data?

$\endgroup$
1
$\begingroup$

Are you talking about (X_train,y_train) and (X_test,y_test). If yes, then X represents the data(features) and y represents the labels of that data. That's why you get a pair when you divide it into training and test data

| improve this answer | |
$\endgroup$
1
$\begingroup$

For any Machine Learning model, the available data is usually split into three sets:

Training Set:

The part of data used to train the model and learn the parameters of the network.

The data that remains after allocation of the Training Dataset, is split into the Validation and Test sets.

Validation Set:

This sample of data is used to provide an unbiased evaluation of a model fit on the training dataset. This helps in tuning model hyperparameters to improve the model performance. Eg: Changing the number of clusters ($k$) in a K-Means algorithm, or the pooling layers in a CNN.

Test Set:

After training, this part of the dataset is used to used to test how well the model generalizes to unseen data and estimate its performance.

Another possibility (going by your question), is the use of Cross-Validation. This is performed when the dataset is too small. In such a case, a random split is performed on the dataset resulting in k non-overlapping subsets. The test error is then estimated by taking the average test error across k trials. kFold [Image Source]

| improve this answer | |
$\endgroup$
  • $\begingroup$ You're not answering the question "While we split data in training and test data, why we have to pairs of each?" or addressing the confusion completely. The OP thinks that both the training and test datasets are again divided into two chunks. $\endgroup$ – nbro Feb 7 at 13:38
  • $\begingroup$ As far as I know, I have never come across a scenario where such a split has been used. I clarified his doubt by answering the general approach to splitting a dataset (without explicitly hinting at the fact that he might have a misconception). If you are aware of a case where such a splitting condition is used, you are free to elucidate on it. $\endgroup$ – SoumyadeepB Feb 7 at 13:59
0
$\begingroup$

Usually we are splinting the data into 3 chunks for example 70% for training, 10% for validation and 20% for testing. The first two chunks are going to be used for training. The reason you need the validation dataset is to tune your hyper-parameters and see how well your model can generalize. Once you have a model that achieves a fairly good performance on the validation dataset, you're measuring its accuracy on the test dataset.

| improve this answer | |
$\endgroup$
  • $\begingroup$ You're not answering the question "While we split data in training and test data, why we have to pairs of each?" or addressing the confusion completely. The OP thinks that both the training and test datasets are again divided into two chunks. $\endgroup$ – nbro Feb 7 at 13:39
  • $\begingroup$ Btw, I didn't downvote this answer and I am not sure why someone would. $\endgroup$ – nbro Feb 9 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.