6

In machine learning, you normally split your data into 3 parts (80-10-10%). The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset. The second part (10%) is the development set (or dataset), aka validation set. This is used as measuring your performance with various hyperparameters (e.g. in ...


4

Short answer To select the proper dataset to construct, you should first figure out a metric to use to compare, and then select the dataset construction that gives the better metric. There is no single best metric, it depends on the task and your interpretation on what type of error is more important. If you believe it is important that errors should not be ...


3

Essentially, any data you use to train or develop the model shouldn't be used as test data. In principle, "unseen" data gives a good estimate for the generalisation performance of the model; but this is only valid if the data really is unseen and hasn't been used in the model development process. If you've been tuning a model to increase its ...


3

There are a few issues you need to address first. Normalise your data. You should try and keep your values for each input in a good range, otherwise you're never going to train anything useful. A simple way of doing this could be to divide each value by the maximum value for that input. This will ensure they are between 0 and 1, or you could divide by the ...


2

for the first point I'm very sorry that I cannot give you any literature on this, but I might be able to explain you, why you don't take PCA on both datasets independently. Principal components analysis is simply a transformation of your data into another (less dimensional) coordinate system. The axis for your new coordinate system are defined by the ...


1

I think that these terms may be used inconsistently across sources. If someone says held-out dataset, I would immediately think of a dataset that is not used for training, but can be used for anything else, validation (hyper-parameter tuning or early stopping) or testing; so, to determine what they are referring to, I would probably take into account the ...


1

I am not sure whether that solves your problem at hand, but one approach you could look into is k-fold Cross Validation (CV). In this approach, you split your combined train, development, and test data into $k$ randomized and equally-sized partitions. Afterwards, you train and evaluate your model $k$ times. In the $i^{th}$ iteration, you train your model on ...


Only top voted, non community-wiki answers of a minimum length are eligible