Right now, I'm planning to make a deep neural network for classifying taste of crystals, with their molecular structure, which includes information like the number of atoms or the mass of each atom.

How should I make a data set for training, testing, and validation?


1 Answer 1


You can cluster all your features in one matrix X, in which each line would be one element of the data set you want to construct, and each column would be a different feature of this element.

You construct then a Y vector containing the different target classes, where the i-th element will be the target class of the i-th X element.

For the following I suppose you use python

Create train and test sets using sklearn wrapper train_test_split on X and Y just built, in which you can shuffle your X and Y (non-avoidable in this case) :

from sklearn.model_selection import train_test_split

BTW, I recommend you to rescale your data with sklearn with for example one of :

from sklearn.preprocessing import MinMaxScaler, StandardScaler

It allows better learning and so better generalization

  • 2
    $\begingroup$ Well done on train and test split and scaling. However, @stain also asked about validation set. Some packages (eg most of Keras supervised learning algorithms) can consider a portion (by with absolute or relative size) of training set as the validation set. Otherwise, one can perform train_test_split twice (one on whole data and once on the train set). See datascience.stackexchange.com/a/15136/46505 $\endgroup$ Mar 21, 2018 at 2:09
  • 1
    $\begingroup$ I also thought of mentioning cross-validation using k-splits (kfolds) generalizing the concept of training. As it is mentioned in your link @O_o answer seems also relevant here. $\endgroup$
    – nsaura
    Mar 21, 2018 at 10:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .