0
$\begingroup$

I have developed, trained and tested an NLP model. It is persisted in a pickle file. The model contains the data preprocessing function that includes text cleaning and new features engineered with word2vec.

With the trained model, I want to make predictions on a new text. The new text data, after preprocessing, won't contain the same engineered features of the training dataset.

Therefore my question is, how can the trained model make predictions on the new dataset as it has different engineered features (different numbers of columns and different columns)?

Should I preprocess the new text data and the training dataset as one dataset?

$\endgroup$
2
  • $\begingroup$ Your question/problem is not entirely clear to me. You say that, after preprocessing the test data, you get the test dataset, which has different features than the features of the training dataset. So, why would preprocessing both the training and test datasets, as a whole, fix this issue? Are you currently using a different preprocessing procedure for the training and test datasets? If that's the case, why? If that's not the case, then why do you get different features for the training and test datasets? $\endgroup$
    – nbro
    May 16 at 11:00
  • $\begingroup$ @nbro I trained and tested my model and preprocessed train and test as a whole. So both having the same features. My issue rises when I want to make predictions on a new data set different from train and test. My question is, do I need to append the new data set to train and test, preprocess them as a whole dataset and then make predictions on the new dataset (without training the model again)? $\endgroup$
    – Annalix
    May 16 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.