What is the meaning of test data set in naive bayes classifier or decision trees?

Question

What is the benefit of a test data set, especially for naive bayes estimator or decision tree construction?

When using a naive bayes classifier the probabilities are a fact. As far as I know there is nothing one could tune (like the weights in a neural net). So what is the purpose of the test data set? Simply to know if one can apply naive bayes or not?

Similiarly what is the benefit of the test data set when constructing a decision tree. We alread use the gini impurity to construct the best possibe decision tree and there is nothing we could do when we get bad results with the test data set.

Answer 1

Your assumption about the test data is not correct completely. Maybe you use the test data to tune your learning algorithm to work better on the test data, but it's not the whole thing. Sometimes you need to know that the ML method is working or not and have a sense about how much does it work!

You have other scenarios that you want to evaluate your method:

1. Compare the result of the leaner with other techniques. For example, you are considering DT versus an SVM classifier over a data set. If you want to compare them, you need a value to found such a sense about the comparison.

2. Sometimes you are using an ensemble method and you want to tune some parameters to balance between using different ML methods. Hence, you need to evaluate these learning methods (DT, Naive Bayes) to improve the ensemble method.

Answer 2

In machine learning, we can use all the datasets as training data in a model. But if there are too many data sets, or too much data, and we do not split them up, our model may be not produce acceptable results.

Why?

Because if the model studies too much training data, it may be overfitted.

(Just like when you cram for a test, and get overloaded with too much information!)

What I mean is, your model is only familiar with the data you provide, not for the new data.

So we need to use test data to train our algorithm. Naive Bayes and Decision Tree Classifier are no exception because they can produce an overfitted model based on train data.

So we test it on the data test to know how well the method works in relation to the problem.

Most data scientists divide their data (with answers, that is historical data) into three portions: training data, cross-validation data and testing data. The training data is used to make sure the machine recognizes patterns in the data, the cross-validation data is used to ensure better accuracy and efficiency of the algorithm used to train the machine, and the test data is used to see how well the machine can predict new answers based on its training.

_{SOURCE: https://www.researchgate.net/post/What_is_training_and_testing_data_in_machine_learning}

Answer 3

In the field of ML and AI, you should always remember that before choosing any algorithm you should know the data .One should always start with Data Analyzing, which itself is field for this critical job. Decision Tree can never work with its best optimization without tuning the datasets. Here is a great article that you can refer : Tuning Decision Tree

Purpose of test data in naive byes classifier: 1) It's necessity to check the accuracy, hits, hit rates, coverage, diversity, novelty, etc. metrics.

2) It hypertune your testdata(as anti_train_set) also by using mean, standard deviation, variance.

I really think that you should try other algorithms to train your datasets. I can't name all of them. However, in Neural network, rnn, cnn, rbm are some great algorithm to work with.

Please always remember that Machine learning is like an art, where datasets (test, train, evaluated) are colors and its up-to us to use the right amount of them.

Answer 4

I actually pondered this question a few months ago, so i understand your point of view!

You are correct in assuming that if you already build your tree or calculate your probabilities, what is the point of using test data? Because your model is fixed the way they are, no matter if you use test data or not. Well, the purpose of test data is not only to test your model against unseen data and get some evaluation score. But it is also to test if your model is the right fit for your problem.

One of the main reason why we all build ML/AI model in the first place is to extract insights that can be used to solve problems, make decisions etc. If you don't test your naive bayes or decision tree model with test data, you won't know if the information that are given to you by those model mean anything. They may not even help you solve problems or give you relevant information. Yes they may spout out big numbers and classify things, but are those result what you're looking for? Are the result relevant to your problem? Can the result be used to solve what you are trying to do?

Using test data gives you the opportunity to see if your model gives you the best insights and the best solution to your problems. So here are the takeaways from my answer:

• Test data can be used to test your model against unseen data
• You can gain score (evaluation) on your model when you test it with test data, which in turn can be used to fine tune your model
• You can see if the answer the model gives you is any relevant to your initial problem. It its not, then it may be best to use some other algorithm.

Answer 5

When we train a model using a data train, sometimes the resulting score is very high. This makes us believe that our model is very good. But when predicting actual data, the resulting score is very low. Why?? This means that the trained model is overfit (to data train) and fail to predict anything useful on yet-unseen data.

That's why we have to check our model to test data (predict test data) and compare the accuracy between data train and data test. If the accuracy is not too far away, then our model does not overfit.

Later, we can improve our model with Cross Validation (Reference) that split our data train to n-split data and take one of that to became data train. Then we take the average of Cross Validation score.

Answer 6

One way to test the accuracy of trained model is by testing it using data test. By testing we are able to check the accuracy. Whether your model is a good model or not depends heavily on the accuracy, if your accuracy is too low or too high (eg. up to 99%~100%), there could be some problem in your model.

For further information on the example in data test, you can access https://jakevdp.github.io/PythonDataScienceHandbook/05.05-naive-bayes.html

Hope this helps