I am working on a classification problem.
I have a dataset $S$ and I am training several prediction algorithms using S: Naive Bayes, SVM, classification trees.
Intuitively, I was planning to combine my models, and, for each data point in the test sample $S'$, take the majority vote as my prediction.
Does that make sense? I feel this is a very simplistic way to combine different models.
These are generally known as ensemble methods. Your method is essentially what Scikit-Learn's VotingClassifier does, which is perfectly reasonable and might give you better results. Of course, if you have an ensemble of classifiers and some of them perform quite poorly, the ensemble might not be able to beat the best classifier: you'll need to check this in your cross-validation. Be aware that any confidence or probability estimates may not be well-calibrated, so the predictions from the ensemble may not be particularly meaningful.
There are more elaborate ways of ensembling classifiers. Random forest classifiers, for example, are just ensembles of decision trees; a technique called bagging is also used here to improve performance.
The technique of using a second model to weight the predictions from an ensemble is known as stacked generalisation. It was introduced by Wolpert in 1991 [1], and you can find plenty of interesting examples of using the technique, e.g. on Kaggle.
[1] David H. Wolpert. Stacked generalization. Neural Networks 5.2 (1992), pp. 241-259.
It is a simple way to do it but it is not wrong.
If you are getting probabilities for each model, then, you can average them. Then, you can do the classification.
Also, you assign weights to each model based on a validation set and regressing the weights for each models prediction.
