3
$\begingroup$

I've selected more than 10 discriminative (Classification) models, each wrapped with a BaggingClassifier object, optimized with a GridSearchCV, and all of them placed within a VotingClassifier object.

Alone they all bring around 70% accuracy, on a data set which is about half normal/uniform distributed, and half one-hot distributed. Together they provide 80% accuracy, which isn't good enough, to the 95%< needed.

The models: DecisionTreeClassifier, ExtraTreesClassifier, KNeighborsClassifier, GradientBoostingClassifier, LogisticRegression, SVC, Perceptron, and a few more classifiers.

How do I check if the combination is good?

$\endgroup$
  • 2
    $\begingroup$ Yeah, you can’t. If it works well it’s good. It’s a fairly odd combination tbh $\endgroup$ – Andreas Storvik Strauman Jun 23 '18 at 19:21
  • $\begingroup$ The accuracy value can only be compared with other models. Having a goal is good, but unless you have a strong indication that your goal is achievable given the data in question, then just quoting a raw value like 70% or 99.9% is meaningless. How do human experts do given the same raw data? $\endgroup$ – Neil Slater Jun 23 '18 at 20:28
  • $\begingroup$ Neil - It's for a HW assignment, and they claim 95%< is possible, probably too many features for a human to do 95%< . Andreas - I just took all the models I've found, with 70%< accuracy. What would be a good way to choose models ? and how many are probably needed (18 features data, half normal/uniform half one-hots) ? $\endgroup$ – Miko Diko Jun 23 '18 at 21:21
  • $\begingroup$ OK, so if you trust your course materials, and 95% has been done, then you are missing something major. Were you expected to make a large ensemble as part of the task? I think you may be expected to do some feature engineering too - what course materials have you studied, and have you been told which ones are relevant to the task? $\endgroup$ – Neil Slater Jun 24 '18 at 17:28
1
$\begingroup$

Goodness is subjective. Reliable knowledge isn't possible with that flimsy a quality objective.

The sturdy objective criteria you gave is 95%, so it is bad by that criteria. (I'm assuming that the 95% is expected for a given data set or a randomized sample from a given data set.)

However, the 80% accuracy is good by the criteria where the you sum the measures of the accuracy of the individual models, divide by the number of models, and find you have gained ten percentage points of accuracy over that average with your aggregated execution strategy. (I'm assuming here that you used a defined set of network meta-parameters, layers depths and widths, starting parameters, activation model mapping to layers, inter-network connectivity, and loss/error methods for each model that is similar to the aggregated execution strategy.)

I have four questions. (My apologies that this question leads to the two assumptions above and a bunch more questions.)

  • Is the 80% accuracy also well over the maximum of the accuracies of the set of accuracies from the individual models?
  • Is the computing resource draw to achieve 80% accuracy achievable with additional run time of the best of the individual models, using less than or equal to the computing resource draw to achieve 80% that way?
  • Have you run your evaluation with a full set of meta parameter vectors to check the entire meta-space for best case?
  • What economic, contractual, or operational hard stop is dictating the 90%?

If we know these answers, we may be able to respond more effectively and possibly find a loop hole in the logic that appears to leave you with an undesirable foregone conclusion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.