1
$\begingroup$

I've selected more than 10 discriminative (classification) models, each wrapped with a BaggingClassifier object, optimized with a GridSearchCV, and all of them placed within a VotingClassifier object.

Alone, they all bring around 70% accuracy, on a data set which is about half normal/uniform distributed, and half one-hot distributed. Together, they provide 80% accuracy, which isn't good enough, given that I was told that 95%< is achievable.

The models: DecisionTreeClassifier, ExtraTreesClassifier, KNeighborsClassifier, GradientBoostingClassifier, LogisticRegression, SVC, Perceptron, and a few more classifiers.

How do I check if the combination is good?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Yeah, you can’t. If it works well it’s good. It’s a fairly odd combination tbh $\endgroup$
    – Andreas Storvik Strauman
    Jun 23, 2018 at 19:21
  • $\begingroup$ The accuracy value can only be compared with other models. Having a goal is good, but unless you have a strong indication that your goal is achievable given the data in question, then just quoting a raw value like 70% or 99.9% is meaningless. How do human experts do given the same raw data? $\endgroup$
    – Neil Slater
    Jun 23, 2018 at 20:28
  • $\begingroup$ Neil - It's for a HW assignment, and they claim 95%< is possible, probably too many features for a human to do 95%< . Andreas - I just took all the models I've found, with 70%< accuracy. What would be a good way to choose models ? and how many are probably needed (18 features data, half normal/uniform half one-hots) ? $\endgroup$
    – Miko Diko
    Jun 23, 2018 at 21:21
  • 1
    $\begingroup$ OK, so if you trust your course materials, and 95% has been done, then you are missing something major. Were you expected to make a large ensemble as part of the task? I think you may be expected to do some feature engineering too - what course materials have you studied, and have you been told which ones are relevant to the task? $\endgroup$
    – Neil Slater
    Jun 24, 2018 at 17:28

1 Answer 1

Reset to default
2
$\begingroup$

Goodness is subjective. Reliable knowledge isn't possible with that flimsy a quality objective.

The sturdy objective criteria you gave is 95%, so it is bad by that criteria. (I'm assuming that the 95% is expected for a given data set or a randomized sample from a given data set.)

However, the 80% accuracy is good by the criteria where the you sum the measures of the accuracy of the individual models, divide by the number of models, and find you have gained ten percentage points of accuracy over that average with your aggregated execution strategy. (I'm assuming here that you used a defined set of network meta-parameters, layers depths and widths, starting parameters, activation model mapping to layers, inter-network connectivity, and loss/error methods for each model that is similar to the aggregated execution strategy.)

I have four questions. (My apologies that this question leads to the two assumptions above and a bunch more questions.)

  • Is the 80% accuracy also well over the maximum of the accuracies of the set of accuracies from the individual models?
  • Is the computing resource draw to achieve 80% accuracy achievable with additional run time of the best of the individual models, using less than or equal to the computing resource draw to achieve 80% that way?
  • Have you run your evaluation with a full set of meta parameter vectors to check the entire meta-space for best case?
  • What economic, contractual, or operational hard stop is dictating the 90%?

If we know these answers, we may be able to respond more effectively and possibly find a loop hole in the logic that appears to leave you with an undesirable foregone conclusion.

Improve this answer
$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .