16 events
when toggle format what by license comment
Jun 25 '20 at 0:44 comment added Sports_Stats The hardest part to fathom is the fact that after 815 results from A System, which had a 58% Accuracy when only needing to be 52.5% accurate, is that those results can't be trusted to be better than 52.5% over the next 800 results?
Jun 24 '20 at 20:13 comment added Sports_Stats Oops....I imported a different csv....here are the correct results for A System: Cross Validation Accuracy: 0.54 (+/- 0.10) / K-Fold Accuracy: 0.50 (+/- 0.12) / Stratified K-Fold Accuracy: 0.48 (+/- 0.11)
Jun 24 '20 at 20:07 comment added Neil Slater Hmm, I would expect those standard errors to be lower (around 0.05 if a single measure was around 0.16). I'm not sure what's up there, I don't know your library and how it reports things.
Jun 24 '20 at 20:06 comment added Sports_Stats Ok.......I think I figured out where I went wrong and in addition to running K-Fold, I also ran Stratified K-Fold both with n_splits=10 and random_state=0, here are the results (including my original cross_val_score: Cross Validation Accuracy: 0.55 (+/- 0.18) / K-Fold Accuracy: 0.49 (+/- 0.16) / Stratified K-Fold Accuracy: 0.52 (+/- 0.12).
Jun 24 '20 at 18:59 comment added Neil Slater @Sports_Stats: That doesn't seem right. If you run k fold, you should simply get multiple Accuracy values, and be told a mean Accuracy and (reduced) standard error. For instance if you ran 10-fold cross validation on your data set, I would expect you see a mean accuracy somewhere around 0.50 (50%) +- 0.03 (+- 3%) - it's the reduction of the standard error that you care about, it means you can trust the CV values a bit more.
Jun 24 '20 at 18:49 comment added Sports_Stats That unfortunately looks like the case as I just received a KFold Validation of -0.251765602234286 with a 57% Testing Accuracy. Which I guess I subtract the KFold Validation from the Testing Accuracy and receive a 32% Accuracy?
Jun 24 '20 at 18:18 comment added Neil Slater I hope you do well. In all likelihood you are about to discover a deeper rabbit hole as the kinds of predictions you are looking at are hard to get right, and model accuracy can drift a lot over time as factors you are not including change. I think FiveThirtyEight might make some good reading for you - fivethirtyeight.com/tag/sports-betting Thanks for the offer of sharing results, but I am not really a betting person so I would not use them. If they ever get good, keep them to yourself :-)
Jun 24 '20 at 17:45 comment added Sports_Stats Much Appreciated Neil! I truly thank you for all your responses and helpfulness. I'll go ahead and implement the K-Fold Validation and see what results I can come up with. Also, if I can get this algorithm to prove that it can be profitable on future predictions, and if you're interested, I'll send you my future picks :)
Jun 24 '20 at 17:36 comment added Neil Slater @Sports_Stats: The +-0.16 is standard error on the CV Accuracy, which is not reported as a percentage but a probability, so it's a lot worse than that. It means you are 95% confident that the "true" accuracy is somewhere between 41% or 73%. This is complicated by you then using the value of 0.57 to select this as "the best", so there is also an unmeasured (and difficult to assess) bias in there. That range is very large, probably because you don't have much data to spare for cross validation, which is why I suggest k-fold validation.
Jun 24 '20 at 17:28 comment added Sports_Stats Hey Neil, The Cross Validation Accuracy for C System is 57% with a Standard Deviation of (+/- 0.16)​. Doesn't that indicate that 95% of future predictions will be between 57-0.16= 56.84% and 57 + 0.16 = 57.16% ?
Jun 23 '20 at 17:35 comment added Neil Slater @Sports_Stats: Given your SD on the estimates is +- 0.08 I would not get too excited about small improvements you see by tweaking params - they could just be luck and not represent true values. You could maybe look into k-fold cross validation to try and get that variance in the estimate down
Jun 23 '20 at 17:21 comment added Sports_Stats Sorry for the confusion but to clear things up: The Accuracy label under the CONFUSION MATRIX label is == Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2) and scores = cross_val_score(clf, X, y, cv=10) Also, when I change the random_state from 42 to 0, my new results are much better: Train Accuracy: 0.6306306306306306 Test Accuracy: 0.5803571428571429 Cross Validation Accuracy: 0.55 (+/- 0.08)
Jun 23 '20 at 17:15 comment added Neil Slater @Sports_Stats: Actually your question does not mention CV and states "Train Accuracy"? There is an addional "Accuracy" value at the end - is that the CV accuracy? If so, the CV results are also pushing you to pick model C, so you are in the clear to simply trust what the test accuracy is there and not otherthink things :-)
Jun 23 '20 at 17:13 comment added Neil Slater @Sports_Stats: Sorry I didn't spot you were already using CV. In general CV accuracy is tainted by using it to select the best agent, so it will be an over-estimate. So yes you should use test accuracy as a guage for how well that model will do "in production". You also need to keep selecting your best agent according to CV, don't select by test accuracy (otherwise you cannot trust it is unbiased). The worst thing you could do here is look through the models, compare test accuracy and pick the best.
Jun 23 '20 at 17:08 comment added Sports_Stats Hi Neil, Thanks for responding! I am not referring to my Training Accuracy. Taking a looking at my A System Results: Train Accuracy: 0.6211656441717791 Test Accuracy: 0.5153374233128835 F1 Score: 0.52 CONFUSION MATRIX: [[16 50] [29 68]] Cross Validation Accuracy: 0.55 (+/- 0.10) Should I trust the Test Accuracy over the Cross Validation Accuracy?
Jun 23 '20 at 7:41 history answered Neil Slater CC BY-SA 4.0