0
$\begingroup$

What I want to achieve is this: If my desired outputs are [1, 2, 3, 4] I would rather have my network produce this output:

[0.99, 2.01, 999, 4.01]

than say this:

[0.94, 1.88, 3.12, 4.1]

So I'd rather have a few very accurate outputs and the rest completely off, than have them all be decent but no more than that. My question is, is there a known way to do this? If not, would it make sense to remove the inputs that produce poor outputs, and redo the learning phase?

Improve this question
$\endgroup$
3
$\begingroup$

I assume [1, 2, 3, 4] are the desired outputs for different examples for a regression task. Sound like you need a different loss function. From your description it seems you don't care how big the error is if it's bigger than some value. Try the Huber loss(in Pytorch and TensorFlow). Examples that are far from the expected value won't produce big gradients (:

Improve this answer
$\endgroup$
4
  • $\begingroup$ Yes that's correct. Thank, I will check that out $\endgroup$ – user1477107 Sep 3 '20 at 19:29
  • $\begingroup$ Can I just make the gradient 0 if the error is too large? So I have 2 loss funtions, depending on the error. One is zero and the other one is quadratic or whatever else $\endgroup$ – user1477107 Sep 5 '20 at 14:52
  • $\begingroup$ If you implement the loss function to be a constant, when the error is too large, you’ll get this behavior. The best way to achieve it is using torch.where/tf.where. Note that this might not be optimal. If many examples have an error larger than the threshold, the model won’t train or it will overfit on a subset of the examples. Can you provide more context? $\endgroup$ – dbalchev Sep 6 '20 at 15:08
  • $\begingroup$ I'm trying to predict stock prices so the idea is after an initial phase of learning, when no error is very large and the network has a decent grasp of the function, to ditch the worst outputs in order to get better accuracy on the good outputs $\endgroup$ – user1477107 Sep 6 '20 at 16:40
0
$\begingroup$

Use genetic algorithm. Run like 25 neural networks at once and choose the most successful one. This method is similar to evolution, which is why it is very effective. I created a model like this with similar sized training data as yours and it reached an overall error rate of 0.06% in a second. Don’t get rid of nodes. Instead, eliminate the bad networks. However, this doesn’t produce extremely high error rates if that is what you want.

Improve this answer
$\endgroup$
3
  • $\begingroup$ They don't have to be high, I just wanted to say that if error is larger than x it might as well be a gazillion, I don't care. But you're just saying to run the network many times, not necessarily at the same time? $\endgroup$ – user1477107 Sep 3 '20 at 8:57
  • $\begingroup$ Yes. Also consider increasing or decreasing your mutation amount (random increase to weights) based on how well the networks are performing. $\endgroup$ – Kral Sep 3 '20 at 15:41
  • $\begingroup$ This will help increase network accuracy, but doesn't address the specific question about prioritising a few correct outputs over punishing very incorrect outputs. Addressing this ultimately comes down to the loss function, as that determines how to handle bad outputs. $\endgroup$ – Recessive Sep 4 '20 at 2:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.