Let's say there's a ball with features position, velocity, acceleration.

These three are all concatenated as inputs to my neural network.

However, I have prior knowledge that position is way more predictive than the other features.

How do I weight the position feature much more strongly than the others? Would just applying a large scalar coefficient to it as preprocessing work? Seems unprincipled...


If you are training a neural network, it should learn correct weights to use the most predictive feature without any interference. For a strongly predictive feature matched with weaker ones, most NNs will learn the stronger association very quickly. I'm not sure whether there is an easy way to add this as a prior. You could do things such as inject the most influential feature in one or more hidden layers, but that does seem ugly, and only really helps if the predictive feature has certain types of relationship to output (it would probably help most if there is a strong linear relationship).

Training problems are usually in the opposite direction - how to extract the small amount of information from more noisy and less influential features that may still add up and sometimes contradict the main predictor.

The only thing that springs to mind for NNs that may help in this situation (mostly about fixing the problem of using the less reliable features) is the architecture of residual neural networks. The use of skip connections builds the neural network in such a way that a default "do nothing" identity function is encouraged as layers are added, and this allows for combining weak but complex relationships with stronger simple ones. A residual NN can do this with less compromise than changing network hyper parameters to find best fit on other NNs.

However, residual neural networks are good for managing different degrees of complexity that manifest within a problem, which is not necessarily the same as different degrees of predictive power between features.

Would just applying a large scalar coefficient to it as preprocessing work?

No, this would usually be counter-productive for training a neural network. You should instead be normalising all inputs - a common and effective technique for neural networks is to ensure that each feature is scaled and offset so it has mean 0, standard deviation 1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.