How to transform inputs and extract useful outputs in a Neural Network?

Question

So I've been trying to understand neural networks ever since I came across Adam Geitgey's blog on machine learning. I've read as much as I can on the subject (that I can grasp) and believe I understand all the broad concepts and some of the workings (despite being very weak in maths), neurons, synapses, weights, cost functions, backpropagation etc. However, I've not been able to figure out how to translate real world problems into a neural network solution.

Case in point, Adam Geitgey gives as an example usage, a house price prediction system where given a data set containing No. of bedrooms, Sq. feet, Neighborhood and Sale price you can train a neural network to be able to predict the price of a house. However he stops short of actually implementing a possible solution in code. The closest he gets, by way of an example, is basic a function demonstrating how you'd implement weights:

def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):
  price = 0

  # a little pinch of this
  price += num_of_bedrooms * 1.0

  # and a big pinch of that
  price += sqft * 1.0

  # maybe a handful of this
  price += neighborhood * 1.0

  # and finally, just a little extra salt for good measure
  price += 1.0

  return price 

Other resources seem to focus more heavily on the maths and the only basic code example I could find that I understand (i.e. that isn't some all singing, all dancing image classification codebase) is an implementation that trains a neural network to be an XOR gate that deals only in 1's and 0's.

So there's a gap in my knowledge that I just can't seem to bridge. If we return to the house price prediction problem, hows does one make the data suitable for feeding into a neural network? For example:

• No. of bedrooms: 3
• Sq. feet: 2000
• Neighborhood: Normaltown
• Sale price: $250,000

Can you just feed 3 and 2000 directly into the neural network because they are numbers? Or do you need to transform them into something else? Similarly what about the Normaltown value, that's a string, how do you go about translating it into a value a neural network can understand? Can you just pick a number, like an index, so long as it's consistent throughout the data?

Most of the neural network examples I've seen the numbers passing between layers are either 0 to 1 or -1 to 1. So at the end of processing, how do you transform the output value to something usable like $185,000?

I know the house price prediction example probably isn't a particularly useful problem given that it's been massively oversimplified to just three data points. But I just feel that if I could get over this hurdle and write an extremely basic app that trains using pseudo real-life data and spits out a pseudo real-life answer than I'll have broken the back of it and be able to kick on and delve further into machine learning.

Answer 1

This is a good question which I wrestled with myself when first trying to code an ANN.

Below is a good general-purpose solution, and it's the one I implemented in my code for trying to predict well-behaved numerical data. If your data is not well-behaved (i.e. fraught with outliers) then you may need to do more work normalizing the inputs and outputs. Some of the more advanced methods are described here.

Note: I will assume that you are using f(x) = tanh(x) as your activation function. If you aren't, you should still be able to reason through how to normalize your data after reading this.

How to prepare the input data:

The basic idea is that you want a significant variation in each input parameter to be reflected by a significant variation in the activation of the neuron those inputs are being fed into. By looking at a plot of the derivative of the tanh(x) actiavtion function, you'll see that the region of significant slope is within a distance of one or two from the origin. This means that whether the input to the activation function is 2000 or 3000 (values of x for which the derivative is negligibly small), the output of the activation will be almost identical...so your neuron's state will be independent of the difference between 2000 and 3000, and your network will never produce any predictive power from values in that range.

So if you want to input the square footage of the house into a neuron, you need to normalize the square footage so that the network can tell the difference between 2000 and 3000. One way to do this so that all of the significant variations in your data are 'noticed' by the neuron is to z-score-normalize the inputs.

• Gather all of your square footage values (from your training set) and calculate the mean and standard deviation. Store the mean and standard deviation---you'll need this information to normalize new square footage values when testing.

• Normalize the vector of square footage values by subtracting the mean and then dividing the result by the standard deviation (all operations element-wise of course). Subtracting the mean centers your data at the origin, and dividing by the standard deviation makes sure most of it is between -1 and 1, where the neuron's output is most sensitive to its input. This is called z-score normalization because each input value is replaced by its z-score.

• Do the above for each input variable.

Now, when you put each input value through a neuron, the output of the neuron is an activation between -1 and 1 (look at the image of tanh(x)). Since this is already in the 'sensitive' range of the activation function, you don't need to worry about altering the output of the input-layer neurons before sending them to the first hidden layer. Just give any hidden layer neurons the outputs of the previous layer directly---they will be able to handle them just fine.

When you reach the last layer (the output neuron(s)), what you get is again another activation between -1 and 1. You have to convert this back into a value for the house in question, whether that value will be used as a prediction in a test set or to calculate error during training. However you do this, you just have to be consistent and use the same de-normalization procedure in training and testing. One way to think about it is: when the output neuron(s) returns 1, that means the network is returning the maximum possible house value as its prediction. What should the highest value the network can estimate be? The right approach here simply depends on your application. This is what I did:

• Calculate the mean of [the/each] output variable and store it.
• Calculate the maximum deviation of the output variable from the mean. Python: MaxDev = max([abs(DataPoint-numpy.mean(TrainingData)) for DataPoint in TrainingData])
• When the network returns output(s) between -1 and 1, multiply the output by MaxDev and add it to the mean.

Two basic quick checks you can do to see if your normalization-renormalization scheme is suitable (these are necessary, but perhaps not sufficient conditions):

1. If all the input values are average (e.g. average no. of bedrooms, average sq.feet, etc), is the network's output equal to the average of the output variable (e.g. house value) as well? (It should be.)
2. If all the input values are unusually high/low, is the network's output unusually high/low as well? (This only works if all the inputs are positively related to the output...if some of them are inversely related related, you will have to think a bit more).

Observe that the scheme presented here satisfies these two conditions.

Notice that this scheme would allow your network to only predict house values inside the range of house values in your training data set. Depending on the application, this behavior can be desirable or undesirable.

For example: you may want to make it impossible for your network to predict negative house values. Think about how you would do this. De-normalize the output so that -1 is mapped to 0.

If you want to set no limit on the values your network can predict, then you can run the network's output through a function that maps the [-1,1] range to all real numbers...like arctanh(x)! As long as you do this during training your network will adjust its weights to accommodate this.

I hope this was helpful. Let me know if you have further questions. My ANN module is in Python, by the way, so I might have language-specific advice.