I'd like to ask for any kind of assistance regarding the following problem:

I was given the following training data: 100 numbers, each one is a parameter, they together define a number X(also given).This is one instance,I have 20 000 instances for training.Next, I have 5000 lines given, each containing the 100 numbers as parameters.My task is to predict the number X for these 5000 instances.

I am stuck because I only know of the sigmoid activation function so far, and I assume it's not suitable for cases like this where the output values aren't either 0 or 1.

So my question is this : What's a good choice for an activation function and how does one go about implementing a neural network for a problem such as this?


4 Answers 4


Lets mock some data up.

"100 numbers, each one is a parameter, they together define a number X(also given)"

# i.e. size of X_train -> [n x d]
# i.e. size of X_train -> [??? x 100]  , when d = 100

# "I have 20000 instances for training"
# i.e. size of X_train -> [20000 x 100], when n = 20000

import torch
import numpy as np

X_train = torch.rand((20000, 100))
X_train = np.random.rand(20000, 100) # Or using numpy

But what is your Y?

# Since the definition of a regression task, 
# loosely means to predict an output real number
# given an input of d dimension

# So the appropriate Y_train would 
# be of dimension [n x 1] 
# and look like this:

y_train = torch.rand((20000, 1))

y_train = np.random.rand(20000, 1) # Or using numpy

What is a linear perceptron?

Taking definition from this tutorial

enter image description here

Thus, in picture:

enter image description here

Next we need to define training routine,

For now take it as biblical truth that this is an okay routine to train a neural net model (this isn't the only way but easiest or supervised learning):

enter image description here

In code:

import math
import numpy as np

def sigmoid(x): # Returns values that sums to one.
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(sx): 
    # See https://math.stackexchange.com/a/1225116
    # Hint: let sx = sigmoid(x)
    return sx * (1 - sx)

def cost(predicted, truth):
    return np.abs(truth - predicted)

num_epochs = 10000 # No. of times to iterate.
learning_rate = 0.03 # How large a step to take per iteration.

# Lets standardize and call our inputs X and outputs Y
X = np.array(torch.rand((20000, 100)))
Y = or_output

for _ in range(num_epochs):
    layer0 = X

    # Step 2a: Multiply the weights vector with the inputs, sum the products, i.e. s
    # Step 2b: Put the sum through the sigmoid, i.e. f()
    # Inside the perceptron, Step 2. 
    layer1 = sigmoid(np.dot(X, W))

    # Back propagation.
    # Step 3a: Compute the errors, i.e. difference between expected output and predictions
    # How much did we miss?
    layer1_error = cost(layer1, Y)

    # Step 3b: Multiply the error with the derivatives to get the delta
    # multiply how much we missed by the slope of the sigmoid at the values in layer1
    layer1_delta = layer1_error * sigmoid_derivative(layer1)

    # Step 3c: Multiply the delta vector with the inputs, sum the product (use np.dot)
    # Step 4: Multiply the learning rate with the output of Step 3c.
    W +=  learning_rate * np.dot(layer0.T, layer1_delta)

Now that we learn the model, i.e. the W.

When we see the data points that we need to use the model on, we apply the same forward propagation step, i.e. layer1 = sigmoid(np.dot(X, W))

Since we have:

I have 5000 lines given, each containing the 100 numbers as parameters.My task is to predict the number X for these 5000 instances.

And in code:

# If we mock up the data,
# it should be the same internal dimension. 
X_test = np.random.rand(5000, 100)

# The desired output just needs to pass through the W and the activation:
# the shape of `output` -> [5000 x 1] , 
# where there's 1 output value for each input.
output = sigmoid(np.dot(X_test, W))
  • $\begingroup$ There's some weird things happening here when we design our network with a sigmoid, we naturally force the output values to be between the (0, 1) range. But this is partly because when we mock up the data using np.random.rand, the values are expected to fall between the (0,1). $\endgroup$
    – alvas
    May 11, 2020 at 16:00
  • $\begingroup$ A normal regression task where the output has arbitrary range of values, simply skip the sigmoid layer for the forward propagation and then the error derivatives in the sigmoid_derivative(layer1) would just be 1. $\endgroup$
    – alvas
    May 11, 2020 at 16:01

The quick answer is that you want to use an activation function on the output layer that does not compress values to $(0,1)$. Depending on your software, this might be called "linear" or "identity". It looks like Keras just wants you to leave off the activation function: model.add(Dense(1)).

The typical way of thinking of a neural network as a classifier (let's say a binary classifier) is just extending a logistic regression. In fact, when you use a sigmoid activation function on the output node, you're (sort of) running logistic regression on the final hidden layer.

A logistic regression is one type of generalized linear model. The gist of GLM is that some transformation of the the value of interest is a linear function of the feature space.

Let $X$ be the data matrix for the feature space. Let $\beta$ be a parameter vector. Then $\hat{y} = \mathbb{E}[y] = X\beta$ is the linear model, and $g(\mathbb{E}[y]) = X\beta$ is the generalized linear model (vectorized, so apply $g$ to each $y_i$).

But we could extend this to a nonlinear transformation, and when a neural network is a binary classifier, this is precisely what we're doing. Instead of the transformation of $X$ being given by $\beta$ and thus linear, we apply some nonlinear transformation $f$ and get $g(\mathbb{E}[y]) = f(X)$.

The terminology in GLM is "link function", but that is essentially the activation function on the final node(s) of the neural network. Consequently, all of the GLM link functions are in play, and one of those link functions is the identity function. For a GLM, that's just linear regression. For your neural network, it will be a neural network (nonlinear) regression, which sounds like what you want.


Usually you're normalizing the data first, meaning that your whole dataset will be in between 0 and 1. Afterwords after you're having the model predictions, when computing the cost function or evaluating the model, you can apply the inverse of the normalization function.

  • $\begingroup$ I'm a little confused now, the point of the sigmoid function is to put the data set between 0 and 1? $\endgroup$ Dec 11, 2019 at 23:35
  • 1
    $\begingroup$ No, you use sigmoid to introduce non-linearity in your model. In addition sigmoid is especially used for models where we have to predict the probability as an output since it's defined in the range of 0 and 1. For normalizing the data i would suggest you to look at min-max normalization or z-score normalization. $\endgroup$
    – razvanc92
    Dec 12, 2019 at 8:13

for regression, you can use a hidden layer with sigmoid, then a LINEAR output layer, where the weighted sum goes straight through, without modification.

this way your output is not restricted to 0-1

  • $\begingroup$ Alternatively, you may use ReLu activation funtion. ReLu stands for Rectified Linear units and is defined as : max(x, 0) So , when dealing with problem where the output is not restricted just to 0-1, you may want to use "ReLu" activation function for all the hidden layers and "linear" activation function for the output layer. $\endgroup$ May 11, 2020 at 18:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .