Background I have tried to fit a logistic regression model - written using a forward / back propagation approach (as part of Andrew Ng's deep learning course) - to a very non-linear data set (see picture below). Of course, it totally fails; in Andrew Ng's course, the failure of logistic regression to fit to this motivates developing a neural net - which works quite nicely. But my question concerns what my logistic model is doing and why.

non-linear data set

The problem My logistic regression model's cost increases, even after massively reducing the learning rate. But at the same time my accuracy (slowly) increases. I simply cannot see why.

To confuse matters even more - if I resort to a negative learning rate (essentially trying to force the calibration to higher cost values) the cost then decreases for a time until the accuracy hits 50%. After this point, the cost then inexorably increases - but the accuracy stays equal to 50%. The solution so found is to set all points to either red or blue (a reasonable fit given logistic regression simply cannot work on this data).

My questions and thoughts on answers I have reproduced the Python code below - hopefully it's clear. My questions are:

  1. Is there a mistake in the model that explains why negative learning rates seem to work better?
  2. On the topic of why the cost increases even as accuracy asymptotes to 50%: is the issue that once the model has discovered the "all points equal to either red or blue" solution the parameters "w" and "b" just get larger and larger (in absolute terms) - driving all of the predictions closer to 1 (or conversely if it predicts all points are 0)?

To explain this second question a bit more: imagine red points are defined by y = 1. Suppose parameters w, b are chosen such that the probability for every point equals 0.9. Then the model predicts all points are red - which is correct for half the points. The model can then improve half the predictions by driving w and b up (so that sigmoid ( w*x + b) --> 1). But of course, this makes half the predictions (the blue points) more and more wrong - which causes the cost function for those points - log(1 - prob) - to diverge. I don't truly see why gradient descent would do this but it's all I can think of for the peculiar behaviour of the algorithm.

Hope this all makes sense. Hit me up if not.

import numpy as np
import matplotlib.pyplot as plt

# function to create a flower-like arrangement of 1s and 0s
def load_planar_dataset():
    m = 400 # number of examples
    N = int(m/2) # number of points per class
    D = 2 # dimensionality / i.e. work in 2d plane - so X is a set of (x,y) coordinate points
    X = np.zeros((m,D)) # data matrix where each row is a single example
    Y = np.zeros((m,1), dtype='uint8') # labels vector (0 for red, 1 for blue)
    a = 4 # maximum ray of the flower

    for j in range(2):
        ix = range(N*j,N*(j+1))
        t = np.linspace(j*3.12,(j+1)*3.12,N) + np.random.randn(N)*0.2 # theta / random element mixes up some of the petals so you get mostly blue with some red petals and vice-versa
        r = a*np.sin(4*t) + np.random.randn(N)*0.2 # radius / again random element alters  shape of flower slightly
        X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
        Y[ix] = j
    X = X.T # transpose so columns = training example as per standard in lectures
    Y = Y.T

    return X, Y

# function to plot the above data plus a modelled decision boundary - works by applying model to grid of points and colouring accordingly
def plot_decision_boundary(model, X, y):
    # Set min and max values and give it some padding
    x_min, x_max = X[0, :].min() - 1, X[0, :].max() + 1
    y_min, y_max = X[1, :].min() - 1, X[1, :].max() + 1
    h = 0.01
    # Generate a grid of points with distance h between them
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Predict the function value for the whole grid
    Z = model(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
    plt.scatter(X[0, :], X[1, :], c=y, cmap=plt.cm.Spectral)

# sigmoid function as per sandard linear regression
def sigmoid(z):
    Compute the sigmoid of z

    z -- A scalar or numpy array of any size.

    s -- sigmoid(z)

    s = 1. / (1. + np.exp(-z))
    return s

def propagate(w, b, X, Y):
    Implement the cost function and its gradient for the propagation explained above

    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b

    m = X.shape[1];

    # forward prop
    Z = np.dot(w.T, X) + b; 
    A = sigmoid(Z); # activiation = the prediction of the model

    # compute cost
    cost =  - 1. / m * np.sum( (Y * np.log(A) + (1. - Y) * np.log(1. - A)  ) )

    #back prop for gradient descent

    da = - Y / A + (1. - Y) / (1. - A)  
    dz = da * A * (1. - A) # = - Y (1-A) + (1. - Y) A =  A - Y  
    dw = 1. / m * np.dot( X, dz.T )
    db = 1. / m * np.sum(dz)

    grads = {"dw": dw,
                "db": db}

    return grads, cost

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
    This function optimizes w and b by running a gradient descent algorithm
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of shape (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- True to print the loss every 100 steps
    params -- dictionary containing the weights w and bias b
    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
    costs = []

    for i in range(num_iterations):
        # cost /gradient calculation
        grads, cost = propagate(w, b, X, Y)

        #retrieve derivatives
        dw = grads["dw"]
        db = grads["db"]

        # update values according to gradient descent algorithm
        w = w - learning_rate * dw
        b = b - learning_rate * db

        # record the costs
        if i % 100 == 0:

            # Print the cost every 100 training iterations
            if print_cost:
                print("Cost after iteration %i: %f" %(i, cost))

    params = {  "w": w,
                "b": b}

    grads = {   "dw": dw,
                "db": db}

    return params, grads, costs

def predict(w, b, X):
    Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
    Z = np.dot(w.T, X) + b
    A = sigmoid(Z)

    Y_prediction = (A >= 0.5).astype(int)

    return Y_prediction

np.random.seed(1) # set a seed so that the results are consistent

X, Y = load_planar_dataset()

# Visualize the data:

plt.scatter(X[0, :], X[1, :], c=Y, s=40, cmap=plt.cm.Spectral); # s = size of points; cmap are nicer colours

shape_X = X.shape
shape_Y = Y.shape
m = shape_Y[1]  # training set size
n = shape_X[0] # number of features (2)

# initialise parameters
w = np.random.rand(n, 1)
b = 0

# print accuracy of initial parameters by comparing prediction to 
print("train accuracy: {} %".format(100 - np.mean(np.abs(predict(w, b, X) - Y)) * 100))

# fit model and print out costs every 100 iterations of the forward / back prop
parameters, grads, costs = optimize(w, b, X, Y, num_iterations = 10000, learning_rate = 0.000005, print_cost = True)

# return the prediction
Y_prediction = predict(parameters["w"], parameters["b"], X)

# print accuracy of fitted model
print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction - Y)) * 100))

# print parameters for interest
print( parameters["w"] , parameters["b"] )

# plot decision boundary
plot_decision_boundary(lambda x: predict(parameters["w"], parameters["b"], x.T), X, Y)

List item

Just realised the issue - super subtle (at least for a Python novice like me) - I implemented numerical gradient checking (as I should have done from the start) and saw the gradient descent was working incorrectly.

The dataset Y was created as a datatype "uint8". In the line of the back-propagation where I calculate the derivative with respect to "a" ("da") the statement "-Y" was returning 255 for those values of Y that were 1 - this appears to be a feature of these unsigned binary numbers.

By replacing "uint8" with "int8" in the definition of Y - or else forcing the correct behaviour by writing "-1.*Y" in the calculation of "da" (which I think casts "Y" as an integer rather than an unsigned integer) I managed to produce the correct behaviour.

This in turn means the gradient descent is doing what it should and the logistic regression converges to a stable value of the cost. Accuracy then tracks cost as well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.