Background I have tried to fit a logistic regression model - written using a forward / back propagation approach (as part of Andrew Ng's deep learning course) - to a very non-linear data set (see picture below). Of course, it totally fails; in Andrew Ng's course, the failure of logistic regression to fit to this motivates developing a neural net - which works quite nicely. But my question concerns what my logistic model is doing and why.
The problem My logistic regression model's cost increases, even after massively reducing the learning rate. But at the same time my accuracy (slowly) increases. I simply cannot see why.
To confuse matters even more - if I resort to a negative learning rate (essentially trying to force the calibration to higher cost values) the cost then decreases for a time until the accuracy hits 50%. After this point, the cost then inexorably increases - but the accuracy stays equal to 50%. The solution so found is to set all points to either red or blue (a reasonable fit given logistic regression simply cannot work on this data).
My questions and thoughts on answers I have reproduced the Python code below - hopefully it's clear. My questions are:
- Is there a mistake in the model that explains why negative learning rates seem to work better?
- On the topic of why the cost increases even as accuracy asymptotes to 50%: is the issue that once the model has discovered the "all points equal to either red or blue" solution the parameters "w" and "b" just get larger and larger (in absolute terms) - driving all of the predictions closer to 1 (or conversely if it predicts all points are 0)?
To explain this second question a bit more: imagine red points are defined by y = 1. Suppose parameters w, b are chosen such that the probability for every point equals 0.9. Then the model predicts all points are red - which is correct for half the points. The model can then improve half the predictions by driving w and b up (so that sigmoid ( w*x + b) --> 1). But of course, this makes half the predictions (the blue points) more and more wrong - which causes the cost function for those points - log(1 - prob) - to diverge. I don't truly see why gradient descent would do this but it's all I can think of for the peculiar behaviour of the algorithm.
Hope this all makes sense. Hit me up if not.
import numpy as np
import matplotlib.pyplot as plt
# function to create a flower-like arrangement of 1s and 0s
def load_planar_dataset():
np.random.seed(1)
m = 400 # number of examples
N = int(m/2) # number of points per class
D = 2 # dimensionality / i.e. work in 2d plane - so X is a set of (x,y) coordinate points
X = np.zeros((m,D)) # data matrix where each row is a single example
Y = np.zeros((m,1), dtype='uint8') # labels vector (0 for red, 1 for blue)
a = 4 # maximum ray of the flower
for j in range(2):
ix = range(N*j,N*(j+1))
t = np.linspace(j*3.12,(j+1)*3.12,N) + np.random.randn(N)*0.2 # theta / random element mixes up some of the petals so you get mostly blue with some red petals and vice-versa
r = a*np.sin(4*t) + np.random.randn(N)*0.2 # radius / again random element alters shape of flower slightly
X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
Y[ix] = j
X = X.T # transpose so columns = training example as per standard in lectures
Y = Y.T
return X, Y
# function to plot the above data plus a modelled decision boundary - works by applying model to grid of points and colouring accordingly
def plot_decision_boundary(model, X, y):
# Set min and max values and give it some padding
x_min, x_max = X[0, :].min() - 1, X[0, :].max() + 1
y_min, y_max = X[1, :].min() - 1, X[1, :].max() + 1
h = 0.01
# Generate a grid of points with distance h between them
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
# Predict the function value for the whole grid
Z = model(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
# Plot the contour and training examples
plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
plt.ylabel('x2')
plt.xlabel('x1')
plt.scatter(X[0, :], X[1, :], c=y, cmap=plt.cm.Spectral)
# sigmoid function as per sandard linear regression
def sigmoid(z):
"""
Compute the sigmoid of z
Arguments:
z -- A scalar or numpy array of any size.
Return:
s -- sigmoid(z)
"""
s = 1. / (1. + np.exp(-z))
return s
#
def propagate(w, b, X, Y):
"""
Implement the cost function and its gradient for the propagation explained above
Arguments:
w -- weights, a numpy array of size (num_px * num_px * 3, 1)
b -- bias, a scalar
X -- data of size (num_px * num_px * 3, number of examples)
Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)
Return:
cost -- negative log-likelihood cost for logistic regression
dw -- gradient of the loss with respect to w, thus same shape as w
db -- gradient of the loss with respect to b, thus same shape as b
"""
m = X.shape[1];
# forward prop
Z = np.dot(w.T, X) + b;
A = sigmoid(Z); # activiation = the prediction of the model
# compute cost
cost = - 1. / m * np.sum( (Y * np.log(A) + (1. - Y) * np.log(1. - A) ) )
#back prop for gradient descent
da = - Y / A + (1. - Y) / (1. - A)
dz = da * A * (1. - A) # = - Y (1-A) + (1. - Y) A = A - Y
dw = 1. / m * np.dot( X, dz.T )
db = 1. / m * np.sum(dz)
grads = {"dw": dw,
"db": db}
return grads, cost
def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
"""
This function optimizes w and b by running a gradient descent algorithm
Arguments:
w -- weights, a numpy array of size (num_px * num_px * 3, 1)
b -- bias, a scalar
X -- data of shape (num_px * num_px * 3, number of examples)
Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
num_iterations -- number of iterations of the optimization loop
learning_rate -- learning rate of the gradient descent update rule
print_cost -- True to print the loss every 100 steps
Returns:
params -- dictionary containing the weights w and bias b
grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
"""
costs = []
for i in range(num_iterations):
# cost /gradient calculation
grads, cost = propagate(w, b, X, Y)
#retrieve derivatives
dw = grads["dw"]
db = grads["db"]
# update values according to gradient descent algorithm
w = w - learning_rate * dw
b = b - learning_rate * db
# record the costs
if i % 100 == 0:
costs.append(cost)
# Print the cost every 100 training iterations
if print_cost:
print("Cost after iteration %i: %f" %(i, cost))
params = { "w": w,
"b": b}
grads = { "dw": dw,
"db": db}
return params, grads, costs
def predict(w, b, X):
'''
Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)
Arguments:
w -- weights, a numpy array of size (num_px * num_px * 3, 1)
b -- bias, a scalar
X -- data of size (num_px * num_px * 3, number of examples)
Returns:
Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
'''
Z = np.dot(w.T, X) + b
A = sigmoid(Z)
Y_prediction = (A >= 0.5).astype(int)
return Y_prediction
np.random.seed(1) # set a seed so that the results are consistent
X, Y = load_planar_dataset()
# Visualize the data:
plt.scatter(X[0, :], X[1, :], c=Y, s=40, cmap=plt.cm.Spectral); # s = size of points; cmap are nicer colours
plt.show()
shape_X = X.shape
shape_Y = Y.shape
m = shape_Y[1] # training set size
n = shape_X[0] # number of features (2)
# initialise parameters
w = np.random.rand(n, 1)
b = 0
# print accuracy of initial parameters by comparing prediction to
print("train accuracy: {} %".format(100 - np.mean(np.abs(predict(w, b, X) - Y)) * 100))
# fit model and print out costs every 100 iterations of the forward / back prop
parameters, grads, costs = optimize(w, b, X, Y, num_iterations = 10000, learning_rate = 0.000005, print_cost = True)
# return the prediction
Y_prediction = predict(parameters["w"], parameters["b"], X)
# print accuracy of fitted model
print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction - Y)) * 100))
# print parameters for interest
print( parameters["w"] , parameters["b"] )
# plot decision boundary
plot_decision_boundary(lambda x: predict(parameters["w"], parameters["b"], x.T), X, Y)
plt.show()
List item
