In some part of my online ML course, I should build a code for plotting a learning curve, to be more specific, it's just a linear regression implementation.

So the first thing to consider is the expressions of training and Cross-validation error:

$$ J_{train} = \frac{1}{2.m_{train}}\sum_{i=1}^n (h_{\theta}(x^{(i)} - y^{(i)})^2 $$ $$ J_{CV} = \frac{1}{2.m_{CV}}\sum_{i=1}^n (h_{\theta}(x^{(i)} - y^{(i)})^2 $$

the instructor give for my the following instruction:

%Note: You should evaluate the training error on the first i training
%       examples (i.e., X(1:i, :) and y(1:i)).
%       For the cross-validation error, you should instead evaluate on
%       the _entire_ cross validation set (Xval and yval).
% Note: If you are using your cost function (linearRegCostFunction)
%       to compute the training and cross validation error, you should 
%       call the function with the lambda argument set to 0. 
%       Do note that you will still need to use lambda when running
%       the training to obtain the theta parameters.
% Hint: You can loop over the examples with the following:
%       for i = 1:m
%           % Compute train/cross validation errors using training examples 
%           % X(1:i, :) and y(1:i), storing the result in 
%           % error_train(i) and error_val(i)
%           ....
%       end

And then, in accordance what I understood, I build my MATLAB code:

function [error_train, error_val] = ...
    learningCurve(X, y, Xval, yval, lambda)
grad = 0;
theta = trainLinearReg(Xval,yval,lambda);
m = size(X,1);
h_theta = X*theta;

for i = 1:m
    error_train(i) = linearRegCostFunction(X(1:i,:), y(1:i,:), theta, 0);
    error_val(i) = linearRegCostFunction(Xval(1:i,:), yval(1:i,:), theta, 0);


And I got the following result as learning curves: enter image description here

I think that there's something wrong and I got some misunderstanding about this subject because second the instructor the learning curves should look like: enter image description here

Could someone help me understand what I did or understood wrongly?


function [J, grad] = linearRegCostFunction(X, y, theta, lambda)
J = 0;
grad = zeros(size(theta));

h_theta  = X*theta;

J = (1/(2*m))*sum((h_theta - y).^2,1) + (lambda/(2*m))*sum(theta.^2,1);

grad(1) = (1/m)*sum((h_theta - y)'*X(:,1));
grad(2) = (1/m)*(h_theta - y)'*X(:,2)  + sum((lambda/m).*theta);

Look a the CV description I just posted at SE CV. In the first sentence there is a link to Kohavi, which explains bootstrap bias, or estimating error as a function of increasing sample size -- which is what you want. There's also a link to my blog "Help my Neural Network Does Not Work."


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.