I am trying to solve a classification problem by implementing the Least Squares algorithm in Python. To solve this problem, I am implementing the linear algebra formula to train the classifier, which is $w = (X^TX)X^Ty$, where $w$ is the final weight vector of the classification function, $X$ is an input matrix of training data and $y$ a matrix of training labels. The classifier must be able to classify into three classes. As seen in the following snippet, during the preparation of the data, I gather my training data in matrix $X$, adding the number one at the end of each sample. I also gather my training labels in matrix $y$, coding each class as a sequence of -1 and 1.
X = np.matrix(np.zeros((len(train_set),4))) y = np.matrix(np.zeros((len(train_set),3))) for i, row in train_set.iterrows(): X[i] = [row, row, row, 1] if row == 'H': y[i] = [1, -1, -1] elif row == 'D': y[i] = [-1, 1, -1] else: y[i] = [-1, -1, 1]
What we have in matrix $y$ in the end, is a matrix that each column is a representation of each class and can tell us which samples belong to the class of the corresponding column. Having explained what each matrix in my program contains, my question is this, which of the following two implementations of the training process is correct and why?
At first, I went with the implementation seen in the snippet below.
Xtranspose = X.T dotProduct = Xtranspose.dot(X) inverse = np.linalg.pinv(dotProduct) A = inverse.dot(Xtranspose) w = A.dot(y) for i, row in test_set.iterrows(): r = np.matrix([row, row, row, 1]).dot(w)
As you can see, considering that $A = (X^TX)X^T$, I multiply $A$ with the label matrix $y$ and then I use the weight vector $w$ in the loop to test the classifier on some test data. Later, though, after some research on the internet, I found this second implementation, which actually has a higher success rate.
for i, row in test_set.iterrows(): r = np.zeros() j = 0 for column in y.T: w = A.dot(column.T) r[j] = np.matrix([row, row, row, 1]).dot(w) j += 1
Having calculated $A$ beforehand, I now calculate the weights for each column of $y$, for each class, separately. This second method, has a 10% greater success rate than the first one. So, why does the second training method have a better success rate? Is the second method the right training method?