Which of the following two implementations of a Least Squares classifier in Python is correct?

Question

I am trying to solve a classification problem by implementing the Least Squares algorithm in Python. To solve this problem, I am implementing the linear algebra formula to train the classifier, which is $w = (X^TX)X^Ty$, where $w$ is the final weight vector of the classification function, $X$ is an input matrix of training data and $y$ a matrix of training labels. The classifier must be able to classify into three classes. As seen in the following snippet, during the preparation of the data, I gather my training data in matrix $X$, adding the number one at the end of each sample. I also gather my training labels in matrix $y$, coding each class as a sequence of -1 and 1.

    X = np.matrix(np.zeros((len(train_set),4)))
    y = np.matrix(np.zeros((len(train_set),3)))

    for i, row in train_set.iterrows():
        X[i] = [row[1], row[2], row[3], 1]
        if row[0] == 'H':
            y[i] = [1, -1, -1]
        elif row[0] == 'D':
            y[i] = [-1, 1, -1]
        else:
            y[i] = [-1, -1, 1]

What we have in matrix $y$ in the end, is a matrix that each column is a representation of each class and can tell us which samples belong to the class of the corresponding column. Having explained what each matrix in my program contains, my question is this, which of the following two implementations of the training process is correct and why?

At first, I went with the implementation seen in the snippet below.

    Xtranspose = X.T
    dotProduct = Xtranspose.dot(X)
    inverse = np.linalg.pinv(dotProduct)
    A = inverse.dot(Xtranspose)
    w = A.dot(y)
    
    for i, row in test_set.iterrows():
        r = np.matrix([row[1], row[2], row[3], 1]).dot(w)

As you can see, considering that $A = (X^TX)X^T$, I multiply $A$ with the label matrix $y$ and then I use the weight vector $w$ in the loop to test the classifier on some test data. Later, though, after some research on the internet, I found this second implementation, which actually has a higher success rate.

    for i, row in test_set.iterrows():
        r = np.zeros([3])
        j = 0
        for column in y.T:
            w = A.dot(column.T)
            r[j] = np.matrix([row[1], row[2], row[3], 1]).dot(w)
            j += 1

Having calculated $A$ beforehand, I now calculate the weights for each column of $y$, for each class, separately. This second method, has a 10% greater success rate than the first one. So, why does the second training method have a better success rate? Is the second method the right training method?

Answer 1

The first implementation is better. In the second one, you calculate the weights based on information about a single class only, the $w$ inside the loop will be primed to find the classes. This is equivalent to training three models each trying to predict confidence in one class only. The first one would be able to do it for three classes.

A couple of extra points:

• It's probably better to use np.array instead of np.matrix (Checkout the note here https://numpy.org/doc/stable/reference/generated/numpy.matrix.html)
• For classification, it's more common to use cross entropy as the loss function, instead of the L2 loss. The normal equations work with the L2 loss.
• I can't see that from the example, but it's worth normalising the features.