# About the Matrix Multiplication in Fully Connected Neural Networks

I've learned some machine learning networks, and I think they're not that interesting. Sometimes I feel like they're just messing around with formulas and creating something that doesn't make sense. And some tutorials only tell you how to do it, but not why it's designed that way.

This is especially true in neural networks and AI. It's frustrating. I've been trying to understand the relationship between fully connected neural networks and matrix multiplication, but I've been getting some confusing feelings that I couldn't quite put into words. But now I think I've found some problems, so I'd like to ask and see how you respond.

Below are some simple examples of matrix multiplication, which are similar to the so-called "neural networks" that simulate the human brain. There are some circles, and then the input layer.

The key point is the input layer. I think there's a problem here.

For example, let's say we have a 3x5 matrix X and a 5x10 matrix W. We can ignore the bias term b for now. W is the weight matrix that transforms the input X into the neural network. Let's look at how matrix multiplication simulates the neural network. For example, the first row of X is X00, X01,..., X04. According to matrix multiplication, this row needs to be multiplied element-wise with each column of W, and then summed to get a new row, which is Y00, Y01,..., Y09.

Let's look at the neural network corresponding to W. This layer has 5 neurons, because each row represents a neuron, and each neuron has 10 inputs, because there are 10 columns.

So when we multiply the first row of X with each column of W, we find that X00 is always multiplied with W00, W01,..., W09, which means that X00 is always input to the first neuron's all inputs.

Do you think this is correct?

Assuming X is an image, then X00 is just one pixel of the image. This pixel is always input to the first neuron's all inputs, but never to the second neuron, for example, W10, because one row is just one neuron.

So my question is, if X is an image, don't you think that X00 is important for the second, third, fourth neurons, and so on? Because of matrix multiplication, X00 can't be input to any other neurons in the first layer.

I see this and I think there's a problem.

If we manually rotate the image X by 180 degrees, then X00 will be input to W40, which is a different neuron.

If the network produces a very different output, then there's a problem, because rotating an image by 180 degrees shouldn't change its content.

So?

I've looked at the source code of LLaMA3 and other neural networks, and every time I see this, I get confused. Since there's uncertainty, why not just combine all possible situations? I think that's the only way to make sure that the network is correct. Instead of programmers writing code that they think should work, but only considering a limited perspective, and then training the AI.

You can't guarantee that your own code is correct, you can only try to include all possible situations and then rely on computational power, luck, and other factors.

So, regarding the above situation, my idea is that X00 should be input to all neurons' all inputs, not just some of them.

Am I correct?

• Can you put your specific question in the title? If you have multiple questions, create multiple posts
– nbro
Commented Aug 2 at 8:14