I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard neural network as np.dot(x, W) and sometimes I see it as np.dot(W.T, x) and sometimes np.dot(W, x).

Take this image for example. They represent the input data as a matrix of [NxD] and weight data as [DxH] where H is the number of neurons in the hidden layer. This seems the most natural since input data will often be in tabular format with rows as samples and columns as features.


Now an example from the CS231n course notes. They talk about this below example and cite the code used to compute it as:

f = lambda x: 1.0/(1.0 + np.exp(-x)) # activation function (use sigmoid)
x = np.random.randn(3, 1) # random input vector of three numbers (3x1)
h1 = f(np.dot(W1, x) + b1) # calculate first hidden layer activations (4x1)
h2 = f(np.dot(W2, h1) + b2) # calculate second hidden layer activations (4x1)
out = np.dot(W3, h2) + b3 # output neuron (1x1)

Where W is [4x3] and x is [3x1]. I would expect the weight matrix to have dimensions equal to [n_features, n_hidden_neurons] but in this example it just seems like they transposed it naturally before it was used.


I guess I am just confused about general nomenclature in how data should be shaped and used consistently when computing neural network forward passes. Sometimes I see transpose, sometimes I don't. Is there a standard, preferred way to represent data in accordance to a diagram like these This question may be silly but I just wanted to discuss it a bit. Thank you.


I don't think there's a "standard way" of expressing the forward pass: you use the transpose when you need to use it, and this depends on how you define the weights and inputs matrices, and on the architecture of your neural network. For example, in a fully connected feedforward neural network, you know that every neuron in the previous layer is connected to every neuron in the current layer, so, as long as this is satisfied when you multiply the matrices, it does not matter whether you use transposes or not, and I don't think that, in computational terms, it makes any difference if you use transposes or not. (By the way, if you are writing something, I suggest that you always specify the dimensions of your matrices and your conventions).

Of course, if you want to use a library like TensorFlow, you will probably need to follow the conventions of the library, but this is another story.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.