I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard neural network as np.dot(x, W) and sometimes I see it as np.dot(W.T, x) and sometimes np.dot(W, x).
Take this image for example. They represent the input data as a matrix of [NxD] and weight data as [DxH] where H is the number of neurons in the hidden layer. This seems the most natural since input data will often be in tabular format with rows as samples and columns as features.
Now an example from the CS231n course notes. They talk about this below example and cite the code used to compute it as:
f = lambda x: 1.0/(1.0 + np.exp(-x)) # activation function (use sigmoid)
x = np.random.randn(3, 1) # random input vector of three numbers (3x1)
h1 = f(np.dot(W1, x) + b1) # calculate first hidden layer activations (4x1)
h2 = f(np.dot(W2, h1) + b2) # calculate second hidden layer activations (4x1)
out = np.dot(W3, h2) + b3 # output neuron (1x1)
Where W is [4x3] and x is [3x1]. I would expect the weight matrix to have dimensions equal to [n_features, n_hidden_neurons] but in this example it just seems like they transposed it naturally before it was used.
I guess I am just confused about general nomenclature in how data should be shaped and used consistently when computing neural network forward passes. Sometimes I see transpose, sometimes I don't. Is there a standard, preferred way to represent data in accordance to a diagram like these This question may be silly but I just wanted to discuss it a bit. Thank you.
