0
$\begingroup$

I understand that the batch size is the number of examples you pass into the neural network (NN). If the batch size is 10, it means you feed the NN 10 examples at once.

Assuming I have an NN with a single Dense layer. This Dense layer of 20 units has an input shape (10, 3). This means that I am feeding the NN 10 examples at once, with every example being represented by 3 values. This Dense layer will have an output shape of (10, 20).

I understand that the 20 in the 2nd dimension comes from the number of units in the Dense layer. However, what does the 10 (Batch Size) in the first dimension mean? Does this mean that the NN learns 10 separate sets of weights (with each set of weights corresponding to one example, and one set of weights being a matrix of 60 values:3 features x 20 units)?

$\endgroup$
  • $\begingroup$ Note that, in general, implementation-related questions (including questions about software libraries) are off-topic here. Please, have a look at ai.stackexchange.com/help/on-topic for more info. $\endgroup$ – nbro Aug 29 at 1:00
0
$\begingroup$

The Dense layers outputs 20 values per example. And since you have 10 examples in the batch the output is (10, 20) (one set of 20 values per example in the batch). The nn doesn't learn 10 separate sets of weights. Each set of 20 values is computed with the same weight (and bias if you have any). So if say example 2 and 5 had the same input values, they'll always have the same output values.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Yes I understand where you are coming from. There will be 20 different values per example because its features (lets call them x1,x2 and x3) are different. Then, there will be 10 examples because of mini batch gradient descent - the average of the gradients of all 10 examples will be taken into account for the weights update. Is this correct? $\endgroup$ – jaksnak Aug 27 at 0:37
  • $\begingroup$ No. The gradients of the weights would be the sum (instead of average) of the gradients for each example. $\endgroup$ – dbalchev Aug 28 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.