1
$\begingroup$

I understand that the batch size is the number of examples you pass into the neural network (NN). If the batch size is 10, it means you feed the NN 10 examples at once.

Assuming I have an NN with a single Dense layer. This Dense layer of 20 units has an input shape (10, 3). This means that I am feeding the NN 10 examples at once, with every example being represented by 3 values. This Dense layer will have an output shape of (10, 20).

I understand that the 20 in the 2nd dimension comes from the number of units in the Dense layer. However, what does the 10 (Batch Size) in the first dimension mean? Does this mean that the NN learns 10 separate sets of weights (with each set of weights corresponding to one example, and one set of weights being a matrix of 60 values:3 features x 20 units)?

Improve this question
$\endgroup$
1
  • $\begingroup$ Note that, in general, implementation-related questions (including questions about software libraries) are off-topic here. Please, have a look at ai.stackexchange.com/help/on-topic for more info. $\endgroup$
    – nbro
    Commented Aug 29, 2020 at 1:00

1 Answer 1

Reset to default
1
$\begingroup$

The Dense layers outputs 20 values per example. And since you have 10 examples in the batch the output is (10, 20) (one set of 20 values per example in the batch). The nn doesn't learn 10 separate sets of weights. Each set of 20 values is computed with the same weight (and bias if you have any). So if say example 2 and 5 had the same input values, they'll always have the same output values.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes I understand where you are coming from. There will be 20 different values per example because its features (lets call them x1,x2 and x3) are different. Then, there will be 10 examples because of mini batch gradient descent - the average of the gradients of all 10 examples will be taken into account for the weights update. Is this correct? $\endgroup$
    – jaksnak
    Commented Aug 27, 2020 at 0:37
  • $\begingroup$ No. The gradients of the weights would be the sum (instead of average) of the gradients for each example. $\endgroup$
    – dbalchev
    Commented Aug 28, 2020 at 10:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .