I am trying to figure out how multiprocessing works in neural networks.

In the example I've seen, the database is split into $x$ parts (depending on how many workers you have) and each worker is responsible to train the network using a different part of the database. I am confused regarding the optimization part:

let's say worker 1 finished first to calculate the gradient, now it will update the network accordingly

Then worker 2 finished the calculation and it will also attempt to update the weights, however the gradient it calculated was for the network before it was updated by the first worker. Now the second worker will attempt to update the network with a bad gradient.

Did I miss something?

I am trying to figure out how multiprocessing works in neural networks.

In the example I've seen, the database is split into $x$ parts (depending on how many workers you have) and each worker is responsible to train the network using a different part of the database.

I am confused regarding the optimization part:

Let's say worker 1 finished first to calculate the gradient, now it will update the network accordingly.

Then worker 2 finished the calculation and it will also attempt to update the weights. However, the gradient it calculated was for the network before it was updated by the first worker. Now, the second worker will attempt to update the network with a bad gradient.

Did I miss something?

I am trying to figure out how the multiprocessing works in neural network.

In the example I've seen, the database is split to x parts (depending how many workers you have) and each worker is responsible to train the network using a different part of the database. I am confused regarding the optimization part:

lets say the worker 1 finished first to calculate the gradient, now it will update the network accordingly

Then worker 2 finished the calculation and it will also attempt to update the weights, however the gradient it calculated was for the network before it was updated by the first worker. Now the second worker will attempt to update the network with bad gradient.

Did I miss something? thanks

I am trying to figure out how multiprocessing works in neural networks.

In the example I've seen, the database is split into $x$ parts (depending on how many workers you have) and each worker is responsible to train the network using a different part of the database. I am confused regarding the optimization part:

let's say worker 1 finished first to calculate the gradient, now it will update the network accordingly

Then worker 2 finished the calculation and it will also attempt to update the weights, however the gradient it calculated was for the network before it was updated by the first worker. Now the second worker will attempt to update the network with a bad gradient.

Did I miss something?