2
$\begingroup$

I am approaching the implementation of the OpenPose algorithm for realtime human body pose estimation.

According to the official paper OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, $L$ and $S$ fields (body part maps, and part affinity fields) are estimated. These have the same size as the input image, and, according to the paper, these fields should be outputted at a given step in the forward pass (after a given number of $L$ stages, and $S$ stages), but, since before entering these stages the image is passed through the initial layers of the VGG-19 model, the spatial dimension is encoded and the features that finally enter the $L$ and $S$ stages have other dimensionality.

All the network is convolutional, there's no FC layer at all. The VGG-19 part is the only one that contains MaxPooling layers, hence affecting the spatial relations and size of the receptive fields.

My point is, after stage execution, I get tensors of shape [batch_size, filter_number, 28, 28]. The issue is that the paper is not stating how to decode this information into the $L$ and $S$ maps of size $224 \times 224$.

Following a traditional approach and decoding the final tensors with a linear net from, let's say, $15000 \rightarrow (224 * 224 * \text{ number of body parts }) + (224 * 224 * \text{ number of limbs } * 2)$~A very huge number!, is out of question for any domestic computer, I presume I should have the least 128GbRAM installed, and is not the case.

Another solution is to remove the max-pooling layers from the VGG-19 part, but then although the map size is preserved to $224$, instead of $28$, the huge amount of computations and values that need to be stored also lead to memory errors.

So, the problem is, how can I get to a final output of $224 \times 224$ without FC layers, from a tensor of shape [batch_size, bodyparts, 28, 28]?

Not an easy answer. I will check a TensorFlow implementation I have seen around to see how the problem was solved.

Any parallel ideas are greatly welcome.

Improve this question
$\endgroup$
4
  • $\begingroup$ Sorry nbro, but that looked more to my english teacher corrections than a ML related answer. What was so necessary about your corrections?. I am not a native english speaker, but the overall meaning and sense of my question, is fully understandable I guess. $\endgroup$ – Chal.lo Mar 22 '20 at 11:12
  • $\begingroup$ BTW I think in other approaches the output of the S and L stages is upsampled to 224px to make it compatible with annotated images. So what I do is to downsample the annotations to the spatial dimensions of the output filters, (28,28) for the Loss function (MSE) to work. Now it seems to run correctly and there's no need to flatten or decode the results to compare results, which seems the right path to me. Any ideas? $\endgroup$ – Chal.lo Mar 22 '20 at 11:14
  • $\begingroup$ Please, ping me if you want me to see your messages, i.e. use @nbro. I didn't edit your post to offend you, but e.g. to improve the title so that it is more informative. I am sorry if you felt offended or upset. I was just trying to further improve your post. Unfortunately, I am not familiar with OpenPose, so I cannot help you now. I also didn't fully understand your description, probably because I am not familiar with OpenPose. Eventually, you could also ask this question at stats.stackexchange.com or datascience.stackexchange.com. $\endgroup$ – nbro♦ Mar 24 '20 at 0:25
  • $\begingroup$ I didn't feel offended at all. I am aware my english expressivity is not fully accurate. Need to increase my lr. :D... no worries I fully understand... $\endgroup$ – Chal.lo Mar 24 '20 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.