I'm trying to create a convolutional neural network without frameworks (such as PyTorch, TensorFlow, Keras, and so on) with Python.

Here's a description of CNN taken from the Wikipedia article

In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery. They are also known as shift invariant or space invariant artificial neural networks (SIANN), based on their shared-weights architecture and translation invariance characteristics. They have applications in image and video recognition, recommender systems, image classification, medical image analysis, natural language processing, and financial time series.

A CNN has different types of layers, such as convolution, pooling (max or average), flatten and dense (or fully-connected) layers.

I have a few questions.

  1. Should we compute gradients (such as $\frac{\partial L}{\partial A_i}$,$\frac{\partial L}{\partial Z_i}$,$\frac{\partial L}{\partial A_{i-1}}$ and so on) in flatten layer or not?

  2. If no, then how should I compute $\frac{\partial L}{\partial A_i}$ and $\frac{\partial L}{\partial Z_i}$ of first layer of convolutional layer? With $\frac{\partial L}{[\frac{\partial g(A_i)}{\partial x}]}$ or with $\frac{\partial L}{\partial dA_{i+2}}$(P.S. as you know iteration of BackPropagation is reverse, so I used i+n for denote the previous layer)?

  3. Or can I compute derivatives in Flatten layer with $$\frac{\partial J}{\partial A} = W_{i+1}^T Z_{i+1}$$(i+1 denotes prev.layer in BackProp) $$\frac{\partial L}{\partial Z} = \frac{\partial L}{\partial A} *\frac{\partial g(A_i)}{\partial x} $$ and then reshape of Conv2D shape?

P.S. I found questions like mine (names are same), but there're not answer to my question as I asking about formula.

  • $\begingroup$ What are $A$ and $Z$ here? $\endgroup$
    – nbro
    Dec 25 '21 at 12:03

I have found that you should compute derivatives $\frac{\partial L}{\partial A}, \frac{\partial L}{\partial Z}$ in Flatten layer and then reshape Conv2D input shape.

  • 1
    $\begingroup$ Would you share an example of neural network model having Flatten --> Convolution? $\endgroup$
    – Cloud Cho
    Mar 1 '21 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.