1
$\begingroup$

I'm trying to create a convolutional neural network without frameworks (such as PyTorch, TensorFlow, Keras, and so on) with Python.

Here's a description of CNN taken from the Wikipedia article

In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery. They are also known as shift invariant or space invariant artificial neural networks (SIANN), based on their shared-weights architecture and translation invariance characteristics. They have applications in image and video recognition, recommender systems, image classification, medical image analysis, natural language processing, and financial time series.

A CNN has different types of layers, such as convolution, pooling (max or average), flatten and dense (or fully-connected) layers.

I have a few questions.

  1. Should we compute gradients (such as $\frac{\partial L}{\partial A_i}$,$\frac{\partial L}{\partial Z_i}$,$\frac{\partial L}{\partial A_{i-1}}$ and so on) in flatten layer or not?

  2. If no, then how should I compute $\frac{\partial L}{\partial A_i}$ and $\frac{\partial L}{\partial Z_i}$ of first layer of convolutional layer? With $\frac{\partial L}{[\frac{\partial g(A_i)}{\partial x}]}$ or with $\frac{\partial L}{\partial dA_{i+2}}$(P.S. as you know iteration of BackPropagation is reverse, so I used i+n for denote the previous layer)?

  3. Or can I compute derivatives in Flatten layer with $$\frac{\partial J}{\partial A} = W_{i+1}^T Z_{i+1}$$(i+1 denotes prev.layer in BackProp) $$\frac{\partial L}{\partial Z} = \frac{\partial L}{\partial A} *\frac{\partial g(A_i)}{\partial x} $$ and then reshape of Conv2D shape?

P.S. I found questions like mine (names are same), but there're not answer to my question as I asking about formula.

$\endgroup$
1
  • $\begingroup$ What are $A$ and $Z$ here? $\endgroup$
    – nbro
    Dec 25, 2021 at 12:03

1 Answer 1

1
$\begingroup$

I have found that you should compute derivatives $\frac{\partial L}{\partial A}, \frac{\partial L}{\partial Z}$ in Flatten layer and then reshape Conv2D input shape.

$\endgroup$
1
  • 1
    $\begingroup$ Would you share an example of neural network model having Flatten --> Convolution? $\endgroup$
    – Cloud Cho
    Mar 1, 2021 at 22:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .