let's say I have a conv layer i with 64 feature maps and a filter size of 3x3. The previous conv layer i-1 has 32 feature map. Step-size is 2 and padding 1.
My question is now how to know how many neurons of the previous layer are connected to each of the neuron in i?
My first thought was 32 because there are 32 neurons/filters and I thought that number of feature maps = number of filters.
But I am not sure
Firstly, step size and padding have no influence on the number of parameters, they just determine where to apply your parameters, i.e. the convolution operator.
My first thought was 32 because there are 32 neurons/filters [...].
If you have 32 input channels and a kernel size of $3 \times 3$ then each entry of your $3\times3$-filter acts on a 32-dimensional vector. Therefore the number of parameters of your kernel is $3 \cdot 3 \cdot 32$. Or a more visual explanation: One convolutional filter squashes a $k \times k \times C^{in}$ cube of data into a single scalar, where each number in the cube has its own scalar in the kernel.
If you want 64 output channels, you have to have 64 of these kernels, so the convolution you describe should have $3 \cdot 3 \cdot 32 \cdot 64$ parameters.
As a generic formula: $\text{#Parameters}(\mathbf{W}) = k^h \cdot k^w \cdot C^{in} \cdot C^{out}$. Here $k^h$ and $k^w$ denote kernel height and kernel width respectively, and $C^{in}$ and $C^{out}$ denote the number of in- and output channels.
[...] and I thought that number of feature maps = number of filters
That's right, but each filter $w$ is not actually a 2-dimensional window, but a 3-dimensional window, i.e. a rank 3 tensor $\mathbf{W} \in \mathbb{R}^{k \times k \times C^{in}}$. You can see it in some visualizations of CNNs (s. image bloew). The input image has 3 channels and the first filter has 3-channels too and computes a single output pixel with only 1 channel. For further reading, this blog post seems to summarize convolution well.
Image taken from here
