How to calculate the number of parameters in CNN?

Calculating the number of parameters in Convolutional Neural Networks (CNNs) is important for understanding the model complexity, computational requirements, and potential overfitting.

Parameters in CNNs are primarily the weights and biases learned during training. This article will walk you through calculating these parameters in various layers of a CNN.

To calculate the total number of parameters in a 2D convolutional neural network, sum the parameters from all layers, including convolutional, fully connected, and batch normalization layers, while excluding pooling layers as they contribute zero parameters.

1. Convolutional Layer

For a convolutional layer, the number of parameters is determined by the size of the filters (kernels), the number of filters, and the number of input channels.

We can calculate the parameter in the following manner:

Parameters=[Tex](k_w \times k_h \times C_{in} +1)\times C_{out}[/Tex]

Where:

• [Tex]k_w[/Tex] = kernel width
• [Tex]k_h[/Tex] = kernel height
• [Tex]C_{in}[/Tex] = number of input channels
• [Tex]C_{out}[/Tex]= number of filters (output channels)

The “+ 1” accounts for the bias term for each filter.

Let’s consider an example, where the convolutional layer with 32 filters of size 3×3 is given, and has an input with 3 channels:

[Tex] Parameters=(3×3×3+1)×32 =(27+1)×32=28×32 =896[/Tex]

2. Fully Connected (Dense) Layers

For a fully connected layer, the number of parameters is given by the number of input units times the number of output units, plus one bias term for each output unit.

[Tex]Parameters=(\text{input units} × \text{output units})+ \text{output units}[/Tex]

Let’s take an example, where the fully connected layer has 128 input units and 64 output units the computed parameters are:

[Tex]Parameters=(128×64)+64=8192+64=8256 [/Tex]

3. Batch Normalization Layers

For batch normalization layers, each feature channel has two parameters (gamma and beta).

[Tex]Parameters=2×\text{num features}[/Tex]

Let’s suppose that batch normalization layer is applied to an output of 64 channels then parameters can be computed as:

[Tex]Parameters = 2 \times 64 =128 [/Tex]

4. Pooling Layers

Pooling layers (e.g., max pooling, average pooling) do not have learnable parameters, so they contribute 0 to the parameter count.

5. Combining All Layers

To find the total number of parameters in the CNN, sum the parameters from all the layers calculated above.

Consider a simple CNN with the following layers:

1. Conv layer: 16 filters, 3×3 size, 3 input channels
2. Conv layer: 32 filters, 3×3 size, 16 input channels
3. Fully connected layer: 128 input units, 64 output units
4. Batch normalization after each convolutional layer

Conv Layer 1:

[Tex]\text{Parameters}=(3×3×3+1)×16=(27+1)×16=448[/Tex]

Batch Norm 1:

[Tex]\text{Parameters}=2×16=32[/Tex]

Conv Layer 2:

[Tex]\text{Parameters}=(3×3×16+1)×32=(144+1)×32=4640[/Tex]

Batch Norm 2:

[Tex]\text{Parameters}=2×32=64[/Tex]

Fully Connected Layer:

[Tex]\text{Parameters}=(128×64)+64=8256[/Tex]

Total Parameters:

[Tex]\text{Total Parameters} = 448 + 32 + 4640 +64 + 8256 = 13440 [/Tex]

So, the total number of parameters in this simple CNN example is 13,440.

For a 3D convolutional layer, the number of parameters depends on the size of the filters (kernels), the number of filters, and the number of input channels.

[Tex]\text{Parameters} = (k_d \times k_h \times k_w \times C_{in} + 1) \times C_{out}[/Tex]

Where:

• [Tex]k_d[/Tex]= kernel depth (size along the depth dimension)
• [Tex]k_h[/Tex] = kernel height (size along the height dimension)
• [Tex]k_w[/Tex]​ = kernel width (size along the width dimension)
• [Tex]C_{in}[/Tex] = number of input channels
• [Tex]C_{out}[/Tex] = number of filters (output channels)

The “+ 1” accounts for the bias term for each filter.

Let’s consider an example for 3D Convolution Neural Network where we will compute the parameters for:

1. Conv layer: 16 filters, size 3x3x3, 3 input channels
2. Conv layer: 32 filters, size 3x3x3, 16 input channels
3. Fully connected layer: 128 input units, 64 output units
4. Batch normalization after each convolutional layer

Conv Layer 1:

[Tex]\text{Parameters}=(3×3×3×3+1)×16=82×16=1312[/Tex]

Batch Norm 1:

[Tex]\text{Parameters}=2×16=32[/Tex]

Conv Layer 2:

[Tex]\text{Parameters}=(3×3×3×16+1)×32=(432+1)×32=433×32=13856 [/Tex]

Batch Norm 2:

[Tex]\text{Parameters}=2×32=64[/Tex]

Fully Connected Layer:

[Tex]\text{Parameters}=(128×64)+64=8256[/Tex]

Total Parameters:

[Tex]\text{Total Parameters} = 1312+32+13856+64+8256=23520[/Tex]

So, the total number of parameters in this simple 3D CNN example is 23,520.

1. Filter size: Larger filters have more parameters.
2. Number of filters: More filters increase the parameters.
3. Stride and padding: These do not affect the number of parameters but affect the output size.
4. Number of layers: More layers increase the total parameters.
5. Input size: Larger input sizes can lead to larger fully connected layers if the output of the final convolutional layer is large.

Calculating the number of parameters in CNNs is fundamental for designing and understanding neural network models. By systematically calculating the parameters for each layer, one can evaluate the complexity and feasibility of the model for a given application.