Statistical Nature of the Learning Process in Neural Networks

Understanding the statistical nature of the learning process in neural networks (NNs) is pivotal for optimizing their performance. This article aims to provide a comprehensive understanding of the statistical nature of the learning process in NNs. It will delve into the concepts of bias and variance, the bias-variance trade-off, and how these factors influence the performance of NNs. By the end, readers will have a deeper understanding of how to optimize NNs for better performance.

Neural networks are computational models inspired by the human brain. They consist of layers of interconnected nodes (neurons), where each connection (synapse) has an associated weight. Through training, NNs can learn complex patterns from data, making them powerful tools for classification, regression, and pattern recognition tasks.

Importance of Understanding the Learning Process

Understanding the learning process in NNs is essential for:

1. Improving Performance: Optimizing parameters and architectures to enhance accuracy and efficiency.
2. Diagnosing Issues: Identifying and addressing problems such as overfitting and underfitting.
3. Ensuring Robustness: Making NNs more reliable and generalizable across different datasets and tasks.

This analysis focuses on the deviation between a target function f(x) and the actual function F(x,w) derived by the NN, where x denotes the input signal. By examining this deviation, we can gain insights into the effectiveness of the NN and identify areas for improvement.

Environment Setting

Let’s consider a scenario with N realization of a random vector X, denoted by: [Tex]\{ x_i \}_{i=1}^{N}[/Tex] and corresponding set of random scalar D denoted by: [Tex]\{ d_i \}_{i=1}^N[/Tex].

These measurements constitute the training sample, that is denoted by:

[Tex]\mathcal{T} = \{ (x_i, d_i) \}_{i=1}^N[/Tex]

We assume the regressive model:

[Tex]D = f(X) +\epsilon[/Tex]

Where

• f(X) is the deterministic function of its argument vector
• [Tex]\epsilon [/Tex] is the random expectational error

Properties of Regressive Model

Regressive model has two properties:

1. Zero Mean Error

• The mean value of expectational error is zero: [Tex]\mathbb{E}[\epsilon \mid x] = 0[/Tex] where, E is statistical expectation operator.
• Based on this property, we can say regression function f(x) is the conditional mean of model output D, given that input X=x shown by: [Tex]f(x) = \mathbb{E}[D \mid x][/Tex]

2. Orthogonality Principle

• The expectational error is uncorrelated with regression function f(X): [Tex]\mathbb{E}[\epsilon f(X)] = 0[/Tex]
• This property is called principle of orthogonality which states that all the information about D available to us through input X has been encoded into regression function f(X).
• [Tex]E[\epsilon f(X)] = E[ E[ \epsilon f(X) \mid x ]][/Tex] [Tex]=[/Tex][Tex] E[ f(X) E[\epsilon \mid x ]][/Tex] = [Tex]E[f(X) \cdot 0 ][/Tex] = 0

Neural Network Response

The actual response Y of the NN to input X is:

[Tex]Y=F(X,w) [/Tex]

where [Tex]F(X,w)[/Tex] is the NN’s input-output function.

Cost function

The cost function [Tex]ε(w)[/Tex] is the squared difference between the desired response ?d and the actual response y of the NN, averaged over the training data [Tex]\mathcal{T}[/Tex]:

[Tex]\varepsilon(w) = \frac{1}{2} \mathbb{E}_{\mathcal{T}} \left[ (d – F(x, \mathcal{T}))^2 \right][/Tex]

[Tex]\mathbb{E}_{\mathcal{T}}[/Tex] is average operator taken over training sample [Tex]\mathcal{T}[/Tex].

By adding and subtracting f(x) to the argument [Tex](d – F(x,\mathcal{T}))[/Tex] , we can write it as:

[Tex]d – F(x, \mathcal{T}) = (d – f(x)) + (f(x) – F(x, \mathcal{T})) = \epsilon + (f(x) – F(x, \mathcal{T}))[/Tex]

Substituting this equation in cost function:

[Tex]\varepsilon(w) = \frac{1}{2} \mathbb{E}_{\mathcal{T}}[\epsilon^2] + \frac{1}{2} \mathbb{E}_{\mathcal{T}}[(f(x) – F(x,\mathcal{T}))^2] + \mathbb{E}_{\mathcal{T}}[\epsilon(f(x) – F(x,\mathcal{T}))][/Tex]

The last expectation term on the right hand side is zero for two reasons:

So, the equation is reduced to:

[Tex]\varepsilon(w) = \frac{1}{2} \mathbb{E}_{\mathcal{T}}[\epsilon^2] + \frac{1}{2} \mathbb{E}_{\mathcal{T}}[(f(x) – F(x,\mathcal{T}))^2][/Tex]

where, [Tex]\frac{1}{2} \mathbb{E}_{\mathcal{T}}[\epsilon^2][/Tex] represents intrinsic error because it is independent of weight vector w.

Bias-Variance Trade-Off

The natural measure of effectiveness of [Tex]F(x,w)[/Tex] as a predictor of desired response d is defined by:

[Tex]\mathcal{L}_{\text{av}}(f(x), F(x,w)) = \mathbb{E}_{\mathcal{T}}[(f(x) – F(x,\mathcal{T}))^2][/Tex]

This result provides mathematical basis for trade off b/w the bias and variance resulting from the use of [Tex]F(x,w)[/Tex].

Average value of estimation error b/w regression function [Tex]f(x) = \mathbb{E}[D \mid X=x][/Tex] and approximation function [Tex]F(x,w)[/Tex] is:

[Tex]\mathcal{L}_{\text{av}}(f(x), F(x,w)) = \mathbb{E}_{\mathcal{T}} \left[ \left( \mathbb{E}[D \mid X=x] – F(x,\mathcal{T}) \right)^2 \right] [/Tex]has a constant expectation with respect to the training data sample

The expectational error [Tex]\epsilon[/Tex] is uncorrelated with the regression function f(x).

The expectational error [Tex]\epsilon[/Tex] pertains to the regressive model whereas approximating function [Tex]F(x,w) [/Tex]pertains to the neural network model.

Next we find that:

[Tex]\mathbb{E}[D \mid X=x] – F(x,\mathcal{T}) = (\mathbb{E}[D \mid X=x] – \mathbb{E}_{\mathcal{T}}[F(x,\mathcal{T})]) + (\mathbb{E}_{\mathcal{T}}[F(x,\mathcal{T})] – F(x,\mathcal{T}))[/Tex]

where we simply added and subtracted the average [Tex]\mathbb{E}_{\mathcal{T}}[F(x,\mathcal{T})][/Tex]

[Tex]\mathcal{L}_{\text{av}}(f(x), F(x,w)) = B^2(w) + V(w) [/Tex]

where,

• [Tex]B(w) = \mathbb{E}_{\mathcal{T}}[F(x, \mathcal{T})] – \mathbb{E}[D \mid X=x][/Tex]
• [Tex]V(w) = \mathbb{E}_{\mathcal{T}}\left[(F(x,\mathcal{T}) – \mathbb{E}_{\mathcal{T}}[F(x,\mathcal{T})])^2\right][/Tex]

Observations:

1. The term B(w) is the bias of the average value of the approximating function [Tex]F(x, \mathcal{T})[/Tex], measured with respect to the regression function f[Tex](x) = \mathbb{E}[D \mid X=x][/Tex]. This term represents the inability of the neural network defined by the function [Tex]F(x,w)[/Tex] to accurately approximate the regression function [Tex]f(x) = \mathbb{E}[D \mid X=x][/Tex]. We may therefore view the bias B(w) as an approximation error.
2. The term V(w) is the variance of the approximating function [Tex]F(x,w)[/Tex], measured over the entire training sample [Tex]\mathcal{T}[/Tex]. This second term represents the inadequacy of the information contained in the training sample [Tex]\mathcal{T}[/Tex] about the regression function f(x). We may therefore view the variance V(w) as the manifestation of an estimation error.

To achieve good overall performance, bias B(w) and variance V(w) of approximating function [Tex]F(x,w) = F(x,\mathcal{T})[/Tex] would both have to be small. In neural networks, achieving a small bias leads to a large variance. However, if we have an infinitely large training sample for a single neural network, we can reduce both bias and variance. This leads to bias/variance dilemma, resulting in very slow convergence.

To address this dilemma, we can intentionally introduce bias into the network, which allows us to reduce or eliminate variance. It’s important to ensure that this bias is harmless and only contributes to mean-square error if we try to infer regressions outside the expected class. Bias needs to be designed for each specific application. To achieve this, we use a constrained network architecture, which performs better than a general-purpose architecture.

Understanding the statistical nature of the learning process in neural networks is essential for optimizing their performance. By analyzing the bias-variance trade-off, we can design better network architectures and training strategies. This balance is key to developing robust and efficient neural networks.