GenAIHub
EN | PT
← Back to Technical Section

Neural Networks

The Foundation of Modern AI and Deep Learning

What are Neural Networks?

Neural Networks are computational models inspired by the human brain. They consist of interconnected nodes (neurons) organized in layers that learn to recognize patterns in data through training. Neural networks form the backbone of modern AI systems, from image recognition to large language models.

πŸ’‘ Key Insight: A neural network learns by adjusting the weights of connections between neurons through a process called backpropagation, gradually improving its ability to make accurate predictions.

🧠

Deep Learning

Many layers

πŸ–ΌοΈ

CNNs

Images

πŸ“

RNNs

Sequences

⚑

Transformers

Attention

Basic Architecture

Input Layer x₁ xβ‚‚ x₃ Hidden Layer 1 Hidden Layer 2 Output Layer y₁ yβ‚‚ Each connection has a learnable weight (w)

Input Layer

Receives raw data (pixels, text embeddings, numbers). Each node represents one feature.

Hidden Layers

Learn intermediate representations. More layers = deeper network = more complex patterns.

Output Layer

Produces predictions (classification, regression, generation). Size depends on task.

Key Concepts

Activation Functions

Non-linear functions (ReLU, Sigmoid, Tanh) that enable networks to learn complex patterns.

Backpropagation

Algorithm that calculates gradients and updates weights by propagating error backwards.

Loss Function

Measures how wrong predictions are. Training minimizes this function (MSE, Cross-Entropy).

Gradient Descent

Optimization algorithm that adjusts weights in the direction that reduces loss.

Types of Neural Networks

Type Best For Examples
Feedforward (MLP) Tabular data, classification Fraud detection, pricing
CNN Images, spatial patterns ResNet, VGG, EfficientNet
RNN / LSTM Sequential data, time series Speech recognition
Transformer Language, attention-based GPT, BERT, T5
GAN Generation, adversarial StyleGAN, DALL-E

Activation Functions (Deep Dive)

Activation functions introduce non-linearity into neural networks, allowing them to learn complex patterns. Without activation functions, a neural network would be equivalent to a single linear transformation.

πŸ’‘ Why Non-Linearity Matters: Real-world data has complex, non-linear relationships. Linear functions can only draw straight lines/planes to separate data, but activation functions enable networks to learn curved, intricate decision boundaries.

R

ReLU (Rectified Linear Unit)

The most widely used activation function in deep learning. Simple yet effective.

Formula:

f(x) = max(0, x)
  • βœ… Computationally efficient
  • βœ… Reduces vanishing gradient problem
  • βœ… Sparse activation (neurons can be "off")
  • ⚠️ "Dying ReLU" problem (neurons stuck at 0)
x y
Οƒ

Sigmoid

Squashes any input to a value between 0 and 1. Historically important, now mainly used for binary outputs.

Formula:

Οƒ(x) = 1 / (1 + e⁻ˣ)
  • βœ… Output interpretable as probability
  • βœ… Smooth gradient
  • ⚠️ Vanishing gradients for extreme values
  • ⚠️ Not zero-centered
x 1 0
th

Tanh (Hyperbolic Tangent)

Similar to Sigmoid but outputs values between -1 and 1. Zero-centered, which helps training.

Formula:

tanh(x) = (eˣ - e⁻ˣ) / (eˣ + e⁻ˣ)
  • βœ… Zero-centered output
  • βœ… Stronger gradients than Sigmoid
  • βœ… Good for RNNs/LSTMs
  • ⚠️ Still has vanishing gradient issue
x +1 -1
S

Softmax

Converts a vector of raw scores into a probability distribution. Essential for multi-class classification.

Formula:

softmax(xᡒ) = eˣⁱ / Σⱼ eˣʲ
  • βœ… Outputs sum to 1 (probability)
  • βœ… Differentiable for backprop
  • βœ… Standard for classification output
  • 🎯 Used with Cross-Entropy loss

Example: [2.0, 1.0, 0.1]

0.659

Cat

0.242

Dog

0.099

Bird

Sum = 1.0 βœ“

# PyTorch activation functions

import torch
import torch.nn.functional as F

x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])

relu_out = F.relu(x)           # [0, 0, 0, 1, 2]
sigmoid_out = torch.sigmoid(x) # [0.12, 0.27, 0.5, 0.73, 0.88]
tanh_out = torch.tanh(x)       # [-0.96, -0.76, 0, 0.76, 0.96]

# Softmax on a batch of logits
logits = torch.tensor([2.0, 1.0, 0.1])
probs = F.softmax(logits, dim=0)  # [0.659, 0.242, 0.099]

Training Process (Deep Dive)

Training a neural network is an iterative optimization process. The network makes predictions, measures errors, and adjusts its parameters to improve over time.

1. Forward Pass Input β†’ Predictions Ε· = f(x, W) 2. Loss Calculation Error Measurement L = loss(Ε·, y) 3. Backward Pass Compute Gradients βˆ‚L/βˆ‚W (chain rule) 4. Update Weights Gradient Descent W = W - Ξ·Β·βˆ‡L 5. Repeat Until Convergence epochs Γ— batches

1️⃣ Forward Pass

Data flows from input through each layer to produce an output prediction. 

# For each layer:
z = W Β· x + b          # Linear transformation
a = activation(z)      # Apply non-linearity
x = a                  # Output becomes next input

# Example with 2 layers:
z1 = W1 Β· input + b1   β†’  a1 = ReLU(z1)
z2 = W2 Β· a1 + b2      β†’  output = softmax(z2)

2️⃣ Loss Calculation

Quantifies how wrong the prediction is compared to the true label.

MSE (Regression)

L = (1/n) Ξ£(y - Ε·)Β²

Cross-Entropy (Classification)

L = -Ξ£ yα΅’ log(Ε·α΅’)

3️⃣ Backward Pass (Backpropagation)

Uses the chain rule of calculus to compute gradients of the loss with respect to each weight. 

# Chain rule propagates error backwards:
βˆ‚L/βˆ‚W2 = βˆ‚L/βˆ‚output Β· βˆ‚output/βˆ‚z2 Β· βˆ‚z2/βˆ‚W2
βˆ‚L/βˆ‚W1 = βˆ‚L/βˆ‚output Β· βˆ‚output/βˆ‚a1 Β· βˆ‚a1/βˆ‚z1 Β· βˆ‚z1/βˆ‚W1

# Each layer receives gradient from next layer
# and passes modified gradient to previous layer

4️⃣ Update Weights (Gradient Descent)

Adjust weights in the direction that reduces the loss. The learning rate (Ξ·) controls step size.

W_new = W_old - Ξ· Β· βˆ‚L/βˆ‚W

SGD

Basic gradient descent

Adam

Adaptive learning rate

AdamW

Adam + weight decay

# Complete PyTorch training loop

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Sequential(
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

for epoch in range(num_epochs):
    for inputs, labels in dataloader:
        # 1. Forward pass
        outputs = model(inputs)
        
        # 2. Calculate loss
        loss = criterion(outputs, labels)
        
        # 3. Backward pass
        optimizer.zero_grad()  # Clear previous gradients
        loss.backward()        # Compute gradients
        
        # 4. Update weights
        optimizer.step()       # Apply gradients
    
    print(f'Epoch {epoch+1}, Loss: {loss.item():.4f}')

Popular Frameworks

Related Topics

Transformers Pretraining Embeddings Vision Models

Test Your Knowledge

Score 8/10 or higher to pass