What are Recurrent Neural Networks?
Recurrent Neural Networks (RNNs) are a class of neural networks designed for processing sequence data (like text, time series, or DNA). Unlike Feedforward networks, RNNs have a "memory" enabling them to retain information from previous inputs in the sequence.
⚠️ The Vanishing Gradient Problem: Standard RNNs struggle to learn long-range dependencies because gradients diminish exponentially as they propagate back through time, causing the network to "forget" early inputs.
LSTMs and GRUs
To solve the short-term memory issue of standard RNNs, specialized architectures were developed:
Long Short-Term Memory (LSTM)
Introduced by Hochreiter & Schmidhuber (1997).
Uses a complex system of gates (Input, Forget, Output) and a cell state highway to regulate the flow of information, allowing it to remember important data over long sequences.
Gated Recurrent Unit (GRU)
Introduced by Cho et al. (2014).
A simplified version of LSTM with fewer parameters (Update and Reset gates), often achieving similar performance with faster training.
RNNs vs Transformers
| Feature | RNN / LSTM | Transformer |
|---|---|---|
| Processing | Sequential (Slow) | Parallel (Fast) |
| Long Context | Limited | Excellent (Attention) |
| Computations | O(n) - Linear | O(n²) - Quadratic |
Code Example (Keras/TensorFlow)
Creating a simple LSTM layer for text classification.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense
vocab_size = 10000
embedding_dim = 64
max_length = 100
model = Sequential([
Embedding(vocab_size, embedding_dim, input_length=max_length),
# LSTM layer with 128 units
LSTM(128, return_sequences=False),
Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()