Reinforcement Learning (RL)

Learning through interaction, trial, and error.

Overview

Reinforcement Learning (RL) is a type of machine learning where an Agent learns to make decisions by performing actions in an Environment and receiving Rewards (or penalties). unlike supervised learning, there are no "correct" labels provided upfront—the agent must discover the best strategy (policy) to maximize cumulative reward.

🤖

Agent

The learner/actor

🌍

Environment

The world it lives in

⚡

Action

What the agent does

💎

Reward

Feedback signal

RLHF (RL from Human Feedback)

This is the critical step that made ChatGPT and other modern LLMs helpful and safe. After pre-training (next-token prediction), models are fine-tuned using human preferences.

The RLHF Pipeline:

SFT (Supervised Fine-Tuning): Train the model on high-quality instruction-response pairs.
Reward Modeling: Train a separate model (Reward Model) to predict human preference scores (e.g., "Response A is better than B").
PPO (Proximal Policy Optimization): Use RL to optimize the LLM policy to maximize the score from the Reward Model, while not drifting too far from the original model.

Key Algorithms

PPO (Proximal Policy Optimization)

The standard for RLHF. Stable and efficient because it limits how much the policy can change in a single update.

DPO (Direct Preference Optimization)

A newer, simpler approach (2023) that optimizes the policy directly from preference data without needing a separate Reward Model or complex PPO loop. Replacing PPO in many pipelines.

RL Loop Concept

A simplified Python-like pseudocode of an RL training loop (typical in libraries like Gymnasium).

import gymnasium as gym

# 1. Create Environment
env = gym.make("CartPole-v1")
state, info = env.reset()

for step in range(1000):
    # 2. Agent chooses action (Policy)
    action = agent.predict(state)
    
    # 3. Environment executes action
    next_state, reward, terminated, truncated, info = env.step(action)
    
    # 4. Agent learns (Update weights based on reward)
    agent.learn(state, action, reward, next_state)
    
    state = next_state
    
    if terminated or truncated:
        state, info = env.reset()